Maintaining the url tree structure for ZAP tool

When we use zap tool one thing that we have to do is maintain the url tree structure. Normally when we run the zap tool[1]  we perform all the UI actions manually so zap tool can trace them. Zap tool will use these urls when the tool starting to attack. So if we want more accurate result with the required strength, the url tree structure should be maintained properly. Other wise there is a chance that zap tool will miss some urls where there are actual threats that could harm the application.

But the system i am developing is a cloud computing solution. So there is no one  to do the UI actions manually. But some how this should be done. I have came up with some ideas that will make this possible.

Request for a selenium script.

When we develop web applications most of us use selenium to automate web applications . Even though selenium is used as a testing automation tool we could use that same tool to over come this issue. As mentioned in my other posts i am going to create a front end web application. There i can ask for a selenium script as well. Once i pull the source code and deploy it in my sever i can run the selenium script and create the url tree.  Also i can use same selenium script to verify whether  the deployment is successfully done or not as well.

Creating a crawler.

while i search for this problem i found a tool called linkchecker[2]. That is a application written in python to find broken URLs. But  it is capable of saving all the URLs with their parent URL  within a given depth.

some Features of  linkchecker.

• recursive and multi threaded checking and site crawling
• restriction of link checking with regular expression filters for URLs
• proxy support
• honors robots.txt exclusion protocol
• Cookie support
• Different interfaces: command line, GUI and web interface

 I was able to generate a XML report of a small web application.

With the XML file and by asking values for the forms in the front end application i will create a script that will perform the UI actions. With that script i can create the .session file and run the zap scan.


Request for .session file

There will be some developers who have knowledge about zap and know how to perform the security scan by them self. But they won't do the security scan by them self because of the time and resource problems. If the situation is like that we can ask them to do the UI actions manually (They will most likely do this because they want more accurate results.) when the zap tool is running. Once zap tool finishes tracing the UI actions end users can upload the .session file as well. we can use that .session file in our servers to run the ZAP tool. But everyone will not be able to that.

  File -> Session properties

[1]https://pavithraprbd.blogspot.com/2017/03/owasp-zap-zed-attack-proxy.html 

[2]https://wummel.github.io/linkchecker

OWASP ZAP (Zed Attack Proxy)

OWASP  ZAP aka Zed Attack Proxy is one of the world’s most popular free security tools and it can find security vulnerabilities in web applications. Its also a great tool for experienced pen testers to use for manual security testing. OWASP ZAP is a completely free and open source software. This is a cross-platform tool which is written in Java and is available in all of the popular operating systems including Microsoft Windows, Linux and Mac OS.

ZAP provides automated scanners as well as a set of tools that allow you to find security vulnerabilities manually. Some of the built in features include: Intercepting proxy server, Traditional and AJAX Web crawlers, Automated scanner, Passive scanner, Forced browsing, Fuzzer, WebSocket support, Scripting languages, and Plug-n-Hack support. It has a plugin-based architecture and an online ‘marketplace’ which allows new or updated features to be added[2].



In this post I am going to show you how to install and run OWASP  ZAP tool and do a basic attack in GUI  mode. (there are  many other options and configurations are available in zap tool. This is just to show the basic options and configurations of zap tool.)




First you need to download zap tool from here.

Once it finishes downloading you can extract it. To do that go to your download place and execute this command.

tar -zxvf ZAP_2.5.0_Linux.tar.gz

When it finishes you can go to that folder and run the zap.sh file

Once you accept to their terms and conditions the tool will start. This is the interface you get when you run zap tool.

Here you can select the first option and click on start. If you want to run a quick test on an application you could just simply select the url and click on attack button. But in order to get best results we need to configure the tool a bit.

First what you need to do is increase the heap size for JVM to run the zap tool. To do that open the zap.sh file in a text editor and change the value Xmx512m to Xmx4096m.If you system memory is less than 1500mb instead of changing Xmx512m you need to change Xmx256m or Xmx128m depending on your memory.

Once it done you can use a policy to fine tune the zap tool [3]. To import that to your zap tool go to analyze -> Scan Policy Manager and you can import the test policy you downloaded from the opened window.

Then go to tool->option and goto local proxy and change the address and the port since i am using port 9090 for my tomcat server here i am adding 8080 if you are using port 8080 for tomcat or some other application you will have to add a another port.

 Then open your Firefox browser and goto preferences -> advanced -> network and click on settings

Here select the option manual proxy configuration change the HTTP proxy and port as you configured in the zap tool .(if the localhost or 127.0.0.1 or host name is in the no proxy for field remove them also, other wise zap tool will not trace them.)

once done open the application you want run the scan in the Firefox browser.it will be identified by the zap tool.

Then we have to include the site to the context to do that right click on the site and select include in context -> default context

keep the default configurations and click ok. Then goto session properties to mange sessions of the application and click on authentication and change the selected authentication method to form based authentication. 

Here in this form insert the URL for the login page of the site to zap to identity the login page and then enter names of the user name field  and password filed and variable holders for them. Then from the drop down box in username parameter select the user name field and in password parameter select password field. Here in this case id is my username.

id={%username%}&password={%password%}

Regex pattern identified in logged in response message is to zap to identify if  the logging is successful. Here we can add anything in regex format but it should make sure that the user is logged in successfully. I am adding the bellow regex since logout URL  appear only when a user logged in.

  \Q<a href="logout.jsp">logout</a>\E

Regex pattern identified in logged out response message is to zap to identify if  the logout is successful

  \Q<a href="adminlogin.jsp">login</a>\E

Instead of filling the form you can do this automatically also by logging into the application manually. Zap tool will trace the login and then you can flag that action as  form based authentication as well

 Then we need to add  users. To do that select users tab and click on add and give the user name and the passwords and click on add button.we can define multiple users as well.

After everything configured we can perform all the actions manually so zap will trace these actions and will use the when spider the site. To do a spider you can right click on the site and click  attack -> spider and then click start scan.

If you want to do a active scan, same as earlier you can right click on  select attack -> active scan.

Here if you wants to add the test policy we downloaded[3] you can click on show advanced options then goto policy and select the policy from the drop down menu. Then you can click on start scan. It will take lot of time to finish the scan.

Once the scan finishes you can generate reports by clicking Report -> Generate HTML Report or XML report

report output

Reference

[1]https://medium.com/@PrakhashS/dynamic-scanning-with-owasp-zap-for-identifying-security-threats-complete-guide-52b3643eee04#.6damz0ert

[2]https://en.wikipedia.org/wiki/OWASP_ZAP

[3] https://gist.github.com/anonymous/240783a9dd4aa963a6b5101ac17b0bdf

My solution

Main objective of this project is to provide a cloud based service for developers to perform security scans for their web applications without having a prior knowledge or prior experience in using and configuring the security testing tools.As mentioned in the WSO2 GSoC Project Proposals there will be mainly 2 steps in this project.

In the first step I am going to create per configured docker images for web vulnerability scanners (OWASP Dependency Check, OWASP ZAP, FindSecBugs).So anyone can pull the preconfigured   docker images, run containers from them and simply  perform the security testing minimizing the hard work. Once its done as the second step these docker images will be used to create a cloud computing solution where developers/testers can provide their source code and get the security testing done for them.

In order to provide the cloud computing solution i am going to create a web application for end users where they can provide their source code or URL of their web application (If they provide the URL of the application they will be able to do dynamic security testing only.).End user will be able to provide their source code as a ZIP file,WAR file or provide GitHub, Google drive or Dropbox links.Whatever the way they provide the source code , system will download/pull or extract the application code in the most suitable server (Since this is a cloud computing system we could use a distributed computing solution to get high performance and efficiency) or we can run these security scans parallely in the same server as well.

It is good if end users can provide the url to the application if they want to do a dynamic scan with ZAP tool but in case they have not hosted their application, the system will deploy the application in the server(This deployment will happen in the server which is going to perform FindSecBugs and OWASP Dependency Check security scan since ZAP only needs the URL of the application.So we can start ZAP tool in another server parallely with the OWASP Dependency Check and FindSecBugs ).When we use ZAP tool to perform dynamic testing we need to manage sessions.To do that in  the front end application we can ask for the url of login page and names of username field and the password field and the credentials for test users.the system will use these data to login to the application when prefoming zap tool 

http://zap/JSON/authentication/action/setAuthenticationMethod/?zapapiformat=JSON&contextId=1&authMethodName=formBasedAuthentication&authMethodConfigParams=loginUrl%3Dhttp%253A%252F%252Fexample.com%252Flogin%252F%26loginRequestData%3Dusername%253D%257B%2525username%2525%257D%2526password%253D%257B%2525password%2525%257D

Once every thing is done the system will start the security scan and will generate reports. Since the web vulnerability scanning is a very time consuming process (especially ZAP tool) once the scans are done the system will email the reports to the end user, also since all the tools generate XML reports we can let end users to log into our front end application and show the results there also. If we do that (not related to GSOC project) by getting their feed backs on the results and by identifying the patterns and situations of false positives may be we will be able to create a way to remove most of the false positives from the reports by using machine learning mechanisms.