Automated Security Testing with soapUI and Hudson
Test and Monitor | Posted November 08, 2011

You’ve seen the conference session, attended the webinar, and now here is the blog post that gives you an overview of the security testing features in soapUI 4.0 and how you can automate them using more or less any scheduling tool you might prefer (we like Hudson).


When we reached out to our users with an online survey a couple of years ago the one feature that was requested most was “Security Testing” – something that caught us by surprise but makes a lot of sense when you come to think of it; just like for example performance, security is a non-functional requirement that can have severe consequences if neglected – look at Sony, Facebook, Nokia, MySQL, etc. – all major corporations that in the last years have been subject to relatively simple SQL Injection attacks which have exposed data not meant to be public – most notably in Sony’s case where over 1 million user records including addresses, etc. were exposed and leaked to the public.

Our philosophy behind the Security Testing features in soapUI is the same as for soapUI LoadTests; make it easy (and free!) for our users to add basic security tests for their services allowing them to scan for common vulnerabilities without requiring too much investment in time. This will build both awareness of security-related problems and provide a starting-point for digging deeper into security issues with soapUI or other tools more fit for the specific need. soapUI Pro makes the process even easier by providing a number of related wizards but the core functionality described below is fully available in the free version also.

What is Functional Security Testing Anyway?

Let’s back up a bit and try to define our domain here; when malicious requests are sent to a system a number of technical layers can be targeted, including;

  1. The NIC (Network interface Card) and its drivers
  2. The Operating System itself as it processes the incoming request from the NIC
  3. The target application server that handles the request (for example Apache, IIS, etc.)
  4. The application itself which provides some kind of functionality to the end user

While the attacks targeted at the first 2 layers often use brute force to exploit vulnerabilities in handling of connections, memory, etc., attacks at layer 3 and 4 usually target at vulnerabilities related to the actual functionality of the application itself – for example an application server might have a security flaw related to how it handles session identifiers while an application might have vulnerabilities related to how it performs specific functionality to the client (for example searching, updating of data, etc.). When it comes to the first 3 layers there are already a number of great tools out there (both free and commercial) that help you to identify related vulnerabilities, and since soapUI already has support for functional and performance testing of applications and their APIs, adding security testing to the mix was a natural choice, especially because this also happens to be a very common target for malicious attacks.

The model in soapUI for identifying functional security vulnerabilities is built on top of existing functional tests; in soapUI a Security Test extends an existing TestCase by allowing you to add a number of Security Scans after the desired TestSteps – these scans are executed after the corresponding TestStep and aim at identifying any vulnerabilities related to its input or output – the results for the Security Scan is asserted by using any of the built in Security Assertions (or any of the standard assertions if that fits better). The following picture can be used to illustrate;

When running a Security Test, the underlying TestCase is thus executed as it would be if it were running as a standalone functional test, and the configured Security Scans will be executed and their results asserted to find any corresponding vulnerabilities.

The following screenshot shows a Security Test for a simple login-logout TestCase with SQL Injection and Fuzzing Security Scans configured and executed for the login request;

(Head over to the soapUI website to read more about soapUI SecurityTests)

How to Identify Vulnerabilities?

As hinted above, soapUI uses assertions to check for vulnerabilities – let’s discuss this in a little more detail before we move on to looking at some actual Security Scans.

When you come to think of it – how would you identify security vulnerabilities in the first place? What response would you get back when trying (for example) a SQL Injection attack that would allow you to identify that a security vulnerability is at hand? Actually, this isn’t always as easy as it sounds. An obvious sign of a “successful” attack is that your target service stops responding to requests (i.e. it crashes), but this is a relatively unusual finding (but very valuable since you found it first!). More often you will get an error message back that tells you that the service was not able to process the (malicious) request – this might be a good thing during development but it is also one of those situations where you need to take on the hat of the hacker; does the error message tell the client something they shouldn’t know about your system? For example which database you are using? Which version? Which language or framework? Maybe it exposes internal IP numbers (“failed to connect to…”)? Any of these might be exactly what the client is looking for; the information allows them to target specific attacks at your systems infrastructure (there are public databases of known security issues in most software) which in turn might give them the backdoor they have been looking for. soapUI provides a specific assertion for detecting these kinds of responses, we call it the “Sensitive Information Exposure” assertion; it searches messages returned by your service for any information that might expose system details to the client – version numbers, software technologies, etc. The list is of course configurable so if you come up with some text to search for that you absolutely don’t want to be exposed by mistake in your services you can add this to the global configuration and make sure that it isn’t exposed by any of your services during a security test.

Unfortunately it gets harder: often injection attacks can be more-or-less silent, meaning that the response might not give any indication of that the attack actually succeeded at all – or it might be something very specific to your service. For example, if we configure a SQL Injection Scan to target a login request, what we don’t want to get from the service while testing is a successful login. Here a standard soapUI XPath assertion fits nicely; it can be configured to check the response for each SQL Injection attempt to check that it is not returning a sessionId but instead an error message (that of course doesn’t expose any unnecessary system information). In the case of the response not giving any hint at all, maybe a script assertion is needed to check the state of the database after the request to ensure that all is OK.

So to recap;

  • Security Tests resulting in a Denial of Service are unusual but valuable in the sense that you found them first and can make sure that they won’t happen again
  • Security Tests resulting in error messages or responses that somehow expose system information that could be misused by a client can be identified by using the Sensitive Information Exposure assertion
  • Many times the actual responses to a Security Test and how to interpret them from a security point of view is up to the tester and requires corresponding system knowledge and understanding to be able to create corresponding assertions that would pinpoint exploits in responses (without resulting in unnecessary false positives)

What can we Test For?

soapUI 4.0 comes with a total of 10 Security Scans that mimic some of the most commons Security Attacks being used today;

  • SQL and XPath Injection Security Scans try to provoke corresponding vulnerabilities in the server code.
  • Invalid Data, Boundary Check, Fuzzing and Malicious Attachment Security Scans send invalid input to provoke error unexpected errors and corresponding error messages that might exposes system information
  • Malformed XML and XML Bomb Security Scans try to exploit vulnerabilities in the target systems XML processing – XML Bombs could potential result in a DOS attack if the server is not correctly protected or configured.
  • Cross Site Scripting Scan helps you check for both persistent and non-persistent XSS vulnerabilities
  • Script Security Scan (in true soapUI spirit) allows you to script your own parameter mutations in the groovy scripting language

Supported Protocols and APIs

As mentioned above, soapUI layers Security Scans on top of existing requests in your TestCases, which applies to any requests that are parameterized or contain XML as the payload. This includes

  • SOAP Requests – any element or attribute in the request can be used as a parameter by the security scans. This is for requests sent over HTTP, HTTPS or JMS
  • REST / HTTP Requests that send XML in their body (POST and PUT) – the same parameterization applies here as for SOAP requests
  • REST / HTTP / AMF Requests with parameters – these parameters can be targeted by Security Scans as well

Before we head on to the idea and practice behind automation of Security Tests, let’s have a quick look at two of the most common Security Attacks and how they can be tested for with soapUI.

SQL Injection Attacks

SQL Injection attacks are the most common attacks against modern websites. The reason for this being (at least) twofold;

  • Most sites are backed by a database which needs to be accessed from the web/service application. Sloppy coding (which is very common) at this level will make way for exploits.
  • These Databases often hold data that is of value to hackers (email addresses, credit-card numbers, etc.) making them more attractive than attacks that don’t provide any “value” to the attackers.

To illustrate how simple a SQL Injection attack is consider the following SQL Statement used to validate the username/password sent to a login script;

  • select * from tb_user where username = ‘%uname%’  and password = ‘%pwd%’
  • If the input to the script is a valid uname/pwd combination, the actual SQL code executed will be as follows;
    • For uname = bob, pwd = fish
      • select * from tb_user where username= ’bob’ and password = ’fish’

If there is such a username/password combination in the database, the user will be found and logged in to the system, all is fine.

Now let’s look what happens if the input is the following;

  • For uname = bob, pwd = ’ or ‘1’ = ‘1
    • select * from tb_user where username= ’bob’ and password = ’’ or ‘1’ = ‘1’

The value provided as a password actually “melts into” the existing SQL used to validate the user and results in a SQL statement that will always return rows from the database, no matter what username is provided, resulting in the user always being logged in – pure evil!

Obviously building SQL statements as shown above should not be a viable option, but unfortunately this “concatenation” approach is indeed very common, especially in more modern (and therefore often immature) frameworks / platforms, which partially explains why these vulnerabilities are still so prevalent.

So how would you detect this in soapUI? Well, as already hinted above, a more specific XPath assertion could be used to detect that an invalid SQL statement does not result in a successful login. The complete setup in soapUI would be as follows;

From top to bottom in this (somewhat confusing) screenshot we can see the following;

  1. A Security Test that performs a SQL Injection Scan for the login request
  2. The SQL Injection Scan Configuration Dialog which has two parameters (username and password) configured into which it will insert a number of preconfigured SQL statements to mimic a SQL Injection attack. The XPath Match assertion for the Security Scan is visible at the bottom of the configuration dialog.
  3. The XPath Match Configuration dialog contains an XPath statement that counts the number of return values in the response (session ids) and expects that value to be 0 (otherwise the request would have succeeded in logging in).

Running this test gives the following entries in the Security Run log;

Here you can see that the first injection actually succeeded in logging in (it used the same SQL hack as I did in my above code example). The following injections that all sent possible exploits did not succeed in generating a valid session id.

Cross Site Scripting

Cross-Site Scripting attacks (“XSS” attacks) are also one of the most common attacks targeted at both web applications and their APIs. The attack usually exploits deficiencies in a client web application where it injects a script that is executed in the browser and there can record user actions, passwords, etc and relay them to a separate server without the user ever noticing.

The attack can be performed in two main ways;

    • Non-Persistent Attacks: include a malicious client-side script in a request that will then be part of the response web page and executed in the client browser. This is squarely targeted at web applications, for example a search page might display the actual search string in the response page (“you searched for…”) without escaping it; if the search string is a full-blown script it would be executed in the client and could perform corresponding unwanted actions.

  • Persistent Attacks: stores a malicious client-side script into server-side storage (for example a database) from each it will later be displayed in another web-page. This attack can target both web interfaces and their APIs that may be exposed to update the underlying data; for example a REST interface might be available for updating database records via JSON – inserting malicious content here might result in it being displayed in a totally separate web page showing database content, and if that page is insecure, just as the page in the example for non-persistent attacks, the same exploits could apply.

Testing for these two is relatively easy but prone to false positives; the test basically makes a request containing some non-malicious script and then checks either the response page or other pages (for persistent attacks) if they contain these scripts unmodified. The trick here is that scripts can be returned in a web page unmodified but still escaped which will result in false positives if simple checking for the string itself.

The soapUI Cross-Site Scripting Scan does just this; it inserts a number of known simple scripts into the configured request parameters and will then check both the immediate response and secondary web-pages to see if they contain the script unmodified. The configuration is as follows;

  • At the top you can see the search parameter configured in the Cross Site Scripting Scan dialog
  • The dedicated “Cross Site Scripting Detection” assertion has been added and its configuration dialog is shown below – here you can select to

  • check the immediate response for the inserted content 
  • specify a custom script that builds a list of secondary URLs that will be checked for the inserted content; the reason this being a script is that you might need to include some kind of id or other dynamic parameters into these URLs based on the immediate response received by the initial request (for example the response might contain the unique ID of an item that the request created and this must be passed as a parameter to the secondary requests).

You can hopefully see from both of these examples that the exploit mechanics are actually not very advanced or hard to understand, which actually stands true for most security vulnerabilities and hopefully gives you the incitement to both consider and test for them during development and testing phases of your product lifecycle.

Automating Security Tests

So finally we arrive at the automation part of this article. The idea behind automating security test execution is straight-forward; run your security tests repeatedly to detect any security vulnerability regressions that might be introduced into your code or system while it is being developed. Changes that might trigger such a regression could be:

  • Component Updates – changing a third-party component or library might for example change error handling or error messages to be more verbose and expose unnecessary information to the client
  • Infrastructure Changes  - upgrading or introducing key components into an existing infrastructure might change how requests are processed or handled
  • Refactoring of Code – just like unit-tests catch functional errors introduced by refactoring, security tests will hopefully find errors like error messages, injection or XSS exploits, etc.

soapUI provides a command-line securityrunner.bat/.sh file that allows you to run your security tests in a headless environment and will generate a corresponding report for you. Automating this with Hudson is (almost) a no-brainer;

    1. Create a new free-style software project job in Hudson

    1. Configure an “Execute Windows batch command” build step and correspond triggers and Post-build actions:

  1. (Make sure you have –j option specified for the command-line runner so it outputs the JUnit-compatible reports used by the Post-build action further down)

  2. If you are hosting the soapUI project in a code repository configure the corresponding connections under the “Source Code Management” section of the Job configuration

Now you are all set! If you run the Job in Hudson the soapUI securitytestrunner will execute the configured Security Tests allowing you to follow the output in the Console;

Any errors will be shown in the log as the tests run;

And the end of the log will show the status of the execution;

Going back to the Project view in Hudson will show an aggregated view of the historic executions;

And digging into the last result will show you

Piece of cake!


Final Words

I hope I have gotten these points across;

  • Neglecting possible security vulnerabilities and related issues in your services and APIs can put your data (and your business!) at serious risk.
  • Taking security seriously and building basic expertise around it is an investment well worth making.
  • The core mechanics of most security vulnerabilities are easy to understand, test for, and prevent.
  • With the new Security Testing features in soapUI 4.0 there is really no good reason to not do at least a very basic Security Testing baseline checking for the most common vulnerabilities.
  • Automating the execution of your Security Tests (soapUI or not) further builds your awareness of and defense against unnecessary vulnerabilities.

So, head right over to the soapUI website to download the latest version (you might even opt for a soapUI Pro Trial if you want to cut some corners) and get grooving with your first security test.


Thanks for your time!






And be sure to follow and share with us on Twitter, Facebook and Linkedin.


By submitting this form, you agree to our
Terms of Use and Privacy Policy

Thanks for Subscribing

Keep an eye on your inbox for more great content.

Continue Reading

Add a little SmartBear to your life

Stay on top of your Software game with the latest developer tips, best practices and news, delivered straight to your inbox