Making sure it still works: Regression Testing at XING

Agile development processes like SCRUM (confer Scrum @ XING – a case study) raise the bar for quality assurance professionals. The biggest challenge we need to overcome is getting used to release cycles of two to four weeks.

Even in teams that are running smoothly there’s often not enough time besides the regular quality assurance (QA) of newly developed features to execute manual regression tests. Every new line of code that’s added to a program has the potential to break existing functionality. Therefore it’s not enough to test new functionality – the existing one also needs to be tested. This is done by re-executing test cases from previous development cycles and called “regression testing”. Of course engineers write automated tests themselves that should prevent existing code from breaking but those tend to be on a unit or component level. System tests that use the frontend to enter data and verify all layers take a lot of time to run and would hinder engineers in their daily development work.
The usual solution for this problem consists of testers creating automated regression tests.

The tools for regression testing

The QA department at XING has developed a framework, which is used by testers inside project teams to create regression tests. This framework is based on TestNG and Selenium.

TestNG is a JUnit-like test framework that (besides the usual unit test capabilities) makes it possible to assign test methods to groups.  A test method that’s testing a search functionality in a backend interface will be assigned to “search” and “backend”. Since the test method is run in short acceptance tests it will also be assigned to “acceptance”. These groups can be used to create different suites, filtering all test methods for certain groups. That way we’re able to run short acceptance tests (filtering for group “acceptance”) and full-blown system tests (no group filtering) from the same test code. It’s also possible to run a test that focuses on parts of the system, e.g. the search.

Selenium is a test engine for most common browsers. While a lot of the web testing frameworks operate on a HTTP level and verify against the HTML code of a website, Selenium acts like a real user inside the browser. This is what actually allows us to test websites like XING with a high amount of JavaScript and AJAX.

We have now extended these tools quite heavily, although they were very powerful in the first place. The extensions start with improved logging mechanisms (results are written both in a database and our wiki) and don’t stop with default methods for the core functionalities of the XING platform (e.g. login and logout).

Automated test cases and their advantages

A usual automated test case looks like this:

@Description("TC-UC5-01-01\nsearch for string that is not available (neither company nor have)")
@Test(groups = { "frontend", "search" })
public void TC_UC5_01_01() {
String keyword = "abcdefghijklmnop";
login(TestUser.dont_care);
openCompanyPages();
searchCompanyByKeyword(keyword);
verifyCompanyCount(0);
verifyCompanySearchNoResult(keyword);
}

This has a couple of advantages:

1. The first thing to notice in this example is the speaking names of the methods. These shall ensure that even people without a deep technical knowledge understand what the test is about. This is especially helpful in case of a failing test.
2. Another advantage of these methods is the separation of test workflow and actual HTML frontend. Maintenance efforts are greatly reduced this way, since frontend changes lead to a change in a single isolation method instead of many changes in multiple test workflows (see below)

public void openCompanyPages() throws IOException {
String locator = "//*[@href='/companies/']";
waitForElementIsPresent(locator, Timeout);
clickAndWait(locator, Timeout);
}

3. The third advantage can be found while writing the automated tests. Since the workflows use the isolation methods, they can be created without having to use the actual frontend of the application. As soon as the frontend is available the isolation methods need to be implemented and the tests can be run.

In some sprints we managed to have the automated tests up and running before the manual testing finished.

The automated regression tests created by the project teams are executed on a daily basis in addition to the tests written by the engineers, to achieve a maximum of resistance against regression defects.

A recent example: Company profiles

Regression tests tend to grow over the lifespan of a project. In the past half year we’ve been automating more than 500 test cases that form part of the “Company Profiles” project. This is equivalent to about 60% of the specified manual test cases and takes about two hours to execute. The tests written by engineers, by comparison, consist of 330 test cases and take 83 seconds to run.

As you can see by the spikes in this chart, the regression tests found some errors which were then successfully fixed before the release. Most of the “noise” coming from errors and failures persistent throughout the whole chart can be attributed to an unstable server that the tests were executed on. This noise remains constant even as the amount of test cases increases, keeping the investigation efforts also at a constant level.
Using the powerful logging mechanisms we have in place it’s easy to determine which tests do more harm than good. These will be made more robust and if that’s not possible they will be deactivated.

Having an extensive suite of automated regression tests in place gives QA professionals both freedom and security: freedom to concentrate on the newly developed features and security that the old features are still working as intended. Using the techniques mentioned above, the costs for maintenance are lower than the costs for manual regression tests and a lot lower than the costs for bug fixes.

One week after the GoLive of the second version of Company Profiles we can see that the effort involved in regression test automation and manual testing was well worth it. There was a total of three reported bugs, all of them minor and fixed within a week. Compared to 624 person days of development effort, three bugs really are negligible.