Does ATDD really save you time?

Details

Jan20

Posted by Super User

Acceptance Test Driven Development (ATDD) is a very effective development practice that essentially involves writing specifications in the form of documented and automated examples. These automated examples become automated acceptance tests that validate the features being delivered. The process of writing these examples encourages teams to focus on where the business value of a feature is coming from, which in turn helps developers aim for the most appropriate solutions in business terms.

When I help folks out with ATDD and TDD practices, one common question people ask me is this: Does using ATDD cost more in development time?

My gut feeling is that when you practice Acceptance Test Driven Development, developers spend maybe 20% extra time writing automated tests, compared to a project with no automated testing at all As far as TDD goes, I generally find that overall an experienced TDD practitioner will take similar or slighly less time to implement code using TDD than without using TDD.

However, that is far from the whole picture. Anecdotal evidence, and my own experience, suggest significant other advantages in adopting ATDD practices, and in fact projects using ATDD tend to deliver faster and with significantly fewer defectsthan projects using more conventional approaches. For example:

• Writing specifications in the form of automated tests helps clarify the requirements earlier rather than later, and ATDD-style reporting also makes it very visible what is actully being delivered. This helps keep development efforts focused on building features that will provide real value, which in turn reduces waste.
• Code written using TDD (especially using a BDD-style approach) not only has fewer bugs, but it tends to be much easier to understand and maintain, which makes changes easier (and less scary) to implement, and makes ongoing maintenance cheaper.

Automated Acceptance and Regression Testing (where automated tests are written after the features are delivered, but the tests do not play a driving role in the development process) are a great way to improve code quality and reduce the risk of embarrassing regressions and outages. However automated tests by themselves do not help focus and validate the development efforts the way ATDD does.

A recent study seems to confirm this. In this very interesting study, teams using agile development practices including in particular ATDD and TDD delivered projects 31% faster with 4 times fewer defects. Another study, back in 2008 found that using TDD reduced defects by between 40% and 90%, although it increased initial development time by 15-35%. The teams did agree however that this increase was offset by lower maintenance costs.

And yet another, from 2006, found that TDD significant improvements in the quality of the code (in terms of the defect rate), but were inconclusive with regards to productivity gains.

These studies are certainly very interesting. However I think that the earlier ones actually tend to understate the advantages of TDD, for two reasons:

• They are often done with teams that were realtively new to TDD practices: However like many practices, TDD works better when you have already had some practice. In my experience, time spen
• They do not factor in the time gains in the maintenance phase: indeed, anyone with any experience using TDD will confirm that code written using good TDD practices is significantly easier to understand and maintain.

Modern development practices such as ATDD and TDD are not easy: they require significant efforts in championing, training, mentoring, and managerial support. However there is little doubt that, when well applied, these techniques significantly improve code quality and reduce maintenance costs, but can also contribute to faster project delivery and thus reduced project costs.

atdd

bdd

Functional Test Coverage - taking BDD reporting to the next level

Details

Jan20

Posted by Super User

Conventional test reports, generated by tools such as JUnit or TestNG, naturally focus on what tests have been executed, and whether they passed or failed. While this is certainly useful from a testing perspective, these reports are far from telling the whole picture.

BDD reporting tools like Cucumber and JBehave take things a step further, introducing the concept of "pending" tests. A pending test is one that has been specified (for example, as an acceptance criteria for a user story), but which has not been implemented yet.

In BDD, we describe the expected behaviour of our application using concrete examples, that eventually form the basis of the "acceptance criteria" for the user stories we are implementing. BDD tools such as Cucumber and JBehave not only report on test results: they also report on the user stories that these tests validate.

However this reporting is still limited for large projects, where the numbers of user stories can become unwieldy. User stories are not created in isolation: rather, user stories help describe features, which support capabilities that need to be implemented to achieve the business goals of the application. So it makes sense to be able to report on test results not only at the user story level, but also at higher levels, for example in terms of features and capabilities. This makes it easier to report on not only what stories have been implemented, but also what features and capabilities remain to be done. An example of such a report is shown in Figure 1 (or see the full report here).

In agile projects, it is generally considered that a user story is not complete until all of its automated acceptance tests pass. Similarly, a feature cannot be considered ready to deliver until all of the acceptance criteria for the underlying user stories have been specified and implemented. However, sensible teams shy away from trying to define all of the acceptance criteria up-front, leaving this until the "last responsible moment", often shortly before the user story is scheduled to be implemented. For this reason, reports that relate project progress and status only in terms of test results are missing out on the big picture.

To get a more accurate idea of what features have been delivered, which ones are in progress, and what work remains to be done, we must think not in terms of test results, but in terms of the requirements as we currently understand them, matching the currently implemented tests to these requirements, but also pointing out what requirements currently have no acceptance criteria defined. And when graphs and reports illustrate how much progress has been made, the requirements with no acceptance criteria must also be part of the picture.

Requirements-level BDD reporting with Thucydides

Thucydides puts some of these concepts into practice. Thucydides reports not only on how the tests did, but also fits them into the broader picture, showing what requirements have been tested and, just as importantly, what requirements haven't.

During the rest of this article, we will see how to report on both your requirements and your test results using a very simple directory-based approach. You can follow along with this example by cloning the Github project at https://github.com/thucydides-webtests/thucydides-simple-demo

Simple requirements in Thucydides - a directory-based approach

Thucydides can integrate with many different requirement management systems, and it is easy to write your own plugin to tailor the integration to suite your particular environment. A popular approach, for example, is to store requirements in JIRA and to use Thucydides to read the requirements hierarcy directly from the JIRA cards. However the simplest approach, which uses a directory-based approach, is probably the easiest to use to get started, and it is that approach that we will be looking at here.

Requirements can usually be organized in a hierarchial structure. By default, Thucydides uses a three-level hierarchy of requirements. At the top level, capabilities represent a high-level capacity that the application must provide to meet the application's business goals. At the next level down, features help deliver these capabilities. To make implementation easier, a feature can be broken up into user stories, each of which in turn can contain a number of acceptance criteria.

Of course, you don't have to use this structure if it doesn't suit you. You can override the thucydides.capability.types system property to provide your own hierarchy. For example, if you wanted a hierarchy with modules,epics, and features, you would just set thucydides.capability.types to "module,epic,feature".

When we use the default directory-based requirements strategy in Thucydides, the requirements are stored in a hierarchial directory structure that matches the requirements hierarchy. At the lowest level, a user story is represented by a JBehave *.story file, an easyb story, or a JUnit test. All of the other requirements are represented as directories (see Figure 2 for an example of such a structure).

In each requirements directory, you can optionally place a file called narrative.txt, which contains a free-text summary of the requirement. This will appear in the reports, with the first line appearing as the requirement title. A typical narrative text is illustrated in the following example:

Learn the meaning of a word
In order to learn the meaning of a word that I don't know
As an online reader
I want to be able to find out the meaning of the word

If you are implementing the acceptance criteria as JUnit tests, just place the JUnit tests in the package that matches the correspoinding requirement. You need to use the thucydides.test.root system property to specify the root package of your requirements. For the example in Figure 2, this value should be set to nz.govt.nzqa.lssu.stories.

If you are using JBehave, just place the *.story files in the src/test/resources/storiesdirectory, again respecting a directory structure that corresponds to your requirement hierarchy. Thenarrative.txt files also work for JBehave requirements.

Progress is measured by the total number of passing, failing or pending acceptance criteria, either for the whole project (at the top level), or within a particular requirement as you drill down the requirements hierarchy. For the purposes of reporting, a requirement with no acceptance criteria is attributed an arbitrary number of "imaginary" pending acceptance criteria. Thucydides uses 4 per requirement by default, but you can override this value using the thucydides.estimated.tests.per.requirement system property.

Conclusion

BDD is an excellent approach for communicating with, and reporting back to, stakeholders. However, for accurate acceptance test reporting on real-world projects, you need to go beyond the story level, and cater for the whole requirements hierarchy. In particular, you need to not only report on tests that have been executed, but also allow for the tests that haven't been written yet.

Thucydides puts these concepts into practice: using a simple directory-based convention, you can easily integrate your requirements hierarcy into your acceptance tests.

bdd

Code coverage metrics and Functional Test Coverage

Details

Jul21

Posted by Super User

There have been some articles and tweets about code coverage recently, and it seems that many developers are still laboring under a few misconceptions in this area.

Code coverage can be a very useful metric. However you need to know how, and when, to use it. The link between code coverage and test quality is tenuous at best - in short, high code coverage is, in itself, no guarantee of well tested code. And increasing code coverage for the sake of code coverage will not necessarily improve either the quality of your tests or the quality of your application. It is easy (and obviously a largely futile exercise) to achieve high code coverage metrics without actually testing anything at all.

Now don't go thinking I'm not a fan of test coverage. For the record, I am a huge fan of high test coverage, though I don't write tests explicitly with this aim (as I will discuss further on). As a metric, code coverage has its limitations, and should not be used for purposes for which it is poorly suited. Test coverage is excellent at indicated what code has not been exercised by your unit tests. Indeed, if high code coverage does not prove, in itself, that your code is well tested, low code coverage provides fairly conclusive evidence that your code is untested. An experienced developer will know how to use this information to complete her tests to cover important edge cases and boundary conditions.

But what of the broader picture? How do code coverage metrics help you deliver a useful, high quality product to your users?

Well tested applications tend to be more reliable, easier to understand, easier to maintain and in the end faster to develop. This seems a no-brainer, but it is also the practical experience of countless TDD practitioners, and the results of quite a few academic studies. And, in my experience, the single most effective way to achieve high test quality comes from using a combination of ATDD and TDD/BDD. Techniques such as Acceptance-Test Driven Development (ATDD) and Example-based specifications are an excellent way to to drive and track the development process. This process drills down and fans out into Test-Driven Development (TDD), often with a behavioural flavour to it (BDD) at a lower level. This holistic approach has the major advantage of giving you confidence in your code both on a functional (does it does what the client wants) and a technical (does it work) level.

So what of test coverage? For a product owner, or for someone from QA, the notion of 90% test coverage is abstract at best. It may be able to indicate that all of the classes in thecom.acme.gizmo.widgetpackage have been exercised during the unit (and possibly the integration). However what is more useful is to know how many requirements, or story cards, or features, have been demonstrably implemented and tested.

Each story has a set of acceptance criteria, typically expressed as examples of how the feature would work in different scenarios. Developers or testers automate these acceptance criteria (for web applications, this could involve using Selenium or WebDriver tests, for example). These tests are then run automatically, for example whenever the code changes (ideally), or on a nightly basis (if the tests take a very long time to run). The reports generated by these test runs give the product owners a very clear idea of which features have been implemented, which work, and how many are still pending implementation. BDD tools such as easyb or cucumber are a great help implementing this sort of tests.

So where does that leave us with code coverage? In short, you really need both functional and technical test coverage metrics. However high code coverage should be the natural outcome of good testing practices, not a goal to be aimed for. For this reason, I am not a big fan of aiming for a certain percentage of code coverage. But, if I am working on a project using proper ATDD, BDD and TDD practices and the code coverage drops below say 90-95%, I will investigate, as it may be an indicator of an underlying problem or an area where good testing practices have not been followed.

Managing state between steps

Details

Jul21

Posted by Super User

Sometimes it's useful to be able to pass information between steps. For example, you might need to check that a client's details entered into a registration form appear correctly on a confirmation page later on.

You could do this by passing values from one step to another, however this tends to clutter up the steps. Another approach is to use the Thucydides test session, which is essentially a hash map where you can store variables for the duration of a single test. You can obtain this session map using the Thucydides.getCurrentSession() static method.

As illustrated here, you can p

@Step
public void notes_publication_name_and_date() {
    PublicationDatesPage page = pages().get(PublicationDatesPage.class);
    String publicationName = page.getPublicationName();
    DateTime publicationDate = page.getPublicationDate();

    Thucydides.getCurrentSession().put("publicationName", publicationName);
    Thucydides.getCurrentSession().put("publicationDate", publicationDate);
}

Then, in a step invoked later on in the test, you can check the values stored in the session:

public void checks_publication_details_on_confirmation_page() {

    ConfirmationPage page = pages().get(ConfirmationPage.class);

    String selectedPublicationName = (String) 
Thucydides.getCurrentSession().get("publicationName");
    DateTime selectedPublicationDate = (DateTime) 
Thucydides.getCurrentSession().get("publicationDate");

    assertThat(page.getPublicationDate(), is(selectedPublicationName));
    assertThat(page.getPublicationName(), is(selectedPublicationDate));

}

If no variable is found with the requested name, the test will fail. The test session is cleared at the start of each test.