Testers must understand the differences
between testing in traditional lifecycle models (e.g., sequential such as the
V-model or iterative such as RUP) and Agile lifecycles in order to work
effectively and efficiently. The Agile models differ in terms of the way
testing and development activities are integrated, the project work products,
the names, entry and exit criteria used for various levels of testing, the use
of tools, and how independent testing can be effectively utilized.
Testers should remember that organizations
vary considerably in their implementation of lifecycles. Deviation from the
ideals of Agile lifecycles may represent intelligent customization and
adaptation of the practices. The ability to adapt to the context of a given
project, including the software development practices actually followed, is a
key success factor for testers.
Testing and Development Activities
One of the main differences between
traditional lifecycles and Agile lifecycles is the idea of very short iterations, each iteration resulting in working
software that delivers features of value to business stakeholders. At the
beginning of the project, there is a release planning period. This is followed
by a sequence of iterations. At the beginning of each iteration, there is an
iteration planning period. Once iteration scope is established, the selected
user stories are developed, integrated with the system, and tested. These
iterations are highly dynamic, with development, integration, and testing
activities taking place throughout each iteration, and with considerable
parallelism and overlap. Testing activities occur throughout the iteration, not
as a final activity.
Testers, developers, and business
stakeholders all have a role in testing, as with traditional lifecycles.
- Developers perform unit tests
as they develop features from the user stories.
- Testers then test those
features. Business stakeholders also test the stories during implementation.
- Business stakeholders might use
written test cases, but they also might simply experiment with and use the
feature in order to provide fast feedback to the development team.
In some cases, hardening or stabilization iterations occur periodically to
resolve any lingering defects and other forms of technical debt. However, the
best practice is that no feature is considered done until it has been
integrated and tested with the system. Another good practice is to address
defects remaining from the previous iteration at the beginning of the next
iteration, as part of the backlog for that iteration (referred to as “fix bugs
first”).
However, some complain that this practice
results in a situation where the total work to be done in the iteration is
unknown and it will be more difficult to estimate when the remaining features
can be done. At the end of the sequence of iterations, there can be a set of
release activities to get the software ready for delivery, though in some cases
delivery occurs at the end of each iteration.
When risk-based
testing is used as one of the test strategies, a high-level risk
analysis occurs during release planning, with testers often driving that
analysis. However, the specific quality risks associated with each iteration
are identified and assessed in iteration planning. This risk analysis can
influence the sequence of development as well as the priority and depth of
testing for the features. It also influences the estimation of the test effort
required for each feature.
In some Agile practices (e.g., Extreme
Programming), pairing is used. Pairing
can involve testers working together in twos to test a feature. Pairing can
also involve a tester working collaboratively with a developer to develop and
test a feature. Pairing can be difficult when the test team is distributed, but
processes and tools can help enable distributed pairing.
Testers may also serve as testing and
quality coaches within the team, sharing testing knowledge and supporting
quality assurance work within the team. This promotes a sense of collective
ownership of quality of the product.
Test automation at all levels of testing
occurs in many Agile teams, and this can mean that testers spend time creating,
executing, monitoring, and maintaining automated tests and results. Because of
the heavy use of test automation, a higher percentage of the manual testing on
Agile projects tends to be done using experience-based and defect-based
techniques such as software attacks, exploratory testing, and error guessing.
While developers will focus on creating unit tests, testers should focus on creating
automated integration,
system, and system integration tests. This leads to a tendency for Agile teams
to favour testers with a strong technical and test automation background.
One core Agile principle is that change may
occur throughout the project. Therefore, lightweight work product documentation
is favoured in Agile projects. Changes to existing features have testing
implications, especially regression testing implications. The use of automated
testing is one way of managing the amount of test effort associated with
change.
However, it’s important that the rate of
change not exceed the project team’s ability to deal with the risks associated
with those changes.
Project Work Products
Project work products of immediate interest
to Agile testers typically fall into three categories:
- Business-oriented work products
that describe what is needed (e.g., requirements specifications) and how to use
it (e.g., user documentation)
- Development work products that
describe how the system is built (e.g., database entity-relationship diagrams),
that actually implement the system (e.g., code), or that evaluate individual
pieces of code (e.g., automated unit tests)
- Test work products that
describe how the system is tested (e.g., test strategies and plans), that
actually test the system (e.g., manual and automated tests), or that present
test results (e.g., test dashboards)
In a typical Agile project, it is a common
practice to avoid producing vast amounts of documentation. Instead, focus is
more on having working software, together with automated tests that demonstrate
conformance to requirements. This encouragement to reduce documentation applies
only to documentation that does not deliver value to the customer. In a
successful Agile project, a balance is struck between increasing efficiency by
reducing documentation and providing sufficient documentation to support
business, testing, development, and maintenance activities. The team must make
a decision during release planning about which work products are required and
what level of work product documentation is needed.
Typical business-oriented work products on
Agile projects include user stories and acceptance criteria. User stories are
the Agile form of requirements specifications, and should explain how the
system should behave with respect to a single, coherent feature or function.
A user story should define a feature small
enough to be completed in a single iteration. Larger collections of related
features, or a collection of sub-features that make up a single complex feature, may be referred to as “epics”. Epics may include user
stories for different development teams. For example, one user story can
describe what is required at the API-level (middleware) while another story
describes what is needed at the UI-level (application). These collections may
be developed over a series of sprints. Each epic and its user stories should
have associated acceptance criteria.
Epics can be easily broken down into user
stories by slicing them vertically and they comply to INVEST criteria. Each
Epic and user story contains acceptance criteria with necessary business rules
and mockups attached to them.
Typical developer work products on Agile
projects include code. Agile developers also often create automated unit tests.
These tests might be created after the development of code. In some cases,
though, developers create tests incrementally, before each portion of the code
is written, in order to provide a way of verifying, once that portion of code
is written, whether it works as expected. While this approach is referred to as
test first or test-driven development, in reality the tests are more a form of
executable low-level design specifications rather than tests.
Typical tester work products on Agile
projects include automated tests, as well as documents such as test plans,
quality risk catalogs, manual tests, defect reports, and test results logs. The
documents are captured in as lightweight a fashion as possible, which is often
also true of these documents in traditional lifecycles. Testers will also
produce test metrics from defect reports and test results logs, and again there
is an emphasis on a lightweight approach.
In some Agile implementations, especially
regulated, safety critical, distributed, or highly complex projects and
products, further formalization of these work products is required. For
example, some teams transform user stories and acceptance criteria into more
formal requirements specifications. Vertical and horizontal traceability
reports may be prepared to satisfy auditors, regulations, and other
requirements.
In some regulated agile projects, more
inspection of the documentation that is being produced during the project
execution may be required.
For example: Federal drug administration
(FDA) projects require lot of audit reports, strategy reports and compliance
reports that need to be produced during the project execution.
