Quality assurance has one of the most important parts to play in the process of software development. In order to make sure that the developed product is fully prepared to meet the needs of the market, the quality experts run a wide variety of tests, checking if the technical requirements are met, making sure that there are no bugs or passing them back to the development team upon discovery, before signing off on the product.
Many software companies on a global level use Lean and Agile methodologies as a framework for their QA process. The main difference between the two methodologies is between the way they present the flow of work. Agile is an iterative methodology and everything is planned on the basis of sprints, meaning that a team commits to completing a predefined number of tasks in a specific amount of time. Lean is flow-based and aims to finish as many tasks as possible to the highest quality in the shortest amount of time. Simply said, with Agile, everything may be completed way before the end of the sprint, but it won’t be released until the predefined date. As opposed to Agile, the due date of a Lean project is based on customer demand and his or her ability to receive the new addition. In Lean software development, the features are prepared, tested and as soon as they are done, may be released to the customer.
Why might a flow-based method mapped on a Kanban board work better than iterations when it comes to QA? There are several reasons applying a Lean method, such as Kanban, best suits the QA process.
As QA is among the final steps before the release of a new product or feature, there is a lot of hand off back and forth between developers and QA. For example, the QA team takes a feature that has been developed by the dev team and has to test it. The feature comes with the respective documentation, which was handed off to QA by the developer in charge. With this hand off comes the risk of generating waste (activities that do not bring value and yet consume time and resources), caused by differences in the documentation and the actual feature. In order to minimize the waste and avoid unnecessary complications, the whole process needs to be visualized as clearly as possible and optimized.
How do we do QA at Kanbanize?
This is a visualization of the set up of our QA team’s board in Kanbanize:
The columns are as follows:
REQUESTED:
To do: Tasks that should be started next are placed here. The tasks are placed according to their priority from right to left and then from top to bottom.
Plan: The QA expert plans what functions of the feature require testing and afterward moves on to coding the actual tests. If the team member creating the tests has questions, he or she collaborates with the rest of the QA team and tries to keep up communication within the comments of the card so that it can be tracked in the system. While exploring the new feature during the planning phase, a member of the QA team might find an application bug before any testing has been done. In these cases, a high priority bug card is created on the dev team board and the feature only proceeds to full testing after the bug has been removed. When planning is complete, the card is moved to the next column.
IN PROGRESS:
Coding: In this first phase, the member of the QA team assigned to this feature writes the code for the test. When finished, the changes are committed in the source repository and the card moves to the next column.The goal here is to regularly add new tests to the whole suite to keep it updated.
Waiting for tests to pass: The appropriate test suites are run. The team is waiting for the tests to pass on the test environment which is used for running the automated tests. This is done because the hardware in every machine is different and even if the tests pass successfully on the personal environment, they may fail on the machines used for testing at night. If the tests pass, the card becomes ready for review and on the contrary upon failure a card is generated automatically in the expedite swimlane.
Ready for Review: Cards that are ready for review. Every person on the QA team is allowed to contribute to code reviews. The only condition is that the reviewing member must have not worked on the card before and represents, in a sense, a fresh pair of eyes. The main purpose of this practice is to boost knowledge-sharing and to comply with the best practices for coding. During review, the less experienced members have the chance to learn from their colleagues and gain more awareness about the structure of the code and its details. In order to keep track of the team rules, members abide by a review and coding guide put together by the project managers and the team itself.
Review: Review of the card and code review. If the reviewer has questions or thinks that there are more test cases that must be covered, a comment should be added. If the card is for a specific scenario, a bug, for example, the reviewer should only validate that the test code fulfills the required scenario and might not go further to put the test in the context of the entirety of the application.
DONE: All tests have passed. Documentation cards are automatically moved to the QA swimlane in the Know-How board, a dedicated board to preserve the knowledge we gain in the work process. It is meant to be used in order to enable the team members to be able to learn without asking another member of the team for guidance.
The swimlanes are as follows:
EXPEDITE
There are three main reasons for cards to appear in the Expedite Lane in a QA Board:
1. Card from the Dev board needs a QA review, while the feature listed inside is still being developed, meaning QA should prioritize it to avoid blocking the developers’ progress.
2. Card from the Dev board is ready to be tested on production.
3. A test has failed.
*It is important to note that WIP Limit rules do not apply in this lane!
Miscellaneous: This swimlane holds items of various types – regression testing, helping another team, investigation, documentation, etc. Our QA uses the email integration functionality in Kanbanize so that when a message is sent to a predefined e-mail, a card is created in this swimlane. This email address is one for the entire QA team and it is used for internal communication with other teams and integrations. This is precisely how other teams request help.
Infrastructure: All tasks related to QA infrastructure changes or interference (Continuous Integration Server updates, etc.) are tracked here.
Technical Debt: Our test framework is like any other software application and all technical debt on QA tests is tracked in this swimlane.
Bugs: When a customer bug that has been reported or a bug that was discovered internally is fixed and ready to be deployed back on production, a card for creating an automated regression test will be created here.
*We differentiate between customer bug reports and bugs found internally because we want to be sure to prioritize the former. The customers are most important , which makes bugs reported by customers of the highest priority.
Features: This swimlane contains cards for creating automated tests that validate the behavior of different features. The backlog of this swimlane will never be empty. You can always find something to test in the software.
Our QA team is mainly focused on automation.
Automation is on focus, meaning that environment configurations, application configurations, tests execution and testing steps must run autonomously and, at the end, the team should have access to data for: passed, failed and skipped tests. The whole team believes that the application developers are the ones that give value to the customer and the QA’s duty is to help them as much as possible. So, if the dev team needs help, this is considered with a higher priority than creating new automated tests.
A typical day in the QA team revolves around the creation and maintenance of testing scripts. This is the main value that the team provides to the group.
Our test suites consist of E2E (End-to-End) tests which check if the flow of a feature is performing as designed from start to finish.
We have divided the E2E tests into four categories:
Webapp – General features in the web app are tested.
Runtime Policies – New runtime policies or changes to existing ones are tested.
Email Integration – Features interacting with the email integration in Kanbanize are tested.
API – Changes to the application programming interface are tested.
All the suites are executed during the night, in order to check for regressions that occurred during the previous day. They serve as a kind of health check for the entire software. The team collects historical data about the count of executed tests, how many of them were disabled or are failing and what is the reason for the failure. The suites are executed twice per night in order to ensure that the team does not allow unstable tests to slip between the cracks. If there is a difference in the amount of failed tests between the two executions, then the conclusion is that some of them are unstable and the team takes the necessary action to update them.
The entire QA process is visualized on the dedicated QA board in Kanbanize.
On some days, the first person who comes into the office in the morning will find a card in the Expedite lane that has been generated automatically to indicate failing tests in the overnight executions. When a nightly or a continuous integration test job fails, the CI (Continuous Integration) server uses the Kanbanize API to automatically create a card in the Expedite swimlane of the QA board. The person who gets to the card first becomes the assignee and begins investigating why the test has failed.
There are four reasons for the tests to fail. Therefore, there are four actions to be taken by the team in order to solve the problem and run the tests successfully:
Reason: A bug has been found.
QA Solution: A member of the team creates a bug card in the dev team board and set tags ‘automation’ and ‘regression’.
Reason: A change elsewhere in the application has caused an older test to fail.
QA Solution:The tests are updated.
Reason: A test is passing and failing erratically, is classified as unstable.
QA Solution: The team tries to stabilize the test.
Reason: A test has not run properly, test is classified as broken.
QA Solution: Action is taken towards fixing it.
How does the QA team deal with global Application Changes?
It sometimes happens that a failed test card, even in the expedite lane, ends up accumulating a high cycle time. Unfortunately, this is the burden of having a lot of end-to-end tests. The more of them you have, the more time you spend on maintenance and investigation because one bug could break many tests and one app change could affect several test suites. For example, in the beginning of 2016 there was a big update in Kanbanize concerning Card Details. Style, design, behaviour, new features, etc. were changed. The majority of the QA team’s tests have steps which go through those details and most of them ended up broken after the update.
In these circumstances, our team made the conscious decision not to fully disable the broken tests but aim to update them one by one as soon as possible. We considered our options and decided that, if the tests were disabled and updated later in time the possibility for a bug to “sneak in” and to be noticed later in the process might increase.
The Quality Assurance team is also responsible for three types of work items that are constantly decreasing:
QA Reviews
Not so long ago, the Kanbanize RnD decided to follow the “NO QA” approach to development in order to cultivate the mindset that all developers should aim for the highest quality before they pass a feature onto QA. This might even include building small tests of their own before a hand off to the QA team.
Before this shift in the team approach, very few of the developers were contributing to the testing process by validating that the new features were working correctly or a bug fix had not broken anything else. When a feature was considered ready, it was passed for QA check and, in most cases, was returned back after the planning phase because an obvious bug was discovered.
This is bound to happen once in awhile, however, the real problem was that this was a recurring action for almost every card. The result of this became that the cycle time of cards was getting significantly higher than anticipated and this was considered pure waste.
Eventually, the developers had to learn how to handle manual validation and leave the QA team to concentrate on automation.
Now, the application developers are writing testing notes during the tech design phase. The testing notes are reviewed from a QA team and if they consider that the proposed notes are not enough, more are added.
Regression Testing
The Kanbanize deployment rate for new versions is once a month. It has been like this for more than two years! In the week before each release, our QA team is constantly testing all features in order to identify potential problems which might have been missed. The invested effort for regression testing a feature depends on how well it is or is not covered by the automated test. Due to the development of our automation testing suite, the time it takes the team for regression testing is constantly shrinking.
Testing on Production
Our current situation is such that we do not apply automated tests to the new features or bug fixes. Therefore, validation that the deployment is successful depends on the RnD department. On the first working day after the release, the whole RnD team dedicates time for sanity testing the deployed changes. This is done in order to confirm that the bugs have been fixed and no further issues are introduced due to these changes. This happens pretty quickly because all people are involved in the process.
When asked what feature of Kanbanize is most useful to them, the QA team pointed out the runtime policies.
“The runtime policies are definitely the most helpful feature of all. We are currently using 12 policies that are sparing us from a lot of manual, repetitive work and helping us focus only on the stuff that actually gives value. As for the process, it is always evolving. Any changes we make are greatly motivated by metrics and our main change drivers are the Heat Map, Trends and Flows charts.”
Our quality experts are making use of the following runtime policies from the business rules engine in Kanbanize:
CARD IS MOVED:
When a card is moved to Ready for Review by QA in the Development Board, create a card in the Expedite swimlane of the QA team’s board.
When a card is Ready to be Tested on Production, create a card in the Expedite swimlane of the QA team’s board.
If a card has no assignee, assign the card to the person who starts working on it, meaning that the card is assigned automatically to the person who moves it to “In Progress”.
Create a card for automation of customer issue or expedite issue.
Create a card for automation of Bug.
Create a card for automation of Feature.
*The three policies above are shared between our Development and QA boards. When a card on the Dev board is moved to the column named “Ready for Production” then the above-mentioned cards are created on the QA board.
When a card on the development board is placed in the Ready for Code Review column, send a notification to the QA team.
When a Documentation Card is reviewed and moved to Done, move it to the QA swimlane of the dedicated Know-How board. Usually these cards hold answers to a problem or a question that gave one of our colleagues a hard time.
CARD IS BLOCKED:
Automatically Block the card if the required fields are not filled.
CARD IS CREATED:
Send a notification to every QA team member when a card is created in the QA Expedite lane. The notification is received as an in app notification as well as email to the inbox of each user in the QA board.
CARD IS UPDATED:
On our Development board, we have a card with information about server changes that need to be made for every new release. This policy is set to send a notification to every QA team member when the New Version of the card is updated.
RECURRING CARD IS UPDATED:
Done cards from yesterday are moved to Archive. In other words, at the end of the day, all cards with criteria indicating they were Moved to Done Date, Not Today are archived.
*Hint: The next day’s stand up meeting is much easier this way!
In conclusion, it is fair to mention that there are several ways to perform the process of quality assurance and each one has its merits. At Kanbanize, we have found that applying the Kanban method has kept our QA and Development teams as efficient as possible and always striving to remove more and more waste from their processes. The most important things for any QA team, no matter what method it makes use of in its day-to-day process, is that it aims to continuously develop, integrate, test and release new products at a cadence and quality that its customers as well as team members find optimal. We hope that by reaching this stage of the article you have learned something new that will help your QA team be even more effective using Kanban for its processes!