Confirmation and regression tests – how do they work in practice?

Well-described bug

There is no such thing as a perfect application. You can find an error or a defect in each one. Imagine that you are a manual tester, and you check the correctness of creating a package for a courier company based on the form. You have to manually fill in the most important fields: sender’s data, recipient’s data, phone number, choose the size of the shipment, and finally click Save. Suddenly, it turns out that after saving, the new shipment does not appear on the list, and you have to report – as all testers like to say – a bug.

How to describe the defect correctly?

1. Environment

Depending on the project, it may be useful to have the entire URL of a given environment on which we are testing the application or the environment name itself to know where to look for the cause. This is important because there are usually even several testing environments.

2. User

You need to specify the user name and the role they perform when creating the application. As a rule, these users are various with a certain range of activities to perform.

3. Reproduction steps

Describe, preferably in bullet points, all actions that were performed until the error occurred. The description of these steps is extremely important. Let’s assume that the bug occurs, for example, only for the largest shipments. Without this information, all possible options would be checked, and, as a result, the developer, while bug fixing, would unnecessarily lose time.

4. Result

Then, after the steps needed for reproduction has been taken, you have to save the result, which appeared in the application. For example, “The form for creating a package has been closed. The parcel does not appear on the list.”

5. Expected result

It is very important to determine how the tested functionality should work properly after following the steps. For example, “The form for creating a package has been closed. A new shipment with an ID number appears on the list.”

6. Bug proof

A screenshot or a video from a manual test performed is proof that a given bug exists. If the bug does not contain a large number of steps, a screenshot showing the final result and highlighting the most important information is enough. I personally recommend the Screenpresso, which easily allows you to take a correct screenshot. When the bug is more complicated, I record a video, so I do not miss anything. To create test videos, I use Snagit. This application is paid, but in my opinion, it is worth investing in.

It is also worth to include logs from the web console, which is one of the development tools. Logs allow a developer to find the cause of a defect faster. The console can be easily turned on or off using the F12 button. I used to use the Firebug extension to Mozilla Firefox most often, but, since it is no longer developed, I use a web console with Google Chrome.

7. Priority

The bugs found by the tester are of different nature and affect the application operation in different ways. In most test management tools, a priority field with a drop-down list is used to determine the report validity level. It usually includes three levels: low, medium, and high. On this basis, the appropriate bug fixing queue is determined.

With the example of our parcel creation form, I can explain how to evaluate defects according to these 3 categories:

• Low – applications have little application operation impact and often involve minor visual changes, typos, and so on; incorrect shipment form name, for example.
• Medium –  bug category makes the application very difficult to run but allow us to continue testing; no visible sender data field on the shipments list, for example.
• High –  bugs blocking the tests and make it impossible for us to check the product’s performance further; no new shipment on the shipment list, for example.

It happens that the priority field is not available. In such situations, I managed to add such a category at the end of the application name or set colors, to green meaning low, yellow meaning medium, red meaning high, for example.

In one of the projects, I met with a division into priority and severity fields. The priority field concerned developers’ repair priority (managed by the project manager). The severity field concerned the application’s operation bug impact (filled in by the tester creating the bug report).

8. Label

Often in the test management tools, as in very popular Jira, there is a label field. This is a label that makes it easier to filter, sort bugs, and so on. In this field, depending on the project, you can add terms such as sprint name (when working in Agile methodology), release number, name of the test phase in which the defect was found, and, for larger projects, an explanation whether the bug is functional or concerns bad data.

Figure – Software test – process.

Confirmation tests as a testing basis

After reporting the bug and receiving information that it has been fixed, we can proceed to the so-called confirmation test. To perform it correctly, you should choose the same environment, the same user, and follow the reproduction steps described in the report. We do not extend the test scope; it covers only the areas described in the bug report.

After the test has been properly performed and the expected result appears in the application, we post a screenshot or a video and add a comment to the submission. After these steps are taken, we can consider the bug as fixed and close the report.

In my opinion, it is worth considering adding a comment and a screenshot or a video of the retest. In one of the projects I took part in, manual tests without such information were considered invalid.

Regression tests –  finding new bugs

All manual tests checking that the latest fixes did not violate the previously functioning product areas are considered as regression tests. Regression tests are an ideal opportunity to find an application’s new defects.

Here is a good example. After checking that a new shipment appears on the list and after closing the report, we also test the form cancellation option to creating such a shipment (before reporting the bug, it worked correctly). After testing, it turns out that the form cannot be canceled, and the application window is still open. This is the reason why it is worth doing this kind of test. I know from experience that you can easily damage other related product areas when you improve a given functionality.

Depending on the size of the project, regression tests can be performed in different ways. When the application is not complex, it is best to make a list of its most important areas using so-called keywords, such as: creating a shipment, canceling a shipment, and so on, and perform such tests after each bug is fixed. In the case of a complex application, when there are more areas, and there is not enough time to check all of them, it is best to agree with the project leader on the highest priority. Then, you can adjust to your needs and perform only those regression tests that are most urgent.

To conclude, it depends on the range of manual tests performed, whether they are considered confirmation or regressive. Of course, it must be remembered that a correctly described defect is the basis for good cooperation between the tester and the developer. Thanks to this, we respect our time and do not lose it for additional arrangements resulting from a bad description of the report.

I hope that my suggestions prove useful and help you to improve the description of bug reports in the application, thus to better distinguish between confirmation and regression tests.