How can we minimise the risk of defects and additional costs of Dynamics 365 updates by using regression tests and the RSAT tool?

What is regression and when does it occur in a system?

Regression is the opposite of progression. Progress is synonymous with advancement, implies positive change and constitutes something desirable. Conversely, regression is something we would rather avoid. Unfortunately, regression can also affect IT infrastructure, including the ERP system which facilitates many processes important to the functioning of an organisation.

Nowadays, business applications are synchronised with many other tools and are constantly changing. Microsoft makes every effort to provide its customers with regular product updates. Approximately seven Dynamics 365 system updates are released annually, and customers must implement at least two of them. In addition, many businesses choose to customise the system by changing its standard features.

Attempting to update the system can have the opposite effect and negatively affect the proper running of processes. In a software context, such a scenario is called regression, meaning that something previously worked, but after modifying the application code, it stopped working. Such defects can occur as a result of a software upgrade, patch, or modification, as each of these actions causes a change in the structure of the code.

What effects can the regressions have?

We need to remember that a key aspect of system operation is to ensure process continuity, and the implementation of new functions must not be at the expense of system reliability and stability. However, what if such a defect occurs in the system?

Imagine a manufacturer who has just implemented a certain modification to his Microsoft Dynamics 365 system. The next day, it turns out that there is a problem in the vendor accounts module. At the same time, the company has many pending orders for its products and purchase orders for raw materials needed for production. They cannot be processed due to a defect. Production is delayed, as are many of the dependent processes. Chaos is increasing, with employees seeking alternative ways of completing standard tasks. Thus, a minor change has disrupted people's daily routines, and eliminating the damage requires additional work.

Of course, the occurrence of a regression does not necessarily result in such a scenario, but it will always involve inconvenience.

What are regression tests and what is their purpose?

So, should we not implement any system changes? Of course not, that would be unwise. An ERP system should be flexible and constantly adapt to new business requirements. However, remember to undertake its modernisation wisely, and be aware of the possibility of undesirable outcomes. As the saying goes: "Prevention is better than cure" and this also applies in this case.

A good tactic is to test the system for regression. This involves regularly running tests to verify whether the processes are still performing correctly. In practice, a tester or a special program executes a particular process step by step, marking whether each process has succeeded. If the test fails, rapid detection and reporting of the regression will significantly reduce the time to eliminate it.

Such tests can be carried out at any stage of the system's operation, but are particularly recommended after a software upgrade, a change in the specifications of existing functions or the implementation of new functions or patches. The testing process requires people with knowledge of various business operations, often specialising in specific modules. The team should be diverse so that the tests cover various aspects of the system, including those financial or compliance-related for a particular country or territorial area.

RSAT - the essential assistant in regression testing

Now we know not to neglect regression testing. This naturally raises the question of how to implement it effectively in a company.

For example, take manual testing - it requires the preparation of multiple Test Suites, each of which contain Test Cases. A Test Case consists of a list of steps that need to be performed to ensure that a particular Dynamics 365 functionality works properly.

We can task the designated team to carry out the testing process, giving them a 'test manual' so that each member tests their assigned module. The entire process will likely consume a vast number of hours. Obviously, the more complex or modified our system is, the more time it will take to test it. In addition, the need to repeat this time-consuming task will arise again when we again upgrade the application environment or decide to make changes to our ERP system.

It does not encourage manual regression testing. What is the alternative? To automate the described process. The Regression Suite Automation Tool, or RSAT for short, which is a Microsoft tool dedicated to automatic regression testing of Dynamics 365, is here to help.

RSAT works in a simple way - just provide it with an 'instruction' containing a description of the process and it will execute it automatically. The built-in Dynamics 365 Task Recorder helps to create such an instruction. With this tool, we can record almost any business process and save it as a recording.

‍

Another integral part of RSAT is the fully integrated Azure DevOps platform, and its service - Test Plans - makes it easy to manage tests and their results.

How does RSAT work?

Let us look at some technical aspects of RSAT-based testing.

• Once RSAT is configured and a process recording created using Task Recorder, we can move on to creating tests.
• Each test case is based on a separate recording, from which RSAT generates a set of files that includes executable files with C# code and a configurable Excel file holding the test parameters.
• When running the test, RSAT, with the Selenium framework, automatically opens a browser, navigates to our system, and repeats all the steps contained in the indicated recording.
• If the reproduced process runs smoothly, the test is successful.
• If the programme fails to execute any of the steps correctly in our environment, the test will fail, and an error message will be displayed.

‍

‍

RSAT takes the workload out of the hands of a human and crawls the system in search of anomalies.

Failed tests require attention, but with the possibility of uploading the results to Azure DevOps, analysis becomes simpler. People involved in the testing process can track the test progress and manage test plans, of which there may be several for a project. What is more, by integrating RSAT with Azure Pipelines, we gain the ability to create a test schedule - tests can be launched periodically at set times with it.

How to implement RSAT?

If you have the Dynamics 365 Finance and Operations platform of version 15 or later, you will only need an Azure DevOps Test Manager or Test Plans license and a Microsoft Excel spreadsheet. Then download the latest version of RSAT from the Microsoft website for free. The tool can be installed on any computer running Windows 10, Windows 11, or Windows Server 2016, as it always connects to the test environment via a web browser. And that is all.

The most important issue is to develop a comprehensive test suite tailored to your ERP system. Consider which components of the application are critical or particularly error prone. Testing of critical functions should have a higher priority and be carried out first. The correct recording of the processes the tests will be based on is half the success. Tests created from the recordings can later be modified and run by different users of the system. Some tests based on universal processes can be run in different modules, as well as in multiple companies existing in the system.

Summary

The Task Recorder records processes executed in Dynamics 365; RSAT creates automated tests based on the recordings, which we can manage in Azure DevOps Test Plans. With this toolkit, we can map repeatable processes and check if the running functionality has been broken when new functionality is implemented. The tests performed by RSAT will help to quickly detect anomalies and thus cut the costs of fixes. Therefore, if you care about business process continuity, it is worth implementing regression testing in your system as soon as possible.