Once you have a regression suite set up, you will need to run it. When you have a smaller number of tests that need to be run on a specified day, that won’t be a problem, and the tests will successfully finish running within the allocated time period. However, as the suite becomes larger and larger, so does the amount of time required to run all the tests. In some cases, you can even get beyond a 24-hour test run. So, what can you do to optimize the test run time? You can use two strategies: parallelize and split.
Parallelization
Running tests in parallel can be done in several ways. One of the most common ways is to have your tests run in parallel by using, for example, TestNG’s parallelization mechanism, when using a testng.xml type of file for selecting what tests to run. It basically allows you to define on what level you want your tests to be run in parallel: on a class level, or maybe a method level. Ideally, from my experience, you will use the class level, since a class can have some setup or teardown methods that would need to be run before or after either just one test, or the entire class. If you choose to parallelize at method level, these setup and teardown methods might behave unexpectedly and might work properly for one of the running tests, but would impact another running test. A class is kind of a self-sufficient unit, so it’s better to run it all at once.
Also, for this TestNG based approach, you can specify the number of threads on which to run the parallelization. This determines how many units (classes in this case) will be run in parallel at one time. Keep in mind that setting the number of threads to a high value will cause the system on which the tests are running to become very unresponsive, since more of its’ resources will be allocated to the test run. That means that if you are running on your work station, you will have a hard time using it at all. But also, if let’s say a browser is used, it will open and behave in a sluggish manner, which will eventually slow the tests down. The higher the number of parallel threads, the slower the test run becomes, and the time spent running the parallelization will tend to be closer to the regular test run time. However, a number like 5 is a common value for the number of threads, which allows for a good number of tests to be run in parallel.
Another parallelization mechanism is offered by TestNG’s DataProvider functionality, but this is a less time saver than the above approach. Simply because you don’t have that many tests that use a DataProvider. However, for such tests where you do use it, you can specify that you want parallelism to be in place, which will cause your test (which will be a method) to be run in parallel, by going through all the values from the data provider. Also note that there is a default limit set in the Data Provider, that will allow only for a chunk of the methods to run in parallel at once, and when any method run finishes a new one is started. Basically, at any given time, there are only that many tests running in parallel as the threshold allows it.
If you choose to go with parallelization, first of all, make sure you don’t exaggerate with it. Meaning you don’t over-parallelize. For example, combining both the above approaches might lead to unexpected results and weird reporting. Also, make sure that any tests you plan to run in parallel are immune to this way of running. Have independent tests, that can be run at any time, even if other tests for the same functionality, or in the same area of the functionality, are running at that time.
Splitting
Another method of running the automated tests is by splitting them into several runs. This means that during an allocated timeframe, let’s say for example 3 days, you will run the entire regression suite of tests, but not all of the tests during the same day. You won’t parallelize (or if you want to you can), but instead you will choose, based on some criteria, what tests to run on the first day, then what tests to run on the second, while the remainder will be run on the third day. Of course, this timeframe depends on what is reasonable for the quantity of tests you have and how much it takes for them to run.
In order to pick which tests to run when, you can choose some criteria like: priority or feature. If you go with priorities, that sounds nice: it will allow you to run the most important tests first and identify the major issues early. However usually it is a bit tricky to separate important tests from not so important tests, especially if they cover several features or several areas of a feature. They are probably spread across several test classes and several packages. You could use TestNG’s groups for making the selection, but you need to be careful not to forget adding the correct groups to any of the tests you want to run.
Another approach for splitting is to simply run all the tests from a class or all the test from a package at once, since normally a package would be used for storing tests for the same or related functionality. This is the easiest approach, since you are only required to group the tests properly when you write them, and there is no need to add any TestNG groups to them. With this approach, instead of validating bits of the entire application during one day of test run, bits which are spread throughout the application, you are validating an entire part of the application, which you don’t need to look into during the following days of test runs.
Whatever strategies you choose for creating and running your regression tests, don’t forget to keep them updated when the features that they test also get updated.