The What
I have a bunch of tests that i would like to run faster, by making them execute in parallel. In my tests:
I am not using a DataProvider and only want to make the same test run several times.
I am using a DataProvider and want my test to run with the provided values from the provider, but in parallel.
I am not using DataProviders, but my tests are ran by using the textng.xml file that specifies which tests to run (as per this article).
The How
I have a single test, which i want to run a number of times, but i do not want to change anything about it. In this case, the @Test annotation for the test method will be provided 2 attributes: one is the invocationCount, which tells the test execution engine how many times i want to run the test, and the threadPoolSize, to tell it how many execution threads i want to use at any given time. If only the threadPoolSize attribute is specified, no parallelism will be available (also if invocationCount is the only attribute present, the test will execute sequentually the number of times this parameter dictates). Dependent methods will also run in parallel (for example, if there is a @BeforeMethod method present, it will run before each test method will run, therefore ensuring that the tests' pre-requisites have been fulfilled). An example of such a setting can be found below:
@Test(invocationCount = 10, threadPoolSize = 3) 2. I am using a dataProvider for my test, but since it contains a large number of entries, i want to parallelize the test execution to run the tests faster. The test itself does not need to be updated, however the @DataProvider annotation will take an extra parameter, namely parallel. If this attribute is not specified, by default the tests do not run in parallel. The default number of threads for the parallel execution is 10. With this type of configuration, the depending methods are also ran (like the @BeforeMethod annotated method). Such a configuration is depicted below:
@DataProvider(name = "aDataProvider", parallel = true) 3. I am running the tests by means of a testng.xml file. In it i specified which are the tests that i want to run. In order to parallelize the test execution, in the beginning of this file, within the <suite> tag, i will add 2 attributes: the first one is parallel, which specifies the parallelism level - whether i want to run classes in parallel (and the methods within each class to be run sequentially, each class being associated with an execution thread), or have each method run in parallel (each method in its' thread; in the same thread, the currently running methods' depending methods will be also executed). The second attribute to be added specifies the number of threads to be allocated to the test execution, and is called thread-count. Two such examples are found below:
<suite name="Suite1" parallel="methods" thread-count="4"> <suite name="Suite1" parallel="classes" thread-count="5"> A couple of possible disadvantages
Although it is quite a benefit to speed up the test execution, there are two possible downsides to this. First of all, the performance of the machine where tests are running can become severely degraded, depending on how many execution threads have been defined. You might not be able to use that machine properly during test run. The second one is that, due to parallelism, tests might become unreliable or unstable (unexpected behavior could be observed by running them, like weird looking and interlaced console output or condition validation issues). Before opting to apply parallelism, the chosen tests should be ran several times to observe their behavior with the parallel configurations applied.