Overview

You can access the presentation here.

The source code and the eclipse project for the TestNG demo is available here.

Feel free to contact me at ujugmonal@gmail.com if you have any questions.

-- Monal Daxini

TestNG and Eclipse

The TestNG Eclipse plugin is included if you downloaded and installed Eclipse from http://icslib/eclipse. If you got the eclipse installation package from somewhere else, you can install the plugin using "Find and Install" feature with http://beust.com/eclipse as the update url.

How to use the Eclipse plugin?

  • You can run a single test by right clicking on the test file and then clicking on Run As -> TestNG Test.
  • You can run a TestNG xml file specifying suite of tests (TestNG suite xml format) by right clicking and then clicking on Run As -> TestNG Suite.
  • For running groups you will have to create a new "Run Configuration". The plugins allows you to browse through a list of groups, which it automatically builds by scanning your test files. You can then choose the group(s) to run. Alternatively, you can specify the groups to run in the TestNG suite xml file, and run the file as mentioned above.

TestNG Eclipse plugin gotchas

Currently there are three issues with the TestNG eclipse plugin, which the users need to be aware of:
  • If you have setup a "Run Configuration" for TestNG, and you run it using Run As -> "name of Run Configuration", at times this does not work right and throws errors although it should. If this occurs, just click on Run As -> TestNG Suite, thus not using the defined "Run Configuration".
  • The number of tests run by and as reported by the TestNG eclipse plugin is different from that reported by the maven-surefire-plugin when the tests are run using maven.
  • When you run a TestNG xml suite file direclty in Eclipse it scans all the classes in the packages specified (including your source classes, which may be in the same package as your test classes)in the suite xml file. The plugin throws exceptions for classes that are not TestNG or Junit classes. However, it does run all the tests successfully and shows the results in the TestNG view. Our recommendation is to watch if the results are all green and ignore the text output in the console, and use the console only if you have errors in your tests. The errors cannot be avoided easily because TestNG xml suite file does not support the specification of classes to include or exclude using a reqular expression. However, it does support listing individual classes to include, which is not feasible if you have large number of tests. So, if you run the suite xml without individually listing the test classes then all the classes (including source classes) are scanned and included in the TestNG Test Runner.
This project has a TestNG suite xml file called testng-suite.xml. You can define additioanl TestNG xml suite files if needed. All the output from eclipses TestNG plugin by default goes to test-output folder, which can be customized in Windows -> Preferences options.

TestNG Junit and Maven

This project uses maven-surefire-plugin 2.3 that supports both TestNG and Junit. However, there are some quirks with this plugin, which this document outlines. By default - The pom.xml of your project has everything configured to automatically pick up all JUnit and TestNG tests that exist in your project's src/test/java directory. All test classes having a package name that matches "*.integration.*" pattern are treated as integration tests and run during the "integration-test" phase of the maven lifecycle, and unit tests are run in the "test" lifecycle phase. You can specifically invoke a certain phase of the lifecycle, and thus automatically execute all the goals bound to that phase, using the following command: 
        mvn <phase>
        Examples: mvn test
                  mvn integration-test
The following are the inclusion and exclusion patterns for the test classes run in the test phase: 
        
          <includes>
            <include>**/*Tests.java</include>
            <include>**/*Test.java</include>
          </includes>
          <excludes>
            <exclude>**/*$*</exclude>
            <exclude>**/integration/**</exclude>
          </excludes>
The following are the inclusion and exclusion patterns for test classes executed in the integration-test phase: 
        
          <includes>
            <include>**/integration/**/*Tests.java</include>
            <include>**/integration/**/*Test.java</include>
          </includes>
          <excludes>
            <exclude>**/*$*</exclude>
          </excludes>

Grouping Tests - TestNG and JUnit

The afore mentioned maven-surefire-plugin does allow "groups" and "excludeGroups" elements, but they do not work (a bug). Therefore, the maven-surefire-plugin in project's pom.xml, created from the archetype, is not configured to run or exclude groups. Hence, the best way to run or exclude groups defined in your test classes is specifying the TestNG suite xml file in pom.xml. The snippets of xml that have to be added to pom.xml for running a certain set of groups (via TestNG suite xml files) during test phase and another set during the integration-test phase of maven's lifecycle is listed below.

For tests to be run in the test phase add everything between and inclusive of the plugin tag into your pom.xml from the following snippet: 

        
          <build>
            <plugins>
              ...
              <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-surefire-plugin</artifactId>
                <version>2.3</version>
                <!--  We cannot use the executions element to specify this configuration
                and the one	specified for integration tests in the profiles defined below
                as it does not work. -->
                <configuration>
                  <forkMode>once</forkMode>
                  <argLine>-Djava.awt.headless=true -XX:MaxPermSize=128m -Xmx128m</argLine>
                  <!--
                  <includes>
                    <include>**/*Tests.java</include>
                    <include>**/*Test.java</include>
                  </includes>
                  <excludes>
                    <exclude>**/*$*</exclude>
                    <exclude>**/integration/**</exclude>
                  </excludes>
                  -->
                  <suiteXmlFiles>
                    <suiteXmlFile>
                      src/test/resources/testng/unit-testng.xml
                    </suiteXmlFile>
                  </suiteXmlFiles>
                </configuration>
              </plugin>
              ...
            </plugins>
          </build>

For tests to be run in the integration-test phase add the following into you pom.xml. However, if your pom.xml already has a profiles tag then only add everything between and inclusive of the profile tag. 

        
          <profiles>
            <profile>
              <id>itest</id>
              <activation>
                <activeByDefault>true</activeByDefault>
              </activation>
              <build>
                <plugins>
                  <plugin>
                    <groupId>org.apache.maven.plugins</groupId>
                    <artifactId>maven-surefire-plugin</artifactId>
                    <version>2.3</version>
                    <executions>
                      <execution>
                        <id>surefire-integration-tests</id>
                        <phase>integration-test</phase>
                        <goals>
                          <goal>test</goal>
                        </goals>
                        <configuration>
                          <forkMode>once</forkMode>
                          <argLine> -Djava.awt.headless=true -XX:MaxPermSize=128m -Xmx128m </argLine>
                          <!--
                          <includes>
                            <include>**/integration/**/*Tests.java</include>
                            <include>**/integration/**/*Test.java</include>
                          </includes>
                          <excludes>
                            <exclude>**/*$*</exclude>
                          </excludes>
                          -->
                          <suiteXmlFiles>
                            <suiteXmlFile>
                              src/test/resources/testng/integration-testng.xml
                            </suiteXmlFile>
                          </suiteXmlFiles>
                          </configuration>
                      </execution>
                    </executions>
                  </plugin>
                </plugins>
              </build>
            </profile>
          </profiles>

You can pass additional parameters to TestNG by specifying them in the argLine tag. More information about these parameters, is described in the Running TestNG documentation .

You can specify groups that you want to run in the suite xml file, provided you have your tests annotated with the groups appropriately. For example, specify the "unit" group to be run in the suite unit-testng.xml file (see above) to execute all unit tests, and specify "integration-test in integration-testng file to execute all integration tests. TestNG does not allow you to define a new group containing a set of methods or classes in the suite xml file. However, it does allow you to speciy groups of groups, and the groups to run.

The only way to group tests in JUnit is to organize test classes in different packages and then use inclusion and exclusion filters. It is very cumbersome to manage and maintain a pacakage based approach to gropuing tests, and at times it may be impossible to move a class into a different package because it tests a class having package scope.

One problem with the configuration above is that specifying the "suiteXmlFiles" element renders the inclusion and exclusion patterns ineffective. This is a bug in the maven-surefire-plugin, hence, the inclusion and exclusion patterns are shown but commented. TestNG suite xml file allows one to specify the pacakges to be included, and the classes to be included. However, the classes to be included cannot be specified as a pattern, they can only be listed individually. Therefore, the option of specifying the packages and classes in the TestNG xml file is not as powerful as the inclusion and exclusion pattern of the maven-surefire-plugin.

Please see the section "TestNG and Eclipse" above to run groups in Eclipse using the TestNG plugin.

Reference

TestNG documentatoin