Skip to content
This repository was archived by the owner on Nov 17, 2017. It is now read-only.

Latest commit

 

History

History

metamer

Folders and files

NameName
Last commit message
Last commit date

parent directory

..
application
application
 
 
ftest
ftest
 
 
README.adoc
README.adoc
 
 
TESTS.adoc
TESTS.adoc
 
 
build.sh
build.sh
 
 
pom.xml
pom.xml
 
 

README.adoc

Metamer

Metamer is a testing application and test suite which covers all of RichFaces 4 components and it is able to display them in different templates (tables, accordion, ui:repeat, …​)

1. Metamer Maven Structure

Metamer consists of root module and 2 submodules:

  • root - parent for application, ftest

    • application - Java-EE application

    • ftest - functional test suite using Selenium 2, contains all tests

2. Metamer Application Structure

Metamer is Maven based web application which can be deployed onto Wildfly 8 (and possibly any other Java EE container) and Tomcat 7 and 8.

Source code is available at GitHub.

2.1. JSF Page per Component/Feature

  • Templates

    • configurable nesting of templates using various container components:

    • root: webapp/templates/template.xhtml

    • templates:

      • plain

      • iteration - a4jRepeat, hDataTable, richDataTable…​

      • panels - richPanel, richTabPanel, richTogglePanel, richAccordion…​

  • Page Parts

    • template.xhtml - nesting mechanism (see above)

    • header.xhtml - contains general controls

    • footer.xhtml - contains output of VersionBean

  • Customization per component (ui:insert in template.xhtml):

    • outOfTemplateBefore - usually contains additional controls for advanced features of components which can’t be covered by customization of attribute values

    • content - contains component nested in all configured templates

    • outOfTemplateAfter - as for outOfTemplateBefore, usually contains attribute list

  • Attributes

    • attributes for each component are bound directly from page using following notation:

      • #{componentBean.attributes[‘attributeName’].value}

      • where attributes stands for class Attributes extends Map<String, Attribute>

      • attributes are loaded to map from richfaces’s faces-config.xml

      • attributes can have predefined enumeration values using ComponentBean.properties files

  • Separation

    • each component has own package (named like Bean for given component) for component samples

    • in each package there is:

      • list.xhtml - the links to all samples

      • simple.xhtml - default component usage (not compulsory)

      • {feature}.xhtml - feature sample

2.2. Managed Beans

  • per component (and some per component/feature)

  • bean.* packages

  • RichBean

    • serves common functionality for general tests (actions, action listeners)

  • TemplatesBean

    • serves list of templates for user session

  • PhasesBean

    • holds the list of phases which request goes through

    • phases provided by RichPhaseListener

  • VersionBean

    • provides set of runtime and browser versions

    • output on each Metamer page ⇒ can be copied to JIRA’s environment

2.3. Model

  • provides models collections for iteration

  • loaded from XMLs using JAXB

  • model.* packages

  • exposed in bean Model

2.4. Additional Resources

  • any other resources are located under webapp/resources/*

3. Building and running Metamer

3.1. Building Metamer

Metamer can be built in several profiles to satisfy compatibility with various runtimes and JSF implementations. To build the application, run bash script build.sh in root directory of Metamer. This script generates several WAR archives with application in directory application/target which can be used in various scenarios:

  • metamer-jee6.war - to deploy on Wildfly, EAP

  • metamer-tomcat-mojarra.war - with bundled JSF Mojarra reference implementation to deploy on Tomcat 7

  • metamer-tomcat-myfaces.war - with bundled JSF MyFaces to deploy on Tomcat 7

To build the application with one specific profile run mvn clean install -DskipTests -P{profileName} where profileName can be found in pom.xml in root directory. Created WAR file will be located in application/target folder.

3.2. Deploying Metamer

To deploy the application just copy the WAR file into server deployment folder. Optionally, you can import the project into IDE (Eclipse, JBDS,…​) and deploy directly from IDE. This is particularly useful when you need to do a lot of changes and re-deploy often.

4. Functional Test Development

4.1. Prerequisites for running the tests

  • SW:

    • should be OS independent, but needs graphical interface to run Firefox browser — tested with Windows 7/8/10, Fedora 20-23, RHEL 5/6/7

    • Firefox browser

    • Maven 3

    • JDK 1.7 and up

  • HW:

    • at least 4GB RAM

    • dual core processor with 2,5Ghz

4.2. Running a test

  • switch to metamer/ftest directory and run mvn clean verify -PcontainerProfileName [-Dtest=testName | -Dtestng.suite.xml=pathToSuite] [-Dtemplates=templateName] [-DpathToEAPZip=/home/someUser/somePath/eap.zip] [-DpathToPatch=/home/someuser/somepath/BZXYZ.zip] [-Dversion.eap=x.y.z] [-Deap.ws.enabled=false] [-Dbrowser=browserName] [-Drepeats=numberOfRepeats] [-Dconfigurator.skip.reverse] [-Dconfigurator.skip.case=caseName] [-Dconfigurator.skip.enabled=false] [-Dmyfaces]

    • containerProfileName is a name of a container you want to use, see pom.xml in parent for their names

      • e.g. -Pwildfly-remote-10-0

      • when using some jbosseap* profile, you can change the path to EAP zip file (if the path generated by the qa-maven-plugin does not suite you) by using system property pathToEAPZip, e.g. mvn clean verify -Pjbosseap-managed-6-4 -DpathToEAPZip=/home/me/eap/6.4.x/eap.zip

      • when using some jbosseap* or wildfly* profile, you can also apply a one-off patch to actual container using system property pathToPatch, e.g. mvn clean verify -Pjbosseap-managed-6-4 -DpathToPatch=/home/me/patches/BZXYZ.zip

      • when using some jbosseap* profile, you have to change the used EAP version with system property version.eap (e.g. 6.4.7, 6.4.7.CP.CR1), for example mvn clean verify -Pjbosseap-managed-6-4 -Dversion.eap=6.4.7.CP.CR1. Beware, that you should use the same major and minor version as in the name of the activated profile (e.g. activated profile jbosseap-managed-6-4, use version.eap=6.4.?).

      • when using some jbosseap* profile, you can turn off the RichFaces push over WebSocket protocol by property eap.ws.enabled=false (can influence the stability of push tests).

    • testName is a name of a specific test to run

      • e.g. -Dtest=TestEditor for all tests in TestEditor class or -Dtest=TestEditor#testRendered to run just one method

      • regular expression can be used, for instance -Dtest=TestA* will run all test classes which name begins with 'TestA'

      • omit -Dtest if you want to run all tests (not recommended, the test suite is huge)

    • pathToSuite is a path to an existing testng-suite.xml (see content of folder metamer/ftest/src/test/resources/testng/ for available suites)

      • e.g. -Dtestng.suite.xml=src/test/resources/testng/testng-output.xml

      • the pathToSuite configuration will be ignored, when you specify the -Dtest property

      • the suites are (usually) defined by the focus of individual components in RichFaces (e.g. output, input, dnd)

      • to see which tests are run with any particular TestNG suite, open the suite in metamer/ftest/src/test/resources/testng/

      • the default value is set to src/test/resources/testng/testng-all.xml (link)

    • templateName is one or more of the templates such as plain, richPanel, a4jRepeat, uiRepeat

      • the template is wrapper in which will be the tested component placed. It is used for checking the correct behavior between components (RichFaces component inside of other/same RichFaces component or inside of a JSF component).

      • list of all templates used in tests can be found in AbstractMetamerTest class in the annotation @Templates over the field template.

      • tested component can be placed in template (possible values are):

        • plain, the tested component is not wrapped

        • richAccordion, RichFaces switchable panel component (rich:accordion)

        • richCollapsibleSubTable, RichFaces switchable table component (rich:collapsibleSubTable)

        • richExtendedDataTable, RichFaces complex table component (rich:extendedDataTable)

        • richDataGrid, RichFaces iteration component (rich:dataGrid)

        • richCollapsiblePanel, RichFaces switchable panel component (rich:collapsiblePanel)

        • richTabPanel, RichFaces switchable panel component (rich:tabPanel)

        • richPopupPanel, RichFaces pop-up panel component (rich:popupPanel)

        • a4jRegion, non-visual RichFaces component, used for marking processed zones during ajax requests (a4j:region)

        • a4jRepeat, basic RichFaces iteration component (a4j:repeat)

        • uiRepeat, basic JSF iteration component (ui:repeat)

      • all templates are case insensitive and have aliases (can be found/edited in Template class)

      • to run tests in:

        • all templates: use * or all

        • no template (default value): use either of plain, no, none (they are equivalent) or do not use the templates property

        • more templates concurrently: use + for separating the templates, e.g. popup+accordion+edt (ExtendedDataTable in Accordion in PopupPanel)

        • more templates separately: use , for separating the templates, e.g. accordion,popup,popup+edt. Each test in test suite will be executed in each specified template(s) (if it can run in such template).

    • browser will set the used browser, download necessary driver binaries, download specific browser binary (Firefox only, see below), download and extract EAP when some jbosseap-managed profile activated and will kill container’s and driver’s processes before testing. If the Jenkins environment is detected (see ProcessMojo#isOnJenkins in qa-maven-plugin project), then everything to download will be downloaded or linked from the network drive. This profile is activated by default with value firefox, ergo all previous tasks are done by default (by qa-maven-plugin). The profile can be disabled using property -Dbrowser.disabled. Additional info can be found in the browser profile in the metamer/ftest/pom.xml.

      • browserName is case-insensitive name of the browser to be used. Supported browsers are Internet Explorer (browserName contains ie, internetExplorer or explorer), Chrome (browserName contains cr or chrome), Firefox (browserName contains ff or firefox). If the browser is Firefox, then the name can be followed by a number (or dash and number), which will be used to specify browser’s version.

        • some examples:

          • -Dbrowser=ff to run tests with system’s Firefox browser.

          • -Dbrowser=cr to run tests with system’s Chrome browser.

          • -Dbrowser=ie9 or -Dbrowser=ie-9 to run tests with system’s IE browser (the version 9 will be ignored)

          • -Dbrowser=ff30 or -Dbrowser=ff-30 to run tests with Firefox 30. If working in non-Jenkins environment, then the browser binary will be downloaded and stored at {user.home}/selenium, otherwise the binary will be linked from network drive.

          • -Dbrowser=ff45esr to run tests with Firefox 45esr. If working in non-Jenkins environment, then the browser binary will be downloaded and stored at {user.home}/selenium, otherwise the binary will be linked from network drive.

      • there are three parts of the qa-maven-plugin which can be turned off by providing a system property:

        • -Densure.browser=false will skip the download and extraction of needed drivers and browser and will not set up paths to them. You have to specify following properties:

          • arq.extension.webdriver.browser name of the used browser, will be passed to Arquillian to instantiate the correct drivers

          • webdriver.firefox.bin path to Firefox binary (needed when using Firefox)

          • webdriver.chrome.driver path to Chrome WebDriver driver (needed when using Chrome)

          • webdriver.ie.driver path to Internet Explorer WebDriver driver (needed when using Internet Explorer)

        • -Densure.cleanup=false will skip the the tasks cleanup before running tests. It will not kill running containers processes.

        • -Densure.eap=false will skip the downloading and unpacking of EAP. You have to specify following property:

          • jbossEAP{XY}Home path to home directory of the used EAP. The {XY} should be replaced by the correct major and minor version, e.g. jbossEAP64Home.

        • you can also use multiple parts (or a single part) of the plugin and then override the properties returned from it by specifying a proper system property (see properties before).

    • repeats will invoke each test multiple times (set by this property value; usefull for debugging unstable tests)

    • configurator.skip.reverse will run only those tests, which are marked with @Skip annotation (see section About annotations) and its condition is met (see can depend on current OS, used container, JSF implementation…​)

    • configurator.skip.case=caseName will run only those tests, which are marked with @Skip annotation (see section About annotations) with case-sensitive value equal to the selected caseName class name. Use e.g. -Dconfigurator.skip.case=MyFaces to run only tests with annotation @Skip(On.JSF.MyFaces.class), -Dconfigurator.skip.case=UIRepeatSetIndexIssue to run only tests with annotation @Skip(BecauseOf.UIRepeatSetIndexIssue.class)

    • configurator.skip.enabled=false will turn the skip configurator off and all tests marked with the @Skip annotation will be normally executed

    • myfaces will set the JSF implementation to MyFaces, but this works only for Tomcat containers.

    • examples:

      • mvn clean verify -Pwildfly-managed-10-0 will run the whole test suite (src/test/resources/testng/testng-all.xml) in plain template with system’s Firefox browser on WildFly 10

      • mvn clean verify -Ptomcat-managed-8 -Dmyfaces=true -Dbrowser=cr -Dsmoke -Dtemplates=edt,pp will run the smoke test suite (src/test/resources/testng/testng-smoke.xml) in richExtendedDataTable(edt) and richPopupPanel(pp) template with system’s Chrome browser on Tomcat 8 with MyFaces implemention of JSF (bundled in metamer-tomcat-myfaces.war)

4.3. Debugging a test

  • set a breakpoint in code

  • to run debugging from terminal simply add another switch -Dmaven.surefire.debug test

4.4. Creating new tests

  • create or modify .xhtml in the application/src/main/webapp/components, this is the facelet which will be loaded in the test

  • create .java test class in the package for the component, in the ftest/src/test/java/…​

  • we are using Arquillian Graphene 2, tests have these specifics:

    • tests should extend AbstractWebDriverTest, have a look at: method with @Deployment annotation, @Drone WebDriver browser injection point, @ArquillianResource contextRoot, @BeforeMethod loadPage

    • we are using Page Object pattern, have a look at @Page annotation

    • we are using Page Fragments pattern, have a look at @FindBy annotations, for more information see Graphene documentation

  • see TESTS document for some basics.

4.5. Creating issue reproducers in Metamer

  • when creating new facelet for the issue, create it under affected component with the name of issue number (e.g. rf-15422.xhtml)

    • the newly created issue should also contain a steps to reproduce the issue, e.g. <rich:panel header="steps to reproduce">1. blah blah <br /> 2. blah blah</rich:panel>

    • list the newly created facelet also in the list.xhtml, together with the issue description and reference

  • create also a test class with the same name (e.g. TestRF15422.java) under the package for the affected component

  • annotate test method with:

    • @Skip (the skip condition can be specified, see @Skip annotation in About annotations)

    • @IssueTracking("https://issues.jboss.org/browse/RF-15422")

  • try to reuse existing backing beans, otherwise create new ones in the package application/src/main/java/org/richfaces/tests/metamer/bean/issues

  • make a comment in the JIRA issue what facelet reproduce the issue, and also steps to reproduce it

4.6. About annotations

In tests there are several important annotations we use:

  • @IssueTracking("https://issues.jboss.org/browse/RF-007")

    • IssueTracking is a marker annotation for test methods and classes

    • it does not influence which tests will be executed

    • it is used to define, that the test is somehow connected with a tracked issue (e.g. tests it, is blocked by)

    • if such issue is resolved (the test method is not marked with @Skip) and the test fails, we know it is a regression problem

  • @Templates(value = {"templateName"}, exclude = {"anotherTemplateName"})

    • this indicates in which templates you want or you do not want the test to be executed

    • test will not run when it is not executed with correct template

    • beware that it work only for templates listed in AbstractMetamerTest class in the annotation @Templates over the field template

    • it is usually used for marking tests, which are failing in certain templates, because of some issue in RichFaces or JSF. In such case it is often followed by @Skip and @IssueTracking annotations (see below)

    • some tests are marked to run only in plain template, because there is no profit to run them in other templates (e.g. only a client side case)

  • @UseForAllTests

    • this indicates that the field will be used as injection point in all tests from the class where it is placed to all classes extending it

    • all test methods in this hierarchy can run multiple times each time with different value injected to the annotated field (values and how to get them is specified in annotation parameters, see the javadoc for more informations)

    • this annotation influences the execution of tests (increases number of executions of one test method)

  • @UseWithField

    • this indicates that the test method will be used with a field to which the values specified in annotation attributes will be injected

    • uses the first field which will be found in class hierarchy from current class to the Object class

    • such marked test method can run multiple times each time with different value injected to the field which is using (field, values and how to get them is specified in annotation parameters, see the javadoc for more informations)

    • this annotation influences the execution of tests (increases number of executions of one test method)

  • @Uses

    • this is helper annotation for using multiple @UseWithField annotations in a single test method

  • @CoversAttributes

    • this indicates that the test covers testing of RichFaces component’s attribute (e.g. oncomplete, status, styleClass) or multiple attributes. The actual component is obtained from test package or from specified parameter.

    • this annotation is used in CoverageCollectorTool (see About Tools) to automatize the report of all covered attributes of all RichFaces components.

    • this annotation does not influencethe execution of tests

  • @Skip

    • this indicates that the test method will be skipped if given configuration/condition occurs. Without any value specified, the test with this annotation will always be skipped. See the javadoc of SkipConfigurator for more details and examples.

  • @Unstable

    • this indicates unstable test method. Such annotated method can be executed more than once, until the first success occurs or the maximum count of retry attempts is reached (see the javadoc for more details).

5. Metamer RichFaces Bug Report Guidelines

  1. Find if JIRA already exists (use filters to ease your work)

  2. File new RF/RFPL project Bug. RF is for bugs related to RichFaces (e.g. malfunctioning component). RFPL is for Metamer related problems (e.g. create/fix tests, update app dependency etc.).

    • choose components (probably component-* for specific component bug)

    • choose affects versions

    • for Metamer:

    • otherwise:

      • Description with code sample (Bean + JSF)

      • use {code} for formatting code samples and stack traces (see JIRA formating rules for more info on formating)

      • eventually provide specific version control revision or WAR directly

6. About Tools

There are a few tools to simplify/automatize some work in Metamer:

  • ComponentsAttributesCollectorTool

    • updates or creates java component attributes file (e.g. NotifyMessageAttributes.java) for each RichFaces component

    • tool should be runnable from IDE with test classpath (e.g. in NetBeans simply 'Run File') or by invoking maven command from 'metamer/ftest' mvn test -Pupdate-attributes

  • CoverageCollectorTool

    • creates a coverage report of all RichFaces components attributes

    • for creating report it uses component attributes file (e.g. NotifyMessageAttributes.java) and all test methods (annotated with @Test) with @CoversAttributes annotation. So it is important to have correctly annotated test methods and to have updated all of the components attributes files.

    • tool should be runnable from IDE with test classpath (e.g. in NetBeans simply 'Run File')

7. Testing with other JSF implementation

The test suite was executed with both Mojarra and MyFaces JSF implementations. The test suite is working with Mojarra provided in all EE aplication servers, which are present in the root pom.xml (WildFlys, EAPs) and with Mojarra bundled in the war for Tomcat containers (version is defined in upstream artifact richfaces-build, version can be changed using property version.org.glassfish.javax.faces, while building the Metamer application). The test suite is also working with MyFaces bundled in the war for Tomcat containers (version is defined in upstream artifact richfaces-build, version can be changed using property version.myfaces, while building the Metamer application).

In order to run the test suite against e.g. EAP with MyFaces (aka modified EAP), you have at least these possibilities how to do that, you can:

  • run the test suite with property jbossEAP{XY}Home pointing to the home directory of the modified EAP (property is already explained in this document)

  • run the test suite with property pathToEAPZip pointing to the zip file of the modified EAP (property is already explained in this document)

  • create a custom maven profile (e.g. jbosseap-remote) and use it with the started modified EAP (inspiration for the profile can be found at https://github.com/richfaces/richfaces/blob/4.5.17.Final/pom.xml#L1016-L1041).

8. Known Issues

Issue Problem description and probably solution

browser window does not start in given timeout

Unknown reason. You can:

  • increase the timeout, configurable through Arquillian Drone properties (refer to the Drone documentation)

RequestGuardException in tests

Selenium tries to click element which is not on screen. You can:

  • increase desktop resolution

  • move to the element before invoking the request (e.g. before clicking the element)

  • update Selenium and/or Arquillian dependencies

Updated Firefox and the tests does not run

There is a problem in Selenium. Each Selenium release works with limited number of Firefox releases (see Platforms Supported by Selenium — support for Firefox is the latest release, the previous release, the latest ESR release and the previous ESR release). You can:

  • use older Firefox, e.g. add property -Dbrowser=firefox45esr to the maven command

  • update Selenium and/or Arquillian dependencies

  • upgrade Firefox driver to Marionette

Related Selenium issues:

UnreachableBrowserException during the test execution

Browser process probably died (or was killed) during the test execution. Might be a problem of particular machine running tests.

  • watch browser process during test execution to find whether it fails always on the same test, after some time, memory consumption of browser, etc.

View cannot be restored or ViewExpiredException in tests

This happens when a HTTP request is sent while there was already an ajax request in progress. You can:

  • use Graphene.guardAjax() on the action which resolves in ajax request to wait until the request is done

  • wait for Metamer’s page status component (RichFaces component placed in header of the Metamer page, which extends the Metamer template template="/templates/template.xhtml", to track the process of the ajax requests) to change to the stop state with method getMetamerPage().getStatus().advanced().waitUntilStatusStateChanges(StatusState.STOP). This solution can be less reliable than previous step

Test fails randomly with something like "Request type 'XHR' was expected, but type 'NONE' was done instead"

Most of tests can run when browser window is not focused, i.e. when it is not top-level window. However, tests using focus and blur browser events require that the browser window is active.

  • Disable all programs that have pop-up windows (e.g. Windows update) while running tests.

  • Another workaround is to use Xvfb, start your tests using xvfb-run mvn clean verify…​

The test suite freezes before the Metamer war is deployed to EAP 6.4

This problem is caused when enabling WebSocket protocol in EAP in method enableWebSocketsInEAP63_or_64(war, temporaryJBossWebXML); in AbstractMetamerTest. You can:

  • skip enabling of the WebSocket protocol by using system property eap.ws.enabled=false, beware that it can influence the stability of tests in a4jPush package

Assertion in test fails randomly even though the logic of test is correct and there is no bug in JSF/RichFaces.

There is probably some race condition

  • use Graphene.guardAjax() on the action which resolves in Ajax request to wait until the request is done

  • in some cases, JavaScript needs to be executed after successful Ajax request so you need additional wait so that page is updated

  • workaround is to mark the test with annotation @Unstable (see section About annotations)

org.openqa.selenium.TimeoutException: Timed out after 4 seconds: (some reason)

The application doesn’t behave as expected

  • double-check that the logic of test is correct

  • it might indicate that server and/or browser is too slow to execute desired actions, try to watch the test execution

  • workaround is to mark the test with annotation @Unstable (see section About annotations)

  • it might also indicate that the test is dependent on focus or blur events, try runnit your tests using xvfb-run mvn clean verify…​

ElementNotVisibleException is thrown during test execution

Selenium cannot interact with not visible elements on page (i.e. when hidden by CSS or when inside a container with scrollers)

  • make sure that you are not trying to use element that is supposed to be hidden

  • use Graphene.guardAjax() on the action which resolves in Ajax request to wait until the request is done

  • make sure to use high enough resolution of your display so that majority of page is visible, especially if your test is running inside VNC (in view-only mode) or Xvfb

  • if your test is failing inside template richPopupPanel (or its alias popup), the problem will be probably caused by scrollers inside popup panel, Selenium cannot scroll the container (div HTML element), try enlarging the popup panel in metamer/application/src/main/webapp/templates/richPopupPanel.xhtml