Support for junit xml generated from ctest #2203

pkp124 · 2023-11-14T21:39:34Z

pkp124
Nov 14, 2023

Hi all,

I am quite new to Allure and I am trying to understand its capabilities. My project is C++ based, we work a lot with ctest/gtest frameworks. Is it possible to use allure html reporting for a junit xml file generated from the tool CTest or Gtest.

Thanks

baev · 2023-11-16T12:51:30Z

baev
Nov 16, 2023
Maintainer

Allure supports multiple dialects of junit.xml format out-of-the-box — specify the directory with XML files to the report generator, and it will generate the report.

Although CTest or Gtest XML results format may differ. In that case, kindly provide an example of the result so we can see the difference and consider adding support to it.

0 replies

pkp124 · 2023-12-15T08:12:53Z

pkp124
Dec 15, 2023
Author

Thanks for the reply :) Really appreciate it ! I did try the ctest generated junit xml with allure. I only have one complaint and that is in the timeline. I have tried allure with pytest and I understand that, maybe the timeline depends on some allure generated data. But it would be really great, if allure can support ctest to the same extent as pytest. :)

For your reference, attached are the following:

report.xml : junit report from a small ctest run
allure test reports: generated using the command line: allure generate . -o allure-report-ctest --clean

Allure version used: 2.24.1

Please let me know if you need any more info.
allure-report-ctest.zip

5 replies

pkp124 Dec 15, 2023
Author

Some more information, Ctest also produces some additional information in the form of xml files. One such xml file for the attached results of the previous comment is shown here:

<?xml version="1.0" encoding="UTF-8"?>
<Site BuildName="Test build"
	BuildStamp="20231214-2200-Nightly"
	Name="Test"
	Generator="ctest-3.21.3"
	Append="true"
	CompilerName=""
	CompilerVersion=""
	OSName="Linux"
	Hostname="test"
	OSRelease="3.10.0-1160.31.1.el7.x86_64"
	OSVersion="#1 SMP Thu Jun 10 13:32:12 UTC 2021"
	OSPlatform="x86_64"
	Is64Bits="1"
	VendorString="GenuineIntel"
	VendorID="Intel Corporation"
	FamilyID="6"
	ModelID="85"
	ProcessorCacheSize="16896"
	NumberOfLogicalCPU="24"
	NumberOfPhysicalCPU="24"
	TotalVirtualMemory="192054"
	TotalPhysicalMemory="191902"
	LogicalProcessorsPerPhysical="1"
	ProcessorClockFrequency="2700.08"
	>
	<Testing>
		<StartDateTime>Dec 15 09:14 CET</StartDateTime>
		<StartTestTime>1702628079</StartTestTime>
		<TestList>
			<Test>./test_B</Test>
			<Test>./test_A</Test>
			<Test>./test_C</Test>
			<Test>./test_D</Test>
		</TestList>
		<Test Status="passed">
			<Name>test_B</Name>
			<Path>.</Path>
			<FullName>./test_B</FullName>
			<FullCommandLine>/usr/bin/sleep "20s"</FullCommandLine>
			<Results>
				<NamedMeasurement type="numeric/double" name="Execution Time">
					<Value>20.0024</Value>
				</NamedMeasurement>
				<NamedMeasurement type="numeric/double" name="Processors">
					<Value>1</Value>
				</NamedMeasurement>
				<NamedMeasurement type="text/string" name="Completion Status">
					<Value>Completed</Value>
				</NamedMeasurement>
				<NamedMeasurement type="text/string" name="Command Line">
					<Value>/usr/bin/sleep "20s"</Value>
				</NamedMeasurement>
				<NamedMeasurement type="text/string" name="Environment">
					<Value>CTEST_RESOURCE_GROUP_COUNT=0</Value>
				</NamedMeasurement>
				<Measurement>
					<Value></Value>
				</Measurement>
			</Results>
		</Test>
		<Test Status="passed">
			<Name>test_A</Name>
			<Path>.</Path>
			<FullName>./test_A</FullName>
			<FullCommandLine>/usr/bin/sleep "50s"</FullCommandLine>
			<Results>
				<NamedMeasurement type="numeric/double" name="Execution Time">
					<Value>50.0017</Value>
				</NamedMeasurement>
				<NamedMeasurement type="numeric/double" name="Processors">
					<Value>1</Value>
				</NamedMeasurement>
				<NamedMeasurement type="text/string" name="Completion Status">
					<Value>Completed</Value>
				</NamedMeasurement>
				<NamedMeasurement type="text/string" name="Command Line">
					<Value>/usr/bin/sleep "50s"</Value>
				</NamedMeasurement>
				<NamedMeasurement type="text/string" name="Environment">
					<Value>CTEST_RESOURCE_GROUP_COUNT=0</Value>
				</NamedMeasurement>
				<Measurement>
					<Value></Value>
				</Measurement>
			</Results>
		</Test>
		<Test Status="passed">
			<Name>test_C</Name>
			<Path>.</Path>
			<FullName>./test_C</FullName>
			<FullCommandLine>/usr/bin/sleep "20s"</FullCommandLine>
			<Results>
				<NamedMeasurement type="numeric/double" name="Execution Time">
					<Value>20.0017</Value>
				</NamedMeasurement>
				<NamedMeasurement type="numeric/double" name="Processors">
					<Value>1</Value>
				</NamedMeasurement>
				<NamedMeasurement type="text/string" name="Completion Status">
					<Value>Completed</Value>
				</NamedMeasurement>
				<NamedMeasurement type="text/string" name="Command Line">
					<Value>/usr/bin/sleep "20s"</Value>
				</NamedMeasurement>
				<NamedMeasurement type="text/string" name="Environment">
					<Value>CTEST_RESOURCE_GROUP_COUNT=0</Value>
				</NamedMeasurement>
				<Measurement>
					<Value></Value>
				</Measurement>
			</Results>
		</Test>
		<Test Status="passed">
			<Name>test_D</Name>
			<Path>.</Path>
			<FullName>./test_D</FullName>
			<FullCommandLine>/usr/bin/sleep "20s"</FullCommandLine>
			<Results>
				<NamedMeasurement type="numeric/double" name="Execution Time">
					<Value>20.0036</Value>
				</NamedMeasurement>
				<NamedMeasurement type="numeric/double" name="Processors">
					<Value>1</Value>
				</NamedMeasurement>
				<NamedMeasurement type="text/string" name="Completion Status">
					<Value>Completed</Value>
				</NamedMeasurement>
				<NamedMeasurement type="text/string" name="Command Line">
					<Value>/usr/bin/sleep "20s"</Value>
				</NamedMeasurement>
				<NamedMeasurement type="text/string" name="Environment">
					<Value>CTEST_RESOURCE_GROUP_COUNT=0</Value>
				</NamedMeasurement>
				<Measurement>
					<Value></Value>
				</Measurement>
			</Results>
		</Test>
		<EndDateTime>Dec 15 09:16 CET</EndDateTime>
		<EndTestTime>1702628169</EndTestTime>
		<ElapsedMinutes>1</ElapsedMinutes>
	</Testing>
</Site>

So probably, it is still possible to get a timeline out of ctest generated output files?

baev Dec 18, 2023
Maintainer

xsd:

<?xml version="1.0" standalone="yes"?>
<xs:schema id="NewDataSet" xmlns="" xmlns:xs="http://www.w3.org/2001/XMLSchema">
  <xs:element name="Site">
    <xs:complexType>
      <xs:sequence>
        <xs:element name="Testing" minOccurs="0" maxOccurs="unbounded">
          <xs:complexType>
            <xs:sequence>
              <xs:element name="StartDateTime" type="xs:string" minOccurs="0" />
              <xs:element name="StartTestTime" type="xs:string" minOccurs="0" />
              <xs:element name="TestList" minOccurs="0" maxOccurs="unbounded">
                <xs:complexType>
                  <xs:sequence>
                    <xs:element name="Test" type="xs:string" minOccurs="0"  maxOccurs="unbounded"/>
                  </xs:sequence>
                </xs:complexType>
              </xs:element>
              <xs:element name="Test" minOccurs="0" maxOccurs="unbounded">
                <xs:complexType>
                  <xs:sequence>
                    <xs:element name="Name" type="xs:string" minOccurs="0" />
                    <xs:element name="Path" type="xs:string" minOccurs="0"  />
                    <xs:element name="FullName" type="xs:string" minOccurs="0"  />
                    <xs:element name="FullCommandLine" type="xs:string" minOccurs="0" />
                    <xs:element name="Results" minOccurs="0" maxOccurs="unbounded">
                      <xs:complexType>
                        <xs:sequence>
                          <xs:element name="NamedMeasurement" minOccurs="0" maxOccurs="unbounded">
                            <xs:complexType>
                              <xs:sequence>
				      <xs:element name="Value" type="xs:string" minOccurs="0" />
                              </xs:sequence>
                              <xs:attribute name="type" type="xs:string" />
                              <xs:attribute name="name" type="xs:string" />
                            </xs:complexType>
                          </xs:element>
                          <xs:element name="Measurement" minOccurs="0" maxOccurs="unbounded">
                            <xs:complexType>
                              <xs:sequence>
                                <xs:element name="Value" type="xs:string" minOccurs="0" />
                              </xs:sequence>
                            </xs:complexType>
                          </xs:element>
                        </xs:sequence>
                      </xs:complexType>
                    </xs:element>
                  </xs:sequence>
                  <xs:attribute name="Status" type="xs:string" />
                </xs:complexType>
	</xs:element>
	<xs:element name="EndDateTime" type="xs:string" minOccurs="0" />
    <xs:element name="EndTestTime" type="xs:string" minOccurs="0" />
    <xs:element name="ElapsedMinutes" type="xs:string" minOccurs="0" /> 
            </xs:sequence>
          </xs:complexType>
        </xs:element>
      </xs:sequence>
      <xs:attribute name="BuildName" type="xs:string" />
      <xs:attribute name="BuildStamp" type="xs:string" />
      <xs:attribute name="Name" type="xs:string" />
      <xs:attribute name="Generator" type="xs:string" />
      <xs:attribute name="OSName" type="xs:string" />
      <xs:attribute name="Hostname" type="xs:string" />
      <xs:attribute name="OSRelease" type="xs:string" />
      <xs:attribute name="OSVersion" type="xs:string" />
      <xs:attribute name="OSPlatform" type="xs:string" />
      <xs:attribute name="Is64Bits" type="xs:string" />
      <xs:attribute name="VendorString" type="xs:string" />
      <xs:attribute name="VendorID" type="xs:string" />
      <xs:attribute name="FamilyID" type="xs:string" />
      <xs:attribute name="ModelID" type="xs:string" />
      <xs:attribute name="ProcessorCacheSize" type="xs:string" />
      <xs:attribute name="NumberOfLogicalCPU" type="xs:integer" />
      <xs:attribute name="NumberOfPhysicalCPU" type="xs:integer" />
      <xs:attribute name="TotalVirtualMemory" type="xs:integer" />
      <xs:attribute name="TotalPhysicalMemory" type="xs:integer" />
      <xs:attribute name="LogicalProcessorsPerPhysical" type="xs:integer" />
      <xs:attribute name="ProcessorClockFrequency" type="xs:float" />
    </xs:complexType>
  </xs:element>
  <xs:element name="NewDataSet">
    <xs:complexType>
      <xs:choice minOccurs="0" maxOccurs="unbounded">
        <xs:element ref="Site" />
      </xs:choice>
    </xs:complexType>
  </xs:element>
</xs:schema>

pkp124 Dec 18, 2023
Author

Just to be clear, this is a separate xml file(not junit file) generated by CTest. I was working towards adding this information to the junit xml output of CTest itself, as mentioned in other comments.

baev Dec 18, 2023
Maintainer

We can support that in Allure, that's relatively simple

pkp124 Dec 18, 2023
Author

Oh cool :) Would be great if we can support plots of measurements which are numbers or float.

pkp124 · 2023-12-15T08:27:30Z

pkp124
Dec 15, 2023
Author

Having said that, i see individual tests dont have a start and end time stamps. Will try to get CTest to add this. :)

I just feel allure reports are quite extensive and readable. Lot of C++ projects use Ctest as their test runner and use CDash as the test dashboard. Think it would be good to have this support going both ways :)

0 replies

pkp124 · 2023-12-18T15:17:13Z

pkp124
Dec 18, 2023
Author

Hi @baev ,

Thanks for your response :) I have opened some tickets with CMake community to add some extra features to the junit xml report generated, mentioned below

is a "measurements" property, which is something supported by the CTest + CDash stack to visualize some custom information in CDash. Please see: https://cmake.org/cmake/help/latest/command/ctest_test.html#additional-test-measurements
is "start" and "end" timestamp property per testcase

I would request your/community's guidance on these requirements. Would adding this information to the "properties" element of the test case in the junit xml work?

6 replies

pkp124 Dec 18, 2023
Author

So let me put it another way, What would be needed to support timeline for CTest runs? :)

baev Dec 18, 2023
Maintainer

However, it's the same for CTest.XML. To build the Timeline, we need start & stop timestamps to be specified per test

baev Dec 18, 2023
Maintainer

The best scenario is to have a Unix Epoch timestamp in milliseconds per Test node & duration in milliseconds (or separate stop timestamp). We can parse any other timestamp format, but the limitation of Allure is that you can't specify it during the report generation. It needs to be the same all the time, or we will do our best trying to guess the timezone, etc.

pkp124 Dec 18, 2023
Author

So to summarize, would you agree that adding start and stop time stamp and timezone to the juit xml testcase property be a decent enough way of providing this info to allure?

baev Dec 18, 2023
Maintainer

I would rather support CText.XML, but we can add such a possibility to JUnit.XML as well. The issue with JUnit XML is that there are plenty (10+) different dialects, and no clear schema/schema owner.

baev · 2023-12-18T16:16:00Z

baev
Dec 18, 2023
Maintainer

JUnit XML plugin for Allure: https://github.com/allure-framework/allure2/blob/master/plugins/junit-xml-plugin/src/main/java/io/qameta/allure/junitxml/JunitXmlPlugin.java

1 reply

SaturnIC May 20, 2024

What is the use of this file?

pkp124 · 2023-12-28T14:45:23Z

pkp124
Dec 28, 2023
Author

Hi @baev ,

What do you think about ctest support for allure, from the info provided above and whatever ctest offers currently :)
Do you need anything specific from me? Please let me know if i can help in some way.

Thanks
Phani

1 reply

baev Jan 8, 2024
Maintainer

Until there are no test start/stop timestamps available in any of the results, we can't do much from Allure's side.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Allure Framework

Support for junit xml generated from ctest #2203

{{title}}

Replies: 6 comments 13 replies

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

Allure Framework

Support for junit xml generated from ctest #2203

pkp124 Nov 14, 2023

Replies: 6 comments · 13 replies

baev Nov 16, 2023 Maintainer

pkp124 Dec 15, 2023 Author

pkp124 Dec 15, 2023 Author

baev Dec 18, 2023 Maintainer

pkp124 Dec 18, 2023 Author

baev Dec 18, 2023 Maintainer

pkp124 Dec 18, 2023 Author

pkp124 Dec 15, 2023 Author

pkp124 Dec 18, 2023 Author

pkp124 Dec 18, 2023 Author

baev Dec 18, 2023 Maintainer

baev Dec 18, 2023 Maintainer

pkp124 Dec 18, 2023 Author

baev Dec 18, 2023 Maintainer

baev Dec 18, 2023 Maintainer

SaturnIC May 20, 2024

pkp124 Dec 28, 2023 Author

baev Jan 8, 2024 Maintainer

pkp124
Nov 14, 2023

Replies: 6 comments 13 replies

baev
Nov 16, 2023
Maintainer

pkp124
Dec 15, 2023
Author

pkp124 Dec 15, 2023
Author

baev Dec 18, 2023
Maintainer

pkp124 Dec 18, 2023
Author

baev Dec 18, 2023
Maintainer

pkp124 Dec 18, 2023
Author

pkp124
Dec 15, 2023
Author

pkp124
Dec 18, 2023
Author

pkp124 Dec 18, 2023
Author

baev Dec 18, 2023
Maintainer

baev Dec 18, 2023
Maintainer

pkp124 Dec 18, 2023
Author

baev Dec 18, 2023
Maintainer

baev
Dec 18, 2023
Maintainer

pkp124
Dec 28, 2023
Author

baev Jan 8, 2024
Maintainer