OgsTester. Add test definitions to project files. #2255

endJunction · 2018-11-04T23:28:02Z

Move test definitions into the project files and add a "test mode" to the ogs executable.

bilke · 2018-11-05T09:45:34Z

Note to self:

Remove output
Globbing

chleh

Looks like a good feature: Easier to read than Tests.cmake. And can be extended without having to dig into CMake.

Some general comments:

Renaming of the reference files s.t. they match the solution files seems to be mandatory. That might be inconvenient if one develops a test case or if one wants to compare several time steps to an analytical solution.
There should be proper escaping for the system() calls. Otherwise, e.g., external people could hijack our CI servers by making a PR. OK, they can hijack them anyway, e.g., by adding something nasty to the cmake files. 😄

chleh · 2018-11-05T08:58:44Z

Applications/CLI/ogs.cpp

@@ -86,7 +86,7 @@ int main(int argc, char* argv[])
 #else
                                               "all",
 #endif
-                                               "log level");


That's for the cmdline help text?
Maybe also PROJECT_FILE in L71?

chleh · 2018-11-05T09:00:54Z

Applications/ApplicationsLib/TestDefinition.cpp

+std::string convert_to_string(double const& value)
+{
+    // TODO (naumov) Replace this with fmt library.
+    char buffer[1024];


pretty much for a formatted double value. 😄

chleh · 2018-11-05T09:04:10Z

Applications/ApplicationsLib/TestDefinition.cpp

+    // TODO (naumov) Replace this with fmt library.
+    char buffer[1024];
+    int const chars_written = std::snprintf(buffer, 1024, "%g", value);
+    if (chars_written < 0 || chars_written > 1024)


Off by one: > 1023.
https://en.cppreference.com/w/cpp/io/c/fprintf:

Number of characters that would have been written for a sufficiently large buffer if successful (not including the terminating null character), or a negative value if an error occurred. Thus, the (null-terminated) output has been completely written if and only if the returned value is nonnegative and less than buf_size.

chleh · 2018-11-05T09:04:30Z

Applications/ApplicationsLib/TestDefinition.cpp

+/// Wraps a string into double ticks.
+std::string safeString(std::string s)
+{
+    return "\"" + s + "\"";


Only safe if all " in s are escaped.

chleh · 2018-11-05T09:06:43Z

Applications/ApplicationsLib/TestDefinition.cpp

+        //
+        int const return_value =
+            // TODO (naumov) replace system call with output consuming call.
+            // and extract the vtkdiff call to common function.


... and properly escape any strings occuring in command lines.

chleh · 2018-11-05T09:28:52Z

Tests/Data/Mechanics/Burgers/cube_1e0.prj

+    <test_definition>
+        <vtkdiff>
+            <file>cube_1e0_pcs_0_ts_1_t_0.000100.vtu</file>
+            <field>displacement</field>


As mentioned before: ref_field and field might be good.

chleh · 2018-11-05T09:32:11Z

Tests/Data/Mechanics/Linear/PressureBC/axisymmetric_pipe.prj

@@ -12,6 +12,8 @@
                <youngs_modulus>E</youngs_modulus>
                <poissons_ratio>nu</poissons_ratio>
            </constitutive_relation>
+            <solid_density>rho_sr</solid_density>
+            <specific_body_force>0 0 0</specific_body_force>


So, this was never tested before?

LIS only, and there is no LIS build in CI ;(

chleh · 2018-11-05T09:35:56Z

scripts/cmake/test/OgsTest.cmake

+    endif()
+
+    set(OgsTest_SOURCE_DIR "${Data_SOURCE_DIR}/${OgsTest_DIR}")
+    set(OgsTest_BINARY_DIR "${Data_BINARY_DIR}/${OgsTest_DIR}")


Does it work if OGS is installed systemwide? Or if a test resides outside the source/build directory?
OK, the cmake stuff won't work without source/build dirs. But does the feature itself (ogs --reference ...) work? Or do you need to set some environment variables s.t. it works?

Works as promised from anywhere. Only if vtkdiff is not in current PATH and not in ./bin the VTKDIFF_BINARY should be set...

chleh · 2018-11-05T09:54:35Z

scripts/cmake/test/AddTest.cmake

@@ -61,10 +61,12 @@ function (AddTest)
        set (AddTest_REQUIREMENTS TRUE)
    endif()

+    set(X_PRJ_FILE ${AddTest_EXECUTABLE_ARGS})  # For easier retrival later on


Easier retrieval of what?

chleh · 2018-11-05T09:58:33Z

Applications/ApplicationsLib/TestDefinition.cpp

+                  }
+                  return return_value;
+              });
+    return all_of(begin(return_values), end(return_values),


IMHO the test should fail if return_values is empty (configuration error, no tests defined).

bilke · 2018-11-05T13:58:20Z

@endJunction @chleh Jenkins CI jobs are submitted for members of our GitHub ufz-organization only (https://github.com/orgs/ufz/teams/members/members).

chleh · 2018-11-05T15:23:15Z

@bilke, great!

bilke · 2018-11-09T09:25:14Z

@endJunction I guess for a clean implementation of file globbing we need either boost::filesystem or std::filesystem. Maybe this can be discussed and addressed later on.

chleh · 2018-11-13T13:16:51Z

Applications/ApplicationsLib/TestDefinition.cpp

+        std::string const absolute_tolerance_parameter =
+            absolute_tolerance == boost::none
+                ? ""
+                : "--abs " + convert_to_string(*absolute_tolerance);


btw: the precedence of + over : is probably not working knowledge of everybody.

endJunction · 2018-11-14T11:13:48Z

The conversion scripts, previously added in the AddTest.cmake are in the last commit on OgsTesterConversionScripts branch.

TomFischer · 2018-11-14T11:27:22Z

Applications/ApplicationsLib/TestDefinition.cpp

+    if (reference_path.empty())
+    {
+        OGS_FATAL(
+            "Reference path containting expected result files can not be "


containting -> containing ✔️

Add --reference PATH option to ogs, which when present will execute tests (vtkdiff) constructed from the <test_definiton> section in the project file. This section contains the tester (vtkdiff for now) the file, field, and absolute and relative tolerances. This allows the user to run tests w/o cmake and includes the previously separated test description into the project file.

Convert sigma_.. to vectorial sigma. Update output prefix. Move search length to top.

TomFischer

Looks good.

wenqing

👍

wenqing · 2018-11-14T13:48:08Z

Applications/ApplicationsLib/TestDefinition.cpp

+namespace
+{
+/// Safe conversion of a double to a string using decimal or decimal exponent
+/// notation. See std::snprintf() for details for "%g" conversion specifier.


How about std::ostringstream?

Good point. Even better, I rewrote that part to avoid any conversion. Could you please check the changes in the last commit?

The new changes look good.

wenqing · 2018-11-14T15:37:12Z

Applications/ApplicationsLib/TestDefinition.cpp

            //! \ogs_file_param{prj__test_definition__vtkdiff__absolute_tolerance}
-            vtkdiff_config.getConfigParameterOptional<double>(
-                "absolute_tolerance");
+            vtkdiff_config.getConfigParameter<std::string>("absolute_tolerance",


avoided the conversion by this way, 👍.

Pass the value directly instead and only check if the string convertible to a double if not empty. This avoids possible losses due to conversion.

wenqing

Just compiled with gnu compiler, got one warning and one error.

wenqing · 2018-11-16T10:27:07Z

Applications/ApplicationsLib/TestDefinition.cpp

+            pos, s.c_str());
+    }
+
+    if (std::isnan(value))


compilation error with g++: error: ‘isnan’ is not a member of ‘std’

This has been fixed here 66eb63a

@bilke Thanks.

wenqing · 2018-11-16T10:29:18Z

Applications/ApplicationsLib/TestDefinition.cpp

+bool isConvertibleToDouble(std::string const& s)
+{
+    std::size_t pos = 0;
+    double value;


compilation warning: warning: variable ‘value’ set but not used.

This warning is gone now.

ogsbot · 2020-06-19T12:55:44Z

OpenGeoSys development has been moved to GitLab.

See this pull request on GitLab.

endJunction added discussion 🗣️ WIP 🏗️ stop review 🛑 labels Nov 4, 2018

chleh reviewed Nov 5, 2018

View reviewed changes

endJunction mentioned this pull request Nov 6, 2018

Misc for OgsTester #2257

Merged

endJunction removed the discussion 🗣️ label Nov 9, 2018

endJunction changed the title ~~ogs tester. [Discussion]~~ OgsTester. Add test definitions to project files. Nov 13, 2018

chleh reviewed Nov 13, 2018

View reviewed changes

endJunction added please review and removed stop review 🛑 WIP 🏗️ labels Nov 14, 2018

TomFischer reviewed Nov 14, 2018

View reviewed changes

endJunction and others added 11 commits November 14, 2018 14:07

[BL] Add removeFiles().

7d00c92

[BL] ConfigTree; Add size() to Range's.

313ea74

Add OgsTest.cmake for in prj-embedded tests.

59f32c4

[T] SD; Add <test_definition> to project files.

f453f15

[T] SD; Relax two_material_gravity tol for LIS.

06b9232

[T] SD; Update Ehlers/axisymmetric sphere test.

0249bcc

Convert sigma_.. to vectorial sigma. Update output prefix. Move search length to top.

[T] SD; Rename reference files to match output.

a4e50a0

[PL] SD; Use OgsTest().

c562c58

[doc] Tag descriptions for test_definition subtree

ac0ea39

Use TARGET_FILE generator expression for vtkdiff.

a589380

TomFischer approved these changes Nov 14, 2018

View reviewed changes

wenqing approved these changes Nov 14, 2018

View reviewed changes

wenqing reviewed Nov 14, 2018

View reviewed changes

endJunction removed the please review label Nov 14, 2018

[App] TD; Do not convert tolerances to double.

a0e987b

Pass the value directly instead and only check if the string convertible to a double if not empty. This avoids possible losses due to conversion.

endJunction merged commit 8dd1ce4 into ufz:master Nov 14, 2018

endJunction deleted the OgsTester branch November 14, 2018 22:03

wenqing reviewed Nov 16, 2018

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

OgsTester. Add test definitions to project files. #2255

OgsTester. Add test definitions to project files. #2255

endJunction commented Nov 4, 2018

bilke commented Nov 5, 2018 •

edited

Loading

chleh left a comment •

edited

Loading

chleh Nov 5, 2018

chleh Nov 5, 2018

chleh Nov 5, 2018

chleh Nov 5, 2018

chleh Nov 5, 2018

chleh Nov 5, 2018

chleh Nov 5, 2018

endJunction Nov 5, 2018

chleh Nov 5, 2018

endJunction Nov 5, 2018

chleh Nov 5, 2018

chleh Nov 5, 2018

bilke commented Nov 5, 2018

chleh commented Nov 5, 2018

bilke commented Nov 9, 2018

chleh Nov 13, 2018

endJunction commented Nov 14, 2018

TomFischer Nov 14, 2018 •

edited by endJunction

Loading

TomFischer left a comment

wenqing left a comment

wenqing Nov 14, 2018 •

edited

Loading

endJunction Nov 14, 2018

wenqing Nov 14, 2018

wenqing Nov 14, 2018

wenqing left a comment

wenqing Nov 16, 2018

bilke Nov 16, 2018

wenqing Nov 16, 2018

wenqing Nov 16, 2018

wenqing Nov 20, 2018

ogsbot commented Jun 19, 2020

OgsTester. Add test definitions to project files. #2255

OgsTester. Add test definitions to project files. #2255

Conversation

endJunction commented Nov 4, 2018

bilke commented Nov 5, 2018 • edited Loading

chleh left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bilke commented Nov 5, 2018

chleh commented Nov 5, 2018

bilke commented Nov 9, 2018

Choose a reason for hiding this comment

endJunction commented Nov 14, 2018

TomFischer Nov 14, 2018 • edited by endJunction Loading

Choose a reason for hiding this comment

TomFischer left a comment

Choose a reason for hiding this comment

wenqing left a comment

Choose a reason for hiding this comment

wenqing Nov 14, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wenqing left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ogsbot commented Jun 19, 2020

bilke commented Nov 5, 2018 •

edited

Loading

chleh left a comment •

edited

Loading

TomFischer Nov 14, 2018 •

edited by endJunction

Loading

wenqing Nov 14, 2018 •

edited

Loading