Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add lcov code coverage tool #467

Merged
merged 5 commits into from
Jun 16, 2020
Merged

Add lcov code coverage tool #467

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
5140bdd
add lcov capability
apcraig Jun 11, 2020
c85dca3
update documentation
apcraig Jun 11, 2020
e19220c
update report name and lcov web site org
apcraig Jun 12, 2020
9edb18c
remove space in /bin/csh declaration
apcraig Jun 12, 2020
97db1ad
resolve conflicts with master
apcraig Jun 16, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
update documentation
  • Loading branch information
@apcraig
apcraig committed Jun 16, 2020
commit c85dca3b9a7d3996b648dbb537b9bbf99c064f1b
3 changes: 2 additions & 1 deletion README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
[![Build Status](https://travis-ci.org/CICE-Consortium/CICE.svg?branch=master)](https://travis-ci.org/CICE-Consortium/CICE)
[![Documentation Status](https://readthedocs.org/projects/cice-consortium-cice/badge/?version=master)](http://cice-consortium-cice.readthedocs.io/en/master/?badge=master)
[![codecov](https://codecov.io/gh/apcraig/Test_CICE_Icepack/branch/master/graph/badge.svg)](https://codecov.io/gh/apcraig/Test_CICE_Icepack)
..[![codecov](https://codecov.io/gh/apcraig/Test_CICE_Icepack/branch/master/graph/badge.svg)](https://codecov.io/gh/apcraig/Test_CICE_Icepack)
[![lcov](https://img.shields.io/endpoint?url=https://apcraig.github.io/coverage.json)]

## The CICE Consortium sea-ice model
CICE is a computationally efficient model for simulating the growth, melting, and movement of polar sea ice. Designed as one component of coupled atmosphere-ocean-land-ice global climate models, today’s CICE model is the outcome of more than two decades of community collaboration in building a sea ice model suitable for multiple uses including process studies, operational forecasting, and climate simulation.
Expand Down
7 changes: 5 additions & 2 deletions cice.setup
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -432,7 +432,9 @@ if ( ${dosuite} == 0 ) then

else
set tarrays = `echo ${testsuite} | sed 's/,/ /g' | fmt -1 | sort -u`
set testsuitecnt = 0
foreach tarray ( ${tarrays} )
@ testsuitecnt = ${testsuitecnt} + 1
if (-e ${tarray}) then
cat ${tarray} >> $tsfile
else if (-e ${tarray}.ts) then
Expand Down Expand Up @@ -516,8 +518,9 @@ cat >! ${tsdir}/report_codecov.csh << EOF0

source ${ICE_SCRIPTS}/machines/env.${machcomp}

set rn0 = "${shhash}:${sdate}-${stime}:${testsuite}"
set report_name = \`echo \${rn0} | sed -e 's/ //g'\`
set rn0 = "${shhash}:${sdate}-${stime}:${testsuitecnt}:${testsuite}"
set rn1 = \`echo \${rn0} | sed -e 's/ //g'\`
set report_name = \`echo \${rn1} | sed -e 's/_suite//g'\`

#for codecov
set use_curl = 1
Expand Down
87 changes: 55 additions & 32 deletions doc/source/user_guide/ug_testing.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -381,7 +381,7 @@ following options are valid for suites,
This is only used by ``--suite`` and when set, invokes a script that sends the test results to the results page when all tests are complete. Please see :ref:`testreporting` for more information.

``--coverage``
When invoked, code coverage diagnostics are generated. This will modify the build and reduce optimization. The results can then be processed by **report_codecov.csh** or **report_lcov.csh**. General use is not recommended, this is mainly used as a diagnostic to periodically assess test coverage. Please see :ref:`codecoverage` for more information.
When invoked, code coverage diagnostics are generated. This will modify the build and reduce optimization and generate coverage reports using lcov or codecov tools. General use is not recommended, this is mainly used as a diagnostic to periodically assess test coverage. Please see :ref:`codecoverage` for more information.

``--setup-only``
This is only used by ``--suite`` and when set, just creates the suite testcases. It does not build or submit them to run. By default, the suites do ``--setup-build-submit``.
Expand Down Expand Up @@ -667,40 +667,63 @@ Code Coverage Testing

The ``--coverage`` feature in **cice.setup** provides a method to diagnose code coverage.
This argument turns on special compiler flags including reduced optimization and then
invokes the gcov tool.
invokes the gcov tool. Once runs are complete, either lcov or codecov can be used
to analyze the results.
This option is currently only available with the gnu compiler and on a few systems
with modified Macros files.

Because codecov.io does not support git submodule analysis right now, a customized
repository has to be created to test CICE with Icepack integrated directly. The repository
https://github.com/apcraig/Test_CICE_Icepack serves as the current default test repository.
In general, to setup the code coverage test in CICE, the current CICE master has
to be copied into the Test_CICE_Icepack repository, then the full test suite
can be run with the gnu compiler with the ``--coverage`` argument.

The test suite will run and then a report will be generated and uploaded to
the `codecov.io site <https://codecov.io/gh/apcraig/Test_CICE_Icepack>`_ by the
**report_codecov.csh** script. The env variable CODECOV_TOKEN needs to be defined
either in the environment or in a file named **~/.codecov_cice_token**. That
token provides write permission to the Test_CICE_Icepack codecov.io site and is available
by contacting the Consortium team directly.

A script that carries out the end-to-end testing can be found in
**configuration/scripts/tests/cice_test_codecov.csh**

This is a special diagnostic test and does not constitute proper model testing.
General use is not recommended, this is mainly used as a diagnostic to periodically
assess test coverage. The interaction with codecov.io is not always robust and
can be tricky to manage. Some constraints are that the output generated at runtime
is copied into the directory where compilation took place. That means each
test should be compiled separately. Tests that invoke multiple runs
(such as exact restart and the decomp test) will only save coverage information
for the last run, so some coverage information may be lost. The gcov tool can
be a little slow to run on large test suites, and the codecov.io bash uploader
(that runs gcov and uploads the data to codecov.io) is constantly evolving.
Finally, gcov requires that the diagnostic output be copied into the git sandbox for
analysis. These constraints are handled by the current scripts, but may change
in the future.
At the present time, the ``--coverage`` flag invokes the lcov analysis automatically
by running the **report_lcov.csh** script in the test suite directory. The output
will show up at the `CICE lcov website <https://apcraig.github.io>`__. To
use the tool, you should have write permission for that repository. The lcov tool
should be run on a full multi-suite test suite, and it can
take several hours to process the data once the test runs are complete. A typical
instantiation would be
::

./cice.setup --suite first_suite,base_suite,travis_suite,decomp_suite,reprosum_suite,io_suite,quick_suite --mach cheyenne --env gnu --testid cc01 --coverage

Alternatively, codecov analysis can be carried out by manually running the **report_codecov.csh**
script from the test suite directory, but there are several ongoing problems with this approach and
it is not generally recommended. A script that summarizes the end-to-end process for codecov
analysis can be found in ..**configuration/scripts/tests/cice_test_codecov.csh**. The codecov
analysis is largely identical to the analysis performed by lcov, codecov just provides a nicer
web experience to view the output.

This is a special diagnostic test and is not part of the standard model testing.
General use is not recommended, this is mainly used as a diagnostic to periodically
assess test coverage.

..Because codecov.io does not support git submodule analysis right now, a customized
..repository has to be created to test CICE with Icepack integrated directly. The repository
..https://github.com/apcraig/Test_CICE_Icepack serves as the current default test repository.
..In general, to setup the code coverage test in CICE, the current CICE master has
..to be copied into the Test_CICE_Icepack repository, then the full test suite
..can be run with the gnu compiler with the ``--coverage`` argument.

..The test suite will run and then a report will be generated and uploaded to
..the `codecov.io site <https://codecov.io/gh/apcraig/Test_CICE_Icepack>`_ by the
..**report_codecov.csh** script. The env variable CODECOV_TOKEN needs to be defined
..either in the environment or in a file named **~/.codecov_cice_token**. That
..token provides write permission to the Test_CICE_Icepack codecov.io site and is available
..by contacting the Consortium team directly.

..A script that carries out the end-to-end testing can be found in
..**configuration/scripts/tests/cice_test_codecov.csh**

..This is a special diagnostic test and does not constitute proper model testing.
..General use is not recommended, this is mainly used as a diagnostic to periodically
..assess test coverage. The interaction with codecov.io is not always robust and
..can be tricky to manage. Some constraints are that the output generated at runtime
..is copied into the directory where compilation took place. That means each
..test should be compiled separately. Tests that invoke multiple runs
..(such as exact restart and the decomp test) will only save coverage information
..for the last run, so some coverage information may be lost. The gcov tool can
..be a little slow to run on large test suites, and the codecov.io bash uploader
..(that runs gcov and uploads the data to codecov.io) is constantly evolving.
..Finally, gcov requires that the diagnostic output be copied into the git sandbox for
..analysis. These constraints are handled by the current scripts, but may change
..in the future.


.. _compliance:
Expand Down