Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP] spdlog for multi-level multi-sink logging #462

Open
wants to merge 6 commits into
base: dev
Choose a base branch
from

Conversation

C0nsultant
Copy link
Member

@C0nsultant C0nsultant commented Feb 6, 2019

TBD

Close #461.

To Do

  • push implementation
  • use spdlog 1.4+ and include link compiled .cpp file instead of increasing compile-time via header-only lib?
  • Docs:
    • README.md
    • docs/source/dev/buildoptions.rst
    • docs/source/dev/dependencies.rst

@C0nsultant C0nsultant added tools third party third party libraries that are shipped and/or linked internal backend labels Feb 6, 2019
@C0nsultant C0nsultant self-assigned this Feb 6, 2019
@C0nsultant C0nsultant requested a review from ax3l February 6, 2019 15:22
.travis.yml Outdated Show resolved Hide resolved
@C0nsultant C0nsultant force-pushed the topic-logging branch 2 times, most recently from e01c674 to 36d9c9c Compare February 6, 2019 15:33
@ax3l ax3l assigned ax3l and unassigned C0nsultant Feb 6, 2019
.travis.yml Outdated
# - &style_cpp
# stage: 'Style'
# name: clang-format@5.0.0
# language: python
# python: "2.7"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Python 3 is only needed if the python bindings are build

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well in this case, we require Python due to run-clang-format.py. It's 2019 already and I'd say it's time to leave 2.7 behind altogether.

Copy link
Member

@ax3l ax3l Feb 7, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, then you only need to change the entries in the matrix that use run-clang-format.py if it is not backwards compatible :)
But I think all of this can be done in another PR and is independent of this change, no?
update: just saw the comment above, yep.

CMakeLists.txt Outdated
@@ -292,6 +294,15 @@ if(openPMD_HAVE_PYTHON)
INTERFACE pybind11::pybind11)
endif()

# external library: spdlog (optional)
if(openPMD_USE_LOGGING STREQUAL AUTO)
set(openPMD_HAVE_LOGGING (${CMAKE_BUILD_TYPE} STREQUAL "Debug" OR ${CMAKE_BUILD_TYPE} STREQUAL "RelWithDebInfo"))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can do the control this way, but it's slightly different as the other dependencies then.

For example, if the openPMD_USE_XY is AUTO and set to EXTERNAL and not found, this will not fail in our current logic. Here it would.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shamelessly copied from Pybind11 as that probably is the most consistent with the other CMake _USE_ logic.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I just wonder if we should couple it to the Debug build logic since we will use log levels on various intensity also in Release builds and vice versa, might want to disable it for some Debug runs.

So maybe just handle it's default activation just like any other functional extension? Just a thought.

@ax3l ax3l mentioned this pull request Feb 19, 2019
@C0nsultant C0nsultant force-pushed the topic-logging branch 13 times, most recently from 54c01e3 to 59c592c Compare February 20, 2019 07:58
-------

By default, logging is only enabled in ``Debug`` builds.
In order to enable this feature unconditionally, pass ``-DopenPMD_USE_LOGGING=ON`` or ``-DopenPMD_USE_LOGGING=OFF`` to disable it unconditinoally.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

little typo:

Suggested change
In order to enable this feature unconditionally, pass ``-DopenPMD_USE_LOGGING=ON`` or ``-DopenPMD_USE_LOGGING=OFF`` to disable it unconditinoally.
In order to enable this feature unconditionally, pass ``-DopenPMD_USE_LOGGING=ON`` or ``-DopenPMD_USE_LOGGING=OFF`` to disable it unconditionally.


enum class Level
{
TRACE,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we also add a "timing" mode in this or another level? "throughput" (bytes/s) would also be awesome.
cc @franzpoeschel

Copy link
Member Author

@C0nsultant C0nsultant Feb 26, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Interesting idea. That would mean we count bytes during all backend IO and take timestamps in the fronend triggering calls.

Intuitively, this would result in a different code structure to the other log facilities.

auto series = openPMD::Series("data%T.h5", openPMD::AccessType::CREATE);
RecordComponent ideas = ...

std::shared_prt< double > patch = ...

// assume DEFER
ideas.store_chunk(patch, {0}, {patch_size});
LOG_TIME(series.flush());     /* example.cpp:8 series.flush() took 0.314s */

std::array< std::shared_prt< double >, 42 > lots_of_patches = ...
for( auto patch : lots_of_patches )
    ideas.store_chunk(patch, ...);
LOG_THROUGHPUT(series.flush());  /* example.cpp:13 series.flush() averaged 314MB/s */

I'll see what I can do. This will be pretty invasive, though.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Doesn't need to go into the first log version, we can add those in a follow-up if you like.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backend internal third party third party libraries that are shipped and/or linked tools
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Logging / Tracing / Debugging facilities
3 participants