Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Morpheus changes related to utility consolidation #531

Merged
Merged
Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
69 commits
Select commit Hold shift + click to select a range
b2a976f
Start new update branch
drobison00 Dec 2, 2022
22d63fb
Checkpoint
drobison00 Dec 2, 2022
3aead15
Checkpoint
drobison00 Dec 2, 2022
da9cf59
Checkpoint
drobison00 Dec 2, 2022
3cfe2b9
Baseline conversion to utilities lib works
drobison00 Dec 2, 2022
99048ea
Checkpoint - transitioning morpheus _libs
drobison00 Dec 2, 2022
71493e2
More file consolidation
drobison00 Dec 2, 2022
12f0033
SRF->MRC changes compiling -- still some python errors
drobison00 Dec 6, 2022
4e034ac
Additional SRF->MRC conversions
drobison00 Dec 6, 2022
6d5e221
Add MRC config -- currently working if we install mrc directly and bu…
drobison00 Dec 7, 2022
81afaa0
Merge utilities updates
drobison00 Dec 7, 2022
6ee23e6
Update to use submodules
drobison00 Dec 8, 2022
d59e28d
Remove local cmake Configure
drobison00 Dec 8, 2022
a983414
Fix modules path
drobison00 Dec 8, 2022
bf953f4
Merge branch 'branch-23.01' into devin_issue_496_mrc_refactor
drobison00 Dec 9, 2022
d4dc62b
Build updates
drobison00 Dec 12, 2022
31fe7a5
Merge MRC migration base
drobison00 Dec 12, 2022
edfdea1
Update utilities repo tag, cleanup cmakelists
drobison00 Dec 12, 2022
1c55054
Consolidate scripts
drobison00 Dec 13, 2022
6e93745
Migrate more common scripts to utilities
drobison00 Dec 13, 2022
dabb100
Updates for final merge PR
drobison00 Dec 13, 2022
b91165e
Updates for final merge PR
drobison00 Dec 13, 2022
e60f5b9
Updates for final merge PR
drobison00 Dec 13, 2022
1cd0f04
Merge branch 'branch-23.01' into devin_issue_496_mrc_refactor
drobison00 Dec 13, 2022
1168775
Merge with mrc migration branch
drobison00 Dec 13, 2022
a6c885f
Update utilities
drobison00 Dec 13, 2022
b38058f
Formatting fixes
drobison00 Dec 13, 2022
c40ac86
Merge branch 'devin_issue_496_mrc_refactor' into devin_issue_87_mrc_r…
drobison00 Dec 13, 2022
7597924
Merge with branch-23.01
drobison00 Dec 13, 2022
918b5ab
Merge branch 'devin_issue_496_mrc_refactor' into devin_issue_87_mrc_r…
drobison00 Dec 13, 2022
d088828
Update external utils
drobison00 Dec 13, 2022
c419486
Update formatting
drobison00 Dec 13, 2022
e48d745
Merge branch 'devin_issue_496_mrc_refactor' into devin_issue_87_mrc_r…
drobison00 Dec 13, 2022
9293171
Formatting fixes
drobison00 Dec 14, 2022
ecdb6b7
Merge branch 'branch-23.01' into devin_issue_496_mrc_refactor
drobison00 Dec 14, 2022
88df477
Merge branch 'devin_issue_496_mrc_refactor' into devin_issue_87_mrc_r…
drobison00 Dec 14, 2022
a3f25f8
Bump utilities repo, add cmake debugging flags
drobison00 Dec 14, 2022
3dba9fb
Bump utilities version
drobison00 Dec 14, 2022
de12859
Syncing MRC and Morpheus .clang-format
mdemoret-nv Dec 14, 2022
649daec
Update CONTRIBUTING.md
drobison00 Dec 14, 2022
181ce01
Merge latest 23.01
drobison00 Dec 16, 2022
66514f5
Bump utilities repo version
drobison00 Dec 16, 2022
99196c6
Update CMakeLists.txt
drobison00 Dec 17, 2022
77703a0
PR feedback updates, utilitis repo bump
drobison00 Dec 17, 2022
ad4c3f4
Migrate a few more files to utilities, bump utils repo version
drobison00 Dec 19, 2022
e826167
Docs update
drobison00 Dec 19, 2022
690cb0d
Roll back Sphinx deps as default cuda11.5 environment deps
drobison00 Dec 20, 2022
e0760cb
Update morpheus to use new package creation code, bump utilities version
drobison00 Dec 20, 2022
4694c12
Update git modules versions
drobison00 Dec 20, 2022
8665f04
Bump utils repo version
drobison00 Dec 20, 2022
758426e
Merge branch 'branch-23.01' into devin_issue_87_mrc_rebase_branch
drobison00 Jan 6, 2023
6b13a4b
Merge with branch-23.01, bump utilities version
drobison00 Jan 6, 2023
498b85c
Roll to utilities branch 23.01
drobison00 Jan 6, 2023
7ddd8e6
Formatting fixes
drobison00 Jan 6, 2023
2de7319
Merge 23.01
drobison00 Jan 17, 2023
82e71bd
Update build scripts to update and initialize sub-repos
drobison00 Jan 17, 2023
80dbc8e
Update setup.py to ignore flake8 E402 errors
drobison00 Jan 17, 2023
ac0b506
Build fix
drobison00 Jan 17, 2023
31cf9c8
Formatting fix
drobison00 Jan 17, 2023
ba1fe2b
Update CI to default to cuda arch all
drobison00 Jan 17, 2023
396c06e
Add subset of compute caps to build for in CI
drobison00 Jan 17, 2023
9a236e2
CI tweak
drobison00 Jan 17, 2023
b8c0943
Merge branch-23.01
drobison00 Jan 23, 2023
94f0a54
Update utilities for build_wheel fix, and add Morpheus option for CI …
drobison00 Jan 24, 2023
77633ad
more dist updates
drobison00 Jan 24, 2023
dec6d6c
Merge branch 'branch-23.01' into devin_issue_87_mrc_rebase_branch
drobison00 Jan 24, 2023
dfc1933
Tweak CI morpheus wheel installation
drobison00 Jan 24, 2023
27d0bac
Fix python install to include morpheus testing data
drobison00 Jan 24, 2023
a528e02
Update submodule url to be portable across forks
drobison00 Jan 24, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ set(MORPHEUS_RAPIDS_CMAKE_VERSION "22.08" CACHE STRING "Sets default versions fo
set(MORPHEUS_CACHE_DIR "${CMAKE_SOURCE_DIR}/.cache" CACHE PATH "Directory to contain all CPM and CCache data")
mark_as_advanced(MORPHEUS_CACHE_DIR)

set(CMAKE_CXX_STANDARD 17)
set(CMAKE_CXX_STANDARD 20)
set(CMAKE_CXX_STANDARD_REQUIRED ON)
set(CMAKE_CXX_EXTENSIONS ON)
set(CMAKE_EXPORT_COMPILE_COMMANDS ON)
Expand Down
11 changes: 6 additions & 5 deletions docker/conda/environments/cuda11.5_dev.yml
Original file line number Diff line number Diff line change
Expand Up @@ -29,10 +29,11 @@ dependencies:
- ccache>=3.7
- clangdev=14
- click >=8
- cmake=3.22
- cmake=3.23.1
- configargparse=1.5
- cuda-nvml-dev=11.5
- cuda-python<=11.7.0 # Remove when Issue #251 is closed
- nvcc_linux-64=11.5
- cuda-python<=11.5.0 # Remove when Issue #251 is closed
- cudatoolkit=11.5
- cudf 22.08
- cupy=9.5.0
Expand All @@ -43,7 +44,7 @@ dependencies:
- faker=12.3.0
- flake8
- flatbuffers=2.0
- gcc_linux-64=9.4
- gcc_linux-64=11.2
- gflags=2.2
- git>=2.35.3 # Needed for wildcards on safe.directory
- git-lfs=3.2
Expand All @@ -53,7 +54,7 @@ dependencies:
- grpc-cpp>=1.43
- grpcio
- gtest=1.10
- gxx_linux-64=9.4
- gxx_linux-64=11.2
- include-what-you-use=0.18
- isort
- librdkafka=1.7.0
Expand Down Expand Up @@ -81,7 +82,7 @@ dependencies:
- scikit-build=0.13
- sphinx
- sphinx_rtd_theme
- mrc 23.01.*
- mrc=23.01.*
- sysroot_linux-64=2.17
- tqdm=4
- typing_utils=0.1
Expand Down
63 changes: 63 additions & 0 deletions morpheus/_lib/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,69 @@ include(cmake/python_modules/common.cmake)
#----------morpheus._lib.file_types---------
include(cmake/python_modules/file_types.cmake)

function(print_target_properties)
set(prefix F_ARGV)
set(options WRITE_TO_FILE PRINT_NOTFOUND)
set(singleValueArgs WRITE_TO_FILE_NAME)
set(multiValueArgs TARGETS PROPERTIES)

include(CMakeParseArguments)
cmake_parse_arguments(${prefix}
"${options}"
"${singleValueArgs}"
"${multiValueArgs}"
${ARGN})

list(SORT F_ARGV_PROPERTIES)
list(REMOVE_DUPLICATES F_ARGV_PROPERTIES)
foreach(CURRENT_TARGET ${F_ARGV_TARGETS})
set(current_properties_list "")

message(STATUS "[Target Properties '${CURRENT_TARGET}']")
message(STATUS "=====================================================")
foreach(CURRENT_PROPERTY ${F_ARGV_PROPERTIES})
string(STRIP ${CURRENT_PROPERTY} CURRENT_PROPERTY) # Remove trailing/leading whitespace
string(REGEX MATCHALL "^[^#].*$" CURRENT_PROPERTY ${CURRENT_PROPERTY}) # Comments hack

if (NOT CURRENT_PROPERTY)
continue()
endif()

get_target_property(CURRENT_PROPERTY_VALUE ${CURRENT_TARGET} ${CURRENT_PROPERTY})
if((NOT F_ARGV_PRINT_NOTFOUND) AND ("${CURRENT_PROPERTY_VALUE}" STREQUAL "CURRENT_PROPERTY_VALUE-NOTFOUND"))
else()
if ("${CURRENT_PROPERTY_VALUE}" STREQUAL "")
set(CURRENT_PROPERTY_VALUE "<EMPTY_STR>")
else()
set(CURRENT_PROPERTY_VALUE "${CURRENT_PROPERTY_VALUE}")
endif()

string(JOIN ", " CURRENT_PROPERTY_VALUE ${CURRENT_PROPERTY_VALUE})
list(APPEND current_properties_list "${CURRENT_PROPERTY} -> \"${CURRENT_PROPERTY_VALUE}\"")
endif()
endforeach()

foreach(PROPERTY_PAIR ${current_properties_list})
if (F_ARGV_WRITE_TO_FILE)
FILE(APPEND "${CMAKE_BINARY_DIR}/morpheus_utils_debug_target_properties.txt" ${PROPERTY_PAIR})
endif()
message(STATUS "${PROPERTY_PAIR}")
endforeach()
endforeach()
endfunction()

file(STRINGS "${CMAKE_SOURCE_DIR}/test.cmake.props" PROPS_TO_GET)
print_target_properties(
TARGETS
cudf::cudf
matx::matx
mrc::pymrc
cuda_utils_objs
cuda_utils
PROPERTIES
${PROPS_TO_GET}
)

if (MORPHEUS_BUILD_TESTS)
add_subdirectory(tests)
endif()
Expand Down
28 changes: 20 additions & 8 deletions morpheus/_lib/cmake/libraries/cuda_utils.cmake
Original file line number Diff line number Diff line change
Expand Up @@ -18,18 +18,21 @@ find_package(pybind11 REQUIRED)

# Place the two cuda sources in their own target and disable IWYU for that target.
add_library(cuda_utils_objs
OBJECT
${MORPHEUS_LIB_ROOT}/src/utilities/matx_util.cu
${MORPHEUS_LIB_ROOT}/src/utilities/type_util.cu
OBJECT
${MORPHEUS_LIB_ROOT}/src/utilities/matx_util.cu
${MORPHEUS_LIB_ROOT}/src/utilities/type_util.cu
)


set_target_properties(
cuda_utils_objs
cuda_utils_objs
PROPERTIES
C_INCLUDE_WHAT_YOU_USE ""
CXX_INCLUDE_WHAT_YOU_USE ""
EXPORT_COMPILE_COMMANDS OFF
CXX_STANDARD 17
CXX_STANDARD_REQUIRED ON
CUDA_STANDARD 17
CUDA_STANDARD_REQUIRED ON
C_INCLUDE_WHAT_YOU_USE ""
CXX_INCLUDE_WHAT_YOU_USE ""
EXPORT_COMPILE_COMMANDS OFF
)

target_include_directories(cuda_utils_objs
Expand Down Expand Up @@ -71,6 +74,15 @@ target_link_libraries(cuda_utils
pybind11::pybind11
)

set_target_properties(
cuda_utils
PROPERTIES
CXX_STANDARD 17
CXX_STANDARD_REQUIRED ON
CUDA_STANDARD 17
CUDA_STANDARD_REQUIRED ON
)

set_target_properties(cuda_utils
PROPERTIES OUTPUT_NAME ${PROJECT_NAME}_utils
)
Expand Down