Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Integrated training: Framework updates and Modular DFP pipeline implementation #760

Merged
merged 198 commits into from
Mar 30, 2023
Merged

Integrated training: Framework updates and Modular DFP pipeline implementation #760

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
198 commits
Select commit Hold shift + click to select a range
d5b16cc
Start sketching out control messages, add temporary fix for Prealloca…
drobison00 Feb 7, 2023
4c630a6
Bug fixes, and control message PoC impl + cpp/py unit tests
drobison00 Feb 8, 2023
7eb09a5
Bring naming changes in line with other messages, add more python uni…
drobison00 Feb 8, 2023
5db3a4f
Checkpoint
drobison00 Feb 9, 2023
02a38c3
DataLoader is working in basic principal, need to move edge converter…
drobison00 Feb 10, 2023
1b7053f
Add DataLoaderModule and morpheus module integration, fixed all modul…
drobison00 Feb 10, 2023
9f8889b
Something is broken with the data loader test -- why is there a segfa…
drobison00 Feb 10, 2023
026d263
Add c++ side tests for data loader module, fix some bugs related to u…
drobison00 Feb 10, 2023
a982c99
Add loader sketches, more unit tests
drobison00 Feb 10, 2023
5e12d99
Add unit tests, work around interpreter conflicts with cudf
drobison00 Feb 14, 2023
1b63cfa
Add file loader paths for supported files, fix tests, add python tests
drobison00 Feb 15, 2023
5aab213
broadcast stage for dfp pipeline
bsuryadevara Feb 15, 2023
8ff9ca7
broadcast stage for dfp pipeline
bsuryadevara Feb 15, 2023
9119102
More unit tests, add loader registry, various other improvements
drobison00 Feb 16, 2023
17aa7b3
Test fixes
drobison00 Feb 16, 2023
c1f4286
Added non linear modules stage for dfp pipeline
bsuryadevara Feb 16, 2023
2003b7e
Added non linear modules stage for dfp pipeline
bsuryadevara Feb 16, 2023
9babdf4
Merge remote-tracking branch 'upstream/branch-23.03' into 698-fea-cre…
bsuryadevara Feb 16, 2023
71adcfa
Merge remote-tracking branch 'upstream/branch-23.03' into 698-fea-cre…
bsuryadevara Feb 16, 2023
769c85d
style check
bsuryadevara Feb 16, 2023
b53b4df
modified variable naming conventions
bsuryadevara Feb 16, 2023
b8026c0
Checkpoint loader registry is working, all pieces in place for initia…
drobison00 Feb 16, 2023
c14406e
Add c++ side LoaderRegistry unittests
drobison00 Feb 16, 2023
da15331
Add more unittests
drobison00 Feb 16, 2023
93069a4
Merge branch 'branch-23.03' into devin_integrated_training_595
drobison00 Feb 16, 2023
56153f2
Merge remote-tracking branch 'devin/devin_integrated_training_595' in…
bsuryadevara Feb 17, 2023
78d67e8
Switch data loaders to consume and produce control messages. Anything…
drobison00 Feb 17, 2023
db2111c
Naming fixes, add the ability to list factory objects in the loader r…
drobison00 Feb 17, 2023
18ca626
Merge remote-tracking branch 'devin/devin_integrated_training_595' in…
bsuryadevara Feb 17, 2023
fabcaa8
Fix template 'definition not available' warning. Add file list loader
drobison00 Feb 17, 2023
a4e6be9
Add file list loader
drobison00 Feb 17, 2023
4022a19
Merge remote-tracking branch 'devin/devin_integrated_training_595' in…
bsuryadevara Feb 21, 2023
856b813
Factory updates
drobison00 Feb 21, 2023
5994b7e
Update to allow config passing to loaders
drobison00 Feb 21, 2023
b684568
data loader integration to dfp pipeline
bsuryadevara Feb 21, 2023
53b034e
Merge remote-tracking branch 'devin/devin_integrated_training_595' in…
bsuryadevara Feb 21, 2023
41e17ee
data loader integration to dfp pipeline
bsuryadevara Feb 21, 2023
6ce3981
Update loaders to take configurations
drobison00 Feb 21, 2023
fdda9f6
Merge remote-tracking branch 'bhargav_morpheus/bhargav_integrated_tra…
drobison00 Feb 21, 2023
8c6d5d4
Shift to new implicit control message schema
drobison00 Feb 21, 2023
5b577ba
Start adding pieces for schema validation
drobison00 Feb 22, 2023
01da852
Checkpoint, maybe inference wasn't so bad!
drobison00 Feb 22, 2023
8cf6823
data loader integration to dfp pipeline
bsuryadevara Feb 23, 2023
a8401ff
data loader integration to dfp pipeline
bsuryadevara Feb 23, 2023
f574e07
updated message meta
bsuryadevara Feb 23, 2023
ddf5e71
Initial conversion of whole pipeline to use control messages
drobison00 Feb 23, 2023
d722e66
Updates to fix missing caches
drobison00 Feb 24, 2023
0314abf
Merge remote-tracking branch 'devin/devin_integrated_training_595' in…
bsuryadevara Feb 24, 2023
d5f3451
dfp inference pipeline with control messages
bsuryadevara Feb 24, 2023
e3917de
added logging support for the modules
bsuryadevara Feb 24, 2023
537b69e
control message source implementation
bsuryadevara Feb 25, 2023
0bc3e38
checkpoint
drobison00 Feb 25, 2023
9889e91
Checkpoint -- fixed multi-load per CM
drobison00 Feb 25, 2023
fc3aa37
Pipeline updates to handle different training categories -- often nee…
drobison00 Feb 26, 2023
f7f06cd
Merge Bhargav's test branch, updates, and quick input test sample
drobison00 Feb 26, 2023
bba2fb5
Fix inference messages
drobison00 Feb 26, 2023
b89334c
added fsspec dataloader
bsuryadevara Feb 27, 2023
1b43dfc
forked pipeline to run training and inference in parallel
bsuryadevara Feb 27, 2023
80f093f
Merge pull request #2 from bsuryadevara/bhargav_integrated_training_716
drobison00 Feb 28, 2023
53c80d8
removed print statement
bsuryadevara Feb 28, 2023
304b772
Remove spurrious print, tweak some things for testing
drobison00 Feb 28, 2023
969975e
Merge remote-tracking branch 'devin/devin_integrated_training_595' in…
bsuryadevara Feb 28, 2023
9db7d7e
Update config_generator workflow to reduce redundant parameters
drobison00 Feb 28, 2023
3181adb
More config parameter pruning
drobison00 Feb 28, 2023
3f1535e
Remove debug code
drobison00 Feb 28, 2023
6246fda
Cleanup file_batcher module
drobison00 Feb 28, 2023
a39de93
Cleanup file_batcher module
drobison00 Feb 28, 2023
29eecf3
fix to dfp benchmarks
bsuryadevara Feb 28, 2023
ee25ef3
Merge remote-tracking branch 'devin/devin_integrated_training_595' in…
bsuryadevara Feb 28, 2023
897d361
fix to dfp benchmarks
bsuryadevara Feb 28, 2023
ba8734d
Update conftest.py
bsuryadevara Mar 1, 2023
56477f4
Merge pull request #3 from bsuryadevara/fix-dfp-benchmarks
drobison00 Mar 1, 2023
af4ba04
Tweak to make benchmarks run
drobison00 Mar 1, 2023
01a6ecb
Performance improvements -- reducing data copies
drobison00 Mar 1, 2023
3398a20
Additional performance optimizations, start removing pandas wherever …
drobison00 Mar 2, 2023
ef271b3
dfp deployment module splitting
bsuryadevara Mar 2, 2023
ca32d7c
updates to dfp deployment module
bsuryadevara Mar 2, 2023
7ee3a9b
Merge remote-tracking branch 'devin/devin_integrated_training_595' in…
bsuryadevara Mar 2, 2023
8f665dd
restructured benchmark tests
bsuryadevara Mar 3, 2023
8c39fe5
restructured benchmark tests
bsuryadevara Mar 3, 2023
dcd272f
updates to dfp benchmarks
bsuryadevara Mar 3, 2023
76fa276
Update file_batcher.py
drobison00 Mar 3, 2023
2f25576
Merge 23.03
drobison00 Mar 4, 2023
f9e3563
added kafka source stage for control messages
bsuryadevara Mar 5, 2023
c61b4e7
added kafka source stage for control messages
bsuryadevara Mar 5, 2023
98d0869
Merge remote-tracking branch 'devin/devin_integrated_training_595' in…
bsuryadevara Mar 5, 2023
de4edea
added kafka source stage for control messages
bsuryadevara Mar 5, 2023
e90c736
added control message filter module
bsuryadevara Mar 6, 2023
1bf2047
changes to control message kafka source
bsuryadevara Mar 6, 2023
a9ad0d6
Merge pull request #4 from bsuryadevara/bhargav_integrated_training_716
drobison00 Mar 6, 2023
b04b581
Merge latest from Bhargav 716, locking down control messages
drobison00 Mar 7, 2023
2f219c0
Fix control message typing issue
drobison00 Mar 7, 2023
3f3eb65
Add task_type python bindings for control messages
drobison00 Mar 7, 2023
3ebc9e7
removed deprecated statements
bsuryadevara Mar 7, 2023
44ac55a
dfp demo gui
bsuryadevara Mar 9, 2023
c0377f8
Checkpoint. Mostly through config rework
drobison00 Mar 9, 2023
c0f793a
Configuration updates finished -- still need more cleanup
drobison00 Mar 9, 2023
f22f1de
More cleanup, config fixes
drobison00 Mar 10, 2023
1badf56
Fix some incorrect pipeline default values
drobison00 Mar 10, 2023
7696fa8
added UI
bsuryadevara Mar 12, 2023
6890400
trivial changes to demo gui
bsuryadevara Mar 12, 2023
cf703fb
static ui for training messages
bsuryadevara Mar 13, 2023
4287f47
Update README.md
bsuryadevara Mar 13, 2023
48502ce
Update README.md
bsuryadevara Mar 13, 2023
43fd691
updated file names
bsuryadevara Mar 13, 2023
821874e
updated readme.md
bsuryadevara Mar 13, 2023
5eef02d
Merge with Bhargav's development branch
drobison00 Mar 14, 2023
325bc31
Naming tweaks
drobison00 Mar 14, 2023
308a253
file batcher bug fix
drobison00 Mar 14, 2023
d85f245
Merge branch-23.03
drobison00 Mar 14, 2023
f1ff7cb
Docs update
drobison00 Mar 14, 2023
9b74754
Comments updates
drobison00 Mar 14, 2023
1e0df31
Config generator fix
drobison00 Mar 14, 2023
5b70ded
Comment cleanup, improved error handling
drobison00 Mar 14, 2023
37510db
First pass markdown files for new modules
drobison00 Mar 15, 2023
429041c
Docs updates
drobison00 Mar 15, 2023
27c7fe1
Add top level module documentation file
drobison00 Mar 15, 2023
c617371
trivial changes to dfp demo files
bsuryadevara Mar 16, 2023
e893322
renamed multiport file name
bsuryadevara Mar 16, 2023
4e3ff46
Add simple module user guide
drobison00 Mar 16, 2023
b100160
Add simple c++ module example
drobison00 Mar 16, 2023
d7bddd7
Morpheus modules python updates
drobison00 Mar 16, 2023
a94671b
Docs naming tweak
drobison00 Mar 16, 2023
5701239
Remove dfp benchmarking results -- they're not comparing the right th…
drobison00 Mar 16, 2023
ce38b6f
Benchmark fixes
drobison00 Mar 16, 2023
19d88ee
trivial changes
bsuryadevara Mar 17, 2023
789759e
added human in loop dfp demo gui
bsuryadevara Mar 17, 2023
c6691b6
Merge remote-tracking branch 'devin/devin_integrated_training_595' in…
bsuryadevara Mar 17, 2023
b58cb0b
removed file_batcher test
bsuryadevara Mar 17, 2023
06bac3c
Merge pull request #5 from bsuryadevara/bhargav_integrated_training_716
drobison00 Mar 17, 2023
c02d5cd
Merge Bhargav's PR, add start of control messages docs
drobison00 Mar 17, 2023
2429689
Update copyright headers
drobison00 Mar 17, 2023
831bf36
Sphinx updates, formatting fixes, docs updates, etc...
drobison00 Mar 17, 2023
50cccd9
Docs update
drobison00 Mar 17, 2023
264b5d3
Merge branch-23.03, fix unit tests
drobison00 Mar 21, 2023
70d3a0d
added control message documentation
bsuryadevara Mar 21, 2023
8f77b67
added control message documentation
bsuryadevara Mar 21, 2023
e9f43d4
added control message documentation
bsuryadevara Mar 21, 2023
2825930
added control message documentation
bsuryadevara Mar 21, 2023
95ccc20
added control message documentation
bsuryadevara Mar 21, 2023
b29cdcd
trivial changes to loader docs
bsuryadevara Mar 21, 2023
4ec9879
trivial changes to loader docs
bsuryadevara Mar 21, 2023
f390d76
Merge pull request #6 from bsuryadevara/bhargav_integrated_training_716
drobison00 Mar 21, 2023
8d95b4d
updated demo GUI and added docs
Mar 23, 2023
46a5930
Checkpoint -- Modular DFP pipeline walkthrough
drobison00 Mar 23, 2023
332ef0f
Checkpoint Modular Pipeline examples updates
drobison00 Mar 23, 2023
4ed5870
Fix table of contents
drobison00 Mar 23, 2023
dad6e62
added demo GUI docs
Mar 23, 2023
b4531fa
Update submit_messages.md
bsuryadevara Mar 23, 2023
595b46b
Update README.md
bsuryadevara Mar 23, 2023
4ec2241
Merge remote-tracking branch 'devin/devin_integrated_training_595' in…
Mar 24, 2023
f7b76ee
Merge branch 'bhargav_integrated_training_716' of https://github.com/…
Mar 24, 2023
0767968
Update README.md
bsuryadevara Mar 24, 2023
0c4a2f2
Update submit_messages.md
bsuryadevara Mar 24, 2023
b228419
Merge pull request #7 from bsuryadevara/bhargav_integrated_training_716
drobison00 Mar 24, 2023
eb2742b
added deafult config parameter values for modules
Mar 24, 2023
ba445fb
changes to module docs
Mar 24, 2023
b7deaaf
Merge remote-tracking branch 'devin/devin_integrated_training_595' in…
Mar 24, 2023
0c0591f
Add data loader module docs
drobison00 Mar 24, 2023
49a4980
Merge remote-tracking branch 'devin/devin_integrated_training_595' in…
Mar 24, 2023
24070d6
added header to dfp files
Mar 24, 2023
efd20bd
Update module docs to support tabular formatting
drobison00 Mar 24, 2023
d82b6d7
added header to dfp files
Mar 24, 2023
3abf196
Merge remote-tracking branch 'devin/devin_integrated_training_595' in…
Mar 27, 2023
82552a4
remove broadcast stage implementation
Mar 27, 2023
0d5de30
PR feedback updates
drobison00 Mar 27, 2023
cc3c9c0
More PR feedback updates
drobison00 Mar 27, 2023
e75196c
Checkpoint before merge
drobison00 Mar 28, 2023
fbf8efe
Merge remote-tracking branch 'devin/devin_integrated_training_595' in…
Mar 28, 2023
c111f6b
Merge latest 23.03 and resolve pipeline breakages.
drobison00 Mar 28, 2023
a875db8
Formatting and IWYU fixes
drobison00 Mar 28, 2023
88fe3aa
Merge remote-tracking branch 'devin/devin_integrated_training_595' in…
Mar 28, 2023
851a95d
Flake8 updates, docs cleanup
drobison00 Mar 28, 2023
f76fdd3
Merge branch 23.03
drobison00 Mar 28, 2023
d78278a
updated module schema docs
Mar 28, 2023
52705f4
Merge remote-tracking branch 'devin/devin_integrated_training_595' in…
Mar 28, 2023
f947fb1
updated module schema docs
Mar 29, 2023
1b19c8b
module doc updates
Mar 29, 2023
64635e8
module doc updates
Mar 29, 2023
3d041f9
Merge pull request #8 from bsuryadevara/bhargav_integrated_training_595
drobison00 Mar 29, 2023
fa124c9
Merge branch 'branch-23.03' into devin_integrated_training_595
drobison00 Mar 29, 2023
dabccbb
module doc updates
Mar 29, 2023
b76c5ad
Merge remote-tracking branch 'devin/devin_integrated_training_595' in…
Mar 29, 2023
0e376e8
Merge pull request #9 from bsuryadevara/bhargav_integrated_training_595
drobison00 Mar 29, 2023
3b0191c
Merge branch-23.03
drobison00 Mar 29, 2023
c2713f4
IWYU fixes
drobison00 Mar 29, 2023
a26b608
Formatting tweaks -- local iwyu is a bit different from CI. Fix pytho…
drobison00 Mar 30, 2023
15efb8b
Switch modules version to use morphues version.hpp
drobison00 Mar 30, 2023
efeaeeb
Formatting fix
drobison00 Mar 30, 2023
49a2f62
Be a bit more specific about importing morpheus.common in loader test…
drobison00 Mar 30, 2023
1396c72
Add xfails to test_loader_registry tests to see if we get ci passing
drobison00 Mar 30, 2023
f0b0dbe
Fix in-place python builds
dagardner-nv Mar 30, 2023
2f2c8f3
Remove xfails
dagardner-nv Mar 30, 2023
bca5a95
Remove unused import
dagardner-nv Mar 30, 2023
d165718
Skip known failing tests
dagardner-nv Mar 30, 2023
455286c
Xfail fix
drobison00 Mar 30, 2023
47e4028
isort fix
drobison00 Mar 30, 2023
613530b
Fix missing symbols in release build
drobison00 Mar 30, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
remove broadcast stage implementation
  • Loading branch information
Bhargav Suryadevara authored and Bhargav Suryadevara committed Mar 27, 2023
commit 82552a43f6d98af0013d447ff28b752f715a55d6
97 changes: 0 additions & 97 deletions morpheus/stages/general/broadcast_stage.py
View file
Open in desktop

This file was deleted.

52 changes: 0 additions & 52 deletions tests/test_broadcast_stage.py
View file
Open in desktop

This file was deleted.