Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adds support to read and write to Databricks delta tables #630

Merged
merged 35 commits into from
Oct 5, 2023
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
35 commits
Select commit Hold shift + click to select a range
56108cc
Added databricks dependency
pthalasta Jan 17, 2023
c3598e7
Added jdk dependency installation
pthalasta Jan 17, 2023
9f989d7
Added delta lake source stage
pthalasta Jan 24, 2023
dbafd69
Added output stage to write to delta lake table
pthalasta Jan 24, 2023
323a077
Merge branch 'nv-morpheus:branch-23.01' into branch-23.01
pthalasta Jan 24, 2023
04d75d9
Merge branch 'branch-23.01' into branch-23.01
pthalasta Jan 25, 2023
e5b7930
Updated source reader with formatting changes
pthalasta Jan 26, 2023
54930f2
Updated sink writer with formatting changes
pthalasta Jan 26, 2023
ac47465
Merge branch 'branch-23.01' into branch-23.01
pthalasta Jan 26, 2023
0fad59f
updated source stage with pagination and moved config creation to uti…
pthalasta Sep 1, 2023
3aebcbb
Renamed classes of source and sink to match databrick-deltalake naming
pthalasta Sep 1, 2023
c0a2670
Changed Dockerfile and requirements as per latest changes in 23.11 br…
pthalasta Sep 1, 2023
03c9758
Updated branch changes to reflect 23.11
pthalasta Sep 1, 2023
cdc5403
Removed databricks-connect from runtime and example yaml requirements…
pthalasta Sep 5, 2023
cc3bdc7
Removed databricks-connect from runtime and example yaml requirements…
pthalasta Sep 5, 2023
7833498
Removed databricks-connect from runtime and example yaml requirements…
pthalasta Sep 5, 2023
82677cf
Upgraded databricks connect to 13.3 and modified code as per new version
pthalasta Sep 27, 2023
bee1a55
Upgraded databricks connect to 13.3 and modified code as per new version
pthalasta Sep 27, 2023
88f958a
Upgraded databricks connect to 13.3 and modified code as per new version
pthalasta Sep 28, 2023
86183e9
Added test case for databricks delta lake source stage
pthalasta Oct 2, 2023
39d7f60
Added test case for databricks delta lake source stage
pthalasta Oct 2, 2023
f6018bf
Added test case for databricks delta lake source stage
pthalasta Oct 2, 2023
25aec2f
Merge branch 'branch-23.11' of https://github.com/nv-morpheus/Morpheu…
pthalasta Oct 2, 2023
d30e924
Merge branch 'branch-23.11' into branch-23.01
pthalasta Oct 2, 2023
377456b
Added copyright info for test file
pthalasta Oct 2, 2023
cdb75c6
Merge branch 'branch-23.01' of https://github.com/pthalasta/Morpheus …
pthalasta Oct 2, 2023
ad6c6f6
Bug fixes in test stage
pthalasta Oct 2, 2023
58889ec
Bug fixes in test stage
pthalasta Oct 2, 2023
9e70e79
Added sink test case and additional cleanup of code
pthalasta Oct 3, 2023
cf7029e
Bug fixes in sink test code
pthalasta Oct 3, 2023
279c695
formatting changes
pthalasta Oct 3, 2023
40a2ca5
ci suggested fixes to test/code
Oct 3, 2023
4aa07b3
ci suggested fixes to test/code
Oct 4, 2023
e738677
doc generation fixes
Oct 4, 2023
acb796c
added PreallocatorMixin imports in source
Oct 4, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
doc generation fixes
  • Loading branch information
Pradeep Thalasta committed Oct 4, 2023
commit e738677f415ae14752797322146c9175e5c7cc54
1 change: 1 addition & 0 deletions docs/source/conf.py
Original file line number Diff line number Diff line change
Expand Up @@ -164,6 +164,7 @@
autodoc_mock_imports = [
"cudf", # Avoid loading GPU libraries during the documentation build
"cupy", # Avoid loading GPU libraries during the documentation build
"databricks.connect",
"merlin",
"morpheus.cli.commands", # Dont document the CLI in Sphinx
"nvtabular",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -83,10 +83,6 @@ def name(self) -> str:
def accepted_types(self) -> typing.Tuple:
"""
Returns accepted input types for this stage.
Returns
-------
typing.Tuple(`morpheus.pipeline.messages.MessageMeta`, )
Accepted input types.
"""
return (MessageMeta, )

Expand Down
Loading