Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

add sagemaker predict PoC to eland #2

Merged
merged 30 commits into from
Sep 9, 2022
Merged

add sagemaker predict PoC to eland #2

merged 30 commits into from
Sep 9, 2022

Conversation

LEFTA98
Copy link
Contributor

@LEFTA98 LEFTA98 commented Sep 6, 2022

  • Add integration with sagemaker for doing real-time prediction with opensearch data
  • Add custom index to filter opensearch-py-ml DataFrames by instead of defaulting to _doc as indexing field

Original PR, for those with access: https://github.com/LEFTA98/opensearch-py-ml-thomaym/pull/2

Please note that the CI checks are failing; this is expected due to the renaming of many objects. We will add our own CI checks back in future release.

LEFTA98 and others added 21 commits July 8, 2022 10:44
@LEFTA98
Merge pull request #3 from LEFTA98/data-setup-fix
83a6230 
reverted many erroneous "fixes" to tests
@LEFTA98
added initial connection to predicting with sagemaker
3644f84
@LEFTA98
added sagemaker predict api
fd017db
@LEFTA98
added band-aid to fix iterating over rows
55b2f4f
@LEFTA98
debugging indexing issue
5f2d9aa
@LEFTA98
reverted indexing change for sagemaker predict
5679b13
@LEFTA98
Merge pull request #4 from LEFTA98/sagemaker_integration
727c0db 
Sagemaker integration
@LEFTA98
added deprecation warnings to ml module
35ccb3b
@LEFTA98
refactoring elasticsearch names to opensearch
396d7ec
@LEFTA98
continued renaming opensearch variables
53df988
@LEFTA98
more renaming changes
e4881d8
@LEFTA98
first commit for ml common integration
410ad0e
@LEFTA98
PoC for model upload
7f3d7dd
@LEFTA98
renamed model chunk uploading path
a3bf832
@LEFTA98
added total chunks to model upload
1b79f62
@LEFTA98
fixed docstring typo
18ebcaa
@LEFTA98
added first iteration of custom model load supprot
2d60689
@LEFTA98
PR feedback
ed1f7f9
@LEFTA98
implement PR feedback
564f071
@LEFTA98
removed unsupported features
96024af
@LEFTA98
Merge pull request #5 from opensearch-project/custom_model_fixes
06138ac 
Custom model fixes
ylwu-amzn
ylwu-amzn reviewed Sep 8, 2022
View reviewed changes
eland/operations.py
):
df = query_compiler._es_results_to_pandas(hits)
df = self._apply_df_post_processing(df, post_processing)
# i += 1
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remove debug code?

eland/sagemaker_tools.py Outdated

from sagemaker import RealTimePredictor, Session

DEFAULT_UPLOAD_CHUNK_SIZE = 1000
Copy link
Collaborator

@ylwu-amzn ylwu-amzn Sep 8, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is for uploading data to sagemaker only? If yes, how about change this constant name to "DEFAULT_SAGEMAKER_UPLOAD_CHUNK_SIZE"

Update: ignore this comment, just saw you have changed in another PR #4

@LEFTA98 LEFTA98 merged commit a813c64 into eland_to_opensearch Sep 9, 2022
LEFTA98 added a commit that referenced this pull request Sep 9, 2022
@LEFTA98
Merge pull request #2 from LEFTA98/dimension-fixes
0b89f1f 
fixed aggregatable field name tests
@dhrubo-os dhrubo-os deleted the sagemaker_predict branch February 8, 2023 18:00
@Assylnur Assylnur mentioned this pull request Mar 30, 2023
Closed
dhrubo-os pushed a commit that referenced this pull request Aug 22, 2023
@thanawan-atc
[Automating Model Tracing and Uploading] PR2 Model Listing & Uploading (
205a0fd 
#210)

* Initiate PR #2 Model Listing & Uploading

Signed-off-by: Thanawan Atchariyachanvanit <latchari@amazon.com>

* Add comment

Signed-off-by: Thanawan Atchariyachanvanit <latchari@amazon.com>

* Correct linting

Signed-off-by: Thanawan Atchariyachanvanit <latchari@amazon.com>

* Update CHANGELOG.md

Signed-off-by: Thanawan Atchariyachanvanit <latchari@amazon.com>

* Update CHANGELOG.md

Signed-off-by: Thanawan Atchariyachanvanit <latchari@amazon.com>

* Revert "Correct demo_ml_commons_integration.ipynb (#208)"

This reverts commit c67f969.

Signed-off-by: Thanawan Atchariyachanvanit <latchari@amazon.com>

* Removed a modified file from pull request

Signed-off-by: Thanawan Atchariyachanvanit <latchari@amazon.com>

* Removed a modified file from pull request

Signed-off-by: Thanawan Atchariyachanvanit <latchari@amazon.com>

* Add steps.checkout_pr_branch.outcome

Signed-off-by: Thanawan Atchariyachanvanit <latchari@amazon.com>

* Remove old config json

Signed-off-by: Thanawan Atchariyachanvanit <latchari@amazon.com>

* Minor change

Signed-off-by: Thanawan Atchariyachanvanit <latchari@amazon.com>

* Allow non-st

Signed-off-by: Thanawan Atchariyachanvanit <latchari@amazon.com>

* Add tests for PR2

Signed-off-by: Thanawan Atchariyachanvanit <latchari@amazon.com>

* Create __init__.py

Signed-off-by: Thanawan Atchariyachanvanit <latchari@amazon.com>

* Create __init__.py

Signed-off-by: Thanawan Atchariyachanvanit <latchari@amazon.com>

* Update noxfile.py

Signed-off-by: Thanawan Atchariyachanvanit <latchari@amazon.com>

* Update noxfile.py

Signed-off-by: Thanawan Atchariyachanvanit <latchari@amazon.com>

* Update setup.cfg

Signed-off-by: Thanawan Atchariyachanvanit <latchari@amazon.com>

* Update noxfile.py

Signed-off-by: Thanawan Atchariyachanvanit <latchari@amazon.com>

* Add more test to improve coverage

Signed-off-by: Thanawan Atchariyachanvanit <latchari@amazon.com>

* Improve test cov

Signed-off-by: Thanawan Atchariyachanvanit <latchari@amazon.com>

* Make it extensible

Signed-off-by: Thanawan Atchariyachanvanit <latchari@amazon.com>

* Update update_model_listing.yml

Signed-off-by: Thanawan Atchariyachanvanit <latchari@amazon.com>

* Avoid crashing if folder does not exists

Signed-off-by: Thanawan Atchariyachanvanit <latchari@amazon.com>

* Debug update_model_listing.yml

Signed-off-by: Thanawan Atchariyachanvanit <latchari@amazon.com>

* Update update_model_listing.yml

Signed-off-by: Thanawan Atchariyachanvanit <latchari@amazon.com>

* Wrap with update_pretrained_model_listing_main

Signed-off-by: Thanawan Atchariyachanvanit <latchari@amazon.com>

* Update update_model_listing.yml

Signed-off-by: Thanawan Atchariyachanvanit <latchari@amazon.com>

* Remove unused variables

Signed-off-by: Thanawan Atchariyachanvanit <latchari@amazon.com>

* Remove unnecessary variable

Signed-off-by: Thanawan Atchariyachanvanit <latchari@amazon.com>

---------

Signed-off-by: Thanawan Atchariyachanvanit <latchari@amazon.com>
opensearch-trigger-bot bot pushed a commit that referenced this pull request Aug 22, 2023
@thanawan-atc @github-actions
[Automating Model Tracing and Uploading] PR2 Model Listing & Uploading (
3b330d7 
#210)

* Initiate PR #2 Model Listing & Uploading

Signed-off-by: Thanawan Atchariyachanvanit <latchari@amazon.com>

* Add comment

Signed-off-by: Thanawan Atchariyachanvanit <latchari@amazon.com>

* Correct linting

Signed-off-by: Thanawan Atchariyachanvanit <latchari@amazon.com>

* Update CHANGELOG.md

Signed-off-by: Thanawan Atchariyachanvanit <latchari@amazon.com>

* Update CHANGELOG.md

Signed-off-by: Thanawan Atchariyachanvanit <latchari@amazon.com>

* Revert "Correct demo_ml_commons_integration.ipynb (#208)"

This reverts commit c67f969.

Signed-off-by: Thanawan Atchariyachanvanit <latchari@amazon.com>

* Removed a modified file from pull request

Signed-off-by: Thanawan Atchariyachanvanit <latchari@amazon.com>

* Removed a modified file from pull request

Signed-off-by: Thanawan Atchariyachanvanit <latchari@amazon.com>

* Add steps.checkout_pr_branch.outcome

Signed-off-by: Thanawan Atchariyachanvanit <latchari@amazon.com>

* Remove old config json

Signed-off-by: Thanawan Atchariyachanvanit <latchari@amazon.com>

* Minor change

Signed-off-by: Thanawan Atchariyachanvanit <latchari@amazon.com>

* Allow non-st

Signed-off-by: Thanawan Atchariyachanvanit <latchari@amazon.com>

* Add tests for PR2

Signed-off-by: Thanawan Atchariyachanvanit <latchari@amazon.com>

* Create __init__.py

Signed-off-by: Thanawan Atchariyachanvanit <latchari@amazon.com>

* Create __init__.py

Signed-off-by: Thanawan Atchariyachanvanit <latchari@amazon.com>

* Update noxfile.py

Signed-off-by: Thanawan Atchariyachanvanit <latchari@amazon.com>

* Update noxfile.py

Signed-off-by: Thanawan Atchariyachanvanit <latchari@amazon.com>

* Update setup.cfg

Signed-off-by: Thanawan Atchariyachanvanit <latchari@amazon.com>

* Update noxfile.py

Signed-off-by: Thanawan Atchariyachanvanit <latchari@amazon.com>

* Add more test to improve coverage

Signed-off-by: Thanawan Atchariyachanvanit <latchari@amazon.com>

* Improve test cov

Signed-off-by: Thanawan Atchariyachanvanit <latchari@amazon.com>

* Make it extensible

Signed-off-by: Thanawan Atchariyachanvanit <latchari@amazon.com>

* Update update_model_listing.yml

Signed-off-by: Thanawan Atchariyachanvanit <latchari@amazon.com>

* Avoid crashing if folder does not exists

Signed-off-by: Thanawan Atchariyachanvanit <latchari@amazon.com>

* Debug update_model_listing.yml

Signed-off-by: Thanawan Atchariyachanvanit <latchari@amazon.com>

* Update update_model_listing.yml

Signed-off-by: Thanawan Atchariyachanvanit <latchari@amazon.com>

* Wrap with update_pretrained_model_listing_main

Signed-off-by: Thanawan Atchariyachanvanit <latchari@amazon.com>

* Update update_model_listing.yml

Signed-off-by: Thanawan Atchariyachanvanit <latchari@amazon.com>

* Remove unused variables

Signed-off-by: Thanawan Atchariyachanvanit <latchari@amazon.com>

* Remove unnecessary variable

Signed-off-by: Thanawan Atchariyachanvanit <latchari@amazon.com>

---------

Signed-off-by: Thanawan Atchariyachanvanit <latchari@amazon.com>
(cherry picked from commit 205a0fd)
dhrubo-os pushed a commit that referenced this pull request Aug 22, 2023
@opensearch-trigger-bot @thanawan-atc
[Automating Model Tracing and Uploading] PR2 Model Listing & Uploading (
5071bfb 
#210) (#238)

* Initiate PR #2 Model Listing & Uploading

Signed-off-by: Thanawan Atchariyachanvanit <latchari@amazon.com>

* Add comment

Signed-off-by: Thanawan Atchariyachanvanit <latchari@amazon.com>

* Correct linting

Signed-off-by: Thanawan Atchariyachanvanit <latchari@amazon.com>

* Update CHANGELOG.md

Signed-off-by: Thanawan Atchariyachanvanit <latchari@amazon.com>

* Update CHANGELOG.md

Signed-off-by: Thanawan Atchariyachanvanit <latchari@amazon.com>

* Revert "Correct demo_ml_commons_integration.ipynb (#208)"

This reverts commit c67f969.

Signed-off-by: Thanawan Atchariyachanvanit <latchari@amazon.com>

* Removed a modified file from pull request

Signed-off-by: Thanawan Atchariyachanvanit <latchari@amazon.com>

* Removed a modified file from pull request

Signed-off-by: Thanawan Atchariyachanvanit <latchari@amazon.com>

* Add steps.checkout_pr_branch.outcome

Signed-off-by: Thanawan Atchariyachanvanit <latchari@amazon.com>

* Remove old config json

Signed-off-by: Thanawan Atchariyachanvanit <latchari@amazon.com>

* Minor change

Signed-off-by: Thanawan Atchariyachanvanit <latchari@amazon.com>

* Allow non-st

Signed-off-by: Thanawan Atchariyachanvanit <latchari@amazon.com>

* Add tests for PR2

Signed-off-by: Thanawan Atchariyachanvanit <latchari@amazon.com>

* Create __init__.py

Signed-off-by: Thanawan Atchariyachanvanit <latchari@amazon.com>

* Create __init__.py

Signed-off-by: Thanawan Atchariyachanvanit <latchari@amazon.com>

* Update noxfile.py

Signed-off-by: Thanawan Atchariyachanvanit <latchari@amazon.com>

* Update noxfile.py

Signed-off-by: Thanawan Atchariyachanvanit <latchari@amazon.com>

* Update setup.cfg

Signed-off-by: Thanawan Atchariyachanvanit <latchari@amazon.com>

* Update noxfile.py

Signed-off-by: Thanawan Atchariyachanvanit <latchari@amazon.com>

* Add more test to improve coverage

Signed-off-by: Thanawan Atchariyachanvanit <latchari@amazon.com>

* Improve test cov

Signed-off-by: Thanawan Atchariyachanvanit <latchari@amazon.com>

* Make it extensible

Signed-off-by: Thanawan Atchariyachanvanit <latchari@amazon.com>

* Update update_model_listing.yml

Signed-off-by: Thanawan Atchariyachanvanit <latchari@amazon.com>

* Avoid crashing if folder does not exists

Signed-off-by: Thanawan Atchariyachanvanit <latchari@amazon.com>

* Debug update_model_listing.yml

Signed-off-by: Thanawan Atchariyachanvanit <latchari@amazon.com>

* Update update_model_listing.yml

Signed-off-by: Thanawan Atchariyachanvanit <latchari@amazon.com>

* Wrap with update_pretrained_model_listing_main

Signed-off-by: Thanawan Atchariyachanvanit <latchari@amazon.com>

* Update update_model_listing.yml

Signed-off-by: Thanawan Atchariyachanvanit <latchari@amazon.com>

* Remove unused variables

Signed-off-by: Thanawan Atchariyachanvanit <latchari@amazon.com>

* Remove unnecessary variable

Signed-off-by: Thanawan Atchariyachanvanit <latchari@amazon.com>

---------

Signed-off-by: Thanawan Atchariyachanvanit <latchari@amazon.com>
(cherry picked from commit 205a0fd)

Co-authored-by: Thanawan Atchariyachanvanit <latchari@amazon.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ylwu-amzn ylwu-amzn ylwu-amzn left review comments

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@LEFTA98 @ylwu-amzn