Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[GSK-1426] Use WebSocket to replace GRPC for worker-backend communication #1238

Merged
merged 168 commits into from
Aug 17, 2023
Merged

[GSK-1426] Use WebSocket to replace GRPC for worker-backend communication #1238

merged 168 commits into from
Aug 17, 2023

Conversation

Inokinoki
Copy link
Member

@Inokinoki Inokinoki commented Jul 11, 2023 •
edited
Loading

Description

Related Issue

Type of Change

  • 📚 Examples / docs / tutorials / dependencies update
  • 🔧 Bug fix (non-breaking change which fixes an issue)
  • 🥂 Improvement (non-breaking change which improves an existing feature)
  • 🚀 New feature (non-breaking change which adds functionality)
  • 💥 Breaking change (fix or feature that would cause existing functionality to change)
  • 🔐 Security fix

Checklist

  • I've read the CODE_OF_CONDUCT.md document.
  • I've read the CONTRIBUTING.md guide.
  • I've updated the code style using make codestyle.
  • I've written tests for all new methods and classes that I created.
  • I've written the docstring in Google format for all the methods and classes that I used.

Inokinoki added 24 commits July 6, 2023 11:59
@Inokinoki
Open WebSocket connections from workers to server
0b8f56b
@Inokinoki
Add basic WebSocket-based worker-backend comm
15b0793
@Inokinoki
Separate one- and multi-shot messages in WS Worker
9a97e08
@Inokinoki
Allow to observe connectivities of ML Workers
178fe0b
@Inokinoki
Add basic definitions for WebSocket ML Worker DTO
045416f
@Inokinoki
Add WebSocket comm service for ML Workers
6ac6092
@Inokinoki
Migrate ML Worker getInfo method to WebSocket
bd64bfd
@Inokinoki
Separate listener code from MLWorker definitions
881c1bc
@Inokinoki
Add websocket_actor decorator to register handler
fe39dd3
@Inokinoki
Fix getInfo error due to too many arguments
118731b
@Inokinoki
Migrate ML Worker stopWorker method to WebSocket
d28986e
@Inokinoki
Add client to MLWorker to prepare removing tunnel
12a0c76
@Inokinoki
Migrate ML Worker runModel method to WebSocket
33345f9
@Inokinoki
Migrate runModelForDataFrame method to WebSocket
5dc8528 
In ML Worker
@Inokinoki
Migrate ML Worker explain method to WebSocket
0b0c7a8
@Inokinoki
Migrate ML Worker explainText method to WebSocket
f523e47
@Inokinoki
Migrate ML Worker getCatalog method to WebSocket
1e4755e
@Inokinoki
Migrate ML Worker datasetProcesing to WebSocket
598ed02
@Inokinoki
Migrate ML Worker runAdHocTest method to WebSocket
04cace1
@Inokinoki
Migrate ML Worker runTestSuite method to WebSocket
6cc8d71
@Inokinoki
Migrate ML Worker generateTestSuite to WebSocket
038683f
@Inokinoki
Fix runAdHocTest return value
063d355
@Inokinoki
Migrate ML Worker echo method to WebSocket
a5facf1
@Inokinoki
Remove wrong return-value annotations for actors
b0cd8cd
@Inokinoki Inokinoki changed the title Feature/Use WebSocket to replace GRPC for worker-backend communication [Draft] Feature/Use WebSocket to replace GRPC for worker-backend communication Jul 11, 2023
@Inokinoki Inokinoki marked this pull request as draft July 11, 2023 21:07
@Inokinoki Inokinoki changed the title [Draft] Feature/Use WebSocket to replace GRPC for worker-backend communication Feature/Use WebSocket to replace GRPC for worker-backend communication Jul 11, 2023
@github-actions
Copy link

Please add the 'safe for build' label in order to perform the sonar analysis!

@Inokinoki
Fix bug due to False bool function argument
28ef9fb 
Detect not `None` in `parse_function_arguments` to determine the type.
@Inokinoki
Allow project_key to be None in ArtifactRef
b7d5ad5 
Fix validation error when deserializing slicing and transformation funcs
@Inokinoki Inokinoki mentioned this pull request Aug 9, 2023
Closed
11 tasks
@Inokinoki
Rename potentialInternalWorkerId to pending...
6a27b1a 
For better understanding:
1. We validate the token to allow the subscription
2. After a successful subscription, we ensure that a worker is connected
This can avoid sending action command to a pending internal worker.
@Inokinoki
Polish HF access token acquirement
5d2898c 
- Remove base URL rom Hugging Face axios.
- Change popover to a card integrated into TestSuites and
MLWorker instructions.
- Fetch HF Spaces token each time.
- Do not show HF token in a public HF Spaces.
@Inokinoki
Change --token arg to --hf-token in worker cli
0f4e298
@Inokinoki
Fix StartWorkerInstructions state in non HFSpaces
f8fcfb7
@Inokinoki
Extract HF access token acquirement common code
21a4e6a
@Inokinoki
Polish HF access token input by user test feedback
af13782 
Add outline to text field.
Open HF Settings page in a new tab/window.
@Inokinoki
Merge branch 'main' into feature/websocket-replacing-grpc
d5d3eba
@Inokinoki Inokinoki requested a review from andreybavt August 10, 2023 10:11
@andreybavt
Copy link
Contributor

@Inokinoki can you also fix sonar complaints please? Those that are not relevant can be ignored (by adding NOSONAR for example)

Inokinoki and others added 14 commits August 14, 2023 18:34
@Inokinoki
Polish Java Backend based on SonarLint scan
088c7e0 
Refactor `WebSocketSecurityConfig` to reduce the understanding complexity.
Change the names of `MLWorkerAction` enums to match Java naming convention.
Remove unused legacy functions and imports.
Remove duplicated error type in `MLWorkerIllegalReplyException`.
Remove `MLWorkerService`, `GRPCMapper` references in WebSocket.
@Inokinoki
Fix sonar lint in Python
7dbfff5
@Inokinoki
Polish Java code in WebSocket ML Worker
7dbfafc 
Refactor DataProcessing action to reduce complexity.
Add comments to DB transaction annotation.
Remove unused.
@Inokinoki
Remove model parameter in run_other_model
181f323 
Make SonarLint happy
@Inokinoki
Merge branch 'main' of https://github.com/Giskard-AI/giskard into fea…
3e93085 
…ture/websocket-replacing-grpc
@Inokinoki
Remove duplicated code detected by SonarLint
3db3af9 
Make `DatasetProcessFunctionMetaDTO` child of `FunctionMetaDTO` in Java.
@andreybavt
fixed the lockfile
28ba7de
@andreybavt
fix build
15758e2
@Inokinoki
Sync certifi with pdm resolved one in lockfile
74b9dc6
@Inokinoki
Use isinstance instead of type comparison
e3c4b45 
To make recent `pdm lint` and pre-commit happy.
@andreybavt
get rid of grpc and jhipster
bdb08d1
@andreybavt
fix tests
e5f2294
@andreybavt
restored /ml-worker-connect
77061b6
@andreybavt
Sonar
b0b2dc7
@sonarqubecloud
Copy link

Kudos, SonarCloud Quality Gate passed!    Quality Gate passed

Bug A 0 Bugs
Vulnerability A 0 Vulnerabilities
Security Hotspot A 0 Security Hotspots
Code Smell A 2 Code Smells

17.1% 17.1% Coverage
0.7% 0.7% Duplication

@andreybavt andreybavt merged commit b865652 into main Aug 17, 2023
@Hartorn Hartorn deleted the feature/websocket-replacing-grpc branch September 22, 2023 10:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@andreybavt andreybavt Awaiting requested review from andreybavt

Assignees
No one assigned
Labels
Backend Frontend Python Pull requests that update Python code safe for build
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
7 participants
@Inokinoki @andreybavt @rabah-khalek @alexcombessie @kevinmessiaen @AbSsEnT @mathieuroques