Cluster mode #8
Open
Cluster mode #8
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…the structure from the CountTable
…rk Workers to communicate with S3
…orExploration to work with Dataframes. Up until now, metadata and stats query works correctly in cluster
…to see how the filters are behaving.
…tor. The idea is that Connectors that work with spark will provide a connection to the Dataset for each Table. Then SparkDatabaseRepository will be responsible to utilize them in order to implement the various methods required by the DatabaseRepository. In this case we put trust in spark that it will run the query in an optimized manner. Also, this makes extremely easier to add new database as it wil only require 4-5 methods.
…istence and unordered.
…istence and unordered.
…st that change the get function
…at it can work on a cluster with both modes
…issue was that the events were read from SingleTable and handled as EventBoth. Since EventBoth extends EventTs it was handled as a timestamp event and the position information was missed.
Fixed issue with Kleene* when working in PatternDetectionSingle.
…vent type. The issue was that since there was no et-pairs, causing issue with the detection queries.
|
Fixed #6 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Query now works on a spark cluster. The RDD framework has been swapped to Datasets. Additionally, the DeltaController has been integrated with S3Connector. Using try-catch can now read both formats. Cluster mode has been tested for both Delta and Parquet.