Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Issue 264: Update qviz for multiblock files #437

Merged
Merged
Changes from 1 commit
Commits
Show all changes
39 commits
Select commit Hold shift + click to select a range
42e11d6
initial commit
jorgeMarin1 Sep 6, 2024
b0fda5d
Merge branch 'Qbeast-io:main' into qviz-bug
jorgeMarin1 Sep 10, 2024
0fe8b08
initial commit
jorgeMarin1 Sep 6, 2024
5fb0956
added process table function using delta tables
jorgeMarin1 Sep 10, 2024
5f60426
create qviz using Delta tables
jorgeMarin1 Sep 10, 2024
0ee8232
added custom table
jorgeMarin1 Sep 10, 2024
30256f8
added custom table
jorgeMarin1 Sep 10, 2024
cf9cfc3
removed ecommerce300k_2019
jorgeMarin1 Sep 10, 2024
cbcbaed
fixed visualization
jorgeMarin1 Sep 12, 2024
92cf732
deleted parquet files and added a new folder for table test
jorgeMarin1 Sep 12, 2024
e9559f7
added comments on the code
jorgeMarin1 Sep 12, 2024
ee95b87
added comments to the code
jorgeMarin1 Sep 12, 2024
2f0aea8
Merge branch 'main' into qviz-bug
jorgeMarin1 Sep 12, 2024
a267f87
deleted code and files that won't be used
jorgeMarin1 Sep 12, 2024
032077c
added unit test
jorgeMarin1 Sep 13, 2024
398494d
addressed changes
jorgeMarin1 Sep 17, 2024
353011f
addressed changes
jorgeMarin1 Sep 18, 2024
13e95a7
snake case
jorgeMarin1 Sep 18, 2024
31ac8e1
edited example of json log file
jorgeMarin1 Sep 19, 2024
8ecfdae
raised exception when failed to create delta table
jorgeMarin1 Sep 19, 2024
6f824d4
edited name of the method that extracted cubes from delta tables
jorgeMarin1 Sep 19, 2024
cb0f1a5
made sure blocks_str is indeed a string
jorgeMarin1 Sep 19, 2024
5f06c45
put definition of cube_string before if statement
jorgeMarin1 Sep 19, 2024
d565fff
added else statement for block_string
jorgeMarin1 Sep 25, 2024
0178d1e
deleted non-necessary comments
jorgeMarin1 Sep 25, 2024
a1d75a7
changed path to test table in README
jorgeMarin1 Sep 25, 2024
512dd67
default value of sampling fraction is 0.02 and that value can be modi…
jorgeMarin1 Sep 25, 2024
22ff6b5
deleted unnecessary files
jorgeMarin1 Sep 25, 2024
1faa7c2
switched to version 3.2.0 of Delta Lake, which is compatible with 3.5…
jorgeMarin1 Sep 25, 2024
cda8c6d
update size of cubes using path of blocks
jorgeMarin1 Sep 26, 2024
b407ada
WIP
Jiaweihu08 Oct 14, 2024
1b5ee6b
Remove large file
Jiaweihu08 Oct 14, 2024
aa486ca
WPI, first working version
Jiaweihu08 Oct 16, 2024
4bce299
Organize code, add tests
Jiaweihu08 Oct 16, 2024
b6855b0
Update README
Jiaweihu08 Oct 17, 2024
db3ebe9
Add comments
Jiaweihu08 Oct 17, 2024
a6b88a7
Update poetry lock and toml
Jiaweihu08 Oct 17, 2024
bc4ef3c
Update README
Jiaweihu08 Oct 17, 2024
2727e64
Correct stats computation
Jiaweihu08 Oct 18, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
put definition of cube_string before if statement
  • Loading branch information
jorgeMarin1 committed Sep 19, 2024
commit 5f06c451359d2bf3f4c7f35fb040f89aa74f59fb
6 changes: 3 additions & 3 deletions utils/visualizer/qviz/content_loader.py
Original file line number Diff line number Diff line change
Expand Up @@ -133,8 +133,9 @@ def extract_cubes_from_delta_table(delta_table: DeltaTable, symbol_count: int) -
# Check if we already have a cube in the dictionary with the cube ID of this block
# If we already have a cube with this ID, we update the atributes of the cube
# Since objects are passed by reference, they can be modified in situ
if block["cubeId"] in cubes_dict.keys():
dict_cube = cubes_dict[block["cubeId"]]
cube_string = block["cubeId"]
if cube_string in cubes_dict.keys():
dict_cube = cubes_dict[cube_string]
normalized_max_weight = normalize_weight(
int(block["maxWeight"])
)
Expand All @@ -144,7 +145,6 @@ def extract_cubes_from_delta_table(delta_table: DeltaTable, symbol_count: int) -
# If we don't have this cube in the dictionary, a new cube is created
else:
# Calculate depth of the cube
cube_string = block["cubeId"]
depth = len(cube_string) // symbol_count
# Assign max_weight, min_weight, elemnt_count a value for the new cube
max_weight = block["maxWeight"]
Expand Down