fix(package): Remove `archive_storage_type` column from the `datasets` table since it should be a cluster-level setting. #1029

haiqi96 · 2025-06-23T21:29:45Z

Description

This PR removes the storage_type column from the datasets table beause:

The storage type can be inferred from the package config.
Keeping the column may confuses user and make them feel that CLP supports datasets from different storage type at the sametime, which is not true.

Checklist

The PR satisfies the contribution guidelines.
This is a breaking change and that has been indicated in the PR title, OR this isn't a
breaking change.
Necessary docs have been updated, OR no docs need to be updated.

Validation performed

Started a clp-s package with default settings. Compressed and viewed a log and confirmed there is no error.
Manually checked the database with gui viewer and confirmed that the column is removed.
Searched for archive_storage_type in the codebase to ensure no other code references to it.

Summary by CodeRabbit

Refactor
- Removed the storage type field from dataset handling, simplifying dataset creation and management.
Documentation
- Updated metadata database design guide to reflect removal of storage type field and clarify archive directory details.

coderabbitai · 2025-06-23T21:30:08Z

Walkthrough

This change removes the archive_storage_type column from the datasets table schema and eliminates all related code references, including the parameter in the add_dataset function and its usage in function calls. The function signature and SQL statements are updated accordingly. Documentation describing the datasets table schema is also updated to reflect these removals.

Changes

Files/Paths	Change Summary
clp_py_utils/clp_metadata_db_utils.py	Removed `archive_storage_type` column and parameter from table schema, SQL, and `add_dataset`.
job_orchestration/scheduler/compress/compression_scheduler.py	Updated `add_dataset` call to omit the `archive_storage_type` argument.
docs/src/dev-guide/design-metadata-db.md	Removed `archive_storage_type` column from datasets table schema description and updated related text.

Possibly related PRs

feat(datasets): Add storage-type column to datasets table, and document datasets table. #923: Introduces the archive_storage_type column to the datasets table schema, which this PR subsequently removes, making them closely connected.
feat(clp-json): Use dataset-specific tables and archive directories for compression, decompression, and search. #868: Adds and uses the archive_storage_type column and dataset-specific table prefixes for dataset isolation in the metadata DB; this PR reverses those changes.

Suggested reviewers

kirkrodrigues

📜 Recent review details

Configuration used: CodeRabbit UI
Review profile: ASSERTIVE
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between a74c494 and 34f2adc.

📒 Files selected for processing (1)

docs/src/dev-guide/design-metadata-db.md (1 hunks)

⏰ Context from checks skipped due to timeout of 90000ms (2)

GitHub Check: lint-check (ubuntu-latest)
GitHub Check: lint-check (macos-latest)

🔇 Additional comments (1)

docs/src/dev-guide/design-metadata-db.md (1)

17-20: Clarify residual mention of “storage type” & sweep for stale references

The new description still refers to “the storage type (e.g., s3)”, which could leave readers wondering where that type is specified now that the archive_storage_type column is gone. Consider explicitly stating that the storage type is configured at the cluster/package level to avoid ambiguity.

While you’re at it, please run a quick sweep to ensure no other docs (or code comments) still reference archive_storage_type.
#!/bin/bash
# Find any lingering mentions of the removed column
rg -n "archive_storage_type"

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

Review comments: Directly reply to a review comment made by CodeRabbit. Example:
- I pushed a fix in commit <commit_id>, please review it.
- Explain this complex logic.
- Open a follow-up GitHub issue for this discussion.
Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
- @coderabbitai explain this code block.
- @coderabbitai modularize this function.
PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
- @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
- @coderabbitai read src/utils.ts and explain its main purpose.
- @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
- @coderabbitai help me debug CodeRabbit configuration file.

Support

Need help? Create a ticket on our support page for assistance with any issues or questions.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (Invoked using PR comments)

@coderabbitai pause to pause the reviews on a PR.
@coderabbitai resume to resume the paused reviews.
@coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
@coderabbitai full review to do a full review from scratch and review all the files again.
@coderabbitai summary to regenerate the summary of the PR.
@coderabbitai generate docstrings to generate docstrings for this PR.
@coderabbitai generate sequence diagram to generate a sequence diagram of the changes in this PR.
@coderabbitai resolve resolve all the CodeRabbit review comments.
@coderabbitai configuration to show the current CodeRabbit configuration for the repository.
@coderabbitai help to get help.

Other keywords and placeholders

Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
Add @coderabbitai anywhere in the PR title to generate the title automatically.

CodeRabbit Configuration File (`.coderabbit.yaml`)

You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
Please see the configuration documentation for more information.
If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

Visit our Documentation for detailed information on how to use CodeRabbit.
Join our Discord Community to get help, request features, and share feedback.
Follow us on X/Twitter for updates and announcements.

kirkrodrigues

We need to update the docs.

For the PR title, how about:

fix(package): Remove `archive_storage_type` column from the `datasets` table since it should be a cluster-level setting.

kirkrodrigues · 2025-06-24T20:12:29Z

Please also fill out the checklist.

haiqi96 · 2025-06-24T21:02:03Z

docs/src/dev-guide/design-metadata-db.md

+| Column name               | Type          | Description                                                                     |
+|---------------------------|---------------|---------------------------------------------------------------------------------|
+| name                      | VARCHAR(255)  | The *unique* name of the dataset.                                               |
+| archive_storage_directory | VARCHAR(4096) | The directory (on the CLP's storage type, e.g. `s3`) where archives are stored. |


or "(on the CLP's storage, e.g. `s3`)"?

docs/src/dev-guide/design-metadata-db.md

Co-authored-by: kirkrodrigues <2454684+kirkrodrigues@users.noreply.github.com>

Super simple fix

485971a

haiqi96 marked this pull request as ready for review June 23, 2025 21:29

haiqi96 requested a review from a team as a code owner June 23, 2025 21:29

missing change

26b3de5

linter so it doesn't kill me

6c33cd2

kirkrodrigues requested changes Jun 24, 2025

View reviewed changes

Update doc

a74c494

haiqi96 changed the title ~~fix(package): Remove storage_type column from the datasets table. (fix #1023)~~ fix(package): fix(package): Remove archive_storage_type column from the datasets table since it should be a cluster-level setting. (fix #1023) Jun 24, 2025

haiqi96 commented Jun 24, 2025

View reviewed changes

kirkrodrigues requested changes Jun 24, 2025

View reviewed changes

docs/src/dev-guide/design-metadata-db.md Outdated Show resolved Hide resolved

haiqi96 changed the title ~~fix(package): Remove archive_storage_type column from the datasets table since it should be a cluster-level setting. (fix #1023)~~ fix(package): Remove archive_storage_type column from the datasets table since it should be a cluster-level setting. Jun 24, 2025

Update docs/src/dev-guide/design-metadata-db.md

34f2adc

Co-authored-by: kirkrodrigues <2454684+kirkrodrigues@users.noreply.github.com>

kirkrodrigues approved these changes Jun 25, 2025

View reviewed changes

Merge branch 'main' into refactor_database_table

78e3654

haiqi96 merged commit 8ccccce into y-scope:main Jun 25, 2025
7 checks passed

coderabbitai bot mentioned this pull request Jun 30, 2025

fix(package): Refactor dataset handling in the package. #1023

Merged

3 tasks

coderabbitai bot mentioned this pull request Aug 13, 2025

feat(package): Add dataset-manager scripts to support listing datasets, and deleting them entirely. #1144

Merged

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

fix(package): Remove `archive_storage_type` column from the `datasets` table since it should be a cluster-level setting. #1029

fix(package): Remove `archive_storage_type` column from the `datasets` table since it should be a cluster-level setting. #1029

Uh oh!

haiqi96 commented Jun 23, 2025 •

edited

Loading

Uh oh!

coderabbitai bot commented Jun 23, 2025 •

edited

Loading

Chat

Support

CodeRabbit Commands (Invoked using PR comments)

Other keywords and placeholders

CodeRabbit Configuration File (`.coderabbit.yaml`)

Documentation and Community

Uh oh!

kirkrodrigues left a comment

Uh oh!

kirkrodrigues commented Jun 24, 2025

Uh oh!

haiqi96 Jun 24, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

fix(package): Remove archive_storage_type column from the datasets table since it should be a cluster-level setting. #1029

fix(package): Remove archive_storage_type column from the datasets table since it should be a cluster-level setting. #1029

Uh oh!

Conversation

haiqi96 commented Jun 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Checklist

Validation performed

Summary by CodeRabbit

Uh oh!

coderabbitai bot commented Jun 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Possibly related PRs

Suggested reviewers

Chat

Support

CodeRabbit Commands (Invoked using PR comments)

Other keywords and placeholders

CodeRabbit Configuration File (.coderabbit.yaml)

Documentation and Community

Uh oh!

kirkrodrigues left a comment

Choose a reason for hiding this comment

Uh oh!

kirkrodrigues commented Jun 24, 2025

Uh oh!

haiqi96 Jun 24, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

fix(package): Remove `archive_storage_type` column from the `datasets` table since it should be a cluster-level setting. #1029

fix(package): Remove `archive_storage_type` column from the `datasets` table since it should be a cluster-level setting. #1029

haiqi96 commented Jun 23, 2025 •

edited

Loading

coderabbitai bot commented Jun 23, 2025 •

edited

Loading

CodeRabbit Configuration File (`.coderabbit.yaml`)