Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Check and improve versioning s3tests #472

Closed
0xavi0 opened this issue Apr 24, 2023 · 2 comments · Fixed by aquarist-labs/ceph#153
Closed

Check and improve versioning s3tests #472

0xavi0 opened this issue Apr 24, 2023 · 2 comments · Fixed by aquarist-labs/ceph#153
Assignees
@0xavi0
Labels
kind/bug Something isn't working kind/quality Quality improvements, Refactoring, Automation via CI, E2E, Integration, CLI or REST API
Milestone
v0.18.0

Comments

@0xavi0
Copy link
Contributor

0xavi0 commented Apr 24, 2023 •
edited
Loading

Right now we are running only 7 out of 21 s3 tests for versioning, because the rest are either failing or not checked for a long time.

It would be good to have a better s3 tests versioning coverage, which would mean that we are following the S3 standard for that particular area.

This is the list of tests not being executed

  • test_bucket_list_return_data_versioning
  • test_object_copy_versioning_multipart_upload
  • test_versioning_obj_create_read_remove
  • test_versioning_obj_create_read_remove_head
  • test_versioning_obj_plain_null_version_overwrite
  • test_versioning_obj_suspend_versions
  • test_versioning_obj_create_versions_remove_all
  • test_versioning_obj_create_versions_remove_special_names
  • test_versioning_obj_create_overwrite_multipart
  • test_versioning_obj_list_marker
  • test_versioning_copy_obj_version
  • test_versioning_multi_object_delete
  • test_versioning_multi_object_delete_with_maker
@0xavi0 0xavi0 added kind/bug Something isn't working kind/quality Quality improvements, Refactoring, Automation via CI, E2E, Integration, CLI or REST API labels Apr 24, 2023
@jhmarina jhmarina changed the title [BUG] Check and improve versioning s3tests Check and improve versioning s3tests May 9, 2023
@0xavi0 0xavi0 self-assigned this May 15, 2023
@jhmarina jhmarina added this to the v0.17.0 milestone May 15, 2023
@jhmarina jhmarina mentioned this issue May 15, 2023
Closed
@irq0
Copy link
Contributor

irq0 commented May 16, 2023

Also interesting:

test_versioned_concurrent_object_create_and_remove fail
test_versioned_concurrent_object_create_concurrent_remove fail

They test list versions

@irq0 irq0 added the triage/next-candidate This could be moved to the next milestone label May 17, 2023
@0xavi0
Copy link
Contributor Author

0xavi0 commented May 18, 2023

Implementing as in https://github.com/aquarist-labs/s3gw/pull/497

@jhmarina jhmarina removed the triage/next-candidate This could be moved to the next milestone label May 19, 2023
@jhmarina jhmarina modified the milestones: v0.17.0, v0.18.0 Jun 7, 2023
0xavi0 referenced this issue in 0xavi0/ceph Jun 7, 2023
@0xavi0
rgw/sfs: Implement new versioning design
5d350d9 
Implements new versioning design.

Includes `rgw::sal::Attrs` type binding in `sqliteorm` code, so
VersionedObject code for conversions is no longer needed.

Adds transactions for the actions that require more than 1 database
access.

Simplifies the way a new version is created and moves its logic to a
sqlite transaction to avoid race conditions when creating N objects with
the same name at once from several threads.

Fixes: https://github.com/aquarist-labs/s3gw/issues/378
Fixes: https://github.com/aquarist-labs/s3gw/issues/472
Fixes: https://github.com/aquarist-labs/s3gw/issues/547
Fixes: https://github.com/aquarist-labs/s3gw/issues/526
Fixes: https://github.com/aquarist-labs/s3gw/issues/524
Fixes: https://github.com/aquarist-labs/s3gw/issues/519

Signed-off-by: Xavi Garcia <xavi.garcia@suse.com>
0xavi0 referenced this issue in 0xavi0/ceph Jun 7, 2023
@0xavi0
rgw/sfs: Implement new versioning design
2ff1d10 
Implements new versioning design.

Includes `rgw::sal::Attrs` type binding in `sqliteorm` code, so
VersionedObject code for conversions is no longer needed.

Adds transactions for the actions that require more than 1 database
access.

Simplifies the way a new version is created and moves its logic to a
sqlite transaction to avoid race conditions when creating N objects with
the same name at once from several threads.

Fixes: https://github.com/aquarist-labs/s3gw/issues/378
Fixes: https://github.com/aquarist-labs/s3gw/issues/472
Fixes: https://github.com/aquarist-labs/s3gw/issues/547
Fixes: https://github.com/aquarist-labs/s3gw/issues/526
Fixes: https://github.com/aquarist-labs/s3gw/issues/524
Fixes: https://github.com/aquarist-labs/s3gw/issues/519

Signed-off-by: Xavi Garcia <xavi.garcia@suse.com>
0xavi0 referenced this issue in 0xavi0/ceph Jun 7, 2023
@0xavi0
rgw/sfs: Implement new versioning design
19633fa 
Implements new versioning design.

Includes `rgw::sal::Attrs` type binding in `sqliteorm` code, so
VersionedObject code for conversions is no longer needed.

Adds transactions for the actions that require more than 1 database
access.

Simplifies the way a new version is created and moves its logic to a
sqlite transaction to avoid race conditions when creating N objects with
the same name at once from several threads.

Fixes: https://github.com/aquarist-labs/s3gw/issues/378
Fixes: https://github.com/aquarist-labs/s3gw/issues/472
Fixes: https://github.com/aquarist-labs/s3gw/issues/547
Fixes: https://github.com/aquarist-labs/s3gw/issues/526
Fixes: https://github.com/aquarist-labs/s3gw/issues/524
Fixes: https://github.com/aquarist-labs/s3gw/issues/519

Signed-off-by: Xavi Garcia <xavi.garcia@suse.com>
0xavi0 referenced this issue in 0xavi0/ceph Jun 7, 2023
@0xavi0
rgw/sfs: Implement new versioning design
f426f46 
Implements new versioning design.

Includes `rgw::sal::Attrs` type binding in `sqliteorm` code, so
VersionedObject code for conversions is no longer needed.

Adds transactions for the actions that require more than 1 database
access.

Simplifies the way a new version is created and moves its logic to a
sqlite transaction to avoid race conditions when creating N objects with
the same name at once from several threads.

Fixes: https://github.com/aquarist-labs/s3gw/issues/378
Fixes: https://github.com/aquarist-labs/s3gw/issues/472
Fixes: https://github.com/aquarist-labs/s3gw/issues/547
Fixes: https://github.com/aquarist-labs/s3gw/issues/526
Fixes: https://github.com/aquarist-labs/s3gw/issues/524
Fixes: https://github.com/aquarist-labs/s3gw/issues/519

Signed-off-by: Xavi Garcia <xavi.garcia@suse.com>
0xavi0 referenced this issue in 0xavi0/ceph Jun 7, 2023
@0xavi0
rgw/sfs: Implement new versioning design
41442b4 
Implements new versioning design.

Includes `rgw::sal::Attrs` type binding in `sqliteorm` code, so
VersionedObject code for conversions is no longer needed.

Adds transactions for the actions that require more than 1 database
access.

Simplifies the way a new version is created and moves its logic to a
sqlite transaction to avoid race conditions when creating N objects with
the same name at once from several threads.

Fixes: https://github.com/aquarist-labs/s3gw/issues/378
Fixes: https://github.com/aquarist-labs/s3gw/issues/472
Fixes: https://github.com/aquarist-labs/s3gw/issues/547
Fixes: https://github.com/aquarist-labs/s3gw/issues/526
Fixes: https://github.com/aquarist-labs/s3gw/issues/524
Fixes: https://github.com/aquarist-labs/s3gw/issues/519

Signed-off-by: Xavi Garcia <xavi.garcia@suse.com>
0xavi0 referenced this issue in 0xavi0/ceph Jun 7, 2023
@0xavi0
rgw/sfs: Implement new versioning design
09f046d 
Implements new versioning design.

Includes `rgw::sal::Attrs` type binding in `sqliteorm` code, so
VersionedObject code for conversions is no longer needed.

Adds transactions for the actions that require more than 1 database
access.

Simplifies the way a new version is created and moves its logic to a
sqlite transaction to avoid race conditions when creating N objects with
the same name at once from several threads.

Fixes: https://github.com/aquarist-labs/s3gw/issues/378
Fixes: https://github.com/aquarist-labs/s3gw/issues/472
Fixes: https://github.com/aquarist-labs/s3gw/issues/547
Fixes: https://github.com/aquarist-labs/s3gw/issues/526
Fixes: https://github.com/aquarist-labs/s3gw/issues/524
Fixes: https://github.com/aquarist-labs/s3gw/issues/519

Signed-off-by: Xavi Garcia <xavi.garcia@suse.com>
@0xavi0 0xavi0 mentioned this issue Jun 7, 2023
Merged
11 tasks
0xavi0 referenced this issue in 0xavi0/ceph Jun 7, 2023
@0xavi0
rgw/sfs: Implement new versioning design
c988114 
Implements new versioning design.

Includes `rgw::sal::Attrs` type binding in `sqliteorm` code, so
VersionedObject code for conversions is no longer needed.

Adds transactions for the actions that require more than 1 database
access.

Simplifies the way a new version is created and moves its logic to a
sqlite transaction to avoid race conditions when creating N objects with
the same name at once from several threads.

Fixes: https://github.com/aquarist-labs/s3gw/issues/378
Fixes: https://github.com/aquarist-labs/s3gw/issues/472
Fixes: https://github.com/aquarist-labs/s3gw/issues/547
Fixes: https://github.com/aquarist-labs/s3gw/issues/526
Fixes: https://github.com/aquarist-labs/s3gw/issues/524
Fixes: https://github.com/aquarist-labs/s3gw/issues/519

Signed-off-by: Xavi Garcia <xavi.garcia@suse.com>
0xavi0 referenced this issue in 0xavi0/ceph Jun 8, 2023
@0xavi0
rgw/sfs: Implement new versioning design
6d645aa 
Implements new versioning design.

Includes `rgw::sal::Attrs` type binding in `sqliteorm` code, so
VersionedObject code for conversions is no longer needed.

Adds transactions for the actions that require more than 1 database
access.

Simplifies the way a new version is created and moves its logic to a
sqlite transaction to avoid race conditions when creating N objects with
the same name at once from several threads.

Fixes: https://github.com/aquarist-labs/s3gw/issues/378
Fixes: https://github.com/aquarist-labs/s3gw/issues/472
Fixes: https://github.com/aquarist-labs/s3gw/issues/547
Fixes: https://github.com/aquarist-labs/s3gw/issues/526
Fixes: https://github.com/aquarist-labs/s3gw/issues/524
Fixes: https://github.com/aquarist-labs/s3gw/issues/519

Signed-off-by: Xavi Garcia <xavi.garcia@suse.com>
0xavi0 referenced this issue in 0xavi0/ceph Jun 8, 2023
@0xavi0
rgw/sfs: Implement new versioning design
9aa8030 
Implements new versioning design.

Includes `rgw::sal::Attrs` type binding in `sqliteorm` code, so
VersionedObject code for conversions is no longer needed.

Adds transactions for the actions that require more than 1 database
access.

Simplifies the way a new version is created and moves its logic to a
sqlite transaction to avoid race conditions when creating N objects with
the same name at once from several threads.

Fixes: https://github.com/aquarist-labs/s3gw/issues/378
Fixes: https://github.com/aquarist-labs/s3gw/issues/472
Fixes: https://github.com/aquarist-labs/s3gw/issues/547
Fixes: https://github.com/aquarist-labs/s3gw/issues/526
Fixes: https://github.com/aquarist-labs/s3gw/issues/524
Fixes: https://github.com/aquarist-labs/s3gw/issues/519

Signed-off-by: Xavi Garcia <xavi.garcia@suse.com>
0xavi0 referenced this issue in 0xavi0/ceph Jun 8, 2023
@0xavi0
rgw/sfs: Implement new versioning design
def1dd6 
Implements new versioning design.

Includes `rgw::sal::Attrs` type binding in `sqliteorm` code, so
VersionedObject code for conversions is no longer needed.

Adds transactions for the actions that require more than 1 database
access.

Simplifies the way a new version is created and moves its logic to a
sqlite transaction to avoid race conditions when creating N objects with
the same name at once from several threads.

Fixes: https://github.com/aquarist-labs/s3gw/issues/378
Fixes: https://github.com/aquarist-labs/s3gw/issues/472
Fixes: https://github.com/aquarist-labs/s3gw/issues/547
Fixes: https://github.com/aquarist-labs/s3gw/issues/526
Fixes: https://github.com/aquarist-labs/s3gw/issues/524
Fixes: https://github.com/aquarist-labs/s3gw/issues/519

Signed-off-by: Xavi Garcia <xavi.garcia@suse.com>
0xavi0 referenced this issue in 0xavi0/ceph Jun 8, 2023
@0xavi0
rgw/sfs: Implement new versioning design
ada0866 
Implements new versioning design.

Includes `rgw::sal::Attrs` type binding in `sqliteorm` code, so
VersionedObject code for conversions is no longer needed.

Adds transactions for the actions that require more than 1 database
access.

Simplifies the way a new version is created and moves its logic to a
sqlite transaction to avoid race conditions when creating N objects with
the same name at once from several threads.

Fixes: https://github.com/aquarist-labs/s3gw/issues/378
Fixes: https://github.com/aquarist-labs/s3gw/issues/472
Fixes: https://github.com/aquarist-labs/s3gw/issues/547
Fixes: https://github.com/aquarist-labs/s3gw/issues/526
Fixes: https://github.com/aquarist-labs/s3gw/issues/524
Fixes: https://github.com/aquarist-labs/s3gw/issues/519

Signed-off-by: Xavi Garcia <xavi.garcia@suse.com>
0xavi0 referenced this issue in 0xavi0/ceph Jun 8, 2023
@0xavi0
rgw/sfs: Implement new versioning design
a32f63e 
Implements new versioning design.

Includes `rgw::sal::Attrs` type binding in `sqliteorm` code, so
VersionedObject code for conversions is no longer needed.

Adds transactions for the actions that require more than 1 database
access.

Simplifies the way a new version is created and moves its logic to a
sqlite transaction to avoid race conditions when creating N objects with
the same name at once from several threads.

Fixes: https://github.com/aquarist-labs/s3gw/issues/378
Fixes: https://github.com/aquarist-labs/s3gw/issues/472
Fixes: https://github.com/aquarist-labs/s3gw/issues/547
Fixes: https://github.com/aquarist-labs/s3gw/issues/526
Fixes: https://github.com/aquarist-labs/s3gw/issues/524
Fixes: https://github.com/aquarist-labs/s3gw/issues/519

Signed-off-by: Xavi Garcia <xavi.garcia@suse.com>
0xavi0 referenced this issue in 0xavi0/ceph Jun 8, 2023
@0xavi0
rgw/sfs: Implement new versioning design
2a17d35 
Implements new versioning design.

Includes `rgw::sal::Attrs` type binding in `sqliteorm` code, so
VersionedObject code for conversions is no longer needed.

Adds transactions for the actions that require more than 1 database
access.

Simplifies the way a new version is created and moves its logic to a
sqlite transaction to avoid race conditions when creating N objects with
the same name at once from several threads.

Fixes: https://github.com/aquarist-labs/s3gw/issues/378
Fixes: https://github.com/aquarist-labs/s3gw/issues/472
Fixes: https://github.com/aquarist-labs/s3gw/issues/547
Fixes: https://github.com/aquarist-labs/s3gw/issues/526
Fixes: https://github.com/aquarist-labs/s3gw/issues/524
Fixes: https://github.com/aquarist-labs/s3gw/issues/519

Signed-off-by: Xavi Garcia <xavi.garcia@suse.com>
0xavi0 referenced this issue in 0xavi0/ceph Jun 8, 2023
@0xavi0
rgw/sfs: Implement new versioning design
eb150a2 
Implements new versioning design.

Includes `rgw::sal::Attrs` type binding in `sqliteorm` code, so
VersionedObject code for conversions is no longer needed.

Adds transactions for the actions that require more than 1 database
access.

Simplifies the way a new version is created and moves its logic to a
sqlite transaction to avoid race conditions when creating N objects with
the same name at once from several threads.

Fixes: https://github.com/aquarist-labs/s3gw/issues/378
Fixes: https://github.com/aquarist-labs/s3gw/issues/472
Fixes: https://github.com/aquarist-labs/s3gw/issues/547
Fixes: https://github.com/aquarist-labs/s3gw/issues/526
Fixes: https://github.com/aquarist-labs/s3gw/issues/524
Fixes: https://github.com/aquarist-labs/s3gw/issues/519

Signed-off-by: Xavi Garcia <xavi.garcia@suse.com>
0xavi0 referenced this issue in 0xavi0/ceph Jun 8, 2023
@0xavi0
rgw/sfs: Implement new versioning design
8bd78d5 
Implements new versioning design.

Includes `rgw::sal::Attrs` type binding in `sqliteorm` code, so
VersionedObject code for conversions is no longer needed.

Adds transactions for the actions that require more than 1 database
access.

Simplifies the way a new version is created and moves its logic to a
sqlite transaction to avoid race conditions when creating N objects with
the same name at once from several threads.

Fixes: https://github.com/aquarist-labs/s3gw/issues/378
Fixes: https://github.com/aquarist-labs/s3gw/issues/472
Fixes: https://github.com/aquarist-labs/s3gw/issues/547
Fixes: https://github.com/aquarist-labs/s3gw/issues/526
Fixes: https://github.com/aquarist-labs/s3gw/issues/524
Fixes: https://github.com/aquarist-labs/s3gw/issues/519

Signed-off-by: Xavi Garcia <xavi.garcia@suse.com>
0xavi0 referenced this issue in 0xavi0/ceph Jun 9, 2023
@0xavi0
rgw/sfs: Implement new versioning design
86792d9 
Implements new versioning design.

Includes `rgw::sal::Attrs` type binding in `sqliteorm` code, so
VersionedObject code for conversions is no longer needed.

Adds transactions for the actions that require more than 1 database
access.

Simplifies the way a new version is created and moves its logic to a
sqlite transaction to avoid race conditions when creating N objects with
the same name at once from several threads.

Fixes: https://github.com/aquarist-labs/s3gw/issues/378
Fixes: https://github.com/aquarist-labs/s3gw/issues/472
Fixes: https://github.com/aquarist-labs/s3gw/issues/547
Fixes: https://github.com/aquarist-labs/s3gw/issues/526
Fixes: https://github.com/aquarist-labs/s3gw/issues/524
Fixes: https://github.com/aquarist-labs/s3gw/issues/519

Signed-off-by: Xavi Garcia <xavi.garcia@suse.com>
0xavi0 referenced this issue in aquarist-labs/ceph Oct 5, 2023
@0xavi0
rgw/sfs: Implement new versioning design
83e527b 
Implements new versioning design.

Includes `rgw::sal::Attrs` type binding in `sqliteorm` code, so
VersionedObject code for conversions is no longer needed.

Adds transactions for the actions that require more than 1 database
access.

Simplifies the way a new version is created and moves its logic to a
sqlite transaction to avoid race conditions when creating N objects with
the same name at once from several threads.

Fixes: https://github.com/aquarist-labs/s3gw/issues/378
Fixes: https://github.com/aquarist-labs/s3gw/issues/472
Fixes: https://github.com/aquarist-labs/s3gw/issues/547
Fixes: https://github.com/aquarist-labs/s3gw/issues/526
Fixes: https://github.com/aquarist-labs/s3gw/issues/524
Fixes: https://github.com/aquarist-labs/s3gw/issues/519

Signed-off-by: Xavi Garcia <xavi.garcia@suse.com>
0xavi0 referenced this issue in 0xavi0/ceph Oct 5, 2023
@0xavi0
rgw/sfs: Implement new versioning design
b1a0a11 
Implements new versioning design.

Includes `rgw::sal::Attrs` type binding in `sqliteorm` code, so
VersionedObject code for conversions is no longer needed.

Adds transactions for the actions that require more than 1 database
access.

Simplifies the way a new version is created and moves its logic to a
sqlite transaction to avoid race conditions when creating N objects with
the same name at once from several threads.

Fixes: https://github.com/aquarist-labs/s3gw/issues/378
Fixes: https://github.com/aquarist-labs/s3gw/issues/472
Fixes: https://github.com/aquarist-labs/s3gw/issues/547
Fixes: https://github.com/aquarist-labs/s3gw/issues/526
Fixes: https://github.com/aquarist-labs/s3gw/issues/524
Fixes: https://github.com/aquarist-labs/s3gw/issues/519

Signed-off-by: Xavi Garcia <xavi.garcia@suse.com>
0xavi0 referenced this issue in aquarist-labs/ceph Oct 18, 2023
@0xavi0
rgw/sfs: Implement new versioning design
3e14e1d 
Implements new versioning design.

Includes `rgw::sal::Attrs` type binding in `sqliteorm` code, so
VersionedObject code for conversions is no longer needed.

Adds transactions for the actions that require more than 1 database
access.

Simplifies the way a new version is created and moves its logic to a
sqlite transaction to avoid race conditions when creating N objects with
the same name at once from several threads.

Fixes: https://github.com/aquarist-labs/s3gw/issues/378
Fixes: https://github.com/aquarist-labs/s3gw/issues/472
Fixes: https://github.com/aquarist-labs/s3gw/issues/547
Fixes: https://github.com/aquarist-labs/s3gw/issues/526
Fixes: https://github.com/aquarist-labs/s3gw/issues/524
Fixes: https://github.com/aquarist-labs/s3gw/issues/519

Signed-off-by: Xavi Garcia <xavi.garcia@suse.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@0xavi0 0xavi0

Labels
kind/bug Something isn't working kind/quality Quality improvements, Refactoring, Automation via CI, E2E, Integration, CLI or REST API
Projects
None yet
Milestone
v0.18.0
Development

Successfully merging a pull request may close this issue.

rgw/sfs: Implement new versioning design 0xavi0/ceph
3 participants
@irq0 @jhmarina @0xavi0