Generalize TestLakectlImport to support all object stores #10089
Conversation
Remove the TODO and update the test to work with S3, GS, and Azure blockstores by adding a helper function that selects the appropriate import path based on the configured blockstore type. https://claude.ai/code/session_01K9z1uofTKKktVVnRgeycG2
|
|
nopcoder
added
exclude-changelog
The previous paths (gs://lakectl-ingest-test-data and Azure equivalent) don't exist. Use the esti-system-testing-data/import-test-data/prefix-1/ paths which are already set up for all providers (S3, GS, Azure) and contain 2100 objects each. https://claude.ai/code/session_01K9z1uofTKKktVVnRgeycG2
| s3LakectlImportPath = "s3://esti-system-testing-data/import-test-data/prefix-1/" | ||
| gsLakectlImportPath = "gs://esti-system-testing-data/import-test-data/prefix-1/" | ||
| azureLakectlImportPath = "https://esti.blob.core.windows.net/esti-system-testing-data/import-test-data/prefix-1/" | ||
| lakectlImportObjects = "2100" |
There was a problem hiding this comment.
I wonder if it'd be possible to upload under a temporary prefix and import from it? That does mean we'll have to cleanup temporary prefixes from time to time.
That'd make it easier for me to test locally with Minio as I would be able to run all the tests in a single session.
There was a problem hiding this comment.
We can open the test to enable custom prefix - but it will also need to the verify the number of objects we import.
If I remember this test - it was not support to create the source for the import as it took long time to produce.
Still not happy from the result of the fix, so I'm still eval if we like this change in or not.
Remove the TODO and update the test to work with S3, GS, and Azure
blockstores by adding a helper function that selects the appropriate
import path based on the configured blockstore type.
https://claude.ai/code/session_01K9z1uofTKKktVVnRgeycG2