From d366659bfbd0bcff13d4dc0f03350ad0e0edb72b Mon Sep 17 00:00:00 2001 From: Tamer Sherif <69483382+tasherif-msft@users.noreply.github.com> Date: Wed, 16 Aug 2023 13:53:17 -0700 Subject: [PATCH] [AzDatalake] APIView Feedback + Samples + Doc + CI/Live issues cleanup (#21348) * Enable gocritic during linting (#20715) Enabled gocritic's evalOrder to catch dependencies on undefined behavior on return statements. Updated to latest version of golangci-lint. Fixed issue in azblob flagged by latest linter. * Cosmos DB: Enable merge support (#20716) * Adding header and value * Wiring and tests * format * Fixing value * change log * [azservicebus, azeventhubs] Stress test and logging improvement (#20710) Logging improvements: * Updating the logging to print more tracing information (per-link) in prep for the bigger release coming up. * Trimming out some of the verbose logging, seeing if I can get it a bit more reasonable. Stress tests: * Add a timestamp to the log name we generate and also default to append, not overwrite. * Use 0.5 cores, 0.5GB as our baseline. Some pods use more and I'll tune them more later. * update proxy version (#20712) Co-authored-by: Scott Beddall * Return an error when you try to send a message that's too large. (#20721) This now works just like the message batch - you'll get an ErrMessageTooLarge if you attempt to send a message that's too large for the link's configured size. NOTE: there's a patch to `internal/go-amqp/Sender.go` to match what's in go-amqp's main so it returns a programmatically useful error when the message is too large. Fixes #20647 * Changes in test that is failing in pipeline (#20693) * [azservicebus, azeventhubs] Treat 'entity full' as a fatal error (#20722) When the remote entity is full we get a resource-limit-exceeded condition. This isn't something we should keep retrying on and it's best to just abort and let the user know immediately, rather than hoping it might eventually clear out. This affected both Event Hubs and Service Bus. Fixes #20647 * [azservicebus/azeventhubs] Redirect stderr and stdout to tee (#20726) * Update changelog with latest features (#20730) * Update changelog with latest features Prepare for upcoming release. * bump minor version * pass along the artifact name so we can override it later (#20732) Co-authored-by: scbedd <45376673+scbedd@users.noreply.github.com> * [azeventhubs] Fixing checkpoint store race condition (#20727) The checkpoint store wasn't guarding against multiple owners claiming for the first time - fixing this by using IfNoneMatch Fixes #20717 * Fix azidentity troubleshooting guide link (#20736) * [Release] sdk/resourcemanager/paloaltonetworksngfw/armpanngfw/0.1.0 (#20437) * [Release] sdk/resourcemanager/paloaltonetworksngfw/armpanngfw/0.1.0 generation from spec commit: 85fb4ac6f8bfefd179e6c2632976a154b5c9ff04 * client factory * fix * fix * update * add sdk/resourcemanager/postgresql/armpostgresql live test (#20685) * add sdk/resourcemanager/postgresql/armpostgresql live test * update assets.json * set subscriptionId default value * format * add sdk/resourcemanager/eventhub/armeventhub live test (#20686) * add sdk/resourcemanager/eventhub/armeventhub live test * update assets * add sdk/resourcemanager/compute/armcompute live test (#20048) * add sdk/resourcemanager/compute/armcompute live test * skus filter * fix subscriptionId default value * fix * gofmt * update recording * sdk/resourcemanager/network/armnetwork live test (#20331) * sdk/resourcemanager/network/armnetwork live test * update subscriptionId default value * update recording * add sdk/resourcemanager/cosmos/armcosmos live test (#20705) * add sdk/resourcemanager/cosmos/armcosmos live test * update assets.json * update assets.json * update assets.json * update assets.json * Increment package version after release of azcore (#20740) * [azeventhubs] Improperly resetting etag in the checkpoint store (#20737) We shouldn't be resetting the etag to nil - it's what we use to enforce a "single winner" when doing ownership claims. The bug here was two-fold: I had bad logic in my previous claim ownership, which I fixed in a previous PR, but we need to reflect that same constraint properly in our in-memory checkpoint store for these tests. * Eng workflows sync and branch cleanup additions (#20743) Co-authored-by: James Suplizio * [azeventhubs] Latest start position can also be inclusive (ie, get the latest message) (#20744) * Update GitHubEventProcessor version and remove pull_request_review procesing (#20751) Co-authored-by: James Suplizio * Rename DisableAuthorityValidationAndInstanceDiscovery (#20746) * fix (#20707) * AzFile (#20739) * azfile: Fixing connection string parsing logic (#20798) * Fixing connection string parse logic * Update README * [azadmin] fix flaky test (#20758) * fix flaky test * charles suggestion * Prepare azidentity v1.3.0 for release (#20756) * Fix broken podman link (#20801) Co-authored-by: Wes Haggard * [azquery] update doc comments (#20755) * update doc comments * update statistics and visualization generation * prep-for-release * Fixed contribution section (#20752) Co-authored-by: Bob Tabor * [azeventhubs,azservicebus] Some API cleanup, renames (#20754) * Adding options to UpdateCheckpoint(), just for future potential expansion * Make Offset an int64, not a *int64 (it's not optional, it'll always come back with ReceivedEvents) * Adding more logging into the checkpoint store. * Point all imports at the production go-amqp * Add supporting features to enable distributed tracing (#20301) (#20708) * Add supporting features to enable distributed tracing This includes new internal pipeline policies and other supporting types. See the changelog for a full description. Added some missing doc comments. * fix linter issue * add net.peer.name trace attribute sequence custom HTTP header policy before logging policy. sequence logging policy after HTTP trace policy. keep body download policy at the end. * add span for iterating over pages * Restore ARM CAE support for azcore beta (#20657) This reverts commit 902097226ff3fe2fc6c3e7fc50d3478350253614. * Upgrade to stable azcore (#20808) * Increment package version after release of data/azcosmos (#20807) * Updating changelog (#20810) * Add fake package to azcore (#20711) * Add fake package to azcore This is the supporting infrastructure for the generated SDK fakes. * fix doc comment * Updating CHANGELOG.md (#20809) * changelog (#20811) * Increment package version after release of storage/azfile (#20813) * Update changelog (azblob) (#20815) * Updating CHANGELOG.md * Update the changelog with correct version * [azquery] migration guide (#20742) * migration guide * Charles feedback * Richard feedback --------- Co-authored-by: Charles Lowell <10964656+chlowell@users.noreply.github.com> * Increment package version after release of monitor/azquery (#20820) * [keyvault] prep for release (#20819) * prep for release * perf tests * update date * small cleanup * rename to FileSystem * regenrated * removed metadata option and fixed generated * fixed tests * fixed tests * fixed tests * fixed tests * added more tests * doc cleanup * doc failures * doc failures * doc failures * changelog * samples * mod tidy * removed client from rename response * fixed deserialize * cleanup * readme * live pipe fix * live pipe fix * live pipe fix * doc fix * doc fix * doc fix * model fix * model fix * doc fix * added doc.go * added file examples * apiview model fixes * version and changelog * changelog fix * fixed export --------- Co-authored-by: Joel Hendrix Co-authored-by: Matias Quaranta Co-authored-by: Richard Park <51494936+richardpark-msft@users.noreply.github.com> Co-authored-by: Azure SDK Bot <53356347+azure-sdk@users.noreply.github.com> Co-authored-by: Scott Beddall Co-authored-by: siminsavani-msft <77068571+siminsavani-msft@users.noreply.github.com> Co-authored-by: scbedd <45376673+scbedd@users.noreply.github.com> Co-authored-by: Charles Lowell <10964656+chlowell@users.noreply.github.com> Co-authored-by: Peng Jiahui <46921893+Alancere@users.noreply.github.com> Co-authored-by: James Suplizio Co-authored-by: Sourav Gupta <98318303+souravgupta-msft@users.noreply.github.com> Co-authored-by: gracewilcox <43627800+gracewilcox@users.noreply.github.com> Co-authored-by: Wes Haggard Co-authored-by: Bob Tabor Co-authored-by: Bob Tabor --- sdk/storage/azdatalake/CHANGELOG.md | 7 + sdk/storage/azdatalake/README.md | 280 +++++++ sdk/storage/azdatalake/assets.json | 2 +- .../azdatalake/datalakeerror/error_codes.go | 13 +- sdk/storage/azdatalake/directory/client.go | 78 +- .../azdatalake/directory/client_test.go | 582 ++++++-------- sdk/storage/azdatalake/directory/constants.go | 1 + .../azdatalake/directory/examples_test.go | 166 ++++ sdk/storage/azdatalake/directory/models.go | 28 +- sdk/storage/azdatalake/directory/responses.go | 7 +- sdk/storage/azdatalake/doc.go | 218 ++++++ sdk/storage/azdatalake/file/client.go | 65 +- sdk/storage/azdatalake/file/client_test.go | 721 ++++++++---------- sdk/storage/azdatalake/file/constants.go | 2 + sdk/storage/azdatalake/file/examples_test.go | 389 ++++++++++ sdk/storage/azdatalake/file/models.go | 79 +- sdk/storage/azdatalake/file/responses.go | 8 +- sdk/storage/azdatalake/file/retry_reader.go | 1 + .../azdatalake/file/retry_reader_test.go | 433 +++++++++++ sdk/storage/azdatalake/filesystem/client.go | 38 +- .../azdatalake/filesystem/client_test.go | 480 +++++++----- .../azdatalake/filesystem/constants.go | 2 +- .../azdatalake/filesystem/examples_test.go | 330 ++++++++ sdk/storage/azdatalake/filesystem/models.go | 65 +- .../azdatalake/filesystem/responses.go | 24 +- sdk/storage/azdatalake/go.mod | 8 + sdk/storage/azdatalake/go.sum | 11 + .../azdatalake/internal/base/clients.go | 6 +- .../internal/exported/access_conditions.go | 2 +- .../azdatalake/internal/exported/exported.go | 2 +- .../exported/shared_key_credential.go | 2 +- .../azdatalake/internal/exported/version.go | 2 +- .../azdatalake/internal/generated/autorest.md | 19 +- .../internal/generated/filesystem_client.go | 4 +- .../internal/generated/zz_models.go | 4 +- .../internal/generated/zz_models_serde.go | 11 +- .../internal/generated_blob/autorest.md | 2 +- .../generated_blob/zz_container_client.go | 14 +- .../internal/generated_blob/zz_models.go | 8 +- .../azdatalake/internal/path/models.go | 45 +- .../azdatalake/internal/path/responses.go | 45 ++ .../azdatalake/internal/shared/shared.go | 2 +- .../internal/testcommon/clients_auth.go | 12 +- .../azdatalake/internal/testcommon/common.go | 6 +- sdk/storage/azdatalake/lease/client.go | 26 +- sdk/storage/azdatalake/lease/client_test.go | 100 +-- sdk/storage/azdatalake/lease/examples_test.go | 92 +++ sdk/storage/azdatalake/lease/models.go | 46 +- sdk/storage/azdatalake/lease/responses.go | 20 +- sdk/storage/azdatalake/sas/query_params.go | 22 +- sdk/storage/azdatalake/sas/service.go | 26 +- sdk/storage/azdatalake/sas/service_test.go | 93 ++- sdk/storage/azdatalake/sas/url_parts.go | 10 +- sdk/storage/azdatalake/sas/url_parts_test.go | 8 +- sdk/storage/azdatalake/service/client.go | 47 +- sdk/storage/azdatalake/service/client_test.go | 96 +-- sdk/storage/azdatalake/service/constants.go | 2 +- .../azdatalake/service/examples_test.go | 335 ++++++++ sdk/storage/azdatalake/service/models.go | 76 +- sdk/storage/azdatalake/service/responses.go | 12 +- sdk/storage/azdatalake/test-resources.json | 9 +- 61 files changed, 3778 insertions(+), 1466 deletions(-) create mode 100644 sdk/storage/azdatalake/CHANGELOG.md create mode 100644 sdk/storage/azdatalake/README.md create mode 100644 sdk/storage/azdatalake/directory/examples_test.go create mode 100644 sdk/storage/azdatalake/doc.go create mode 100644 sdk/storage/azdatalake/file/examples_test.go create mode 100644 sdk/storage/azdatalake/file/retry_reader_test.go create mode 100644 sdk/storage/azdatalake/filesystem/examples_test.go create mode 100644 sdk/storage/azdatalake/lease/examples_test.go create mode 100644 sdk/storage/azdatalake/service/examples_test.go diff --git a/sdk/storage/azdatalake/CHANGELOG.md b/sdk/storage/azdatalake/CHANGELOG.md new file mode 100644 index 000000000000..5ed8f538c47e --- /dev/null +++ b/sdk/storage/azdatalake/CHANGELOG.md @@ -0,0 +1,7 @@ +# Release History + +## 0.1.0-beta.1 (2023-08-16) + +### Features Added + +* This is the initial preview release of the `azdatalake` library diff --git a/sdk/storage/azdatalake/README.md b/sdk/storage/azdatalake/README.md new file mode 100644 index 000000000000..cf1600dd706c --- /dev/null +++ b/sdk/storage/azdatalake/README.md @@ -0,0 +1,280 @@ +# ADLS Gen2 Storage SDK for Go + +> Service Version: 2020-10-02 + +Azure Data Lake Storage Gen2 (ADLS Gen2) is Microsoft's hierarchical object storage solution for the cloud with converged capabilities with Azure Blob Storage. +For example, Data Lake Storage Gen2 provides file system semantics, file-level security, and scale. +Because these capabilities are built on Blob storage, you also get low-cost, tiered storage, with high availability/disaster recovery capabilities. +ADLS Gen2 makes Azure Storage the foundation for building enterprise data lakes on Azure. +Designed from the start to service multiple petabytes of information while sustaining hundreds of gigabits of throughput, ADLS Gen2 allows you to easily manage massive amounts of data. + +[Source code][source] | [API reference documentation][docs] | [REST API documentation][rest_docs] + +## Getting started + +### Install the package + +Install the ADLS Gen2 Storage SDK for Go with [go get][goget]: + +```Powershell +go get github.com/Azure/azure-sdk-for-go/sdk/storage/azdatalake +``` + +If you're going to authenticate with Azure Active Directory (recommended), install the [azidentity][azidentity] module. +```Powershell +go get github.com/Azure/azure-sdk-for-go/sdk/azidentity +``` + +### Prerequisites + +A supported [Go][godevdl] version (the Azure SDK supports the two most recent Go releases). + +You need an [Azure subscription][azure_sub] and a +[Storage Account][storage_account_docs] to use this package. + +To create a new Storage Account, you can use the [Azure Portal][storage_account_create_portal], +[Azure PowerShell][storage_account_create_ps], or the [Azure CLI][storage_account_create_cli]. +Here's an example using the Azure CLI: + +```Powershell +az storage account create --name MyStorageAccount --resource-group MyResourceGroup --location westus --sku Standard_LRS +``` + +### Authenticate the client + +In order to interact with the ADLS Gen2 Storage service, you'll need to create an instance of the `Client` type. The [azidentity][azidentity] module makes it easy to add Azure Active Directory support for authenticating Azure SDK clients with their corresponding Azure services. + +```go +// create a credential for authenticating with Azure Active Directory +cred, err := azidentity.NewDefaultAzureCredential(nil) +// TODO: handle err + +// create a service.Client for the specified storage account that uses the above credential +client, err := service.NewClient("https://MYSTORAGEACCOUNT.dfs.core.windows.net/", cred, nil) +// TODO: handle err +// you can also create filesystem, file and directory clients +``` + +Learn more about enabling Azure Active Directory for authentication with Azure Storage in [our documentation][storage_ad] and [our samples](#next-steps). + +## Key concepts + +ADLS Gen2 provides: +- Hadoop-compatible access +- Hierarchical directory structure +- Optimized cost and performance +- Finer grain security model +- Massive scalability + +ADLS Gen2 storage is designed for: + +- Serving images or documents directly to a browser. +- Storing files for distributed access. +- Streaming video and audio. +- Writing to log files. +- Storing data for backup and restore, disaster recovery, and archiving. +- Storing data for analysis by an on-premises or Azure-hosted service. + +ADLS Gen2 storage offers three types of resources: + +- The _storage account_ +- One or more _filesystems_ in a storage account +- One or more _files_ or _directories_ in a filesystem + +Instances of the `Client` type provide methods for manipulating filesystems and paths within a storage account. +The storage account is specified when the `Client` is constructed. The clients available are referenced below. +Use the appropriate client constructor function for the authentication mechanism you wish to use. + +### Goroutine safety +We guarantee that all client instance methods are goroutine-safe and independent of each other ([guideline](https://azure.github.io/azure-sdk/golang_introduction.html#thread-safety)). This ensures that the recommendation of reusing client instances is always safe, even across goroutines. + +### About metadata +ADLS Gen2 metadata name/value pairs are valid HTTP headers and should adhere to all restrictions governing HTTP headers. Metadata names must be valid HTTP header names, may contain only ASCII characters, and should be treated as case-insensitive. Base64-encode or URL-encode metadata values containing non-ASCII characters. + +### Additional concepts + +[Client options](https://pkg.go.dev/github.com/Azure/azure-sdk-for-go/sdk/azcore/policy#ClientOptions) | +[Accessing the response](https://pkg.go.dev/github.com/Azure/azure-sdk-for-go/sdk/azcore/runtime#WithCaptureResponse) | +[Handling failures](https://pkg.go.dev/github.com/Azure/azure-sdk-for-go/sdk/azcore#ResponseError) | +[Logging](https://pkg.go.dev/github.com/Azure/azure-sdk-for-go/sdk/azcore/log) + + +## Examples + +### Creating and uploading a file (assuming filesystem exists) + +```go +const ( + path = "https://MYSTORAGEACCOUNT.dfs.core.windows.net/sample-fs/sample-file" +) + +// authenticate with Azure Active Directory +cred, err := azidentity.NewDefaultAzureCredential(nil) +// TODO: handle error + +// create a client for the specified storage account +client, err := file.NewClient(path, cred, nil) +// TODO: handle error + +_, err = client.Create(context.TODO(), nil) +// TODO: handle error + +// open the file for reading +fh, err := os.OpenFile(sampleFile, os.O_RDONLY, 0) +// TODO: handle error +defer fh.Close() + +// upload the file to the specified filesystem with the specified file name +_, err = client.UploadFile(context.TODO(), fh, nil) +// TODO: handle error +``` + +### Downloading a file + +```go +const ( + path = "https://MYSTORAGEACCOUNT.dfs.core.windows.net/sample-fs/cloud.jpg" +) + +// authenticate with Azure Active Directory +cred, err := azidentity.NewDefaultAzureCredential(nil) +// TODO: handle error + +// create a client for the specified storage account +client, err := file.NewClient(path, cred, nil) +// TODO: handle error + +// create or open a local file where we can download the file +file, err := os.Create("cloud.jpg") +// TODO: handle error +defer file.Close() + +// download the file +_, err = client.DownloadFile(context.TODO(), file, nil) +// TODO: handle error +``` + +### Creating and deleting a filesystem + +```go +const ( + fs = "https://MYSTORAGEACCOUNT.dfs.core.windows.net/sample-fs" +) + +// authenticate with Azure Active Directory +cred, err := azidentity.NewDefaultAzureCredential(nil) +// TODO: handle error + +// create a client for the specified storage account +client, err := filesystem.NewClient(fs, cred, nil) +// TODO: handle error + +_, err = client.Create(context.TODO(), nil) +// TODO: handle error + +_, err = client.Delete(context.TODO(), nil) +// TODO: handle error +``` + +### Enumerating paths (assuming filesystem exists) + +```go +const ( + fs = "https://MYSTORAGEACCOUNT.dfs.core.windows.net/sample-fs" +) + +// authenticate with Azure Active Directory +cred, err := azidentity.NewDefaultAzureCredential(nil) +// TODO: handle error + +// create a filesystem client for the specified storage account +client, err := filesystem.NewClient(fs, cred, nil) +// TODO: handle error + +// path listings are returned across multiple pages +pager := client.NewListPathsPager(true, nil) + +// continue fetching pages until no more remain +for pager.More() { + // advance to the next page + page, err := pager.NextPage(context.TODO()) + // TODO: handle error + + // print the path names for this page + for _, path := range page.PathList.Paths { + fmt.Println(*path.Name) + fmt.Println(*path.IsDirectory) + } +} +``` + +## Troubleshooting + +All Datalake service operations will return an +[*azcore.ResponseError][azcore_response_error] on failure with a +populated `ErrorCode` field. Many of these errors are recoverable. +The [datalakeerror][datalake_error] package provides the possible Storage error codes +along with various helper facilities for error handling. + + +### Specialized clients + +The ADLS Gen2 Storage SDK for Go provides specialized clients in various subpackages. + +The [file][file] package contains APIs related to file path types. + +The [directory][directory] package contains APIs related to directory path types. + +The [lease][lease] package contains clients for managing leases on paths (paths represent both directory and file paths) and filesystems. Please see the [reference docs](https://docs.microsoft.com/rest/api/storageservices/lease-blob#remarks) for general information on leases. + +The [filesystem][filesystem] package contains APIs specific to filesystems. This includes APIs setting access policies or properties, and more. + +The [service][service] package contains APIs specific to Datalake service. This includes APIs for manipulating filesystems, retrieving account information, and more. + +The [sas][sas] package contains utilities to aid in the creation and manipulation of Shared Access Signature tokens. +See the package's documentation for more information. + + +You can find additional context and examples in our samples for each subpackage (named examples_test.go). + +## Contributing + +See the [Storage CONTRIBUTING.md][storage_contrib] for details on building, +testing, and contributing to this library. + +This project welcomes contributions and suggestions. Most contributions require +you to agree to a Contributor License Agreement (CLA) declaring that you have +the right to, and actually do, grant us the rights to use your contribution. For +details, visit [cla.microsoft.com][cla]. + +This project has adopted the [Microsoft Open Source Code of Conduct][coc]. +For more information see the [Code of Conduct FAQ][coc_faq] +or contact [opencode@microsoft.com][coc_contact] with any +additional questions or comments. + + +[source]: https://github.com/Azure/azure-sdk-for-go/tree/main/sdk/storage/azdatalake +[docs]: https://pkg.go.dev/github.com/Azure/azure-sdk-for-go/sdk/storage/azdatalake +[rest_docs]: https://docs.microsoft.com/rest/api/storageservices/data-lake-storage-gen2 +[godevdl]: https://go.dev/dl/ +[goget]: https://pkg.go.dev/cmd/go#hdr-Add_dependencies_to_current_module_and_install_them +[storage_account_docs]: https://docs.microsoft.com/azure/storage/common/storage-account-overview +[storage_account_create_ps]: https://docs.microsoft.com/azure/storage/common/storage-quickstart-create-account?tabs=azure-powershell +[storage_account_create_cli]: https://docs.microsoft.com/azure/storage/common/storage-quickstart-create-account?tabs=azure-cli +[storage_account_create_portal]: https://docs.microsoft.com/azure/storage/common/storage-quickstart-create-account?tabs=azure-portal +[azure_sub]: https://azure.microsoft.com/free/ +[azidentity]: https://pkg.go.dev/github.com/Azure/azure-sdk-for-go/sdk/azidentity +[storage_ad]: https://docs.microsoft.com/azure/storage/common/storage-auth-aad +[azcore_response_error]: https://pkg.go.dev/github.com/Azure/azure-sdk-for-go/sdk/azcore#ResponseError +[datalake_error]: https://github.com/Azure/azure-sdk-for-go/tree/main/sdk/storage/azdatalake/datalakeerror/error_codes.go +[filesystem]: https://github.com/Azure/azure-sdk-for-go/tree/main/sdk/storage/azdatalake/filesystem/client.go +[lease]: https://github.com/Azure/azure-sdk-for-go/tree/main/sdk/storage/azdatalake/lease +[file]: https://github.com/Azure/azure-sdk-for-go/tree/main/sdk/storage/azdatalake/file/client.go +[directory]: https://github.com/Azure/azure-sdk-for-go/tree/main/sdk/storage/azdatalake/directory/client.go +[sas]: https://github.com/Azure/azure-sdk-for-go/tree/main/sdk/storage/azdatalake/sas +[service]: https://github.com/Azure/azure-sdk-for-go/tree/main/sdk/storage/azdatalake/service/client.go +[storage_contrib]: https://github.com/Azure/azure-sdk-for-go/blob/main/CONTRIBUTING.md +[cla]: https://cla.microsoft.com +[coc]: https://opensource.microsoft.com/codeofconduct/ +[coc_faq]: https://opensource.microsoft.com/codeofconduct/faq/ +[coc_contact]: mailto:opencode@microsoft.com diff --git a/sdk/storage/azdatalake/assets.json b/sdk/storage/azdatalake/assets.json index 3ec59c60cd99..6794bca42de8 100644 --- a/sdk/storage/azdatalake/assets.json +++ b/sdk/storage/azdatalake/assets.json @@ -2,5 +2,5 @@ "AssetsRepo": "Azure/azure-sdk-assets", "AssetsRepoPrefixPath": "go", "TagPrefix": "go/storage/azdatalake", - "Tag": "go/storage/azdatalake_ec80a91bf3" + "Tag": "go/storage/azdatalake_c3c16cffab" } \ No newline at end of file diff --git a/sdk/storage/azdatalake/datalakeerror/error_codes.go b/sdk/storage/azdatalake/datalakeerror/error_codes.go index b482c9b3f929..0c703a242606 100644 --- a/sdk/storage/azdatalake/datalakeerror/error_codes.go +++ b/sdk/storage/azdatalake/datalakeerror/error_codes.go @@ -30,6 +30,7 @@ func HasCode(err error, codes ...StorageErrorCode) bool { return false } +// StorageErrorCode - Error codes returned by the service type StorageErrorCode string // dfs errors @@ -79,10 +80,10 @@ const ( CannotVerifyCopySource StorageErrorCode = "CannotVerifyCopySource" ConditionHeadersNotSupported StorageErrorCode = "ConditionHeadersNotSupported" ConditionNotMet StorageErrorCode = "ConditionNotMet" - FilesystemAlreadyExists StorageErrorCode = "FilesystemAlreadyExists" - FilesystemBeingDeleted StorageErrorCode = "FilesystemBeingDeleted" - FilesystemDisabled StorageErrorCode = "FilesystemDisabled" - FilesystemNotFound StorageErrorCode = "FilesystemNotFound" + FileSystemAlreadyExists StorageErrorCode = "FileSystemAlreadyExists" + FileSystemBeingDeleted StorageErrorCode = "FileSystemBeingDeleted" + FileSystemDisabled StorageErrorCode = "FileSystemDisabled" + FileSystemNotFound StorageErrorCode = "FileSystemNotFound" ContentLengthLargerThanTierLimit StorageErrorCode = "ContentLengthLargerThanTierLimit" CopyAcrossAccountsNotSupported StorageErrorCode = "CopyAcrossAccountsNotSupported" CopyIDMismatch StorageErrorCode = "CopyIdMismatch" @@ -119,7 +120,7 @@ const ( LeaseAlreadyBroken StorageErrorCode = "LeaseAlreadyBroken" LeaseAlreadyPresent StorageErrorCode = "LeaseAlreadyPresent" LeaseIDMismatchWithPathOperation StorageErrorCode = "LeaseIdMismatchWithPathOperation" - LeaseIDMismatchWithFilesystemOperation StorageErrorCode = "LeaseIdMismatchWithFilesystemOperation" + LeaseIDMismatchWithFileSystemOperation StorageErrorCode = "LeaseIdMismatchWithFileSystemOperation" LeaseIDMismatchWithLeaseOperation StorageErrorCode = "LeaseIdMismatchWithLeaseOperation" LeaseIDMissing StorageErrorCode = "LeaseIdMissing" LeaseIsBreakingAndCannotBeAcquired StorageErrorCode = "LeaseIsBreakingAndCannotBeAcquired" @@ -127,7 +128,7 @@ const ( LeaseIsBrokenAndCannotBeRenewed StorageErrorCode = "LeaseIsBrokenAndCannotBeRenewed" LeaseLost StorageErrorCode = "LeaseLost" LeaseNotPresentWithPathOperation StorageErrorCode = "LeaseNotPresentWithPathOperation" - LeaseNotPresentWithFilesystemOperation StorageErrorCode = "LeaseNotPresentWithFilesystemOperation" + LeaseNotPresentWithFileSystemOperation StorageErrorCode = "LeaseNotPresentWithFileSystemOperation" LeaseNotPresentWithLeaseOperation StorageErrorCode = "LeaseNotPresentWithLeaseOperation" MD5Mismatch StorageErrorCode = "Md5Mismatch" CRC64Mismatch StorageErrorCode = "Crc64Mismatch" diff --git a/sdk/storage/azdatalake/directory/client.go b/sdk/storage/azdatalake/directory/client.go index dc7a6d0e63b6..45c9c21b6696 100644 --- a/sdk/storage/azdatalake/directory/client.go +++ b/sdk/storage/azdatalake/directory/client.go @@ -118,7 +118,7 @@ func NewClientWithSharedKeyCredential(directoryURL string, cred *SharedKeyCreden blobClientOpts := blockblob.ClientOptions{ ClientOptions: options.ClientOptions, } - blobSharedKey, err := cred.ConvertToBlobSharedKey() + blobSharedKey, err := exported.ConvertToBlobSharedKey(cred) if err != nil { return nil, err } @@ -188,8 +188,8 @@ func (d *Client) BlobURL() string { return d.generatedDirClientWithBlob().Endpoint() } -// NewFileClient creates a new file.Client object by concatenating fileName to the end of this Client's URL. -// The new file.Client uses the same request policy pipeline as the Client. +// NewFileClient creates a new directory.Client object by concatenating directoryName to the end of this Client's URL. +// The new directory.Client uses the same request policy pipeline as the Client. func (d *Client) NewFileClient(fileName string) (*file.Client, error) { fileName = url.PathEscape(fileName) fileURL := runtime.JoinPaths(d.DFSURL(), fileName) @@ -199,7 +199,7 @@ func (d *Client) NewFileClient(fileName string) (*file.Client, error) { if d.identityCredential() != nil { newBlobClient, err = blockblob.NewClient(newBlobURL, *d.identityCredential(), nil) } else if d.sharedKey() != nil { - blobSharedKey, _ := d.sharedKey().ConvertToBlobSharedKey() + blobSharedKey, _ := exported.ConvertToBlobSharedKey(d.sharedKey()) newBlobClient, err = blockblob.NewClientWithSharedKeyCredential(newBlobURL, blobSharedKey, nil) } else { newBlobClient, err = blockblob.NewClientWithNoCredential(newBlobURL, nil) @@ -210,7 +210,7 @@ func (d *Client) NewFileClient(fileName string) (*file.Client, error) { return (*file.Client)(base.NewPathClient(fileURL, newBlobURL, newBlobClient, d.generatedDirClientWithDFS().InternalClient().WithClientName(shared.FileClient), d.sharedKey(), d.identityCredential(), d.getClientOptions())), nil } -// Create creates a new directory (dfs1). +// Create creates a new directory. func (d *Client) Create(ctx context.Context, options *CreateOptions) (CreateResponse, error) { lac, mac, httpHeaders, createOpts, cpkOpts := options.format() resp, err := d.generatedDirClientWithDFS().Create(ctx, createOpts, httpHeaders, lac, mac, nil, cpkOpts) @@ -218,7 +218,7 @@ func (d *Client) Create(ctx context.Context, options *CreateOptions) (CreateResp return resp, err } -// Delete deletes directory and any path under it (dfs1). +// Delete deletes directory and any path under it. func (d *Client) Delete(ctx context.Context, options *DeleteOptions) (DeleteResponse, error) { lac, mac, deleteOpts := path.FormatDeleteOptions(options, true) resp, err := d.generatedDirClientWithDFS().Delete(ctx, deleteOpts, lac, mac) @@ -226,7 +226,7 @@ func (d *Client) Delete(ctx context.Context, options *DeleteOptions) (DeleteResp return resp, err } -// GetProperties gets the properties of a directory (blob3) +// GetProperties gets the properties of a directory. func (d *Client) GetProperties(ctx context.Context, options *GetPropertiesOptions) (GetPropertiesResponse, error) { opts := path.FormatGetPropertiesOptions(options) var respFromCtx *http.Response @@ -245,36 +245,37 @@ func (d *Client) renamePathInURL(newName string) (string, string, string) { // Split the string based on the last occurrence of the separator firstPart := endpoint[:lastIndex] // From the beginning of the string to the last occurrence of the separator newBlobURL, newPathURL := shared.GetURLs(runtime.JoinPaths(firstPart, newName)) - parsedNewURL, _ := url.Parse(d.DFSURL()) - return parsedNewURL.Path, newPathURL, newBlobURL + oldURL, _ := url.Parse(d.DFSURL()) + return oldURL.Path, newPathURL, newBlobURL } -// Rename renames a directory (dfs1) +// Rename renames a directory. The original directory will no longer exist and the client will be stale. func (d *Client) Rename(ctx context.Context, newName string, options *RenameOptions) (RenameResponse, error) { - newPathWithoutURL, newPathURL, newBlobURL := d.renamePathInURL(newName) - lac, mac, smac, createOpts := path.FormatRenameOptions(options, newPathWithoutURL) + oldURL, newPathURL, newBlobURL := d.renamePathInURL(newName) + lac, mac, smac, createOpts := path.FormatRenameOptions(options, oldURL) var newBlobClient *blockblob.Client var err error if d.identityCredential() != nil { newBlobClient, err = blockblob.NewClient(newBlobURL, *d.identityCredential(), nil) } else if d.sharedKey() != nil { - blobSharedKey, _ := d.sharedKey().ConvertToBlobSharedKey() + blobSharedKey, _ := exported.ConvertToBlobSharedKey(d.sharedKey()) newBlobClient, err = blockblob.NewClientWithSharedKeyCredential(newBlobURL, blobSharedKey, nil) } else { newBlobClient, err = blockblob.NewClientWithNoCredential(newBlobURL, nil) } if err != nil { - return RenameResponse{}, err + return RenameResponse{}, exported.ConvertToDFSError(err) } newDirClient := (*Client)(base.NewPathClient(newPathURL, newBlobURL, newBlobClient, d.generatedDirClientWithDFS().InternalClient().WithClientName(shared.DirectoryClient), d.sharedKey(), d.identityCredential(), d.getClientOptions())) resp, err := newDirClient.generatedDirClientWithDFS().Create(ctx, createOpts, nil, lac, mac, smac, nil) - return RenameResponse{ - Response: resp, - NewDirectoryClient: newDirClient, - }, exported.ConvertToDFSError(err) + //return RenameResponse{ + // Response: resp, + // NewDirectoryClient: newDirClient, + //}, exported.ConvertToDFSError(err) + return path.FormatRenameResponse(&resp), exported.ConvertToDFSError(err) } -// SetAccessControl sets the owner, owning group, and permissions for a directory (dfs1). +// SetAccessControl sets the owner, owning group, and permissions for a directory. func (d *Client) SetAccessControl(ctx context.Context, options *SetAccessControlOptions) (SetAccessControlResponse, error) { opts, lac, mac, err := path.FormatSetAccessControlOptions(options) if err != nil { @@ -319,7 +320,7 @@ func (d *Client) setAccessControlPager(mode generated.PathSetAccessControlRecurs } -func (d *Client) setAccessControlRecursiveHelper(mode generated.PathSetAccessControlRecursiveMode, listOptions *generated.PathClientSetAccessControlRecursiveOptions, options *SetAccessControlRecursiveOptions) (SetAccessControlRecursiveResponse, error) { +func (d *Client) setAccessControlRecursiveHelper(ctx context.Context, mode generated.PathSetAccessControlRecursiveMode, listOptions *generated.PathClientSetAccessControlRecursiveOptions, options *SetAccessControlRecursiveOptions) (SetAccessControlRecursiveResponse, error) { pager := d.setAccessControlPager(mode, listOptions) counter := *options.MaxBatches continueOnFailure := listOptions.ForceFlag @@ -333,7 +334,7 @@ func (d *Client) setAccessControlRecursiveHelper(mode generated.PathSetAccessCon FailedEntries: []*ACLFailedEntry{}, } for pager.More() && counter != 0 { - resp, err := pager.NextPage(context.Background()) + resp, err := pager.NextPage(ctx) if err != nil { return finalResponse, exported.ConvertToDFSError(err) } @@ -349,34 +350,34 @@ func (d *Client) setAccessControlRecursiveHelper(mode generated.PathSetAccessCon return finalResponse, nil } -// SetAccessControlRecursive sets the owner, owning group, and permissions for a directory (dfs1). -func (d *Client) SetAccessControlRecursive(ACL string, options *SetAccessControlRecursiveOptions) (SetAccessControlRecursiveResponse, error) { +// SetAccessControlRecursive sets the owner, owning group, and permissions for a directory. +func (d *Client) SetAccessControlRecursive(ctx context.Context, ACL string, options *SetAccessControlRecursiveOptions) (SetAccessControlRecursiveResponse, error) { if options == nil { options = &SetAccessControlRecursiveOptions{} } mode, listOptions := options.format(ACL, "set") - return d.setAccessControlRecursiveHelper(mode, listOptions, options) + return d.setAccessControlRecursiveHelper(ctx, mode, listOptions, options) } -// UpdateAccessControlRecursive updates the owner, owning group, and permissions for a directory (dfs1). -func (d *Client) UpdateAccessControlRecursive(ACL string, options *UpdateAccessControlRecursiveOptions) (SetAccessControlRecursiveResponse, error) { +// UpdateAccessControlRecursive updates the owner, owning group, and permissions for a directory. +func (d *Client) UpdateAccessControlRecursive(ctx context.Context, ACL string, options *UpdateAccessControlRecursiveOptions) (SetAccessControlRecursiveResponse, error) { if options == nil { options = &UpdateAccessControlRecursiveOptions{} } mode, listOptions := options.format(ACL, "modify") - return d.setAccessControlRecursiveHelper(mode, listOptions, options) + return d.setAccessControlRecursiveHelper(ctx, mode, listOptions, options) } -// RemoveAccessControlRecursive removes the owner, owning group, and permissions for a directory (dfs1). -func (d *Client) RemoveAccessControlRecursive(ACL string, options *RemoveAccessControlRecursiveOptions) (SetAccessControlRecursiveResponse, error) { +// RemoveAccessControlRecursive removes the owner, owning group, and permissions for a directory. +func (d *Client) RemoveAccessControlRecursive(ctx context.Context, ACL string, options *RemoveAccessControlRecursiveOptions) (SetAccessControlRecursiveResponse, error) { if options == nil { options = &RemoveAccessControlRecursiveOptions{} } mode, listOptions := options.format(ACL, "remove") - return d.setAccessControlRecursiveHelper(mode, listOptions, options) + return d.setAccessControlRecursiveHelper(ctx, mode, listOptions, options) } -// GetAccessControl gets the owner, owning group, and permissions for a directory (dfs1). +// GetAccessControl gets the owner, owning group, and permissions for a directory. func (d *Client) GetAccessControl(ctx context.Context, options *GetAccessControlOptions) (GetAccessControlResponse, error) { opts, lac, mac := path.FormatGetAccessControlOptions(options) resp, err := d.generatedDirClientWithDFS().GetProperties(ctx, opts, lac, mac) @@ -384,15 +385,15 @@ func (d *Client) GetAccessControl(ctx context.Context, options *GetAccessControl return resp, err } -// SetMetadata sets the metadata for a directory (blob3). -func (d *Client) SetMetadata(ctx context.Context, options *SetMetadataOptions) (SetMetadataResponse, error) { - opts, metadata := path.FormatSetMetadataOptions(options) +// SetMetadata sets the metadata for a directory. +func (d *Client) SetMetadata(ctx context.Context, metadata map[string]*string, options *SetMetadataOptions) (SetMetadataResponse, error) { + opts := path.FormatSetMetadataOptions(options) resp, err := d.blobClient().SetMetadata(ctx, metadata, opts) err = exported.ConvertToDFSError(err) return resp, err } -// SetHTTPHeaders sets the HTTP headers for a directory (blob3). +// SetHTTPHeaders sets the HTTP headers for a directory. func (d *Client) SetHTTPHeaders(ctx context.Context, httpHeaders HTTPHeaders, options *SetHTTPHeadersOptions) (SetHTTPHeadersResponse, error) { opts, blobHTTPHeaders := path.FormatSetHTTPHeadersOptions(options, httpHeaders) resp, err := d.blobClient().SetHTTPHeaders(ctx, blobHTTPHeaders, opts) @@ -402,7 +403,7 @@ func (d *Client) SetHTTPHeaders(ctx context.Context, httpHeaders HTTPHeaders, op return newResp, err } -// GetSASURL is a convenience method for generating a SAS token for the currently pointed at blob. +// GetSASURL is a convenience method for generating a SAS token for the currently pointed at directory. // It can only be used if the credential supplied during creation was a SharedKeyCredential. func (d *Client) GetSASURL(permissions sas.DirectoryPermissions, expiry time.Time, o *GetSASURLOptions) (string, error) { if d.sharedKey() == nil { @@ -419,7 +420,7 @@ func (d *Client) GetSASURL(permissions sas.DirectoryPermissions, expiry time.Tim qps, err := sas.DatalakeSignatureValues{ DirectoryPath: urlParts.PathName, - FilesystemName: urlParts.FilesystemName, + FileSystemName: urlParts.FileSystemName, Version: sas.Version, Permissions: permissions.String(), StartTime: st, @@ -435,3 +436,6 @@ func (d *Client) GetSASURL(permissions sas.DirectoryPermissions, expiry time.Tim return endpoint, nil } + +// TODO: Undelete() +// TODO: ListPaths() diff --git a/sdk/storage/azdatalake/directory/client_test.go b/sdk/storage/azdatalake/directory/client_test.go index f9b0ea71f857..9ff90a56b9d4 100644 --- a/sdk/storage/azdatalake/directory/client_test.go +++ b/sdk/storage/azdatalake/directory/client_test.go @@ -66,10 +66,10 @@ func (s *UnrecordedTestSuite) TestCreateDirAndDeleteWithConnectionString() { connectionString, _ := testcommon.GetGenericConnectionString(testcommon.TestAccountDatalake) - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) @@ -90,10 +90,10 @@ func (s *RecordedTestSuite) TestBlobURLAndDFSURL() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) @@ -116,10 +116,10 @@ func (s *RecordedTestSuite) TestCreateDirAndDelete() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) @@ -139,10 +139,10 @@ func (s *RecordedTestSuite) TestGetAndCreateFileClient() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) @@ -169,10 +169,10 @@ func (s *RecordedTestSuite) TestCreateDirWithNilAccessConditions() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) @@ -200,10 +200,10 @@ func (s *RecordedTestSuite) TestCreateDirIfModifiedSinceTrue() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) @@ -237,10 +237,10 @@ func (s *RecordedTestSuite) TestCreateDirIfModifiedSinceFalse() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) @@ -274,10 +274,10 @@ func (s *RecordedTestSuite) TestCreateDirIfUnmodifiedSinceTrue() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) @@ -311,10 +311,10 @@ func (s *RecordedTestSuite) TestCreateDirIfUnmodifiedSinceFalse() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) @@ -349,10 +349,10 @@ func (s *RecordedTestSuite) TestCreateDirIfETagMatch() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) @@ -386,10 +386,10 @@ func (s *RecordedTestSuite) TestCreateDirIfETagMatchFalse() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) @@ -420,68 +420,14 @@ func (s *RecordedTestSuite) TestCreateDirIfETagMatchFalse() { testcommon.ValidateErrorCode(_require, err, datalakeerror.ConditionNotMet) } -func (s *RecordedTestSuite) TestCreateDirWithMetadataNotNil() { - _require := require.New(s.T()) - testName := s.T().Name() - - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) - _require.NoError(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) - - createDirOpts := &directory.CreateOptions{ - Metadata: testcommon.BasicMetadata, - } - - _, err = fsClient.Create(context.Background(), nil) - _require.Nil(err) - - dirName := testcommon.GenerateDirName(testName) - dirClient, err := testcommon.GetDirClient(filesystemName, dirName, s.T(), testcommon.TestAccountDatalake, nil) - _require.NoError(err) - - defer testcommon.DeleteDir(context.Background(), _require, dirClient) - - resp, err := dirClient.Create(context.Background(), createDirOpts) - _require.Nil(err) - _require.NotNil(resp) -} - -func (s *RecordedTestSuite) TestCreateDirWithEmptyMetadata() { - _require := require.New(s.T()) - testName := s.T().Name() - - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) - _require.NoError(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) - - createDirOpts := &directory.CreateOptions{ - Metadata: nil, - } - - _, err = fsClient.Create(context.Background(), nil) - _require.Nil(err) - - dirName := testcommon.GenerateDirName(testName) - dirClient, err := testcommon.GetDirClient(filesystemName, dirName, s.T(), testcommon.TestAccountDatalake, nil) - _require.NoError(err) - - defer testcommon.DeleteDir(context.Background(), _require, dirClient) - - resp, err := dirClient.Create(context.Background(), createDirOpts) - _require.Nil(err) - _require.NotNil(resp) -} - func (s *RecordedTestSuite) TestCreateDirWithNilHTTPHeaders() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) createDirOpts := &directory.CreateOptions{ HTTPHeaders: nil, @@ -505,10 +451,10 @@ func (s *RecordedTestSuite) TestCreateDirWithHTTPHeaders() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) createDirOpts := &directory.CreateOptions{ HTTPHeaders: &testcommon.BasicHeaders, @@ -532,10 +478,10 @@ func (s *RecordedTestSuite) TestCreateDirWithLease() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) createFileOpts := &directory.CreateOptions{ ProposedLeaseID: proposedLeaseIDs[0], @@ -567,12 +513,12 @@ func (s *RecordedTestSuite) TestCreateDirWithPermissions() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) perms := "0777" umask := "0000" - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) createFileOpts := &directory.CreateOptions{ Permissions: &perms, @@ -600,14 +546,14 @@ func (s *RecordedTestSuite) TestCreateDirWithOwnerGroupACLUmask() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) owner := "4cf4e284-f6a8-4540-b53e-c3469af032dc" group := owner acl := "user::rwx,group::r-x,other::rwx" umask := "0000" - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) createFileOpts := &directory.CreateOptions{ Owner: &owner, @@ -633,10 +579,10 @@ func (s *RecordedTestSuite) TestDeleteDirWithNilAccessConditions() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) @@ -661,10 +607,10 @@ func (s *RecordedTestSuite) TestDeleteDirIfModifiedSinceTrue() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) @@ -695,10 +641,10 @@ func (s *RecordedTestSuite) TestDeleteDirIfModifiedSinceFalse() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) @@ -730,10 +676,10 @@ func (s *RecordedTestSuite) TestDeleteDirIfUnmodifiedSinceTrue() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) @@ -765,10 +711,10 @@ func (s *RecordedTestSuite) TestDeleteDirIfUnmodifiedSinceFalse() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) @@ -801,10 +747,10 @@ func (s *RecordedTestSuite) TestDeleteDirIfETagMatch() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) @@ -836,10 +782,10 @@ func (s *RecordedTestSuite) TestDeleteDirIfETagMatchFalse() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) @@ -872,10 +818,10 @@ func (s *RecordedTestSuite) TestDirSetAccessControlNil() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) @@ -899,13 +845,13 @@ func (s *RecordedTestSuite) TestDirSetAccessControl() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) owner := "4cf4e284-f6a8-4540-b53e-c3469af032dc" group := owner acl := "user::rwx,group::r-x,other::rwx" - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) opts := &directory.SetAccessControlOptions{ Owner: &owner, @@ -932,13 +878,13 @@ func (s *RecordedTestSuite) TestDirSetAccessControlWithNilAccessConditions() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) owner := "4cf4e284-f6a8-4540-b53e-c3469af032dc" group := owner acl := "user::rwx,group::r-x,other::rwx" - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) opts := &directory.SetAccessControlOptions{ Owner: &owner, @@ -966,13 +912,13 @@ func (s *RecordedTestSuite) TestDirSetAccessControlIfModifiedSinceTrue() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) owner := "4cf4e284-f6a8-4540-b53e-c3469af032dc" group := owner acl := "user::rwx,group::r-x,other::rwx" - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) @@ -1005,13 +951,13 @@ func (s *RecordedTestSuite) TestDirSetAccessControlIfModifiedSinceFalse() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) owner := "4cf4e284-f6a8-4540-b53e-c3469af032dc" group := owner acl := "user::rwx,group::r-x,other::rwx" - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) @@ -1045,13 +991,13 @@ func (s *RecordedTestSuite) TestDirSetAccessControlIfUnmodifiedSinceTrue() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) owner := "4cf4e284-f6a8-4540-b53e-c3469af032dc" group := owner acl := "user::rwx,group::r-x,other::rwx" - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) @@ -1083,13 +1029,13 @@ func (s *RecordedTestSuite) TestDirSetAccessControlIfUnmodifiedSinceFalse() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) owner := "4cf4e284-f6a8-4540-b53e-c3469af032dc" group := owner acl := "user::rwx,group::r-x,other::rwx" - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) @@ -1125,13 +1071,13 @@ func (s *RecordedTestSuite) TestDirSetAccessControlIfETagMatch() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) owner := "4cf4e284-f6a8-4540-b53e-c3469af032dc" group := owner acl := "user::rwx,group::r-x,other::rwx" - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) @@ -1164,13 +1110,13 @@ func (s *RecordedTestSuite) TestDirSetAccessControlIfETagMatchFalse() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) owner := "4cf4e284-f6a8-4540-b53e-c3469af032dc" group := owner acl := "user::rwx,group::r-x,other::rwx" - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) @@ -1203,11 +1149,11 @@ func (s *RecordedTestSuite) TestDirGetAccessControl() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) acl := "user::rwx,group::r-x,other::rwx" - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) createOpts := &directory.CreateOptions{ ACL: &acl, @@ -1232,11 +1178,11 @@ func (s *UnrecordedTestSuite) TestDirGetAccessControlWithSAS() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) acl := "user::rwx,group::r-x,other::rwx" - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) createOpts := &directory.CreateOptions{ ACL: &acl, @@ -1276,10 +1222,10 @@ func (s *UnrecordedTestSuite) TestDeleteWithSAS() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) @@ -1315,11 +1261,11 @@ func (s *RecordedTestSuite) TestDirGetAccessControlWithNilAccessConditions() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) acl := "user::rwx,group::r-x,other::rwx" - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) createOpts := &directory.CreateOptions{ ACL: &acl, @@ -1348,11 +1294,11 @@ func (s *RecordedTestSuite) TestDirGetAccessControlIfModifiedSinceTrue() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) acl := "user::rwx,group::r-x,other::rwx" - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) createOpts := &directory.CreateOptions{ ACL: &acl, @@ -1386,11 +1332,11 @@ func (s *RecordedTestSuite) TestDirGetAccessControlIfModifiedSinceFalse() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) acl := "user::rwx,group::r-x,other::rwx" - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) createOpts := &directory.CreateOptions{ ACL: &acl, @@ -1424,11 +1370,11 @@ func (s *RecordedTestSuite) TestDirGetAccessControlIfUnmodifiedSinceTrue() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) acl := "user::rwx,group::r-x,other::rwx" - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) createOpts := &directory.CreateOptions{ ACL: &acl, @@ -1461,11 +1407,11 @@ func (s *RecordedTestSuite) TestDirGetAccessControlIfUnmodifiedSinceFalse() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) acl := "user::rwx,group::r-x,other::rwx" - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) createOpts := &directory.CreateOptions{ ACL: &acl, @@ -1501,11 +1447,11 @@ func (s *RecordedTestSuite) TestDirGetAccessControlIfETagMatch() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) acl := "user::rwx,group::r-x,other::rwx" - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) createOpts := &directory.CreateOptions{ ACL: &acl, @@ -1539,11 +1485,11 @@ func (s *RecordedTestSuite) TestDirGetAccessControlIfETagMatchFalse() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) acl := "user::rwx,group::r-x,other::rwx" - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) createOpts := &directory.CreateOptions{ ACL: &acl, @@ -1578,11 +1524,11 @@ func (s *RecordedTestSuite) TestDirSetAccessControlRecursive() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) acl := "user::rwx,group::r-x,other::rwx" - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) @@ -1595,7 +1541,7 @@ func (s *RecordedTestSuite) TestDirSetAccessControlRecursive() { _require.Nil(err) _require.NotNil(resp) - resp1, err := dirClient.SetAccessControlRecursive(acl, nil) + resp1, err := dirClient.SetAccessControlRecursive(context.Background(), acl, nil) _require.Nil(err) _require.Equal(resp1.DirectoriesSuccessful, to.Ptr(int32(1))) @@ -1611,11 +1557,11 @@ func (s *RecordedTestSuite) TestDirSetAccessControlRecursiveWithBadContinuation( _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) acl := "user::rwx,group::r-x,other::rwx" - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) @@ -1629,7 +1575,7 @@ func (s *RecordedTestSuite) TestDirSetAccessControlRecursiveWithBadContinuation( _require.NotNil(resp) opts := &directory.SetAccessControlRecursiveOptions{Marker: to.Ptr("garbage")} - resp1, err := dirClient.SetAccessControlRecursive(acl, opts) + resp1, err := dirClient.SetAccessControlRecursive(context.Background(), acl, opts) _require.Nil(err) _require.Equal(resp1.DirectoriesSuccessful, to.Ptr(int32(0))) @@ -1641,11 +1587,11 @@ func (s *RecordedTestSuite) TestDirSetAccessControlRecursiveWithEmptyOpts() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) acl := "user::rwx,group::r-x,other::rwx" - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) @@ -1659,7 +1605,7 @@ func (s *RecordedTestSuite) TestDirSetAccessControlRecursiveWithEmptyOpts() { _require.NotNil(resp) opts := &directory.SetAccessControlRecursiveOptions{} - resp1, err := dirClient.SetAccessControlRecursive(acl, opts) + resp1, err := dirClient.SetAccessControlRecursive(context.Background(), acl, opts) _require.Nil(err) _require.Equal(resp1.DirectoriesSuccessful, to.Ptr(int32(1))) @@ -1675,11 +1621,11 @@ func (s *RecordedTestSuite) TestDirSetAccessControlRecursiveWithMaxResults() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) acl := "user::rwx,group::r-x,other::rwx" - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) @@ -1711,7 +1657,7 @@ func (s *RecordedTestSuite) TestDirSetAccessControlRecursiveWithMaxResults() { _require.Nil(err) opts := &directory.SetAccessControlRecursiveOptions{BatchSize: to.Ptr(int32(2)), MaxBatches: to.Ptr(int32(1)), ContinueOnFailure: to.Ptr(true), Marker: nil} - resp2, err := dirClient.SetAccessControlRecursive(acl, opts) + resp2, err := dirClient.SetAccessControlRecursive(context.Background(), acl, opts) _require.Nil(err) // we expect only one file to have been updated not both since our batch size is 2 and max batches is 1 @@ -1728,11 +1674,11 @@ func (s *RecordedTestSuite) TestDirSetAccessControlRecursiveWithMaxResults2() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) acl := "user::rwx,group::r-x,other::rwx" - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) @@ -1764,7 +1710,7 @@ func (s *RecordedTestSuite) TestDirSetAccessControlRecursiveWithMaxResults2() { _require.Nil(err) opts := &directory.SetAccessControlRecursiveOptions{ContinueOnFailure: to.Ptr(true), Marker: nil} - resp2, err := dirClient.SetAccessControlRecursive(acl, opts) + resp2, err := dirClient.SetAccessControlRecursive(context.Background(), acl, opts) _require.Nil(err) // we expect only one file to have been updated not both since our batch size is 2 and max batches is 1 @@ -1781,11 +1727,11 @@ func (s *RecordedTestSuite) TestDirSetAccessControlRecursiveWithMaxResults3() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) acl := "user::rwx,group::r-x,other::rwx" - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) @@ -1817,7 +1763,7 @@ func (s *RecordedTestSuite) TestDirSetAccessControlRecursiveWithMaxResults3() { _require.Nil(err) opts := &directory.SetAccessControlRecursiveOptions{BatchSize: to.Ptr(int32(1)), ContinueOnFailure: to.Ptr(true), Marker: nil} - resp2, err := dirClient.SetAccessControlRecursive(acl, opts) + resp2, err := dirClient.SetAccessControlRecursive(context.Background(), acl, opts) _require.Nil(err) // we expect only one file to have been updated not both since our batch size is 2 and max batches is 1 @@ -1834,12 +1780,12 @@ func (s *RecordedTestSuite) TestDirUpdateAccessControlRecursive() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) acl := "user::rwx,group::r-x,other::rwx" acl1 := "user::rwx,group::r--,other::r--" - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) createOpts := &directory.CreateOptions{ ACL: &acl, @@ -1855,7 +1801,7 @@ func (s *RecordedTestSuite) TestDirUpdateAccessControlRecursive() { _require.Nil(err) _require.NotNil(resp) - resp1, err := dirClient.UpdateAccessControlRecursive(acl1, nil) + resp1, err := dirClient.UpdateAccessControlRecursive(context.Background(), acl1, nil) _require.Nil(err) _require.Equal(resp1.DirectoriesSuccessful, to.Ptr(int32(1))) @@ -1869,13 +1815,13 @@ func (s *RecordedTestSuite) TestDirRemoveAccessControlRecursive() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) acl := "mask," + "default:user,default:group," + "user:ec3595d6-2c17-4696-8caa-7e139758d24a,group:ec3595d6-2c17-4696-8caa-7e139758d24a," + "default:user:ec3595d6-2c17-4696-8caa-7e139758d24a,default:group:ec3595d6-2c17-4696-8caa-7e139758d24a" - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) @@ -1888,74 +1834,19 @@ func (s *RecordedTestSuite) TestDirRemoveAccessControlRecursive() { _require.Nil(err) _require.NotNil(resp) - resp1, err := dirClient.RemoveAccessControlRecursive(acl, nil) + resp1, err := dirClient.RemoveAccessControlRecursive(context.Background(), acl, nil) _require.Nil(err) _require.Equal(resp1.DirectoriesSuccessful, to.Ptr(int32(1))) } -func (s *RecordedTestSuite) TestDirSetMetadataNil() { - _require := require.New(s.T()) - testName := s.T().Name() - - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) - _require.NoError(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) - - _, err = fsClient.Create(context.Background(), nil) - _require.Nil(err) - - dirName := testcommon.GenerateDirName(testName) - dirClient, err := testcommon.GetDirClient(filesystemName, dirName, s.T(), testcommon.TestAccountDatalake, nil) - _require.NoError(err) - - defer testcommon.DeleteDir(context.Background(), _require, dirClient) - - resp, err := dirClient.Create(context.Background(), nil) - _require.Nil(err) - _require.NotNil(resp) - - _, err = dirClient.SetMetadata(context.Background(), nil) - _require.Nil(err) -} - -func (s *RecordedTestSuite) TestDirSetMetadataWithEmptyOpts() { - _require := require.New(s.T()) - testName := s.T().Name() - - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) - _require.NoError(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) - - _, err = fsClient.Create(context.Background(), nil) - _require.Nil(err) - - dirName := testcommon.GenerateDirName(testName) - dirClient, err := testcommon.GetDirClient(filesystemName, dirName, s.T(), testcommon.TestAccountDatalake, nil) - _require.NoError(err) - - defer testcommon.DeleteDir(context.Background(), _require, dirClient) - - resp, err := dirClient.Create(context.Background(), nil) - _require.Nil(err) - _require.NotNil(resp) - - opts := &directory.SetMetadataOptions{ - Metadata: nil, - } - _, err = dirClient.SetMetadata(context.Background(), opts) - _require.Nil(err) -} - func (s *RecordedTestSuite) TestDirSetMetadataWithBasicMetadata() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) @@ -1970,10 +1861,7 @@ func (s *RecordedTestSuite) TestDirSetMetadataWithBasicMetadata() { _require.Nil(err) _require.NotNil(resp) - opts := &directory.SetMetadataOptions{ - Metadata: testcommon.BasicMetadata, - } - _, err = dirClient.SetMetadata(context.Background(), opts) + _, err = dirClient.SetMetadata(context.Background(), testcommon.BasicMetadata, nil) _require.Nil(err) } @@ -1981,10 +1869,10 @@ func (s *RecordedTestSuite) TestDirSetMetadataWithAccessConditions() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) @@ -2002,14 +1890,13 @@ func (s *RecordedTestSuite) TestDirSetMetadataWithAccessConditions() { currentTime := testcommon.GetRelativeTimeFromAnchor(resp.Date, -10) opts := &directory.SetMetadataOptions{ - Metadata: testcommon.BasicMetadata, AccessConditions: &directory.AccessConditions{ ModifiedAccessConditions: &directory.ModifiedAccessConditions{ IfModifiedSince: ¤tTime, }, }, } - _, err = dirClient.SetMetadata(context.Background(), opts) + _, err = dirClient.SetMetadata(context.Background(), testcommon.BasicMetadata, opts) _require.Nil(err) } @@ -2023,10 +1910,10 @@ func (s *RecordedTestSuite) TestDirSetHTTPHeaders() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) @@ -2048,10 +1935,10 @@ func (s *RecordedTestSuite) TestDirSetHTTPHeadersWithNilAccessConditions() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) @@ -2077,10 +1964,10 @@ func (s *RecordedTestSuite) TestDirSetHTTPHeadersIfModifiedSinceTrue() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) @@ -2111,10 +1998,10 @@ func (s *RecordedTestSuite) TestDirSetHTTPHeadersIfModifiedSinceFalse() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) @@ -2145,10 +2032,10 @@ func (s *RecordedTestSuite) TestDirSetHTTPHeadersIfUnmodifiedSinceTrue() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) @@ -2179,10 +2066,10 @@ func (s *RecordedTestSuite) TestDirSetHTTPHeadersIfUnmodifiedSinceFalse() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) @@ -2213,10 +2100,10 @@ func (s *RecordedTestSuite) TestDirSetHTTPHeadersIfETagMatch() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) @@ -2246,10 +2133,10 @@ func (s *RecordedTestSuite) TestDirSetHTTPHeadersIfETagMatchFalse() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) @@ -2280,10 +2167,10 @@ func (s *RecordedTestSuite) TestDirRenameNoOptions() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) @@ -2296,20 +2183,21 @@ func (s *RecordedTestSuite) TestDirRenameNoOptions() { _require.Nil(err) _require.NotNil(resp) - resp1, err := dirClient.Rename(context.Background(), "newName", nil) + //resp1, err := dirClient.Rename(context.Background(), "newName", renameFileOpts) + _, err = dirClient.Rename(context.Background(), "newName", nil) _require.Nil(err) - _require.NotNil(resp1) - _require.Contains(resp1.NewDirectoryClient.DFSURL(), "newName") + //_require.NotNil(resp1) + //_require.Contains(resp1.NewDirectoryClient.DFSURL(), "newName") } func (s *RecordedTestSuite) TestRenameDirWithNilAccessConditions() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) @@ -2326,20 +2214,21 @@ func (s *RecordedTestSuite) TestRenameDirWithNilAccessConditions() { AccessConditions: nil, } - resp1, err := dirClient.Rename(context.Background(), "newName", renameFileOpts) + //resp1, err := dirClient.Rename(context.Background(), "newName", renameFileOpts) + _, err = dirClient.Rename(context.Background(), "newName", renameFileOpts) _require.Nil(err) - _require.NotNil(resp1) - _require.Contains(resp1.NewDirectoryClient.DFSURL(), "newName") + //_require.NotNil(resp1) + //_require.Contains(resp1.NewDirectoryClient.DFSURL(), "newName") } func (s *RecordedTestSuite) TestRenameDirIfModifiedSinceTrue() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) @@ -2361,20 +2250,21 @@ func (s *RecordedTestSuite) TestRenameDirIfModifiedSinceTrue() { }, }, } - resp1, err := dirClient.Rename(context.Background(), "newName", renameFileOpts) + //resp1, err := dirClient.Rename(context.Background(), "newName", renameFileOpts) + _, err = dirClient.Rename(context.Background(), "newName", renameFileOpts) _require.Nil(err) - _require.NotNil(resp1) - _require.Contains(resp1.NewDirectoryClient.DFSURL(), "newName") + //_require.NotNil(resp1) + //_require.Contains(resp1.NewDirectoryClient.DFSURL(), "newName") } func (s *RecordedTestSuite) TestRenameDirIfModifiedSinceFalse() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) @@ -2397,7 +2287,9 @@ func (s *RecordedTestSuite) TestRenameDirIfModifiedSinceFalse() { }, } + //_, err = dirClient.Rename(context.Background(), "newName", renameFileOpts) _, err = dirClient.Rename(context.Background(), "newName", renameFileOpts) + _require.NotNil(err) testcommon.ValidateErrorCode(_require, err, datalakeerror.SourceConditionNotMet) } @@ -2406,10 +2298,10 @@ func (s *RecordedTestSuite) TestRenameDirIfUnmodifiedSinceTrue() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) @@ -2432,20 +2324,21 @@ func (s *RecordedTestSuite) TestRenameDirIfUnmodifiedSinceTrue() { }, } - resp1, err := dirClient.Rename(context.Background(), "newName", renameFileOpts) + //resp1, err := dirClient.Rename(context.Background(), "newName", renameFileOpts) + _, err = dirClient.Rename(context.Background(), "newName", renameFileOpts) _require.Nil(err) - _require.NotNil(resp1) - _require.Contains(resp1.NewDirectoryClient.DFSURL(), "newName") + //_require.NotNil(resp1) + //_require.Contains(resp1.NewDirectoryClient.DFSURL(), "newName") } func (s *RecordedTestSuite) TestRenameDirIfUnmodifiedSinceFalse() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) @@ -2468,7 +2361,9 @@ func (s *RecordedTestSuite) TestRenameDirIfUnmodifiedSinceFalse() { }, } + //_, err = dirClient.Rename(context.Background(), "newName", renameFileOpts) _, err = dirClient.Rename(context.Background(), "newName", renameFileOpts) + _require.NotNil(err) testcommon.ValidateErrorCode(_require, err, datalakeerror.SourceConditionNotMet) } @@ -2477,10 +2372,10 @@ func (s *RecordedTestSuite) TestRenameDirIfETagMatch() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) @@ -2503,20 +2398,21 @@ func (s *RecordedTestSuite) TestRenameDirIfETagMatch() { }, } - resp1, err := dirClient.Rename(context.Background(), "newName", renameFileOpts) + //resp1, err := dirClient.Rename(context.Background(), "newName", renameFileOpts) + _, err = dirClient.Rename(context.Background(), "newName", renameFileOpts) _require.Nil(err) - _require.NotNil(resp1) - _require.Contains(resp1.NewDirectoryClient.DFSURL(), "newName") + //_require.NotNil(resp1) + //_require.Contains(resp1.NewDirectoryClient.DFSURL(), "newName") } func (s *RecordedTestSuite) TestRenameDirIfETagMatchFalse() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) @@ -2539,7 +2435,9 @@ func (s *RecordedTestSuite) TestRenameDirIfETagMatchFalse() { }, } + //_, err = dirClient.Rename(context.Background(), "newName", renameFileOpts) _, err = dirClient.Rename(context.Background(), "newName", renameFileOpts) + _require.NotNil(err) testcommon.ValidateErrorCode(_require, err, datalakeerror.SourceConditionNotMet) } diff --git a/sdk/storage/azdatalake/directory/constants.go b/sdk/storage/azdatalake/directory/constants.go index 99aae6d16704..25bfba6e776d 100644 --- a/sdk/storage/azdatalake/directory/constants.go +++ b/sdk/storage/azdatalake/directory/constants.go @@ -10,6 +10,7 @@ import ( "github.com/Azure/azure-sdk-for-go/sdk/storage/azdatalake/internal/path" ) +// EncryptionAlgorithmType defines values for EncryptionAlgorithmType. type EncryptionAlgorithmType = path.EncryptionAlgorithmType const ( diff --git a/sdk/storage/azdatalake/directory/examples_test.go b/sdk/storage/azdatalake/directory/examples_test.go new file mode 100644 index 000000000000..627317f8a62f --- /dev/null +++ b/sdk/storage/azdatalake/directory/examples_test.go @@ -0,0 +1,166 @@ +//go:build go1.18 +// +build go1.18 + +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT License. See License.txt in the project root for license information. + +package directory_test + +import ( + "context" + "fmt" + "github.com/Azure/azure-sdk-for-go/sdk/azcore/to" + "github.com/Azure/azure-sdk-for-go/sdk/storage/azdatalake" + "github.com/Azure/azure-sdk-for-go/sdk/storage/azdatalake/directory" + "log" + "os" + "time" +) + +func handleError(err error) { + if err != nil { + log.Fatal(err.Error()) + } +} + +// make sure you create the filesystem before running this example +func Example_directory_CreateAndDelete() { + accountName, accountKey := os.Getenv("AZURE_STORAGE_ACCOUNT_NAME"), os.Getenv("AZURE_STORAGE_ACCOUNT_KEY") + + // Create a directory client + u := fmt.Sprintf("https://%s.dfs.core.windows.net/fs/dir1", accountName) + credential, err := azdatalake.NewSharedKeyCredential(accountName, accountKey) + handleError(err) + dirClient, err := directory.NewClientWithSharedKeyCredential(u, credential, nil) + handleError(err) + + _, err = dirClient.Create(context.Background(), nil) + handleError(err) + + _, err = dirClient.Delete(context.Background(), nil) + handleError(err) +} + +// This examples shows how to set a directory HTTP Headers, how to read, and how to update the directory's HTTP headers. +func Example_directory_HTTPHeaders() { + // make sure you create the filesystem and directory before running this example + accountName, accountKey := os.Getenv("AZURE_STORAGE_ACCOUNT_NAME"), os.Getenv("AZURE_STORAGE_ACCOUNT_KEY") + + // Create a dir client + u := fmt.Sprintf("https://%s.dfs.core.windows.net/fs/dir1", accountName) + credential, err := azdatalake.NewSharedKeyCredential(accountName, accountKey) + handleError(err) + dirClient, err := directory.NewClientWithSharedKeyCredential(u, credential, nil) + handleError(err) + + // Create a directory with HTTP headers + _, err = dirClient.SetHTTPHeaders(context.TODO(), directory.HTTPHeaders{ + ContentType: to.Ptr("text/html; charset=utf-8"), + ContentDisposition: to.Ptr("attachment"), + }, nil) + handleError(err) + + get, err := dirClient.GetProperties(context.TODO(), nil) + handleError(err) + + fmt.Println(get.ContentType) + fmt.Println(get.ContentDisposition) +} + +// make sure you create the filesystem and directory before running this example +func Example_dir_Client_SetMetadata() { + accountName, accountKey := os.Getenv("AZURE_STORAGE_ACCOUNT_NAME"), os.Getenv("AZURE_STORAGE_ACCOUNT_KEY") + // Create a dir client + u := fmt.Sprintf("https://%s.dfs.core.windows.net/fs/dir1", accountName) + credential, err := azdatalake.NewSharedKeyCredential(accountName, accountKey) + handleError(err) + + dirClient, err := directory.NewClientWithSharedKeyCredential(u, credential, nil) + handleError(err) + + _, err = dirClient.SetMetadata(context.TODO(), map[string]*string{"author": to.Ptr("Tamer")}, nil) + handleError(err) + + // Query the directory's properties and metadata + get, err := dirClient.GetProperties(context.TODO(), nil) + handleError(err) + + // Show the directory's metadata + if get.Metadata == nil { + log.Fatal("No metadata returned") + } + + for k, v := range get.Metadata { + fmt.Print(k + "=" + *v + "\n") + } +} + +// make sure you create the filesystem before running this example +func Example_directory_Rename() { + accountName, accountKey := os.Getenv("AZURE_STORAGE_ACCOUNT_NAME"), os.Getenv("AZURE_STORAGE_ACCOUNT_KEY") + + // Create a directory client + u := fmt.Sprintf("https://%s.dfs.core.windows.net/fs/dir1", accountName) + credential, err := azdatalake.NewSharedKeyCredential(accountName, accountKey) + handleError(err) + dirClient, err := directory.NewClientWithSharedKeyCredential(u, credential, nil) + handleError(err) + + _, err = dirClient.Create(context.Background(), nil) + handleError(err) + + _, err = dirClient.Rename(context.Background(), "renameDir", nil) + handleError(err) +} + +// for this example make sure to create paths within the dir so you can recursively set the ACL on them +func Example_directory_SetACLRecursive() { + accountName, accountKey := os.Getenv("AZURE_STORAGE_ACCOUNT_NAME"), os.Getenv("AZURE_STORAGE_ACCOUNT_KEY") + + // Create a directory client + acl := "user::rwx,group::r-x,other::rwx" + u := fmt.Sprintf("https://%s.dfs.core.windows.net/fs/dir1", accountName) + credential, err := azdatalake.NewSharedKeyCredential(accountName, accountKey) + handleError(err) + dirClient, err := directory.NewClientWithSharedKeyCredential(u, credential, nil) + handleError(err) + + _, err = dirClient.SetAccessControlRecursive(context.Background(), acl, &directory.SetAccessControlRecursiveOptions{ + BatchSize: to.Ptr(int32(2)), MaxBatches: to.Ptr(int32(1)), ContinueOnFailure: to.Ptr(true), Marker: nil}) + handleError(err) +} + +func getRelativeTimeFromAnchor(anchorTime *time.Time, amount time.Duration) time.Time { + return anchorTime.Add(amount * time.Second) +} + +// make sure you create the filesystem before running this example +func Example_directory_SetAccessControlIfUnmodifiedSinceTrue() { + accountName, accountKey := os.Getenv("AZURE_STORAGE_ACCOUNT_NAME"), os.Getenv("AZURE_STORAGE_ACCOUNT_KEY") + + // Create a directory client + owner := "4cf4e284-f6a8-4540-b53e-c3469af032dc" + group := owner + acl := "user::rwx,group::r-x,other::rwx" + u := fmt.Sprintf("https://%s.dfs.core.windows.net/fs/dir1", accountName) + credential, err := azdatalake.NewSharedKeyCredential(accountName, accountKey) + handleError(err) + dirClient, err := directory.NewClientWithSharedKeyCredential(u, credential, nil) + handleError(err) + resp, err := dirClient.Create(context.Background(), nil) + handleError(err) + + currentTime := getRelativeTimeFromAnchor(resp.Date, 10) + opts := &directory.SetAccessControlOptions{ + Owner: &owner, + Group: &group, + ACL: &acl, + AccessConditions: &directory.AccessConditions{ + ModifiedAccessConditions: &directory.ModifiedAccessConditions{ + IfUnmodifiedSince: ¤tTime, + }, + }} + + _, err = dirClient.SetAccessControl(context.Background(), opts) + handleError(err) +} diff --git a/sdk/storage/azdatalake/directory/models.go b/sdk/storage/azdatalake/directory/models.go index 9e3016c05322..954ee67c481d 100644 --- a/sdk/storage/azdatalake/directory/models.go +++ b/sdk/storage/azdatalake/directory/models.go @@ -13,12 +13,10 @@ import ( "github.com/Azure/azure-sdk-for-go/sdk/storage/azdatalake/internal/path" ) -// CreateOptions contains the optional parameters when calling the Create operation. dfs endpoint. +// CreateOptions contains the optional parameters when calling the Create operation. type CreateOptions struct { // AccessConditions contains parameters for accessing the file. AccessConditions *AccessConditions - // Metadata is a map of name-value pairs to associate with the file storage object. - Metadata map[string]*string // CPKInfo contains a group of parameters for client provided encryption key. CPKInfo *CPKInfo // HTTPHeaders contains the HTTP headers for path operations. @@ -80,7 +78,7 @@ func (o *CreateOptions) format() (*generated.LeaseAccessConditions, *generated.M type accessControlRecursiveOptions struct { // Max batch size is 2000 paths BatchSize *int32 - // Number of recursive calls to be made to set access control + // Number of recursive calls to be made to set access control. MaxBatches *int32 // ContinueOnFailure indicates whether to continue on failure when the operation encounters an error. ContinueOnFailure *bool @@ -119,43 +117,43 @@ type RemoveAccessControlRecursiveOptions = accessControlRecursiveOptions // ================================= path imports ================================== -// DeleteOptions contains the optional parameters when calling the Delete operation. dfs endpoint +// DeleteOptions contains the optional parameters when calling the Delete operation. type DeleteOptions = path.DeleteOptions // RenameOptions contains the optional parameters when calling the Rename operation. type RenameOptions = path.RenameOptions -// GetPropertiesOptions contains the optional parameters for the Client.GetProperties method +// GetPropertiesOptions contains the optional parameters for the GetProperties method. type GetPropertiesOptions = path.GetPropertiesOptions -// SetAccessControlOptions contains the optional parameters when calling the SetAccessControl operation. dfs endpoint +// SetAccessControlOptions contains the optional parameters when calling the SetAccessControl operation. type SetAccessControlOptions = path.SetAccessControlOptions // GetAccessControlOptions contains the optional parameters when calling the GetAccessControl operation. type GetAccessControlOptions = path.GetAccessControlOptions -// CPKInfo contains a group of parameters for the PathClient.Download method. +// CPKInfo contains a group of parameters for client provided encryption key. type CPKInfo = path.CPKInfo -// GetSASURLOptions contains the optional parameters for the Client.GetSASURL method. +// GetSASURLOptions contains the optional parameters for the GetSASURL method. type GetSASURLOptions = path.GetSASURLOptions -// SetHTTPHeadersOptions contains the optional parameters for the Client.SetHTTPHeaders method. +// SetHTTPHeadersOptions contains the optional parameters for the SetHTTPHeaders method. type SetHTTPHeadersOptions = path.SetHTTPHeadersOptions // HTTPHeaders contains the HTTP headers for path operations. type HTTPHeaders = path.HTTPHeaders -// SetMetadataOptions provides set of configurations for Set Metadata on path operation +// SetMetadataOptions provides set of configurations for SetMetadata. type SetMetadataOptions = path.SetMetadataOptions // SharedKeyCredential contains an account's name and its primary or secondary key. type SharedKeyCredential = path.SharedKeyCredential -// AccessConditions identifies blob-specific access conditions which you optionally set. +// AccessConditions identifies path-specific access conditions which you optionally set. type AccessConditions = path.AccessConditions -// SourceAccessConditions identifies blob-specific access conditions which you optionally set. +// SourceAccessConditions identifies source path-specific access conditions which you optionally set. type SourceAccessConditions = path.SourceAccessConditions // LeaseAccessConditions contains optional parameters to access leased entity. @@ -164,10 +162,10 @@ type LeaseAccessConditions = path.LeaseAccessConditions // ModifiedAccessConditions contains a group of parameters for specifying access conditions. type ModifiedAccessConditions = path.ModifiedAccessConditions -// SourceModifiedAccessConditions contains a group of parameters for specifying access conditions. +// SourceModifiedAccessConditions contains a group of parameters for specifying source access conditions. type SourceModifiedAccessConditions = path.SourceModifiedAccessConditions -// CPKScopeInfo contains a group of parameters for the PathClient.SetMetadata method. +// CPKScopeInfo contains a group of parameters for client provided encryption scope. type CPKScopeInfo path.CPKScopeInfo // ACLFailedEntry contains the failed ACL entry (response model). diff --git a/sdk/storage/azdatalake/directory/responses.go b/sdk/storage/azdatalake/directory/responses.go index ce5a222f298b..0a3cd654da49 100644 --- a/sdk/storage/azdatalake/directory/responses.go +++ b/sdk/storage/azdatalake/directory/responses.go @@ -10,11 +10,8 @@ import ( "github.com/Azure/azure-sdk-for-go/sdk/storage/azdatalake/internal/path" ) -// RenameResponse contains the response fields for the Create operation. -type RenameResponse struct { - Response CreateResponse - NewDirectoryClient *Client -} +// RenameResponse contains the response fields for the Rename operation. +type RenameResponse = path.RenameResponse type setAccessControlRecursiveResponse struct { DirectoriesSuccessful *int32 diff --git a/sdk/storage/azdatalake/doc.go b/sdk/storage/azdatalake/doc.go new file mode 100644 index 000000000000..50b93e555d49 --- /dev/null +++ b/sdk/storage/azdatalake/doc.go @@ -0,0 +1,218 @@ +//go:build go1.18 +// +build go1.18 + +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT License. See License.txt in the project root for license information. + +/* + +Package azdatalake can access an Azure Data Lake Service Gen2 (ADLS Gen2). + +The azdatalake package is capable of :- + - Creating, deleting, and querying filesystems in an account + - Creating, deleting, and querying files and directories in a filesystem + - Creating Shared Access Signature for authentication + +Types of Resources + +The azdatalake package allows you to interact with three types of resources :- + +* Azure storage accounts. +* filesystems within those storage accounts. +* files and directories within those filesystems. + +ADLS Gen2 client library for Go allows you to interact with each of these components through the use of a dedicated client object. +To create a client object, you will need the account's ADLS Gen2 service endpoint URL and a credential that allows you to access the account. + +Types of Credentials + +The clients support different forms of authentication. +The azdatalake library supports any of the `azcore.TokenCredential` interfaces, authorization via a Connection String, +or authorization with a Shared Access Signature token. + +Using a Shared Key + +To use an account shared key (aka account key or access key), provide the key as a string. +This can be found in your storage account in the Azure Portal under the "Access Keys" section. + +Use the key as the credential parameter to authenticate the client: + + accountName, ok := os.LookupEnv("AZURE_STORAGE_ACCOUNT_NAME") + if !ok { + panic("AZURE_STORAGE_ACCOUNT_NAME could not be found") + } + accountKey, ok := os.LookupEnv("AZURE_STORAGE_ACCOUNT_KEY") + if !ok { + panic("AZURE_STORAGE_ACCOUNT_KEY could not be found") + } + + serviceURL := fmt.Sprintf("https://%s.dfs.core.windows.net/", accountName) + + cred, err := azdatalake.NewSharedKeyCredential(accountName, accountKey) + handle(err) + + serviceClient, err := service.NewClientWithSharedKey(serviceURL, cred, nil) + handle(err) + + // get the underlying dfs endpoint + fmt.Println(serviceClient.DFSURL()) + + // get the underlying blob endpoint + fmt.Println(serviceClient.BlobURL()) + +Using a Connection String + +Depending on your use case and authorization method, you may prefer to initialize a client instance with a connection string instead of providing the account URL and credential separately. +To do this, pass the connection string to the service client's `NewClientFromConnectionString` method. +The connection string can be found in your storage account in the Azure Portal under the "Access Keys" section. + + connStr := "DefaultEndpointsProtocol=https;AccountName=;AccountKey=;EndpointSuffix=core.windows.net" + serviceClient, err := service.NewClientFromConnectionString(connStr, nil) + +Using a Shared Access Signature (SAS) Token + +To use a shared access signature (SAS) token, provide the token at the end of your service URL. +You can generate a SAS token from the Azure Portal under Shared Access Signature or use the ServiceClient.GetSASToken() functions. + + accountName, ok := os.LookupEnv("AZURE_STORAGE_ACCOUNT_NAME") + if !ok { + panic("AZURE_STORAGE_ACCOUNT_NAME could not be found") + } + accountKey, ok := os.LookupEnv("AZURE_STORAGE_ACCOUNT_KEY") + if !ok { + panic("AZURE_STORAGE_ACCOUNT_KEY could not be found") + } + serviceURL := fmt.Sprintf("https://%s.dfs.core.windows.net/", accountName) + + cred, err := azdatalake.NewSharedKeyCredential(accountName, accountKey) + handle(err) + serviceClient, err := service.NewClientWithSharedKey(serviceURL, cred, nil) + handle(err) + fmt.Println(serviceClient.DFSURL()) + + // Alternatively, you can create SAS on the fly + + resources := sas.AccountResourceTypes{Service: true} + permission := sas.AccountPermissions{Read: true} + start := time.Now() + expiry := start.AddDate(0, 0, 1) + serviceURLWithSAS, err := serviceClient.GetSASURL(resources, permission, start, expiry) + handle(err) + + serviceClientWithSAS, err := service.NewClientWithNoCredential(serviceURLWithSAS, nil) + handle(err) + + fmt.Println(serviceClientWithSAS.DFSURL()) + fmt.Println(serviceClientWithSAS.BlobURL()) + +Types of Clients + +There are three different clients provided to interact with the various components of the ADLS Gen2 Service: + +1. **`ServiceClient`** + * Get and set account settings. + * Query, create, list and delete filesystems within the account. + +2. **`FileSystemClient`** + * Get and set filesystem access settings, properties, and metadata. + * Create, delete, and query files/directories within the filesystem. + * `FileSystemLeaseClient` to support filesystem lease management. + +3. **`FileClient`** + * Get and set file properties. + * Perform CRUD operations on a given file. + * Set ACLs on a given file. + * `PathLeaseClient` to support file lease management. + +4. **`DirectoryClient`** + * Get and set directory properties. + * Perform CRUD operations on a given directory. + * Set ACLs on a given directory and recursively on all subdirectories and files. + * `PathLeaseClient` to support directory lease management. + +Examples + + // Your account name and key can be obtained from the Azure Portal. + accountName, ok := os.LookupEnv("AZURE_STORAGE_ACCOUNT_NAME") + if !ok { + panic("AZURE_STORAGE_ACCOUNT_NAME could not be found") + } + + accountKey, ok := os.LookupEnv("AZURE_STORAGE_ACCOUNT_KEY") + if !ok { + panic("AZURE_STORAGE_ACCOUNT_KEY could not be found") + } + cred, err := azdatalake.NewSharedKeyCredential(accountName, accountKey) + handle(err) + + // The service URL for dfs endpoints is usually in the form: http(s)://.dfs.core.windows.net/ + serviceClient, err := service.NewClientWithSharedKey(fmt.Sprintf("https://%s.dfs.core.windows.net/", accountName), cred, nil) + handle(err) + + // ===== 1. Create a filesystem ===== + + // First, create a filesystem client, and use the Create method to create a new filesystem in your account + fsClient, err := serviceClient.NewFileSystemClient("testfs") + handle(err) + + // All APIs have an options' bag struct as a parameter. + // The options' bag struct allows you to specify optional parameters such as metadata, public access types, etc. + // If you want to use the default options, pass in nil. + _, err = fsClient.Create(context.TODO(), nil) + handle(err) + + // ===== 2. Upload and Download a file ===== + uploadData := "Hello world!" + + // Create a new file from the fsClient + fileClient, err := fsClient.NewFileClient("HelloWorld.txt") + handle(err) + + + _, err = fileClient.UploadStream(context.TODO(), streaming.NopCloser(strings.NewReader(uploadData)), nil) + handle(err) + + // Download the file's contents and ensure that the download worked properly + fileDownloadResponse, err := fileClient.DownloadStream(context.TODO(), nil) + handle(err) + + // Use the bytes.Buffer object to read the downloaded data. + // RetryReaderOptions has a lot of in-depth tuning abilities, but for the sake of simplicity, we'll omit those here. + reader := fileDownloadResponse.Body(nil) + downloadData, err := io.ReadAll(reader) + handle(err) + if string(downloadData) != uploadData { + handle(errors.New("Uploaded data should be same as downloaded data")) + } + + if err = reader.Close(); err != nil { + handle(err) + return + } + + // ===== 3. List paths ===== + // List methods returns a pager object which can be used to iterate over the results of a paging operation. + // To iterate over a page use the NextPage(context.Context) to fetch the next page of results. + // PageResponse() can be used to iterate over the results of the specific page. + // Always check the Err() method after paging to see if an error was returned by the pager. A pager will return either an error or the page of results. + + pager := fsClient.NewListPathsPager(nil) + page, err := pager.NextPage(context.TODO()) + handle(err) + + // print the path names for this page + for _, path := range page.PathList.Paths { + fmt.Println(*path.Name) + fmt.Println(*path.IsDirectory) + } + + // Delete the file. + _, err = fileClient.Delete(context.TODO(), nil) + handle(err) + + // Delete the filesystem. + _, err = fsClient.Delete(context.TODO(), nil) + handle(err) +*/ + +package azdatalake diff --git a/sdk/storage/azdatalake/file/client.go b/sdk/storage/azdatalake/file/client.go index 0a84b5af7cba..113750ae05e9 100644 --- a/sdk/storage/azdatalake/file/client.go +++ b/sdk/storage/azdatalake/file/client.go @@ -125,7 +125,7 @@ func NewClientWithSharedKeyCredential(fileURL string, cred *SharedKeyCredential, blobClientOpts := blockblob.ClientOptions{ ClientOptions: options.ClientOptions, } - blobSharedKey, err := cred.ConvertToBlobSharedKey() + blobSharedKey, err := exported.ConvertToBlobSharedKey(cred) if err != nil { return nil, err } @@ -196,7 +196,7 @@ func (f *Client) BlobURL() string { return f.generatedFileClientWithBlob().Endpoint() } -// Create creates a new file (dfs1). +// Create creates a new file. func (f *Client) Create(ctx context.Context, options *CreateOptions) (CreateResponse, error) { lac, mac, httpHeaders, createOpts, cpkOpts := options.format() resp, err := f.generatedFileClientWithDFS().Create(ctx, createOpts, httpHeaders, lac, mac, nil, cpkOpts) @@ -204,7 +204,7 @@ func (f *Client) Create(ctx context.Context, options *CreateOptions) (CreateResp return resp, err } -// Delete deletes a file (dfs1). +// Delete deletes a file. func (f *Client) Delete(ctx context.Context, options *DeleteOptions) (DeleteResponse, error) { lac, mac, deleteOpts := path.FormatDeleteOptions(options, false) resp, err := f.generatedFileClientWithDFS().Delete(ctx, deleteOpts, lac, mac) @@ -212,7 +212,7 @@ func (f *Client) Delete(ctx context.Context, options *DeleteOptions) (DeleteResp return resp, err } -// GetProperties gets the properties of a file (blob3) +// GetProperties gets the properties of a file. func (f *Client) GetProperties(ctx context.Context, options *GetPropertiesOptions) (GetPropertiesResponse, error) { opts := path.FormatGetPropertiesOptions(options) var respFromCtx *http.Response @@ -231,33 +231,34 @@ func (f *Client) renamePathInURL(newName string) (string, string, string) { // Split the string based on the last occurrence of the separator firstPart := endpoint[:lastIndex] // From the beginning of the string to the last occurrence of the separator newBlobURL, newPathURL := shared.GetURLs(runtime.JoinPaths(firstPart, newName)) - parsedNewURL, _ := url.Parse(f.DFSURL()) - return parsedNewURL.Path, newPathURL, newBlobURL + oldURL, _ := url.Parse(f.DFSURL()) + return oldURL.Path, newPathURL, newBlobURL } -// Rename renames a file (dfs1) +// Rename renames a file. The original file will no longer exist and the client will be stale. func (f *Client) Rename(ctx context.Context, newName string, options *RenameOptions) (RenameResponse, error) { - newPathWithoutURL, newPathURL, newBlobURL := f.renamePathInURL(newName) - lac, mac, smac, createOpts := path.FormatRenameOptions(options, newPathWithoutURL) + oldPathWithoutURL, newPathURL, newBlobURL := f.renamePathInURL(newName) + lac, mac, smac, createOpts := path.FormatRenameOptions(options, oldPathWithoutURL) var newBlobClient *blockblob.Client var err error if f.identityCredential() != nil { newBlobClient, err = blockblob.NewClient(newBlobURL, *f.identityCredential(), nil) } else if f.sharedKey() != nil { - blobSharedKey, _ := f.sharedKey().ConvertToBlobSharedKey() + blobSharedKey, _ := exported.ConvertToBlobSharedKey(f.sharedKey()) newBlobClient, err = blockblob.NewClientWithSharedKeyCredential(newBlobURL, blobSharedKey, nil) } else { newBlobClient, err = blockblob.NewClientWithNoCredential(newBlobURL, nil) } if err != nil { - return RenameResponse{}, err + return RenameResponse{}, exported.ConvertToDFSError(err) } newFileClient := (*Client)(base.NewPathClient(newPathURL, newBlobURL, newBlobClient, f.generatedFileClientWithDFS().InternalClient().WithClientName(shared.FileClient), f.sharedKey(), f.identityCredential(), f.getClientOptions())) resp, err := newFileClient.generatedFileClientWithDFS().Create(ctx, createOpts, nil, lac, mac, smac, nil) - return RenameResponse{ - Response: resp, - NewFileClient: newFileClient, - }, exported.ConvertToDFSError(err) + //return RenameResponse{ + // Response: resp, + // NewFileClient: newFileClient, + //}, exported.ConvertToDFSError(err) + return path.FormatRenameResponse(&resp), exported.ConvertToDFSError(err) } // SetExpiry operation sets an expiry time on an existing file (blob2). @@ -274,7 +275,7 @@ func (f *Client) SetExpiry(ctx context.Context, expiryValues SetExpiryValues, o return resp, err } -// SetAccessControl sets the owner, owning group, and permissions for a file or directory (dfs1). +// SetAccessControl sets the owner, owning group, and permissions for a file. func (f *Client) SetAccessControl(ctx context.Context, options *SetAccessControlOptions) (SetAccessControlResponse, error) { opts, lac, mac, err := path.FormatSetAccessControlOptions(options) if err != nil { @@ -285,7 +286,7 @@ func (f *Client) SetAccessControl(ctx context.Context, options *SetAccessControl return resp, err } -// UpdateAccessControl updates the owner, owning group, and permissions for a file or directory (dfs1). +// UpdateAccessControl updates the owner, owning group, and permissions for a file. func (f *Client) UpdateAccessControl(ctx context.Context, ACL string, options *UpdateAccessControlOptions) (UpdateAccessControlResponse, error) { opts, mode := options.format(ACL) resp, err := f.generatedFileClientWithDFS().SetAccessControlRecursive(ctx, mode, opts) @@ -293,7 +294,7 @@ func (f *Client) UpdateAccessControl(ctx context.Context, ACL string, options *U return resp, err } -// GetAccessControl gets the owner, owning group, and permissions for a file or directory (dfs1). +// GetAccessControl gets the owner, owning group, and permissions for a file. func (f *Client) GetAccessControl(ctx context.Context, options *GetAccessControlOptions) (GetAccessControlResponse, error) { opts, lac, mac := path.FormatGetAccessControlOptions(options) resp, err := f.generatedFileClientWithDFS().GetProperties(ctx, opts, lac, mac) @@ -301,7 +302,7 @@ func (f *Client) GetAccessControl(ctx context.Context, options *GetAccessControl return resp, err } -// RemoveAccessControl removes the owner, owning group, and permissions for a file or directory (dfs1). +// RemoveAccessControl removes the owner, owning group, and permissions for a file. func (f *Client) RemoveAccessControl(ctx context.Context, ACL string, options *RemoveAccessControlOptions) (RemoveAccessControlResponse, error) { opts, mode := options.format(ACL) resp, err := f.generatedFileClientWithDFS().SetAccessControlRecursive(ctx, mode, opts) @@ -309,15 +310,15 @@ func (f *Client) RemoveAccessControl(ctx context.Context, ACL string, options *R return resp, err } -// SetMetadata sets the metadata for a file or directory (blob3). -func (f *Client) SetMetadata(ctx context.Context, options *SetMetadataOptions) (SetMetadataResponse, error) { - opts, metadata := path.FormatSetMetadataOptions(options) +// SetMetadata sets the metadata for a file. +func (f *Client) SetMetadata(ctx context.Context, metadata map[string]*string, options *SetMetadataOptions) (SetMetadataResponse, error) { + opts := path.FormatSetMetadataOptions(options) resp, err := f.blobClient().SetMetadata(ctx, metadata, opts) err = exported.ConvertToDFSError(err) return resp, err } -// SetHTTPHeaders sets the HTTP headers for a file or directory (blob3). +// SetHTTPHeaders sets the HTTP headers for a file. func (f *Client) SetHTTPHeaders(ctx context.Context, httpHeaders HTTPHeaders, options *SetHTTPHeadersOptions) (SetHTTPHeadersResponse, error) { opts, blobHTTPHeaders := path.FormatSetHTTPHeadersOptions(options, httpHeaders) resp, err := f.blobClient().SetHTTPHeaders(ctx, blobHTTPHeaders, opts) @@ -327,7 +328,7 @@ func (f *Client) SetHTTPHeaders(ctx context.Context, httpHeaders HTTPHeaders, op return newResp, err } -// GetSASURL is a convenience method for generating a SAS token for the currently pointed at blob. +// GetSASURL is a convenience method for generating a SAS token for the currently pointed at file. // It can only be used if the credential supplied during creation was a SharedKeyCredential. func (f *Client) GetSASURL(permissions sas.FilePermissions, expiry time.Time, o *GetSASURLOptions) (string, error) { if f.sharedKey() == nil { @@ -344,7 +345,7 @@ func (f *Client) GetSASURL(permissions sas.FilePermissions, expiry time.Time, o qps, err := sas.DatalakeSignatureValues{ FilePath: urlParts.PathName, - FilesystemName: urlParts.FilesystemName, + FileSystemName: urlParts.FileSystemName, Version: sas.Version, Permissions: permissions.String(), StartTime: st, @@ -361,6 +362,7 @@ func (f *Client) GetSASURL(permissions sas.FilePermissions, expiry time.Time, o return endpoint, nil } +// AppendData appends data to existing file with a given offset. func (f *Client) AppendData(ctx context.Context, offset int64, body io.ReadSeekCloser, options *AppendDataOptions) (AppendDataResponse, error) { appendDataOptions, leaseAccessConditions, cpkInfo, err := options.format(offset, body) if err != nil { @@ -377,6 +379,7 @@ func (f *Client) AppendData(ctx context.Context, offset int64, body io.ReadSeekC return resp, exported.ConvertToDFSError(err) } +// FlushData commits appended data to file func (f *Client) FlushData(ctx context.Context, offset int64, options *FlushDataOptions) (FlushDataResponse, error) { flushDataOpts, modifiedAccessConditions, leaseAccessConditions, httpHeaderOpts, cpkInfoOpts, err := options.format(offset) if err != nil { @@ -453,7 +456,7 @@ func (f *Client) uploadFromReader(ctx context.Context, reader io.ReaderAt, actua return exported.ConvertToDFSError(err) } -// UploadBuffer uploads a buffer in chunks to an Azure file. +// UploadBuffer uploads a buffer in chunks to a file. func (f *Client) UploadBuffer(ctx context.Context, buffer []byte, options *UploadBufferOptions) error { uploadOptions := uploadFromReaderOptions{} if options != nil { @@ -462,7 +465,7 @@ func (f *Client) UploadBuffer(ctx context.Context, buffer []byte, options *Uploa return exported.ConvertToDFSError(f.uploadFromReader(ctx, bytes.NewReader(buffer), int64(len(buffer)), &uploadOptions)) } -// UploadFile uploads a file in chunks to an Azure file. +// UploadFile uploads a file in chunks to a file. func (f *Client) UploadFile(ctx context.Context, file *os.File, options *UploadFileOptions) error { stat, err := file.Stat() if err != nil { @@ -486,7 +489,7 @@ func (f *Client) UploadStream(ctx context.Context, body io.Reader, options *Uplo return exported.ConvertToDFSError(err) } -// DownloadStream reads a range of bytes from a blob. The response also includes the blob's properties and metadata. +// DownloadStream reads a range of bytes from a file. The response also includes the file's properties and metadata. // For more information, see https://docs.microsoft.com/rest/api/storageservices/get-blob. func (f *Client) DownloadStream(ctx context.Context, o *DownloadStreamOptions) (DownloadStreamResponse, error) { if o == nil { @@ -506,14 +509,14 @@ func (f *Client) DownloadStream(ctx context.Context, o *DownloadStreamOptions) ( return fullResp, exported.ConvertToDFSError(err) } -// DownloadBuffer downloads an Azure blob to a buffer with parallel. +// DownloadBuffer downloads an Azure file to a buffer with parallel. func (f *Client) DownloadBuffer(ctx context.Context, buffer []byte, o *DownloadBufferOptions) (int64, error) { opts := o.format() val, err := f.blobClient().DownloadBuffer(ctx, shared.NewBytesWriter(buffer), opts) return val, exported.ConvertToDFSError(err) } -// DownloadFile downloads an Azure blob to a local file. +// DownloadFile downloads a datalake file to a local file. // The file would be truncated if the size doesn't match. func (f *Client) DownloadFile(ctx context.Context, file *os.File, o *DownloadFileOptions) (int64, error) { opts := o.format() @@ -521,4 +524,4 @@ func (f *Client) DownloadFile(ctx context.Context, file *os.File, o *DownloadFil return val, exported.ConvertToDFSError(err) } -// TODO: add undelete +// TODO: Undelete() diff --git a/sdk/storage/azdatalake/file/client_test.go b/sdk/storage/azdatalake/file/client_test.go index 8ab3897c6d9d..23a9f03d4c35 100644 --- a/sdk/storage/azdatalake/file/client_test.go +++ b/sdk/storage/azdatalake/file/client_test.go @@ -85,10 +85,10 @@ func (s *UnrecordedTestSuite) TestCreateFileAndDeleteWithConnectionString() { connectionString, _ := testcommon.GetGenericConnectionString(testcommon.TestAccountDatalake) - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) @@ -109,10 +109,10 @@ func (s *RecordedTestSuite) TestCreateFileAndDelete() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) @@ -132,10 +132,10 @@ func (s *RecordedTestSuite) TestCreateFileWithNilAccessConditions() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) @@ -163,10 +163,10 @@ func (s *RecordedTestSuite) TestCreateFileIfModifiedSinceTrue() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) @@ -200,10 +200,10 @@ func (s *RecordedTestSuite) TestCreateFileIfModifiedSinceFalse() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) @@ -237,10 +237,10 @@ func (s *RecordedTestSuite) TestCreateFileIfUnmodifiedSinceTrue() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) @@ -274,10 +274,10 @@ func (s *RecordedTestSuite) TestCreateFileIfUnmodifiedSinceFalse() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) @@ -312,10 +312,10 @@ func (s *RecordedTestSuite) TestCreateFileIfETagMatch() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) @@ -349,10 +349,10 @@ func (s *RecordedTestSuite) TestCreateFileIfETagMatchFalse() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) @@ -383,17 +383,17 @@ func (s *RecordedTestSuite) TestCreateFileIfETagMatchFalse() { testcommon.ValidateErrorCode(_require, err, datalakeerror.ConditionNotMet) } -func (s *RecordedTestSuite) TestCreateFileWithMetadataNotNil() { +func (s *RecordedTestSuite) TestCreateFileWithNilHTTPHeaders() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) createFileOpts := &file.CreateOptions{ - Metadata: testcommon.BasicMetadata, + HTTPHeaders: nil, } _, err = fsClient.Create(context.Background(), nil) @@ -410,17 +410,21 @@ func (s *RecordedTestSuite) TestCreateFileWithMetadataNotNil() { _require.NotNil(resp) } -func (s *RecordedTestSuite) TestCreateFileWithEmptyMetadata() { +func (s *UnrecordedTestSuite) TestCreateFileWithExpiryAbsolute() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) + expiryTimeAbsolute := time.Now().Add(8 * time.Second) createFileOpts := &file.CreateOptions{ - Metadata: nil, + Expiry: file.CreateExpiryValues{ + ExpiryType: file.CreateExpiryTypeAbsolute, + ExpiresOn: time.Now().Add(8 * time.Second).UTC().Format(http.TimeFormat), + }, } _, err = fsClient.Create(context.Background(), nil) @@ -435,49 +439,25 @@ func (s *RecordedTestSuite) TestCreateFileWithEmptyMetadata() { resp, err := fClient.Create(context.Background(), createFileOpts) _require.Nil(err) _require.NotNil(resp) -} - -func (s *RecordedTestSuite) TestCreateFileWithNilHTTPHeaders() { - _require := require.New(s.T()) - testName := s.T().Name() - - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) - _require.NoError(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) - - createFileOpts := &file.CreateOptions{ - HTTPHeaders: nil, - } - - _, err = fsClient.Create(context.Background(), nil) - _require.Nil(err) - - fileName := testcommon.GenerateFileName(testName) - fClient, err := testcommon.GetFileClient(filesystemName, fileName, s.T(), testcommon.TestAccountDatalake, nil) - _require.NoError(err) - defer testcommon.DeleteFile(context.Background(), _require, fClient) - - resp, err := fClient.Create(context.Background(), createFileOpts) + resp1, err := fClient.GetProperties(context.Background(), nil) _require.Nil(err) - _require.NotNil(resp) + _require.NotNil(resp1.ExpiresOn) + _require.Equal(expiryTimeAbsolute.UTC().Format(http.TimeFormat), (*resp1.ExpiresOn).UTC().Format(http.TimeFormat)) } -func (s *UnrecordedTestSuite) TestCreateFileWithExpiryAbsolute() { +func (s *RecordedTestSuite) TestCreateFileWithExpiryNever() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) - expiryTimeAbsolute := time.Now().Add(8 * time.Second) createFileOpts := &file.CreateOptions{ Expiry: file.CreateExpiryValues{ - ExpiryType: file.CreateExpiryTypeAbsolute, - ExpiresOn: time.Now().Add(8 * time.Second).UTC().Format(http.TimeFormat), + ExpiryType: file.CreateExpiryTypeNeverExpire, }, } @@ -488,26 +468,23 @@ func (s *UnrecordedTestSuite) TestCreateFileWithExpiryAbsolute() { fClient, err := testcommon.GetFileClient(filesystemName, fileName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) - defer testcommon.DeleteFile(context.Background(), _require, fClient) - resp, err := fClient.Create(context.Background(), createFileOpts) _require.Nil(err) _require.NotNil(resp) - resp1, err := fClient.GetProperties(context.Background(), nil) + _, err = fClient.Delete(context.Background(), nil) _require.Nil(err) - _require.NotNil(resp1.ExpiresOn) - _require.Equal(expiryTimeAbsolute.UTC().Format(http.TimeFormat), (*resp1.ExpiresOn).UTC().Format(http.TimeFormat)) + } func (s *RecordedTestSuite) TestCreateFileWithExpiryRelativeToNow() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) createFileOpts := &file.CreateOptions{ Expiry: file.CreateExpiryValues{ @@ -540,10 +517,10 @@ func (s *RecordedTestSuite) TestCreateFileWithNeverExpire() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) createFileOpts := &file.CreateOptions{ Expiry: file.CreateExpiryValues{ @@ -571,10 +548,10 @@ func (s *RecordedTestSuite) TestCreateFileWithLease() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) createFileOpts := &file.CreateOptions{ ProposedLeaseID: proposedLeaseIDs[0], @@ -606,12 +583,12 @@ func (s *RecordedTestSuite) TestCreateFileWithPermissions() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) perms := "0777" umask := "0000" - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) createFileOpts := &file.CreateOptions{ Permissions: &perms, @@ -639,14 +616,14 @@ func (s *RecordedTestSuite) TestCreateFileWithOwnerGroupACLUmask() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) owner := "4cf4e284-f6a8-4540-b53e-c3469af032dc" group := owner acl := "user::rwx,group::r-x,other::rwx" umask := "0000" - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) createFileOpts := &file.CreateOptions{ Owner: &owner, @@ -672,10 +649,10 @@ func (s *RecordedTestSuite) TestDeleteFileWithNilAccessConditions() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) @@ -700,10 +677,10 @@ func (s *RecordedTestSuite) TestDeleteFileIfModifiedSinceTrue() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) @@ -734,10 +711,10 @@ func (s *RecordedTestSuite) TestDeleteFileIfModifiedSinceFalse() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) @@ -769,10 +746,10 @@ func (s *RecordedTestSuite) TestDeleteFileIfUnmodifiedSinceTrue() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) @@ -804,10 +781,10 @@ func (s *RecordedTestSuite) TestDeleteFileIfUnmodifiedSinceFalse() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) @@ -840,10 +817,10 @@ func (s *RecordedTestSuite) TestDeleteFileIfETagMatch() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) @@ -875,10 +852,10 @@ func (s *RecordedTestSuite) TestDeleteFileIfETagMatchFalse() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) @@ -911,10 +888,10 @@ func (s *RecordedTestSuite) TestFileSetAccessControlNil() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) @@ -938,13 +915,13 @@ func (s *RecordedTestSuite) TestFileSetAccessControl() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) owner := "4cf4e284-f6a8-4540-b53e-c3469af032dc" group := owner acl := "user::rwx,group::r-x,other::rwx" - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) opts := &file.SetAccessControlOptions{ Owner: &owner, @@ -971,13 +948,13 @@ func (s *RecordedTestSuite) TestFileSetAccessControlWithNilAccessConditions() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) owner := "4cf4e284-f6a8-4540-b53e-c3469af032dc" group := owner acl := "user::rwx,group::r-x,other::rwx" - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) opts := &file.SetAccessControlOptions{ Owner: &owner, @@ -1005,13 +982,13 @@ func (s *RecordedTestSuite) TestFileSetAccessControlIfModifiedSinceTrue() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) owner := "4cf4e284-f6a8-4540-b53e-c3469af032dc" group := owner acl := "user::rwx,group::r-x,other::rwx" - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) @@ -1044,13 +1021,13 @@ func (s *RecordedTestSuite) TestFileSetAccessControlIfModifiedSinceFalse() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) owner := "4cf4e284-f6a8-4540-b53e-c3469af032dc" group := owner acl := "user::rwx,group::r-x,other::rwx" - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) @@ -1084,13 +1061,13 @@ func (s *RecordedTestSuite) TestFileSetAccessControlIfUnmodifiedSinceTrue() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) owner := "4cf4e284-f6a8-4540-b53e-c3469af032dc" group := owner acl := "user::rwx,group::r-x,other::rwx" - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) @@ -1122,13 +1099,13 @@ func (s *RecordedTestSuite) TestFileSetAccessControlIfUnmodifiedSinceFalse() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) owner := "4cf4e284-f6a8-4540-b53e-c3469af032dc" group := owner acl := "user::rwx,group::r-x,other::rwx" - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) @@ -1164,13 +1141,13 @@ func (s *RecordedTestSuite) TestFileSetAccessControlIfETagMatch() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) owner := "4cf4e284-f6a8-4540-b53e-c3469af032dc" group := owner acl := "user::rwx,group::r-x,other::rwx" - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) @@ -1203,13 +1180,13 @@ func (s *RecordedTestSuite) TestFileSetAccessControlIfETagMatchFalse() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) owner := "4cf4e284-f6a8-4540-b53e-c3469af032dc" group := owner acl := "user::rwx,group::r-x,other::rwx" - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) @@ -1242,11 +1219,11 @@ func (s *RecordedTestSuite) TestFileGetAccessControl() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) acl := "user::rwx,group::r-x,other::rwx" - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) createOpts := &file.CreateOptions{ ACL: &acl, @@ -1271,11 +1248,11 @@ func (s *UnrecordedTestSuite) TestFileGetAccessControlWithSAS() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) acl := "user::rwx,group::r-x,other::rwx" - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) createOpts := &file.CreateOptions{ ACL: &acl, @@ -1315,10 +1292,10 @@ func (s *UnrecordedTestSuite) TestFileDeleteWithSAS() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) @@ -1354,11 +1331,11 @@ func (s *RecordedTestSuite) TestFileGetAccessControlWithNilAccessConditions() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) acl := "user::rwx,group::r-x,other::rwx" - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) createOpts := &file.CreateOptions{ ACL: &acl, @@ -1387,11 +1364,11 @@ func (s *RecordedTestSuite) TestFileGetAccessControlIfModifiedSinceTrue() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) acl := "user::rwx,group::r-x,other::rwx" - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) createOpts := &file.CreateOptions{ ACL: &acl, @@ -1425,11 +1402,11 @@ func (s *RecordedTestSuite) TestFileGetAccessControlIfModifiedSinceFalse() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) acl := "user::rwx,group::r-x,other::rwx" - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) createOpts := &file.CreateOptions{ ACL: &acl, @@ -1463,11 +1440,11 @@ func (s *RecordedTestSuite) TestFileGetAccessControlIfUnmodifiedSinceTrue() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) acl := "user::rwx,group::r-x,other::rwx" - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) createOpts := &file.CreateOptions{ ACL: &acl, @@ -1500,11 +1477,11 @@ func (s *RecordedTestSuite) TestFileGetAccessControlIfUnmodifiedSinceFalse() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) acl := "user::rwx,group::r-x,other::rwx" - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) createOpts := &file.CreateOptions{ ACL: &acl, @@ -1540,11 +1517,11 @@ func (s *RecordedTestSuite) TestFileGetAccessControlIfETagMatch() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) acl := "user::rwx,group::r-x,other::rwx" - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) createOpts := &file.CreateOptions{ ACL: &acl, @@ -1578,11 +1555,11 @@ func (s *RecordedTestSuite) TestFileGetAccessControlIfETagMatchFalse() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) acl := "user::rwx,group::r-x,other::rwx" - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) createOpts := &file.CreateOptions{ ACL: &acl, @@ -1615,12 +1592,12 @@ func (s *RecordedTestSuite) TestFileUpdateAccessControl() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) acl := "user::rwx,group::r-x,other::rwx" acl1 := "user::rwx,group::r--,other::r--" - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) createOpts := &file.CreateOptions{ ACL: &acl, @@ -1648,13 +1625,13 @@ func (s *RecordedTestSuite) TestFileRemoveAccessControl() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) acl := "mask," + "default:user,default:group," + "user:ec3595d6-2c17-4696-8caa-7e139758d24a,group:ec3595d6-2c17-4696-8caa-7e139758d24a," + "default:user:ec3595d6-2c17-4696-8caa-7e139758d24a,default:group:ec3595d6-2c17-4696-8caa-7e139758d24a" - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) @@ -1671,69 +1648,14 @@ func (s *RecordedTestSuite) TestFileRemoveAccessControl() { _require.Nil(err) } -func (s *RecordedTestSuite) TestFileSetMetadataNil() { - _require := require.New(s.T()) - testName := s.T().Name() - - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) - _require.NoError(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) - - _, err = fsClient.Create(context.Background(), nil) - _require.Nil(err) - - fileName := testcommon.GenerateFileName(testName) - fClient, err := testcommon.GetFileClient(filesystemName, fileName, s.T(), testcommon.TestAccountDatalake, nil) - _require.NoError(err) - - defer testcommon.DeleteFile(context.Background(), _require, fClient) - - resp, err := fClient.Create(context.Background(), nil) - _require.Nil(err) - _require.NotNil(resp) - - _, err = fClient.SetMetadata(context.Background(), nil) - _require.Nil(err) -} - -func (s *RecordedTestSuite) TestFileSetMetadataWithEmptyOpts() { - _require := require.New(s.T()) - testName := s.T().Name() - - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) - _require.NoError(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) - - _, err = fsClient.Create(context.Background(), nil) - _require.Nil(err) - - fileName := testcommon.GenerateFileName(testName) - fClient, err := testcommon.GetFileClient(filesystemName, fileName, s.T(), testcommon.TestAccountDatalake, nil) - _require.NoError(err) - - defer testcommon.DeleteFile(context.Background(), _require, fClient) - - resp, err := fClient.Create(context.Background(), nil) - _require.Nil(err) - _require.NotNil(resp) - - opts := &file.SetMetadataOptions{ - Metadata: nil, - } - _, err = fClient.SetMetadata(context.Background(), opts) - _require.Nil(err) -} - func (s *RecordedTestSuite) TestFileSetMetadataWithBasicMetadata() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) @@ -1748,10 +1670,7 @@ func (s *RecordedTestSuite) TestFileSetMetadataWithBasicMetadata() { _require.Nil(err) _require.NotNil(resp) - opts := &file.SetMetadataOptions{ - Metadata: testcommon.BasicMetadata, - } - _, err = fClient.SetMetadata(context.Background(), opts) + _, err = fClient.SetMetadata(context.Background(), testcommon.BasicMetadata, nil) _require.Nil(err) } @@ -1759,10 +1678,10 @@ func (s *RecordedTestSuite) TestFileSetMetadataWithAccessConditions() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) @@ -1780,14 +1699,13 @@ func (s *RecordedTestSuite) TestFileSetMetadataWithAccessConditions() { currentTime := testcommon.GetRelativeTimeFromAnchor(resp.Date, -10) opts := &file.SetMetadataOptions{ - Metadata: testcommon.BasicMetadata, AccessConditions: &file.AccessConditions{ ModifiedAccessConditions: &file.ModifiedAccessConditions{ IfModifiedSince: ¤tTime, }, }, } - _, err = fClient.SetMetadata(context.Background(), opts) + _, err = fClient.SetMetadata(context.Background(), testcommon.BasicMetadata, opts) _require.Nil(err) } @@ -1801,10 +1719,10 @@ func (s *RecordedTestSuite) TestFileSetHTTPHeaders() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) @@ -1826,10 +1744,10 @@ func (s *RecordedTestSuite) TestFileSetHTTPHeadersWithNilAccessConditions() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) @@ -1855,10 +1773,10 @@ func (s *RecordedTestSuite) TestFileSetHTTPHeadersIfModifiedSinceTrue() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) @@ -1889,10 +1807,10 @@ func (s *RecordedTestSuite) TestFileSetHTTPHeadersIfModifiedSinceFalse() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) @@ -1923,10 +1841,10 @@ func (s *RecordedTestSuite) TestFileSetHTTPHeadersIfUnmodifiedSinceTrue() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) @@ -1957,10 +1875,10 @@ func (s *RecordedTestSuite) TestFileSetHTTPHeadersIfUnmodifiedSinceFalse() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) @@ -1991,10 +1909,10 @@ func (s *RecordedTestSuite) TestFileSetHTTPHeadersIfETagMatch() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) @@ -2024,10 +1942,10 @@ func (s *RecordedTestSuite) TestFileSetHTTPHeadersIfETagMatchFalse() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) @@ -2058,10 +1976,10 @@ func (s *RecordedTestSuite) TestRenameNoOptions() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) @@ -2074,20 +1992,21 @@ func (s *RecordedTestSuite) TestRenameNoOptions() { _require.Nil(err) _require.NotNil(resp) - resp1, err := fClient.Rename(context.Background(), "newName", nil) + //resp1, err := fClient.Rename(context.Background(), "newName", renameFileOpts) + _, err = fClient.Rename(context.Background(), "newName", nil) _require.Nil(err) - _require.NotNil(resp1) - _require.Contains(resp1.NewFileClient.DFSURL(), "newName") + //_require.NotNil(resp1) + //_require.Contains(resp1.NewFileClient.DFSURL(), "newName") } func (s *RecordedTestSuite) TestRenameFileWithNilAccessConditions() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) @@ -2104,20 +2023,21 @@ func (s *RecordedTestSuite) TestRenameFileWithNilAccessConditions() { AccessConditions: nil, } - resp1, err := fClient.Rename(context.Background(), "newName", renameFileOpts) + //resp1, err := fClient.Rename(context.Background(), "newName", renameFileOpts) + _, err = fClient.Rename(context.Background(), "newName", renameFileOpts) _require.Nil(err) - _require.NotNil(resp1) - _require.Contains(resp1.NewFileClient.DFSURL(), "newName") + //_require.NotNil(resp1) + //_require.Contains(resp1.NewFileClient.DFSURL(), "newName") } func (s *RecordedTestSuite) TestRenameFileIfModifiedSinceTrue() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) @@ -2139,20 +2059,21 @@ func (s *RecordedTestSuite) TestRenameFileIfModifiedSinceTrue() { }, }, } - resp1, err := fClient.Rename(context.Background(), "newName", renameFileOpts) + //resp1, err := fClient.Rename(context.Background(), "newName", renameFileOpts) + _, err = fClient.Rename(context.Background(), "newName", renameFileOpts) _require.Nil(err) - _require.NotNil(resp1) - _require.Contains(resp1.NewFileClient.DFSURL(), "newName") + //_require.NotNil(resp1) + //_require.Contains(resp1.NewFileClient.DFSURL(), "newName") } func (s *RecordedTestSuite) TestRenameFileIfModifiedSinceFalse() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) @@ -2175,7 +2096,9 @@ func (s *RecordedTestSuite) TestRenameFileIfModifiedSinceFalse() { }, } + //_, err = fClient.Rename(context.Background(), "newName", renameFileOpts) _, err = fClient.Rename(context.Background(), "newName", renameFileOpts) + _require.NotNil(err) testcommon.ValidateErrorCode(_require, err, datalakeerror.SourceConditionNotMet) } @@ -2184,10 +2107,10 @@ func (s *RecordedTestSuite) TestRenameFileIfUnmodifiedSinceTrue() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) @@ -2210,20 +2133,21 @@ func (s *RecordedTestSuite) TestRenameFileIfUnmodifiedSinceTrue() { }, } - resp1, err := fClient.Rename(context.Background(), "newName", renameFileOpts) + //resp1, err := fClient.Rename(context.Background(), "newName", renameFileOpts) + _, err = fClient.Rename(context.Background(), "newName", renameFileOpts) _require.Nil(err) - _require.NotNil(resp1) - _require.Contains(resp1.NewFileClient.DFSURL(), "newName") + //_require.NotNil(resp1) + //_require.Contains(resp1.NewFileClient.DFSURL(), "newName") } func (s *RecordedTestSuite) TestRenameFileIfUnmodifiedSinceFalse() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) @@ -2246,7 +2170,9 @@ func (s *RecordedTestSuite) TestRenameFileIfUnmodifiedSinceFalse() { }, } + //_, err = fClient.Rename(context.Background(), "newName", renameFileOpts) _, err = fClient.Rename(context.Background(), "newName", renameFileOpts) + _require.NotNil(err) testcommon.ValidateErrorCode(_require, err, datalakeerror.SourceConditionNotMet) } @@ -2255,10 +2181,10 @@ func (s *RecordedTestSuite) TestRenameFileIfETagMatch() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) @@ -2281,20 +2207,21 @@ func (s *RecordedTestSuite) TestRenameFileIfETagMatch() { }, } - resp1, err := fClient.Rename(context.Background(), "newName", renameFileOpts) + //resp1, err := fClient.Rename(context.Background(), "newName", renameFileOpts) + _, err = fClient.Rename(context.Background(), "newName", renameFileOpts) _require.Nil(err) - _require.NotNil(resp1) - _require.Contains(resp1.NewFileClient.DFSURL(), "newName") + //_require.NotNil(resp1) + //_require.Contains(resp1.NewFileClient.DFSURL(), "newName") } func (s *RecordedTestSuite) TestRenameFileIfETagMatchFalse() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) @@ -2317,7 +2244,9 @@ func (s *RecordedTestSuite) TestRenameFileIfETagMatchFalse() { }, } + //_, err = fClient.Rename(context.Background(), "newName", renameFileOpts) _, err = fClient.Rename(context.Background(), "newName", renameFileOpts) + _require.NotNil(err) testcommon.ValidateErrorCode(_require, err, datalakeerror.SourceConditionNotMet) } @@ -2326,10 +2255,10 @@ func (s *UnrecordedTestSuite) TestFileUploadDownloadStream() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) @@ -2376,10 +2305,10 @@ func (s *RecordedTestSuite) TestFileUploadDownloadSmallStream() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) @@ -2423,10 +2352,10 @@ func (s *RecordedTestSuite) TestFileUploadTinyStream() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) @@ -2470,10 +2399,10 @@ func (s *UnrecordedTestSuite) TestFileUploadFile() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) @@ -2538,10 +2467,10 @@ func (s *RecordedTestSuite) TestSmallFileUploadFile() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) @@ -2605,10 +2534,10 @@ func (s *RecordedTestSuite) TestTinyFileUploadFile() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) @@ -2671,10 +2600,10 @@ func (s *UnrecordedTestSuite) TestFileUploadBuffer() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) @@ -2719,10 +2648,10 @@ func (s *RecordedTestSuite) TestFileUploadSmallBuffer() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) @@ -2765,10 +2694,10 @@ func (s *RecordedTestSuite) TestFileAppendAndFlushData() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) @@ -2800,10 +2729,10 @@ func (s *UnrecordedTestSuite) TestFileAppendAndFlushDataWithValidation() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) @@ -2844,10 +2773,10 @@ func (s *RecordedTestSuite) TestFileAppendAndFlushDataWithEmptyOpts() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) @@ -2882,10 +2811,10 @@ func (s *RecordedTestSuite) TestFileAppendAndFlushDataWithLeasedFile() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) @@ -2937,10 +2866,10 @@ func (s *RecordedTestSuite) TestFileAppendAndFlushAndDownloadDataWithLeasedFile( _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) @@ -3000,10 +2929,10 @@ func (s *RecordedTestSuite) TestAppendAndFlushFileWithHTTPHeaders() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) @@ -3038,10 +2967,10 @@ func (s *RecordedTestSuite) TestFlushWithNilAccessConditions() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) @@ -3076,10 +3005,10 @@ func (s *RecordedTestSuite) TestFlushWithEmptyAccessConditions() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) @@ -3116,10 +3045,10 @@ func (s *RecordedTestSuite) TestFlushIfModifiedSinceTrue() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) @@ -3159,10 +3088,10 @@ func (s *RecordedTestSuite) TestFlushIfModifiedSinceFalse() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) @@ -3199,10 +3128,10 @@ func (s *RecordedTestSuite) TestFlushIfUnmodifiedSinceTrue() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) @@ -3243,10 +3172,10 @@ func (s *RecordedTestSuite) TestFlushIfUnmodifiedSinceFalse() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) @@ -3283,10 +3212,10 @@ func (s *RecordedTestSuite) TestFlushIfEtagMatch() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) @@ -3326,10 +3255,10 @@ func (s *RecordedTestSuite) TestFlushIfEtagMatchFalse() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) @@ -3371,10 +3300,10 @@ func (s *UnrecordedTestSuite) TestFileDownloadFile() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) @@ -3435,10 +3364,10 @@ func (s *RecordedTestSuite) TestFileUploadDownloadSmallFile() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) @@ -3514,10 +3443,10 @@ func (s *RecordedTestSuite) TestFileUploadDownloadSmallFileWithRange() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) @@ -3597,10 +3526,10 @@ func (s *RecordedTestSuite) TestFileUploadDownloadSmallFileWithAccessConditions( _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) @@ -3686,10 +3615,10 @@ func (s *RecordedTestSuite) TestFileUploadDownloadWithProgress() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) @@ -3747,10 +3676,10 @@ func (s *UnrecordedTestSuite) TestFileDownloadBuffer() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) @@ -3798,10 +3727,10 @@ func (s *RecordedTestSuite) TestFileDownloadSmallBuffer() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) @@ -3847,10 +3776,10 @@ func (s *RecordedTestSuite) TestFileDownloadSmallBufferWithHTTPRange() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) @@ -3900,10 +3829,10 @@ func (s *RecordedTestSuite) TestFileDownloadSmallBufferWithAccessConditions() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) diff --git a/sdk/storage/azdatalake/file/constants.go b/sdk/storage/azdatalake/file/constants.go index da5ec3d2b065..77729179d3d4 100644 --- a/sdk/storage/azdatalake/file/constants.go +++ b/sdk/storage/azdatalake/file/constants.go @@ -14,6 +14,7 @@ import ( "github.com/Azure/azure-sdk-for-go/sdk/storage/azdatalake/internal/path" ) +// EncryptionAlgorithmType defines values for EncryptionAlgorithmType. type EncryptionAlgorithmType = path.EncryptionAlgorithmType const ( @@ -23,6 +24,7 @@ const ( // response models: +// ImmutabilityPolicyMode Specifies the immutability policy mode to set on the file. type ImmutabilityPolicyMode = path.ImmutabilityPolicyMode const ( diff --git a/sdk/storage/azdatalake/file/examples_test.go b/sdk/storage/azdatalake/file/examples_test.go new file mode 100644 index 000000000000..213adfb3f173 --- /dev/null +++ b/sdk/storage/azdatalake/file/examples_test.go @@ -0,0 +1,389 @@ +//go:build go1.18 +// +build go1.18 + +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT License. See License.txt in the project root for license information. + +package file_test + +import ( + "bytes" + "context" + "crypto/md5" + "encoding/binary" + "fmt" + "github.com/Azure/azure-sdk-for-go/sdk/azcore/streaming" + "github.com/Azure/azure-sdk-for-go/sdk/azcore/to" + "github.com/Azure/azure-sdk-for-go/sdk/storage/azdatalake" + "github.com/Azure/azure-sdk-for-go/sdk/storage/azdatalake/directory" + "github.com/Azure/azure-sdk-for-go/sdk/storage/azdatalake/file" + "github.com/Azure/azure-sdk-for-go/sdk/storage/azdatalake/internal/shared" + "hash/crc64" + "io" + "log" + "os" + "strconv" + "strings" + "time" +) + +func handleError(err error) { + if err != nil { + log.Fatal(err.Error()) + } +} + +// make sure you create the filesystem before running this example +func Example_file_CreateAndDelete() { + accountName, accountKey := os.Getenv("AZURE_STORAGE_ACCOUNT_NAME"), os.Getenv("AZURE_STORAGE_ACCOUNT_KEY") + + // Create a file client + u := fmt.Sprintf("https://%s.dfs.core.windows.net/fs/file.txt", accountName) + credential, err := azdatalake.NewSharedKeyCredential(accountName, accountKey) + handleError(err) + fileClient, err := file.NewClientWithSharedKeyCredential(u, credential, nil) + handleError(err) + + _, err = fileClient.Create(context.Background(), nil) + handleError(err) + + _, err = fileClient.Delete(context.Background(), nil) + handleError(err) +} + +// This examples shows how to create a file with HTTP Headers, how to read, and how to update the file's HTTP headers. +// make sure you create the filesystem and file before running this example. +func Example_file_HTTPHeaders() { + accountName, accountKey := os.Getenv("AZURE_STORAGE_ACCOUNT_NAME"), os.Getenv("AZURE_STORAGE_ACCOUNT_KEY") + + // Create a file client + u := fmt.Sprintf("https://%s.dfs.core.windows.net/fs/file.txt", accountName) + credential, err := azdatalake.NewSharedKeyCredential(accountName, accountKey) + handleError(err) + fileClient, err := file.NewClientWithSharedKeyCredential(u, credential, nil) + handleError(err) + + // Create a directory with HTTP headers + _, err = fileClient.SetHTTPHeaders(context.TODO(), file.HTTPHeaders{ + ContentType: to.Ptr("text/html; charset=utf-8"), + ContentDisposition: to.Ptr("attachment"), + }, nil) + handleError(err) + + get, err := fileClient.GetProperties(context.TODO(), nil) + handleError(err) + + fmt.Println(get.ContentType) + fmt.Println(get.ContentDisposition) +} + +// make sure you create the filesystem before running this example +func Example_file_CreateFileWithExpiryRelativeToNow() { + accountName, accountKey := os.Getenv("AZURE_STORAGE_ACCOUNT_NAME"), os.Getenv("AZURE_STORAGE_ACCOUNT_KEY") + + // Create a file client + u := fmt.Sprintf("https://%s.dfs.core.windows.net/fs/file.txt", accountName) + credential, err := azdatalake.NewSharedKeyCredential(accountName, accountKey) + handleError(err) + fClient, err := file.NewClientWithSharedKeyCredential(u, credential, nil) + handleError(err) + + createFileOpts := &file.CreateOptions{ + Expiry: file.CreateExpiryValues{ + ExpiryType: file.CreateExpiryTypeRelativeToNow, + ExpiresOn: strconv.FormatInt((8 * time.Second).Milliseconds(), 10), + }, + } + + _, err = fClient.Create(context.Background(), createFileOpts) + handleError(err) + + resp, err := fClient.GetProperties(context.Background(), nil) + handleError(err) + fmt.Println(*resp.ExpiresOn) + + time.Sleep(time.Second * 10) + _, err = fClient.GetProperties(context.Background(), nil) + // we expect datalakeerror.PathNotFound + handleError(err) +} + +// make sure you create the filesystem before running this example +func Example_file_CreateFileWithNeverExpire() { + accountName, accountKey := os.Getenv("AZURE_STORAGE_ACCOUNT_NAME"), os.Getenv("AZURE_STORAGE_ACCOUNT_KEY") + + // Create a file client + u := fmt.Sprintf("https://%s.dfs.core.windows.net/fs/file.txt", accountName) + credential, err := azdatalake.NewSharedKeyCredential(accountName, accountKey) + handleError(err) + fClient, err := file.NewClientWithSharedKeyCredential(u, credential, nil) + handleError(err) + + createFileOpts := &file.CreateOptions{ + Expiry: file.CreateExpiryValues{ + ExpiryType: file.CreateExpiryTypeNeverExpire, + }, + } + + _, err = fClient.Create(context.Background(), createFileOpts) + handleError(err) + + resp, err := fClient.GetProperties(context.Background(), nil) + handleError(err) + // should be empty since we never expire + fmt.Println(*resp.ExpiresOn) +} + +// make sure you create the filesystem and file before running this example +func Example_file_Client_SetMetadata() { + accountName, accountKey := os.Getenv("AZURE_STORAGE_ACCOUNT_NAME"), os.Getenv("AZURE_STORAGE_ACCOUNT_KEY") + + // Create a file client + u := fmt.Sprintf("https://%s.dfs.core.windows.net/fs/file.txt", accountName) + credential, err := azdatalake.NewSharedKeyCredential(accountName, accountKey) + handleError(err) + + fileClient, err := file.NewClientWithSharedKeyCredential(u, credential, nil) + handleError(err) + + _, err = fileClient.SetMetadata(context.TODO(), map[string]*string{"author": to.Ptr("Tamer")}, nil) + handleError(err) + + // Query the directory's properties and metadata + get, err := fileClient.GetProperties(context.TODO(), nil) + handleError(err) + + // Show the directory's metadata + if get.Metadata == nil { + log.Fatal("No metadata returned") + } + + for k, v := range get.Metadata { + fmt.Print(k + "=" + *v + "\n") + } +} + +// make sure you create the filesystem before running this example +func Example_file_Rename() { + accountName, accountKey := os.Getenv("AZURE_STORAGE_ACCOUNT_NAME"), os.Getenv("AZURE_STORAGE_ACCOUNT_KEY") + + // Create a file client + u := fmt.Sprintf("https://%s.dfs.core.windows.net/fs/file.txt", accountName) + credential, err := azdatalake.NewSharedKeyCredential(accountName, accountKey) + handleError(err) + fileClient, err := file.NewClientWithSharedKeyCredential(u, credential, nil) + handleError(err) + + _, err = fileClient.Create(context.Background(), nil) + handleError(err) + + _, err = fileClient.Rename(context.Background(), "renameFile", nil) + handleError(err) +} + +// set acl on a file +// make sure you create the filesystem and file before running this example +func Example_file_SetACL() { + accountName, accountKey := os.Getenv("AZURE_STORAGE_ACCOUNT_NAME"), os.Getenv("AZURE_STORAGE_ACCOUNT_KEY") + + // Create a file client + acl := "user::rwx,group::r-x,other::rwx" + u := fmt.Sprintf("https://%s.dfs.core.windows.net/fs/file.txt", accountName) + credential, err := azdatalake.NewSharedKeyCredential(accountName, accountKey) + handleError(err) + fileClient, err := file.NewClientWithSharedKeyCredential(u, credential, nil) + handleError(err) + + _, err = fileClient.SetAccessControl(context.Background(), &file.SetAccessControlOptions{ + ACL: &acl, + }) + handleError(err) +} + +func getRelativeTimeFromAnchor(anchorTime *time.Time, amount time.Duration) time.Time { + return anchorTime.Add(amount * time.Second) +} + +// make sure you create the filesystem before running this example +func Example_file_SetAccessControlIfUnmodifiedSinceTrue() { + accountName, accountKey := os.Getenv("AZURE_STORAGE_ACCOUNT_NAME"), os.Getenv("AZURE_STORAGE_ACCOUNT_KEY") + + // Create a directory client + owner := "4cf4e284-f6a8-4540-b53e-c3469af032dc" + group := owner + acl := "user::rwx,group::r-x,other::rwx" + u := fmt.Sprintf("https://%s.dfs.core.windows.net/fs/file.txt", accountName) + credential, err := azdatalake.NewSharedKeyCredential(accountName, accountKey) + handleError(err) + fileClient, err := directory.NewClientWithSharedKeyCredential(u, credential, nil) + handleError(err) + resp, err := fileClient.Create(context.Background(), nil) + handleError(err) + + currentTime := getRelativeTimeFromAnchor(resp.Date, 10) + opts := &directory.SetAccessControlOptions{ + Owner: &owner, + Group: &group, + ACL: &acl, + AccessConditions: &directory.AccessConditions{ + ModifiedAccessConditions: &directory.ModifiedAccessConditions{ + IfUnmodifiedSince: ¤tTime, + }, + }} + + _, err = fileClient.SetAccessControl(context.Background(), opts) + handleError(err) +} + +const random64BString string = "2SDgZj6RkKYzJpu04sweQek4uWHO8ndPnYlZ0tnFS61hjnFZ5IkvIGGY44eKABov" + +func generateData(sizeInBytes int) (io.ReadSeekCloser, []byte) { + data := make([]byte, sizeInBytes) + _len := len(random64BString) + if sizeInBytes > _len { + count := sizeInBytes / _len + if sizeInBytes%_len != 0 { + count = count + 1 + } + copy(data[:], strings.Repeat(random64BString, count)) + } else { + copy(data[:], random64BString) + } + return streaming.NopCloser(bytes.NewReader(data)), data +} + +// make sure you create the filesystem before running this example +func Example_file_UploadFileAndDownloadStream() { + accountName, accountKey := os.Getenv("AZURE_STORAGE_ACCOUNT_NAME"), os.Getenv("AZURE_STORAGE_ACCOUNT_KEY") + + // Create a file client + u := fmt.Sprintf("https://%s.dfs.core.windows.net/fs/file.txt", accountName) + credential, err := azdatalake.NewSharedKeyCredential(accountName, accountKey) + handleError(err) + fClient, err := file.NewClientWithSharedKeyCredential(u, credential, nil) + handleError(err) + var fileSize int64 = 10 * 1024 + + _, err = fClient.Create(context.Background(), nil) + handleError(err) + + // create local file + _, content := generateData(int(fileSize)) + err = os.WriteFile("testFile", content, 0644) + handleError(err) + + defer func() { + err = os.Remove("testFile") + handleError(err) + }() + + fh, err := os.Open("testFile") + handleError(err) + + defer func(fh *os.File) { + err := fh.Close() + handleError(err) + }(fh) + + // get md5 hash to compare against after download + hash := md5.New() + _, err = io.Copy(hash, fh) + handleError(err) + contentMD5 := hash.Sum(nil) + + // upload the file + err = fClient.UploadFile(context.Background(), fh, &file.UploadFileOptions{ + Concurrency: 5, + ChunkSize: 2 * 1024, + }) + handleError(err) + + gResp2, err := fClient.GetProperties(context.Background(), nil) + handleError(err) + fmt.Println(*gResp2.ContentLength, fileSize) + + dResp, err := fClient.DownloadStream(context.Background(), nil) + handleError(err) + + data, err := io.ReadAll(dResp.Body) + handleError(err) + + downloadedMD5Value := md5.Sum(data) + downloadedContentMD5 := downloadedMD5Value[:] + + // compare the hashes + fmt.Println(downloadedContentMD5, contentMD5) +} + +// make sure you create the filesystem before running this example +func Example_file_UploadBufferAndDownloadStream() { + accountName, accountKey := os.Getenv("AZURE_STORAGE_ACCOUNT_NAME"), os.Getenv("AZURE_STORAGE_ACCOUNT_KEY") + + // Create a file client + u := fmt.Sprintf("https://%s.dfs.core.windows.net/fs/file.txt", accountName) + credential, err := azdatalake.NewSharedKeyCredential(accountName, accountKey) + handleError(err) + fClient, err := file.NewClientWithSharedKeyCredential(u, credential, nil) + handleError(err) + var fileSize int64 = 10 * 1024 + + _, content := generateData(int(fileSize)) + md5Value := md5.Sum(content) + contentMD5 := md5Value[:] + + err = fClient.UploadBuffer(context.Background(), content, &file.UploadBufferOptions{ + Concurrency: 5, + ChunkSize: 2 * 1024, + }) + handleError(err) + gResp2, err := fClient.GetProperties(context.Background(), nil) + handleError(err) + fmt.Println(*gResp2.ContentLength, fileSize) + + dResp, err := fClient.DownloadStream(context.Background(), nil) + handleError(err) + + data, err := io.ReadAll(dResp.Body) + handleError(err) + + downloadedMD5Value := md5.Sum(data) + downloadedContentMD5 := downloadedMD5Value[:] + + fmt.Println(downloadedContentMD5, contentMD5) +} + +// make sure you create the filesystem before running this example +func Example_file_AppendAndFlushDataWithValidation() { + accountName, accountKey := os.Getenv("AZURE_STORAGE_ACCOUNT_NAME"), os.Getenv("AZURE_STORAGE_ACCOUNT_KEY") + + // Create a file client + u := fmt.Sprintf("https://%s.dfs.core.windows.net/fs/file.txt", accountName) + credential, err := azdatalake.NewSharedKeyCredential(accountName, accountKey) + handleError(err) + fClient, err := file.NewClientWithSharedKeyCredential(u, credential, nil) + handleError(err) + + contentSize := 1024 * 8 // 8KB + content := make([]byte, contentSize) + body := bytes.NewReader(content) + rsc := streaming.NopCloser(body) + contentCRC64 := crc64.Checksum(content, shared.CRC64Table) + + // validate data using crc64 + opts := &file.AppendDataOptions{ + TransactionalValidation: file.TransferValidationTypeComputeCRC64(), + } + putResp, err := fClient.AppendData(context.Background(), 0, rsc, opts) + handleError(err) + fmt.Println(putResp.ContentCRC64) + fmt.Println(binary.LittleEndian.Uint64(putResp.ContentCRC64), contentCRC64) + + // after appending data, flush it + _, err = fClient.FlushData(context.Background(), int64(contentSize), nil) + handleError(err) + + // compare content length as well + gResp2, err := fClient.GetProperties(context.Background(), nil) + handleError(err) + fmt.Println(*gResp2.ContentLength, int64(contentSize)) +} diff --git a/sdk/storage/azdatalake/file/models.go b/sdk/storage/azdatalake/file/models.go index 928772d28e16..aa4852e7c9e8 100644 --- a/sdk/storage/azdatalake/file/models.go +++ b/sdk/storage/azdatalake/file/models.go @@ -22,19 +22,17 @@ const ( _1MiB = 1024 * 1024 CountToEnd = 0 - // MaxAppendBytes indicates the maximum number of bytes that can be updated in a call to Client.UploadRange. + // MaxAppendBytes indicates the maximum number of bytes that can be updated in a call to Client.AppendData. MaxAppendBytes = 100 * 1024 * 1024 // 100iB // MaxFileSize indicates the maximum size of the file allowed. MaxFileSize = 4 * 1024 * 1024 * 1024 * 1024 // 4 TiB ) -// CreateOptions contains the optional parameters when calling the Create operation. dfs endpoint. +// CreateOptions contains the optional parameters when calling the Create operation. type CreateOptions struct { // AccessConditions contains parameters for accessing the file. AccessConditions *AccessConditions - // Metadata is a map of name-value pairs to associate with the file storage object. - Metadata map[string]*string // CPKInfo contains a group of parameters for client provided encryption key. CPKInfo *CPKInfo // HTTPHeaders contains the HTTP headers for path operations. @@ -175,12 +173,17 @@ type UploadFileOptions = uploadFromReaderOptions // FlushDataOptions contains the optional parameters for the Client.FlushData method. type FlushDataOptions struct { + // AccessConditions contains parameters for accessing the file. AccessConditions *AccessConditions - // HTTPHeaders contains the optional path HTTP headers to set when the file is created. // CPKInfo contains optional parameters to perform encryption using customer-provided key. - CPKInfo *CPKInfo - HTTPHeaders *HTTPHeaders - Close *bool + CPKInfo *CPKInfo + // HTTPHeaders contains the HTTP headers for path operations. + HTTPHeaders *HTTPHeaders + // Close This event has a property indicating whether this is the final change to distinguish the + // difference between an intermediate flush to a file stream and the final close of a file stream. + Close *bool + // RetainUncommittedData if "true", uncommitted data is retained after the flush operation + // completes, otherwise, the uncommitted data is deleted after the flush operation. RetainUncommittedData *bool } @@ -231,7 +234,7 @@ func (o *FlushDataOptions) format(offset int64) (*generated.PathClientFlushDataO return flushDataOpts, modifiedAccessConditions, leaseAccessConditions, httpHeaderOpts, cpkInfoOpts, nil } -// AppendDataOptions contains the optional parameters for the Client.UploadRange method. +// AppendDataOptions contains the optional parameters for the Client.AppendData method. type AppendDataOptions struct { // TransactionalValidation specifies the transfer validation type to use. // The default is nil (no transfer validation). @@ -342,18 +345,19 @@ func (u *UploadStreamOptions) getFlushDataOptions() *FlushDataOptions { } } -// DownloadStreamOptions contains the optional parameters for the Client.Download method. +// DownloadStreamOptions contains the optional parameters for the Client.DownloadStream method. type DownloadStreamOptions struct { // When set to true and specified together with the Range, the service returns the MD5 hash for the range, as long as the // range is less than or equal to 4 MB in size. RangeGetContentMD5 *bool - // Range specifies a range of bytes. The default value is all bytes. Range *HTTPRange - + // AccessConditions contains parameters for accessing the file. AccessConditions *AccessConditions - CPKInfo *CPKInfo - CPKScopeInfo *CPKScopeInfo + // CPKInfo contains optional parameters to perform encryption using customer-provided key. + CPKInfo *CPKInfo + // CPKScopeInfo contains a group of parameters for client provided encryption scope. + CPKScopeInfo *CPKScopeInfo } func (o *DownloadStreamOptions) format() *blob.DownloadStreamOptions { @@ -383,26 +387,19 @@ func (o *DownloadStreamOptions) format() *blob.DownloadStreamOptions { type DownloadBufferOptions struct { // Range specifies a range of bytes. The default value is all bytes. Range *HTTPRange - - // ChunkSize specifies the block size to use for each parallel download; the default size is DefaultDownloadBlockSize. + // ChunkSize specifies the chunk size to use for each parallel download; the default size is 4MB. ChunkSize int64 - // Progress is a function that is invoked periodically as bytes are received. Progress func(bytesTransferred int64) - - // BlobAccessConditions indicates the access conditions used when making HTTP GET requests against the blob. + // AccessConditions indicates the access conditions used when making HTTP GET requests against the file. AccessConditions *AccessConditions - // CPKInfo contains a group of parameters for client provided encryption key. CPKInfo *CPKInfo - // CPKScopeInfo contains a group of parameters for client provided encryption scope. CPKScopeInfo *CPKScopeInfo - - // Concurrency indicates the maximum number of blocks to download in parallel (0=default). + // Concurrency indicates the maximum number of chunks to download in parallel (0=default). Concurrency uint16 - - // RetryReaderOptionsPerChunk is used when downloading each block. + // RetryReaderOptionsPerChunk is used when downloading each chunk. RetryReaderOptionsPerChunk *RetryReaderOptions } @@ -444,24 +441,19 @@ func (o *DownloadBufferOptions) format() *blob.DownloadBufferOptions { type DownloadFileOptions struct { // Range specifies a range of bytes. The default value is all bytes. Range *HTTPRange - - // ChunkSize specifies the block size to use for each parallel download; the default size is DefaultDownloadBlockSize. + // ChunkSize specifies the chunk size to use for each parallel download; the default size is 4MB. ChunkSize int64 - // Progress is a function that is invoked periodically as bytes are received. Progress func(bytesTransferred int64) - - // BlobAccessConditions indicates the access conditions used when making HTTP GET requests against the blob. + // AccessConditions indicates the access conditions used when making HTTP GET requests against the file. AccessConditions *AccessConditions - - // ClientProvidedKeyOptions indicates the client provided key by name and/or by value to encrypt/decrypt data. - CPKInfo *CPKInfo + // CPKInfo contains a group of parameters for client provided encryption key. + CPKInfo *CPKInfo + // CPKScopeInfo contains a group of parameters for client provided encryption scope. CPKScopeInfo *CPKScopeInfo - - // Concurrency indicates the maximum number of blocks to download in parallel. The default value is 5. + // Concurrency indicates the maximum number of chunks to download in parallel. The default value is 5. Concurrency uint16 - - // RetryReaderOptionsPerChunk is used when downloading each block. + // RetryReaderOptionsPerChunk is used when downloading each chunk. RetryReaderOptionsPerChunk *RetryReaderOptions } @@ -522,11 +514,14 @@ type SetExpiryOptions struct { // placeholder for future options } +// HTTPRange defines a range of bytes within an HTTP resource, starting at offset and +// ending at offset+count. A zero-value HTTPRange indicates the entire resource. An HTTPRange +// which has an offset but no zero value count indicates from the offset to the resource's end. type HTTPRange = exported.HTTPRange // ================================= path imports ================================== -// DeleteOptions contains the optional parameters when calling the Delete operation. dfs endpoint +// DeleteOptions contains the optional parameters when calling the Delete operation. type DeleteOptions = path.DeleteOptions // RenameOptions contains the optional parameters when calling the Rename operation. @@ -535,13 +530,13 @@ type RenameOptions = path.RenameOptions // GetPropertiesOptions contains the optional parameters for the Client.GetProperties method type GetPropertiesOptions = path.GetPropertiesOptions -// SetAccessControlOptions contains the optional parameters when calling the SetAccessControl operation. dfs endpoint +// SetAccessControlOptions contains the optional parameters when calling the SetAccessControl operation. type SetAccessControlOptions = path.SetAccessControlOptions // GetAccessControlOptions contains the optional parameters when calling the GetAccessControl operation. type GetAccessControlOptions = path.GetAccessControlOptions -// CPKInfo contains a group of parameters for the PathClient.Download method. +// CPKInfo contains CPK related information. type CPKInfo = path.CPKInfo // GetSASURLOptions contains the optional parameters for the Client.GetSASURL method. @@ -559,10 +554,10 @@ type SetMetadataOptions = path.SetMetadataOptions // SharedKeyCredential contains an account's name and its primary or secondary key. type SharedKeyCredential = path.SharedKeyCredential -// AccessConditions identifies blob-specific access conditions which you optionally set. +// AccessConditions identifies file-specific access conditions which you optionally set. type AccessConditions = path.AccessConditions -// SourceAccessConditions identifies blob-specific access conditions which you optionally set. +// SourceAccessConditions identifies file-specific source access conditions which you optionally set. type SourceAccessConditions = path.SourceAccessConditions // LeaseAccessConditions contains optional parameters to access leased entity. diff --git a/sdk/storage/azdatalake/file/responses.go b/sdk/storage/azdatalake/file/responses.go index 801c9d03b0c0..f72747584ab7 100644 --- a/sdk/storage/azdatalake/file/responses.go +++ b/sdk/storage/azdatalake/file/responses.go @@ -26,15 +26,13 @@ type AppendDataResponse = generated.PathClientAppendDataResponse // FlushDataResponse contains the response from method Client.FlushData. type FlushDataResponse = generated.PathClientFlushDataResponse -// RenameResponse contains the response fields for the Create operation. -type RenameResponse struct { - Response CreateResponse - NewFileClient *Client -} +// RenameResponse contains the response fields for the Rename operation. +type RenameResponse = path.RenameResponse // DownloadStreamResponse contains the response from the DownloadStream method. // To read from the stream, read from the Body field, or call the NewRetryReader method. type DownloadStreamResponse struct { + // DownloadResponse contains response fields from DownloadStream. DownloadResponse client *Client getInfo httpGetterInfo diff --git a/sdk/storage/azdatalake/file/retry_reader.go b/sdk/storage/azdatalake/file/retry_reader.go index 66e3f35edf0d..139278e54b85 100644 --- a/sdk/storage/azdatalake/file/retry_reader.go +++ b/sdk/storage/azdatalake/file/retry_reader.go @@ -21,6 +21,7 @@ type httpGetter func(ctx context.Context, i httpGetterInfo) (io.ReadCloser, erro // HTTPGetterInfo is passed to an HTTPGetter function passing it parameters // that should be used to make an HTTP GET request. type httpGetterInfo struct { + // HTTPRange contains range Range *HTTPRange // ETag specifies the resource's etag that should be used when creating diff --git a/sdk/storage/azdatalake/file/retry_reader_test.go b/sdk/storage/azdatalake/file/retry_reader_test.go new file mode 100644 index 000000000000..61013cd1de46 --- /dev/null +++ b/sdk/storage/azdatalake/file/retry_reader_test.go @@ -0,0 +1,433 @@ +//go:build go1.18 +// +build go1.18 + +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT License. See License.txt in the project root for license information. + +package file + +import ( + "context" + "crypto/rand" + "errors" + "fmt" + "io" + "net" + "net/http" + "testing" + "time" + + "github.com/stretchr/testify/require" +) + +// Testings for RetryReader +// This reader return one byte through each Read call +type perByteReader struct { + RandomBytes []byte // Random generated bytes + + byteCount int // Bytes can be returned before EOF + currentByteIndex int // Bytes that have already been returned. + doInjectError bool + doInjectErrorByteIndex int + doInjectTimes int + injectedError error + + // sleepDuration and closeChannel are only use in "forced cancellation" tests + sleepDuration time.Duration + closeChannel chan struct{} +} + +func newPerByteReader(byteCount int) *perByteReader { + perByteReader := perByteReader{ + byteCount: byteCount, + closeChannel: nil, + } + + perByteReader.RandomBytes = make([]byte, byteCount) + _, _ = rand.Read(perByteReader.RandomBytes) + + return &perByteReader +} + +func newSingleUsePerByteReader(contents []byte) *perByteReader { + perByteReader := perByteReader{ + byteCount: len(contents), + closeChannel: make(chan struct{}, 10), + } + + perByteReader.RandomBytes = contents + + return &perByteReader +} + +func (r *perByteReader) Read(b []byte) (n int, err error) { + if r.doInjectError && r.doInjectErrorByteIndex == r.currentByteIndex && r.doInjectTimes > 0 { + r.doInjectTimes-- + return 0, r.injectedError + } + + if r.currentByteIndex < r.byteCount { + n = copy(b, r.RandomBytes[r.currentByteIndex:r.currentByteIndex+1]) + r.currentByteIndex += n + + // simulate a delay, which may be successful or, if we're closed from another go-routine, may return an + // error + select { + case <-r.closeChannel: + return n, errors.New(ReadOnClosedBodyMessage) + case <-time.After(r.sleepDuration): + return n, nil + } + } + + return 0, io.EOF +} + +func (r *perByteReader) Close() error { + if r.closeChannel != nil { + r.closeChannel <- struct{}{} + } + return nil +} + +// Test normal retry succeed, note initial response not provided. +// Tests both with and without notification of failures +func TestRetryReaderReadWithRetry(t *testing.T) { + // Test twice, the second time using the optional "logging"/notification callback for failed tries + // We must test both with and without the callback, since be testing without + // we are testing that it is, indeed, optional to provide the callback + for _, logThisRun := range []bool{false, true} { + + // Extra setup for testing notification of failures (i.e. of unsuccessful tries) + failureMethodNumCalls := 0 + failureWillRetryCount := 0 + failureLastReportedFailureCount := int32(-1) + var failureLastReportedError error = nil + failureMethod := func(failureCount int32, lastError error, rnge HTTPRange, willRetry bool) { + failureMethodNumCalls++ + if willRetry { + failureWillRetryCount++ + } + failureLastReportedFailureCount = failureCount + failureLastReportedError = lastError + } + + // Main test setup + byteCount := 1 + body := newPerByteReader(byteCount) + body.doInjectError = true + body.doInjectErrorByteIndex = 0 + body.doInjectTimes = 1 + body.injectedError = &net.DNSError{IsTemporary: true} + + getter := func(ctx context.Context, info httpGetterInfo) (io.ReadCloser, error) { + r := http.Response{} + body.currentByteIndex = int(info.Range.Offset) + r.Body = body + + return r.Body, nil + } + + httpGetterInfo := httpGetterInfo{ + Range: &HTTPRange{ + Count: int64(byteCount), + }, + } + initResponse, err := getter(context.Background(), httpGetterInfo) + require.NoError(t, err) + + rrOptions := RetryReaderOptions{MaxRetries: 1} + if logThisRun { + rrOptions.OnFailedRead = failureMethod + } + retryReader := newRetryReader(context.Background(), initResponse, httpGetterInfo, getter, rrOptions) + + // should fail and succeed through retry + can := make([]byte, 1) + n, err := retryReader.Read(can) + require.Equal(t, n, 1) + require.NoError(t, err) + + // check "logging", if it was enabled + if logThisRun { + // We only expect one failed try in this test + // And the notification method is not called for successes + require.Equal(t, failureMethodNumCalls, 1) // this is the number of calls we counted + require.Equal(t, failureWillRetryCount, 1) // the sole failure was retried + require.Equal(t, failureLastReportedFailureCount, int32(1)) // this is the number of failures reported by the notification method + require.Error(t, failureLastReportedError) + } + // should return EOF + n, err = retryReader.Read(can) + require.Equal(t, n, 0) + require.Equal(t, err, io.EOF) + } +} + +// Test normal retry succeed, note initial response not provided. +// Tests both with and without notification of failures +func TestRetryReaderWithRetryIoUnexpectedEOF(t *testing.T) { + // Test twice, the second time using the optional "logging"/notification callback for failed tries + // We must test both with and without the callback, since be testing without + // we are testing that it is, indeed, optional to provide the callback + for _, logThisRun := range []bool{false, true} { + + // Extra setup for testing notification of failures (i.e. of unsuccessful tries) + failureMethodNumCalls := 0 + failureWillRetryCount := 0 + failureLastReportedFailureCount := int32(-1) + var failureLastReportedError error = nil + failureMethod := func(failureCount int32, lastError error, rnge HTTPRange, willRetry bool) { + failureMethodNumCalls++ + if willRetry { + failureWillRetryCount++ + } + failureLastReportedFailureCount = failureCount + failureLastReportedError = lastError + } + + // Main test setup + byteCount := 1 + body := newPerByteReader(byteCount) + body.doInjectError = true + body.doInjectErrorByteIndex = 0 + body.doInjectTimes = 1 + body.injectedError = io.ErrUnexpectedEOF + + getter := func(ctx context.Context, info httpGetterInfo) (io.ReadCloser, error) { + r := http.Response{} + body.currentByteIndex = int(info.Range.Offset) + r.Body = body + + return r.Body, nil + } + + httpGetterInfo := httpGetterInfo{ + Range: &HTTPRange{ + Count: int64(byteCount), + }, + } + initResponse, err := getter(context.Background(), httpGetterInfo) + require.NoError(t, err) + + rrOptions := RetryReaderOptions{MaxRetries: 1} + if logThisRun { + rrOptions.OnFailedRead = failureMethod + } + retryReader := newRetryReader(context.Background(), initResponse, httpGetterInfo, getter, rrOptions) + + // should fail and succeed through retry + can := make([]byte, 1) + n, err := retryReader.Read(can) + require.Equal(t, n, 1) + require.NoError(t, err) + + // check "logging", if it was enabled + if logThisRun { + // We only expect one failed try in this test + // And the notification method is not called for successes + require.Equal(t, failureMethodNumCalls, 1) // this is the number of calls we counted + require.Equal(t, failureWillRetryCount, 1) // the sole failure was retried + require.Equal(t, failureLastReportedFailureCount, int32(1)) // this is the number of failures reported by the notification method + require.Error(t, failureLastReportedError) + } + // should return EOF + n, err = retryReader.Read(can) + require.Equal(t, n, 0) + require.Equal(t, err, io.EOF) + } +} + +// Test normal retry fail as retry Count not enough. +func TestRetryReaderReadNegativeNormalFail(t *testing.T) { + // Extra setup for testing notification of failures (i.e. of unsuccessful tries) + failureMethodNumCalls := 0 + failureWillRetryCount := 0 + failureLastReportedFailureCount := int32(-1) + var failureLastReportedError error = nil + failureMethod := func(failureCount int32, lastError error, rnge HTTPRange, willRetry bool) { + failureMethodNumCalls++ + if willRetry { + failureWillRetryCount++ + } + failureLastReportedFailureCount = failureCount + failureLastReportedError = lastError + } + + // Main test setup + byteCount := 1 + body := newPerByteReader(byteCount) + body.doInjectError = true + body.doInjectErrorByteIndex = 0 + body.doInjectTimes = 2 + body.injectedError = &net.DNSError{IsTemporary: true} + + startResponse := body + + getter := func(ctx context.Context, info httpGetterInfo) (io.ReadCloser, error) { + r := http.Response{} + body.currentByteIndex = int(info.Range.Offset) + r.Body = body + + return r.Body, nil + } + + rrOptions := RetryReaderOptions{ + MaxRetries: 1, + OnFailedRead: failureMethod, + } + httpGetterInfo := httpGetterInfo{ + Range: &HTTPRange{ + Count: int64(byteCount), + }, + } + retryReader := newRetryReader(context.Background(), startResponse, httpGetterInfo, getter, rrOptions) + + // should fail + can := make([]byte, 1) + n, err := retryReader.Read(can) + require.Equal(t, n, 0) + require.Equal(t, err, body.injectedError) + + // Check that we received the right notification callbacks + // We only expect two failed tries in this test, but only one + // of the would have had willRetry = true + require.Equal(t, failureMethodNumCalls, 2) // this is the number of calls we counted + require.Equal(t, failureWillRetryCount, 1) // only the first failure was retried + require.Equal(t, failureLastReportedFailureCount, int32(2)) // this is the number of failures reported by the notification method + require.Error(t, failureLastReportedError) +} + +// Test boundary case when Count equals to 0 and fail. +func TestRetryReaderReadCount0(t *testing.T) { + byteCount := 1 + body := newPerByteReader(byteCount) + body.doInjectError = true + body.doInjectErrorByteIndex = 1 + body.doInjectTimes = 1 + body.injectedError = &net.DNSError{IsTemporary: true} + + startResponseBody := body + + getter := func(ctx context.Context, info httpGetterInfo) (io.ReadCloser, error) { + r := http.Response{} + body.currentByteIndex = int(info.Range.Offset) + r.Body = body + + return r.Body, nil + } + + httpGetterInfo := httpGetterInfo{ + Range: &HTTPRange{ + Count: int64(byteCount), + }, + } + + retryReader := newRetryReader(context.Background(), startResponseBody, httpGetterInfo, getter, RetryReaderOptions{MaxRetries: 1}) + + // should consume the only byte + can := make([]byte, 1) + n, err := retryReader.Read(can) + require.Equal(t, n, 1) + require.NoError(t, err) + + // should not read when Count=0, and should return EOF + n, err = retryReader.Read(can) + require.Equal(t, n, 0) + require.Equal(t, err, io.EOF) +} + +func TestRetryReaderReadNegativeNonRetriableError(t *testing.T) { + byteCount := 1 + body := newPerByteReader(byteCount) + body.doInjectError = true + body.doInjectErrorByteIndex = 0 + body.doInjectTimes = 1 + body.injectedError = fmt.Errorf("not retriable error") + + startResponseBody := body + + getter := func(ctx context.Context, info httpGetterInfo) (io.ReadCloser, error) { + r := http.Response{} + body.currentByteIndex = int(info.Range.Offset) + r.Body = body + + return r.Body, nil + } + + httpGetterInfo := httpGetterInfo{ + Range: &HTTPRange{ + Count: int64(byteCount), + }, + } + + retryReader := newRetryReader(context.Background(), startResponseBody, httpGetterInfo, getter, RetryReaderOptions{MaxRetries: 2}) + + dest := make([]byte, 1) + _, err := retryReader.Read(dest) + require.Equal(t, err, body.injectedError) +} + +// Test the case where we programmatically force a retry to happen, via closing the body early from another goroutine +// Unlike the retries orchestrated elsewhere in this test file, which simulate network failures for the +// purposes of unit testing, here we are testing the cancellation mechanism that is exposed to +// consumers of the API, to allow programmatic forcing of retries (e.g. if the consumer deems +// the read to be taking too long, they may force a retry in the hope of better performance next time). +func TestRetryReaderReadWithForcedRetry(t *testing.T) { + for _, enableRetryOnEarlyClose := range []bool{false, true} { + + // use the notification callback, so we know that the retry really did happen + failureMethodNumCalls := 0 + failureMethod := func(failureCount int32, lastError error, rnge HTTPRange, willRetry bool) { + failureMethodNumCalls++ + } + + // Main test setup + byteCount := 10 // so multiple passes through read loop will be required + sleepDuration := 100 * time.Millisecond + randBytes := make([]byte, byteCount) + _, _ = rand.Read(randBytes) + getter := func(ctx context.Context, info httpGetterInfo) (io.ReadCloser, error) { + body := newSingleUsePerByteReader(randBytes) // make new one every time, since we force closes in this test, and it is unusable after a close + body.sleepDuration = sleepDuration + r := http.Response{} + body.currentByteIndex = int(info.Range.Offset) + r.Body = body + + return r.Body, nil + } + + httpGetterInfo := httpGetterInfo{ + Range: &HTTPRange{ + Count: int64(byteCount), + }, + } + initResponse, err := getter(context.Background(), httpGetterInfo) + require.NoError(t, err) + + rrOptions := RetryReaderOptions{MaxRetries: 2, EarlyCloseAsError: !enableRetryOnEarlyClose} + rrOptions.OnFailedRead = failureMethod + retryReader := newRetryReader(context.Background(), initResponse, httpGetterInfo, getter, rrOptions) + + // set up timed cancellation from separate goroutine + go func() { + time.Sleep(sleepDuration * 5) + err := retryReader.Close() + if err != nil { + return + } + }() + + // do the read (should fail, due to forced cancellation, and succeed through retry) + output := make([]byte, byteCount) + n, err := io.ReadFull(retryReader, output) + if enableRetryOnEarlyClose { + require.Equal(t, n, byteCount) + require.NoError(t, err) + require.EqualValues(t, output, randBytes) + require.Equal(t, failureMethodNumCalls, 1) // assert that the cancellation did indeed happen + } else { + require.Error(t, err) + } + } +} diff --git a/sdk/storage/azdatalake/filesystem/client.go b/sdk/storage/azdatalake/filesystem/client.go index b42803edb8cf..40dc0768aaad 100644 --- a/sdk/storage/azdatalake/filesystem/client.go +++ b/sdk/storage/azdatalake/filesystem/client.go @@ -47,7 +47,7 @@ func NewClient(filesystemURL string, cred azcore.TokenCredential, options *Clien } base.SetPipelineOptions((*base.ClientOptions)(conOptions), &plOpts) - azClient, err := azcore.NewClient(shared.FilesystemClient, exported.ModuleVersion, plOpts, &conOptions.ClientOptions) + azClient, err := azcore.NewClient(shared.FileSystemClient, exported.ModuleVersion, plOpts, &conOptions.ClientOptions) if err != nil { return nil, err } @@ -59,7 +59,7 @@ func NewClient(filesystemURL string, cred azcore.TokenCredential, options *Clien ClientOptions: options.ClientOptions, } blobContainerClient, _ := container.NewClient(containerURL, cred, &containerClientOpts) - fsClient := base.NewFilesystemClient(filesystemURL, containerURL, blobContainerClient, azClient, nil, &cred, (*base.ClientOptions)(conOptions)) + fsClient := base.NewFileSystemClient(filesystemURL, containerURL, blobContainerClient, azClient, nil, &cred, (*base.ClientOptions)(conOptions)) return (*Client)(fsClient), nil } @@ -74,7 +74,7 @@ func NewClientWithNoCredential(filesystemURL string, options *ClientOptions) (*C plOpts := runtime.PipelineOptions{} base.SetPipelineOptions((*base.ClientOptions)(conOptions), &plOpts) - azClient, err := azcore.NewClient(shared.FilesystemClient, exported.ModuleVersion, plOpts, &conOptions.ClientOptions) + azClient, err := azcore.NewClient(shared.FileSystemClient, exported.ModuleVersion, plOpts, &conOptions.ClientOptions) if err != nil { return nil, err } @@ -86,7 +86,7 @@ func NewClientWithNoCredential(filesystemURL string, options *ClientOptions) (*C ClientOptions: options.ClientOptions, } blobContainerClient, _ := container.NewClientWithNoCredential(containerURL, &containerClientOpts) - fsClient := base.NewFilesystemClient(filesystemURL, containerURL, blobContainerClient, azClient, nil, nil, (*base.ClientOptions)(conOptions)) + fsClient := base.NewFileSystemClient(filesystemURL, containerURL, blobContainerClient, azClient, nil, nil, (*base.ClientOptions)(conOptions)) return (*Client)(fsClient), nil } @@ -104,7 +104,7 @@ func NewClientWithSharedKeyCredential(filesystemURL string, cred *SharedKeyCrede } base.SetPipelineOptions((*base.ClientOptions)(conOptions), &plOpts) - azClient, err := azcore.NewClient(shared.FilesystemClient, exported.ModuleVersion, plOpts, &conOptions.ClientOptions) + azClient, err := azcore.NewClient(shared.FileSystemClient, exported.ModuleVersion, plOpts, &conOptions.ClientOptions) if err != nil { return nil, err } @@ -115,12 +115,12 @@ func NewClientWithSharedKeyCredential(filesystemURL string, cred *SharedKeyCrede containerClientOpts := container.ClientOptions{ ClientOptions: options.ClientOptions, } - blobSharedKey, err := cred.ConvertToBlobSharedKey() + blobSharedKey, err := exported.ConvertToBlobSharedKey(cred) if err != nil { return nil, err } blobContainerClient, _ := container.NewClientWithSharedKeyCredential(containerURL, blobSharedKey, &containerClientOpts) - fsClient := base.NewFilesystemClient(filesystemURL, containerURL, blobContainerClient, azClient, cred, nil, (*base.ClientOptions)(conOptions)) + fsClient := base.NewFileSystemClient(filesystemURL, containerURL, blobContainerClient, azClient, cred, nil, (*base.ClientOptions)(conOptions)) return (*Client)(fsClient), nil } @@ -201,7 +201,7 @@ func (fs *Client) NewFileClient(filePath string) *file.Client { return (*file.Client)(base.NewPathClient(fileURL, blobURL, fs.containerClient().NewBlockBlobClient(filePath), fs.generatedFSClientWithDFS().InternalClient().WithClientName(shared.FileClient), fs.sharedKey(), fs.identityCredential(), fs.getClientOptions())) } -// Create creates a new filesystem under the specified account. (blob3). +// Create creates a new filesystem under the specified account. func (fs *Client) Create(ctx context.Context, options *CreateOptions) (CreateResponse, error) { opts := options.format() resp, err := fs.containerClient().Create(ctx, opts) @@ -209,7 +209,7 @@ func (fs *Client) Create(ctx context.Context, options *CreateOptions) (CreateRes return resp, err } -// Delete deletes the specified filesystem and any files or directories it contains. (blob3). +// Delete deletes the specified filesystem and any files or directories it contains. func (fs *Client) Delete(ctx context.Context, options *DeleteOptions) (DeleteResponse, error) { opts := options.format() resp, err := fs.containerClient().Delete(ctx, opts) @@ -217,17 +217,17 @@ func (fs *Client) Delete(ctx context.Context, options *DeleteOptions) (DeleteRes return resp, err } -// GetProperties returns all user-defined metadata, standard HTTP properties, and system properties for the filesystem. (blob3). +// GetProperties returns all user-defined metadata, standard HTTP properties, and system properties for the filesystem. func (fs *Client) GetProperties(ctx context.Context, options *GetPropertiesOptions) (GetPropertiesResponse, error) { opts := options.format() newResp := GetPropertiesResponse{} resp, err := fs.containerClient().GetProperties(ctx, opts) - formatFilesystemProperties(&newResp, &resp) + formatFileSystemProperties(&newResp, &resp) err = exported.ConvertToDFSError(err) return newResp, err } -// SetMetadata sets one or more user-defined name-value pairs for the specified filesystem. (blob3). +// SetMetadata sets one or more user-defined name-value pairs for the specified filesystem. func (fs *Client) SetMetadata(ctx context.Context, options *SetMetadataOptions) (SetMetadataResponse, error) { opts := options.format() resp, err := fs.containerClient().SetMetadata(ctx, opts) @@ -235,7 +235,7 @@ func (fs *Client) SetMetadata(ctx context.Context, options *SetMetadataOptions) return resp, err } -// SetAccessPolicy sets the permissions for the specified filesystem or the files and directories under it. (blob3). +// SetAccessPolicy sets the permissions for the specified filesystem or the files and directories under it. func (fs *Client) SetAccessPolicy(ctx context.Context, options *SetAccessPolicyOptions) (SetAccessPolicyResponse, error) { opts := options.format() resp, err := fs.containerClient().SetAccessPolicy(ctx, opts) @@ -243,7 +243,7 @@ func (fs *Client) SetAccessPolicy(ctx context.Context, options *SetAccessPolicyO return resp, err } -// GetAccessPolicy returns the permissions for the specified filesystem or the files and directories under it. (blob3). +// GetAccessPolicy returns the permissions for the specified filesystem or the files and directories under it. func (fs *Client) GetAccessPolicy(ctx context.Context, options *GetAccessPolicyOptions) (GetAccessPolicyResponse, error) { opts := options.format() newResp := GetAccessPolicyResponse{} @@ -255,7 +255,7 @@ func (fs *Client) GetAccessPolicy(ctx context.Context, options *GetAccessPolicyO // TODO: implement undelete path in fs client as well -// NewListPathsPager operation returns a pager of the shares under the specified account. (dfs1) +// NewListPathsPager operation returns a pager of the shares under the specified account. // For more information, see https://learn.microsoft.com/en-us/rest/api/storageservices/list-shares func (fs *Client) NewListPathsPager(recursive bool, options *ListPathsOptions) *runtime.Pager[ListPathsSegmentResponse] { listOptions := options.format() @@ -291,7 +291,7 @@ func (fs *Client) NewListPathsPager(recursive bool, options *ListPathsOptions) * }) } -// NewListDeletedPathsPager operation returns a pager of the shares under the specified account. (dfs op/blob2). +// NewListDeletedPathsPager operation returns a pager of the shares under the specified account. // For more information, see https://learn.microsoft.com/en-us/rest/api/storageservices/list-shares func (fs *Client) NewListDeletedPathsPager(options *ListDeletedPathsOptions) *runtime.Pager[ListDeletedPathsSegmentResponse] { listOptions := options.format() @@ -327,9 +327,9 @@ func (fs *Client) NewListDeletedPathsPager(options *ListDeletedPathsOptions) *ru }) } -// GetSASURL is a convenience method for generating a SAS token for the currently pointed at container. +// GetSASURL is a convenience method for generating a SAS token for the currently pointed at filesystem. // It can only be used if the credential supplied during creation was a SharedKeyCredential. -func (fs *Client) GetSASURL(permissions sas.FilesystemPermissions, expiry time.Time, o *GetSASURLOptions) (string, error) { +func (fs *Client) GetSASURL(permissions sas.FileSystemPermissions, expiry time.Time, o *GetSASURLOptions) (string, error) { if fs.sharedKey() == nil { return "", datalakeerror.MissingSharedKeyCredential } @@ -342,7 +342,7 @@ func (fs *Client) GetSASURL(permissions sas.FilesystemPermissions, expiry time.T qps, err := sas.DatalakeSignatureValues{ Version: sas.Version, Protocol: sas.ProtocolHTTPS, - FilesystemName: urlParts.FilesystemName, + FileSystemName: urlParts.FileSystemName, Permissions: permissions.String(), StartTime: st, ExpiryTime: expiry.UTC(), diff --git a/sdk/storage/azdatalake/filesystem/client_test.go b/sdk/storage/azdatalake/filesystem/client_test.go index 3cf5cae5ae0a..571bbce8055e 100644 --- a/sdk/storage/azdatalake/filesystem/client_test.go +++ b/sdk/storage/azdatalake/filesystem/client_test.go @@ -24,6 +24,8 @@ import ( "github.com/stretchr/testify/suite" ) +var proposedLeaseIDs = []*string{to.Ptr("c820a799-76d7-4ee2-6e15-546f19325c2c"), to.Ptr("326cc5e1-746e-4af8-4811-a50e6629a8ca")} + func Test(t *testing.T) { recordMode := recording.GetRecordMode() t.Logf("Running datalake Tests in %s mode\n", recordMode) @@ -61,21 +63,21 @@ type UnrecordedTestSuite struct { suite.Suite } -func validateFilesystemDeleted(_require *require.Assertions, filesystemClient *filesystem.Client) { +func validateFileSystemDeleted(_require *require.Assertions, filesystemClient *filesystem.Client) { _, err := filesystemClient.GetAccessPolicy(context.Background(), nil) _require.NotNil(err) - testcommon.ValidateErrorCode(_require, err, datalakeerror.FilesystemNotFound) + testcommon.ValidateErrorCode(_require, err, datalakeerror.FileSystemNotFound) } func (s *RecordedTestSuite) TestCreateFilesystem() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) @@ -85,17 +87,17 @@ func (s *RecordedTestSuite) TestCreateFilesystemWithOptions() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) + filesystemName := testcommon.GenerateFileSystemName(testName) testStr := "hello" metadata := map[string]*string{"foo": &testStr, "bar": &testStr} - access := filesystem.Filesystem + access := filesystem.FileSystem opts := filesystem.CreateOptions{ Metadata: metadata, Access: &access, } - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) _, err = fsClient.Create(context.Background(), &opts) _require.Nil(err) @@ -103,14 +105,14 @@ func (s *RecordedTestSuite) TestCreateFilesystemWithOptions() { props, err := fsClient.GetProperties(context.Background(), nil) _require.NoError(err) _require.NotNil(props.Metadata) - _require.Equal(*props.PublicAccess, filesystem.Filesystem) + _require.Equal(*props.PublicAccess, filesystem.FileSystem) } func (s *RecordedTestSuite) TestCreateFilesystemWithFileAccess() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) + filesystemName := testcommon.GenerateFileSystemName(testName) testStr := "hello" metadata := map[string]*string{"foo": &testStr, "bar": &testStr} access := filesystem.File @@ -118,9 +120,9 @@ func (s *RecordedTestSuite) TestCreateFilesystemWithFileAccess() { Metadata: metadata, Access: &access, } - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) _, err = fsClient.Create(context.Background(), &opts) _require.Nil(err) @@ -134,16 +136,16 @@ func (s *RecordedTestSuite) TestCreateFilesystemEmptyMetadata() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) + filesystemName := testcommon.GenerateFileSystemName(testName) metadata := map[string]*string{"foo": nil, "bar": nil} - access := filesystem.Filesystem + access := filesystem.FileSystem opts := filesystem.CreateOptions{ Metadata: metadata, Access: &access, } - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) _, err = fsClient.Create(context.Background(), &opts) _require.Nil(err) @@ -151,14 +153,14 @@ func (s *RecordedTestSuite) TestCreateFilesystemEmptyMetadata() { props, err := fsClient.GetProperties(context.Background(), nil) _require.NoError(err) _require.Nil(props.Metadata) - _require.Equal(*props.PublicAccess, filesystem.Filesystem) + _require.Equal(*props.PublicAccess, filesystem.FileSystem) } func (s *RecordedTestSuite) TestFilesystemCreateInvalidName() { _require := require.New(s.T()) - fsClient, err := testcommon.GetFilesystemClient("foo bar", s.T(), testcommon.TestAccountDatalake, nil) + fsClient, err := testcommon.GetFileSystemClient("foo bar", s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) _, err = fsClient.Create(context.Background(), nil) @@ -170,27 +172,27 @@ func (s *RecordedTestSuite) TestFilesystemCreateNameCollision() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) _, err = fsClient.Create(context.Background(), nil) _require.NotNil(err) - testcommon.ValidateErrorCode(_require, err, datalakeerror.FilesystemAlreadyExists) + testcommon.ValidateErrorCode(_require, err, datalakeerror.FileSystemAlreadyExists) } func (s *RecordedTestSuite) TestFilesystemGetProperties() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) @@ -201,12 +203,61 @@ func (s *RecordedTestSuite) TestFilesystemGetProperties() { _require.Nil(resp.Metadata) } +func (s *RecordedTestSuite) TestFilesystemGetPropertiesWithEmptyOpts() { + _require := require.New(s.T()) + testName := s.T().Name() + + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + _require.NoError(err) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) + + _, err = fsClient.Create(context.Background(), nil) + _require.Nil(err) + + opts := &filesystem.GetPropertiesOptions{} + resp, err := fsClient.GetProperties(context.Background(), opts) + _require.Nil(err) + _require.NotNil(resp.ETag) + _require.Nil(resp.Metadata) +} + +func (s *RecordedTestSuite) TestFilesystemGetPropertiesWithLease() { + _require := require.New(s.T()) + testName := s.T().Name() + + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + _require.NoError(err) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) + + _, err = fsClient.Create(context.Background(), nil) + _require.Nil(err) + + fsLeaseClient, err := lease.NewFileSystemClient(fsClient, &lease.FileSystemClientOptions{LeaseID: proposedLeaseIDs[0]}) + _require.Nil(err) + _, err = fsLeaseClient.AcquireLease(context.Background(), int32(60), nil) + _require.Nil(err) + + opts := &filesystem.GetPropertiesOptions{LeaseAccessConditions: &filesystem.LeaseAccessConditions{ + LeaseID: fsLeaseClient.LeaseID(), + }} + + resp, err := fsClient.GetProperties(context.Background(), opts) + _require.Nil(err) + _require.NotNil(resp.ETag) + _require.Nil(resp.Metadata) + + _, err = fsLeaseClient.ReleaseLease(context.Background(), nil) + _require.Nil(err) +} + func (s *RecordedTestSuite) TestFilesystemDelete() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) _, err = fsClient.Create(context.Background(), nil) @@ -215,29 +266,29 @@ func (s *RecordedTestSuite) TestFilesystemDelete() { _, err = fsClient.Delete(context.Background(), nil) _require.Nil(err) - validateFilesystemDeleted(_require, fsClient) + validateFileSystemDeleted(_require, fsClient) } func (s *RecordedTestSuite) TestFilesystemDeleteNonExistent() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) _, err = fsClient.Delete(context.Background(), nil) _require.NotNil(err) - testcommon.ValidateErrorCode(_require, err, datalakeerror.FilesystemNotFound) + testcommon.ValidateErrorCode(_require, err, datalakeerror.FileSystemNotFound) } func (s *RecordedTestSuite) TestFilesystemDeleteIfModifiedSinceTrue() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) resp, err := fsClient.Create(context.Background(), nil) @@ -245,40 +296,40 @@ func (s *RecordedTestSuite) TestFilesystemDeleteIfModifiedSinceTrue() { currentTime := testcommon.GetRelativeTimeFromAnchor(resp.Date, -10) - deleteFilesystemOptions := filesystem.DeleteOptions{ + deleteFileSystemOptions := filesystem.DeleteOptions{ AccessConditions: &filesystem.AccessConditions{ ModifiedAccessConditions: &filesystem.ModifiedAccessConditions{ IfModifiedSince: ¤tTime, }, }, } - _, err = fsClient.Delete(context.Background(), &deleteFilesystemOptions) + _, err = fsClient.Delete(context.Background(), &deleteFileSystemOptions) _require.Nil(err) - validateFilesystemDeleted(_require, fsClient) + validateFileSystemDeleted(_require, fsClient) } func (s *RecordedTestSuite) TestFilesystemDeleteIfModifiedSinceFalse() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) resp, err := fsClient.Create(context.Background(), nil) _require.Nil(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) currentTime := testcommon.GetRelativeTimeFromAnchor(resp.Date, 10) - deleteFilesystemOptions := filesystem.DeleteOptions{ + deleteFileSystemOptions := filesystem.DeleteOptions{ AccessConditions: &filesystem.AccessConditions{ ModifiedAccessConditions: &filesystem.ModifiedAccessConditions{ IfModifiedSince: ¤tTime, }, }, } - _, err = fsClient.Delete(context.Background(), &deleteFilesystemOptions) + _, err = fsClient.Delete(context.Background(), &deleteFileSystemOptions) _require.NotNil(err) testcommon.ValidateErrorCode(_require, err, datalakeerror.ConditionNotMet) } @@ -287,8 +338,8 @@ func (s *RecordedTestSuite) TestFilesystemDeleteIfUnModifiedSinceTrue() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) resp, err := fsClient.Create(context.Background(), nil) @@ -296,41 +347,41 @@ func (s *RecordedTestSuite) TestFilesystemDeleteIfUnModifiedSinceTrue() { currentTime := testcommon.GetRelativeTimeFromAnchor(resp.Date, 10) - deleteFilesystemOptions := filesystem.DeleteOptions{ + deleteFileSystemOptions := filesystem.DeleteOptions{ AccessConditions: &filesystem.AccessConditions{ ModifiedAccessConditions: &filesystem.ModifiedAccessConditions{ IfUnmodifiedSince: ¤tTime, }, }, } - _, err = fsClient.Delete(context.Background(), &deleteFilesystemOptions) + _, err = fsClient.Delete(context.Background(), &deleteFileSystemOptions) _require.Nil(err) - validateFilesystemDeleted(_require, fsClient) + validateFileSystemDeleted(_require, fsClient) } func (s *RecordedTestSuite) TestFilesystemDeleteIfUnModifiedSinceFalse() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) resp, err := fsClient.Create(context.Background(), nil) _require.Nil(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) currentTime := testcommon.GetRelativeTimeFromAnchor(resp.Date, -10) - deleteFilesystemOptions := filesystem.DeleteOptions{ + deleteFileSystemOptions := filesystem.DeleteOptions{ AccessConditions: &filesystem.AccessConditions{ ModifiedAccessConditions: &filesystem.ModifiedAccessConditions{ IfUnmodifiedSince: ¤tTime, }, }, } - _, err = fsClient.Delete(context.Background(), &deleteFilesystemOptions) + _, err = fsClient.Delete(context.Background(), &deleteFileSystemOptions) _require.NotNil(err) testcommon.ValidateErrorCode(_require, err, datalakeerror.ConditionNotMet) @@ -340,13 +391,13 @@ func (s *RecordedTestSuite) TestFilesystemSetMetadataNonEmpty() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) opts := filesystem.SetMetadataOptions{ Metadata: testcommon.BasicMetadata, @@ -366,13 +417,13 @@ func (s *RecordedTestSuite) TestFilesystemSetMetadataEmpty() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) opts := filesystem.SetMetadataOptions{ Metadata: map[string]*string{}, @@ -390,13 +441,13 @@ func (s *RecordedTestSuite) TestFilesystemSetMetadataNil() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) _, err = fsClient.SetMetadata(context.Background(), nil) _require.Nil(err) @@ -410,13 +461,13 @@ func (s *RecordedTestSuite) TestFilesystemSetMetadataInvalidField() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) opts := filesystem.SetMetadataOptions{ Metadata: map[string]*string{"!nval!d Field!@#%": to.Ptr("value")}, @@ -430,27 +481,27 @@ func (s *RecordedTestSuite) TestFilesystemSetMetadataNonExistent() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) _, err = fsClient.SetMetadata(context.Background(), nil) _require.NotNil(err) - testcommon.ValidateErrorCode(_require, err, datalakeerror.FilesystemNotFound) + testcommon.ValidateErrorCode(_require, err, datalakeerror.FileSystemNotFound) } func (s *RecordedTestSuite) TestFilesystemSetEmptyAccessPolicy() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) _, err = fsClient.SetAccessPolicy(context.Background(), &filesystem.SetAccessPolicyOptions{}) _require.Nil(err) @@ -460,13 +511,13 @@ func (s *RecordedTestSuite) TestFilesystemSetNilAccessPolicy() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) _, err = fsClient.SetAccessPolicy(context.Background(), nil) _require.Nil(err) @@ -476,13 +527,13 @@ func (s *RecordedTestSuite) TestFilesystemSetAccessPolicy() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) start := time.Date(2020, 1, 1, 0, 0, 0, 0, time.UTC) expiration := time.Date(2024, 1, 1, 0, 0, 0, 0, time.UTC) @@ -499,7 +550,7 @@ func (s *RecordedTestSuite) TestFilesystemSetAccessPolicy() { }, ID: &id, }) - options := filesystem.SetAccessPolicyOptions{FilesystemACL: signedIdentifiers} + options := filesystem.SetAccessPolicyOptions{FileSystemACL: signedIdentifiers} _, err = fsClient.SetAccessPolicy(context.Background(), &options) _require.Nil(err) } @@ -508,13 +559,13 @@ func (s *RecordedTestSuite) TestFilesystemSetMultipleAccessPolicies() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) id := "empty" @@ -546,7 +597,7 @@ func (s *RecordedTestSuite) TestFilesystemSetMultipleAccessPolicies() { Permission: &permission3, }, }) - options := filesystem.SetAccessPolicyOptions{FilesystemACL: signedIdentifiers} + options := filesystem.SetAccessPolicyOptions{FileSystemACL: signedIdentifiers} _, err = fsClient.SetAccessPolicy(context.Background(), &options) _require.Nil(err) @@ -560,13 +611,13 @@ func (s *RecordedTestSuite) TestFilesystemSetNullAccessPolicy() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) id := "null" @@ -574,7 +625,7 @@ func (s *RecordedTestSuite) TestFilesystemSetNullAccessPolicy() { signedIdentifiers = append(signedIdentifiers, &filesystem.SignedIdentifier{ ID: &id, }) - options := filesystem.SetAccessPolicyOptions{FilesystemACL: signedIdentifiers} + options := filesystem.SetAccessPolicyOptions{FileSystemACL: signedIdentifiers} _, err = fsClient.SetAccessPolicy(context.Background(), &options) _require.Nil(err) @@ -587,13 +638,13 @@ func (s *RecordedTestSuite) TestFilesystemGetAccessPolicyWithEmptyOpts() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) id := "null" @@ -601,7 +652,7 @@ func (s *RecordedTestSuite) TestFilesystemGetAccessPolicyWithEmptyOpts() { signedIdentifiers = append(signedIdentifiers, &filesystem.SignedIdentifier{ ID: &id, }) - options := filesystem.SetAccessPolicyOptions{FilesystemACL: signedIdentifiers} + options := filesystem.SetAccessPolicyOptions{FileSystemACL: signedIdentifiers} _, err = fsClient.SetAccessPolicy(context.Background(), &options) _require.Nil(err) @@ -616,8 +667,8 @@ func (s *RecordedTestSuite) TestFilesystemGetAccessPolicyOnLeasedFilesystem() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) _, err = fsClient.Create(context.Background(), nil) @@ -625,7 +676,7 @@ func (s *RecordedTestSuite) TestFilesystemGetAccessPolicyOnLeasedFilesystem() { var proposedLeaseIDs = []*string{to.Ptr("c820a799-76d7-4ee2-6e15-546f19325c2c"), to.Ptr("326cc5e1-746e-4af8-4811-a50e6629a8ca")} - fsLeaseClient, err := lease.NewFilesystemClient(fsClient, &lease.FilesystemClientOptions{ + fsLeaseClient, err := lease.NewFileSystemClient(fsClient, &lease.FileSystemClientOptions{ LeaseID: proposedLeaseIDs[0], }) _require.Nil(err) @@ -636,7 +687,7 @@ func (s *RecordedTestSuite) TestFilesystemGetAccessPolicyOnLeasedFilesystem() { signedIdentifiers = append(signedIdentifiers, &filesystem.SignedIdentifier{ ID: &id, }) - options := filesystem.SetAccessPolicyOptions{FilesystemACL: signedIdentifiers} + options := filesystem.SetAccessPolicyOptions{FileSystemACL: signedIdentifiers} _, err = fsClient.SetAccessPolicy(context.Background(), &options) _require.Nil(err) @@ -668,13 +719,13 @@ func (s *RecordedTestSuite) TestFilesystemGetSetPermissionsMultiplePolicies() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) // Define the policies start, err := time.Parse(time.UnixDate, "Fri Jun 11 20:00:00 UTC 2021") @@ -700,7 +751,7 @@ func (s *RecordedTestSuite) TestFilesystemGetSetPermissionsMultiplePolicies() { }, }, } - options := filesystem.SetAccessPolicyOptions{FilesystemACL: permissions} + options := filesystem.SetAccessPolicyOptions{FileSystemACL: permissions} _, err = fsClient.SetAccessPolicy(context.Background(), &options) _require.Nil(err) @@ -714,8 +765,8 @@ func (s *RecordedTestSuite) TestFilesystemGetPermissionsPublicAccessNotNone() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) access := filesystem.File @@ -724,7 +775,7 @@ func (s *RecordedTestSuite) TestFilesystemGetPermissionsPublicAccessNotNone() { } _, err = fsClient.Create(context.Background(), &createContainerOptions) // We create the container explicitly so we can be sure the access policy is not empty _require.Nil(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) resp, err := fsClient.GetAccessPolicy(context.Background(), nil) @@ -738,13 +789,13 @@ func (s *RecordedTestSuite) TestFilesystemSetPermissionsPublicAccessTypeFile() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) setAccessPolicyOptions := filesystem.SetAccessPolicyOptions{ Access: to.Ptr(filesystem.File), } @@ -760,36 +811,36 @@ func (s *RecordedTestSuite) TestFilesystemSetPermissionsPublicAccessFilesystem() _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) setAccessPolicyOptions := filesystem.SetAccessPolicyOptions{ - Access: to.Ptr(filesystem.Filesystem), + Access: to.Ptr(filesystem.FileSystem), } _, err = fsClient.SetAccessPolicy(context.Background(), &setAccessPolicyOptions) _require.Nil(err) resp, err := fsClient.GetAccessPolicy(context.Background(), nil) _require.Nil(err) - _require.Equal(*resp.PublicAccess, filesystem.Filesystem) + _require.Equal(*resp.PublicAccess, filesystem.FileSystem) } func (s *RecordedTestSuite) TestFilesystemSetPermissionsACLMoreThanFive() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) start, err := time.Parse(time.UnixDate, "Fri Jun 11 20:00:00 UTC 2021") _require.Nil(err) @@ -813,7 +864,7 @@ func (s *RecordedTestSuite) TestFilesystemSetPermissionsACLMoreThanFive() { setAccessPolicyOptions := filesystem.SetAccessPolicyOptions{ Access: &access, } - setAccessPolicyOptions.FilesystemACL = permissions + setAccessPolicyOptions.FileSystemACL = permissions _, err = fsClient.SetAccessPolicy(context.Background(), &setAccessPolicyOptions) _require.NotNil(err) @@ -824,13 +875,13 @@ func (s *RecordedTestSuite) TestFilesystemSetPermissionsDeleteAndModifyACL() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) start, err := time.Parse(time.UnixDate, "Fri Jun 11 20:00:00 UTC 2021") _require.Nil(err) @@ -854,7 +905,7 @@ func (s *RecordedTestSuite) TestFilesystemSetPermissionsDeleteAndModifyACL() { setAccessPolicyOptions := filesystem.SetAccessPolicyOptions{ Access: &access, } - setAccessPolicyOptions.FilesystemACL = permissions + setAccessPolicyOptions.FileSystemACL = permissions _, err = fsClient.SetAccessPolicy(context.Background(), &setAccessPolicyOptions) _require.Nil(err) @@ -868,7 +919,7 @@ func (s *RecordedTestSuite) TestFilesystemSetPermissionsDeleteAndModifyACL() { setAccessPolicyOptions1 := filesystem.SetAccessPolicyOptions{ Access: &access, } - setAccessPolicyOptions1.FilesystemACL = permissions + setAccessPolicyOptions1.FileSystemACL = permissions _, err = fsClient.SetAccessPolicy(context.Background(), &setAccessPolicyOptions1) _require.Nil(err) @@ -882,13 +933,13 @@ func (s *RecordedTestSuite) TestFilesystemSetPermissionsDeleteAllPolicies() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) start, err := time.Parse(time.UnixDate, "Fri Jun 11 20:00:00 UTC 2021") _require.Nil(err) @@ -911,7 +962,7 @@ func (s *RecordedTestSuite) TestFilesystemSetPermissionsDeleteAllPolicies() { setAccessPolicyOptions := filesystem.SetAccessPolicyOptions{ Access: to.Ptr(filesystem.File), } - setAccessPolicyOptions.FilesystemACL = permissions + setAccessPolicyOptions.FileSystemACL = permissions _, err = fsClient.SetAccessPolicy(context.Background(), &setAccessPolicyOptions) _require.Nil(err) @@ -923,7 +974,7 @@ func (s *RecordedTestSuite) TestFilesystemSetPermissionsDeleteAllPolicies() { setAccessPolicyOptions = filesystem.SetAccessPolicyOptions{ Access: to.Ptr(filesystem.File), } - setAccessPolicyOptions.FilesystemACL = []*filesystem.SignedIdentifier{} + setAccessPolicyOptions.FileSystemACL = []*filesystem.SignedIdentifier{} _, err = fsClient.SetAccessPolicy(context.Background(), &setAccessPolicyOptions) _require.Nil(err) @@ -936,13 +987,13 @@ func (s *RecordedTestSuite) TestFilesystemSetPermissionsInvalidPolicyTimes() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) // Swap start and expiry expiry, err := time.Parse(time.UnixDate, "Fri Jun 11 20:00:00 UTC 2021") @@ -966,7 +1017,7 @@ func (s *RecordedTestSuite) TestFilesystemSetPermissionsInvalidPolicyTimes() { setAccessPolicyOptions := filesystem.SetAccessPolicyOptions{ Access: to.Ptr(filesystem.File), } - setAccessPolicyOptions.FilesystemACL = permissions + setAccessPolicyOptions.FileSystemACL = permissions _, err = fsClient.SetAccessPolicy(context.Background(), &setAccessPolicyOptions) _require.Nil(err) } @@ -975,13 +1026,13 @@ func (s *RecordedTestSuite) TestFilesystemSetPermissionsNilPolicySlice() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) _, err = fsClient.SetAccessPolicy(context.Background(), nil) _require.Nil(err) @@ -991,13 +1042,13 @@ func (s *RecordedTestSuite) TestFilesystemSetPermissionsSignedIdentifierTooLong( _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) id := "" for i := 0; i < 65; i++ { @@ -1022,7 +1073,7 @@ func (s *RecordedTestSuite) TestFilesystemSetPermissionsSignedIdentifierTooLong( setAccessPolicyOptions := filesystem.SetAccessPolicyOptions{ Access: to.Ptr(filesystem.File), } - setAccessPolicyOptions.FilesystemACL = permissions + setAccessPolicyOptions.FileSystemACL = permissions _, err = fsClient.SetAccessPolicy(context.Background(), &setAccessPolicyOptions) _require.NotNil(err) @@ -1033,13 +1084,13 @@ func (s *RecordedTestSuite) TestFilesystemSetPermissionsIfModifiedSinceTrue() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) resp, err := fsClient.Create(context.Background(), nil) _require.Nil(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) currentTime := testcommon.GetRelativeTimeFromAnchor(resp.Date, -10) @@ -1060,13 +1111,13 @@ func (s *RecordedTestSuite) TestFilesystemSetPermissionsIfModifiedSinceFalse() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) resp, err := fsClient.Create(context.Background(), nil) _require.Nil(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) currentTime := testcommon.GetRelativeTimeFromAnchor(resp.Date, 10) @@ -1085,13 +1136,13 @@ func (s *RecordedTestSuite) TestFilesystemSetPermissionsIfUnModifiedSinceTrue() _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) resp, err := fsClient.Create(context.Background(), nil) _require.Nil(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) currentTime := testcommon.GetRelativeTimeFromAnchor(resp.Date, 10) @@ -1112,13 +1163,13 @@ func (s *RecordedTestSuite) TestFilesystemSetPermissionsIfUnModifiedSinceFalse() _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) resp, err := fsClient.Create(context.Background(), nil) _require.Nil(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) currentTime := testcommon.GetRelativeTimeFromAnchor(resp.Date, -10) @@ -1137,13 +1188,13 @@ func (s *UnrecordedTestSuite) TestFilesystemSetAccessPoliciesInDifferentTimeForm _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) id := "timeInEST" permission := "rw" @@ -1187,7 +1238,7 @@ func (s *UnrecordedTestSuite) TestFilesystemSetAccessPoliciesInDifferentTimeForm Permission: &permission3, }, }) - options := filesystem.SetAccessPolicyOptions{FilesystemACL: signedIdentifiers} + options := filesystem.SetAccessPolicyOptions{FileSystemACL: signedIdentifiers} _, err = fsClient.SetAccessPolicy(context.Background(), &options) _require.Nil(err) @@ -1202,13 +1253,13 @@ func (s *RecordedTestSuite) TestFilesystemSetAccessPolicyWithNullId() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) signedIdentifiers := make([]*filesystem.SignedIdentifier, 0) signedIdentifiers = append(signedIdentifiers, &filesystem.SignedIdentifier{ @@ -1217,7 +1268,7 @@ func (s *RecordedTestSuite) TestFilesystemSetAccessPolicyWithNullId() { }, }) - options := filesystem.SetAccessPolicyOptions{FilesystemACL: signedIdentifiers} + options := filesystem.SetAccessPolicyOptions{FileSystemACL: signedIdentifiers} _, err = fsClient.SetAccessPolicy(context.Background(), &options) _require.NotNil(err) testcommon.ValidateErrorCode(_require, err, datalakeerror.InvalidXMLDocument) @@ -1227,20 +1278,20 @@ func (s *RecordedTestSuite) TestFilesystemSetAccessPolicyWithNullId() { _require.Len(resp1.SignedIdentifiers, 0) } -func (s *UnrecordedTestSuite) TestSASFilesystemClient() { +func (s *UnrecordedTestSuite) TestSASFileSystemClient() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) // Adding SAS and options - permissions := sas.FilesystemPermissions{ + permissions := sas.FileSystemPermissions{ Read: true, Add: true, Write: true, @@ -1262,10 +1313,10 @@ func (s *RecordedTestSuite) TestFilesystemListPathsWithRecursive() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) @@ -1288,6 +1339,45 @@ func (s *RecordedTestSuite) TestFilesystemListPathsWithRecursive() { resp, err := pager.NextPage(context.Background()) _require.Nil(err) _require.Equal(5, len(resp.Paths)) + _require.NotNil(resp.PathList.Paths[0].IsDirectory) + + if err != nil { + break + } + } +} + +func (s *RecordedTestSuite) TestFilesystemListPathsWithRecursiveNoPrefix() { + _require := require.New(s.T()) + testName := s.T().Name() + + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + _require.NoError(err) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) + + _, err = fsClient.Create(context.Background(), nil) + _require.Nil(err) + + client := fsClient.NewFileClient("file1") + _, err = client.Create(context.Background(), nil) + _require.Nil(err) + client = fsClient.NewFileClient("file2") + _, err = client.Create(context.Background(), nil) + _require.Nil(err) + dirClient := fsClient.NewDirectoryClient("dir1") + _, err = dirClient.Create(context.Background(), nil) + _require.Nil(err) + dirClient = fsClient.NewDirectoryClient("dir2") + _, err = dirClient.Create(context.Background(), nil) + _require.Nil(err) + + pager := fsClient.NewListPathsPager(true, nil) + for pager.More() { + resp, err := pager.NextPage(context.Background()) + _require.Nil(err) + _require.Equal(4, len(resp.Paths)) + _require.NotNil(resp.PathList.Paths[0].IsDirectory) if err != nil { break } @@ -1298,10 +1388,10 @@ func (s *RecordedTestSuite) TestFilesystemListPathsWithoutRecursive() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) @@ -1334,10 +1424,10 @@ func (s *RecordedTestSuite) TestFilesystemListPathsWithMaxResults() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) @@ -1376,10 +1466,10 @@ func (s *RecordedTestSuite) TestFilesystemListPathsWithPrefix() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) @@ -1415,10 +1505,10 @@ func (s *RecordedTestSuite) TestFilesystemListPathsWithContinuation() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) @@ -1460,10 +1550,10 @@ func (s *RecordedTestSuite) TestFilesystemListDeletedPaths() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) @@ -1498,10 +1588,10 @@ func (s *RecordedTestSuite) TestFilesystemListDeletedPathsWithMaxResults() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) @@ -1548,10 +1638,10 @@ func (s *RecordedTestSuite) TestFilesystemListDeletedPathsWithPrefix() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) @@ -1595,10 +1685,10 @@ func (s *RecordedTestSuite) TestFilesystemListDeletedPathsWithContinuation() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) @@ -1644,20 +1734,20 @@ func (s *RecordedTestSuite) TestFilesystemListDeletedPathsWithContinuation() { _require.Equal("", *resp.NextMarker) } -func (s *UnrecordedTestSuite) TestSASFilesystemCreateAndDeleteFile() { +func (s *UnrecordedTestSuite) TestSASFileSystemCreateAndDeleteFile() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) // Adding SAS and options - permissions := sas.FilesystemPermissions{ + permissions := sas.FileSystemPermissions{ Read: true, Add: true, Write: true, @@ -1682,20 +1772,20 @@ func (s *UnrecordedTestSuite) TestSASFilesystemCreateAndDeleteFile() { _require.Nil(err) } -func (s *UnrecordedTestSuite) TestSASFilesystemCreateAndDeleteDirectory() { +func (s *UnrecordedTestSuite) TestSASFileSystemCreateAndDeleteDirectory() { _require := require.New(s.T()) testName := s.T().Name() - filesystemName := testcommon.GenerateFilesystemName(testName) - fsClient, err := testcommon.GetFilesystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) + filesystemName := testcommon.GenerateFileSystemName(testName) + fsClient, err := testcommon.GetFileSystemClient(filesystemName, s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) // Adding SAS and options - permissions := sas.FilesystemPermissions{ + permissions := sas.FileSystemPermissions{ Read: true, Add: true, Write: true, diff --git a/sdk/storage/azdatalake/filesystem/constants.go b/sdk/storage/azdatalake/filesystem/constants.go index 1b2916076c46..037d182790e0 100644 --- a/sdk/storage/azdatalake/filesystem/constants.go +++ b/sdk/storage/azdatalake/filesystem/constants.go @@ -16,7 +16,7 @@ type PublicAccessType = azblob.PublicAccessType const ( File PublicAccessType = azblob.PublicAccessTypeBlob - Filesystem PublicAccessType = azblob.PublicAccessTypeContainer + FileSystem PublicAccessType = azblob.PublicAccessTypeContainer ) // StatusType defines values for StatusType diff --git a/sdk/storage/azdatalake/filesystem/examples_test.go b/sdk/storage/azdatalake/filesystem/examples_test.go new file mode 100644 index 000000000000..ad8e7b293c0f --- /dev/null +++ b/sdk/storage/azdatalake/filesystem/examples_test.go @@ -0,0 +1,330 @@ +//go:build go1.18 +// +build go1.18 + +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT License. See License.txt in the project root for license information. + +package filesystem_test + +import ( + "bytes" + "context" + "fmt" + "io" + "log" + "net/http" + "os" + "strings" + "time" + + "github.com/Azure/azure-sdk-for-go/sdk/azcore/streaming" + "github.com/Azure/azure-sdk-for-go/sdk/azcore/to" + "github.com/Azure/azure-sdk-for-go/sdk/azidentity" + "github.com/Azure/azure-sdk-for-go/sdk/storage/azdatalake" + "github.com/Azure/azure-sdk-for-go/sdk/storage/azdatalake/filesystem" + "github.com/Azure/azure-sdk-for-go/sdk/storage/azdatalake/sas" +) + +func handleError(err error) { + if err != nil { + log.Fatal(err.Error()) + } +} + +func Example_fs_NewClient() { + accountName, ok := os.LookupEnv("AZURE_STORAGE_ACCOUNT_NAME") + if !ok { + panic("AZURE_STORAGE_ACCOUNT_NAME could not be found") + } + fsName := "testfs" + fsURL := fmt.Sprintf("https://%s.dfs.core.windows.net/%s", accountName, fsName) + + cred, err := azidentity.NewDefaultAzureCredential(nil) + handleError(err) + + fsClient, err := filesystem.NewClient(fsURL, cred, nil) + handleError(err) + fmt.Println(fsClient.DFSURL()) +} + +func Example_fs_NewClientWithSharedKeyCredential() { + accountName, ok := os.LookupEnv("AZURE_STORAGE_ACCOUNT_NAME") + if !ok { + panic("AZURE_STORAGE_ACCOUNT_NAME could not be found") + } + accountKey, ok := os.LookupEnv("AZURE_STORAGE_ACCOUNT_KEY") + if !ok { + panic("AZURE_STORAGE_ACCOUNT_KEY could not be found") + } + fsName := "testfs" + fsURL := fmt.Sprintf("https://%s.dfs.core.windows.net/%s", accountName, fsName) + + cred, err := azdatalake.NewSharedKeyCredential(accountName, accountKey) + handleError(err) + fsClient, err := filesystem.NewClientWithSharedKeyCredential(fsURL, cred, nil) + handleError(err) + fmt.Println(fsClient.DFSURL()) +} + +func Example_fs_NewClientWithNoCredential() { + accountName, ok := os.LookupEnv("AZURE_STORAGE_ACCOUNT_NAME") + if !ok { + panic("AZURE_STORAGE_ACCOUNT_NAME could not be found") + } + sharedAccessSignature, ok := os.LookupEnv("AZURE_STORAGE_SHARED_ACCESS_SIGNATURE") + if !ok { + panic("AZURE_STORAGE_SHARED_ACCESS_SIGNATURE could not be found") + } + fsName := "testfs" + fsURL := fmt.Sprintf("https://%s.dfs.core.windows.net/%s?%s", accountName, fsName, sharedAccessSignature) + + fsClient, err := filesystem.NewClientWithNoCredential(fsURL, nil) + handleError(err) + fmt.Println(fsClient.DFSURL()) +} + +func Example_fs_NewClientFromConnectionString() { + // Your connection string can be obtained from the Azure Portal. + connectionString, ok := os.LookupEnv("AZURE_STORAGE_CONNECTION_STRING") + if !ok { + log.Fatal("the environment variable 'AZURE_STORAGE_CONNECTION_STRING' could not be found") + } + fsName := "testfs" + fsClient, err := filesystem.NewClientFromConnectionString(connectionString, fsName, nil) + handleError(err) + fmt.Println(fsClient.DFSURL()) +} + +func Example_fs_ClientNewFileClient() { + accountName, ok := os.LookupEnv("AZURE_STORAGE_ACCOUNT_NAME") + if !ok { + panic("AZURE_STORAGE_ACCOUNT_NAME could not be found") + } + fsName := "testfs" + fsURL := fmt.Sprintf("https://%s.dfs.core.windows.net/%s", accountName, fsName) + + cred, err := azidentity.NewDefaultAzureCredential(nil) + handleError(err) + + fsClient, err := filesystem.NewClient(fsURL, cred, nil) + handleError(err) + + fileClient := fsClient.NewFileClient("test_File") + handleError(err) + fmt.Println(fileClient.DFSURL()) +} + +func Example_fs_ClientCreate() { + accountName, ok := os.LookupEnv("AZURE_STORAGE_ACCOUNT_NAME") + if !ok { + panic("AZURE_STORAGE_ACCOUNT_NAME could not be found") + } + fsName := "testfs" + fsURL := fmt.Sprintf("https://%s.dfs.core.windows.net/%s", accountName, fsName) + + cred, err := azidentity.NewDefaultAzureCredential(nil) + handleError(err) + + fsClient, err := filesystem.NewClient(fsURL, cred, nil) + handleError(err) + + fsCreateResponse, err := fsClient.Create(context.TODO(), &filesystem.CreateOptions{ + Metadata: map[string]*string{"Foo": to.Ptr("Bar")}, + }) + handleError(err) + fmt.Println(fsCreateResponse) +} + +func Example_fs_ClientDelete() { + accountName, ok := os.LookupEnv("AZURE_STORAGE_ACCOUNT_NAME") + if !ok { + panic("AZURE_STORAGE_ACCOUNT_NAME could not be found") + } + fsName := "testfs" + fsURL := fmt.Sprintf("https://%s.dfs.core.windows.net/%s", accountName, fsName) + + cred, err := azidentity.NewDefaultAzureCredential(nil) + handleError(err) + + fsClient, err := filesystem.NewClient(fsURL, cred, nil) + handleError(err) + + fsDeleteResponse, err := fsClient.Delete(context.TODO(), nil) + handleError(err) + fmt.Println(fsDeleteResponse) +} + +func Example_fs_ClientListPaths() { + accountName, ok := os.LookupEnv("AZURE_STORAGE_ACCOUNT_NAME") + if !ok { + panic("AZURE_STORAGE_ACCOUNT_NAME could not be found") + } + fsName := "testfs" + fsURL := fmt.Sprintf("https://%s.dfs.core.windows.net/%s", accountName, fsName) + + cred, err := azidentity.NewDefaultAzureCredential(nil) + handleError(err) + + fsClient, err := filesystem.NewClient(fsURL, cred, nil) + handleError(err) + + pager := fsClient.NewListPathsPager(true, nil) + + for pager.More() { + resp, err := pager.NextPage(context.TODO()) + if err != nil { + log.Fatal(err) + } + for _, path := range resp.Paths { + fmt.Println(*path.Name) + } + } +} + +func Example_fs_ClientListDeletedPaths() { + accountName, ok := os.LookupEnv("AZURE_STORAGE_ACCOUNT_NAME") + if !ok { + panic("AZURE_STORAGE_ACCOUNT_NAME could not be found") + } + fsName := "testfs" + fsURL := fmt.Sprintf("https://%s.dfs.core.windows.net/%s", accountName, fsName) + + cred, err := azidentity.NewDefaultAzureCredential(nil) + handleError(err) + + fsClient, err := filesystem.NewClient(fsURL, cred, nil) + handleError(err) + + pager := fsClient.NewListDeletedPathsPager(nil) + + for pager.More() { + resp, err := pager.NextPage(context.TODO()) + if err != nil { + log.Fatal(err) + } + for _, path := range resp.Segment.PathItems { + fmt.Println(*path.Name) + } + } +} + +func Example_fs_ClientGetSASURL() { + accountName, ok := os.LookupEnv("AZURE_STORAGE_ACCOUNT_NAME") + if !ok { + panic("AZURE_STORAGE_ACCOUNT_NAME could not be found") + } + fsName := "testfs" + fsURL := fmt.Sprintf("https://%s.dfs.core.windows.net/%s", accountName, fsName) + + cred, err := azidentity.NewDefaultAzureCredential(nil) + handleError(err) + + fsClient, err := filesystem.NewClient(fsURL, cred, nil) + handleError(err) + + permission := sas.FileSystemPermissions{Read: true} + start := time.Now() + expiry := start.AddDate(1, 0, 0) + options := filesystem.GetSASURLOptions{StartTime: &start} + sasURL, err := fsClient.GetSASURL(permission, expiry, &options) + handleError(err) + _ = sasURL +} + +// This example shows how to manipulate a fs's permissions. +func Example_fs_ClientSetAccessPolicy() { + accountName, ok := os.LookupEnv("AZURE_STORAGE_ACCOUNT_NAME") + if !ok { + panic("AZURE_STORAGE_ACCOUNT_NAME could not be found") + } + fsName := "testfs" + fsURL := fmt.Sprintf("https://%s.dfs.core.windows.net/%s", accountName, fsName) + + cred, err := azidentity.NewDefaultAzureCredential(nil) + handleError(err) + + fsClient, err := filesystem.NewClient(fsURL, cred, nil) + handleError(err) + + // Create the fs + _, err = fsClient.Create(context.TODO(), nil) + handleError(err) + + // Upload a simple File. + fileClient := fsClient.NewFileClient("HelloWorld.txt") + handleError(err) + + err = fileClient.UploadStream(context.TODO(), streaming.NopCloser(strings.NewReader("Hello World!")), nil) + handleError(err) + + // Attempt to read the File + get, err := http.Get(fileClient.DFSURL()) + handleError(err) + if get.StatusCode == http.StatusNotFound { + // ChangeLease the File to be public access File + _, err := fsClient.SetAccessPolicy( + context.TODO(), + &filesystem.SetAccessPolicyOptions{ + Access: to.Ptr(filesystem.File), + }, + ) + if err != nil { + log.Fatal(err) + } + + // Now, this works + get, err = http.Get(fileClient.DFSURL()) + if err != nil { + log.Fatal(err) + } + var text bytes.Buffer + _, err = text.ReadFrom(get.Body) + if err != nil { + return + } + defer func(Body io.ReadCloser) { + _ = Body.Close() + }(get.Body) + + fmt.Println("Public access File data: ", text.String()) + } +} + +func Example_fs_ClientSetMetadata() { + accountName, ok := os.LookupEnv("AZURE_STORAGE_ACCOUNT_NAME") + if !ok { + panic("AZURE_STORAGE_ACCOUNT_NAME could not be found") + } + fsName := "testfs" + fsURL := fmt.Sprintf("https://%s.dfs.core.windows.net/%s", accountName, fsName) + + cred, err := azidentity.NewDefaultAzureCredential(nil) + handleError(err) + + fsClient, err := filesystem.NewClient(fsURL, cred, nil) + handleError(err) + + // Create a fs with some metadata, key names are converted to lowercase before being sent to the service. + // You should always use lowercase letters, especially when querying a map for a metadata key. + creatingApp, err := os.Executable() + handleError(err) + _, err = fsClient.Create(context.TODO(), &filesystem.CreateOptions{Metadata: map[string]*string{"author": to.Ptr("azFile"), "app": to.Ptr(creatingApp)}}) + handleError(err) + + // Query the fs's metadata + fsGetPropertiesResponse, err := fsClient.GetProperties(context.TODO(), nil) + handleError(err) + + if fsGetPropertiesResponse.Metadata == nil { + log.Fatal("metadata is empty!") + } + + for k, v := range fsGetPropertiesResponse.Metadata { + fmt.Printf("%s=%s\n", k, *v) + } + + // Update the metadata and write it back to the fs + fsGetPropertiesResponse.Metadata["author"] = to.Ptr("Mohit") + _, err = fsClient.SetMetadata(context.TODO(), &filesystem.SetMetadataOptions{Metadata: fsGetPropertiesResponse.Metadata}) + handleError(err) +} diff --git a/sdk/storage/azdatalake/filesystem/models.go b/sdk/storage/azdatalake/filesystem/models.go index 50944097d146..05c395fc20c3 100644 --- a/sdk/storage/azdatalake/filesystem/models.go +++ b/sdk/storage/azdatalake/filesystem/models.go @@ -13,13 +13,15 @@ import ( "time" ) -// SetAccessPolicyOptions provides set of configurations for Filesystem.SetAccessPolicy operation. +// SetAccessPolicyOptions provides set of configurations for FileSystem.SetAccessPolicy operation. type SetAccessPolicyOptions struct { - // Specifies whether data in the filesystem may be accessed publicly and the level of access. + // Access Specifies whether data in the filesystem may be accessed publicly and the level of access. // If this header is not included in the request, filesystem data is private to the account owner. - Access *PublicAccessType + Access *PublicAccessType + // AccessConditions identifies filesystem-specific access conditions which you optionally set. AccessConditions *AccessConditions - FilesystemACL []*SignedIdentifier + // FileSystemACL sets the access policy for the filesystem. + FileSystemACL []*SignedIdentifier } func (o *SetAccessPolicyOptions) format() *container.SetAccessPolicyOptions { @@ -29,19 +31,17 @@ func (o *SetAccessPolicyOptions) format() *container.SetAccessPolicyOptions { return &container.SetAccessPolicyOptions{ Access: o.Access, AccessConditions: exported.FormatContainerAccessConditions(o.AccessConditions), - ContainerACL: o.FilesystemACL, + ContainerACL: o.FileSystemACL, } } // CreateOptions contains the optional parameters for the Client.Create method. type CreateOptions struct { - // Specifies whether data in the filesystem may be accessed publicly and the level of access. + // Access specifies whether data in the filesystem may be accessed publicly and the level of access. Access *PublicAccessType - - // Optional. Specifies a user-defined name-value pair associated with the filesystem. + // Metadata specifies a user-defined name-value pair associated with the filesystem. Metadata map[string]*string - - // Optional. Specifies the encryption scope settings to set on the filesystem. + // CPKScopeInfo specifies the encryption scope settings to set on the filesystem. CPKScopeInfo *CPKScopeInfo } @@ -58,6 +58,7 @@ func (o *CreateOptions) format() *container.CreateOptions { // DeleteOptions contains the optional parameters for the Client.Delete method. type DeleteOptions struct { + // AccessConditions identifies filesystem-specific access conditions which you optionally set. AccessConditions *AccessConditions } @@ -70,8 +71,9 @@ func (o *DeleteOptions) format() *container.DeleteOptions { } } -// GetPropertiesOptions contains the optional parameters for the FilesystemClient.GetProperties method. +// GetPropertiesOptions contains the optional parameters for the FileSystemClient.GetProperties method. type GetPropertiesOptions struct { + // LeaseAccessConditions contains parameters to access leased filesystem. LeaseAccessConditions *LeaseAccessConditions } @@ -91,7 +93,9 @@ func (o *GetPropertiesOptions) format() *container.GetPropertiesOptions { // SetMetadataOptions contains the optional parameters for the Client.SetMetadata method. type SetMetadataOptions struct { - Metadata map[string]*string + // Metadata sets the metadata key-value pairs to set on the filesystem. + Metadata map[string]*string + // AccessConditions identifies filesystem-specific access conditions which you optionally set. AccessConditions *AccessConditions } @@ -109,6 +113,7 @@ func (o *SetMetadataOptions) format() *container.SetMetadataOptions { // GetAccessPolicyOptions contains the optional parameters for the Client.GetAccessPolicy method. type GetAccessPolicyOptions struct { + // LeaseAccessConditions contains parameters to access leased filesystem. LeaseAccessConditions *LeaseAccessConditions } @@ -126,12 +131,16 @@ func (o *GetAccessPolicyOptions) format() *container.GetAccessPolicyOptions { } } -// ListPathsOptions contains the optional parameters for the Filesystem.ListPaths operation. +// ListPathsOptions contains the optional parameters for the FileSystem.ListPaths operation. type ListPathsOptions struct { - Marker *string + // Marker contains last continuation token returned from the service for listing. + Marker *string + // MaxResults sets the maximum number of paths that will be returned per page. MaxResults *int32 - Prefix *string - Upn *bool + // Prefix filters the results to return only paths whose names begin with the specified prefix path. + Prefix *string + // UPN is the user principal name. + UPN *bool } func (o *ListPathsOptions) format() generated.FileSystemClientListPathsOptions { @@ -143,15 +152,18 @@ func (o *ListPathsOptions) format() generated.FileSystemClientListPathsOptions { Continuation: o.Marker, MaxResults: o.MaxResults, Path: o.Prefix, - Upn: o.Upn, + Upn: o.UPN, } } -// ListDeletedPathsOptions contains the optional parameters for the Filesystem.ListDeletedPaths operation. PLACEHOLDER +// ListDeletedPathsOptions contains the optional parameters for the FileSystem.ListDeletedPaths operation. type ListDeletedPathsOptions struct { - Marker *string + // Marker contains last continuation token returned from the service for listing. + Marker *string + // MaxResults sets the maximum number of paths that will be returned per page. MaxResults *int32 - Prefix *string + // Prefix filters the results to return only paths whose names begin with the specified prefix path. + Prefix *string } func (o *ListDeletedPathsOptions) format() generated.FileSystemClientListBlobHierarchySegmentOptions { @@ -169,6 +181,7 @@ func (o *ListDeletedPathsOptions) format() generated.FileSystemClientListBlobHie // GetSASURLOptions contains the optional parameters for the Client.GetSASURL method. type GetSASURLOptions struct { + // StartTime is the time after which the SAS will become valid. StartTime *time.Time } @@ -186,7 +199,7 @@ func (o *GetSASURLOptions) format() time.Time { return st } -//// UndeletePathOptions contains the optional parameters for the Filesystem.UndeletePath operation. +//// UndeletePathOptions contains the optional parameters for the FileSystem.UndeletePath operation. //type UndeletePathOptions struct { // // placeholder //} @@ -198,7 +211,7 @@ func (o *GetSASURLOptions) format() time.Time { // return &UndeletePathOptions{} //} -// CPKScopeInfo contains a group of parameters for the FilesystemClient.Create method. +// CPKScopeInfo contains a group of parameters for the FileSystemClient.Create method. type CPKScopeInfo = container.CPKScopeInfo // AccessPolicy - An Access policy. @@ -214,7 +227,7 @@ type SignedIdentifier = container.SignedIdentifier // SharedKeyCredential contains an account's name and its primary or secondary key. type SharedKeyCredential = exported.SharedKeyCredential -// AccessConditions identifies blob-specific access conditions which you optionally set. +// AccessConditions identifies filesystem-specific access conditions which you optionally set. type AccessConditions = exported.AccessConditions // LeaseAccessConditions contains optional parameters to access leased entity. @@ -229,11 +242,11 @@ type PathList = generated.PathList // Path contains the path properties from the ListPaths operation type Path = generated.Path -// PathItem contains the response from method FilesystemClient.ListBlobsHierarchySegment. +// PathItem contains the response from method FileSystemClient.ListPathsHierarchySegment. type PathItem = generated.PathItemInternal -// PathProperties contains the response from method FilesystemClient.ListBlobsHierarchySegment. +// PathProperties contains the response from method FileSystemClient.ListPathsHierarchySegment. type PathProperties = generated.PathPropertiesInternal -// PathPrefix contains the response from method FilesystemClient.ListBlobsHierarchySegment. +// PathPrefix contains the response from method FileSystemClient.ListPathsHierarchySegment. type PathPrefix = generated.PathPrefix diff --git a/sdk/storage/azdatalake/filesystem/responses.go b/sdk/storage/azdatalake/filesystem/responses.go index 5651a4a77a9c..15b97decc321 100644 --- a/sdk/storage/azdatalake/filesystem/responses.go +++ b/sdk/storage/azdatalake/filesystem/responses.go @@ -13,7 +13,7 @@ import ( "time" ) -// GetAccessPolicyResponse contains the response from method FilesystemClient.GetAccessPolicy. +// GetAccessPolicyResponse contains the response from method FileSystemClient.GetAccessPolicy. type GetAccessPolicyResponse struct { // PublicAccess contains the information returned from the x-ms-blob-public-access header response. PublicAccess *PublicAccessType @@ -52,7 +52,7 @@ func formatGetAccessPolicyResponse(r *GetAccessPolicyResponse, contResp *contain r.Version = contResp.Version } -// GetPropertiesResponse contains the response from method FilesystemClient.GetProperties. +// GetPropertiesResponse contains the response from method FileSystemClient.GetProperties. type GetPropertiesResponse struct { // BlobPublicAccess contains the information returned from the x-ms-blob-public-access header response. PublicAccess *PublicAccessType @@ -105,7 +105,7 @@ type GetPropertiesResponse struct { } // removes the blob prefix in access type -func formatFilesystemProperties(r *GetPropertiesResponse, contResp *container.GetPropertiesResponse) { +func formatFileSystemProperties(r *GetPropertiesResponse, contResp *container.GetPropertiesResponse) { r.PublicAccess = contResp.BlobPublicAccess r.ClientRequestID = contResp.ClientRequestID r.Date = contResp.Date @@ -124,29 +124,29 @@ func formatFilesystemProperties(r *GetPropertiesResponse, contResp *container.Ge r.Version = contResp.Version } -// CreateResponse contains the response from method FilesystemClient.Create. +// CreateResponse contains the response from method FileSystemClient.Create. type CreateResponse = container.CreateResponse -// DeleteResponse contains the response from method FilesystemClient.Delete. +// DeleteResponse contains the response from method FileSystemClient.Delete. type DeleteResponse = container.DeleteResponse -// SetMetadataResponse contains the response from method FilesystemClient.SetMetadata. +// SetMetadataResponse contains the response from method FileSystemClient.SetMetadata. type SetMetadataResponse = container.SetMetadataResponse -// SetAccessPolicyResponse contains the response from method FilesystemClient.SetAccessPolicy. +// SetAccessPolicyResponse contains the response from method FileSystemClient.SetAccessPolicy. type SetAccessPolicyResponse = container.SetAccessPolicyResponse -// ListPathsSegmentResponse contains the response from method FilesystemClient.ListPathsSegment. +// ListPathsSegmentResponse contains the response from method FileSystemClient.ListPathsSegment. type ListPathsSegmentResponse = generated.FileSystemClientListPathsResponse -// UndeletePathResponse contains the response from method FilesystemClient.UndeletePath. +// UndeletePathResponse contains the response from method FileSystemClient.UndeletePath. type UndeletePathResponse = generated.PathClientUndeleteResponse -// ListDeletedPathsSegmentResponse contains the response from method FilesystemClient.ListPathsSegment. +// ListDeletedPathsSegmentResponse contains the response from method FileSystemClient.ListPathsSegment. type ListDeletedPathsSegmentResponse = generated.FileSystemClientListPathHierarchySegmentResponse -// ListPathsHierarchySegmentResponse contains the response from method FilesystemClient.ListBlobsHierarchySegment. +// ListPathsHierarchySegmentResponse contains the response from method FileSystemClient.ListPathsHierarchySegment. type ListPathsHierarchySegmentResponse = generated.ListPathsHierarchySegmentResponse -// PathHierarchyListSegment contains the response from method FilesystemClient.ListBlobsHierarchySegment. +// PathHierarchyListSegment contains the response from method FileSystemClient.ListPathsHierarchySegment. type PathHierarchyListSegment = generated.PathHierarchyListSegment diff --git a/sdk/storage/azdatalake/go.mod b/sdk/storage/azdatalake/go.mod index 17498bc501b5..e909b1dc0a62 100644 --- a/sdk/storage/azdatalake/go.mod +++ b/sdk/storage/azdatalake/go.mod @@ -4,16 +4,24 @@ go 1.18 require ( github.com/Azure/azure-sdk-for-go/sdk/azcore v1.7.0 + github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.1.0 github.com/Azure/azure-sdk-for-go/sdk/internal v1.3.0 github.com/Azure/azure-sdk-for-go/sdk/storage/azblob v1.0.0 github.com/stretchr/testify v1.7.1 ) require ( + github.com/AzureAD/microsoft-authentication-library-for-go v0.5.1 // indirect github.com/davecgh/go-spew v1.1.1 // indirect github.com/dnaeon/go-vcr v1.1.0 // indirect + github.com/golang-jwt/jwt v3.2.1+incompatible // indirect + github.com/google/uuid v1.1.1 // indirect + github.com/kylelemons/godebug v1.1.0 // indirect + github.com/pkg/browser v0.0.0-20210115035449-ce105d075bb4 // indirect github.com/pmezard/go-difflib v1.0.0 // indirect + golang.org/x/crypto v0.0.0-20220511200225-c6db032c6c88 // indirect golang.org/x/net v0.8.0 // indirect + golang.org/x/sys v0.6.0 // indirect golang.org/x/text v0.8.0 // indirect gopkg.in/yaml.v2 v2.4.0 // indirect gopkg.in/yaml.v3 v3.0.0-20210107192922-496545a6307b // indirect diff --git a/sdk/storage/azdatalake/go.sum b/sdk/storage/azdatalake/go.sum index 3482ec48d6a5..eafa822d8fa4 100644 --- a/sdk/storage/azdatalake/go.sum +++ b/sdk/storage/azdatalake/go.sum @@ -1,30 +1,41 @@ github.com/Azure/azure-sdk-for-go/sdk/azcore v1.7.0 h1:8q4SaHjFsClSvuVne0ID/5Ka8u3fcIHyqkLjcFpNRHQ= github.com/Azure/azure-sdk-for-go/sdk/azcore v1.7.0/go.mod h1:bjGvMhVMb+EEm3VRNQawDMUyMMjo+S5ewNjflkep/0Q= github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.1.0 h1:QkAcEIAKbNL4KoFr4SathZPhDhF4mVwpBMFlYjyAqy8= +github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.1.0/go.mod h1:bhXu1AjYL+wutSL/kpSq6s7733q2Rb0yuot9Zgfqa/0= github.com/Azure/azure-sdk-for-go/sdk/internal v1.3.0 h1:sXr+ck84g/ZlZUOZiNELInmMgOsuGwdjjVkEIde0OtY= github.com/Azure/azure-sdk-for-go/sdk/internal v1.3.0/go.mod h1:okt5dMMTOFjX/aovMlrjvvXoPMBVSPzk9185BT0+eZM= github.com/Azure/azure-sdk-for-go/sdk/storage/azblob v1.0.0 h1:u/LLAOFgsMv7HmNL4Qufg58y+qElGOt5qv0z1mURkRY= github.com/Azure/azure-sdk-for-go/sdk/storage/azblob v1.0.0/go.mod h1:2e8rMJtl2+2j+HXbTBwnyGpm5Nou7KhvSfxOq8JpTag= github.com/AzureAD/microsoft-authentication-library-for-go v0.5.1 h1:BWe8a+f/t+7KY7zH2mqygeUD0t8hNFXe08p1Pb3/jKE= +github.com/AzureAD/microsoft-authentication-library-for-go v0.5.1/go.mod h1:Vt9sXTKwMyGcOxSmLDMnGPgqsUg7m8pe215qMLrDXw4= github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c= github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= github.com/dnaeon/go-vcr v1.1.0 h1:ReYa/UBrRyQdant9B4fNHGoCNKw6qh6P0fsdGmZpR7c= github.com/dnaeon/go-vcr v1.1.0/go.mod h1:M7tiix8f0r6mKKJ3Yq/kqU1OYf3MnfmBWVbPx/yU9ko= github.com/golang-jwt/jwt v3.2.1+incompatible h1:73Z+4BJcrTC+KczS6WvTPvRGOp1WmfEP4Q1lOd9Z/+c= +github.com/golang-jwt/jwt v3.2.1+incompatible/go.mod h1:8pz2t5EyA70fFQQSrl6XZXzqecmYZeUEB8OUGHkxJ+I= +github.com/golang-jwt/jwt/v4 v4.2.0 h1:besgBTC8w8HjP6NzQdxwKH9Z5oQMZ24ThTrHp3cZ8eU= github.com/google/uuid v1.1.1 h1:Gkbcsh/GbpXz7lPftLA3P6TYMwjCLYm83jiFQZF/3gY= +github.com/google/uuid v1.1.1/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo= github.com/kylelemons/godebug v1.1.0 h1:RPNrshWIDI6G2gRW9EHilWtl7Z6Sb1BR0xunSBf0SNc= +github.com/kylelemons/godebug v1.1.0/go.mod h1:9/0rRGxNHcop5bhtWyNeEfOS8JIWk580+fNqagV/RAw= github.com/modocache/gover v0.0.0-20171022184752-b58185e213c5/go.mod h1:caMODM3PzxT8aQXRPkAt8xlV/e7d7w8GM5g0fa5F0D8= +github.com/montanaflynn/stats v0.6.6/go.mod h1:etXPPgVO6n31NxCd9KQUMvCM+ve0ruNzt6R8Bnaayow= github.com/pkg/browser v0.0.0-20210115035449-ce105d075bb4 h1:Qj1ukM4GlMWXNdMBuXcXfz/Kw9s1qm0CLY32QxuSImI= +github.com/pkg/browser v0.0.0-20210115035449-ce105d075bb4/go.mod h1:N6UoU20jOqggOuDwUaBQpluzLNDqif3kq9z2wpdYEfQ= github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM= github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME= github.com/stretchr/testify v1.7.1 h1:5TQK59W5E3v0r2duFAb7P95B6hEeOyEnHRa8MjYSMTY= github.com/stretchr/testify v1.7.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg= golang.org/x/crypto v0.0.0-20220511200225-c6db032c6c88 h1:Tgea0cVUD0ivh5ADBX4WwuI12DUd2to3nCYe2eayMIw= +golang.org/x/crypto v0.0.0-20220511200225-c6db032c6c88/go.mod h1:IxCIyHEi3zRg3s0A5j5BB6A9Jmi73HwBIUl50j+osU4= golang.org/x/net v0.8.0 h1:Zrh2ngAOFYneWTAIAPethzeaQLuHwhuBkuV6ZiRnUaQ= golang.org/x/net v0.8.0/go.mod h1:QVkue5JL9kW//ek3r6jTKnTFis1tRmNAW2P1shuFdJc= +golang.org/x/sys v0.0.0-20210124154548-22da62e12c0c/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.6.0 h1:MVltZSvRTcU2ljQOhs94SXPftV6DCNnZViHeQps87pQ= +golang.org/x/sys v0.6.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/text v0.8.0 h1:57P1ETyNKtuIjB4SRd15iJxuhj8Gc416Y78H3qgMh68= golang.org/x/text v0.8.0/go.mod h1:e1OnstbJyHTd6l/uOt8jFFHp6TRDWZR/bV3emEE/zU8= gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405 h1:yhCVgyC4o1eVCa2tZl7eS0r+SDo693bJlVdllGtEeKM= diff --git a/sdk/storage/azdatalake/internal/base/clients.go b/sdk/storage/azdatalake/internal/base/clients.go index cc4af2b63e93..e5b9e2a300c5 100644 --- a/sdk/storage/azdatalake/internal/base/clients.go +++ b/sdk/storage/azdatalake/internal/base/clients.go @@ -55,10 +55,10 @@ func IdentityCredentialComposite[T, K, U any](client *CompositeClient[T, K, U]) return client.identityCred } -func NewFilesystemClient(fsURL string, fsURLWithBlobEndpoint string, client *container.Client, azClient *azcore.Client, sharedKey *exported.SharedKeyCredential, identityCred *azcore.TokenCredential, options *ClientOptions) *CompositeClient[generated.FileSystemClient, generated.FileSystemClient, container.Client] { +func NewFileSystemClient(fsURL string, fsURLWithBlobEndpoint string, client *container.Client, azClient *azcore.Client, sharedKey *exported.SharedKeyCredential, identityCred *azcore.TokenCredential, options *ClientOptions) *CompositeClient[generated.FileSystemClient, generated.FileSystemClient, container.Client] { return &CompositeClient[generated.FileSystemClient, generated.FileSystemClient, container.Client]{ - innerT: generated.NewFilesystemClient(fsURL, azClient), - innerK: generated.NewFilesystemClient(fsURLWithBlobEndpoint, azClient), + innerT: generated.NewFileSystemClient(fsURL, azClient), + innerK: generated.NewFileSystemClient(fsURLWithBlobEndpoint, azClient), sharedKey: sharedKey, identityCred: identityCred, innerU: client, diff --git a/sdk/storage/azdatalake/internal/exported/access_conditions.go b/sdk/storage/azdatalake/internal/exported/access_conditions.go index 624efc449132..e7b2a8ac7a3f 100644 --- a/sdk/storage/azdatalake/internal/exported/access_conditions.go +++ b/sdk/storage/azdatalake/internal/exported/access_conditions.go @@ -33,7 +33,7 @@ type ModifiedAccessConditions = generated.ModifiedAccessConditions // SourceModifiedAccessConditions contains a group of parameters for specifying access conditions of a source. type SourceModifiedAccessConditions = generated.SourceModifiedAccessConditions -// FormatContainerAccessConditions formats FilesystemAccessConditions into container's LeaseAccessConditions and ModifiedAccessConditions. +// FormatContainerAccessConditions formats FileSystemAccessConditions into container's LeaseAccessConditions and ModifiedAccessConditions. func FormatContainerAccessConditions(b *AccessConditions) *container.AccessConditions { if b == nil { return &container.AccessConditions{ diff --git a/sdk/storage/azdatalake/internal/exported/exported.go b/sdk/storage/azdatalake/internal/exported/exported.go index 0dea8e21223c..c6bbc6fe428d 100644 --- a/sdk/storage/azdatalake/internal/exported/exported.go +++ b/sdk/storage/azdatalake/internal/exported/exported.go @@ -45,7 +45,7 @@ func ConvertToDFSError(err error) error { responseErr.ErrorCode = strings.Replace(responseErr.ErrorCode, "blob", "path", -1) responseErr.ErrorCode = strings.Replace(responseErr.ErrorCode, "Blob", "Path", -1) responseErr.ErrorCode = strings.Replace(responseErr.ErrorCode, "container", "filesystem", -1) - responseErr.ErrorCode = strings.Replace(responseErr.ErrorCode, "Container", "Filesystem", -1) + responseErr.ErrorCode = strings.Replace(responseErr.ErrorCode, "Container", "FileSystem", -1) return responseErr } return err diff --git a/sdk/storage/azdatalake/internal/exported/shared_key_credential.go b/sdk/storage/azdatalake/internal/exported/shared_key_credential.go index d54cdc3a0b76..e75b29f0ad8b 100644 --- a/sdk/storage/azdatalake/internal/exported/shared_key_credential.go +++ b/sdk/storage/azdatalake/internal/exported/shared_key_credential.go @@ -49,7 +49,7 @@ func (c *SharedKeyCredential) AccountName() string { return c.accountName } -func (c *SharedKeyCredential) ConvertToBlobSharedKey() (*azblob.SharedKeyCredential, error) { +func ConvertToBlobSharedKey(c *SharedKeyCredential) (*azblob.SharedKeyCredential, error) { cred, err := azblob.NewSharedKeyCredential(c.accountName, c.accountKeyString) if err != nil { return nil, err diff --git a/sdk/storage/azdatalake/internal/exported/version.go b/sdk/storage/azdatalake/internal/exported/version.go index b570e4653d73..8fdb8eaff91d 100644 --- a/sdk/storage/azdatalake/internal/exported/version.go +++ b/sdk/storage/azdatalake/internal/exported/version.go @@ -8,5 +8,5 @@ package exported const ( ModuleName = "azdatalake" - ModuleVersion = "v0.1.0" + ModuleVersion = "v0.1.0-beta.1" ) diff --git a/sdk/storage/azdatalake/internal/generated/autorest.md b/sdk/storage/azdatalake/internal/generated/autorest.md index 56ce2831277a..edc674717865 100644 --- a/sdk/storage/azdatalake/internal/generated/autorest.md +++ b/sdk/storage/azdatalake/internal/generated/autorest.md @@ -22,7 +22,7 @@ export-clients: true use: "@autorest/go@4.0.0-preview.49" ``` -### Remove Filesystem and PathName from parameter list since they are not needed +### Remove FileSystem and PathName from parameter list since they are not needed ``` yaml directive: - from: swagger-document @@ -253,7 +253,7 @@ directive: transform: >- return $. replace(/PublicAccessTypeBlob/g, 'PublicAccessTypeFile'). - replace(/PublicAccessTypeContainer/g, 'PublicAccessTypeFilesystem'). + replace(/PublicAccessTypeContainer/g, 'PublicAccessTypeFileSystem'). replace(/FileSystemClientListBlobHierarchySegmentResponse/g, 'FileSystemClientListPathHierarchySegmentResponse'). replace(/ListBlobsHierarchySegmentResponse/g, 'ListPathsHierarchySegmentResponse'). replace(/ContainerName\s*\*string/g, 'FileSystemName *string'). @@ -266,13 +266,14 @@ directive: replace(/ContainerProperties/g, 'FileSystemProperties'); ``` -### TODO: FIX THE BELOW IN UNMARSHALASJSON -### Change path props to string +### ``` yaml directive: -- from: swagger-document - where: $.definitions.Path.properties - transform: > - $.isDirectory.type = "string"; - $.contentLength.type = "string"; +- from: + - zz_models_serde.go + where: $ + transform: >- + return $. + replace(/err = unpopulate\((.*), "ContentLength", &p\.ContentLength\)/g, 'var rawVal string\nerr = unpopulate(val, "ContentLength", &rawVal)\nintVal, _ := strconv.ParseInt(rawVal, 10, 64)\np.ContentLength = &intVal'). + replace(/err = unpopulate\((.*), "IsDirectory", &p\.IsDirectory\)/g, 'var rawVal string\nerr = unpopulate(val, "IsDirectory", &rawVal)\nboolVal, _ := strconv.ParseBool(rawVal)\np.IsDirectory = &boolVal'); ``` \ No newline at end of file diff --git a/sdk/storage/azdatalake/internal/generated/filesystem_client.go b/sdk/storage/azdatalake/internal/generated/filesystem_client.go index d35651c781b8..35c7c2e8e8a8 100644 --- a/sdk/storage/azdatalake/internal/generated/filesystem_client.go +++ b/sdk/storage/azdatalake/internal/generated/filesystem_client.go @@ -22,10 +22,10 @@ func (client *FileSystemClient) InternalClient() *azcore.Client { return client.internal } -// NewFilesystemClient creates a new instance of ServiceClient with the specified values. +// NewFileSystemClient creates a new instance of ServiceClient with the specified values. // - endpoint - The URL of the service account, share, directory or file that is the target of the desired operation. // - azClient - azcore.Client is a basic HTTP client. It consists of a pipeline and tracing provider. -func NewFilesystemClient(endpoint string, azClient *azcore.Client) *FileSystemClient { +func NewFileSystemClient(endpoint string, azClient *azcore.Client) *FileSystemClient { client := &FileSystemClient{ internal: azClient, endpoint: endpoint, diff --git a/sdk/storage/azdatalake/internal/generated/zz_models.go b/sdk/storage/azdatalake/internal/generated/zz_models.go index 64f0139f6536..6201c56fa6d4 100644 --- a/sdk/storage/azdatalake/internal/generated/zz_models.go +++ b/sdk/storage/azdatalake/internal/generated/zz_models.go @@ -260,7 +260,7 @@ type ModifiedAccessConditions struct { } type Path struct { - ContentLength *string + ContentLength *int64 CreationTime *string ETag *string @@ -268,7 +268,7 @@ type Path struct { EncryptionScope *string ExpiryTime *string Group *string - IsDirectory *string + IsDirectory *bool LastModified *string Name *string Owner *string diff --git a/sdk/storage/azdatalake/internal/generated/zz_models_serde.go b/sdk/storage/azdatalake/internal/generated/zz_models_serde.go index f118d2a055aa..cb66df1e8173 100644 --- a/sdk/storage/azdatalake/internal/generated/zz_models_serde.go +++ b/sdk/storage/azdatalake/internal/generated/zz_models_serde.go @@ -16,6 +16,7 @@ import ( "github.com/Azure/azure-sdk-for-go/sdk/azcore" "github.com/Azure/azure-sdk-for-go/sdk/azcore/runtime" "reflect" + "strconv" "time" ) @@ -230,7 +231,10 @@ func (p *Path) UnmarshalJSON(data []byte) error { var err error switch key { case "contentLength": - err = unpopulate(val, "ContentLength", &p.ContentLength) + var rawVal string + err = unpopulate(val, "ContentLength", &rawVal) + intVal, _ := strconv.ParseInt(rawVal, 10, 64) + p.ContentLength = &intVal delete(rawMsg, key) case "creationTime": err = unpopulate(val, "CreationTime", &p.CreationTime) @@ -248,7 +252,10 @@ func (p *Path) UnmarshalJSON(data []byte) error { err = unpopulate(val, "Group", &p.Group) delete(rawMsg, key) case "isDirectory": - err = unpopulate(val, "IsDirectory", &p.IsDirectory) + var rawVal string + err = unpopulate(val, "IsDirectory", &rawVal) + boolVal, _ := strconv.ParseBool(rawVal) + p.IsDirectory = &boolVal delete(rawMsg, key) case "lastModified": err = unpopulate(val, "LastModified", &p.LastModified) diff --git a/sdk/storage/azdatalake/internal/generated_blob/autorest.md b/sdk/storage/azdatalake/internal/generated_blob/autorest.md index f0e08dcb686d..5905e9ac68ee 100644 --- a/sdk/storage/azdatalake/internal/generated_blob/autorest.md +++ b/sdk/storage/azdatalake/internal/generated_blob/autorest.md @@ -469,7 +469,7 @@ directive: replace(/PublicAccessTypeContainer/g, 'PublicAccessTypeFileSystem'). replace(/ContainerClientListBlobHierarchySegmentResponse/g, 'FileSystemClientListPathHierarchySegmentResponse'). replace(/ListBlobsHierarchySegmentResponse/g, 'ListPathsHierarchySegmentResponse'). - replace(/(^|[^"])ContainerName/g, '$1FilesystemName'). + replace(/(^|[^"])ContainerName/g, '$1FileSystemName'). replace(/BlobHierarchyListSegment/g, 'PathHierarchyListSegment'). replace(/BlobItems/g, 'PathItems'). replace(/BlobItem/g, 'PathItem'). diff --git a/sdk/storage/azdatalake/internal/generated_blob/zz_container_client.go b/sdk/storage/azdatalake/internal/generated_blob/zz_container_client.go index 778da1718b7c..3f34ec1a47c4 100644 --- a/sdk/storage/azdatalake/internal/generated_blob/zz_container_client.go +++ b/sdk/storage/azdatalake/internal/generated_blob/zz_container_client.go @@ -1001,10 +1001,10 @@ func (client *ContainerClient) releaseLeaseHandleResponse(resp *http.Response) ( // If the operation fails it returns an *azcore.ResponseError type. // // Generated from API version 2020-10-02 -// - sourceFilesystemName - Required. Specifies the name of the container to rename. +// - sourceFileSystemName - Required. Specifies the name of the container to rename. // - options - ContainerClientRenameOptions contains the optional parameters for the ContainerClient.Rename method. -func (client *ContainerClient) Rename(ctx context.Context, sourceFilesystemName string, options *ContainerClientRenameOptions) (ContainerClientRenameResponse, error) { - req, err := client.renameCreateRequest(ctx, sourceFilesystemName, options) +func (client *ContainerClient) Rename(ctx context.Context, sourceFileSystemName string, options *ContainerClientRenameOptions) (ContainerClientRenameResponse, error) { + req, err := client.renameCreateRequest(ctx, sourceFileSystemName, options) if err != nil { return ContainerClientRenameResponse{}, err } @@ -1019,7 +1019,7 @@ func (client *ContainerClient) Rename(ctx context.Context, sourceFilesystemName } // renameCreateRequest creates the Rename request. -func (client *ContainerClient) renameCreateRequest(ctx context.Context, sourceFilesystemName string, options *ContainerClientRenameOptions) (*policy.Request, error) { +func (client *ContainerClient) renameCreateRequest(ctx context.Context, sourceFileSystemName string, options *ContainerClientRenameOptions) (*policy.Request, error) { req, err := runtime.NewRequest(ctx, http.MethodPut, client.endpoint) if err != nil { return nil, err @@ -1035,7 +1035,7 @@ func (client *ContainerClient) renameCreateRequest(ctx context.Context, sourceFi if options != nil && options.RequestID != nil { req.Raw().Header["x-ms-client-request-id"] = []string{*options.RequestID} } - req.Raw().Header["x-ms-source-container-name"] = []string{sourceFilesystemName} + req.Raw().Header["x-ms-source-container-name"] = []string{sourceFileSystemName} if options != nil && options.SourceLeaseID != nil { req.Raw().Header["x-ms-source-lease-id"] = []string{*options.SourceLeaseID} } @@ -1189,8 +1189,8 @@ func (client *ContainerClient) restoreCreateRequest(ctx context.Context, options if options != nil && options.RequestID != nil { req.Raw().Header["x-ms-client-request-id"] = []string{*options.RequestID} } - if options != nil && options.DeletedFilesystemName != nil { - req.Raw().Header["x-ms-deleted-container-name"] = []string{*options.DeletedFilesystemName} + if options != nil && options.DeletedFileSystemName != nil { + req.Raw().Header["x-ms-deleted-container-name"] = []string{*options.DeletedFileSystemName} } if options != nil && options.DeletedContainerVersion != nil { req.Raw().Header["x-ms-deleted-container-version"] = []string{*options.DeletedContainerVersion} diff --git a/sdk/storage/azdatalake/internal/generated_blob/zz_models.go b/sdk/storage/azdatalake/internal/generated_blob/zz_models.go index d60fbd9ff1b3..c07e6bdb3b39 100644 --- a/sdk/storage/azdatalake/internal/generated_blob/zz_models.go +++ b/sdk/storage/azdatalake/internal/generated_blob/zz_models.go @@ -974,7 +974,7 @@ type ContainerClientRenewLeaseOptions struct { // ContainerClientRestoreOptions contains the optional parameters for the ContainerClient.Restore method. type ContainerClientRestoreOptions struct { // Optional. Version 2019-12-12 and later. Specifies the name of the deleted container to restore. - DeletedFilesystemName *string + DeletedFileSystemName *string // Optional. Version 2019-12-12 and later. Specifies the version of the deleted container to restore. DeletedContainerVersion *string // Provides a client-generated, opaque value with a 1 KB character limit that is recorded in the analytics logs when storage @@ -1137,7 +1137,7 @@ type DelimitedTextConfiguration struct { // FilterPathItem - Blob info from a Filter Blobs API call type FilterPathItem struct { // REQUIRED - FilesystemName *string `xml:"ContainerName"` + FileSystemName *string `xml:"ContainerName"` // REQUIRED Name *string `xml:"Name"` @@ -1194,7 +1194,7 @@ type LeaseAccessConditions struct { // ListBlobsFlatSegmentResponse - An enumeration of blobs type ListBlobsFlatSegmentResponse struct { // REQUIRED - FilesystemName *string `xml:"ContainerName,attr"` + FileSystemName *string `xml:"ContainerName,attr"` // REQUIRED Segment *BlobFlatListSegment `xml:"Blobs"` @@ -1210,7 +1210,7 @@ type ListBlobsFlatSegmentResponse struct { // ListPathsHierarchySegmentResponse - An enumeration of blobs type ListPathsHierarchySegmentResponse struct { // REQUIRED - FilesystemName *string `xml:"ContainerName,attr"` + FileSystemName *string `xml:"ContainerName,attr"` // REQUIRED Segment *PathHierarchyListSegment `xml:"Blobs"` diff --git a/sdk/storage/azdatalake/internal/path/models.go b/sdk/storage/azdatalake/internal/path/models.go index a66c7bfc448c..3a25acce5a27 100644 --- a/sdk/storage/azdatalake/internal/path/models.go +++ b/sdk/storage/azdatalake/internal/path/models.go @@ -15,9 +15,9 @@ import ( "time" ) -// DeleteOptions contains the optional parameters when calling the Delete operation. dfs endpoint +// DeleteOptions contains the optional parameters when calling the Delete operation. type DeleteOptions struct { - // AccessConditions contains parameters for accessing the file. + // AccessConditions contains parameters for accessing the path. AccessConditions *AccessConditions } @@ -36,7 +36,7 @@ func FormatDeleteOptions(o *DeleteOptions, recursive bool) (*generated.LeaseAcce type RenameOptions struct { // SourceAccessConditions identifies the source path access conditions. SourceAccessConditions *SourceAccessConditions - // AccessConditions contains parameters for accessing the file. + // AccessConditions contains parameters for accessing the path. AccessConditions *AccessConditions } @@ -68,7 +68,7 @@ func FormatRenameOptions(o *RenameOptions, path string) (*generated.LeaseAccessC return leaseAccessConditions, modifiedAccessConditions, nil, createOpts } -// GetPropertiesOptions contains the optional parameters for the Client.GetProperties method +// GetPropertiesOptions contains the optional parameters for the Client.GetProperties method. type GetPropertiesOptions struct { AccessConditions *AccessConditions CPKInfo *CPKInfo @@ -94,7 +94,7 @@ func FormatGetPropertiesOptions(o *GetPropertiesOptions) *blob.GetPropertiesOpti // ===================================== PATH IMPORTS =========================================== -// SetAccessControlOptions contains the optional parameters when calling the SetAccessControl operation. dfs endpoint +// SetAccessControlOptions contains the optional parameters when calling the SetAccessControl operation. type SetAccessControlOptions struct { // Owner is the owner of the path. Owner *string @@ -148,15 +148,19 @@ func FormatGetAccessControlOptions(o *GetAccessControlOptions) (*generated.PathC }, leaseAccessConditions, modifiedAccessConditions } -// CPKInfo contains a group of parameters for the PathClient.Download method. +// CPKInfo contains CPK related information. type CPKInfo struct { + // EncryptionAlgorithm is the algorithm used to encrypt the data. EncryptionAlgorithm *EncryptionAlgorithmType - EncryptionKey *string + // EncryptionKey is the base64 encoded encryption key. + EncryptionKey *string + // EncryptionKeySHA256 is the base64 encoded SHA256 of the encryption key. EncryptionKeySHA256 *string } // GetSASURLOptions contains the optional parameters for the Client.GetSASURL method. type GetSASURLOptions struct { + // StartTime is the start time for this SAS token. StartTime *time.Time } @@ -176,6 +180,7 @@ func FormatGetSASURLOptions(o *GetSASURLOptions) time.Time { // SetHTTPHeadersOptions contains the optional parameters for the Client.SetHTTPHeaders method. type SetHTTPHeadersOptions struct { + // AccessConditions contains parameters for accessing the path. AccessConditions *AccessConditions } @@ -199,19 +204,19 @@ func FormatSetHTTPHeadersOptions(o *SetHTTPHeadersOptions, httpHeaders HTTPHeade // HTTPHeaders contains the HTTP headers for path operations. type HTTPHeaders struct { - // Sets the path's cache control. If specified, this property is stored with the path and returned with a read request. + // CacheControl Sets the path's cache control. If specified, this property is stored with the path and returned with a read request. CacheControl *string - // Sets the path's Content-Disposition header. + // ContentDisposition Sets the path's Content-Disposition header. ContentDisposition *string - // Sets the path's content encoding. If specified, this property is stored with the blobpath and returned with a read + // ContentEncoding Sets the path's content encoding. If specified, this property is stored with the path and returned with a read // request. ContentEncoding *string - // Set the path's content language. If specified, this property is stored with the path and returned with a read + // ContentLanguage Set the path's content language. If specified, this property is stored with the path and returned with a read // request. ContentLanguage *string - // Specify the transactional md5 for the body, to be validated by the service. + // ContentMD5 Specify the transactional md5 for the body, to be validated by the service. ContentMD5 []byte - // Sets the path's content type. If specified, this property is stored with the path and returned with a read request. + // ContentType Sets the path's content type. If specified, this property is stored with the path and returned with a read request. ContentType *string } @@ -246,15 +251,17 @@ func FormatPathHTTPHeaders(o *HTTPHeaders) *generated.PathHTTPHeaders { // SetMetadataOptions provides set of configurations for Set Metadata on path operation type SetMetadataOptions struct { - Metadata map[string]*string + // AccessConditions contains parameters for accessing the path. AccessConditions *AccessConditions - CPKInfo *CPKInfo - CPKScopeInfo *CPKScopeInfo + // CPKInfo contains CPK related information. + CPKInfo *CPKInfo + // CPKScopeInfo specifies the encryption scope settings. + CPKScopeInfo *CPKScopeInfo } -func FormatSetMetadataOptions(o *SetMetadataOptions) (*blob.SetMetadataOptions, map[string]*string) { +func FormatSetMetadataOptions(o *SetMetadataOptions) *blob.SetMetadataOptions { if o == nil { - return nil, nil + return nil } accessConditions := exported.FormatBlobAccessConditions(o.AccessConditions) opts := &blob.SetMetadataOptions{ @@ -270,7 +277,7 @@ func FormatSetMetadataOptions(o *SetMetadataOptions) (*blob.SetMetadataOptions, if o.CPKScopeInfo != nil { opts.CPKScopeInfo = o.CPKScopeInfo } - return opts, o.Metadata + return opts } // ========================================= constants ========================================= diff --git a/sdk/storage/azdatalake/internal/path/responses.go b/sdk/storage/azdatalake/internal/path/responses.go index 1189ff02d693..1cf49ce01acf 100644 --- a/sdk/storage/azdatalake/internal/path/responses.go +++ b/sdk/storage/azdatalake/internal/path/responses.go @@ -32,6 +32,51 @@ type CreateResponse = generated.PathClientCreateResponse // DeleteResponse contains the response fields for the Delete operation. type DeleteResponse = generated.PathClientDeleteResponse +type RenameResponse struct { + // ContentLength contains the information returned from the Content-Length header response. + ContentLength *int64 + + // Continuation contains the information returned from the x-ms-continuation header response. + Continuation *string + + // Date contains the information returned from the Date header response. + Date *time.Time + + // ETag contains the information returned from the ETag header response. + ETag *azcore.ETag + + // EncryptionKeySHA256 contains the information returned from the x-ms-encryption-key-sha256 header response. + EncryptionKeySHA256 *string + + // IsServerEncrypted contains the information returned from the x-ms-request-server-encrypted header response. + IsServerEncrypted *bool + + // LastModified contains the information returned from the Last-Modified header response. + LastModified *time.Time + + // RequestID contains the information returned from the x-ms-request-id header response. + RequestID *string + + // Version contains the information returned from the x-ms-version header response. + Version *string +} + +// We need to do this now in case we add the new client to renamed response - we don't want to break the cx + +func FormatRenameResponse(createResp *CreateResponse) RenameResponse { + newResp := RenameResponse{} + newResp.ContentLength = createResp.ContentLength + newResp.Continuation = createResp.Continuation + newResp.Date = createResp.Date + newResp.ETag = createResp.ETag + newResp.EncryptionKeySHA256 = createResp.EncryptionKeySHA256 + newResp.IsServerEncrypted = createResp.IsServerEncrypted + newResp.LastModified = createResp.LastModified + newResp.RequestID = createResp.RequestID + newResp.Version = createResp.Version + return newResp +} + // removed BlobSequenceNumber, BlobCommittedBlockCount and BlobType headers from the original response: // GetPropertiesResponse contains the response fields for the GetProperties operation. diff --git a/sdk/storage/azdatalake/internal/shared/shared.go b/sdk/storage/azdatalake/internal/shared/shared.go index 0e54e94827df..f6ae4ea2fe66 100644 --- a/sdk/storage/azdatalake/internal/shared/shared.go +++ b/sdk/storage/azdatalake/internal/shared/shared.go @@ -26,7 +26,7 @@ const ( const ( ServiceClient = "azdatalake/service.Client" - FilesystemClient = "azdatalake/share.Client" + FileSystemClient = "azdatalake/filesystem.Client" DirectoryClient = "azdatalake/directory.Client" FileClient = "azdatalake/file.Client" ) diff --git a/sdk/storage/azdatalake/internal/testcommon/clients_auth.go b/sdk/storage/azdatalake/internal/testcommon/clients_auth.go index 6f93a8517585..acc034183de4 100644 --- a/sdk/storage/azdatalake/internal/testcommon/clients_auth.go +++ b/sdk/storage/azdatalake/internal/testcommon/clients_auth.go @@ -113,7 +113,7 @@ func GetServiceClient(t *testing.T, accountType TestAccountType, options *servic return serviceClient, err } -func GetFilesystemClient(fsName string, t *testing.T, accountType TestAccountType, options *filesystem.ClientOptions) (*filesystem.Client, error) { +func GetFileSystemClient(fsName string, t *testing.T, accountType TestAccountType, options *filesystem.ClientOptions) (*filesystem.Client, error) { if options == nil { options = &filesystem.ClientOptions{} } @@ -178,11 +178,11 @@ func GetDirClient(fsName, dirName string, t *testing.T, accountType TestAccountT return dirClient, err } -func ServiceGetFilesystemClient(filesystemName string, s *service.Client) *filesystem.Client { - return s.NewFilesystemClient(filesystemName) +func ServiceGetFileSystemClient(filesystemName string, s *service.Client) *filesystem.Client { + return s.NewFileSystemClient(filesystemName) } -func DeleteFilesystem(ctx context.Context, _require *require.Assertions, filesystemClient *filesystem.Client) { +func DeleteFileSystem(ctx context.Context, _require *require.Assertions, filesystemClient *filesystem.Client) { _, err := filesystemClient.Delete(ctx, nil) _require.Nil(err) } @@ -207,8 +207,8 @@ func GetGenericConnectionString(accountType TestAccountType) (*string, error) { return &connectionString, nil } -func CreateNewFilesystem(ctx context.Context, _require *require.Assertions, filesystemName string, serviceClient *service.Client) *filesystem.Client { - fsClient := ServiceGetFilesystemClient(filesystemName, serviceClient) +func CreateNewFileSystem(ctx context.Context, _require *require.Assertions, filesystemName string, serviceClient *service.Client) *filesystem.Client { + fsClient := ServiceGetFileSystemClient(filesystemName, serviceClient) _, err := fsClient.Create(ctx, nil) _require.Nil(err) diff --git a/sdk/storage/azdatalake/internal/testcommon/common.go b/sdk/storage/azdatalake/internal/testcommon/common.go index 4cc181b9b297..8c7fb0923b7f 100644 --- a/sdk/storage/azdatalake/internal/testcommon/common.go +++ b/sdk/storage/azdatalake/internal/testcommon/common.go @@ -22,15 +22,15 @@ import ( ) const ( - FilesystemPrefix = "gofs" + FileSystemPrefix = "gofs" FilePrefix = "gotestfile" DirPrefix = "gotestdir" DefaultData = "Godatalakedata" InvalidHeaderErrorSubstring = "invalid header field" // error thrown by the http client ) -func GenerateFilesystemName(testName string) string { - return FilesystemPrefix + GenerateEntityName(testName) +func GenerateFileSystemName(testName string) string { + return FileSystemPrefix + GenerateEntityName(testName) } func GenerateFileName(testName string) string { diff --git a/sdk/storage/azdatalake/lease/client.go b/sdk/storage/azdatalake/lease/client.go index d59dfcf1a7da..8e33db303761 100644 --- a/sdk/storage/azdatalake/lease/client.go +++ b/sdk/storage/azdatalake/lease/client.go @@ -16,39 +16,39 @@ import ( "github.com/Azure/azure-sdk-for-go/sdk/storage/azdatalake/internal/generated" ) -// FilesystemClient provides lease functionality for the underlying filesystem client. -type FilesystemClient struct { +// FileSystemClient provides lease functionality for the underlying filesystem client. +type FileSystemClient struct { containerClient *lease.ContainerClient leaseID *string } -// FilesystemClientOptions contains the optional values when creating a FilesystemClient. -type FilesystemClientOptions = lease.ContainerClientOptions +// FileSystemClientOptions contains the optional values when creating a FileSystemClient. +type FileSystemClientOptions = lease.ContainerClientOptions -// NewFilesystemClient creates a filesystem lease client for the provided filesystem client. +// NewFileSystemClient creates a filesystem lease client for the provided filesystem client. // - client - an instance of a filesystem client // - options - client options; pass nil to accept the default values -func NewFilesystemClient(client *filesystem.Client, options *FilesystemClientOptions) (*FilesystemClient, error) { +func NewFileSystemClient(client *filesystem.Client, options *FileSystemClientOptions) (*FileSystemClient, error) { _, _, containerClient := base.InnerClients((*base.CompositeClient[generated.FileSystemClient, generated.FileSystemClient, container.Client])(client)) containerLeaseClient, err := lease.NewContainerClient(containerClient, options) if err != nil { return nil, exported.ConvertToDFSError(err) } - return &FilesystemClient{ + return &FileSystemClient{ containerClient: containerLeaseClient, leaseID: containerLeaseClient.LeaseID(), }, nil } // LeaseID returns leaseID of the client. -func (c *FilesystemClient) LeaseID() *string { +func (c *FileSystemClient) LeaseID() *string { return c.leaseID } // AcquireLease acquires a lease on the filesystem for write and delete operations. // The lease Duration must be between 15 and 60 seconds, or infinite (-1). // For more information, see https://docs.microsoft.com/rest/api/storageservices/lease-blob. -func (c *FilesystemClient) AcquireLease(ctx context.Context, duration int32, o *FilesystemAcquireOptions) (FilesystemAcquireResponse, error) { +func (c *FileSystemClient) AcquireLease(ctx context.Context, duration int32, o *FileSystemAcquireOptions) (FileSystemAcquireResponse, error) { opts := o.format() resp, err := c.containerClient.AcquireLease(ctx, duration, opts) return resp, exported.ConvertToDFSError(err) @@ -57,7 +57,7 @@ func (c *FilesystemClient) AcquireLease(ctx context.Context, duration int32, o * // BreakLease breaks the filesystem's previously-acquired lease (if it exists). Pass the LeaseBreakDefault (-1) // constant to break a fixed-Duration lease when it expires or an infinite lease immediately. // For more information, see https://docs.microsoft.com/rest/api/storageservices/lease-blob. -func (c *FilesystemClient) BreakLease(ctx context.Context, o *FilesystemBreakOptions) (FilesystemBreakResponse, error) { +func (c *FileSystemClient) BreakLease(ctx context.Context, o *FileSystemBreakOptions) (FileSystemBreakResponse, error) { opts := o.format() resp, err := c.containerClient.BreakLease(ctx, opts) return resp, exported.ConvertToDFSError(err) @@ -65,7 +65,7 @@ func (c *FilesystemClient) BreakLease(ctx context.Context, o *FilesystemBreakOpt // ChangeLease changes the filesystem's lease ID. // For more information, see https://docs.microsoft.com/rest/api/storageservices/lease-blob. -func (c *FilesystemClient) ChangeLease(ctx context.Context, proposedLeaseID string, o *FilesystemChangeOptions) (FilesystemChangeResponse, error) { +func (c *FileSystemClient) ChangeLease(ctx context.Context, proposedLeaseID string, o *FileSystemChangeOptions) (FileSystemChangeResponse, error) { opts := o.format() resp, err := c.containerClient.ChangeLease(ctx, proposedLeaseID, opts) if err != nil { @@ -77,14 +77,14 @@ func (c *FilesystemClient) ChangeLease(ctx context.Context, proposedLeaseID stri // RenewLease renews the filesystem's previously-acquired lease. // For more information, see https://docs.microsoft.com/rest/api/storageservices/lease-blob. -func (c *FilesystemClient) RenewLease(ctx context.Context, o *FilesystemRenewOptions) (FilesystemRenewResponse, error) { +func (c *FileSystemClient) RenewLease(ctx context.Context, o *FileSystemRenewOptions) (FileSystemRenewResponse, error) { opts := o.format() resp, err := c.containerClient.RenewLease(ctx, opts) return resp, exported.ConvertToDFSError(err) } // ReleaseLease releases the filesystem's previously-acquired lease. -func (c *FilesystemClient) ReleaseLease(ctx context.Context, o *FilesystemReleaseOptions) (FilesystemReleaseResponse, error) { +func (c *FileSystemClient) ReleaseLease(ctx context.Context, o *FileSystemReleaseOptions) (FileSystemReleaseResponse, error) { opts := o.format() resp, err := c.containerClient.ReleaseLease(ctx, opts) return resp, exported.ConvertToDFSError(err) diff --git a/sdk/storage/azdatalake/lease/client_test.go b/sdk/storage/azdatalake/lease/client_test.go index 51f2183ea718..9095690f1d15 100644 --- a/sdk/storage/azdatalake/lease/client_test.go +++ b/sdk/storage/azdatalake/lease/client_test.go @@ -68,11 +68,11 @@ func (s *LeaseRecordedTestsSuite) TestFilesystemAcquireLease() { svcClient, err := testcommon.GetServiceClient(s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) - filesystemName := testcommon.GenerateFilesystemName(testName) - filesystemClient := testcommon.CreateNewFilesystem(context.Background(), _require, filesystemName, svcClient) - defer testcommon.DeleteFilesystem(context.Background(), _require, filesystemClient) + filesystemName := testcommon.GenerateFileSystemName(testName) + filesystemClient := testcommon.CreateNewFileSystem(context.Background(), _require, filesystemName, svcClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, filesystemClient) - filesystemLeaseClient, _ := lease.NewFilesystemClient(filesystemClient, &lease.FilesystemClientOptions{ + filesystemLeaseClient, _ := lease.NewFileSystemClient(filesystemClient, &lease.FileSystemClientOptions{ LeaseID: proposedLeaseIDs[0], }) @@ -93,11 +93,11 @@ func (s *LeaseRecordedTestsSuite) TestFilesystemDeleteFilesystemWithoutLeaseId() svcClient, err := testcommon.GetServiceClient(s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) - filesystemName := testcommon.GenerateFilesystemName(testName) - filesystemClient := testcommon.CreateNewFilesystem(context.Background(), _require, filesystemName, svcClient) - defer testcommon.DeleteFilesystem(context.Background(), _require, filesystemClient) + filesystemName := testcommon.GenerateFileSystemName(testName) + filesystemClient := testcommon.CreateNewFileSystem(context.Background(), _require, filesystemName, svcClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, filesystemClient) - filesystemLeaseClient, _ := lease.NewFilesystemClient(filesystemClient, &lease.FilesystemClientOptions{ + filesystemLeaseClient, _ := lease.NewFileSystemClient(filesystemClient, &lease.FileSystemClientOptions{ LeaseID: proposedLeaseIDs[0], }) @@ -130,11 +130,11 @@ func (s *LeaseRecordedTestsSuite) TestFilesystemReleaseLease() { svcClient, err := testcommon.GetServiceClient(s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) - filesystemName := testcommon.GenerateFilesystemName(testName) - filesystemClient := testcommon.CreateNewFilesystem(context.Background(), _require, filesystemName, svcClient) - defer testcommon.DeleteFilesystem(context.Background(), _require, filesystemClient) + filesystemName := testcommon.GenerateFileSystemName(testName) + filesystemClient := testcommon.CreateNewFileSystem(context.Background(), _require, filesystemName, svcClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, filesystemClient) - filesystemLeaseClient, _ := lease.NewFilesystemClient(filesystemClient, &lease.FilesystemClientOptions{ + filesystemLeaseClient, _ := lease.NewFileSystemClient(filesystemClient, &lease.FileSystemClientOptions{ LeaseID: proposedLeaseIDs[0], }) @@ -163,11 +163,11 @@ func (s *LeaseRecordedTestsSuite) TestFilesystemRenewLease() { svcClient, err := testcommon.GetServiceClient(s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) - filesystemName := testcommon.GenerateFilesystemName(testName) - filesystemClient := testcommon.CreateNewFilesystem(context.Background(), _require, filesystemName, svcClient) - defer testcommon.DeleteFilesystem(context.Background(), _require, filesystemClient) + filesystemName := testcommon.GenerateFileSystemName(testName) + filesystemClient := testcommon.CreateNewFileSystem(context.Background(), _require, filesystemName, svcClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, filesystemClient) - filesystemLeaseClient, _ := lease.NewFilesystemClient(filesystemClient, &lease.FilesystemClientOptions{ + filesystemLeaseClient, _ := lease.NewFileSystemClient(filesystemClient, &lease.FileSystemClientOptions{ LeaseID: proposedLeaseIDs[0], }) @@ -193,11 +193,11 @@ func (s *LeaseRecordedTestsSuite) TestFilesystemChangeLease() { svcClient, err := testcommon.GetServiceClient(s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) - fsName := testcommon.GenerateFilesystemName(testName) - fsClient := testcommon.CreateNewFilesystem(context.Background(), _require, fsName, svcClient) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + fsName := testcommon.GenerateFileSystemName(testName) + fsClient := testcommon.CreateNewFileSystem(context.Background(), _require, fsName, svcClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) - fsLeaseClient, _ := lease.NewFilesystemClient(fsClient, &lease.FilesystemClientOptions{ + fsLeaseClient, _ := lease.NewFileSystemClient(fsClient, &lease.FileSystemClientOptions{ LeaseID: proposedLeaseIDs[0], }) @@ -228,9 +228,9 @@ func (s *LeaseRecordedTestsSuite) TestFileAcquireLease() { svcClient, err := testcommon.GetServiceClient(s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) - filesystemName := testcommon.GenerateFilesystemName(testName) - filesystemClient := testcommon.CreateNewFilesystem(context.Background(), _require, filesystemName, svcClient) - defer testcommon.DeleteFilesystem(context.Background(), _require, filesystemClient) + filesystemName := testcommon.GenerateFileSystemName(testName) + filesystemClient := testcommon.CreateNewFileSystem(context.Background(), _require, filesystemName, svcClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, filesystemClient) fileName := testcommon.GenerateFileName(testName) fileClient := testcommon.CreateNewFile(context.Background(), _require, fileName, filesystemClient) @@ -258,9 +258,9 @@ func (s *LeaseRecordedTestsSuite) TestDeleteFileWithoutLeaseId() { svcClient, err := testcommon.GetServiceClient(s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) - filesystemName := testcommon.GenerateFilesystemName(testName) - filesystemClient := testcommon.CreateNewFilesystem(context.Background(), _require, filesystemName, svcClient) - defer testcommon.DeleteFilesystem(context.Background(), _require, filesystemClient) + filesystemName := testcommon.GenerateFileSystemName(testName) + filesystemClient := testcommon.CreateNewFileSystem(context.Background(), _require, filesystemName, svcClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, filesystemClient) fileName := testcommon.GenerateFileName(testName) fileClient := testcommon.CreateNewFile(context.Background(), _require, fileName, filesystemClient) @@ -298,9 +298,9 @@ func (s *LeaseRecordedTestsSuite) TestFileReleaseLease() { svcClient, err := testcommon.GetServiceClient(s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) - filesystemName := testcommon.GenerateFilesystemName(testName) - filesystemClient := testcommon.CreateNewFilesystem(context.Background(), _require, filesystemName, svcClient) - defer testcommon.DeleteFilesystem(context.Background(), _require, filesystemClient) + filesystemName := testcommon.GenerateFileSystemName(testName) + filesystemClient := testcommon.CreateNewFileSystem(context.Background(), _require, filesystemName, svcClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, filesystemClient) fileName := testcommon.GenerateFileName(testName) fileClient := testcommon.CreateNewFile(context.Background(), _require, fileName, filesystemClient) @@ -330,9 +330,9 @@ func (s *LeaseRecordedTestsSuite) TestFileRenewLease() { svcClient, err := testcommon.GetServiceClient(s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) - filesystemName := testcommon.GenerateFilesystemName(testName) - filesystemClient := testcommon.CreateNewFilesystem(context.Background(), _require, filesystemName, svcClient) - defer testcommon.DeleteFilesystem(context.Background(), _require, filesystemClient) + filesystemName := testcommon.GenerateFileSystemName(testName) + filesystemClient := testcommon.CreateNewFileSystem(context.Background(), _require, filesystemName, svcClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, filesystemClient) fileName := testcommon.GenerateFileName(testName) fileClient := testcommon.CreateNewFile(context.Background(), _require, fileName, filesystemClient) @@ -362,9 +362,9 @@ func (s *LeaseRecordedTestsSuite) TestFileChangeLease() { svcClient, err := testcommon.GetServiceClient(s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) - filesystemName := testcommon.GenerateFilesystemName(testName) - filesystemClient := testcommon.CreateNewFilesystem(context.Background(), _require, filesystemName, svcClient) - defer testcommon.DeleteFilesystem(context.Background(), _require, filesystemClient) + filesystemName := testcommon.GenerateFileSystemName(testName) + filesystemClient := testcommon.CreateNewFileSystem(context.Background(), _require, filesystemName, svcClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, filesystemClient) fileName := testcommon.GenerateFileName(testName) fileClient := testcommon.CreateNewFile(context.Background(), _require, fileName, filesystemClient) @@ -398,9 +398,9 @@ func (s *LeaseRecordedTestsSuite) TestDirAcquireLease() { svcClient, err := testcommon.GetServiceClient(s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) - filesystemName := testcommon.GenerateFilesystemName(testName) - filesystemClient := testcommon.CreateNewFilesystem(context.Background(), _require, filesystemName, svcClient) - defer testcommon.DeleteFilesystem(context.Background(), _require, filesystemClient) + filesystemName := testcommon.GenerateFileSystemName(testName) + filesystemClient := testcommon.CreateNewFileSystem(context.Background(), _require, filesystemName, svcClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, filesystemClient) dirName := testcommon.GenerateDirName(testName) dirClient := testcommon.CreateNewDir(context.Background(), _require, dirName, filesystemClient) @@ -428,9 +428,9 @@ func (s *LeaseRecordedTestsSuite) TestDeleteDirWithoutLeaseId() { svcClient, err := testcommon.GetServiceClient(s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) - filesystemName := testcommon.GenerateFilesystemName(testName) - filesystemClient := testcommon.CreateNewFilesystem(context.Background(), _require, filesystemName, svcClient) - defer testcommon.DeleteFilesystem(context.Background(), _require, filesystemClient) + filesystemName := testcommon.GenerateFileSystemName(testName) + filesystemClient := testcommon.CreateNewFileSystem(context.Background(), _require, filesystemName, svcClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, filesystemClient) dirName := testcommon.GenerateDirName(testName) dirClient := testcommon.CreateNewDir(context.Background(), _require, dirName, filesystemClient) @@ -468,9 +468,9 @@ func (s *LeaseRecordedTestsSuite) TestDirReleaseLease() { svcClient, err := testcommon.GetServiceClient(s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) - filesystemName := testcommon.GenerateFilesystemName(testName) - filesystemClient := testcommon.CreateNewFilesystem(context.Background(), _require, filesystemName, svcClient) - defer testcommon.DeleteFilesystem(context.Background(), _require, filesystemClient) + filesystemName := testcommon.GenerateFileSystemName(testName) + filesystemClient := testcommon.CreateNewFileSystem(context.Background(), _require, filesystemName, svcClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, filesystemClient) DirName := testcommon.GenerateDirName(testName) DirClient := testcommon.CreateNewDir(context.Background(), _require, DirName, filesystemClient) @@ -500,9 +500,9 @@ func (s *LeaseRecordedTestsSuite) TestDirRenewLease() { svcClient, err := testcommon.GetServiceClient(s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) - filesystemName := testcommon.GenerateFilesystemName(testName) - filesystemClient := testcommon.CreateNewFilesystem(context.Background(), _require, filesystemName, svcClient) - defer testcommon.DeleteFilesystem(context.Background(), _require, filesystemClient) + filesystemName := testcommon.GenerateFileSystemName(testName) + filesystemClient := testcommon.CreateNewFileSystem(context.Background(), _require, filesystemName, svcClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, filesystemClient) dirName := testcommon.GenerateDirName(testName) dirClient := testcommon.CreateNewDir(context.Background(), _require, dirName, filesystemClient) @@ -532,9 +532,9 @@ func (s *LeaseRecordedTestsSuite) TestDirChangeLease() { svcClient, err := testcommon.GetServiceClient(s.T(), testcommon.TestAccountDatalake, nil) _require.NoError(err) - filesystemName := testcommon.GenerateFilesystemName(testName) - filesystemClient := testcommon.CreateNewFilesystem(context.Background(), _require, filesystemName, svcClient) - defer testcommon.DeleteFilesystem(context.Background(), _require, filesystemClient) + filesystemName := testcommon.GenerateFileSystemName(testName) + filesystemClient := testcommon.CreateNewFileSystem(context.Background(), _require, filesystemName, svcClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, filesystemClient) dirName := testcommon.GenerateDirName(testName) dirClient := testcommon.CreateNewDir(context.Background(), _require, dirName, filesystemClient) diff --git a/sdk/storage/azdatalake/lease/examples_test.go b/sdk/storage/azdatalake/lease/examples_test.go new file mode 100644 index 000000000000..adc847cb3050 --- /dev/null +++ b/sdk/storage/azdatalake/lease/examples_test.go @@ -0,0 +1,92 @@ +//go:build go1.18 +// +build go1.18 + +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT License. See License.txt in the project root for license information. + +package lease_test + +import ( + "context" + "fmt" + "github.com/Azure/azure-sdk-for-go/sdk/storage/azdatalake" + "github.com/Azure/azure-sdk-for-go/sdk/storage/azdatalake/filesystem" + "log" + "os" + + "github.com/Azure/azure-sdk-for-go/sdk/azcore/to" + "github.com/Azure/azure-sdk-for-go/sdk/storage/azdatalake/lease" +) + +func handleError(err error) { + if err != nil { + log.Fatal(err.Error()) + } +} + +// This example shows how to perform various lease operations on a filesystem. +// The same lease operations can be performed on individual files as well. +// A lease on a filesystem prevents it from being deleted by others, while a lease on a file +// protects it from both modifications and deletions. +func Example_lease_FileSystemClient_AcquireLease() { + // From the Azure portal, get your Storage account's name and account key. + accountName, accountKey := os.Getenv("AZURE_STORAGE_ACCOUNT_NAME"), os.Getenv("AZURE_STORAGE_ACCOUNT_KEY") + + // Use your Storage account's name and key to create a credential object; this is used to access your account. + credential, err := azdatalake.NewSharedKeyCredential(accountName, accountKey) + handleError(err) + + // Create an fsClient object that wraps the filesystem's URL and a default pipeline. + filesystemURL := fmt.Sprintf("https://%s.dfs.core.windows.net/myfs", accountName) + fsClient, err := filesystem.NewClientWithSharedKeyCredential(filesystemURL, credential, nil) + handleError(err) + + // Create a unique ID for the lease + // A lease ID can be any valid GUID string format. To generate UUIDs, consider the github.com/google/uuid package + leaseID := "36b1a876-cf98-4eb2-a5c3-6d68489658ff" + filesystemLeaseClient, err := lease.NewFileSystemClient(fsClient, + &lease.FileSystemClientOptions{LeaseID: to.Ptr(leaseID)}) + handleError(err) + + // Now acquire a lease on the filesystem. + // You can choose to pass an empty string for proposed ID so that the service automatically assigns one for you. + duration := int32(60) + acquireLeaseResponse, err := filesystemLeaseClient.AcquireLease(context.TODO(), duration, nil) + handleError(err) + fmt.Println("The filesystem is leased for delete operations with lease ID", *acquireLeaseResponse.LeaseID) + + // The filesystem cannot be deleted without providing the lease ID. + _, err = fsClient.Delete(context.TODO(), nil) + if err == nil { + log.Fatal("delete should have failed") + } + + fmt.Println("The filesystem cannot be deleted while there is an active lease") + + // We can release the lease now and the filesystem can be deleted. + _, err = filesystemLeaseClient.ReleaseLease(context.TODO(), nil) + handleError(err) + fmt.Println("The lease on the filesystem is now released") + + // AcquireLease a lease again to perform other operations. + // Duration is still 60 + acquireLeaseResponse, err = filesystemLeaseClient.AcquireLease(context.TODO(), duration, nil) + handleError(err) + fmt.Println("The filesystem is leased again with lease ID", *acquireLeaseResponse.LeaseID) + + // We can change the ID of an existing lease. + newLeaseID := "6b3e65e5-e1bb-4a3f-8b72-13e9bc9cd3bf" + changeLeaseResponse, err := filesystemLeaseClient.ChangeLease(context.TODO(), newLeaseID, nil) + handleError(err) + fmt.Println("The lease ID was changed to", *changeLeaseResponse.LeaseID) + + // The lease can be renewed. + renewLeaseResponse, err := filesystemLeaseClient.RenewLease(context.TODO(), nil) + handleError(err) + fmt.Println("The lease was renewed with the same ID", *renewLeaseResponse.LeaseID) + + // Finally, the lease can be broken, and we could prevent others from acquiring a lease for a period of time + _, err = filesystemLeaseClient.BreakLease(context.TODO(), &lease.FileSystemBreakOptions{BreakPeriod: to.Ptr(int32(60))}) + handleError(err) + fmt.Println("The lease was broken, and nobody can acquire a lease for 60 seconds") +} diff --git a/sdk/storage/azdatalake/lease/models.go b/sdk/storage/azdatalake/lease/models.go index ce552403c698..fced59eab55c 100644 --- a/sdk/storage/azdatalake/lease/models.go +++ b/sdk/storage/azdatalake/lease/models.go @@ -12,12 +12,13 @@ import ( "github.com/Azure/azure-sdk-for-go/sdk/storage/azdatalake/internal/exported" ) -// FilesystemAcquireOptions contains the optional parameters for the LeaseClient.AcquireLease method. -type FilesystemAcquireOptions struct { +// FileSystemAcquireOptions contains the optional parameters for the LeaseClient.AcquireLease method. +type FileSystemAcquireOptions struct { + // ModifiedAccessConditions contains optional parameters to access filesystem. ModifiedAccessConditions *ModifiedAccessConditions } -func (o *FilesystemAcquireOptions) format() *lease.ContainerAcquireOptions { +func (o *FileSystemAcquireOptions) format() *lease.ContainerAcquireOptions { if o == nil || o.ModifiedAccessConditions == nil { return nil } @@ -31,13 +32,15 @@ func (o *FilesystemAcquireOptions) format() *lease.ContainerAcquireOptions { } } -// FilesystemBreakOptions contains the optional parameters for the LeaseClient.BreakLease method. -type FilesystemBreakOptions struct { - BreakPeriod *int32 +// FileSystemBreakOptions contains the optional parameters for the LeaseClient.BreakLease method. +type FileSystemBreakOptions struct { + // BreakPeriod is the proposed duration of seconds that the lease should continue before it is broken. + BreakPeriod *int32 + // ModifiedAccessConditions contains optional parameters to access filesystem. ModifiedAccessConditions *ModifiedAccessConditions } -func (o *FilesystemBreakOptions) format() *lease.ContainerBreakOptions { +func (o *FileSystemBreakOptions) format() *lease.ContainerBreakOptions { opts := &lease.ContainerBreakOptions{} if o == nil { return opts @@ -57,12 +60,13 @@ func (o *FilesystemBreakOptions) format() *lease.ContainerBreakOptions { } } -// FilesystemChangeOptions contains the optional parameters for the LeaseClient.ChangeLease method. -type FilesystemChangeOptions struct { +// FileSystemChangeOptions contains the optional parameters for the LeaseClient.ChangeLease method. +type FileSystemChangeOptions struct { + // ModifiedAccessConditions contains optional parameters to access filesystem. ModifiedAccessConditions *ModifiedAccessConditions } -func (o *FilesystemChangeOptions) format() *lease.ContainerChangeOptions { +func (o *FileSystemChangeOptions) format() *lease.ContainerChangeOptions { if o == nil || o.ModifiedAccessConditions == nil { return nil } @@ -76,11 +80,13 @@ func (o *FilesystemChangeOptions) format() *lease.ContainerChangeOptions { } } -type FilesystemReleaseOptions struct { +// FileSystemReleaseOptions contains the optional parameters for the LeaseClient.ReleaseLease method. +type FileSystemReleaseOptions struct { + // ModifiedAccessConditions contains optional parameters to access filesystem. ModifiedAccessConditions *ModifiedAccessConditions } -func (o *FilesystemReleaseOptions) format() *lease.ContainerReleaseOptions { +func (o *FileSystemReleaseOptions) format() *lease.ContainerReleaseOptions { if o == nil || o.ModifiedAccessConditions == nil { return nil } @@ -94,11 +100,13 @@ func (o *FilesystemReleaseOptions) format() *lease.ContainerReleaseOptions { } } -type FilesystemRenewOptions struct { +// FileSystemRenewOptions contains the optional parameters for the LeaseClient.RenewLease method. +type FileSystemRenewOptions struct { + // ModifiedAccessConditions contains optional parameters to access filesystem. ModifiedAccessConditions *ModifiedAccessConditions } -func (o *FilesystemRenewOptions) format() *lease.ContainerRenewOptions { +func (o *FileSystemRenewOptions) format() *lease.ContainerRenewOptions { if o == nil || o.ModifiedAccessConditions == nil { return nil } @@ -114,6 +122,7 @@ func (o *FilesystemRenewOptions) format() *lease.ContainerRenewOptions { // PathAcquireOptions contains the optional parameters for the LeaseClient.AcquireLease method. type PathAcquireOptions struct { + // ModifiedAccessConditions contains optional parameters to access path. ModifiedAccessConditions *ModifiedAccessConditions } @@ -133,7 +142,9 @@ func (o *PathAcquireOptions) format() *lease.BlobAcquireOptions { // PathBreakOptions contains the optional parameters for the LeaseClient.BreakLease method. type PathBreakOptions struct { - BreakPeriod *int32 + // BreakPeriod is the proposed duration of seconds that the lease should continue before it is broken. + BreakPeriod *int32 + // ModifiedAccessConditions contains optional parameters to access path. ModifiedAccessConditions *ModifiedAccessConditions } @@ -159,6 +170,7 @@ func (o *PathBreakOptions) format() *lease.BlobBreakOptions { // PathChangeOptions contains the optional parameters for the LeaseClient.ChangeLease method. type PathChangeOptions struct { + // ModifiedAccessConditions contains optional parameters to access path. ModifiedAccessConditions *ModifiedAccessConditions } @@ -176,7 +188,9 @@ func (o *PathChangeOptions) format() *lease.BlobChangeOptions { } } +// PathReleaseOptions contains the optional parameters for the LeaseClient.ReleaseLease method. type PathReleaseOptions struct { + // ModifiedAccessConditions contains optional parameters to access path. ModifiedAccessConditions *ModifiedAccessConditions } @@ -194,7 +208,9 @@ func (o *PathReleaseOptions) format() *lease.BlobReleaseOptions { } } +// PathRenewOptions contains the optional parameters for the LeaseClient.RenewLease method. type PathRenewOptions struct { + // ModifiedAccessConditions contains optional parameters to access path. ModifiedAccessConditions *ModifiedAccessConditions } diff --git a/sdk/storage/azdatalake/lease/responses.go b/sdk/storage/azdatalake/lease/responses.go index 1ff60df3b227..55a174ccd6f8 100644 --- a/sdk/storage/azdatalake/lease/responses.go +++ b/sdk/storage/azdatalake/lease/responses.go @@ -8,20 +8,20 @@ package lease import "github.com/Azure/azure-sdk-for-go/sdk/storage/azblob/lease" -// FilesystemAcquireResponse contains the response from method FilesystemClient.AcquireLease. -type FilesystemAcquireResponse = lease.ContainerAcquireResponse +// FileSystemAcquireResponse contains the response from method FileSystemClient.AcquireLease. +type FileSystemAcquireResponse = lease.ContainerAcquireResponse -// FilesystemBreakResponse contains the response from method FilesystemClient.BreakLease. -type FilesystemBreakResponse = lease.ContainerBreakResponse +// FileSystemBreakResponse contains the response from method FileSystemClient.BreakLease. +type FileSystemBreakResponse = lease.ContainerBreakResponse -// FilesystemChangeResponse contains the response from method FilesystemClient.ChangeLease. -type FilesystemChangeResponse = lease.ContainerChangeResponse +// FileSystemChangeResponse contains the response from method FileSystemClient.ChangeLease. +type FileSystemChangeResponse = lease.ContainerChangeResponse -// FilesystemReleaseResponse contains the response from method FilesystemClient.ReleaseLease. -type FilesystemReleaseResponse = lease.ContainerReleaseResponse +// FileSystemReleaseResponse contains the response from method FileSystemClient.ReleaseLease. +type FileSystemReleaseResponse = lease.ContainerReleaseResponse -// FilesystemRenewResponse contains the response from method FilesystemClient.RenewLease. -type FilesystemRenewResponse = lease.ContainerRenewResponse +// FileSystemRenewResponse contains the response from method FileSystemClient.RenewLease. +type FileSystemRenewResponse = lease.ContainerRenewResponse // PathAcquireResponse contains the response from method PathClient.AcquireLease. type PathAcquireResponse = lease.BlobAcquireResponse diff --git a/sdk/storage/azdatalake/sas/query_params.go b/sdk/storage/azdatalake/sas/query_params.go index ed3cd252d93f..7806687833c0 100644 --- a/sdk/storage/azdatalake/sas/query_params.go +++ b/sdk/storage/azdatalake/sas/query_params.go @@ -14,9 +14,9 @@ import ( "time" ) -// timeFormat represents the format of a SAS start or expiry time. Use it when formatting/parsing a time.Time. +// TimeFormat represents the format of a SAS start or expiry time. Use it when formatting/parsing a time.Time. const ( - timeFormat = "2006-01-02T15:04:05Z" // "2017-07-27T00:00:00Z" // ISO 8601 + TimeFormat = "2006-01-02T15:04:05Z" // "2017-07-27T00:00:00Z" // ISO 8601 ) var ( @@ -26,7 +26,7 @@ var ( // TimeFormats ISO 8601 format. // Please refer to https://docs.microsoft.com/en-us/rest/api/storageservices/constructing-a-service-sas for more details. -var timeFormats = []string{"2006-01-02T15:04:05.0000000Z", timeFormat, "2006-01-02T15:04Z", "2006-01-02"} +var timeFormats = []string{"2006-01-02T15:04:05.0000000Z", TimeFormat, "2006-01-02T15:04Z", "2006-01-02"} // Protocol indicates the http/https. type Protocol string @@ -55,7 +55,7 @@ func formatTimesForSigning(startTime, expiryTime time.Time) (string, string) { // formatTimeWithDefaultFormat format time with ISO 8601 in "yyyy-MM-ddTHH:mm:ssZ". func formatTimeWithDefaultFormat(t *time.Time) string { - return formatTime(t, timeFormat) // By default, "yyyy-MM-ddTHH:mm:ssZ" is used + return formatTime(t, TimeFormat) // By default, "yyyy-MM-ddTHH:mm:ssZ" is used } // formatTime format time with given format, use ISO 8601 in "yyyy-MM-ddTHH:mm:ssZ" by default. @@ -63,7 +63,7 @@ func formatTime(t *time.Time, format string) string { if format != "" { return t.Format(format) } - return t.Format(timeFormat) // By default, "yyyy-MM-ddTHH:mm:ssZ" is used + return t.Format(TimeFormat) // By default, "yyyy-MM-ddTHH:mm:ssZ" is used } // ParseTime try to parse a SAS time string. @@ -314,8 +314,8 @@ func (p *QueryParameters) Encode() string { if p.signedOID != "" { v.Add("skoid", p.signedOID) v.Add("sktid", p.signedTID) - v.Add("skt", p.signedStart.Format(timeFormat)) - v.Add("ske", p.signedExpiry.Format(timeFormat)) + v.Add("skt", p.signedStart.Format(TimeFormat)) + v.Add("ske", p.signedExpiry.Format(TimeFormat)) v.Add("sks", p.signedService) v.Add("skv", p.signedVersion) } @@ -403,9 +403,9 @@ func NewQueryParameters(values url.Values) QueryParameters { case "sktid": p.signedTID = val case "skt": - p.signedStart, _ = time.Parse(timeFormat, val) + p.signedStart, _ = time.Parse(TimeFormat, val) case "ske": - p.signedExpiry, _ = time.Parse(timeFormat, val) + p.signedExpiry, _ = time.Parse(TimeFormat, val) case "sks": p.signedService = val case "skv": @@ -480,9 +480,9 @@ func newQueryParameters(values url.Values, deleteSASParametersFromValues bool) Q case "sktid": p.signedTID = val case "skt": - p.signedStart, _ = time.Parse(timeFormat, val) + p.signedStart, _ = time.Parse(TimeFormat, val) case "ske": - p.signedExpiry, _ = time.Parse(timeFormat, val) + p.signedExpiry, _ = time.Parse(TimeFormat, val) case "sks": p.signedService = val case "skv": diff --git a/sdk/storage/azdatalake/sas/service.go b/sdk/storage/azdatalake/sas/service.go index 92ccaa8101a3..22784206a667 100644 --- a/sdk/storage/azdatalake/sas/service.go +++ b/sdk/storage/azdatalake/sas/service.go @@ -24,15 +24,15 @@ type DatalakeSignatureValues struct { Protocol Protocol `param:"spr"` // See the Protocol* constants StartTime time.Time `param:"st"` // Not specified if IsZero ExpiryTime time.Time `param:"se"` // Not specified if IsZero - Permissions string `param:"sp"` // Create by initializing FilesystemPermissions, FilePermissions or DirectoryPermissions and then call String() + Permissions string `param:"sp"` // Create by initializing FileSystemPermissions, FilePermissions or DirectoryPermissions and then call String() IPRange IPRange `param:"sip"` Identifier string `param:"si"` - FilesystemName string - // Use "" to create a Filesystem SAS + FileSystemName string + // Use "" to create a FileSystem SAS // DirectoryPath will set this to "" if it is passed FilePath string // Not nil for a directory SAS (ie sr=d) - // Use "" to create a Filesystem SAS + // Use "" to create a FileSystem SAS DirectoryPath string CacheControl string // rscc ContentDisposition string // rscd @@ -88,7 +88,7 @@ func (v DatalakeSignatureValues) SignWithSharedKey(sharedKeyCredential *SharedKe v.Permissions, startTime, expiryTime, - getCanonicalName(sharedKeyCredential.AccountName(), v.FilesystemName, v.FilePath, v.DirectoryPath), + getCanonicalName(sharedKeyCredential.AccountName(), v.FileSystemName, v.FilePath, v.DirectoryPath), signedIdentifier, v.IPRange.String(), string(v.Protocol), @@ -157,7 +157,7 @@ func (v DatalakeSignatureValues) SignWithUserDelegation(userDelegationCredential } // make sure the permission characters are in the correct order if resource == "c" { - perms, err := parseFilesystemPermissions(v.Permissions) + perms, err := parseFileSystemPermissions(v.Permissions) if err != nil { return QueryParameters{}, err } @@ -183,7 +183,7 @@ func (v DatalakeSignatureValues) SignWithUserDelegation(userDelegationCredential v.Permissions, startTime, expiryTime, - getCanonicalName(exported.GetAccountName(userDelegationCredential), v.FilesystemName, v.FilePath, v.DirectoryPath), + getCanonicalName(exported.GetAccountName(userDelegationCredential), v.FileSystemName, v.FilePath, v.DirectoryPath), *udk.SignedOID, *udk.SignedTID, udkStart, @@ -259,17 +259,17 @@ func getCanonicalName(account string, filesystemName string, fileName string, di return strings.Join(elements, "") } -// FilesystemPermissions type simplifies creating the permissions string for an Azure Storage container SAS. +// FileSystemPermissions type simplifies creating the permissions string for an Azure Storage container SAS. // Initialize an instance of this type and then call its String method to set BlobSignatureValues' Permissions field. // All permissions descriptions can be found here: https://docs.microsoft.com/en-us/rest/api/storageservices/create-service-sas#permissions-for-a-directory-container-or-blob -type FilesystemPermissions struct { +type FileSystemPermissions struct { Read, Add, Create, Write, Delete, List, Move bool Execute, ModifyOwnership, ModifyPermissions bool // Meant for hierarchical namespace accounts } // String produces the SAS permissions string for an Azure Storage container. // Call this method to set BlobSignatureValues' Permissions field. -func (p *FilesystemPermissions) String() string { +func (p *FileSystemPermissions) String() string { var b bytes.Buffer if p.Read { b.WriteRune('r') @@ -305,8 +305,8 @@ func (p *FilesystemPermissions) String() string { } // Parse initializes ContainerPermissions' fields from a string. -func parseFilesystemPermissions(s string) (FilesystemPermissions, error) { - p := FilesystemPermissions{} // Clear the flags +func parseFileSystemPermissions(s string) (FileSystemPermissions, error) { + p := FileSystemPermissions{} // Clear the flags for _, r := range s { switch r { case 'r': @@ -330,7 +330,7 @@ func parseFilesystemPermissions(s string) (FilesystemPermissions, error) { case 'p': p.ModifyPermissions = true default: - return FilesystemPermissions{}, fmt.Errorf("invalid permission: '%v'", r) + return FileSystemPermissions{}, fmt.Errorf("invalid permission: '%v'", r) } } return p, nil diff --git a/sdk/storage/azdatalake/sas/service_test.go b/sdk/storage/azdatalake/sas/service_test.go index 186eb7e43a9f..be33f9f68035 100644 --- a/sdk/storage/azdatalake/sas/service_test.go +++ b/sdk/storage/azdatalake/sas/service_test.go @@ -11,22 +11,22 @@ import ( "testing" ) -func TestFilesystemPermissions_String(t *testing.T) { +func TestFileSystemPermissions_String(t *testing.T) { testdata := []struct { - input FilesystemPermissions + input FileSystemPermissions expected string }{ - {input: FilesystemPermissions{Read: true}, expected: "r"}, - {input: FilesystemPermissions{Add: true}, expected: "a"}, - {input: FilesystemPermissions{Create: true}, expected: "c"}, - {input: FilesystemPermissions{Write: true}, expected: "w"}, - {input: FilesystemPermissions{Delete: true}, expected: "d"}, - {input: FilesystemPermissions{List: true}, expected: "l"}, - {input: FilesystemPermissions{Move: true}, expected: "m"}, - {input: FilesystemPermissions{Execute: true}, expected: "e"}, - {input: FilesystemPermissions{ModifyOwnership: true}, expected: "o"}, - {input: FilesystemPermissions{ModifyPermissions: true}, expected: "p"}, - {input: FilesystemPermissions{ + {input: FileSystemPermissions{Read: true}, expected: "r"}, + {input: FileSystemPermissions{Add: true}, expected: "a"}, + {input: FileSystemPermissions{Create: true}, expected: "c"}, + {input: FileSystemPermissions{Write: true}, expected: "w"}, + {input: FileSystemPermissions{Delete: true}, expected: "d"}, + {input: FileSystemPermissions{List: true}, expected: "l"}, + {input: FileSystemPermissions{Move: true}, expected: "m"}, + {input: FileSystemPermissions{Execute: true}, expected: "e"}, + {input: FileSystemPermissions{ModifyOwnership: true}, expected: "o"}, + {input: FileSystemPermissions{ModifyPermissions: true}, expected: "p"}, + {input: FileSystemPermissions{ Read: true, Add: true, Create: true, @@ -44,22 +44,22 @@ func TestFilesystemPermissions_String(t *testing.T) { } } -func TestFilesystemPermissions_Parse(t *testing.T) { +func TestFileSystemPermissions_Parse(t *testing.T) { testdata := []struct { input string - expected FilesystemPermissions + expected FileSystemPermissions }{ - {expected: FilesystemPermissions{Read: true}, input: "r"}, - {expected: FilesystemPermissions{Add: true}, input: "a"}, - {expected: FilesystemPermissions{Create: true}, input: "c"}, - {expected: FilesystemPermissions{Write: true}, input: "w"}, - {expected: FilesystemPermissions{Delete: true}, input: "d"}, - {expected: FilesystemPermissions{List: true}, input: "l"}, - {expected: FilesystemPermissions{Move: true}, input: "m"}, - {expected: FilesystemPermissions{Execute: true}, input: "e"}, - {expected: FilesystemPermissions{ModifyOwnership: true}, input: "o"}, - {expected: FilesystemPermissions{ModifyPermissions: true}, input: "p"}, - {expected: FilesystemPermissions{ + {expected: FileSystemPermissions{Read: true}, input: "r"}, + {expected: FileSystemPermissions{Add: true}, input: "a"}, + {expected: FileSystemPermissions{Create: true}, input: "c"}, + {expected: FileSystemPermissions{Write: true}, input: "w"}, + {expected: FileSystemPermissions{Delete: true}, input: "d"}, + {expected: FileSystemPermissions{List: true}, input: "l"}, + {expected: FileSystemPermissions{Move: true}, input: "m"}, + {expected: FileSystemPermissions{Execute: true}, input: "e"}, + {expected: FileSystemPermissions{ModifyOwnership: true}, input: "o"}, + {expected: FileSystemPermissions{ModifyPermissions: true}, input: "p"}, + {expected: FileSystemPermissions{ Read: true, Add: true, Create: true, @@ -71,7 +71,7 @@ func TestFilesystemPermissions_Parse(t *testing.T) { ModifyOwnership: true, ModifyPermissions: true, }, input: "racwdlmeop"}, - {expected: FilesystemPermissions{ + {expected: FileSystemPermissions{ Read: true, Add: true, Create: true, @@ -85,14 +85,14 @@ func TestFilesystemPermissions_Parse(t *testing.T) { }, input: "cpwmreodal"}, // Wrong order parses correctly } for _, c := range testdata { - permissions, err := parseFilesystemPermissions(c.input) + permissions, err := parseFileSystemPermissions(c.input) require.Nil(t, err) require.Equal(t, c.expected, permissions) } } -func TestFilesystemPermissions_ParseNegative(t *testing.T) { - _, err := parseFilesystemPermissions("cpwmreodalz") // Here 'z' is invalid +func TestFileSystemPermissions_ParseNegative(t *testing.T) { + _, err := parseFileSystemPermissions("cpwmreodalz") // Here 'z' is invalid require.NotNil(t, err) require.Contains(t, err.Error(), "122") } @@ -177,6 +177,39 @@ func TestFilePermissions_Parse(t *testing.T) { } } +func TestDirectoryPermissions_String(t *testing.T) { + testdata := []struct { + input DirectoryPermissions + expected string + }{ + {input: DirectoryPermissions{Read: true}, expected: "r"}, + {input: DirectoryPermissions{Add: true}, expected: "a"}, + {input: DirectoryPermissions{Create: true}, expected: "c"}, + {input: DirectoryPermissions{Write: true}, expected: "w"}, + {input: DirectoryPermissions{Delete: true}, expected: "d"}, + {input: DirectoryPermissions{List: true}, expected: "l"}, + {input: DirectoryPermissions{Move: true}, expected: "m"}, + {input: DirectoryPermissions{Execute: true}, expected: "e"}, + {input: DirectoryPermissions{Ownership: true}, expected: "o"}, + {input: DirectoryPermissions{Permissions: true}, expected: "p"}, + {input: DirectoryPermissions{ + Read: true, + Add: true, + Create: true, + Write: true, + Delete: true, + List: true, + Move: true, + Execute: true, + Ownership: true, + Permissions: true, + }, expected: "racwdlmeop"}, + } + for _, c := range testdata { + require.Equal(t, c.expected, c.input.String()) + } +} + func TestParsePermissionsNegative(t *testing.T) { _, err := parsePathPermissions("awecrdlfmo") // Here 'f' is invalid require.NotNil(t, err) diff --git a/sdk/storage/azdatalake/sas/url_parts.go b/sdk/storage/azdatalake/sas/url_parts.go index 8af46a1329ad..bef6a601e451 100644 --- a/sdk/storage/azdatalake/sas/url_parts.go +++ b/sdk/storage/azdatalake/sas/url_parts.go @@ -25,7 +25,7 @@ type URLParts struct { Scheme string // Ex: "https://" Host string // Ex: "account.blob.core.windows.net", "10.132.141.33", "10.132.141.33:80" IPEndpointStyleInfo IPEndpointStyleInfo - FilesystemName string // "" if no container + FileSystemName string // "" if no container PathName string // "" if no blob SAS QueryParameters UnparsedParams string @@ -62,9 +62,9 @@ func ParseURL(u string) (URLParts, error) { filesystemEndIndex := strings.Index(path, "/") // Find the next slash (if it exists) if filesystemEndIndex == -1 { // Slash not found; path has container name & no blob name - up.FilesystemName = path + up.FileSystemName = path } else { - up.FilesystemName = path[:filesystemEndIndex] // The container name is the part between the slashes + up.FileSystemName = path[:filesystemEndIndex] // The container name is the part between the slashes up.PathName = path[filesystemEndIndex+1:] // The blob name is after the container slash } } @@ -84,8 +84,8 @@ func (up URLParts) String() string { path += "/" + up.IPEndpointStyleInfo.AccountName } // Concatenate container & blob names (if they exist) - if up.FilesystemName != "" { - path += "/" + up.FilesystemName + if up.FileSystemName != "" { + path += "/" + up.FileSystemName if up.PathName != "" { path += "/" + up.PathName } diff --git a/sdk/storage/azdatalake/sas/url_parts_test.go b/sdk/storage/azdatalake/sas/url_parts_test.go index f4694b148056..32fa9569d957 100644 --- a/sdk/storage/azdatalake/sas/url_parts_test.go +++ b/sdk/storage/azdatalake/sas/url_parts_test.go @@ -27,7 +27,7 @@ func TestParseURLIPStyle(t *testing.T) { require.Equal(t, blobURLParts.Scheme, "https") require.Equal(t, blobURLParts.Host, "127.0.0.1:5000") require.Equal(t, blobURLParts.IPEndpointStyleInfo.AccountName, "fakestorageaccount") - require.Equal(t, blobURLParts.FilesystemName, "fakecontainer") + require.Equal(t, blobURLParts.FileSystemName, "fakecontainer") urlWithIP = "https://127.0.0.1:5000/fakestorageaccount/fakecontainer/fakeblob" blobURLParts, err = ParseURL(urlWithIP) @@ -35,7 +35,7 @@ func TestParseURLIPStyle(t *testing.T) { require.Equal(t, blobURLParts.Scheme, "https") require.Equal(t, blobURLParts.Host, "127.0.0.1:5000") require.Equal(t, blobURLParts.IPEndpointStyleInfo.AccountName, "fakestorageaccount") - require.Equal(t, blobURLParts.FilesystemName, "fakecontainer") + require.Equal(t, blobURLParts.FileSystemName, "fakecontainer") require.Equal(t, blobURLParts.PathName, "fakeblob") } @@ -54,7 +54,7 @@ func TestParseURL(t *testing.T) { require.Equal(t, blobURLParts.Scheme, "https") require.Equal(t, blobURLParts.Host, host) - require.Equal(t, blobURLParts.FilesystemName, testContainer) + require.Equal(t, blobURLParts.FileSystemName, testContainer) validateSAS(t, sasStr, blobURLParts.SAS) } @@ -66,7 +66,7 @@ func TestParseURL(t *testing.T) { require.Equal(t, blobURLParts.Scheme, "https") require.Equal(t, blobURLParts.Host, host) - require.Equal(t, blobURLParts.FilesystemName, testContainer) + require.Equal(t, blobURLParts.FileSystemName, testContainer) validateSAS(t, sasStr, blobURLParts.SAS) } diff --git a/sdk/storage/azdatalake/service/client.go b/sdk/storage/azdatalake/service/client.go index 9a20d5fdf076..3a5f0c734ab5 100644 --- a/sdk/storage/azdatalake/service/client.go +++ b/sdk/storage/azdatalake/service/client.go @@ -113,7 +113,7 @@ func NewClientWithSharedKeyCredential(serviceURL string, cred *SharedKeyCredenti blobServiceClientOpts := service.ClientOptions{ ClientOptions: options.ClientOptions, } - blobSharedKey, err := cred.ConvertToBlobSharedKey() + blobSharedKey, err := exported.ConvertToBlobSharedKey(cred) if err != nil { return nil, err } @@ -147,12 +147,12 @@ func (s *Client) getClientOptions() *base.ClientOptions { return base.GetCompositeClientOptions((*base.CompositeClient[generated.ServiceClient, generated_blob.ServiceClient, service.Client])(s)) } -// NewFilesystemClient creates a new filesystem.Client object by concatenating filesystemName to the end of this Client's URL. +// NewFileSystemClient creates a new filesystem.Client object by concatenating filesystemName to the end of this Client's URL. // The new filesystem.Client uses the same request policy pipeline as the Client. -func (s *Client) NewFilesystemClient(filesystemName string) *filesystem.Client { +func (s *Client) NewFileSystemClient(filesystemName string) *filesystem.Client { filesystemURL := runtime.JoinPaths(s.generatedServiceClientWithDFS().Endpoint(), filesystemName) containerURL, filesystemURL := shared.GetURLs(filesystemURL) - return (*filesystem.Client)(base.NewFilesystemClient(filesystemURL, containerURL, s.serviceClient().NewContainerClient(filesystemName), s.generatedServiceClientWithDFS().InternalClient().WithClientName(shared.FilesystemClient), s.sharedKey(), s.identityCredential(), s.getClientOptions())) + return (*filesystem.Client)(base.NewFileSystemClient(filesystemURL, containerURL, s.serviceClient().NewContainerClient(filesystemName), s.generatedServiceClientWithDFS().InternalClient().WithClientName(shared.FileSystemClient), s.sharedKey(), s.identityCredential(), s.getClientOptions())) } // GetUserDelegationCredential obtains a UserDelegationKey object using the base ServiceURL object. @@ -205,23 +205,23 @@ func (s *Client) BlobURL() string { return s.generatedServiceClientWithBlob().Endpoint() } -// CreateFilesystem creates a new filesystem under the specified account. (blob3) -func (s *Client) CreateFilesystem(ctx context.Context, filesystem string, options *CreateFilesystemOptions) (CreateFilesystemResponse, error) { - filesystemClient := s.NewFilesystemClient(filesystem) +// CreateFileSystem creates a new filesystem under the specified account. +func (s *Client) CreateFileSystem(ctx context.Context, filesystem string, options *CreateFileSystemOptions) (CreateFileSystemResponse, error) { + filesystemClient := s.NewFileSystemClient(filesystem) resp, err := filesystemClient.Create(ctx, options) err = exported.ConvertToDFSError(err) return resp, err } -// DeleteFilesystem deletes the specified filesystem. (blob3) -func (s *Client) DeleteFilesystem(ctx context.Context, filesystem string, options *DeleteFilesystemOptions) (DeleteFilesystemResponse, error) { - filesystemClient := s.NewFilesystemClient(filesystem) +// DeleteFileSystem deletes the specified filesystem. +func (s *Client) DeleteFileSystem(ctx context.Context, filesystem string, options *DeleteFileSystemOptions) (DeleteFileSystemResponse, error) { + filesystemClient := s.NewFileSystemClient(filesystem) resp, err := filesystemClient.Delete(ctx, options) err = exported.ConvertToDFSError(err) return resp, err } -// SetProperties sets properties for a storage account's File service endpoint. (blob3) +// SetProperties sets properties for a storage account's Datalake service endpoint. func (s *Client) SetProperties(ctx context.Context, options *SetPropertiesOptions) (SetPropertiesResponse, error) { opts := options.format() resp, err := s.serviceClient().SetProperties(ctx, opts) @@ -229,7 +229,7 @@ func (s *Client) SetProperties(ctx context.Context, options *SetPropertiesOption return resp, err } -// GetProperties gets properties for a storage account's File service endpoint. (blob3) +// GetProperties gets properties for a storage account's Datalake service endpoint. func (s *Client) GetProperties(ctx context.Context, options *GetPropertiesOptions) (GetPropertiesResponse, error) { opts := options.format() resp, err := s.serviceClient().GetProperties(ctx, opts) @@ -238,29 +238,30 @@ func (s *Client) GetProperties(ctx context.Context, options *GetPropertiesOption } -// NewListFilesystemsPager operation returns a pager of the shares under the specified account. (blob3) +// NewListFileSystemsPager operation returns a pager of the shares under the specified account. // For more information, see https://learn.microsoft.com/en-us/rest/api/storageservices/list-shares -func (s *Client) NewListFilesystemsPager(o *ListFilesystemsOptions) *runtime.Pager[ListFilesystemsResponse] { +func (s *Client) NewListFileSystemsPager(o *ListFileSystemsOptions) *runtime.Pager[ListFileSystemsResponse] { listOptions := generated_blob.ServiceClientListContainersSegmentOptions{} + defaultInclude := ListFileSystemsInclude{} if o != nil { - if o.Include.Deleted { + if o.Include != defaultInclude && o.Include.Deleted != nil && *o.Include.Deleted { listOptions.Include = append(listOptions.Include, generated_blob.ListContainersIncludeTypeDeleted) } - if o.Include.Metadata { + if o.Include != defaultInclude && o.Include.Metadata != nil && *o.Include.Metadata { listOptions.Include = append(listOptions.Include, generated_blob.ListContainersIncludeTypeMetadata) } - if o.Include.System { + if o.Include != defaultInclude && o.Include.System != nil && *o.Include.System { listOptions.Include = append(listOptions.Include, generated_blob.ListContainersIncludeTypeSystem) } listOptions.Marker = o.Marker listOptions.Maxresults = o.MaxResults listOptions.Prefix = o.Prefix } - return runtime.NewPager(runtime.PagingHandler[ListFilesystemsResponse]{ - More: func(page ListFilesystemsResponse) bool { + return runtime.NewPager(runtime.PagingHandler[ListFileSystemsResponse]{ + More: func(page ListFileSystemsResponse) bool { return page.NextMarker != nil && len(*page.NextMarker) > 0 }, - Fetcher: func(ctx context.Context, page *ListFilesystemsResponse) (ListFilesystemsResponse, error) { + Fetcher: func(ctx context.Context, page *ListFileSystemsResponse) (ListFileSystemsResponse, error) { var req *policy.Request var err error if page == nil { @@ -270,14 +271,14 @@ func (s *Client) NewListFilesystemsPager(o *ListFilesystemsOptions) *runtime.Pag req, err = s.generatedServiceClientWithBlob().ListContainersSegmentCreateRequest(ctx, &listOptions) } if err != nil { - return ListFilesystemsResponse{}, exported.ConvertToDFSError(err) + return ListFileSystemsResponse{}, exported.ConvertToDFSError(err) } resp, err := s.generatedServiceClientWithBlob().InternalClient().Pipeline().Do(req) if err != nil { - return ListFilesystemsResponse{}, exported.ConvertToDFSError(err) + return ListFileSystemsResponse{}, exported.ConvertToDFSError(err) } if !runtime.HasStatusCode(resp, http.StatusOK) { - return ListFilesystemsResponse{}, exported.ConvertToDFSError(runtime.NewResponseError(resp)) + return ListFileSystemsResponse{}, exported.ConvertToDFSError(runtime.NewResponseError(resp)) } resp1, err := s.generatedServiceClientWithBlob().ListContainersSegmentHandleResponse(resp) return resp1, exported.ConvertToDFSError(err) diff --git a/sdk/storage/azdatalake/service/client_test.go b/sdk/storage/azdatalake/service/client_test.go index 6d009463b140..286bc62a4cef 100644 --- a/sdk/storage/azdatalake/service/client_test.go +++ b/sdk/storage/azdatalake/service/client_test.go @@ -78,8 +78,8 @@ func (s *ServiceUnrecordedTestsSuite) TestServiceClientFromConnectionString() { svcClient, err := service.NewClientWithSharedKeyCredential(parsedConnStr.ServiceURL, sharedKeyCred, nil) _require.Nil(err) - fsClient := testcommon.CreateNewFilesystem(context.Background(), _require, testcommon.GenerateFilesystemName(testName), svcClient) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + fsClient := testcommon.CreateNewFileSystem(context.Background(), _require, testcommon.GenerateFileSystemName(testName), svcClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) } func (s *ServiceRecordedTestsSuite) TestSetPropertiesLogging() { @@ -182,6 +182,7 @@ func (s *ServiceRecordedTestsSuite) TestSetPropertiesSetCORSMultiple() { _require.Nil(err) resp, err := svcClient.GetProperties(context.Background(), nil) + _require.NoError(err) for i := 0; i < len(resp.CORS); i++ { if resp.CORS[i].AllowedOrigins == &allowedOrigins1 { _require.Equal(resp.CORS[i].AllowedMethods, &allowedMethods1) @@ -196,7 +197,6 @@ func (s *ServiceRecordedTestsSuite) TestSetPropertiesSetCORSMultiple() { _require.Equal(resp.CORS[i].AllowedHeaders, &allowedHeaders2) } } - _require.Nil(err) } func (s *ServiceRecordedTestsSuite) TestAccountDeleteRetentionPolicy() { @@ -331,7 +331,7 @@ func (s *ServiceUnrecordedTestsSuite) TestSASServiceClient() { serviceClient, err := service.NewClientWithSharedKeyCredential(fmt.Sprintf("https://%s.dfs.core.windows.net/", cred.AccountName()), cred, nil) _require.Nil(err) - fsName := testcommon.GenerateFilesystemName(testName) + fsName := testcommon.GenerateFileSystemName(testName) // Note: Always set all permissions, services, types to true to ensure order of string formed is correct. resources := sas.AccountResourceTypes{ @@ -361,10 +361,10 @@ func (s *ServiceUnrecordedTestsSuite) TestSASServiceClient() { _require.Nil(err) // create fs using SAS - _, err = svcClient.CreateFilesystem(context.Background(), fsName, nil) + _, err = svcClient.CreateFileSystem(context.Background(), fsName, nil) _require.Nil(err) - _, err = svcClient.DeleteFilesystem(context.Background(), fsName, nil) + _, err = svcClient.DeleteFileSystem(context.Background(), fsName, nil) _require.Nil(err) } @@ -464,7 +464,7 @@ func (s *ServiceUnrecordedTestsSuite) TestNoSharedKeyCredError() { } -func (s *ServiceUnrecordedTestsSuite) TestGetFilesystemClient() { +func (s *ServiceUnrecordedTestsSuite) TestGetFileSystemClient() { _require := require.New(s.T()) testName := s.T().Name() accountName := os.Getenv("AZURE_STORAGE_ACCOUNT_NAME") @@ -475,10 +475,10 @@ func (s *ServiceUnrecordedTestsSuite) TestGetFilesystemClient() { serviceClient, err := service.NewClientWithSharedKeyCredential(fmt.Sprintf("https://%s.blob.core.windows.net/", accountName), cred, nil) _require.Nil(err) - fsName := testcommon.GenerateFilesystemName(testName + "1") - fsClient := serviceClient.NewFilesystemClient(fsName) + fsName := testcommon.GenerateFileSystemName(testName + "1") + fsClient := serviceClient.NewFileSystemClient(fsName) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) _, err = fsClient.Create(context.Background(), nil) _require.Nil(err) @@ -486,7 +486,7 @@ func (s *ServiceUnrecordedTestsSuite) TestGetFilesystemClient() { _require.Nil(err) } -func (s *ServiceUnrecordedTestsSuite) TestSASFilesystemClient() { +func (s *ServiceUnrecordedTestsSuite) TestSASFileSystemClient() { _require := require.New(s.T()) testName := s.T().Name() accountName := os.Getenv("AZURE_STORAGE_ACCOUNT_NAME") @@ -497,10 +497,10 @@ func (s *ServiceUnrecordedTestsSuite) TestSASFilesystemClient() { serviceClient, err := service.NewClientWithSharedKeyCredential(fmt.Sprintf("https://%s.blob.core.windows.net/", accountName), cred, nil) _require.Nil(err) - fsName := testcommon.GenerateFilesystemName(testName) - fsClient := serviceClient.NewFilesystemClient(fsName) + fsName := testcommon.GenerateFileSystemName(testName) + fsClient := serviceClient.NewFileSystemClient(fsName) - permissions := sas.FilesystemPermissions{ + permissions := sas.FileSystemPermissions{ Read: true, Add: true, } @@ -519,7 +519,7 @@ func (s *ServiceUnrecordedTestsSuite) TestSASFilesystemClient() { testcommon.ValidateErrorCode(_require, err, datalakeerror.AuthorizationFailure) } -func (s *ServiceUnrecordedTestsSuite) TestSASFilesystem2() { +func (s *ServiceUnrecordedTestsSuite) TestSASFileSystem2() { _require := require.New(s.T()) testName := s.T().Name() accountName := os.Getenv("AZURE_STORAGE_ACCOUNT_NAME") @@ -530,16 +530,16 @@ func (s *ServiceUnrecordedTestsSuite) TestSASFilesystem2() { serviceClient, err := service.NewClientWithSharedKeyCredential(fmt.Sprintf("https://%s.blob.core.windows.net/", accountName), cred, nil) _require.Nil(err) - fsName := testcommon.GenerateFilesystemName(testName) - fsClient := serviceClient.NewFilesystemClient(fsName) + fsName := testcommon.GenerateFileSystemName(testName) + fsClient := serviceClient.NewFileSystemClient(fsName) start := time.Now().Add(-5 * time.Minute).UTC() opts := filesystem.GetSASURLOptions{StartTime: &start} - sasUrlReadAdd, err := fsClient.GetSASURL(sas.FilesystemPermissions{Read: true, Add: true}, time.Now().Add(time.Hour), &opts) + sasUrlReadAdd, err := fsClient.GetSASURL(sas.FileSystemPermissions{Read: true, Add: true}, time.Now().Add(time.Hour), &opts) _require.Nil(err) _, err = fsClient.Create(context.Background(), &filesystem.CreateOptions{Metadata: testcommon.BasicMetadata}) _require.Nil(err) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient) fsClient1, err := filesystem.NewClientWithNoCredential(sasUrlReadAdd, nil) _require.Nil(err) @@ -552,7 +552,7 @@ func (s *ServiceUnrecordedTestsSuite) TestSASFilesystem2() { start = time.Now().Add(-5 * time.Minute).UTC() opts = filesystem.GetSASURLOptions{StartTime: &start} - sasUrlRCWL, err := fsClient.GetSASURL(sas.FilesystemPermissions{Add: true, Create: true, Delete: true, List: true}, time.Now().Add(time.Hour), &opts) + sasUrlRCWL, err := fsClient.GetSASURL(sas.FileSystemPermissions{Add: true, Create: true, Delete: true, List: true}, time.Now().Add(time.Hour), &opts) _require.Nil(err) fsClient2, err := filesystem.NewClientWithNoCredential(sasUrlRCWL, nil) @@ -574,8 +574,8 @@ func (s *ServiceRecordedTestsSuite) TestListFilesystemsBasic() { "bar": to.Ptr("barvalue"), } - fsName := testcommon.GenerateFilesystemName(testName) - fsClient := svcClient.NewFilesystemClient(fsName) + fsName := testcommon.GenerateFileSystemName(testName) + fsClient := svcClient.NewFileSystemClient(fsName) _, err = fsClient.Create(context.Background(), &filesystem.CreateOptions{Metadata: md}) defer func(fsClient *filesystem.Client, ctx context.Context, options *filesystem.DeleteOptions) { _, err := fsClient.Delete(ctx, options) @@ -584,9 +584,9 @@ func (s *ServiceRecordedTestsSuite) TestListFilesystemsBasic() { } }(fsClient, context.Background(), nil) _require.Nil(err) - prefix := testcommon.FilesystemPrefix - listOptions := service.ListFilesystemsOptions{Prefix: &prefix, Include: service.ListFilesystemsInclude{Metadata: true}} - pager := svcClient.NewListFilesystemsPager(&listOptions) + prefix := testcommon.FileSystemPrefix + listOptions := service.ListFileSystemsOptions{Prefix: &prefix, Include: service.ListFileSystemsInclude{Metadata: to.Ptr(true)}} + pager := svcClient.NewListFileSystemsPager(&listOptions) count := 0 for pager.More() { @@ -634,8 +634,8 @@ func (s *ServiceRecordedTestsSuite) TestListFilesystemsBasicUsingConnectionStrin "bar": to.Ptr("barvalue"), } - fsName := testcommon.GenerateFilesystemName(testName) - fsClient := testcommon.ServiceGetFilesystemClient(fsName, svcClient) + fsName := testcommon.GenerateFileSystemName(testName) + fsClient := testcommon.ServiceGetFileSystemClient(fsName, svcClient) _, err = fsClient.Create(context.Background(), &filesystem.CreateOptions{Metadata: md}) defer func(fsClient *filesystem.Client, ctx context.Context, options *filesystem.DeleteOptions) { _, err := fsClient.Delete(ctx, options) @@ -644,9 +644,9 @@ func (s *ServiceRecordedTestsSuite) TestListFilesystemsBasicUsingConnectionStrin } }(fsClient, context.Background(), nil) _require.Nil(err) - prefix := testcommon.FilesystemPrefix - listOptions := service.ListFilesystemsOptions{Prefix: &prefix, Include: service.ListFilesystemsInclude{Metadata: true}} - pager := svcClient.NewListFilesystemsPager(&listOptions) + prefix := testcommon.FileSystemPrefix + listOptions := service.ListFileSystemsOptions{Prefix: &prefix, Include: service.ListFileSystemsInclude{Metadata: to.Ptr(true)}} + pager := svcClient.NewListFileSystemsPager(&listOptions) count := 0 for pager.More() { @@ -690,30 +690,30 @@ func (s *ServiceRecordedTestsSuite) TestListFilesystemsPaged() { testName := s.T().Name() svcClient, err := testcommon.GetServiceClient(s.T(), testcommon.TestAccountDefault, nil) _require.Nil(err) - const numFilesystems = 6 + const numFileSystems = 6 maxResults := int32(2) - const pagedFilesystemsPrefix = "azfilesystempaged" + const pagedFileSystemsPrefix = "azfilesystempaged" - filesystems := make([]*filesystem.Client, numFilesystems) + filesystems := make([]*filesystem.Client, numFileSystems) expectedResults := make(map[string]bool) - for i := 0; i < numFilesystems; i++ { - fsName := pagedFilesystemsPrefix + testcommon.GenerateFilesystemName(testName) + fmt.Sprintf("%d", i) - fsClient := testcommon.CreateNewFilesystem(context.Background(), _require, fsName, svcClient) + for i := 0; i < numFileSystems; i++ { + fsName := pagedFileSystemsPrefix + testcommon.GenerateFileSystemName(testName) + fmt.Sprintf("%d", i) + fsClient := testcommon.CreateNewFileSystem(context.Background(), _require, fsName, svcClient) filesystems[i] = fsClient expectedResults[fsName] = false } defer func() { for i := range filesystems { - testcommon.DeleteFilesystem(context.Background(), _require, filesystems[i]) + testcommon.DeleteFileSystem(context.Background(), _require, filesystems[i]) } }() - prefix := pagedFilesystemsPrefix + testcommon.FilesystemPrefix - listOptions := service.ListFilesystemsOptions{MaxResults: &maxResults, Prefix: &prefix, Include: service.ListFilesystemsInclude{Metadata: true}} + prefix := pagedFileSystemsPrefix + testcommon.FileSystemPrefix + listOptions := service.ListFileSystemsOptions{MaxResults: &maxResults, Prefix: &prefix, Include: service.ListFileSystemsInclude{Metadata: to.Ptr(true)}} count := 0 - results := make([]service.FilesystemItem, 0) - pager := svcClient.NewListFilesystemsPager(&listOptions) + results := make([]service.FileSystemItem, 0) + pager := svcClient.NewListFileSystemsPager(&listOptions) for pager.More() { resp, err := pager.NextPage(context.Background()) @@ -725,8 +725,8 @@ func (s *ServiceRecordedTestsSuite) TestListFilesystemsPaged() { } } - _require.Equal(count, numFilesystems) - _require.Equal(len(results), numFilesystems) + _require.Equal(count, numFileSystems) + _require.Equal(len(results), numFileSystems) // make sure each fs we see is expected for _, ctnr := range results { @@ -748,13 +748,13 @@ func (s *ServiceRecordedTestsSuite) TestAccountListFilesystemsEmptyPrefix() { svcClient, err := testcommon.GetServiceClient(s.T(), testcommon.TestAccountDefault, nil) _require.NoError(err) - fsClient1 := testcommon.CreateNewFilesystem(context.Background(), _require, testcommon.GenerateFilesystemName(testName)+"1", svcClient) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient1) - fsClient2 := testcommon.CreateNewFilesystem(context.Background(), _require, testcommon.GenerateFilesystemName(testName)+"2", svcClient) - defer testcommon.DeleteFilesystem(context.Background(), _require, fsClient2) + fsClient1 := testcommon.CreateNewFileSystem(context.Background(), _require, testcommon.GenerateFileSystemName(testName)+"1", svcClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient1) + fsClient2 := testcommon.CreateNewFileSystem(context.Background(), _require, testcommon.GenerateFileSystemName(testName)+"2", svcClient) + defer testcommon.DeleteFileSystem(context.Background(), _require, fsClient2) count := 0 - pager := svcClient.NewListFilesystemsPager(nil) + pager := svcClient.NewListFileSystemsPager(nil) for pager.More() { resp, err := pager.NextPage(context.Background()) diff --git a/sdk/storage/azdatalake/service/constants.go b/sdk/storage/azdatalake/service/constants.go index 76069177867b..a06f2a41a79a 100644 --- a/sdk/storage/azdatalake/service/constants.go +++ b/sdk/storage/azdatalake/service/constants.go @@ -16,7 +16,7 @@ type PublicAccessType = filesystem.PublicAccessType const ( File PublicAccessType = filesystem.File - Filesystem PublicAccessType = filesystem.Filesystem + FileSystem PublicAccessType = filesystem.FileSystem ) // StatusType defines values for StatusType diff --git a/sdk/storage/azdatalake/service/examples_test.go b/sdk/storage/azdatalake/service/examples_test.go new file mode 100644 index 000000000000..525c86d582d1 --- /dev/null +++ b/sdk/storage/azdatalake/service/examples_test.go @@ -0,0 +1,335 @@ +//go:build go1.18 +// +build go1.18 + +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT License. See License.txt in the project root for license information. + +package service_test + +import ( + "context" + "fmt" + "github.com/Azure/azure-sdk-for-go/sdk/storage/azdatalake" + "github.com/Azure/azure-sdk-for-go/sdk/storage/azdatalake/filesystem" + "log" + "os" + "time" + + "github.com/Azure/azure-sdk-for-go/sdk/azcore" + "github.com/Azure/azure-sdk-for-go/sdk/azcore/to" + "github.com/Azure/azure-sdk-for-go/sdk/azidentity" + "github.com/Azure/azure-sdk-for-go/sdk/storage/azdatalake/sas" + "github.com/Azure/azure-sdk-for-go/sdk/storage/azdatalake/service" +) + +func handleError(err error) { + if err != nil { + log.Fatal(err.Error()) + } +} + +func Example_service_Client_NewClient() { + accountName, ok := os.LookupEnv("AZURE_STORAGE_ACCOUNT_NAME") + if !ok { + panic("AZURE_STORAGE_ACCOUNT_NAME could not be found") + } + serviceURL := fmt.Sprintf("https://%s.dfs.core.windows.net/", accountName) + + cred, err := azidentity.NewDefaultAzureCredential(nil) + handleError(err) + serviceClient, err := service.NewClient(serviceURL, cred, nil) + handleError(err) + fmt.Println(serviceClient.DFSURL()) + fmt.Println(serviceClient.BlobURL()) +} + +func Example_service_Client_NewClientWithSharedKeyCredential() { + accountName, ok := os.LookupEnv("AZURE_STORAGE_ACCOUNT_NAME") + if !ok { + panic("AZURE_STORAGE_ACCOUNT_NAME could not be found") + } + accountKey, ok := os.LookupEnv("AZURE_STORAGE_ACCOUNT_KEY") + if !ok { + panic("AZURE_STORAGE_ACCOUNT_KEY could not be found") + } + serviceURL := fmt.Sprintf("https://%s.dfs.core.windows.net/", accountName) + + cred, err := azdatalake.NewSharedKeyCredential(accountName, accountKey) + handleError(err) + serviceClient, err := service.NewClientWithSharedKeyCredential(serviceURL, cred, nil) + handleError(err) + fmt.Println(serviceClient.DFSURL()) + fmt.Println(serviceClient.BlobURL()) +} + +func Example_service_Client_NewClientWithNoCredential() { + accountName, ok := os.LookupEnv("AZURE_STORAGE_ACCOUNT_NAME") + if !ok { + panic("AZURE_STORAGE_ACCOUNT_NAME could not be found") + } + sharedAccessSignature, ok := os.LookupEnv("AZURE_STORAGE_SHARED_ACCESS_SIGNATURE") + if !ok { + panic("AZURE_STORAGE_SHARED_ACCESS_SIGNATURE could not be found") + } + serviceURL := fmt.Sprintf("https://%s.dfs.core.windows.net/?%s", accountName, sharedAccessSignature) + + serviceClient, err := service.NewClientWithNoCredential(serviceURL, nil) + handleError(err) + fmt.Println(serviceClient.DFSURL()) + fmt.Println(serviceClient.BlobURL()) +} + +func Example_service_Client_NewClientFromConnectionString() { + // Your connection string can be obtained from the Azure Portal. + connectionString, ok := os.LookupEnv("AZURE_STORAGE_CONNECTION_STRING") + if !ok { + log.Fatal("the environment variable 'AZURE_STORAGE_CONNECTION_STRING' could not be found") + } + + serviceClient, err := service.NewClientFromConnectionString(connectionString, nil) + handleError(err) + fmt.Println(serviceClient.DFSURL()) + fmt.Println(serviceClient.BlobURL()) +} + +func Example_service_Client_CreateContainer() { + accountName, ok := os.LookupEnv("AZURE_STORAGE_ACCOUNT_NAME") + if !ok { + panic("AZURE_STORAGE_ACCOUNT_NAME could not be found") + } + serviceURL := fmt.Sprintf("https://%s.dfs.core.windows.net/", accountName) + + cred, err := azidentity.NewDefaultAzureCredential(nil) + handleError(err) + + serviceClient, err := service.NewClient(serviceURL, cred, nil) + handleError(err) + + _, err = serviceClient.CreateFileSystem(context.TODO(), "testfs", nil) + handleError(err) + + // ======== 2. Delete a container ======== + defer func(serviceClient1 *service.Client, ctx context.Context, fsName string, options *filesystem.DeleteOptions) { + _, err = serviceClient1.DeleteFileSystem(ctx, fsName, options) + if err != nil { + log.Fatal(err) + } + }(serviceClient, context.TODO(), "testfs", nil) +} + +func Example_service_Client_DeleteFileSystem() { + accountName, ok := os.LookupEnv("AZURE_STORAGE_ACCOUNT_NAME") + if !ok { + panic("AZURE_STORAGE_ACCOUNT_NAME could not be found") + } + serviceURL := fmt.Sprintf("https://%s.dfs.core.windows.net/", accountName) + + cred, err := azidentity.NewDefaultAzureCredential(nil) + handleError(err) + serviceClient, err := service.NewClient(serviceURL, cred, nil) + handleError(err) + + _, err = serviceClient.DeleteFileSystem(context.TODO(), "testfs", nil) + handleError(err) +} + +func Example_service_Client_ListFileSystems() { + accountName, ok := os.LookupEnv("AZURE_STORAGE_ACCOUNT_NAME") + if !ok { + panic("AZURE_STORAGE_ACCOUNT_NAME could not be found") + } + serviceURL := fmt.Sprintf("https://%s.dfs.core.windows.net/", accountName) + + cred, err := azidentity.NewDefaultAzureCredential(nil) + handleError(err) + serviceClient, err := service.NewClient(serviceURL, cred, nil) + handleError(err) + + listFSOptions := service.ListFileSystemsOptions{ + Include: service.ListFileSystemsInclude{ + Metadata: to.Ptr(true), // Include Metadata + Deleted: to.Ptr(true), // Include deleted containers in the result as well + }, + } + pager := serviceClient.NewListFileSystemsPager(&listFSOptions) + + for pager.More() { + resp, err := pager.NextPage(context.TODO()) + if err != nil { + log.Fatal(err) + } + for _, fs := range resp.FileSystemItems { + fmt.Println(*fs.Name) + } + } +} + +func Example_service_Client_GetSASURL() { + cred, err := azdatalake.NewSharedKeyCredential("myAccountName", "myAccountKey") + handleError(err) + serviceClient, err := service.NewClientWithSharedKeyCredential("https://.dfs.core.windows.net", cred, nil) + handleError(err) + + resources := sas.AccountResourceTypes{Service: true} + permission := sas.AccountPermissions{Read: true} + start := time.Now() + expiry := start.AddDate(1, 0, 0) + options := service.GetSASURLOptions{StartTime: &start} + sasURL, err := serviceClient.GetSASURL(resources, permission, expiry, &options) + handleError(err) + + serviceURL := fmt.Sprintf("https://.dfs.core.windows.net/?%s", sasURL) + serviceClientWithSAS, err := service.NewClientWithNoCredential(serviceURL, nil) + handleError(err) + _ = serviceClientWithSAS +} + +func Example_service_Client_SetProperties() { + accountName, ok := os.LookupEnv("AZURE_STORAGE_ACCOUNT_NAME") + if !ok { + panic("AZURE_STORAGE_ACCOUNT_NAME could not be found") + } + serviceURL := fmt.Sprintf("https://%s.dfs.core.windows.net/", accountName) + + cred, err := azidentity.NewDefaultAzureCredential(nil) + handleError(err) + serviceClient, err := service.NewClient(serviceURL, cred, nil) + handleError(err) + + enabled := true // enabling retention period + days := int32(5) // setting retention period to 5 days + serviceSetPropertiesResponse, err := serviceClient.SetProperties(context.TODO(), &service.SetPropertiesOptions{ + DeleteRetentionPolicy: &service.RetentionPolicy{Enabled: &enabled, Days: &days}, + }) + + handleError(err) + fmt.Println(serviceSetPropertiesResponse) +} + +func Example_service_Client_GetProperties() { + accountName, ok := os.LookupEnv("AZURE_STORAGE_ACCOUNT_NAME") + if !ok { + panic("AZURE_STORAGE_ACCOUNT_NAME could not be found") + } + serviceURL := fmt.Sprintf("https://%s.dfs.core.windows.net/", accountName) + + cred, err := azidentity.NewDefaultAzureCredential(nil) + handleError(err) + + serviceClient, err := service.NewClient(serviceURL, cred, nil) + handleError(err) + + serviceGetPropertiesResponse, err := serviceClient.GetProperties(context.TODO(), nil) + handleError(err) + + fmt.Println(serviceGetPropertiesResponse) +} + +// This example shows how to create and use an Azure Storage account Shared Access Signature (SAS). +func Example_service_SASSignatureValues_Sign() { + accountName, accountKey := os.Getenv("AZURE_STORAGE_ACCOUNT_NAME"), os.Getenv("AZURE_STORAGE_ACCOUNT_KEY") + + credential, err := azdatalake.NewSharedKeyCredential(accountName, accountKey) + handleError(err) + + sasQueryParams, err := sas.AccountSignatureValues{ + Protocol: sas.ProtocolHTTPS, + ExpiryTime: time.Now().UTC().Add(48 * time.Hour), + Permissions: to.Ptr(sas.AccountPermissions{Read: true, List: true}).String(), + ResourceTypes: to.Ptr(sas.AccountResourceTypes{Container: true, Object: true}).String(), + }.SignWithSharedKey(credential) + handleError(err) + + sasURL := fmt.Sprintf("https://%s.dfs.core.windows.net/?%s", accountName, sasQueryParams.Encode()) + + // This URL can be used to authenticate requests now + serviceClient, err := service.NewClientWithNoCredential(sasURL, nil) + handleError(err) + + // You can also break a blob URL up into it's constituent parts + blobURLParts, _ := azdatalake.ParseURL(serviceClient.DFSURL()) + fmt.Printf("SAS expiry time = %s\n", blobURLParts.SAS.ExpiryTime()) +} + +func Example_service_Client_NewClientWithUserDelegationCredential() { + accountName, ok := os.LookupEnv("AZURE_STORAGE_ACCOUNT_NAME") + if !ok { + panic("AZURE_STORAGE_ACCOUNT_NAME could not be found") + } + const containerName = "testContainer" + + // Create Managed Identity (OAuth) Credentials using Client ID + clientOptions := azcore.ClientOptions{} // Fill clientOptions as needed + optsClientID := azidentity.ManagedIdentityCredentialOptions{ClientOptions: clientOptions, ID: azidentity.ClientID("7cf7db0d-...")} + cred, err := azidentity.NewManagedIdentityCredential(&optsClientID) + handleError(err) + clientOptionsService := service.ClientOptions{} // Same as azcore.ClientOptions using service instead + + svcClient, err := service.NewClient(fmt.Sprintf("https://%s.dfs.core.windows.net/", accountName), cred, &clientOptionsService) + handleError(err) + + // Set current and past time and create key + currentTime := time.Now().UTC().Add(-10 * time.Second) + pastTime := currentTime.Add(48 * time.Hour) + info := service.KeyInfo{ + Start: to.Ptr(currentTime.UTC().Format(sas.TimeFormat)), + Expiry: to.Ptr(pastTime.UTC().Format(sas.TimeFormat)), + } + + udc, err := svcClient.GetUserDelegationCredential(context.Background(), info, nil) + handleError(err) + + fmt.Println("User Delegation Key has been created for ", accountName) + + // Create Blob Signature Values with desired permissions and sign with user delegation credential + sasQueryParams, err := sas.DatalakeSignatureValues{ + Protocol: sas.ProtocolHTTPS, + StartTime: time.Now().UTC().Add(time.Second * -10), + ExpiryTime: time.Now().UTC().Add(15 * time.Minute), + Permissions: to.Ptr(sas.FileSystemPermissions{Read: true, List: true}).String(), + FileSystemName: containerName, + }.SignWithUserDelegation(udc) + handleError(err) + + sasURL := fmt.Sprintf("https://%s.dfs.core.windows.net/?%s", accountName, sasQueryParams.Encode()) + + // This URL can be used to authenticate requests now + serviceClient, err := service.NewClientWithNoCredential(sasURL, nil) + handleError(err) + + // You can also break a blob URL up into it's constituent parts + blobURLParts, _ := azdatalake.ParseURL(serviceClient.DFSURL()) + fmt.Printf("SAS expiry time = %s\n", blobURLParts.SAS.ExpiryTime()) + + // Create Managed Identity (OAuth) Credentials using Resource ID + optsResourceID := azidentity.ManagedIdentityCredentialOptions{ClientOptions: clientOptions, ID: azidentity.ResourceID("/subscriptions/...")} + cred, err = azidentity.NewManagedIdentityCredential(&optsResourceID) + handleError(err) + + svcClient, err = service.NewClient(fmt.Sprintf("https://%s.dfs.core.windows.net/", accountName), cred, &clientOptionsService) + handleError(err) + + udc, err = svcClient.GetUserDelegationCredential(context.Background(), info, nil) + handleError(err) + fmt.Println("User Delegation Key has been created for ", accountName) + + // Create Blob Signature Values with desired permissions and sign with user delegation credential + sasQueryParams, err = sas.DatalakeSignatureValues{ + Protocol: sas.ProtocolHTTPS, + StartTime: time.Now().UTC().Add(time.Second * -10), + ExpiryTime: time.Now().UTC().Add(15 * time.Minute), + Permissions: to.Ptr(sas.FileSystemPermissions{Read: true, List: true}).String(), + FileSystemName: containerName, + }.SignWithUserDelegation(udc) + handleError(err) + + sasURL = fmt.Sprintf("https://%s.dfs.core.windows.net/?%s", accountName, sasQueryParams.Encode()) + + // This URL can be used to authenticate requests now + serviceClient, err = service.NewClientWithNoCredential(sasURL, nil) + handleError(err) + + // You can also break a blob URL up into it's constituent parts + blobURLParts, _ = azdatalake.ParseURL(serviceClient.DFSURL()) + fmt.Printf("SAS expiry time = %s\n", blobURLParts.SAS.ExpiryTime()) +} diff --git a/sdk/storage/azdatalake/service/models.go b/sdk/storage/azdatalake/service/models.go index 2c2c27af8d5e..7fec5e4fc6fc 100644 --- a/sdk/storage/azdatalake/service/models.go +++ b/sdk/storage/azdatalake/service/models.go @@ -16,13 +16,13 @@ import ( ) import blobSAS "github.com/Azure/azure-sdk-for-go/sdk/storage/azblob/sas" -// CreateFilesystemOptions contains the optional parameters for the Filesystem Create method. -type CreateFilesystemOptions = filesystem.CreateOptions +// CreateFileSystemOptions contains the optional parameters for the FileSystem Create method. +type CreateFileSystemOptions = filesystem.CreateOptions -// DeleteFilesystemOptions contains the optional parameters for the Filesystem Delete method. -type DeleteFilesystemOptions = filesystem.DeleteOptions +// DeleteFileSystemOptions contains the optional parameters for the FileSystem Delete method. +type DeleteFileSystemOptions = filesystem.DeleteOptions -// GetUserDelegationCredentialOptions contains optional parameters for Service.GetUserDelegationKey method. +// GetUserDelegationCredentialOptions contains optional parameters for GetUserDelegationKey method. type GetUserDelegationCredentialOptions struct { // placeholder for future options } @@ -45,29 +45,23 @@ func (o *GetPropertiesOptions) format() *service.GetPropertiesOptions { // SetPropertiesOptions provides set of options for Client.SetProperties type SetPropertiesOptions struct { - // The set of CORS rules. + // CORS The set of CORS rules. CORS []*CORSRule - - // The default version to use for requests to the Datalake service if an incoming request's version is not specified. Possible + // DefaultServiceVersion The default version to use for requests to the Datalake service if an incoming request's version is not specified. Possible // values include version 2008-10-27 and all more recent versions. DefaultServiceVersion *string - - // the retention policy which determines how long the associated data should persist. + // DeleteRetentionPolicy the retention policy which determines how long the associated data should persist. DeleteRetentionPolicy *RetentionPolicy - - // a summary of request statistics grouped by API in hour or minute aggregates + // HourMetrics a summary of request statistics grouped by API in hour or minute aggregates // If version is not set - we default to "1.0" HourMetrics *Metrics - - // Azure Analytics Logging settings. + // Logging Azure Analytics Logging settings. // If version is not set - we default to "1.0" Logging *Logging - - // a summary of request statistics grouped by API in hour or minute aggregates + // MinuteMetrics a summary of request statistics grouped by API in hour or minute aggregates // If version is not set - we default to "1.0" MinuteMetrics *Metrics - - // The properties that enable an account to host a static website. + // StaticWebsite The properties that enable an account to host a static website. StaticWebsite *StaticWebsite } @@ -86,27 +80,31 @@ func (o *SetPropertiesOptions) format() *service.SetPropertiesOptions { } } -// ListFilesystemsInclude indicates what additional information the service should return with each filesystem. -type ListFilesystemsInclude struct { - // Tells the service whether to return metadata for each filesystem. - Metadata bool - - // Tells the service whether to return soft-deleted filesystems. - Deleted bool - - System bool +// ListFileSystemsInclude indicates what additional information the service should return with each filesystem. +type ListFileSystemsInclude struct { + // Metadata tells the service whether to return metadata for each filesystem. + Metadata *bool + // Deleted tells the service whether to return soft-deleted filesystems. + Deleted *bool + // System tells the service whether to return system filesystems. + System *bool } -// ListFilesystemsOptions contains the optional parameters for the Client.List method. -type ListFilesystemsOptions struct { - Include ListFilesystemsInclude - Marker *string +// ListFileSystemsOptions contains the optional parameters for the ListFileSystems method. +type ListFileSystemsOptions struct { + // Include tells the service whether to return filesystem metadata. + Include ListFileSystemsInclude + // Marker is the continuation token to use when continuing the operation. + Marker *string + // MaxResults sets the maximum number of paths that will be returned per page. MaxResults *int32 - Prefix *string + // Prefix filters the results to return only filesystems whose names begin with the specified prefix. + Prefix *string } // GetSASURLOptions contains the optional parameters for the Client.GetSASURL method. type GetSASURLOptions struct { + // StartTime is the time after which the SAS will become valid. StartTime *time.Time } @@ -174,17 +172,17 @@ type LeaseAccessConditions = exported.LeaseAccessConditions // ModifiedAccessConditions contains a group of parameters for specifying access conditions. type ModifiedAccessConditions = exported.ModifiedAccessConditions -// CPKScopeInfo contains a group of parameters for the FilesystemClient.Create method. +// CPKScopeInfo contains a group of parameters for the FileSystemClient.Create method. type CPKScopeInfo = filesystem.CPKScopeInfo // StorageServiceProperties - Storage Service Properties. Returned in GetServiceProperties call. type StorageServiceProperties = service.StorageServiceProperties -// ListFilesystemsSegmentResponse contains fields from the ListFilesystems operation -type ListFilesystemsSegmentResponse = generated_blob.ListFileSystemsSegmentResponse +// ListFileSystemsSegmentResponse contains fields from the ListFileSystems operation +type ListFileSystemsSegmentResponse = generated_blob.ListFileSystemsSegmentResponse -// FilesystemItem contains fields from the ListFilesystems operation -type FilesystemItem = generated_blob.FileSystemItem +// FileSystemItem contains fields from the ListFileSystems operation +type FileSystemItem = generated_blob.FileSystemItem -// FilesystemProperties contains fields from the ListFilesystems operation -type FilesystemProperties = generated_blob.FileSystemProperties +// FileSystemProperties contains fields from the ListFileSystems operation +type FileSystemProperties = generated_blob.FileSystemProperties diff --git a/sdk/storage/azdatalake/service/responses.go b/sdk/storage/azdatalake/service/responses.go index aa3581babdd7..27ef525cf8d7 100644 --- a/sdk/storage/azdatalake/service/responses.go +++ b/sdk/storage/azdatalake/service/responses.go @@ -15,11 +15,11 @@ import ( "github.com/Azure/azure-sdk-for-go/sdk/storage/azdatalake/internal/generated_blob" ) -// CreateFilesystemResponse contains the response fields for the CreateFilesystem operation. -type CreateFilesystemResponse = filesystem.CreateResponse +// CreateFileSystemResponse contains the response fields for the CreateFileSystem operation. +type CreateFileSystemResponse = filesystem.CreateResponse -// DeleteFilesystemResponse contains the response fields for the DeleteFilesystem operation. -type DeleteFilesystemResponse = filesystem.DeleteResponse +// DeleteFileSystemResponse contains the response fields for the DeleteFileSystem operation. +type DeleteFileSystemResponse = filesystem.DeleteResponse // SetPropertiesResponse contains the response fields for the SetProperties operation. type SetPropertiesResponse = service.SetPropertiesResponse @@ -27,5 +27,5 @@ type SetPropertiesResponse = service.SetPropertiesResponse // GetPropertiesResponse contains the response fields for the GetProperties operation. type GetPropertiesResponse = service.GetPropertiesResponse -// ListFilesystemsResponse contains the response fields for the ListFilesystems operation. -type ListFilesystemsResponse = generated_blob.ServiceClientListFileSystemsSegmentResponse +// ListFileSystemsResponse contains the response fields for the ListFileSystems operation. +type ListFileSystemsResponse = generated_blob.ServiceClientListFileSystemsSegmentResponse diff --git a/sdk/storage/azdatalake/test-resources.json b/sdk/storage/azdatalake/test-resources.json index c6259f7ab02f..88c380e64a8f 100644 --- a/sdk/storage/azdatalake/test-resources.json +++ b/sdk/storage/azdatalake/test-resources.json @@ -229,6 +229,11 @@ "apiVersion": "[variables('mgmtApiVersion')]", "name": "[concat(variables('dataLakeAccountName'), '/default')]", "properties": { + "deleteRetentionPolicy": { + "allowPermanentDelete": false, + "enabled": true, + "days": 7 + }, "containerDeleteRetentionPolicy": { "enabled": true, "days": 1 @@ -262,9 +267,9 @@ "properties": { "isVersioningEnabled": true, "deleteRetentionPolicy": { - "allowPermanentDelete": true, + "allowPermanentDelete": false, "enabled": true, - "days": 1 + "days": 7 }, "containerDeleteRetentionPolicy": { "enabled": true,