Skip to content

Commit

Permalink
[AzDatalake] APIView Feedback + Samples + Doc + CI/Live issues cleanup (
Browse files Browse the repository at this point in the history
#21348)

* Enable gocritic during linting (#20715)

Enabled gocritic's evalOrder to catch dependencies on undefined behavior
on return statements.
Updated to latest version of golangci-lint.
Fixed issue in azblob flagged by latest linter.

* Cosmos DB: Enable merge support (#20716)

* Adding header and value

* Wiring and tests

* format

* Fixing value

* change log

* [azservicebus, azeventhubs] Stress test and logging improvement (#20710)

Logging improvements:

* Updating the logging to print more tracing information (per-link) in prep for the bigger release coming up.
* Trimming out some of the verbose logging, seeing if I can get it a bit more reasonable.

Stress tests:

* Add a timestamp to the log name we generate and also default to append, not overwrite.
* Use 0.5 cores, 0.5GB as our baseline. Some pods use more and I'll tune them more later.

* update proxy version (#20712)

Co-authored-by: Scott Beddall <scbedd@microsoft.com>

* Return an error when you try to send a message that's too large. (#20721)

This now works just like the message batch - you'll get an ErrMessageTooLarge
if you attempt to send a message that's too large for the link's configured
size.

NOTE: there's a patch to `internal/go-amqp/Sender.go` to match what's in go-amqp's
main so it returns a programmatically useful error when the message is too large.

Fixes #20647

* Changes in test that is failing in pipeline (#20693)

* [azservicebus, azeventhubs] Treat 'entity full' as a fatal error (#20722)

When the remote entity is full we get a resource-limit-exceeded condition. This isn't something we should keep retrying on and it's best to just abort and let the user know immediately, rather than hoping it might eventually clear out.

This affected both Event Hubs and Service Bus.

Fixes #20647

* [azservicebus/azeventhubs] Redirect stderr and stdout to tee (#20726)

* Update changelog with latest features (#20730)

* Update changelog with latest features

Prepare for upcoming release.

* bump minor version

* pass along the artifact name so we can override it later (#20732)

Co-authored-by: scbedd <45376673+scbedd@users.noreply.github.com>

* [azeventhubs] Fixing checkpoint store race condition (#20727)

The checkpoint store wasn't guarding against multiple owners claiming for the first time - fixing this by using IfNoneMatch

Fixes #20717

* Fix azidentity troubleshooting guide link (#20736)

* [Release] sdk/resourcemanager/paloaltonetworksngfw/armpanngfw/0.1.0 (#20437)

* [Release] sdk/resourcemanager/paloaltonetworksngfw/armpanngfw/0.1.0 generation from spec commit: 85fb4ac6f8bfefd179e6c2632976a154b5c9ff04

* client factory

* fix

* fix

* update

* add sdk/resourcemanager/postgresql/armpostgresql live test (#20685)

* add sdk/resourcemanager/postgresql/armpostgresql live test

* update assets.json

* set subscriptionId default value

* format

* add sdk/resourcemanager/eventhub/armeventhub live test (#20686)

* add sdk/resourcemanager/eventhub/armeventhub live test

* update assets

* add sdk/resourcemanager/compute/armcompute live test (#20048)

* add sdk/resourcemanager/compute/armcompute live test

* skus filter

* fix subscriptionId default value

* fix

* gofmt

* update recording

* sdk/resourcemanager/network/armnetwork live test (#20331)

* sdk/resourcemanager/network/armnetwork live test

* update subscriptionId default value

* update recording

* add sdk/resourcemanager/cosmos/armcosmos live test (#20705)

* add sdk/resourcemanager/cosmos/armcosmos live test

* update assets.json

* update assets.json

* update assets.json

* update assets.json

* Increment package version after release of azcore (#20740)

* [azeventhubs] Improperly resetting etag in the checkpoint store (#20737)

We shouldn't be resetting the etag to nil - it's what we use to enforce a "single winner" when doing ownership claims.

The bug here was two-fold: I had bad logic in my previous claim ownership, which I fixed in a previous PR, but we need to reflect that same constraint properly in our in-memory checkpoint store for these tests.

* Eng workflows sync and branch cleanup additions (#20743)

Co-authored-by: James Suplizio <jasupliz@microsoft.com>

* [azeventhubs] Latest start position can also be inclusive (ie, get the latest message) (#20744)

* Update GitHubEventProcessor version and remove pull_request_review procesing (#20751)

Co-authored-by: James Suplizio <jasupliz@microsoft.com>

* Rename DisableAuthorityValidationAndInstanceDiscovery (#20746)

* fix (#20707)

* AzFile (#20739)

* azfile: Fixing connection string parsing logic (#20798)

* Fixing connection string parse logic

* Update README

* [azadmin] fix flaky test (#20758)

* fix flaky test

* charles suggestion

* Prepare azidentity v1.3.0 for release (#20756)

* Fix broken podman link (#20801)

Co-authored-by: Wes Haggard <weshaggard@users.noreply.github.com>

* [azquery] update doc comments (#20755)

* update doc comments

* update statistics and visualization generation

* prep-for-release

* Fixed contribution section (#20752)

Co-authored-by: Bob Tabor <rotabor@microsoft.com>

* [azeventhubs,azservicebus] Some API cleanup, renames (#20754)

* Adding options to UpdateCheckpoint(), just for future potential expansion
* Make Offset an int64, not a *int64 (it's not optional, it'll always come back with ReceivedEvents)
* Adding more logging into the checkpoint store.
* Point all imports at the production go-amqp

* Add supporting features to enable distributed tracing (#20301) (#20708)

* Add supporting features to enable distributed tracing

This includes new internal pipeline policies and other supporting types.
See the changelog for a full description.
Added some missing doc comments.

* fix linter issue

* add net.peer.name trace attribute

sequence custom HTTP header policy before logging policy.
sequence logging policy after HTTP trace policy.
keep body download policy at the end.

* add span for iterating over pages

* Restore ARM CAE support for azcore beta (#20657)

This reverts commit 9020972.

* Upgrade to stable azcore (#20808)

* Increment package version after release of data/azcosmos (#20807)

* Updating changelog (#20810)

* Add fake package to azcore (#20711)

* Add fake package to azcore

This is the supporting infrastructure for the generated SDK fakes.

* fix doc comment

* Updating CHANGELOG.md (#20809)

* changelog (#20811)

* Increment package version after release of storage/azfile (#20813)

* Update changelog (azblob) (#20815)

* Updating CHANGELOG.md

* Update the changelog with correct version

* [azquery] migration guide (#20742)

* migration guide

* Charles feedback

* Richard feedback

---------

Co-authored-by: Charles Lowell <10964656+chlowell@users.noreply.github.com>

* Increment package version after release of monitor/azquery (#20820)

* [keyvault] prep for release (#20819)

* prep for release

* perf tests

* update date

* small cleanup

* rename to FileSystem

* regenrated

* removed metadata option and fixed generated

* fixed tests

* fixed tests

* fixed tests

* fixed tests

* added more tests

* doc cleanup

* doc failures

* doc failures

* doc failures

* changelog

* samples

* mod tidy

* removed client from rename response

* fixed deserialize

* cleanup

* readme

* live pipe fix

* live pipe fix

* live pipe fix

* doc fix

* doc fix

* doc fix

* model fix

* model fix

* doc fix

* added doc.go

* added file examples

* apiview model fixes

* version and changelog

* changelog fix

* fixed export

---------

Co-authored-by: Joel Hendrix <jhendrix@microsoft.com>
Co-authored-by: Matias Quaranta <ealsur@users.noreply.github.com>
Co-authored-by: Richard Park <51494936+richardpark-msft@users.noreply.github.com>
Co-authored-by: Azure SDK Bot <53356347+azure-sdk@users.noreply.github.com>
Co-authored-by: Scott Beddall <scbedd@microsoft.com>
Co-authored-by: siminsavani-msft <77068571+siminsavani-msft@users.noreply.github.com>
Co-authored-by: scbedd <45376673+scbedd@users.noreply.github.com>
Co-authored-by: Charles Lowell <10964656+chlowell@users.noreply.github.com>
Co-authored-by: Peng Jiahui <46921893+Alancere@users.noreply.github.com>
Co-authored-by: James Suplizio <jasupliz@microsoft.com>
Co-authored-by: Sourav Gupta <98318303+souravgupta-msft@users.noreply.github.com>
Co-authored-by: gracewilcox <43627800+gracewilcox@users.noreply.github.com>
Co-authored-by: Wes Haggard <weshaggard@users.noreply.github.com>
Co-authored-by: Bob Tabor <bob.tabor@microsoft.com>
Co-authored-by: Bob Tabor <rotabor@microsoft.com>
  • Loading branch information
16 people authored Aug 16, 2023
1 parent 40f7a7f commit d366659
Show file tree
Hide file tree
Showing 61 changed files with 3,778 additions and 1,466 deletions.
7 changes: 7 additions & 0 deletions sdk/storage/azdatalake/CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
# Release History

## 0.1.0-beta.1 (2023-08-16)

### Features Added

* This is the initial preview release of the `azdatalake` library
280 changes: 280 additions & 0 deletions sdk/storage/azdatalake/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,280 @@
# ADLS Gen2 Storage SDK for Go

> Service Version: 2020-10-02
Azure Data Lake Storage Gen2 (ADLS Gen2) is Microsoft's hierarchical object storage solution for the cloud with converged capabilities with Azure Blob Storage.
For example, Data Lake Storage Gen2 provides file system semantics, file-level security, and scale.
Because these capabilities are built on Blob storage, you also get low-cost, tiered storage, with high availability/disaster recovery capabilities.
ADLS Gen2 makes Azure Storage the foundation for building enterprise data lakes on Azure.
Designed from the start to service multiple petabytes of information while sustaining hundreds of gigabits of throughput, ADLS Gen2 allows you to easily manage massive amounts of data.

[Source code][source] | [API reference documentation][docs] | [REST API documentation][rest_docs]

## Getting started

### Install the package

Install the ADLS Gen2 Storage SDK for Go with [go get][goget]:

```Powershell
go get github.com/Azure/azure-sdk-for-go/sdk/storage/azdatalake
```

If you're going to authenticate with Azure Active Directory (recommended), install the [azidentity][azidentity] module.
```Powershell
go get github.com/Azure/azure-sdk-for-go/sdk/azidentity
```

### Prerequisites

A supported [Go][godevdl] version (the Azure SDK supports the two most recent Go releases).

You need an [Azure subscription][azure_sub] and a
[Storage Account][storage_account_docs] to use this package.

To create a new Storage Account, you can use the [Azure Portal][storage_account_create_portal],
[Azure PowerShell][storage_account_create_ps], or the [Azure CLI][storage_account_create_cli].
Here's an example using the Azure CLI:

```Powershell
az storage account create --name MyStorageAccount --resource-group MyResourceGroup --location westus --sku Standard_LRS
```

### Authenticate the client

In order to interact with the ADLS Gen2 Storage service, you'll need to create an instance of the `Client` type. The [azidentity][azidentity] module makes it easy to add Azure Active Directory support for authenticating Azure SDK clients with their corresponding Azure services.

```go
// create a credential for authenticating with Azure Active Directory
cred, err := azidentity.NewDefaultAzureCredential(nil)
// TODO: handle err

// create a service.Client for the specified storage account that uses the above credential
client, err := service.NewClient("https://MYSTORAGEACCOUNT.dfs.core.windows.net/", cred, nil)
// TODO: handle err
// you can also create filesystem, file and directory clients
```

Learn more about enabling Azure Active Directory for authentication with Azure Storage in [our documentation][storage_ad] and [our samples](#next-steps).

## Key concepts

ADLS Gen2 provides:
- Hadoop-compatible access
- Hierarchical directory structure
- Optimized cost and performance
- Finer grain security model
- Massive scalability

ADLS Gen2 storage is designed for:

- Serving images or documents directly to a browser.
- Storing files for distributed access.
- Streaming video and audio.
- Writing to log files.
- Storing data for backup and restore, disaster recovery, and archiving.
- Storing data for analysis by an on-premises or Azure-hosted service.

ADLS Gen2 storage offers three types of resources:

- The _storage account_
- One or more _filesystems_ in a storage account
- One or more _files_ or _directories_ in a filesystem

Instances of the `Client` type provide methods for manipulating filesystems and paths within a storage account.
The storage account is specified when the `Client` is constructed. The clients available are referenced below.
Use the appropriate client constructor function for the authentication mechanism you wish to use.

### Goroutine safety
We guarantee that all client instance methods are goroutine-safe and independent of each other ([guideline](https://azure.github.io/azure-sdk/golang_introduction.html#thread-safety)). This ensures that the recommendation of reusing client instances is always safe, even across goroutines.

### About metadata
ADLS Gen2 metadata name/value pairs are valid HTTP headers and should adhere to all restrictions governing HTTP headers. Metadata names must be valid HTTP header names, may contain only ASCII characters, and should be treated as case-insensitive. Base64-encode or URL-encode metadata values containing non-ASCII characters.

### Additional concepts
<!-- CLIENT COMMON BAR -->
[Client options](https://pkg.go.dev/github.com/Azure/azure-sdk-for-go/sdk/azcore/policy#ClientOptions) |
[Accessing the response](https://pkg.go.dev/github.com/Azure/azure-sdk-for-go/sdk/azcore/runtime#WithCaptureResponse) |
[Handling failures](https://pkg.go.dev/github.com/Azure/azure-sdk-for-go/sdk/azcore#ResponseError) |
[Logging](https://pkg.go.dev/github.com/Azure/azure-sdk-for-go/sdk/azcore/log)
<!-- CLIENT COMMON BAR -->

## Examples

### Creating and uploading a file (assuming filesystem exists)

```go
const (
path = "https://MYSTORAGEACCOUNT.dfs.core.windows.net/sample-fs/sample-file"
)

// authenticate with Azure Active Directory
cred, err := azidentity.NewDefaultAzureCredential(nil)
// TODO: handle error

// create a client for the specified storage account
client, err := file.NewClient(path, cred, nil)
// TODO: handle error

_, err = client.Create(context.TODO(), nil)
// TODO: handle error

// open the file for reading
fh, err := os.OpenFile(sampleFile, os.O_RDONLY, 0)
// TODO: handle error
defer fh.Close()

// upload the file to the specified filesystem with the specified file name
_, err = client.UploadFile(context.TODO(), fh, nil)
// TODO: handle error
```

### Downloading a file

```go
const (
path = "https://MYSTORAGEACCOUNT.dfs.core.windows.net/sample-fs/cloud.jpg"
)

// authenticate with Azure Active Directory
cred, err := azidentity.NewDefaultAzureCredential(nil)
// TODO: handle error

// create a client for the specified storage account
client, err := file.NewClient(path, cred, nil)
// TODO: handle error

// create or open a local file where we can download the file
file, err := os.Create("cloud.jpg")
// TODO: handle error
defer file.Close()

// download the file
_, err = client.DownloadFile(context.TODO(), file, nil)
// TODO: handle error
```

### Creating and deleting a filesystem

```go
const (
fs = "https://MYSTORAGEACCOUNT.dfs.core.windows.net/sample-fs"
)

// authenticate with Azure Active Directory
cred, err := azidentity.NewDefaultAzureCredential(nil)
// TODO: handle error

// create a client for the specified storage account
client, err := filesystem.NewClient(fs, cred, nil)
// TODO: handle error

_, err = client.Create(context.TODO(), nil)
// TODO: handle error

_, err = client.Delete(context.TODO(), nil)
// TODO: handle error
```

### Enumerating paths (assuming filesystem exists)

```go
const (
fs = "https://MYSTORAGEACCOUNT.dfs.core.windows.net/sample-fs"
)

// authenticate with Azure Active Directory
cred, err := azidentity.NewDefaultAzureCredential(nil)
// TODO: handle error

// create a filesystem client for the specified storage account
client, err := filesystem.NewClient(fs, cred, nil)
// TODO: handle error

// path listings are returned across multiple pages
pager := client.NewListPathsPager(true, nil)

// continue fetching pages until no more remain
for pager.More() {
// advance to the next page
page, err := pager.NextPage(context.TODO())
// TODO: handle error

// print the path names for this page
for _, path := range page.PathList.Paths {
fmt.Println(*path.Name)
fmt.Println(*path.IsDirectory)
}
}
```

## Troubleshooting

All Datalake service operations will return an
[*azcore.ResponseError][azcore_response_error] on failure with a
populated `ErrorCode` field. Many of these errors are recoverable.
The [datalakeerror][datalake_error] package provides the possible Storage error codes
along with various helper facilities for error handling.


### Specialized clients

The ADLS Gen2 Storage SDK for Go provides specialized clients in various subpackages.

The [file][file] package contains APIs related to file path types.

The [directory][directory] package contains APIs related to directory path types.

The [lease][lease] package contains clients for managing leases on paths (paths represent both directory and file paths) and filesystems. Please see the [reference docs](https://docs.microsoft.com/rest/api/storageservices/lease-blob#remarks) for general information on leases.

The [filesystem][filesystem] package contains APIs specific to filesystems. This includes APIs setting access policies or properties, and more.

The [service][service] package contains APIs specific to Datalake service. This includes APIs for manipulating filesystems, retrieving account information, and more.

The [sas][sas] package contains utilities to aid in the creation and manipulation of Shared Access Signature tokens.
See the package's documentation for more information.


You can find additional context and examples in our samples for each subpackage (named examples_test.go).

## Contributing

See the [Storage CONTRIBUTING.md][storage_contrib] for details on building,
testing, and contributing to this library.

This project welcomes contributions and suggestions. Most contributions require
you to agree to a Contributor License Agreement (CLA) declaring that you have
the right to, and actually do, grant us the rights to use your contribution. For
details, visit [cla.microsoft.com][cla].

This project has adopted the [Microsoft Open Source Code of Conduct][coc].
For more information see the [Code of Conduct FAQ][coc_faq]
or contact [opencode@microsoft.com][coc_contact] with any
additional questions or comments.

<!-- LINKS -->
[source]: https://github.com/Azure/azure-sdk-for-go/tree/main/sdk/storage/azdatalake
[docs]: https://pkg.go.dev/github.com/Azure/azure-sdk-for-go/sdk/storage/azdatalake
[rest_docs]: https://docs.microsoft.com/rest/api/storageservices/data-lake-storage-gen2
[godevdl]: https://go.dev/dl/
[goget]: https://pkg.go.dev/cmd/go#hdr-Add_dependencies_to_current_module_and_install_them
[storage_account_docs]: https://docs.microsoft.com/azure/storage/common/storage-account-overview
[storage_account_create_ps]: https://docs.microsoft.com/azure/storage/common/storage-quickstart-create-account?tabs=azure-powershell
[storage_account_create_cli]: https://docs.microsoft.com/azure/storage/common/storage-quickstart-create-account?tabs=azure-cli
[storage_account_create_portal]: https://docs.microsoft.com/azure/storage/common/storage-quickstart-create-account?tabs=azure-portal
[azure_sub]: https://azure.microsoft.com/free/
[azidentity]: https://pkg.go.dev/github.com/Azure/azure-sdk-for-go/sdk/azidentity
[storage_ad]: https://docs.microsoft.com/azure/storage/common/storage-auth-aad
[azcore_response_error]: https://pkg.go.dev/github.com/Azure/azure-sdk-for-go/sdk/azcore#ResponseError
[datalake_error]: https://github.com/Azure/azure-sdk-for-go/tree/main/sdk/storage/azdatalake/datalakeerror/error_codes.go
[filesystem]: https://github.com/Azure/azure-sdk-for-go/tree/main/sdk/storage/azdatalake/filesystem/client.go
[lease]: https://github.com/Azure/azure-sdk-for-go/tree/main/sdk/storage/azdatalake/lease
[file]: https://github.com/Azure/azure-sdk-for-go/tree/main/sdk/storage/azdatalake/file/client.go
[directory]: https://github.com/Azure/azure-sdk-for-go/tree/main/sdk/storage/azdatalake/directory/client.go
[sas]: https://github.com/Azure/azure-sdk-for-go/tree/main/sdk/storage/azdatalake/sas
[service]: https://github.com/Azure/azure-sdk-for-go/tree/main/sdk/storage/azdatalake/service/client.go
[storage_contrib]: https://github.com/Azure/azure-sdk-for-go/blob/main/CONTRIBUTING.md
[cla]: https://cla.microsoft.com
[coc]: https://opensource.microsoft.com/codeofconduct/
[coc_faq]: https://opensource.microsoft.com/codeofconduct/faq/
[coc_contact]: mailto:opencode@microsoft.com
2 changes: 1 addition & 1 deletion sdk/storage/azdatalake/assets.json
Original file line number Diff line number Diff line change
Expand Up @@ -2,5 +2,5 @@
"AssetsRepo": "Azure/azure-sdk-assets",
"AssetsRepoPrefixPath": "go",
"TagPrefix": "go/storage/azdatalake",
"Tag": "go/storage/azdatalake_ec80a91bf3"
"Tag": "go/storage/azdatalake_c3c16cffab"
}
13 changes: 7 additions & 6 deletions sdk/storage/azdatalake/datalakeerror/error_codes.go
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,7 @@ func HasCode(err error, codes ...StorageErrorCode) bool {
return false
}

// StorageErrorCode - Error codes returned by the service
type StorageErrorCode string

// dfs errors
Expand Down Expand Up @@ -79,10 +80,10 @@ const (
CannotVerifyCopySource StorageErrorCode = "CannotVerifyCopySource"
ConditionHeadersNotSupported StorageErrorCode = "ConditionHeadersNotSupported"
ConditionNotMet StorageErrorCode = "ConditionNotMet"
FilesystemAlreadyExists StorageErrorCode = "FilesystemAlreadyExists"
FilesystemBeingDeleted StorageErrorCode = "FilesystemBeingDeleted"
FilesystemDisabled StorageErrorCode = "FilesystemDisabled"
FilesystemNotFound StorageErrorCode = "FilesystemNotFound"
FileSystemAlreadyExists StorageErrorCode = "FileSystemAlreadyExists"
FileSystemBeingDeleted StorageErrorCode = "FileSystemBeingDeleted"
FileSystemDisabled StorageErrorCode = "FileSystemDisabled"
FileSystemNotFound StorageErrorCode = "FileSystemNotFound"
ContentLengthLargerThanTierLimit StorageErrorCode = "ContentLengthLargerThanTierLimit"
CopyAcrossAccountsNotSupported StorageErrorCode = "CopyAcrossAccountsNotSupported"
CopyIDMismatch StorageErrorCode = "CopyIdMismatch"
Expand Down Expand Up @@ -119,15 +120,15 @@ const (
LeaseAlreadyBroken StorageErrorCode = "LeaseAlreadyBroken"
LeaseAlreadyPresent StorageErrorCode = "LeaseAlreadyPresent"
LeaseIDMismatchWithPathOperation StorageErrorCode = "LeaseIdMismatchWithPathOperation"
LeaseIDMismatchWithFilesystemOperation StorageErrorCode = "LeaseIdMismatchWithFilesystemOperation"
LeaseIDMismatchWithFileSystemOperation StorageErrorCode = "LeaseIdMismatchWithFileSystemOperation"
LeaseIDMismatchWithLeaseOperation StorageErrorCode = "LeaseIdMismatchWithLeaseOperation"
LeaseIDMissing StorageErrorCode = "LeaseIdMissing"
LeaseIsBreakingAndCannotBeAcquired StorageErrorCode = "LeaseIsBreakingAndCannotBeAcquired"
LeaseIsBreakingAndCannotBeChanged StorageErrorCode = "LeaseIsBreakingAndCannotBeChanged"
LeaseIsBrokenAndCannotBeRenewed StorageErrorCode = "LeaseIsBrokenAndCannotBeRenewed"
LeaseLost StorageErrorCode = "LeaseLost"
LeaseNotPresentWithPathOperation StorageErrorCode = "LeaseNotPresentWithPathOperation"
LeaseNotPresentWithFilesystemOperation StorageErrorCode = "LeaseNotPresentWithFilesystemOperation"
LeaseNotPresentWithFileSystemOperation StorageErrorCode = "LeaseNotPresentWithFileSystemOperation"
LeaseNotPresentWithLeaseOperation StorageErrorCode = "LeaseNotPresentWithLeaseOperation"
MD5Mismatch StorageErrorCode = "Md5Mismatch"
CRC64Mismatch StorageErrorCode = "Crc64Mismatch"
Expand Down
Loading

0 comments on commit d366659

Please sign in to comment.