Skip to content

Commit

Permalink
Readme (#12972)
Browse files Browse the repository at this point in the history 
* Readme

* PR feedback

* updates
  • Loading branch information
@rickle-msft
rickle-msft authored Jul 10, 2020
1 parent 9bc23c3 commit 7612f32
Show file tree
Hide file tree
Showing 6 changed files with 295 additions and 46 deletions.
2 changes: 1 addition & 1 deletion eng/code-quality-reports/src/main/resources/checkstyle/checkstyle-suppressions.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -126,7 +126,7 @@

<!-- Don't apply custom Checkstyle rules to files under samples package -->
<suppress
checks="com\.azure\.tools\.checkstyle\.checks\.(ExternalDependencyExposedCheck|HttpPipelinePolicyCheck|EnforceFinalFieldsCheck|ThrowFromClientLoggerCheck|GoodLoggingCheck)"
checks="com\.azure\.tools\.checkstyle\.checks\.(ExternalDependencyExposedCheck|HttpPipelinePolicyCheck|EnforceFinalFieldsCheck|ThrowFromClientLoggerCheck|GoodLoggingCheck|JavadocThrowsChecks)"
files=".*[/\\]samples[/\\].*\.java"/>

<!-- Don't check for JavaDocPackage in samples or tests -->
Expand Down
225 changes: 188 additions & 37 deletions sdk/storage/azure-storage-blob-nio/README.md
View file
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
# Azure Storage Java NIO Blob plugin library for Java
## This README is not yet updated for this project and is a skeleton copied from blobs
# Azure Storage Blob NIO FileSystemProvider

This package allows you to interact with Azure Blob Storage through the standard Java NIO Filesystem APIs.

[Source code][source] | [API reference documentation][docs] | [REST API documentation][rest_docs] | [Product documentation][product_docs] | [Samples][samples]

Expand Down Expand Up @@ -35,8 +36,8 @@ az storage account create \

### Authenticate the client

In order to interact with the Storage Service (Blob, Queue, Message, MessageId, File) you'll need to create an instance of the Service Client class.
To make this possible you'll need the Account SAS (shared access signature) string of the Storage Account. Learn more at [SAS Token][sas_token]
The simplest way to interact with the Storage Service is to create an instance of the [FileSystem][file_system] class using the [FileSystems][file_systems] API.
To make this possible you'll need the Account SAS (shared access signature) string of the Storage Account or a Shared Key. Learn more at [SAS Token][sas_token] and [Shared Key][shared_key]

#### Get credentials

Expand Down Expand Up @@ -76,55 +77,185 @@ b. Alternatively, get the Account SAS Token from the Azure Portal.

##### **Shared Key Credential**

a. Use Account name and Account key. Account name is your Storage Account name.
Use Account name and Account key. Account name is your Storage Account name.

1. Go to your Storage Account
2. Select `Access keys` from the menu on the left
3. Under `key1`/`key2` copy the contents of the `Key` field

or
## Key concepts

b. Use the connection string.
NIO on top of Blob Storage is designed for:

- Working with Blob Storage as though it were a local file system
- Random access reads on large blobs without downloading the entire blob
- Uploading full files as blobs
- Creating and navigating a directory structure within an account
- Reading and setting attributes on blobs

## Design Notes
It is important to recognize that Azure Blob Storage is not a true FileSystem, nor is it the goal of this project to
force Azure Blob Storage to act like a full-fledged FileSystem. While providing FileSystem APIs on top of Azure Blob
Storage can offer convenience and ease of access in certain cases, trying to force the Storage service to work in
scenarios it is not designed for is bound to introduce performance and stability problems.

To that end, this project will only offer APIs that can be sensibly and cleanly built on top of Azure Blob Storage APIs.
We recognize that this will leave some scenarios unsupported indefinitely, but we would rather offer a product that
works predictably and reliably in its well defined scenarios than eagerly support all possible scenarios at the expense
of quality. Even still, supporting some fundamentally required use cases, such as directories, can result in unexpected
behavior due to the difference between blob storage and a file system. The javadocs on each type and method should
therefore be read and understood for ways in which they may diverge from the standard specified by the JDK.

Moreover, even from within a given application, it should be remembered that using a remote FileSystem introduces higher
latency. Because of this, particular care must be taken when managing concurrency. Race conditions are more likely to
manifest, network failures occur more frequently than disk failures, and other such distributed application scenarios
must be considered when working with this FileSystem. While the AzureFileSystem will ensure it takes appropriate steps
towards robustness and reliability, the application developer must also design around these failure scenarios and have
fallback and retry options available.

The view of the FileSystem from within an instance of the JVM will be consistent, but the AzureFileSystem makes no
guarantees on behavior or state should other processes operate on the same data. The AzureFileSystem will assume that it
has exclusive access to the resources stored in Azure Blob Storage and will behave without regard for potential
interfering applications

Finally, this implementation has currently chosen to always read/write directly to/from Azure Storage without a local
cache. Our team has determined that with the tradeoffs of complexity, correctness, safety, performance, debuggability,
etc. one option is not inherently better than the other and that this choice most directly addresses the current known
use cases for this project. While this has consequences for every API, of particular note is the limitations on writing
data. Data may only be written as an entire file (i.e. random IO or appends are not supported), and data is not
committed or available to be read until the write stream is closed.

1. Go to your Storage Account
2. Select `Access keys` from the menu on the left
3. Under `key1`/`key2` copy the contents of the `Connection string` field
## Examples

## Key concepts
The following sections provide several code snippets covering some of the most common Azure Storage Blob NIO tasks, including:

Blob Storage is designed for:
- [Create a `FileSystem`](#create-a-filesystem)
- [Create a directory](#create-a-directory)
- [Iterate over directory contents](#iterate-over-directory-contents)
- [Read a file](#read-a-file)
- [Write to a file](#write-to-a-file)
- [Copy a file](#copy-a-file)
- [Delete a file](#delete-a-file)
- [Read attributes on a file](#read-attributes-on-a-file)
- [Write attributes to a file](#write-attributes-to-a-file)

- Serving images or documents directly to a browser
- Storing files for distributed access
- Streaming video and audio
- Writing to log files
- Storing data for backup and restore, disaster recovery, and archiving
- Storing data for analysis by an on-premises or Azure-hosted service
### Create a `FileSystem`

## Examples
Create a `FileSystem` using the [`shared key`](#get-credentials) retrieved above.

The following sections provide several code snippets covering some of the most common Azure Storage Blob tasks, including:
Note that you can further configure the file system using constants available in `AzureFileSystem`.
Please see the docs for `AzureFileSystemProvider` for a full explanation of initializing and configuring a filesystem

- [Create a `BlobServiceClient`](#create-a-blobserviceclient)
<!-- embedme ./src/samples/java/com/azure/storage/blob/nio/ReadmeSamples.java#L39-L42 -->
```java
Map<String, Object> config = new HashMap<>();
config.put(AzureFileSystem.AZURE_STORAGE_ACCOUNT_KEY, "<your_account_key>");
config.put(AzureFileSystem.AZURE_STORAGE_FILE_STORES, "<container_names>");
FileSystem myFs = FileSystems.newFileSystem(new URI("azb://?account=<your_account_name"), config);
```

### Create a directory

Create a directory using the `Files` api

<!-- embedme ./src/samples/java/com/azure/storage/blob/nio/ReadmeSamples.java#L46-L47 -->
```java
Path dirPath = myFs.getPath("dir");
Files.createDirectory(dirPath);
```

### Create a `BlobServiceClient`
### Iterate over directory contents

Create a `BlobServiceClient` using the [`sasToken`](#get-credentials) generated above.
Iterate over a directory using a `DirectoryStream`

<!-- embedme ./src/samples/java/com/azure/storage/blob/nio/ReadmeSamples.java#L51-L53 -->
```java
BlobServiceClient blobServiceClient = new BlobServiceClientBuilder()
.endpoint("<your-storage-blob-url>")
.sasToken("<your-sasToken>")
.buildClient();
for (Path p : Files.newDirectoryStream(dirPath)) {
System.out.println(p.toString());
}
```

### Read a file

Read the contents of a file using an `InputStream`. Skipping, marking, and resetting are all supported.

<!-- embedme ./src/samples/java/com/azure/storage/blob/nio/ReadmeSamples.java#L57-L60 -->
```java
Path filePath = myFs.getPath("file");
InputStream is = Files.newInputStream(filePath);
is.read();
is.close();
```

### Write to a file

Write to a file. Only writing whole files is supported. Random IO is not supported. The stream must be closed in order
to guarantee that the data is available to be read.

<!-- embedme ./src/samples/java/com/azure/storage/blob/nio/ReadmeSamples.java#L64-L66 -->
```java
OutputStream os = Files.newOutputStream(filePath);
os.write(0);
os.close();
```

### Copy a file

<!-- embedme ./src/samples/java/com/azure/storage/blob/nio/ReadmeSamples.java#L70-L71 -->
```java
Path destinationPath = myFs.getPath("destinationFile");
Files.copy(filePath, destinationPath, StandardCopyOption.COPY_ATTRIBUTES);
```

### Delete a file

<!-- embedme ./src/samples/java/com/azure/storage/blob/nio/ReadmeSamples.java#L75-L75 -->
```java
Files.delete(filePath);
```

### Read attributes on a file

Read attributes of a file through the `AzureBlobFileAttributes`.

<!-- embedme ./src/samples/java/com/azure/storage/blob/nio/ReadmeSamples.java#L79-L80 -->
```java
AzureBlobFileAttributes attr = Files.readAttributes(filePath, AzureBlobFileAttributes.class);
BlobHttpHeaders headers = attr.blobHttpHeaders();
```

Or read attributes dynamically by specifying a string of desired attributes. This will not improve performance as a call
to retrieve any attribute will always retrieve all of them as an atomic bulk operation. You may specify "*" instead of a
list of specific attributes to have all attributes returned in the map.

<!-- embedme ./src/samples/java/com/azure/storage/blob/nio/ReadmeSamples.java#L84-L84 -->
```java
Map<String, Object> attributes = Files.readAttributes(filePath, "azureBlob:metadata,headers");
```

### Write attributes to a file

Set attributes of a file through the `AzureBlobFileAttributeView`.

<!-- embedme ./src/samples/java/com/azure/storage/blob/nio/ReadmeSamples.java#L88-L89 -->
```java
AzureBlobFileAttributeView view = Files.getFileAttributeView(filePath, AzureBlobFileAttributeView.class);
view.setMetadata(Collections.EMPTY_MAP);
```

Or set an attribute dynamically by specifying the attribute as a string.

<!-- embedme ./src/samples/java/com/azure/storage/blob/nio/ReadmeSamples.java#L93-L93 -->
```java
Files.setAttribute(filePath, "azureBlob:blobHttpHeaders", new BlobHttpHeaders());
```

## Troubleshooting

When interacting with blobs using this Java client library, errors returned by the service correspond to the same HTTP
status codes returned for [REST API][error_codes] requests. For example, if you try to retrieve a container or blob that
doesn't exist in your Storage Account, a `404` error is returned, indicating `Not Found`.
When using the NIO implementation for Azure Blob Storage, errors returned by the service are manifested as an
`IOException` which wraps a `BlobStorageException` having the same HTTP status codes returned for
[REST API][error_codes] requests. For example, if you try to read a file that doesn't exist in your Storage Account, a
`404` error is returned, indicating `Not Found`.

### Default HTTP Client
All client libraries by default use the Netty HTTP client. Adding the above dependency will automatically configure
Expand All @@ -137,12 +268,29 @@ operations. The Boring SSL library is an uber jar containing native libraries fo
better performance compared to the default SSL implementation within the JDK. For more information, including how to
reduce the dependency size, refer to the [performance tuning][performance_tuning] section of the wiki.

## Next steps

Several Storage blob Java SDK samples are available to you in the SDK's GitHub repository. These samples provide example code for additional scenarios commonly encountered while working with Key Vault:

## Next steps Samples
Samples are explained in detail [here][samples_readme].
## Continued development

This project is still actively being developed in an effort to move from preview to GA. Below is a list of features that
are not currently supported but are under consideration and may be added before GA. We welcome feedback and input on
which of these may be most useful and are open to suggestions for items not included in this list. While all of these
items are being considered, they have not been investigated and designed and therefore we cannot confirm their
feasibility within Azure Blob Storage. Therefore, it may be the case that further investigation reveals a feature may
not be possible or otherwise may conflict with established design goals and therefor will not ultimately be supported.

- Symbolic links
- Hard links
- Hidden files
- Random writes
- File locks
- Watches on directory events
- Support for other Azure Storage services such as ADLS Gen 2 (Datalake) and Azure Files (shares)
- Token authentication
- Multi-account filesystems
- Delegating access to single files
- Normalizing directory structure of data upon loading a FileSystem
- Local caching
- Other `OpenOptions` such as append or dsync
- Flags to toggle certain behaviors such as FileStore (container) creation, etc.

## Contributing

Expand All @@ -159,18 +307,21 @@ This project has adopted the [Microsoft Open Source Code of Conduct][coc]. For m
[rest_docs]: https://docs.microsoft.com/rest/api/storageservices/blob-service-rest-api
[product_docs]: https://docs.microsoft.com/azure/storage/blobs/storage-blobs-overview
[sas_token]: https://docs.microsoft.com/azure/storage/common/storage-dotnet-shared-access-signature-part-1
[shared_key]: https://docs.microsoft.com/en-us/rest/api/storageservices/authorize-with-shared-key
[jdk]: https://docs.microsoft.com/java/azure/jdk/
[azure_subscription]: https://azure.microsoft.com/free/
[storage_account]: https://docs.microsoft.com/azure/storage/common/storage-quickstart-create-account?tabs=azure-portal
[storage_account_create_cli]: https://docs.microsoft.com/azure/storage/common/storage-quickstart-create-account?tabs=azure-cli
[storage_account_create_portal]: https://docs.microsoft.com/azure/storage/common/storage-quickstart-create-account?tabs=azure-portal
[identity]: https://github.com/Azure/azure-sdk-for-java/blob/master/sdk/identity/azure-identity/README.md
[error_codes]: https://docs.microsoft.com/rest/api/storageservices/blob-service-error-codes
[samples]: src/samples
[samples]: https://docs.oracle.com/javase/tutorial/essential/io/fileio.html
[cla]: https://cla.microsoft.com
[coc]: https://opensource.microsoft.com/codeofconduct/
[coc_faq]: https://opensource.microsoft.com/codeofconduct/faq/
[coc_contact]: mailto:opencode@microsoft.com
[performance_tuning]: https://github.com/Azure/azure-sdk-for-java/wiki/Performance-Tuning
[file_system]: https://docs.oracle.com/javase/7/docs/api/java/nio/file/FileSystem.html
[file_systems]: https://docs.oracle.com/javase/7/docs/api/java/nio/file/FileSystems.html

![Impressions](https://azure-sdk-impressions.azurewebsites.net/api/impressions/azure-sdk-for-java%2Fsdk%2Fstorage%2Fazure-storage-blob%2FREADME.png)
4 changes: 3 additions & 1 deletion ...torage-blob-nio/src/main/java/com/azure/storage/blob/nio/AzureBasicFileAttributeView.java
View file
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,8 @@
*/
public final class AzureBasicFileAttributeView implements BasicFileAttributeView {

static final String NAME = "azureBasic";

private final Path path;

AzureBasicFileAttributeView(Path path) {
Expand All @@ -33,7 +35,7 @@ public final class AzureBasicFileAttributeView implements BasicFileAttributeView
*/
@Override
public String name() {
return "azureBasic";
return NAME;
}

/**
Expand Down
3 changes: 2 additions & 1 deletion ...storage-blob-nio/src/main/java/com/azure/storage/blob/nio/AzureBlobFileAttributeView.java
View file
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,7 @@ public final class AzureBlobFileAttributeView implements BasicFileAttributeView
private final ClientLogger logger = new ClientLogger(AzureBlobFileAttributeView.class);

static final String ATTR_CONSUMER_ERROR = "Exception thrown by attribute consumer";
static final String NAME = "azureBlob";

private final Path path;

Expand Down Expand Up @@ -74,7 +75,7 @@ static Map<String, Consumer<Object>> setAttributeConsumers(AzureBlobFileAttribut
*/
@Override
public String name() {
return "azureBlob";
return NAME;
}

/**
Expand Down
12 changes: 6 additions & 6 deletions ...re-storage-blob-nio/src/main/java/com/azure/storage/blob/nio/AzureFileSystemProvider.java
View file
Original file line number Diff line number Diff line change
Expand Up @@ -755,9 +755,9 @@ public Map<String, Object> readAttributes(Path path, String s, LinkOption... lin
state that "basic" must be supported, so we funnel to azureBasic.
*/
if (viewType.equals("basic")) {
viewType = "azureBasic";
viewType = AzureBasicFileAttributeView.NAME;
}
if (!viewType.equals("azureBasic") && !viewType.equals("azureBlob")) {
if (!viewType.equals(AzureBasicFileAttributeView.NAME) && !viewType.equals(AzureBlobFileAttributeView.NAME)) {
throw LoggingUtility.logError(logger,
new UnsupportedOperationException("Invalid attribute view: " + viewType));
}
Expand All @@ -768,7 +768,7 @@ public Map<String, Object> readAttributes(Path path, String s, LinkOption... lin
should at least validate that the attribute is available on a basic view.
*/
// TODO: Put these strings in constants
if (viewType.equals("azureBasic")) {
if (viewType.equals(AzureBasicFileAttributeView.NAME)) {
if (!AzureBasicFileAttributes.ATTRIBUTE_STRINGS.contains(attributeName) && !attributeName.equals("*")) {
throw LoggingUtility.logError(logger,
new IllegalArgumentException("Invalid attribute. View: " + viewType
Expand All @@ -786,7 +786,7 @@ public Map<String, Object> readAttributes(Path path, String s, LinkOption... lin
// If "*" is specified, add all of the attributes from the specified set.
if (attributeName.equals("*")) {
Set<String> attributesToAdd;
if (viewType.equals("azureBasic")) {
if (viewType.equals(AzureBasicFileAttributeView.NAME)) {
for (String attr : AzureBasicFileAttributes.ATTRIBUTE_STRINGS) {
results.put(attr, attributeSuppliers.get(attr).get());
}
Expand Down Expand Up @@ -848,15 +848,15 @@ public void setAttribute(Path path, String s, Object o, LinkOption... linkOption
state that "basic" must be supported, so we funnel to azureBasic.
*/
if (viewType.equals("basic")) {
viewType = "azureBasic";
viewType = AzureBasicFileAttributeView.NAME;
}

// We don't actually support any setters on the basic view.
if (viewType.equals("azureBasic")) {
throw LoggingUtility.logError(logger,
new IllegalArgumentException("Invalid attribute. View: " + viewType
+ ". Attribute: " + attributeName));
} else if (viewType.equals("azureBlob")) {
} else if (viewType.equals(AzureBlobFileAttributeView.NAME)) {
Map<String, Consumer<Object>> attributeConsumers = AzureBlobFileAttributeView.setAttributeConsumers(
this.getFileAttributeView(path, AzureBlobFileAttributeView.class, linkOptions));
if (!attributeConsumers.containsKey(attributeName)) {
Expand Down
Loading

0 comments on commit 7612f32

Please sign in to comment.