Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Do not Squash][Storage] Merge master change to Storage branch. #12602

Merged
merged 43 commits into from
Aug 10, 2020
Rate limit · GitHub

Access has been restricted

You have triggered a rate limit.

Please wait a few minutes before you try again;
in some cases this may take up to an hour.

Merged
Changes from 1 commit
Commits
Show all changes
43 commits
Select commit Hold shift + click to select a range
76570c0
[Storage] Add preview code change (#11474)
blueww Apr 3, 2020
6b92565
[Storage] Fix a missing assembly in move preview changes (#11490)
blueww Apr 7, 2020
bfcd434
Storage 1.13.4-preview (#11487)
dingmeng-xue Apr 10, 2020
c37b97b
[Storage] Add RestorePolicy.LastEnabledTime to cmdlet output (#11665)
blueww Apr 24, 2020
7e0fcb6
[Storage] Support Encryption Scope dataplane (#11676)
blueww Apr 24, 2020
2de9c77
Update the version (#11690)
wyunchi-ms Apr 26, 2020
16ae5f7
[Storage] Support ORS (#11752)
blueww Apr 30, 2020
ad8763c
[Storage] Support rounting preferences (#11778)
blueww May 2, 2020
8d731b2
[Storage] remove duplicated changelog from merge conflict
blueww May 6, 2020
b7c5d21
Merge pull request #11809 from wastoresh/mergestorage
dingmeng-xue May 6, 2020
c9c93d9
[Storage] Support share soft delete (#11779)
blueww May 6, 2020
d70577b
fix merge conflicts
blueww May 6, 2020
cbc7c53
Merge branch 'Az.Storage-preview' into mergestorage
blueww May 6, 2020
88992c8
[Storage] fix CI failure
blueww May 6, 2020
0e23495
Merge pull request #11813 from wastoresh/mergestorage
dingmeng-xue May 7, 2020
5004cdf
fix merge conflicts
blueww May 12, 2020
a71b647
Fix a CI failure
blueww May 12, 2020
e250838
Merge pull request #11880 from wastoresh/mergestorage
VeryEarly May 14, 2020
bc21465
[Storage] Support NFS (#11900)
blueww May 15, 2020
df72db5
merge from master
VeryEarly May 21, 2020
4258738
suppress breaking change issues
VeryEarly May 21, 2020
992f77f
bump up version for Az.Storage
VeryEarly May 21, 2020
ca06a5f
Merge pull request #11967 from VeryEarly/Az.Storage-Preview2
VeryEarly May 21, 2020
14db8a9
[Storage] Support share usage
blueww Feb 11, 2020
20a4761
[Storage] Support share Accesstier
blueww Apr 21, 2020
06f5d6b
Merge pull request #12013 from wastoresh/sharetier
VeryEarly May 29, 2020
415bcbb
fix merge conflicts
blueww Jun 2, 2020
0250bd1
Merge pull request #12043 from wastoresh/mergestorage
VeryEarly Jun 3, 2020
99a484a
upgrade Az.Storage to 2.1.1-preview
VeryEarly Jun 3, 2020
42ab0c3
Merge pull request #12053 from VeryEarly/Az.Storage-preview
VeryEarly Jun 5, 2020
a0e4cd1
fix merge conflicts
blueww Jun 30, 2020
fe455c5
Merge pull request #12295 from wastoresh/mergestorage
VeryEarly Jul 1, 2020
c733cde
fix merge conflicts
blueww Jul 7, 2020
ffcd379
Merge pull request #12345 from wastoresh/mergestorage
wyunchi-ms Jul 8, 2020
578135a
fix merge conflicts
blueww Jul 21, 2020
b992178
fix merge conflicts
blueww Jul 22, 2020
edc3568
Merge pull request #12446 from wastoresh/mergestorage
wyunchi-ms Jul 22, 2020
4dd364e
[Storage] Support blob tag (#12475)
blueww Jul 25, 2020
4847d04
fix merge conflicts
blueww Jul 27, 2020
1cfc32b
Merge pull request #12485 from wastoresh/mergestorage
isra-fel Jul 27, 2020
fb1eabc
[Storage] Fix some test cases (#12497)
blueww Aug 3, 2020
4eac030
Bump up version for Storage OOB release (#12499) (#12598)
isra-fel Aug 7, 2020
0992f85
fix merge conflicts
blueww Aug 7, 2020
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
fix merge conflicts
Rate limit · GitHub

Access has been restricted

You have triggered a rate limit.

Please wait a few minutes before you try again;
in some cases this may take up to an hour.

blueww committed May 6, 2020
commit d70577bb89a7f547a4fc5b5aabb7bbe09b240621
Original file line number Diff line number Diff line change
@@ -615,8 +615,8 @@ function Test-NetworkRule
foreach($iprule in $stoacliprule) {
$job = Add-AzStorageAccountNetworkRule -ResourceGroupName $rgname -Name $stoname -IpRule $iprule -AsJob
$job | Wait-Job
# add again should not fail
Add-AzStorageAccountNetworkRule -ResourceGroupName $rgname -Name $stoname -IpRule $iprule
# add again should not fail
Add-AzStorageAccountNetworkRule -ResourceGroupName $rgname -Name $stoname -IpRule $iprule
}

$stoacl = Get-AzStorageAccountNetworkRuleSet -ResourceGroupName $rgname -Name $stoname
@@ -714,52 +714,6 @@ function Test-GetAzureStorageLocationUsage
Assert-AreNotEqual 0 $usage.CurrentValue;
}

<#
.SYNOPSIS
Test Invoke-AzStorageAccountFailover
.DESCRIPTION
Smoke[Broken]Test
#>
function Test-FailoverAzureStorageAccount
{
# Setup
$rgname = Get-StorageManagementTestResourceName;

try
{
# Test
$stoname = 'sto' + $rgname;
$stotype = 'Standard_RAGRS';
$kind = 'StorageV2'

$loc = Get-ProviderLocation_Canary ResourceManagement;
New-AzResourceGroup -Name $rgname -Location $loc;

New-AzStorageAccount -ResourceGroupName $rgname -Name $stoname -Location $loc -Type $stotype -Kind $kind;
$sto = Get-AzStorageAccount -ResourceGroupName $rgname -Name $stoname;
Assert-AreEqual $stoname $sto.StorageAccountName;
Assert-AreEqual $stotype $sto.Sku.Name;
Assert-AreEqual $loc.ToLower().Replace(" ", "") $sto.Location;
Assert-AreEqual $kind $sto.Kind;
$seconcaryLocation = $sto.SecondaryLocation

#Invoke Failover
$job = Invoke-AzStorageAccountFailover -ResourceGroupName $rgname -Name $stoname -Force -AsJob
$job | Wait-Job

$sto = Get-AzStorageAccount -ResourceGroupName $rgname -Name $stoname;
Assert-AreEqual $seconcaryLocation $sto.PrimaryLocation;
Assert-AreEqual 'Standard_LRS' $sto.Sku.Name;

Retry-IfException { Remove-AzStorageAccount -Force -ResourceGroupName $rgname -Name $stoname; }
}
finally
{
# Cleanup
Clean-ResourceGroup $rgname
}
}

<#
.SYNOPSIS
Test New-AzStorageAccountFileStorage
6 changes: 2 additions & 4 deletions src/Storage/Storage.Management/Az.Storage.psd1
Original file line number Diff line number Diff line change
@@ -53,7 +53,7 @@ DotNetFrameworkVersion = '4.7.2'
# ProcessorArchitecture = ''

# Modules that must be imported into the global environment prior to importing this module
RequiredModules = @(@{ModuleName = 'Az.Accounts'; ModuleVersion = '1.7.3'; })
RequiredModules = @(@{ModuleName = 'Az.Accounts'; ModuleVersion = '1.7.5'; })

# Assemblies that must be loaded prior to importing this module
RequiredAssemblies = 'Microsoft.Azure.Management.Storage.dll',
@@ -67,9 +67,7 @@ RequiredAssemblies = 'Microsoft.Azure.Management.Storage.dll',
'Microsoft.Azure.DocumentDB.Core.dll', 'Microsoft.OData.Core.dll',
'Microsoft.OData.Edm.dll', 'Microsoft.Spatial.dll',
'Microsoft.Azure.KeyVault.Core.dll', 'Azure.Storage.Blobs.dll',
'Azure.Storage.Common.dll', 'Azure.Storage.Files.DataLake.dll',
'Azure.Core.dll', 'Microsoft.Bcl.AsyncInterfaces.dll',
'System.Text.Json.dll', 'System.Threading.Tasks.Extensions.dll'
'Azure.Storage.Common.dll', 'Azure.Storage.Files.DataLake.dll'

# Script files (.ps1) that are run in the caller's environment prior to importing this module.
# ScriptsToProcess = @()
23 changes: 23 additions & 0 deletions src/Storage/Storage.Management/ChangeLog.md
Original file line number Diff line number Diff line change
@@ -28,6 +28,29 @@
* Supported RoutingPreference settings in create/update Storage account
- `New-AzStorageAccount`
- `Set-AzStorageAccount`
* Add -Adjob to get/list account cmdlet
- `Get-AzStorageAccount`
* Make KeyVersion to optional when update Storage account with KeyvaultEncryption, to support key auto-rotation
- `Set-AzStorageAccount`
* Fix remove Azure File Directory fail with pipeline
- `Remove-AzStorageDirectory`
* Fixed #9880: Change NetWorkRule DefaultAction value defination to align with swagger.
- `Update-AzStorageAccountNetworkRuleSet`
- `Get-AzStorageAccountNetworkRuleSet`
* Fixed #11624: Skip duplicated rules when add NetworkRules, to avoid server failure
- `Add-AzStorageAccountNetworkRule`
* Upgrade Microsoft.Azure.Cosmos.Table SDK to 1.0.7
* When only part items are returned in list DataLake Gen2 Items, add a warning message to remind user to list again with ContinuationToken
- `Get-AzDataLakeGen2ChildItem`
* Support create or update Storage account with Azure Files Active Directory Domain Service Authentication
- `New-AzStorageAccount`
- `Set-AzStorageAccount`
* Support New or List Kerberos keys of Storage account
- `New-AzStorageAccountKey`
- `Get-AzStorageAccountKey`
* Support failover Storage account
- `Invoke-AzStorageAccountFailover`


## Version 1.14.1
* Supported create container and upload blob with Encryption Scope setting
2 changes: 0 additions & 2 deletions src/Storage/Storage.Management/Models/PSStorageAccount.cs
Original file line number Diff line number Diff line change
@@ -114,8 +114,6 @@ public PSStorageAccount(StorageModels.StorageAccount storageAccount)

public bool? EnableHierarchicalNamespace { get; set; }

public bool? FailoverInProgress { get; set; }

public string LargeFileSharesState { get; set; }

public bool? FailoverInProgress { get; set; }
Original file line number Diff line number Diff line change
@@ -52,9 +52,10 @@ public abstract class StorageAccountBaseCmdlet : AzureRMCmdlet

protected const string StorageUsageNounStr = "AzureRmStorageUsage";

internal const string StandardGZRS = "Standard_GZRS";
internal const string StandardRAGZRS = "Standard_RAGZRS";
internal const string FileStorage = "FileStorage";
internal const string StandardGZRS = "Standard_GZRS";
internal const string StandardRAGZRS = "Standard_RAGZRS";

internal const string FileStorage = "FileStorage";
protected struct AccountAccessTier
{
internal const string Hot = "Hot";
50 changes: 1 addition & 49 deletions src/Storage/Storage.Management/help/New-AzStorageAccount.md
Original file line number Diff line number Diff line change
@@ -38,19 +38,6 @@ New-AzStorageAccount [-ResourceGroupName] <String> [-Name] <String> [-SkuName] <
[-DefaultProfile <IAzureContextContainer>] [-RoutingChoice <String>] [<CommonParameters>]
```

### ActiveDirectoryDomainServicesForFile
```
New-AzStorageAccount [-ResourceGroupName] <String> [-Name] <String> [-SkuName] <String> [-Location] <String>
[-Kind <String>] [-AccessTier <String>] [-CustomDomainName <String>] [-UseSubDomain <Boolean>]
[-Tag <Hashtable>] [-EnableHttpsTrafficOnly <Boolean>] [-AssignIdentity] [-NetworkRuleSet <PSNetworkRuleSet>]
[-EnableHierarchicalNamespace <Boolean>] [-EnableLargeFileShare]
[-EnableActiveDirectoryDomainServicesForFile <Boolean>] [-ActiveDirectoryDomainName <String>]
[-ActiveDirectoryNetBiosDomainName <String>] [-ActiveDirectoryForestName <String>]
[-ActiveDirectoryDomainGuid <String>] [-ActiveDirectoryDomainSid <String>]
[-ActiveDirectoryAzureStorageSid <String>] [-AsJob] [-DefaultProfile <IAzureContextContainer>]
[<CommonParameters>]
```

## DESCRIPTION
The **New-AzStorageAccount** cmdlet creates an Azure Storage account.

@@ -116,40 +103,6 @@ PS C:\>New-AzStorageAccount -ResourceGroupName "MyResourceGroup" -AccountName "m

This command creates a Storage account withenable Files Active Directory Domain Service Authentication.

### Example 7: Create a Storage account with enable Files Active Directory Domain Service Authentication.
```
PS C:\>New-AzStorageAccount -ResourceGroupName "MyResourceGroup" -AccountName "mystorageaccount" -Location "eastus2euap" -SkuName "Standard_LRS" -Kind StorageV2 -EnableActiveDirectoryDomainServicesForFile $true `
-ActiveDirectoryDomainName "mydomain.com" `
-ActiveDirectoryNetBiosDomainName "mydomain.com" `
-ActiveDirectoryForestName "mydomain.com" `
-ActiveDirectoryDomainGuid "12345678-1234-1234-1234-123456789012" `
-ActiveDirectoryDomainSid "S-1-5-21-1234567890-1234567890-1234567890" `
-ActiveDirectoryAzureStorageSid "S-1-5-21-1234567890-1234567890-1234567890-1234"
```

This command creates a Storage account withenable Files Active Directory Domain Service Authentication.

### Example 7: Create a Storage account with Queue and Table Service use account-scoped encryption key.
```powershell
PS C:\>New-AzStorageAccount -ResourceGroupName "MyResourceGroup" -AccountName "mystorageaccount" -Location "eastus2euap" -SkuName "Standard_LRS" -Kind StorageV2 -EncryptionKeyTypeForTable Account -EncryptionKeyTypeForQueue Account

PS C:\>$account = get-AzStorageAccount -ResourceGroupName $rgname -StorageAccountName $accountName

PS C:\>$account.Encryption.Services.Queue

Enabled LastEnabledTime KeyType
------- --------------- -------
True 1/9/2020 6:09:11 AM Account

PS C:\>$account.Encryption.Services.Table

Enabled LastEnabledTime KeyType
------- --------------- -------
True 1/9/2020 6:09:11 AM Account
```

This command creates a Storage account with Queue and Table Service use account-scoped encryption key, so Queue and Table will use same encryption key with Blob and File service. Then get the Storage account properties, and view the encryption keytype of Queue and Table Service.

### Example 8: Create a Storage account with Queue and Table Service use account-scoped encryption key.
```
PS C:\>New-AzStorageAccount -ResourceGroupName "MyResourceGroup" -AccountName "mystorageaccount" -Location "eastus2euap" -SkuName "Standard_LRS" -Kind StorageV2 -EncryptionKeyTypeForTable Account -EncryptionKeyTypeForQueue Account
@@ -171,7 +124,7 @@ Enabled LastEnabledTime KeyType

This command creates a Storage account with Queue and Table Service use account-scoped encryption key, so Queue and Table will use same encryption key with Blob and File service. Then get the Storage account properties, and view the encryption keytype of Queue and Table Service.

### Example 10: Create a Storage account with RoutingPreference setting
### Example 9: Create a Storage account with RoutingPreference setting
```powershell
PS C:\>$account = New-AzStorageAccount -ResourceGroupName "MyResourceGroup" -AccountName "mystorageaccount" -Location "eastus2euap" -SkuName "Standard_LRS" -PublishMicrosoftEndpoint $true -PublishInternetEndpoint $true -RoutingChoice MicrosoftRouting

@@ -604,7 +557,6 @@ Routing Choice defines the kind of network routing opted by the user. Possible v
Type: System.String
Parameter Sets: (All)
Aliases:
Accepted values: MicrosoftRouting, InternetRouting

Required: False
Position: Named
13 changes: 0 additions & 13 deletions src/Storage/Storage.Management/help/Set-AzStorageAccount.md
Original file line number Diff line number Diff line change
@@ -48,18 +48,6 @@ Set-AzStorageAccount [-ResourceGroupName] <String> [-Name] <String> [-Force] [-S
[-DefaultProfile <IAzureContextContainer>] [-RoutingChoice <String>] [-WhatIf] [-Confirm] [<CommonParameters>]
```

### ActiveDirectoryDomainServicesForFile
```
Set-AzStorageAccount [-ResourceGroupName] <String> [-Name] <String> [-Force] [-SkuName <String>]
[-AccessTier <String>] [-CustomDomainName <String>] [-UseSubDomain <Boolean>] [-Tag <Hashtable>]
[-EnableHttpsTrafficOnly <Boolean>] [-AssignIdentity] [-NetworkRuleSet <PSNetworkRuleSet>]
[-UpgradeToStorageV2] [-EnableLargeFileShare] -EnableActiveDirectoryDomainServicesForFile <Boolean>
[-ActiveDirectoryDomainName <String>] [-ActiveDirectoryNetBiosDomainName <String>]
[-ActiveDirectoryForestName <String>] [-ActiveDirectoryDomainGuid <String>]
[-ActiveDirectoryDomainSid <String>] [-ActiveDirectoryAzureStorageSid <String>] [-AsJob]
[-DefaultProfile <IAzureContextContainer>] [-WhatIf] [-Confirm] [<CommonParameters>]
```

## DESCRIPTION
The **Set-AzStorageAccount** cmdlet modifies an Azure Storage account.
You can use this cmdlet to modify the account type, update a customer domain, or set tags on a Storage account.
@@ -596,7 +584,6 @@ Routing Choice defines the kind of network routing opted by the user. Possible v
Type: System.String
Parameter Sets: (All)
Aliases:
Accepted values: MicrosoftRouting, InternetRouting

Required: False
Position: Named
2 changes: 1 addition & 1 deletion src/Storage/Storage.Test/Storage.Test.csproj
Original file line number Diff line number Diff line change
@@ -13,7 +13,7 @@
</PropertyGroup>

<ItemGroup>
<PackageReference Include="Microsoft.Azure.Cosmos.Table" Version="0.10.1-preview" />
<PackageReference Include="Microsoft.Azure.Cosmos.Table" Version="1.0.7" />
<PackageReference Include="Microsoft.Azure.Storage.Blob" Version="11.1.1" />
<PackageReference Include="Microsoft.Azure.Storage.File" Version="11.1.1" />
<PackageReference Include="Microsoft.Azure.Storage.Queue" Version="11.1.1" />
19 changes: 19 additions & 0 deletions src/Storage/Storage/Blob/StorageCloudBlobCmdletBase.cs
Original file line number Diff line number Diff line change
@@ -523,6 +523,25 @@ public static IDictionary<string, string> GetUpdatedMetaData(Hashtable Metadata,
}
}

/// <summary>
/// Get Item string without SAS for confirmation string.
/// </summary>
/// <param name="item"></param>
/// <returns></returns>
protected static string GetDataLakeItemUriWithoutSas(DataLakePathClient item)
{
string uriString = item.Uri.ToString();
int length = uriString.IndexOf("?");
if (length < 0) // Not container "?"
{
return uriString;
}
else
{
return uriString.Substring(0, uriString.IndexOf("?"));
}
}

/// <summary>
/// get the DataLakeFileSystemClient object by name if DataLakeFileSystem exists
/// </summary>
Original file line number Diff line number Diff line change
@@ -125,6 +125,10 @@ public override void ExecuteCmdlet()
Page<PathItem> page;
enumerator.MoveNext();
page = enumerator.Current;
if(page.ContinuationToken != null && (MaxCount == null || page.Values.Count < MaxCount.Value))
{
WriteWarning(string.Format("Not all result returned, to list the left items run this cmdlet again with parameter: '-ContinuationToken {0}'.", page.ContinuationToken));
}
foreach (PathItem item in page.Values)
{
WriteDataLakeGen2Item(localChannel, item, fileSystem, page.ContinuationToken, this.FetchProperty.IsPresent);
Original file line number Diff line number Diff line change
@@ -254,7 +254,15 @@ public override void ExecuteCmdlet()
{
if (!InputObject.IsDirectory)
{
blob = new CloudBlockBlob(InputObject.File.Uri, Channel.StorageContext.StorageAccount.Credentials);
if (Channel.StorageContext.StorageAccount.Credentials.IsSAS)
{
// For SAS, the Uri already contains the sas token, so can't repeatedly inout the credential
blob = new CloudBlockBlob(InputObject.File.Uri);
}
else
{
blob = new CloudBlockBlob(InputObject.File.Uri, Channel.StorageContext.StorageAccount.Credentials);
}
fileClient = InputObject.File;
}
else
Original file line number Diff line number Diff line change
@@ -122,7 +122,7 @@ public override void ExecuteCmdlet()

if (foundAFolder)
{
if (ShouldProcess(srcBlobDir.Uri.ToString(), "Move Directory: "))
if (ShouldProcess(GetDataLakeItemUriWithoutSas(srcBlobDir), "Move Directory: "))
{
// check dest exist
bool destExist = true;
@@ -137,7 +137,7 @@ public override void ExecuteCmdlet()
destExist = false;
}

if (this.Force || !destExist || ShouldContinue(string.Format("Overwrite destination {0}", destBlobDir.Uri), ""))
if (this.Force || !destExist || ShouldContinue(string.Format("Overwrite destination {0}", GetDataLakeItemUriWithoutSas(destBlobDir)), ""))
{
destBlobDir = srcBlobDir.Rename(this.DestPath, this.DestFileSystem).Value;
WriteDataLakeGen2Item(localChannel, destBlobDir);
@@ -146,25 +146,25 @@ public override void ExecuteCmdlet()
}
else
{
if (ShouldProcess(srcBlob.Uri.ToString(), "Move File: "))
if (ShouldProcess(GetDataLakeItemUriWithoutSas(srcBlob), "Move File: "))
{
// check dest exist
bool destExist = true;
DataLakeFileSystemClient destFileSystem = GetFileSystemClientByName(localChannel, this.DestFileSystem != null ? this.DestFileSystem : this.FileSystem);
DataLakeFileClient destBlob = destFileSystem.GetFileClient(this.DestPath);
DataLakeFileClient destFile = destFileSystem.GetFileClient(this.DestPath);
try
{
destBlob.GetProperties();
destFile.GetProperties();
}
catch (RequestFailedException e) when (e.Status == 404)
{
destExist = false;
}

if (this.Force || !destExist || ShouldContinue(string.Format("Overwrite destination {0}", destBlob.Uri), ""))
if (this.Force || !destExist || ShouldContinue(string.Format("Overwrite destination {0}", GetDataLakeItemUriWithoutSas(destFile)), ""))
{
destBlob = srcBlob.Rename(this.DestPath, this.DestFileSystem).Value;
WriteDataLakeGen2Item(localChannel, destBlob);
destFile = srcBlob.Rename(this.DestPath, this.DestFileSystem).Value;
WriteDataLakeGen2Item(localChannel, destFile);
}
}
}
Rate limit · GitHub

Access has been restricted

You have triggered a rate limit.

Please wait a few minutes before you try again;
in some cases this may take up to an hour.

You are viewing a condensed version of this merge commit. You can view the full changes here.