Skip to content

Commit

Permalink
Fix policy test flakiness (Azure#4110)
Browse files Browse the repository at this point in the history
There was another issue in Policy tests that make them flaky. `LeafDevice.CreateAsync` appears to be not idempotent, so there is a race condition, when LeafDevice tries to connect before the policy is updated, and in case of failed authorization, it tries to delete and re-create the device. That screws up cached symmetric keys in EH. And since EH does not refresh the device scopes (and keys) more often that 2min, the test fails after several retries.

The solution is to not to use `LeafDevice` - not to remove the device in case of authorization error, but just retry again.
  • Loading branch information
vadim-kovalyov authored Dec 17, 2020
1 parent 7f5c92d commit f827912
Show file tree
Hide file tree
Showing 2 changed files with 24 additions and 24 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,6 @@ namespace Microsoft.Azure.Devices.Edge.Test.Common
using System.IO;
using System.Linq;
using System.Net;
using System.Reflection.Metadata.Ecma335;
using System.Security.Cryptography.X509Certificates;
using System.Text;
using System.Threading;
Expand Down
47 changes: 24 additions & 23 deletions test/Microsoft.Azure.Devices.Edge.Test/AuthorizationPolicy.cs
Original file line number Diff line number Diff line change
Expand Up @@ -3,8 +3,10 @@ namespace Microsoft.Azure.Devices.Edge.Test
{
using System;
using System.Collections.Generic;
using System.Net;
using System.Threading;
using System.Threading.Tasks;
using Microsoft.Azure.Devices.Client;
using Microsoft.Azure.Devices.Client.Exceptions;
using Microsoft.Azure.Devices.Edge.Test.Common;
using Microsoft.Azure.Devices.Edge.Test.Common.Certs;
Expand Down Expand Up @@ -135,38 +137,37 @@ await edgeHub.WaitForReportedPropertyUpdatesAsync(
},
token);

// Create device manually. We can't use LeafDevice.CreateAsync() since it is not
// idempotent and cannot be retried reliably.
Devices.Device edge = await this.iotHub.GetDeviceIdentityAsync(this.runtime.DeviceId, token);
Devices.Device leaf = new Devices.Device(deviceId2)
{
Authentication = new AuthenticationMechanism
{
Type = AuthenticationType.Sas
},
Scope = edge.Scope
};

leaf = await this.iotHub.CreateDeviceIdentityAsync(leaf, token);
string connectionString =
$"HostName={this.iotHub.Hostname};" +
$"DeviceId={leaf.Id};" +
$"SharedAccessKey={leaf.Authentication.SymmetricKey.PrimaryKey};" +
$"GatewayHostName={Dns.GetHostName().ToLower()}";

// There is no reliable way to signal when the policy
// is updated in $edgehub, so need to retry several times.
//
// DefaultProgressive => 55 sec max.
LeafDevice leaf = null;
await RetryPolicy.DefaultProgressive.ExecuteAsync(
async () =>
{
leaf = await LeafDevice.CreateAsync(
deviceId2,
Protocol.Mqtt,
AuthenticationType.Sas,
Option.Some(this.runtime.DeviceId),
false,
CertificateAuthority.GetQuickstart(),
this.iotHub,
token,
Option.None<string>());
using var client = DeviceClient.CreateFromConnectionString(connectionString);
await client.OpenAsync();
}, token);

// verify device is authorized after policy update.
await TryFinally.DoAsync(
async () =>
{
DateTime seekTime = DateTime.Now;
await leaf.SendEventAsync(token);
await leaf.WaitForEventsReceivedAsync(seekTime, token);
},
async () =>
{
await leaf.DeleteIdentityAsync(token);
});
await this.iotHub.DeleteDeviceIdentityAsync(leaf, token);
}

/// <summary>
Expand Down

0 comments on commit f827912

Please sign in to comment.