Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make resource HealthStatus computed from HealthReports #6368

Merged
Merged
Show file tree
Hide file tree
Changes from 32 commits
Commits
Show all changes
43 commits
Select commit Hold shift + click to select a range
43ac440
Make resource HealthStatus computed from HealthReports
Oct 18, 2024
8b3406a
change HealthStatus back to property, fix test
Oct 18, 2024
c608065
Update HealthStatus when state changes
Oct 18, 2024
d8556f4
Return null HealthStatus when not running, use constant
Oct 18, 2024
8f605c9
add 16 to reserved fields
Oct 18, 2024
d4d6849
add default argument to _healthReports, re-add property
Oct 18, 2024
5955d5f
Say resource is unhealthy if it is running but a health check has not…
Oct 18, 2024
a29cc72
Change Health State -> Health state
Oct 18, 2024
e817af7
Remove redundant condition
Oct 18, 2024
20dd730
clean up
Oct 18, 2024
b864dad
add test for ComputeHealthStatus
Oct 18, 2024
1c0c161
Disambiguate have not received health reports and empty health reports
Oct 18, 2024
0e9aa7e
Revert change, add health reports to test PublishUpdate to reflect ac…
Oct 18, 2024
908ad8d
Set health reports to [] on resources that don't have health checks
Oct 18, 2024
b151230
Merge branch 'main' into dev/adamint/remove-aggregated-healthstatus
Oct 18, 2024
5d2d053
test disabling all rhc tests
Oct 19, 2024
4c5858e
Merge branch 'main' into dev/adamint/remove-aggregated-healthstatus
Oct 19, 2024
f6dd57c
try enabling 4 tests
Oct 19, 2024
3050ff8
enable additional 3 tests
Oct 19, 2024
94ee0cc
re-skip 2 checks
Oct 19, 2024
aeb4618
add initial health snapshots in resource notification service
Oct 19, 2024
b696e47
Fix playground.
mitchdenny Oct 20, 2024
85ac554
set health report on publish
Oct 21, 2024
50c92d9
run CI again
Oct 21, 2024
6f96288
re-add additional test
Oct 21, 2024
4a3f64d
re-enable additional test
Oct 21, 2024
e937b92
re-enable last test
Oct 21, 2024
55b5629
remove the redundant parentheses
Oct 21, 2024
b352068
remove duplicate logic
Oct 21, 2024
6056a29
clean up
Oct 21, 2024
3199d76
remove unnecessary newlines
Oct 21, 2024
b3ecfeb
remove HealthAnnotationsInitialized
Oct 21, 2024
c38a5ff
add comment
Oct 22, 2024
2bb8e07
re-add health status computation in dashboard
Oct 22, 2024
feb1dc7
forgot newline
Oct 22, 2024
5145c38
fix test
Oct 22, 2024
b88279a
Add a healthy check resources in to health check sandbox to showcase …
Oct 22, 2024
5b91f84
Update resource health reports if any have changed after the health c…
Oct 22, 2024
69abad0
move logic into above condition
Oct 22, 2024
f7127eb
extract to local static method
Oct 22, 2024
0129991
use string comparers
Oct 22, 2024
8a02eb5
return true if health check name is not found in existing health reports
Oct 22, 2024
2799a2a
Merge branch 'main' into dev/adamint/remove-aggregated-healthstatus
Oct 22, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 10 additions & 15 deletions playground/HealthChecks/HealthChecksSandbox.AppHost/Program.cs
Original file line number Diff line number Diff line change
Expand Up @@ -2,15 +2,16 @@
// The .NET Foundation licenses this file to you under the MIT license.

using Aspire.Hosting.Lifecycle;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Diagnostics.HealthChecks;

var builder = DistributedApplication.CreateBuilder(args);

builder.Services.TryAddLifecycleHook<TestResourceLifecycleHook>();

AddTestResource("healthy", HealthStatus.Healthy, "I'm fine, thanks for asking.");
AddTestResource("unhealthy", HealthStatus.Unhealthy, "I can't do that, Dave.", exception: GetException("Feeling unhealthy."));
AddTestResource("degraded", HealthStatus.Degraded, "Had better days.", exception: GetException("Feeling degraded."));
AddTestResource("unhealthy", HealthStatus.Unhealthy, "I can't do that, Dave.", exceptionMessage: "Feeling unhealthy.");
AddTestResource("degraded", HealthStatus.Degraded, "Had better days.", exceptionMessage: "Feeling degraded.");

#if !SKIP_DASHBOARD_REFERENCE
// This project is only added in playground projects to support development/debugging
Expand All @@ -24,28 +25,22 @@

builder.Build().Run();

static string GetException(string message)
IResourceBuilder<TestResource> AddTestResource(string name, HealthStatus status, string? description = null, string? exceptionMessage = null)
{
try
{
throw new InvalidOperationException(message);
}
catch (InvalidOperationException ex)
{
return ex.ToString();
}
}
builder.Services.AddHealthChecks()
.AddCheck(
$"{name}_check",
() => new HealthCheckResult(status, description, new InvalidOperationException(exceptionMessage))
);

IResourceBuilder<TestResource> AddTestResource(string name, HealthStatus status, string? description = null, string? exception = null)
{
return builder
.AddResource(new TestResource(name))
.WithHealthCheck($"{name}_check")
.WithInitialState(new()
{
ResourceType = "Test Resource",
State = "Starting",
Properties = [],
HealthReports = [new HealthReportSnapshot($"{name}_check", status, description, exception)]
JamesNK marked this conversation as resolved.
Show resolved Hide resolved
})
.ExcludeFromManifest();
}
Expand Down
6 changes: 0 additions & 6 deletions src/Aspire.Dashboard/Model/ResourceStateViewModel.cs
Original file line number Diff line number Diff line change
Expand Up @@ -62,12 +62,6 @@ internal static ResourceStateViewModel GetStateViewModel(ResourceViewModel resou
icon = new Icons.Filled.Size16.Circle();
color = Color.Info;
}
else if (resource.HealthStatus is null)
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This condition is redundant

{
// If we are waiting for a health check, show a progress bar and consider the resource unhealthy
icon = new Icons.Filled.Size16.CheckmarkCircleWarning();
color = Color.Warning;
}
else if (resource.HealthStatus is not HealthStatus.Healthy)
{
icon = new Icons.Filled.Size16.CheckmarkCircleWarning();
Expand Down
2 changes: 1 addition & 1 deletion src/Aspire.Dashboard/Resources/Resources.Designer.cs

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

4 changes: 2 additions & 2 deletions src/Aspire.Dashboard/Resources/Resources.resx
Original file line number Diff line number Diff line change
Expand Up @@ -233,7 +233,7 @@
<value>View console logs</value>
</data>
<data name="ResourcesDetailsHealthStateProperty" xml:space="preserve">
<value>Health State</value>
<value>Health state</value>
</data>
<data name="ResourcesDetailsStopTimeProperty" xml:space="preserve">
<value>Stop time</value>
Expand Down Expand Up @@ -262,4 +262,4 @@
<data name="ResourceActionTelemetryTooltip" xml:space="preserve">
<value>No telemetry found for this resource.</value>
</data>
</root>
</root>
4 changes: 2 additions & 2 deletions src/Aspire.Dashboard/Resources/xlf/Resources.cs.xlf

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

4 changes: 2 additions & 2 deletions src/Aspire.Dashboard/Resources/xlf/Resources.de.xlf

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

4 changes: 2 additions & 2 deletions src/Aspire.Dashboard/Resources/xlf/Resources.es.xlf

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

4 changes: 2 additions & 2 deletions src/Aspire.Dashboard/Resources/xlf/Resources.fr.xlf

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

4 changes: 2 additions & 2 deletions src/Aspire.Dashboard/Resources/xlf/Resources.it.xlf

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

4 changes: 2 additions & 2 deletions src/Aspire.Dashboard/Resources/xlf/Resources.ja.xlf

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

4 changes: 2 additions & 2 deletions src/Aspire.Dashboard/Resources/xlf/Resources.ko.xlf

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

4 changes: 2 additions & 2 deletions src/Aspire.Dashboard/Resources/xlf/Resources.pl.xlf

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

4 changes: 2 additions & 2 deletions src/Aspire.Dashboard/Resources/xlf/Resources.pt-BR.xlf

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

4 changes: 2 additions & 2 deletions src/Aspire.Dashboard/Resources/xlf/Resources.ru.xlf

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

4 changes: 2 additions & 2 deletions src/Aspire.Dashboard/Resources/xlf/Resources.tr.xlf

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

4 changes: 2 additions & 2 deletions src/Aspire.Dashboard/Resources/xlf/Resources.zh-Hans.xlf

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

4 changes: 2 additions & 2 deletions src/Aspire.Dashboard/Resources/xlf/Resources.zh-Hant.xlf

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

44 changes: 39 additions & 5 deletions src/Aspire.Hosting/ApplicationModel/CustomResourceSnapshot.cs
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,9 @@ namespace Aspire.Hosting.ApplicationModel;
/// </summary>
public sealed record CustomResourceSnapshot
{
private readonly ImmutableArray<HealthReportSnapshot> _healthReports = [];
private readonly ResourceStateSnapshot? _state;

/// <summary>
/// The type of the resource.
/// </summary>
Expand Down Expand Up @@ -40,7 +43,15 @@ public sealed record CustomResourceSnapshot
/// <summary>
/// Represents the state of the resource.
/// </summary>
public ResourceStateSnapshot? State { get; init; }
public ResourceStateSnapshot? State
{
get => _state;
init
{
_state = value;
HealthStatus = ComputeHealthStatus(_healthReports, value?.Text);
}
}

/// <summary>
/// The exit code of the resource.
Expand All @@ -52,11 +63,10 @@ public sealed record CustomResourceSnapshot
/// </summary>
/// <remarks>
/// <para>
/// This value is derived from <see cref="HealthReports"/>. If a resource is known to have a health check
/// and no reports exist, or if a resource does not have a health check, then this value is <see langword="null"/>.
/// This value is derived from <see cref="HealthReports"/>.
/// </para>
/// </remarks>
public HealthStatus? HealthStatus { get; init; }
public HealthStatus? HealthStatus { get; private set; }

/// <summary>
/// The health reports for this resource.
Expand All @@ -65,7 +75,15 @@ public sealed record CustomResourceSnapshot
/// May be zero or more. If there are no health reports, the resource is considered healthy
/// so long as no heath checks are registered for the resource.
/// </remarks>
public ImmutableArray<HealthReportSnapshot> HealthReports { get; init; } = [];
public ImmutableArray<HealthReportSnapshot> HealthReports
{
get => _healthReports;
internal init
{
_healthReports = value;
HealthStatus = ComputeHealthStatus(value, State?.Text);
}
}

/// <summary>
/// The environment variables that should show up in the dashboard for this resource.
Expand All @@ -86,6 +104,22 @@ public sealed record CustomResourceSnapshot
/// The commands available in the dashboard for this resource.
/// </summary>
public ImmutableArray<ResourceCommandSnapshot> Commands { get; init; } = [];

internal static HealthStatus? ComputeHealthStatus(ImmutableArray<HealthReportSnapshot> healthReports, string? state)
{
if (state != KnownResourceStates.Running)
{
return null;
}

return healthReports.Length == 0
// If there are no health reports and the resource is running, assume it's healthy.
? Microsoft.Extensions.Diagnostics.HealthChecks.HealthStatus.Healthy
// If there are health reports, the health status is the minimum of the health status of the reports.
// If any of the reports is null (first health check has not returned), the health status is unhealthy.
: healthReports.MinBy(r => r.Status)?.Status
?? Microsoft.Extensions.Diagnostics.HealthChecks.HealthStatus.Unhealthy;
}
}

/// <summary>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -352,7 +352,6 @@ public Task PublishUpdateAsync(IResource resource, string resourceId, Func<Custo
var newState = stateFactory(previousState);

newState = UpdateCommands(resource, newState);
newState = UpdateHealthStatus(resource, newState);

notificationState.LastSnapshot = newState;

Expand All @@ -377,11 +376,10 @@ public Task PublishUpdateAsync(IResource resource, string resourceId, Func<Custo
{
_logger.LogTrace("Resource {Resource}/{ResourceId} update published: " +
"ResourceType = {ResourceType}, CreationTimeStamp = {CreationTimeStamp:s}, State = {{ Text = {StateText}, Style = {StateStyle} }}, " +
"HealthStatus = {HealthStatus} " +
"ExitCode = {ExitCode}, EnvironmentVariables = {{ {EnvironmentVariables} }}, Urls = {{ {Urls} }}, " +
"Properties = {{ {Properties} }}",
resource.Name, resourceId,
newState.ResourceType, newState.CreationTimeStamp, newState.State?.Text, newState.State?.Style, newState.HealthStatus,
newState.ResourceType, newState.CreationTimeStamp, newState.State?.Text, newState.State?.Style,
newState.ExitCode, string.Join(", ", newState.EnvironmentVariables.Select(e => $"{e.Name} = {e.Value}")), string.Join(", ", newState.Urls.Select(u => $"{u.Name} = {u.Url}")),
string.Join(", ", newState.Properties.Select(p => $"{p.Name} = {p.Value}")));
}
Expand All @@ -390,20 +388,6 @@ public Task PublishUpdateAsync(IResource resource, string resourceId, Func<Custo
return Task.CompletedTask;
}

/// <summary>
/// Update resource snapshot health status if the resource is running with no health checks.
/// </summary>
private static CustomResourceSnapshot UpdateHealthStatus(IResource resource, CustomResourceSnapshot previousState)
{
// A resource is also healthy if it has no health check annotations and is in the running state.
if (previousState.HealthStatus is not HealthStatus.Healthy && !resource.TryGetAnnotationsIncludingAncestorsOfType<HealthCheckAnnotation>(out _) && previousState.State?.Text == KnownResourceStates.Running)
{
return previousState with { HealthStatus = HealthStatus.Healthy };
}

return previousState;
}

/// <summary>
/// Use command annotations to update resource snapshot.
/// </summary>
Expand Down
25 changes: 1 addition & 24 deletions src/Aspire.Hosting/Dashboard/DashboardServiceData.cs
Original file line number Diff line number Diff line change
@@ -1,7 +1,6 @@
// Licensed to the .NET Foundation under one or more agreements.
// The .NET Foundation licenses this file to you under the MIT license.

using System.Collections.Immutable;
using System.Runtime.CompilerServices;
using Aspire.Hosting.ApplicationModel;
using Microsoft.Extensions.Logging;
Expand Down Expand Up @@ -50,31 +49,9 @@ static GenericResourceSnapshot CreateResourceSnapshot(IResource resource, string
State = snapshot.State?.Text,
StateStyle = snapshot.State?.Style,
HealthStatus = snapshot.HealthStatus,
HealthReports = GetOrCreateHealthReports(),
HealthReports = snapshot.HealthReports,
Commands = snapshot.Commands
};

ImmutableArray<HealthReportSnapshot> GetOrCreateHealthReports()
adamint marked this conversation as resolved.
Show resolved Hide resolved
{
if (!resource.TryGetAnnotationsIncludingAncestorsOfType<HealthCheckAnnotation>(out var annotations))
{
return snapshot.HealthReports;
}

var enumeratedAnnotations = annotations.ToList();
if (snapshot.HealthReports.Length == enumeratedAnnotations.Count)
{
return snapshot.HealthReports;
}

var reportsByKey = snapshot.HealthReports.ToDictionary(report => report.Name);
foreach (var healthCheckAnnotation in enumeratedAnnotations.Where(annotation => !reportsByKey.ContainsKey(annotation.Key)))
{
reportsByKey.Add(healthCheckAnnotation.Key, new HealthReportSnapshot(healthCheckAnnotation.Key, null, null, null));
}

return [..reportsByKey.Values];
}
}

var timestamp = DateTime.UtcNow;
Expand Down
Loading
Loading