Using containerized Consul for service discovery

[bart-vmware]

Is there an existing issue for this?

• I have searched the existing issues

Describe the bug

I'm trying to use HashiCorp Consul for DNS-based service discovery in an Aspire Starter App. In doing so, I identified the following limitations:

• DNS SRV queries are always sent to the DNS servers known on the local machine (originating from DHCP or resolve.conf), but there's no way to specify a specific DNS server. To use Consul, the query needs to be sent to the Consul DNS server running in a container.
• The hardcoded format of the SRV query needs to change; see https://developer.hashicorp.com/consul/docs/discover/service/static#rfc-2782-lookup

Expected Behavior

Being able to use Consul DNS for service discovery.

Steps To Reproduce

I'm using the following code to start the container from AppHost:

public static class ConsulDistributedApplicationBuilderExtensions
{
    public static IResourceBuilder<ConsulResource> AddConsulContainer(
        this IDistributedApplicationBuilder builder, [ResourceName] string name)
    {
        ArgumentNullException.ThrowIfNull(builder);
        ArgumentNullException.ThrowIfNull(name);

        var consul = new ConsulResource(name);

        return builder.AddResource(consul)
            .WithImage("hashicorp/consul", "latest")
            .WithImageRegistry("docker.io")
            .WithHttpEndpoint(8500, isProxied: false)
            .WithHttpHealthCheck("http://localhost:8500/")
            .WithUrl("http://localhost:8500/v1/agent/services", "Services JSON")
            .WithUrl("http://localhost:8500/v1/agent/checks", "Checks JSON")
            .WithEndpoint("dns-tcp", ea =>
            {
                ea.Protocol = ProtocolType.Tcp;
                ea.Port = 53;
                ea.TargetPort = 8600;
                ea.IsProxied = false;
            })
            .WithEndpoint("dns-udp", ea =>
            {
                // TODO: Why does dashboard show tcp?
                ea.Protocol = ProtocolType.Udp;
                ea.Port = 53;
                ea.TargetPort = 8600;
                ea.IsProxied = false;
            });
    }
}

public sealed class ConsulResource(string name)
    : ContainerResource(name);

From the app, I'm using the following:

builder.Services.AddDnsSrvServiceEndpointProvider(
    options => options.QuerySuffix = "service.consul");

To make this work, I patched the Aspire sources here, which is obviously not a permanent solution:

// PATCH-START: Change DNS query to work with Consul.
//var srvQuery = $"_{portName}._tcp.{query.ServiceName}.{_querySuffix}";  // example: _default._tcp.fortune-service.service.consul
var srvQuery = $"_{query.ServiceName}._tcp.{_querySuffix}";               // example: _fortune-service._tcp.service.consul
// PATCH-END: Change DNS query to work with Consul.

// PATCH-START: Use the Consul DNS server, instead of the local machine.
//public DnsResolver(TimeProvider timeProvider, ILogger<DnsResolver> logger) : this(timeProvider, logger, OperatingSystem.IsLinux() || OperatingSystem.IsMacOS() ? ResolvConf.GetOptions() : NetworkInfo.GetOptions())
public DnsResolver(TimeProvider timeProvider, ILogger<DnsResolver> logger) : this(timeProvider, logger, new ResolverOptions([new IPEndPoint(IPAddress.Parse("127.0.0.1"), 53)]))
// PATCH-END: Use the Consul DNS server, instead of the local machine.

Exceptions (if any)

No response

.NET Version info

.NET SDK:
 Version:           9.0.305
 Commit:            3fc74f3529
 Workload version:  9.0.300-manifests.6fcb754b
 MSBuild version:   17.14.21+8929ca9e3

Runtime Environment:
 OS Name:     Windows
 OS Version:  10.0.26100
 OS Platform: Windows
 RID:         win-x64
 Base Path:   C:\Program Files\dotnet\sdk\9.0.305\

.NET workloads installed:
There are no installed workloads to display.
Configured to use loose manifests when installing new manifests.

Host:
  Version:      9.0.9
  Architecture: x64
  Commit:       893c2ebbd4

.NET SDKs installed:
  8.0.414 [C:\Program Files\dotnet\sdk]
  9.0.110 [C:\Program Files\dotnet\sdk]
  9.0.205 [C:\Program Files\dotnet\sdk]
  9.0.305 [C:\Program Files\dotnet\sdk]

.NET runtimes installed:
  Microsoft.AspNetCore.App 3.1.32 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 6.0.36 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 8.0.20 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 9.0.6 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 9.0.9 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 3.1.32 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 6.0.36 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 8.0.20 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 9.0.6 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 9.0.9 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.WindowsDesktop.App 3.1.32 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 6.0.36 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 8.0.20 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 9.0.6 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 9.0.9 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]

Other architectures found:
  x86   [C:\Program Files (x86)\dotnet]
    registered at [HKLM\SOFTWARE\dotnet\Setup\InstalledVersions\x86\InstallLocation]

Environment variables:
  Not set

global.json file:
  Not found

Anything else?

I'm interested in opening a PR to enable Consul support, but would like some guidance on how to expose these capabilities. Perhaps make ResolverOptions public, and offer a template property for the DNS query somewhere?

[davidfowl]

I'm interested in opening a PR to enable Consul support, but would like some guidance on how to expose these capabilities

This is perfect for the https://github.com/communityToolkit/aspire

As for the service discovery fixes @radek

[bkoelman]

Did you mean to tag @rzikm?

[davidfowl]

Yep 😄

[rzikm]

As for the service discovery fixes @radek

As for the DNS server address, there is currently no API to specify DNS server explicitly (and I did not want to add it as part of #6104 as the PR was big enough without it). I am not sure what the process is for adding additional API surface in this repo.

Edit: I also notice there is this: #11777, so dotnet/extensions may be a better repo for this issue?

To make this work, I patched the Aspire sources here, which is obviously not a permanent solution:

I don't have experience with Consul, but I remember that the generated querries do not fall to the pit of success with aspir8 either (see #9913), so maybe there is more that could be improved over there.

[davidfowl]

@rzikm we can start here and move the issue if needed.

edit: Wait we moved it! Yes lets move to extensions dotnet/extensions#6868

[bart-vmware]

That's fine if this issue needs to move. Can you provide guidance so I can start working on a fix?

[rzikm]

Getting apps to use custom DNS server should be just matter of adding explicit list of IP addresses in the AddDnsSrvServiceEndpointProvider options. maybe take List and if empty, use the default behavior (autodetect).

I am not sure how to go about changing the SRV query hostname template. That part was originally written by @ReubenBond, is there some middle ground that would work with both Consul and still work with Kubernetes CoreDNS?

If not, then maybe we need Func<..., string> in options to allow custom formatting of the SRV query?

[bart-vmware]

Getting apps to use custom DNS server should be just matter of adding explicit list of IP addresses in the AddDnsSrvServiceEndpointProvider options. maybe take List and if empty, use the default behavior (autodetect).

Would it make sense for an app to use both Consul and k8s? If so, adding the Consul DNS server would disable the auto-detection required for k8s. Should we add public bool AddSystemDnsServers { get; set; } = true to DnsSrvServiceEndpointProviderOptions for flexibility?

I am not sure how to go about changing the SRV query hostname template. That part was originally written by @ReubenBond, is there some middle ground that would work with both Consul and still work with Kubernetes CoreDNS?

I'm not familiar with how this works in Kubernetes, but it looks like the currently used format is incompatible with Consul.

If not, then maybe we need Func<..., string> in options to allow custom formatting of the SRV query?

Well, an app that uses both Consul and k8s would require sending SRV queries in both formats. Would it make sense to have multiple DnsSrvServiceEndpointProviderFactory instances in the container, each bound to separate options?

[xXAvoraXx]

Is there any update for this issue? @bart-vmware @davidfowl

I want to use Microsoft's service discovery package with consul without having to do any additional development, so I'm waiting for that development.

[bart-vmware]

@xXAvoraXx I've built an internal prototype that uses Consul in an Aspire Starter App for service registration and discovery using Steeltoe.Discovery.Consul, which works locally with containers. It utilizes the Steeltoe library for both registration and discovery, which relies on Consul's HTTP API. What I'm exploring here is using the Aspire DNS-based provider for discovery instead, which I only got working by patching the Aspire sources as described above.

I've raised a potential concern above regarding the use of both Consul and other DNS servers in the same application. It would necessitate a different design, which hasn't been responded to yet. I don't mind interpreting the silence as a disregard and propose the public API changes here, followed by a PR after approval.

[xXAvoraXx]

We are not using Aspire. All our microservices successfully register themselves with Consul using Steeltoe.Discovery.Consul, via the HTTP API. For our gateway, we use YARP and want to resolve backend service addresses fully dynamically using Microsoft.Extensions.ServiceDiscovery.Dns (and the YARP extension), directly via Consul’s DNS interface. @bart-vmware

We are blocked by the following technical limitations:

• The Microsoft ServiceDiscovery DNS provider does not allow us to specify the DNS server address (e.g., Consul’s IP or the container DNS) programmatically or via configuration. It only uses DNS servers auto-detected from /etc/resolv.conf or network interfaces, which is typically 127.0.0.11 in container environments, making Consul DNS unreachable.
• The SRV query format is hardcoded in the provider (_{portName}.tcp.{serviceName}.{suffix}), but Consul’s default/static SRV records use a different pattern ({serviceName}._tcp.{suffix}). Because of this, dynamic port resolution fails and we have to fall back to A records with hardcoded ports.
• ResolverOptions and related settings are internal, so there is no way to override timeouts, attempts, SRV query string, or the DNS server list in production code.

As a result, it is technically impossible to use Microsoft’s ServiceDiscovery and DNS provider packages with Consul DNS in a fully dynamic, patchless, config-free way.
What we need: The ability to specify DNS server address(es) and SRV query format via public options on extension methods like AddDnsSrvServiceEndpointProvider.

[davidfowl]

What I'm exploring here is using the Aspire DNS-based provider for discovery instead, which I only got working by patching the Aspire sources as described above.

Open to accept changes here to make it work. Well the implementation has moved, but same idea. Though we shouldn't need to take any external dependencies.

[bart-vmware]

Requesting approval for the following public API additions:

namespace Microsoft.Extensions.ServiceDiscovery.Dns.Resolver
{
    public class ResolverOptions
    {
        public IList<IPEndPoint> Servers { get; set; } = new List<IPEndPoint>();
        public int Attempts { get; set; } = 2;
        public TimeSpan Timeout { get; set; } = TimeSpan.FromSeconds(3);
    }
}

namespace Microsoft.Extensions.ServiceDiscovery.Dns
{
    public class DnsSrvServiceEndpointProviderOptions
    {
        public Func<ServiceEndpointQuery, string>? GetQueryText { get; set; }
        public ResolverOptions ResolverOptions { get; set; } = new();
    }
}