ADR: HTTPX Async-First Architecture with Thin Sync Wrappers

[cbcoutinho]

Summary

• Proposes async-first architecture for caldav v3.0 using HTTPX
• All core logic written as async, with thin sync wrappers using anyio.from_thread.run()
• Backward compatible sync API (no changes for existing users)
• New async API via caldav.aio module

This ADR supersedes the approach in #555, taking a simpler "thin wrapper" approach rather than code generation (unasync).

Key Design Decisions

1. HTTPX + AnyIO: Modern HTTP client with native async support, anyio for clean sync-from-async bridging
2. Single Source of Truth: Only async code is maintained; sync wrappers are minimal delegation
3. No Code Generation: Runtime wrapping is simpler than unasync-style generation
4. Backward Compatible: Sync API remains identical for existing users

Test plan

• Review ADR document for technical accuracy
• Discuss approach with maintainers
• If accepted, implement in phases as outlined

---

This PR was generated with the help of AI, and reviewed by a Human

[cbcoutinho]

Howdy @tobixen, I'm going to take another crack at this refactor since the recent changes upstream. To kick things off, I wrote an ADR describing the target architecture. Would you be able to take a look and provide any feedback?

[tobixen]

The design decisions look good, but I haven't had time to look into the code yet.

This will not make it into the caldav 2.x series, it will eventually be caldav 3.x if it looks good and passes all the testing. It does complicate things a bit as I do have some more changes I want to squeeze into 2.x this month before looking into 3..0.

[tobixen]

Does httpx support DNSSEC?

[cbcoutinho]

What does supporting DNSSEC mean? Would this be part of the caldav library itself?

httpx uses the DNS resolver of the host system, and you can override this to use a specific resolver by creating a subclass of httpcore.SyncHTTPTransport similar to the following:

import httpx
import httpcore
import dns.resolver
import socket

# --- Custom DNSSEC Transport ---

class DNSSECTransport(httpcore.SyncHTTPTransport):
    """
    A custom transport that uses dnspython for DNSSEC-validating resolution.
    """
    def __init__(self, transport, dns_servers=None):
        self.transport = transport
        self.resolver = dns.resolver.Resolver()
        
        # Configure resolver to use specified DNS servers
        if dns_servers:
            self.resolver.nameservers = dns_servers
        
        # NOTE: For true DNSSEC validation, the configured nameservers 
        # must be validating resolvers (like 1.1.1.1 or 8.8.8.8).

    def request(self, method, url, headers, stream, extensions):
        scheme, host_bytes, port, path = url
        host = host_bytes.decode("ascii")

        # 1. Intercept the host and perform DNS lookup using dnspython
        if scheme in (b'http', b'https') and host not in ('localhost', '127.0.0.1'):
            try:
                # Resolve A (IPv4) records. You might also want to resolve AAAA (IPv6).
                # This uses the configured nameservers and performs validation if they support it.
                answer = self.resolver.resolve(host, 'A') 
                ip_address = answer[0].to_text()
                
                # 2. Reconstruct the URL with the IP address instead of the hostname
                # The 'Host' header is automatically set to the original hostname by httpx.
                # Use socket.getservbyname to get the port if not explicitly given.
                if port is None:
                    port = socket.getservbyname(scheme.decode('ascii'))
                
                # The underlying transport is called with the IP address instead of the host
                url_with_ip = (scheme, ip_address.encode('ascii'), port, path)

                # 3. Call the underlying transport with the new URL
                return self.transport.request(
                    method, 
                    url_with_ip, 
                    headers, 
                    stream, 
                    extensions
                )
            except dns.exception.DNSException as e:
                # If DNSSEC validation fails, dnspython will likely raise 
                # a SERVFAIL or similar exception (if configured correctly).
                # We can choose to treat this as an unresolvable domain.
                raise httpx.ConnectError(f"DNSSEC resolution failed for {host}: {e}")
        
        # For non-HTTP schemes or non-standard hosts, fall back to the base transport
        return self.transport.request(method, url, headers, stream, extensions)

# --- Usage Example ---

# Use a well-known validating resolver, e.g., Cloudflare
VALIDATING_DNS = ["1.1.1.1"]

# 1. Create the default transport
default_transport = httpcore.SyncHTTPTransport()

# 2. Wrap it with the custom DNSSEC transport
dnssec_transport = DNSSECTransport(default_transport, dns_servers=VALIDATING_DNS)

# 3. Use the custom transport in the HTTPX Client
with httpx.Client(transport=dnssec_transport) as client:
    try:
        # Request a domain known to be DNSSEC signed
        response = client.get("https://www.dnssec-or-not.com/check/", timeout=10)
        print(f"Status for dnssec-or-not.com: {response.status_code}")
        # If DNSSEC validation fails in the custom transport, a ConnectError is raised
        
        # Test an unsigned or misconfigured domain (requires finding a suitable test case)
        # response_fail = client.get("https://unsigned-domain.com/") 
        
    except httpx.ConnectError as e:
        print(f"Connection failed: {e}")
    except Exception as e:
        print(f"An unexpected error occurred: {e}")

[tobixen]

What does supporting DNSSEC mean? Would this be part of the caldav library itself?

httpx uses the DNS resolver of the host system, and you can override this to use a specific resolver by creating a subclass of httpcore.SyncHTTPTransport similar to the following:

I fell into a rabbit hole the other day, I came to realize that the DNS-based service location lookup actually involves a security risk as it may be trivial to highjack the DNS (i.e. on a public hotspot). Validation of DNSSEC records may be the only way to mitigate this risk - but I'm shelving this project as for now, it was too complicated.

niquests claim to support DNSSEC validation, but in practice this seems to involve turning on "dns-over-http" and explicitly configuring it to use a doh-provider, I'm not sure if that is a good idea.

[tobixen]

So, version 2.2 has been released, and async support is the next on my list. At the other hand, I will most likely have other priorities than the CalDAV library during December.

[tobixen]

niquests does support async operations - have any considerations been done to see if it can be utilized?

The biggest problem with niquests seems to be community acceptance. It depends on modifications to the urllib library that was not accepted by the maintainers, so currently niquest depends on an ugly fork of the urllib library.

[tobixen]

I just had a look at the requests.py file yesterday. The file was introduced by someone else in a pull request as far as I can remember. I'm considering to move request-handling logic from davclient to requests.

[tobixen]

While we're building a completely new API, it's the perfect time to do major changes. The old API has grown a bit organically and I'm sometimes annoyed by inconsistencies, i.e. not very well thought through were to use properties vs where to use methods. I also heard people disliking the implicit conversions done when accessing event.data vs event.icalendar_instance. Deciding on all those details are of course outside the scope of this ADR, but should be made before releasing CalDAV 3.0.

[tobixen]

What benefits does anyio give, compared to the latest versions of asyncio?

Anyway, if httpx depends on anyio, then usage of httpx means we'll implicitly depend on anyio, so there is no extra cost involved in depending on anyio here.

[tobixen]

Is this pattern widely used in other projects? Perhaps it's easier to just use the GitHub tools? At the other hand, it's nice to have things like this recorded in the repository itself, it's nice to avoid depending too much on GItHub.

Next question, where/how is one supposed to discuss the ADRs? Are line-by-line comments through the GitHub review infrastructure a good way to handle it?

[tobixen]

Is docs/source the best place to store ADRs? This directory is primarily used for creating user documentation. Will the markdown files be picked up and published by the sphinx-generated documentation? Do we want it to be published with the documentation?

[tobixen]

Generally, the ADR looks good. Some "food for thoughts" left above.

So, the change to async apparently depends on another ADR that hasn't been made yet. When creating a brand new async API, it would be nice to do a complete review of the current API, look into things that obviously doesn't make sense, change things that needs to be changed, etc. I did open an issue on this in #92

[tobixen]

This will affect most of the code. As mentioned in #589 I'd like to replace the "black style" with ruff. I think that ruff should be implemented immediately after the async changeset.

[tobixen]

I'm aiming for releasing 3.0 with async by the end of January.

I should be able to spend a decent amount of time on the CalDAV library in January. In December and February, probably not so much.