Relative URLs in WHATWG URL API

[TimothyGu]

• Version: v9.x+
• Platform: all
• Subsystem: url

We are on the track to slowly deprecate the non-standard url.parse() (#12168 (comment)) in favor of the new WHATWG standard-based URL API. One use case that currently cannot be migrated over from url.parse() is the handling of relative URLs.

Background

url.parse() accepts incomplete, relative URLs by filling unavailable components of a URL with null.

> url.parse('#hash')
Url {
  protocol: null,
  slashes: null,
  auth: null,
  host: null,
  port: null,
  hostname: null,
  hash: '#hash',
  search: null,
  query: null,
  pathname: null,
  path: null,
  href: '#hash' }

On the other hand, the URL constructor guarantees that all URL objects are fully complete and valid URLs, which means that it throws an exception in case of relative URLs:

> new URL('#hash')
TypeError [ERR_INVALID_URL]: Invalid URL: #hash
    at Object.onParseError (internal/url.js:92:17)
    at parse (internal/url.js:101:11)
    at new URL (internal/url.js:184:5)
    at repl:1:1

WHATWG URL API does have the algorithms necessary to parse relative URLs, however, and that is activated if a base argument is provided:

> new URL('#hash', 'http://complete-url/')
URL {
  href: 'http://complete-url/#hash',
  origin: 'http://complete-url',
  protocol: 'http:',
  username: '',
  password: '',
  host: 'complete-url',
  hostname: 'complete-url',
  port: '',
  pathname: '/',
  search: '',
  searchParams: URLSearchParams {},
  hash: '#hash' }

It is not always the case that a base URL is available, though.

Possible solutions

Do nothing

What this entails is that the currently supported ability to parse relative URLs will die as url.parse() becomes deprecated.

Do not deprecate url.parse(); otherwise do nothing

This is the most obvious actual solution, but from tickets like #12168, I don't see this as a good idea.

Add a non-standard TolerantURL class

This could work if we trick the parser into believing we have a legitimate URL, except there are many conditionals in the URL parser algorithm that provide ad-hoc compatibility fixes with legacy implementations. We would have to make a set of opinionated assumptions about the nature of the URL, such as the URL's scheme.

In addition to parsing, the setters will have awkward semantics. Consider the following:

// Case 1
const relativeURL = new TolerantURL('#hash');
console.log(relativeURL.href);
  // Prints #hash

relativeURL.protocol = 'http:';
console.log(relativeURL.href);
  // Should this print http:#hash (what url.format() does)?
  // http://#hash?
  // Or make the setter a noop and therefore just #hash?

// Case 2
// Assuming we have decided to use http scheme semantics
// for TolerantURL if one is not supplied.
// The URL parser does not allow changing special-ness of
// scheme through the protocol setter.
const relativeURL = new TolerantURL('//username:password@host/');
relativeURL.protocol = 'abc';
console.log(relativeURL.href);
  // Should this print abc://username:password@host/?
  // Or //username:password@host/?

// Compare:
const absoluteURL = new URL('http://username:password@host/');
absoluteURL.protocol = 'abc';
console.log(absoluteURL.href);
  // Prints http://username:password@host/

Something else that's better than what I thought of above...

[jasnell]

I do not suspect that we will be able to actually deprecate url.parse() for quite some time. For the foreseeable future (at least through 10.x) the two implementations will coexist side-by-side.

[domenic]

I vote for "do nothing", at least until it's clear what the use cases for these base-less relative URLs are and how people use them.

If we find out people use them a lot, I think a better solution would be (preferably in user-land) a "RelativeURL" class, not "TolerantURL", which only has pathname/search/searchParams/hash/toString(). Someone would have to specify how this works, but maybe as a first-pass it could have an internal real-URL and parse against https://example.com/, then just re-expose the pathname/search/searchParams/hash.

[benjamingr]

@domenic it's worth pointing out that working with relative URLS is extremely common in Node - the most common use case I can think of is when an incoming HTTP request arrives - you get a relative URL under request.url which you typically url.parse.

I think it would be a shame to keep two URL APIs just for relative URLs. It would be really nice if URL supported relative URLs somehow - how set in store is the spec at this point regarding that?

[domenic]

It just doesn't make sense to have a single API for both relative and absolute URLs---the components and parsing rules are far too different depending on the base used for the rest of the API to make any sense. So I'm pretty sure the spec isn't going to change, just because there's no underlying model that makes sense.

The best you can do if you want to use one API is to make up a base URL.

[benjamingr]

@domenic I realize that this is a hard problem, and one I do not understand very well - but I think having a base URL in Node but not in the browser could cause a lot of incompatibility when people expect code using the same spec to run the same way on both platforms.

I think we can only change URL to add example.com or something similar as a default if browsers do. If that isn't practical - then Node should stop calling url.parse a legacy API and embrace using it for relative URLs.

[jasnell]

Which can be done, of course, using the information in an HTTP request (in the case of request.url) so I'm not overly concerned with that particular case.

The key challenge, of course, is that without a base, it's impossible to say for sure which rules to apply to the relative bits. We either must provide a base or we must provide an equivalent context in order to properly handle the URL. Otherwise the parsing will be best guess at best.

[domenic]

A compromise might be adding request.parsedURL() or similar (a function to show that it's expensive) which uses the request info + the request.url relative URL string to return a properly parsed URL instance with the base constructed from the request info.

[benjamingr]

@jasnell

Which can be done, of course, using the information in an HTTP request (in the case of request.url) so I'm not overly concerned with that particular case.

How? An HTTP server is not aware of its host name, may have several dns host names or none. In fact, I'd argue that the end server should not be concerned with what hostname it's using.

[jasnell]

It can make a best guess using the protocol and host header, both of which may be modified of course, but it provides enough context to provide a base URL when parsing the request URI.

[watson]

I'll not claim to know which, if any, of the 3 suggested solutions are best, but I just wanted to add a few things to the discussion:

I've seen incoming HTTP requests to Node servers that don't have a Host header in the wild a few times. If it's been striped by a proxy or if the client haven't provided it in the first place I don't know, but you can't rely on this header.

The 2nd issue is knowing the protocol. This can be inferred by looking at the request.socket.encrypted boolean, but this is not an exact science as the TLS might have been terminated in a load balancer.

Bottom line is that we can only safely get the path (via request.url). Getting the rest can be worked around, but not in a nice way.

[TimothyGu]

So what I'm seeing is that what people really want a partial URL parser for is to parse the origin form of the request target in an HTTP request, which is defined as:

origin-form    = absolute-path [ "?" query ]

We can introduce a URLAbsolutePath class to parse from path state for this specific issue. The semantics are fairly well-defined for this specific case, since we know the scheme is always special (HTTP or HTTPS). Beyond that (including relative, query-only, or fragment-only URLs), however, users are on their own.

[jasnell]

hmm... I'm hesitant to introduce a new class. This could be approximated in userland fairly easily using something like:

const url = new URL(`https://localhost${absolutePath}?${query}`);

Then look at the bits of url that you care about.

[Jessidhia]

@jasnell that doesn't look very usable on user input 😕

[jasnell]

A userland module can make it more usable. I'd rather avoid adding a convenience class that is not part of the standard