Support relative URLs

[sholladay]

The new URL() constructor currently requires at least one of its arguments to be an absolute URL.

new URL('./page.html', 'https://site.com/help/');  // OK
new URL('./page.html', '/help/');  // Uncaught TypeError: URL constructor: /public_html/ is not a valid URL.

That requirement is painful because determining which absolute URL to use as a base can be difficult or impossible in many circumstances. In a regular browser context, document.baseURI should be used. In Web Workers, self.location should be used. In Deno, window.location should be used but only if the --location command line option was used. In Node, there is no absolute URL to use. Trying to write isomorphic code that satisfies this requirement is quite error prone.

Additionally, in many cases it would be useful to parse and resolve relative URLs against each other without knowing an absolute base URL ahead of time.

// Desired output - these currently do not work
new URL('/to', '/from').toString();  // '/to'
new URL('/to', '//from.com/').toString();  // '//from.com/to'

The lack of support for use cases involving only relative URLs is causing me to remove WHATWG URL from Ky, a popular HTTP request library, in favor of our own string replacement. See: sindresorhus/ky#271

Desired API and whether to update the existing new URL() API or create a new API?

From my perspective, updating the new URL() constructor so it can handle a relative URL in the baseUrl argument would be ideal, i.e. remove the requirement for an absolute base in favor of simply parsing any missing URL parts as empty strings (as is currently done when a URL lacks a query, for example). But I understand that changing new URL() at this point may be difficult and it may be more practical to instead create a new API; perhaps new PartialURL() or split out the validation, parsing, and resolution algorithms into individual methods.

For my purposes, I need to at least be able to parse and serialize a relative URL, without having to provide an absolute base URL. A method that resolves two relative URLs against each other and returns the resulting relative URL would also be useful, e.g. URL.resolve('./from/index.html', './to') -> ./from/to.

[annevk]

Well, its purpose is to create a URL and those are by definition not relative. I could see wanting something specialized for path/query/fragment manipulation though. Are there any popular libraries that handle that we could draw inspiration from?

[sholladay]

Where is it defined that a URL must contain a scheme and a host in order to be a valid URL?

Even if such a definition exists, new URL() is the first API in the web ecosystem that I have encountered that has this limitation, making it quite surprising.

Beyond that, the WHATWG URL spec itself defines relative URLs...

https://url.spec.whatwg.org/#relative-url-string

As for existing implementations, see Node's url.parse() and url.resolve(), among others. I've used these extensively to manipulate URLs where the scheme and/or host is not known ahead of time and will be determined later by the end-user or browser, depending on where the URL is ultimately used.

[annevk]

It defines them as input (though only in the context of a base URL, which at least browsers always use), it doesn't define them as data structures. The data structure is defined at https://url.spec.whatwg.org/#url-representation (though it's fair to say that does make it seem like more is optional than in reality is optional; something to improve).

[sholladay]

I get that browsers need an absolute base URL to actually perform a request. And thus it makes sense for the URL specification to define what an absolute base URL is and discuss resolving relative URLs in the context of an absolute base URL, etc.

What doesn't make sense to me is why new URL() imposes this limitation. I cannot think of anything else on the web platform that does this. Even HTML's <base> tag supports relative URLs, despite the fact that it is specifically meant for defining the base URL.

I can see some value in an API that tests whether a URL is absolute. So perhaps part of the problem here is that new URL() actually does a lot of things: parsing, resolving, and validating. These could be broken down into separate methods. I don't think that is strictly necessary, though it would be one way to solve this.

[annevk]

Browsers only have a single URL parser that works as new URL() does (and as defined at https://url.spec.whatwg.org/#url-parsing). E.g., when parsing <base href> the location of the document is used. And in fact, the entirety of the web platform does this as it all builds upon this standard and its primitives.

[sholladay]

Browsers only have a single URL parser that works as new URL() does

Sure, as I said, it's completely reasonable that a browser needs to resolve to an absolute URL. But I'm not building a browser and I have a suspicion that most new URL() users aren't, either. I'm building software for the web platform that is environment agnostic and needs the same functionality as new URL() even if the scheme or host is not yet known. Use cases and relevant code linked to above.

[mgiuca]

To try and clarify this issue: it seems that you're not asking for a definitional change but an actual behavioural change to the Web-facing URL API.

Specifically, the changes you seem to be asking for are:

1. If the base argument is not supplied, it defaults to document.location (the current page's URL), rather than the current behaviour which requires the url argument to be absolute if base is omitted.
2. If the base argument is not absolute, it is first resolved against document.location (the current page's URL), rather than the current behaviour which unconditionally requires the base argument to be absolute.

So for example, if you executed these on https://github.com/whatwg/url/issues/531, all of the following are currently errors, and they would change to work as follows:

// Proposed API.
> new URL('to');
"https://github.com/whatwg/url/issues/to"

> new URL('to', '/from/');
"https://github.com/from/to"

> new URL('to', '//from.com/');
"https://from.com/to"

Technically, this is all feasible, but I don't think it's necessary or desirable. It's rather trivial to write code using the current API that behaves like this if you want it to:

// Current API.
> new URL('to', document.location);
"https://github.com/whatwg/url/issues/to"

> new URL('to', new URL('/from/', document.location));
"https://github.com/from/to"

> new URL('to', new URL('//from.com/', document.location));
"https://from.com/to"

I personally prefer not to change this. The current API forces you to be explicit about incorporating the current document's location, so it's clear to anyone reading the code that the current page's URL might leak into the result. When you don't use document.location as a base, it's a pure mathematical function of the inputs, and will produce the same output on any web page. That's a good property which I don't think we should break.

[sholladay]

No. I want to be able to parse and resolve relative URLs in an environment-agnostic way, for example on the server. It's completely unacceptable to rely on the DOM. The point of this issue is new functionality, which would behave exactly like new URL() does now, except it would support relative URLs in both arguments and it would return the resolved and parsed relative URL. That's it. I'm not asking for magical implicit resolution to an absolute URL. Just allow baseUrl to be relative and if it is relative, then return a relative URL.

I don't care if this is a change to the constructor or exposed as some new method.

[mgiuca]

Ohh, I see what you want now. (Tip: When filing a bug asking for a change to API behaviour, please give sample input and output so it's clear what you want.)

So am I right in thinking that this is what you want for my three examples:

// Proposed API.
> new URL('to');
"to"

> new URL('to', '/from/');
"/from/to"

> new URL('to', '//from.com/');
"//from.com/to"

(Noting that I'm using strings to represent the output above, but it would actually be a URL object.)

OK that makes sense. It does mean changing the URL object to allow representation of all kinds of relative URLs (scheme-relative, host-relative, path-relative, query-relative and fragment-relative). Though maybe that's helpful in explaining in general all of those different kinds of relative, which currently are not captured in the spec other than as details of the parser algorithm.

[sholladay]

To be fair, I referenced Node's url.resolve() as an example of an existing implementation that produces the expected output (approximately). But point taken. Yes, you are correct about the desired output.

This would be a massive help to a lot of libraries and tools, especially those that aim to be isomorphic.

[masinter]

For multipart/related, we invented a scheme "thismessage:". You could use "thismessage::/" as the base if you didn't have one, and remove it when if was there when done. https://www.iana.org/assignments/uri-schemes/uri-schemes.xhtml#thismessage

[sholladay]

Interesting. I did actually consider something exactly like that using invalid: as a scheme, but it's a hack and we'd like to avoid it. In Ky, we were able to use a regex string replacement for the query part of the URL, which also isn't great, but that was sufficient for the one place we still used new URL() - we removed all other usage of new URL() due to the aforementioned problems. There are other situations I've encountered, though, where something more complicated is needed. Parsing and resolving relative URLs is really something that should be built into the standard web APIs.

[brainkim]

Hi, I’m in a similar situation. I’m prototyping a bundler and I keep running into issues using the WHATWG URL class, specifically because it does not parse origin-relative URLs. The use-case is that I want to specify a common prefix for the public distribution of static files; for instance, the prefix can be the string "/static/", implying that the origin is the same origin as the server, but it can also be an absolute URL on a different origin ("https://mycdn.com/"). Some common operations I need include resolving relative and absolute URLs against this base, detecting if another URL is “outside” the base, and getting the relative path of a URL relative to the base, all of which could be done if an origin relative URL could be passed to the URL constructor, something like new URL("main.js", "/static/").

If anyone has any solutions, I’d love to hear about it. I’m loathe to abandon the URL class completely because of all the work it does in parsing URLs, but right now I have a Frankenstein system with URLs, the path/posix module, and regexes that I’d like to abstract.

[annevk]

@brainkim for that specific case it seems you could work around this by using a fake origin such as https://fakehost.invalid and removing it later on.

Also, if we did something here it would not be by changing new URL(). The output of that has to be "complete" and useful in a wide variety of contexts that expect a scheme and such.