Using UriFormat.SafeUnescaped replaces %20 with space in query string

[bkoelman]

Description

Calling Uri.GetComponents() with UriFormat.SafeUnescaped converts %20 characters in query string parameters to regular spaces, which results in a (presumably) invalid URL.

Reproduction Steps

The sample below demonstrates the problem. QueryString.Create() converts spaces to %20, but then they get converted back to regular spaces by Uri.GetComponents().

using Microsoft.AspNetCore.Http;

const string source = "http://localhost/path?filter=some+text";
Console.WriteLine($"source: {source}");

var parameters = new Dictionary<string, string?>
{
    ["filter"] = "other text"
};

string? queryString = QueryString.Create(parameters).Value;
Console.WriteLine($"queryString: {queryString}");

var builder = new UriBuilder(source)
{
    Query = queryString
};

var uri = builder.Uri;
var text = uri.GetComponents(UriComponents.AbsoluteUri, UriFormat.SafeUnescaped);
Console.WriteLine($"text: {text}");

Prints:

source: http://localhost/path?filter=some+text
queryString: ?filter=other%20text
text: http://localhost/path?filter=other text

Expected behavior

I would have expected text in the repro steps above to become http://localhost/path?filter=other+text or http://localhost/path?filter=other%20text.

Actual behavior

In the repro steps above, text becomes http://localhost/path?filter=other text (note the unescaped space in the query string parameter).

Regression?

No response

Known Workarounds

Use UriFormat.UriEscaped instead of UriFormat.SafeUnescaped, which always escapes everything. Or run a replace-all on the output text to convert " " back to "+" or "%20".

Configuration

dotnet --info

.NET SDK:
 Version:           8.0.202
 Commit:            25674bb2f4
 Workload version:  8.0.200-manifests.8cf8de6d

Runtime Environment:
 OS Name:     Windows
 OS Version:  10.0.22631
 OS Platform: Windows
 RID:         win-x64
 Base Path:   C:\Program Files\dotnet\sdk\8.0.202\

.NET workloads installed:
There are no installed workloads to display.

Host:
  Version:      8.0.3
  Architecture: x64
  Commit:       9f4b1f5d66

.NET SDKs installed:
  8.0.202 [C:\Program Files\dotnet\sdk]

.NET runtimes installed:
  Microsoft.AspNetCore.App 6.0.28 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 8.0.3 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 6.0.28 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 8.0.3 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.WindowsDesktop.App 6.0.28 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 8.0.3 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]

Other architectures found:
  x86   [C:\Program Files (x86)\dotnet]
    registered at [HKLM\SOFTWARE\dotnet\Setup\InstalledVersions\x86\InstallLocation]

Environment variables:
  Not set

global.json file:
  Not found

Other information

From https://www.rfc-editor.org/rfc/rfc2396.html#section-3.4, space is not among the set of characters that require escaping:

Within a query component, the characters ";", "/", "?", ":", "@", "&", "=", "+", ",", and "$" are reserved.

However, the existing implementation already escapes more than the spec requires.

This issue stems from the bug report at json-api-dotnet/JsonApiDotNetCore#1525.

[MihaZupan]

SafeUnescaped is described as

Characters that have a reserved meaning in the requested URI components remain escaped. All others are not escaped.

A space does not have a reserved meaning in a query string, therefore it's unescaped.

You are describing the expected behavior as the value being escaped, and listing UriFormat.UriEscaped as a known workaround. I'm not sure I understand the issue here - it sounds like what you want is an escaped value, and should therefore use UriEscaped?

[bkoelman]

My goal is to apply the minimal required escaping (for readability of the produced URLs), which is why I chose UriFormat.SafeUnescaped. For example, I prefer ?page[size]=1 over ?page%5Bsize%5D=1. According to https://www.sistrix.com/ask-sistrix/technical-seo/site-structure/do-i-have-to-convert-the-spaces-in-urls and https://sitebulb.com/hints/internal/url-contains-whitespace/, spaces are never allowed in URLs. This makes me believe the behavior of UriFormat.SafeUnescaped isn't right. Because why would there be a URL parsing/formatting API that produces invalid URLs?

[MihaZupan]

For the example given, [ and ] would not be unescaped as we treat delimiter characters as not safe to unescape for the purposes of UriFormat.SafeUnescaped.

You are not asking for a valid url, but a semi-unescaped representation of its value. If you ever need a valid url (e.g. something to put on a website / send over a wire), you should be using the escaped values (e.g. UriFormat.UriEscaped or values returned by properties such as PathAndQuery/AbsoluteUri).

For your particular scenario, there are other characters that would likely break your expectations - e.g. %20 isn't the only whitespace character out there that would make the unescaped value "invalid" (e.g. not recognized as part of the url by a browser).

I doubt there's an existing API that does exactly the escaping/unescaping you want. If you have a list of known values that you wish to unescape for display purposes (e.g. [ and ]), I would recommend you manually unescape just those.

[bkoelman]

You're right about [ and ], I probably confused it with something else.

Point taken. Anything that needs to roundtrip should use UriFormat.UriEscaped, so I'll adapt our usage accordingly. I suppose this issue can be closed then, though I wonder when "a semi-unescaped representation" would be appropriate to use. I interpreted "SafeUnescaped" as minimal required escaping for the URL to remain valid. However, what it really means is: "unsafe unescaping", producing a broken URL.

Just out of curiosity, why are brackets escaped at all? I would think they'd never conflict with anything anywhere in a URL. Also, they aren't listed in https://www.rfc-editor.org/rfc/rfc2396.html#section-3.4, which is what the documentation points to.

Edit: corrected link to the documentation.

[MihaZupan]

The brackets are mentioned by the newer 3986 RFC.
They can appear if the url authority is an IPv6 literal.

[bkoelman]

Ah yes, of course!

[antonfirsov]

Triage: seems to be resolved, closing. @bkoelman feel free to reopen if you still need help!