Any change VB will support Unicode escape sequences in Strings

[paul1956]

In CSharp you can do "\u0000" and get a null in a string or with other values get any Unicode Character. It might be another one of those breaking changes. I know in Interpolated Strings I can do $"{ChrW(0)}".

[reduckted]

Duplicate of #81?

[paul1956]

@reduckted #81 is for Interpolated Strings and Unicode Escapes are not needed, ChrW works very well (though it would be nice if the IDE could recognize ChrW() and display the correct character when hovering. I was looking for a solution for regular strings where I would need to split the strings and concatenate the ChrW and I only want 1 string.

 ChrW(&H201C) & "is a Unicode Open Quote"

As an FYI VB today doesn't display Unicode Quotes when embedded in a string and it is hit and miss when Unicode Characters get displayed. They are correctly displayed in Immediate window and sometimes in Source. I have posted this in User Feedback.

[AnthonyDGreen]

Just some background here, it would require a new kind of string. C# has two types of strings: "regular" strings that support Unicode escapes (and other \ escapes) require escaping slashes \\ and don't support embedded newlines, and "verbatim" strings @"Like this". Verbatim string literals in C# work exactly* like all VB strings work. They support embedded newlines, don't support \ escapes, don't require slashes to be escaped, escape quotes by doubling up on the quotes "". With the introduction of interpolated strings that's two more string-like syntaxes: $"" and @$"". So to support what C# does in its strings a new kind of string would need to be added. Not supporting escapes has benefits, which is why C# has verbatim strings, that you wouldn't want to lose by changing VB strings outright (not that that would happen, back-compat). Additionally, you'd probably add a new interpolated string syntax that could use the escapable strings.

Incidentally, while working on the JSON literal prototype I did have to actually implement \ escapes in JSON strings. Obviously, the rules inside JSON would be different than regular VB in the way escapes are a little different inside interpolated strings and XML literals but it does raise this question back up.

So, is it worth adding two new stringy syntaxes to the languages to save having to call ChrW? I don't know the answer to that question and would love to hear from others on it.

* VB string literals are almost exactly like C# verbatim string literals, except VB string literals support full-width double-quote characters, and smart quotes like what Microsoft Word uses for quotes so “This is a valid string in VB” but @“This is not a valid string in C#” because of the smart quotes. This section included for pedants

Look at #299
I think the best formula is:
Dim S = $"a{\u0304\u0308}bc{\u0327}"
and for ASCII:
Dim S = $"a{\030\008}bc{\027}"
It's a non breaking code and easy to understand and parse.

I wrote this function to allow me to use \u0000 unicode and \a000 ASCII in strings:

    Function ComposeString(s As String)
        Try
            ' Hex Unicode \u0000
            s = s.Replace("\U", "\u")
            Do
                Dim i = s.IndexOf("\u")
                If i = -1 Then Exit Do
                Dim u = s.Substring(i, 6)
                Dim n = Convert.ToInt16(u.Replace("\u", ""), 16)
                s = s.Replace(u, ChrW(n))
            Loop

            ' Decimal ASCII \a000
            Do
                Dim i = s.IndexOf("\a")
                If i = -1 Then Exit Do
                Dim a = s.Substring(i, 5)
                Dim n = CByte(a.Replace("\a", ""))
                s = s.Replace(a, Chr(n))
            Loop

        Catch ex As Exception
            Throw New Exception("bad format")
        End Try
        Return s
    End Function

It can be used like this:

        Dim x1 = ComposeString("a\u0304\u0308bc\u0327")
        Console.WriteLine(x1) ' ā̈bç
        Dim x2 = ComposeString("Hello \a034Adam\a034")
        Console.WriteLine(x2) ' Hello "Adam"

I also added the same function as an extension method to rhe String Class with the name Decode:

Imports System.Runtime.CompilerServices

Module StringExt
    <Extension()> Public Function Decode(s As String)
        Try
            ' Hex Unicode \u0000
            s = s.Replace("\U", "\u")
            Do
                Dim i = s.IndexOf("\u")
                If i = -1 Then Exit Do
                Dim u = s.Substring(i, 6)
                Dim n = Convert.ToInt16(u.Replace("\u", ""), 16)
                s = s.Replace(u, ChrW(n))
            Loop

            ' Decimal ASCII \a000
            Do
                Dim i = s.IndexOf("\a")
                If i = -1 Then Exit Do
                Dim a = s.Substring(i, 5)
                Dim n = CByte(a.Replace("\a", ""))
                s = s.Replace(a, Chr(n))
            Loop

        Catch ex As Exception
            Throw New Exception("bad format")
        End Try
        Return s
    End Function

End Module

This can be used like this:

        Dim x3 = "a\u0304\u0308bc\u0327".Decode()
        Console.WriteLine(x3) ' ā̈bç
        Dim x4 = "Hello \a034Adam\a034".Decode()
        Console.WriteLine(x4) ' Hello "Adam"

[paul1956]

@AnthonyDGreen I think adding a third type of string is VB would be good, separate from the two that are these and have it follow the C# escaping rules. There is good reason C# has 3 kinds of strings and all 3 are used. On a side note I have recently been having issues with Unicode double quotes and how they are handled, in C# Unicode open and close quotes are allowed between Ascii ", but in VB they are not unless you use 2. As I looked into this it seems that all 3 double quotes ", \ul201c and \ul201d require escaping I can't do "\ul201c" (where I type the Unicode character for \ul201c). Also VB don't display \ul201c and \il201d consistently sometimes they are displayed as " and sometimes correctly.

[gilfusion]

Since VB already has escaping quotes by doubling them """", would it be possible to put other escapes inside those embedded quotes? (Like "Here is a "u201c"quote"u201d"." or "Line1"n"Line2")

[rskar-git]

@paul1956 @gilfusion

VB source code is saved as a UTF-8 file by default; and if you try to introduce "fancy" characters into a plain-text style *.vb file, Visual Studio will warn you and offer to "...resave this file as Unicode...". Given this capability, why is an escape code scheme essential for printable characters?

For the non-printable characters (like escape, new-line, and horizontal-tab), Chr and ChrW are already treated like keywords, so you can make constant characters (e.g. Public Const new_line = ChrW(13) + ChrW(10)); and, of course, a few of these are already defined, such as vbCr and vbLf and vbCrLf. Mixing these in via interpolated strings generally works nicely.

Multi-line text can be done easily in a number of ways: (1) Multi-line in-code literal string; (2) As a text-file based resource (compiled right in! no need to distribute the file too!); and (3) Via an XML literal.

I think the text-file based resource is a gem that's way too often overlooked! And the ultimate in WYSIWYG! Go with encoding "Unicode - Codepage 1200". (See https://stackoverflow.com/questions/2662623/using-a-txt-file-resource-in-vb, https://stackoverflow.com/questions/8336355/what-exactly-is-unicode-codepage-1200).

[rskar-git]

@paul1956:

Fancy double-quotes should be quite tame-able via xml or resource or interpolation or Chr/ChrW; or "escaping" by doubling-them, as you know.

Side note #1, "\ul201c" isn't valid C# (unrecognized escape sequence, and also \ul201d) - did you mean "\u201c" (which is LEFT DOUBLE QUOTATION MARK, decimal 8220)?

Side note #2, when you said "VB don't display \ul201c and \il201d consistently", is this about when printing to a console window? Because that may be tied to the "Region and Language" settings of the workstation.

[paul1956]

@rskar-git
#1 Yes, \u201c.
#2 printing in immediate Window and it is very confusing. The Editor displays correctly in source.

I am trying to translate strings and don't want to convert a "" string to another type of string but that is what I am doing now and I keep running into more exceptions that I need to deal with.