Add support to ternary operator

[daxian-dbw]

PR Summary

Add support to ternary operator <condition> ? <if-true> : <if-false>. It results in a TernaryExpressionAst.

PR Context

Ternary operator has lower precedence than binary operator, so you can write $a -eq $b ? "Hello World" : [int]::MaxValue.

Implicit line continuance for ? is supported, so you can write

$PSEdition -eq 'Core'
    ? "PowerShell Core"
    : "Windows PowerShell"

Today, 1?2:3 is parsed as a command name as 1?2:3 is considered as a generic token by the tokenizer.
We don't want to break that, but in certain situation, we know we are expecting an expression and a generic token like 1?2:3 is not useful. In those cases, we want to make the ternary operator chars ? and : to force ending a number token scan.
A new bool field ForceEndNumbeOnTernaryOpChars is added to Tokenizer and used when scanning number tokens. When the tokenizer is current in Expression mode, ForceEndNumbeOnTernaryOpChars is true, and the current char is ? or :, we force ending the number token.
With this, we are able to write $a -gt 2?[int]::MaxValue:3, ${true}?3:1, and -not 1?2:3.

Characters ? and : are valid chars for a variable name, so $varName?2:3 will be parsed as a variable. In order to write concise ternary expression with a variable condition, you can do this: ${varName}?2:3.

This PR is a draft, not finished yet

Send out the draft PR to start collecting feedback.
Also, exercise the CI builds to find regressions.
Pending work:

1. Finish examining all visitor usages, and update appropriately. Example, VariableAnalysis needs to be updated to account for the ternary operator.
2. Examine if UpdatePosition is correctly called at the right place.
3. Add many tests.

PR Checklist

• PR has a meaningful title
  • Use the present tense and imperative mood when describing your changes
• Summarized changes
• Make sure all .h, .cpp, .cs, .ps1 and .psm1 files have the correct copyright header
• This PR is ready to merge and is not Work in Progress.
  • If the PR is work in progress, please add the prefix WIP: or [ WIP ] to the beginning of the title (the WIP bot will keep its status check at Pending while the prefix is present) and remove the prefix when the PR is ready.
• Breaking changes
  • None
  • OR
  • Experimental feature(s) needed
    • Experimental feature name(s):
• User-facing changes
  • Not Applicable
  • OR
  • Documentation needed
    • Issue filed:
• Testing - New and feature
  • N/A or can only be tested interactively
  • OR
  • Make sure you've added a new test if existing tests do not effectively test the code changed

[lzybkr]

skipComment: false [](start = 72, length = 18)

Why disallow comments?

[daxian-dbw]

I thought it was not necessary to allow for comments. I changed my mind, as comments are allowed before :.
Will change this.

[KirkMunro]

I'd like comment support in this, but without blank lines, e.g.

$something -eq $someValue
    # Describe what this does
    ? Do-ThisThing
    # Describe what the alternative does
    : Do-ThatThing

[daxian-dbw]

Updated to allow comments.

[lzybkr]

Introducing new look-ahead could hurt parsing performance so be sure to measure the impact.

[daxian-dbw]

Will measure the impact and reply back.

[KirkMunro]

Implicit line continuance for ? is supported

Hooray for implicit line continuance! 🎉

A new bool field ForceEndNumbeOnTernaryOpChars is added to Tokenizer and used when scanning number tokens. When the tokenizer is current in Expression mode, ForceEndNumbeOnTernaryOpChars is true, and the current char is ? or :, we force ending the number token.

The description of this PR includes a mispelling. The bool field should be ForceEndNumberOnTernaryOpChars.

Also that flag sounds a good candidate for an Optional Feature. Even with that proposal just in RFC, it would be useful/helpful to consider how it could be used for this optional way to parse ternary operators, so that we work out any kinks in the proposed optional feature design.

[daxian-dbw]

The description of this PR includes a mispelling. The bool field should be ForceEndNumberOnTernaryOpChars.

@KirkMunro Thanks for pointing out this. @kvprasoon also caught it. It has been fixed.

[KirkMunro]

@daxian-dbw Does the fact that Where-Object has an alias of ? get in the way of implicit continuance using ? at the start of a line? I don't think it's likely, because you would have to be using -InputObject in that scenario, which means working with a collection in Where-Object which is unlikely; however, something obscure like this is possible:

$c = @(gps -name pwsh)
$b = $StopProcess -eq $true
? -InputObject $c Count -gt 0
| % {if ($b) {spps $c -whatif}}

That's messy, and hopefully nothing like that exists in scripts, but I wanted to call out the potential conflict with the ? alias so that the potential issue is known/documented.

[daxian-dbw]

@KirkMunro Good catch. The continuation of ? will break ? -InputObject $c Count -gt 0 in that case, so it will be a potential breaking change.
Personally, I'm not too worried about it. That's a very obscure use of Where-Object and I presume the possibility of actually breaking any scripts is very low.
But we do need the committee review to approve this.

[KirkMunro]

On the potential confusion because of the ? alias for Where-Object, I was thinking this weekend that we just need to change the way we think about ?. Instead of just looking at ? as an alias, we need to teach/look at it as a context-sensitive conditional check. What it checks and what happens as a result depends on the context in which it is used. This shouldn't be much more difficult to learn then an operator that can be used as either an unary operator or a binary operator.

[TobiasPSP]

I thought exactly the same this morning 😀 ? complements the existing ?-alias very well. Von meinem iPhone gesendet

…

Am 21.07.2019 um 14:19 schrieb Kirk Munro ***@***.***>: On the potential confusion because of the ? alias for Where-Object, I was thinking this weekend that we just need to change the way we think about ?. Instead of just looking at ? as an alias, we need to teach/look at it as a context-sensitive conditional check. What it checks and what happens as a result depends on the context in which it is used. This shouldn't be much more difficult to learn then an operator that can be used as either an unary operator or a binary operator. — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

[essentialexch]

"He's dead Jim" - Dr. McCoy.

The so-called "glide path to C#" took a death dive off a cliff in v3 and hasn't been seen since.

[rkeithhill]

"glide path to C#" took a death dive off a cliff in v3 and hasn't been seen since.

Really? How about the support added for classes and interfaces in v5?

[rjmholt]

but if we can make the parser handle ?: then certainly we can make it handle if then else without braces

If you take the braces out and keep else optional (which it isn't in Haskell, and in F#, OCaml and Rust is a type error when else is not included and the then clause has a non unit/void expression type), you face another fundamental grammatical ambiguity issue which I don't think anyone will enjoy explaining to people learning PowerShell for the first time.

The -then/-else operators, being operators, arguably don't require such an explanation since they'd probably just be left-associative (like && and || are in JavaScript/TypeScript).

[Jaykul]

@TobiasPSP said:

Let's be careful with adding arbitrary "exceptions" to the language "rules". We currently have two language elements that start with a hyphen: parameters and operators. -then/-else are neither one, and work very different. If we want these, we'd probably have to call them then/else rather than -then/-else.

But they are an operator, as it says up there in the topic, this thing we're talking about is the "ternary operator" (sometimes called the conditional operator), and -then -else would just be using different symbols for it, like we do for most of the other operators.

I honestly don't love the idea of reverse-justifying the alias by pretending it's similar to the operator -- and then using that as a justification for choose the more terse syntax. 😕

I'll admit I think the ternary will reduce readability for the target audience of PowerShell. Using the ?: syntax, it's hard to look up online or in the help, unless you already know what it is. If you know what it is, then it's wonderful -- until someone nests it a few times and you have to untangle it. Anyway, I think it's a feature that's more for code-golf than for readability.

I've had my say, so I'm going to let this lie. I don't feel strongly one way or another. I'm sure I'll use it in the console all the time if it's there -- at the same time I advise people not to use it in shared scripts and production code 🤣

[TobiasPSP]

But they are an operator, as it says up there in the topic, this thing we're talking about is the "ternary operator" (sometimes called the conditional operator), and -then -else would just be using different symbols for it, like we do for most of the other operators.

Yes and no. the ternary operator is a rare operator that takes three operands. So while "-then" is an operator, "-else" is not.

PowerShell has one of these operators already: -replace. It uses an array on the right side to provide more than one operand, similar to:

$a -eq 1 -then $true, $false

This is coincidentally how the "classic" ternary operator works in many languages, it commonly just uses its own separator token (":") to distinguish the two operands:

$a -eq 1 -then $true : $false
$a -eq 1 ? $true : $false

Whether you like "?" or "-then" better I wouldn't care too much. I'd just ask that we stick to the language rules and not use -else as a mere operand separator token while making it look like a concatenated binary operator:

$a -eq 1 -then $true -else $false

By syntactical standards, -else would be defined as binary operator, but behavioral, it would be a keyword because it is not acting at all on its two operands on either side but it is rather part of -then and sharing one operand with -then. Admittedly, this sounds pretty picky, theoretical and bureaucratic, but it harms the predictability of the language: how would the parser tokenize -else? Would it be tokenized as an operator? If not, and it gets a color different from -then, that's confusing. If it appears as an operator, and gets the same color as -then, that's confusing, too, when it suddenly is no operator anymore once you remove -then, and when it not acts on its operands but rather uses an operand from a "different" operator.

These lines would be inline with existing PowerShell language rules:

$a -eq 1 -then $true, $false
$a -eq 1 -then $true : $false (requires a new keyword ":")
$a -eq 1 ? $true : $false (requires a new keyword ":"

This would be a violation since -else is no operator:

$a -eq 1 -then $true -else $false

When you look at these, they are all really not that much different. I'd go with one of the upper three.

(technically, it would impose different challenges to use "," or ":". "," probably would require some sort of "command mode" or take a command AST as array elements, but that's something easily resolvable in the back-end).

[Taoquitok]

PowerShell has one of these operators already: -replace. It uses an array on the right side to provide more than one operand, similar to:

$a -eq 1 -then $true, $false

This is coincidentally how the "classic" ternary operator works in many languages, it commonly just uses its own separator token (":") to distinguish the two operands:

It's good to see someone else come to this example, I thought I was going mad being the only person to suggest following the same structure used by -replace.

Vexx32 has already added some comments on why this would be difficult in the related suggestion issue, but if time is being spent on adding in new functionality anyways, presumably spending it on a format that's already familiar to the powershell language would be worth while?

I believe the concept of logical check -operator 'true result','false result' fits in far more neatly with existing behaviour, while still meeting the needs of speed and ease to write compared to a full if( ){ } else{ } statement, and if we do follow the same structure as -replace, it would also work across multiple lines, allowing for consistent readability with existing functionality

Talking about readability, and ease of searching to find documentation, rather than -then, why not -ternary ? This would at least make it explicitly clear what's happening to those who know what a ternary is, while making it very easy for the wider powershell community to learn what the new functionality is, and what the common term is for this functionality.

I mentioned this in my earlier comment, quoted the relevant parts below:

Much like with the current behaviour of -replace / .replace(), which can be spread across lines, using this existing operator format would be consistent with existing behaviours, so for example the below works with replace

# Same line
'test' -Replace 'te','re'
# output: rest

# multi-line
'test' -Replace
    'te',
    're'
# output: rest

So why not follow the same pattern for ternary?

# Same line
$a -eq $b -ternary $a,$c

# multi-line
$a -eq $b -ternary 
    $a,
    $c

I must say though that having followed these discussions, I'm now more open to the idea of ? : from the technical and usage side, but I'm definitely still worried that it'll be exceptionally difficult for people who do not have experience with ternaries to find out what on earth this does.
Searching for powershell ? : in google, with any combination of "" and + comes up blank, while you would hope it would find either github issue, while searching for "powershell ternary" comes up with the related suggestion github issue.
Same with searching for "? :" in regards to C#.. it doesn't come up with much, and without knowing you need to search for "?:" you'll have issues trying to find information.

[vexx32]

Though I'm reiterating a bit... I do dislike the idea of a $condition -operator $trueAction, $falseAction syntax -- it means we now have a third usage for a comma. Far more confusing in my opinion, and complicates both parsing and using the syntax when one option or another may potentially be an array value. And we can't just make it a regular array, either, if we want short-circuiting behaviour -- we'd end up with side effects being executed from both branches as the array is evaluated, which is not great. 😕

I definitely see the argument for discoverability though. Maybe we need a midway approach, with something like $condition -choose $trueAction : $falseAction so there's at least a keyword to google.

But again, this kinda muddies the waters with keywords and it having specialised syntax. PS can alleviate some discoverability by simply having a help document labelled about_?: or something, really. It mightn't be a valid filename but it can still be a help topic name iirc.

Yes, a brand new syntax isn't the best for newbies to pick up, but having it look like something they might be familiar with when it simply... isn't... is (I think) a good deal worse, as instead of approaching it with a "what even is this" they'll approach it with a "oh, this is like an operator!" and then have all their previous assumptions about operators fall apart around them.

[joeyaiello]

Reiterating a brief conversation I had on this with @daxian-dbw and @rjmholt last week.

I'm in support of the traditional ?/: ternary operator, working out of two base assumptions (which I'm happy for everyone to jump in and invalidate):

1. PowerShell typically has a recommended, "PowerShell-y" way to do things, and then a bunch of terse shortcuts and syntactical allowances. We have aliases, and we tell people not to use them in scripts. We also tell people not to omit the Get- from invocations like ChildItem. (Just check out Invoke-Obfuscation for an extreme example of all the unreadable stuff we support.)
2. When C# developers (who like the ternary operator, I recognize there's a set who don't) use PowerShell, they're upset when the ternary operator isn't in PowerShell. For me, this is the big one: I want it to be as easy as possible for C# developers (especially those working on SDKs for .NET/C# applications) to dabble in PowerShell. Even if they're writing cmdlets in C#, I'd like to lower the bar for them to write their build/test/CI/CD scripts in PowerShell (hence why we worked with .NET to add pwsh to the .NET SDK containers).

This latter reason is also why I think the -then/-else (with or without -) buys us nothing: the first time a C# dev searches up "PowerShell ternary" and sees that, they're going to be off put. And then who is it for? PowerShell scripters in this thread that are already arguing if/else or more parens/brackets are fine?

If we end up in a situation where C# and other developers are writing so much PowerShell that our community is littered with what traditional PowerShell scripters might call an unreadable ternary operator, I think that's a good problem to have.

Not to mention we can warn or fail their modules on a PSSA run to AvoidTernaryOperator and give them a quick fix for traditional if/else.

The only arguments I've heard thus far against ternary altogether generally roll up to "I don't like it", "it's ugly", or "people won't know what it means". To the first two, I'd say "just don't use it", and for me the last one is addressed by PSSA and community self-enforcement (and if the community takes off with it, then folks will learn it).

[TobiasPSP]

The „target audience“ for this operator is the group of people who asked for it, apparently people with c# background. So we probably shouldn’t waste time by trying to make it attractive and digestible for anyone else and in the end come up with something that is so polished and artificial that it isn‘t working well for anyone anymore. Plus c# has always been Powershells alter ego so c# style helps with translating code from c# to PS, too.

[WithHolm]

I still don't like this idea. This really goes against how powershell has worked and should work. Hell. I dont even like -match and -replace, but that is for other reasons (-Match is messy and -replace is Regex, and not string replace). Im repeateing myself now: IF this was going into powershell i would much rather have it as a cmdlet than a operator. It represents "The powershell way" much better than a new operator. this would then include verb/operator'y alias that could represent this, and the invocation would happen in a scriptblock. This way you get the "Golf-IF" and powershell operates like normal. Also you get the benefit that c# devs learn the powershell syntax. Powershell is not C#, even tho they share common ground. There is a probable reason why the pipeline and the syntax is the way it is.

$this -eq $that|t{"Yes","No"}
or multiline:

$this -eq $that|t{
   "Yes",
   "No"
}

[essentialexch]

@rkeithhill - as an admin scripter, not a dev, I've never seen a reason to use interfaces. So I have no opinion on that.

However, the v5 class implementation is incomplete and the syntax is not C# compatible. It's as close to VBScript as it is to C#.

For all of you C# folks, I encourage you to try to explain this to a C# programmer:

(dir).FullName

As I wrote before - the C# glide path died with v3.

Note: I don't disparage the class implementation or imply negative things about the PowerShell PG whatsoever. But those of you suggesting that PowerShell syntax should adhere to C# syntax and behavior - I find it ridiculous. That ended with the release of v3.

[rismoney]

We have aliases, and we tell people not to use them in scripts

This nonsense rubric needs to end. People use them, have always used them and aliases are a core tenet since v1, and it needs to stop being pushed in social circles. Aliases are, have been, and will continued to be used in scripts. Assume that.

[rkeithhill]

I don't see how the PS class declaration syntax of:

class foo { }

is more similar to VBScript's syntax:

Class foo
End Class

versus the C# syntax of:

class foo { }

??

those of you suggesting that PowerShell syntax should adhere to C# syntax and behavior - I find it ridiculous

The above comparison is actually a good case-in-point. When classes were added to PowerShell, the team did not follow VB/VBScript syntax. Instead, they followed the C# syntax for class declaration and even interface inheritance e.g. class myArgCompleter : System.Management.Automation.IArgumentCompleter {}. That said, the syntax for classes internally is definitely more tuned for PowerShell's approach to specifying (field, parameter, return) types.

I don't think anybody is saying that PowerShell needs to match C# syntax one-for-one. It's just that if PowerShell were to add a language feature that has a direct analog in C#, then perhaps it should use that syntax - if it can without breaking other aspects of PS usage.

RE the "glide path to C#", I think that still exists and has evolved to be a two-way glide path. I suspect more than a few admin scripters have benefitted from various C# code samples on SO and other places. I also know from personal experience that SW devs (particularly C/C#) I work with have had a much easier time writing scripts and modifying my scripts. They do appreciate things like familiar control flow constructs (vs if/fi, case/esac).

I'd like to think that PowerShell "scripters" is a big tent that can accommodate many levels of proficiency. Folks at the lower levels don't have to use more advanced features like classes, try/catch/finally or a ternary operator. But I think it is short-sighted to deny some (and I can't believe I'm saying this about a ternary operator) "advanced" features to those that are further along.

[essentialexch]

You chose only the declaration. There is far more to a class than just the declaration.

I recently spent 2 days taking a C# function from C# to PowerShell. By request of a client. I certainly did not think it a good use of my time. (And I will blog on this topic.)

But this experience certainly re-enforces my perspective that the glide path is dead.

[rkeithhill]

I certainly did not think it a good use of my time

Sorry to hear that. I thought all billable time was a good use of time. 😉

[TobiasPSP]

I'd like to think that PowerShell "scripters" is a big tent that can accommodate many levels of proficiency.

PowerShell is a big tent, couldn't agree more. So if apparently a large group loves to get that construct, why not happily help give it to them?

The very thing that matters to me is that new things must not damage existing things or make life harder for existing users. Adding -else that looks like an operator but is just an extension to -then would damage a rule because beginners could no longer trust that a stand-alone keyword beginning with a hyphen is an operator.

With ?: in comparison, it would be trivial to add a rule to the script analyzer to auto-convert ?: to an if statement for those uncomfortable with it. So I can't see this addition to have a hurting impact on existing users.

And yes, we can build that operator on top of existing PS constructs like cmdlets or operators or arrays, and it's smart to evaluate these opportunities first before adding something new. After due inspection of the alternatives, I come to the conclusion that any effort to try and make the ternary operator more "powershelly" by implementing it via cmdlet or operator/array either kills the initial cause (loses its brevity, simplicity, familiarity), or has the potential of messing up the language rules by "bending" them too much.

I'd love to see ?: plus a new script analyzer rule that auto-converts it to if-else on demand.

[LarryWeiss]

We have aliases, and we tell people not to use them in scripts

This nonsense rubric needs to end. People use them, have always used them and aliases are a core tenet since v1, and it needs to stop being pushed in social circles. Aliases are, have been, and will continued to be used in scripts. Assume that.

I am not a total absolutist on this, and I used to not appreciate the benefit for minimizing the use of aliases in scripts, but now I want consistency enough to get on the bandwagon and vote to normalize scripts by eliminating all aliases. That way, when I'm searching a repository for something I can just use the one spelling and have a good chance on finding all occurrences of the thing. Having said that, I love to use aliases when entering commands at the command prompt. Aliases are a good thing to have on hand.

[rismoney]

The hypocrisy of folks decrying aliases in scripts due to ambiguity are the same advocates of ? : syntax??? I'm in favor of ? : not because of current lang alignment, but because it is largely a conventional thing at this point. The former because dir is the convention for directory listing, not get-childitem. This talk with -then -else operators is just verbosity against the efforts intention.

[HumanEquivalentUnit]

For me, this is the big one: I want it to be as easy as possible for C# developers (especially those working on SDKs for .NET/C# applications) to dabble in PowerShell. Even if they're writing cmdlets in C#, I'd like to lower the bar for them to write their build/test/CI/CD scripts in PowerShell

PowerShell could be a build scripting system for C# devs, but it wasn't originally made for that - is there a push to move it towards that? Would pushing PS towards C# devs raise its profile inside Microsoft?

Assume a future where this push is a great success and it's now "as easy as possible" for C# devs, what else could we admin-scripty people expect to see in this future? Has PowerShell turned completely into a superset of C#?

PowerShell is already much much closer to C# than anything else the devs could be using, isn't it? Batch files, Bash, build systems made of XML, other scripting languages - except possibly CAKE which is a C# DSL and PS will struggle to get closer than that. But they're still resisting dabbling in PowerShell because it's not enough like C# because of ?: ? I assume that's actual feedback instead of speculation? What other changes would lower this bar further?

When C# developers (who like the ternary operator, I recognize there's a set who don't) use PowerShell, they're upset when the ternary operator isn't in PowerShell.

It doesn't automatically follow that the fix should be "put a ternary operator in PowerShell". Different languages are different.

But if that is the fix .. I'm back to asking if there's a push to turn PowerShell completely into a superset of C#?

[SeeminglyScience]

@HumanEquivalentUnit

PowerShell could be a build scripting system for C# devs, but it wasn't originally made for that - is there a push to move it towards that?

Build scripts are just automation like everything else. PowerShell wasn't built for creating chat bots either, doesn't mean it isn't great at it.

Would pushing PS towards C# devs raise its profile inside Microsoft?

Getting more C# devs interested in PS is a net gain for everyone. It means more contributions to the core product and more difficult to create third party projects popping up.

Assume a future where this push is a great success and it's now "as easy as possible" for C# devs, what else could we admin-scripty people expect to see in this future? Has PowerShell turned completely into a superset of C#?

I'm not sure what you're implying. Does borrowing a feature from C# make it harder to add unrelated features? If the concern is that resources were put into this, I think the scope of the change is being overestimated. I don't want to speak for @daxian-dbw, but I would guess with pretty high confidence that this discussion has taken way more resources than actually writing the code.

PowerShell is already much much closer to C# than anything else the devs could be using, isn't it? Batch files, Bash, build systems made of XML, other scripting languages - except possibly CAKE which is a C# DSL and PS will struggle to get closer than that. But they're still resisting dabbling in PowerShell because it's not enough like C# because of ?: ? I assume that's actual feedback instead of speculation? What other changes would lower this bar further?

The way you describe it makes it sound like someone is getting into PS and then abandons everything when they see there's no ternary operator. Instead the reality is that it would probably just make a decent chunk of folks a bit more comfortable.

The decision to give up looking into a language is often (in my experience) cumulative. It's not often one thing that makes folks lose interest, but instead a sort of subconscious pros and cons list that adds up to a general feeling about the language.

When C# developers (who like the ternary operator, I recognize there's a set who don't) use PowerShell, they're upset when the ternary operator isn't in PowerShell.

It doesn't automatically follow that the fix should be "put a ternary operator in PowerShell". Different languages are different.

It does automatically follow that a discussion should be had about whether it has a place in PowerShell. Languages are different, but they're also incredibly similar.

Realistically there are plenty of things from other languages that you'll likely never see a PR for because it's objectively wrong for PowerShell. For instance, I'm sure some devs hate that PowerShell is dynamically typed, but you'll never see the PS team advocating for static typing.

But if that is the fix .. I'm back to asking if there's a push to turn PowerShell completely into a superset of C#?

I doubt it, but there's also not a push to keep features from C# out of PowerShell. It's not a particularly difficult feature to implement and it (in my opinion) fits well in the language. Just seems like an easy win to me.

[iSazonov]

I think we should stop the long cycle discussion (conslusion is clear) and go forward with the PR.

[iSazonov]

@daxian-dbw Maybe make sense to open new PR because here we have over 100 comment which is not related to implementation details.

[ThomasNieto]

Is there a reason why this new language feature was not introduced as a RFC?

[daxian-dbw]

Now that the conclusion has been drawn, I will continue with the prototype and submit an RFC based on the prototype.

[daxian-dbw]

Closing this draft PR. I will submit a ready-for-review PR shortly.