Include System.Linq.Async from https://github.com/dotnet/reactive into corefx

[oliverjanik]

I am very happy we have the new IAsyncEnumerable and some c# sugar to go with it in .net core 3.

I think it would be even better if it had batteries included just like IEnumerable.

👍

[stephentoub]

Thanks. We considered this previously, and currently have no plans to ship this in the platform. It is readily available for anyone who wants to use it simply by adding a nuget package reference to it.
cc: @MadsTorgersen, @jaredpar

[MgSam]

@stephentoub What is the reasoning behind this? You guys put a ton of engineering into making IAsyncEnumerable work. Why ship it in a crippled state without basic LINQ support from the core framework?

[stephentoub]

Why ship it in a crippled state

How is it crippled? Add a one-line package reference to your .csproj, and you're off and running. Why is that insufficient?

[MgSam]

Any package which you guys release separately only a tiny fraction of the overall .NET community ever knows about. And many teams try to limit adding additional dependencies in as much as possible. Finally, if the history of RX is any guide these separate packages often have much lower quality standards and documentation than what is included in the BCL.

The trend recently seems like it's been to bundle more to avoid the problems from separate packages. I don't need a separate package to use Span. Why is this feature being treated differently?

[stephentoub]

I don't need a separate package to use Span. Why is this feature being treated differently?

Span is a fundamental exchange type that is now plumbed through all levels of the system, enabling the next wave of performance enhancements in and on the platform. Core primitives like int, double, and string all have dependencies on it. It's integrated with the runtime itself. It needs to be part of the platform for that to work. The same cannot be said of extension methods that operate on IAsyncEnumerable.

Any package which you guys release separately only a tiny fraction of the overall .NET community ever knows about. And many teams try to limit adding additional dependencies in as much as possible.

We collectively need to get past this for the ecosystem to be successful. It does not scale to include "in the box" all possible functionality developers may want.

[jkotas]

Improving health of .NET ecosystem is part of the .NET Foundation charter. Check out https://github.com/dotnet-foundation/projects .

[MgSam]

Span is a fundamental exchange type that is now plumbed through all levels of the system...

And IAsyncEnumerable was deemed important enough to build into the language itself; an extremely high bar for any feature. There were already excellent 3rd party libraries that provided equivalent functionality without new language features. Yet a small library that is helpful to work with IAsyncEnumerable effectively is not important enough to be in the framework?

We collectively need to get past this for the ecosystem to be successful. It does not scale to include "in the box" all possible functionality developers may want.

This is a noble statement, but I don't think a feature which is pretty essential to what you guys just designed and shipped qualifies as "all possible" functionality.

You didn't actually address any of the 3 issues I pointed out:

1. Most developers do not know about packages you ship separately. I've been closely following this community since it went open source and I myself am barely aware of the existence of a package of extension methods for IAsyncEnumerable. For a normal C# dev who doesn't follow the community at all, there is no chance they'll be aware of it.

2. Many of your biggest users are in locked-down enterprises or locked-down teams where what they can use is what's included in the frameworks, and that's it. This ecosystem isn't Python or JavaScript and never will be Python or JavaScript. And this point of view is actually pretty rational- when you allow individual developers to add whatever packages they feel like it quickly becomes a maintenance nightmare because no one knows how anyone else's code actually works.

3. Rx .NET is a low-quality library not up to the standard of anything in the BCL and this makes it difficult to use. It has every API and the kitchen sink thrown into it with little curation. It has horrible documentation- that which does exist is largely out-of-date 3rd party docs from the many prior versions (with many breaking changes in-between) strewn about the web.

Finally, with you guys shipping the ability to trim unused assemblies from an app that bundles the framework, does saving every byte from going into the BCL really matter? I can see not adding shader processing packages into the BCL; but core methods to use language features with collection types? That seems pretty extreme to leave out.

[stephentoub]

And IAsyncEnumerable was deemed important enough to build into the language itself; an extremely high bar for any feature.

For consuming (foreach) and producing (iterators) IAsyncEnumerable... and the corresponding interfaces and key helpers that enable that were added to the framework. The hundreds upon hundreds of extension methods that make up an async LINQ implementation were not: the existing async LINQ library met all the key needs for it, and we did not need to design/implement/test/document/ship/service/maintain/evolve a competing implementation that would actually detract from rather than enhance the ecosystem. We instead chose to work with the maintainers of that library to ensure it was properly designed and interoperable, a library we could and do recommend as the solution developers employ if they need an async LINQ implementation.

[stephentoub]

cc: @clairernovotny for both .NET Foundation and async LINQ

[clairernovotny]

cc @bartdesmet as one of the main Rx authors.

I'm sorry to hear that you feel that Rx's quality is low. What is it that brings you to that result?

We strive to have high test coverage, and we use code coverage tools to help guide us to this. While we don't have 100% coverage, that's not a realistic goal for any library, but what we do have is very high.

It is true that the docs are severely lacking (mostly non-existent). Rx has been a volunteer effort and there has been no budget to write robust docs. Docs are hard and expensive to write and we'd take any contributions for that.

[Turnerj]

While I personally think System.Linq.Async would be better inside the runtime, I have a different take onto why it should be there than @MgSam.

My problem isn't with having an additional dependency, its that the discoverability of this isn't obvious when non-async LINQ is available without an additional dependency (ie. is someone going to know that async LINQ is a NuGet package). It could be argued that not many people would need to use it compared to normal/sync LINQ so it doesn't need to belong in the runtime and I'd believe that at present. The problem here though is that more and more code is being encouraged to be async, especially ASP.NET Core applications and the various controllers/action methods we write. Not including async LINQ in the runtime feels like there are mixed messages where we are encouraged to be async but are only given half of the tools out of the box to do so.

In terms of having a healthy .NET ecosystem, I feel like there is already a fairly big case of where this has happened - Newtonsoft.Json and System.Text.Json. The former was effectively the defacto standard for JSON in .NET with probably every .NET library needing to do anything with JSON using it. With System.Text.Json, that's a non-trivial addition to the runtime for something that was widely adopted. I wouldn't be surprised if usage of Newtonsoft.Json drops dramatically in the near future (I'm moving my own libraries and projects to System.Text.Json, I imagine many others are too). I don't know all the reasons for System.Text.Json nor do I even think the decision was necessarily bad as if every project is basically including the same library, why have it as a library and not the runtime. This though is where I think async LINQ might be in the not-terribly-distant future - a library that may be effectively added to every project to do the same thing.

I do totally understand the point of being hesitant for doing this as the runtime already has a huge API surface with probably tens of thousands of tests and any less code added would make maintaining it far easier. If integrating System.Linq.Async was a possibility, I could totally see maintainability as a roadblock for it.

I think my feelings for this can be summed up as: async LINQ feels like a second class citizen compared to (normal) LINQ. If the future is async (which it seems to be), it makes sense to me that the runtime provides parity of the existing data query methods as async.

Or maybe I'm wrong with all of this - actual use of async LINQ won't grow because async operations are often relegated to file, socket or DB operations which either don't use IAsyncEnumerable or provide their own custom ways of handling async LINQ (that's if we call a DB layer using IQueryable with a custom ToListAsync extension method "async LINQ").

[oliverjanik]

Having officially MS sanctioned packages means a lot in the enterprise space. Every dependency has to be vetted and approved by security team. Every update to every dependency has to be approved as well.

.NET has always been "batteries included". Splitting ecosystem into thousands of competing packages is evolving .NET towards where node/npm is and that is a regression IMO.

Having put enormous enginneering effort into IAsyncEnumerable and having it ignored be cause it lacks basic convenience methods is a huge waste. It will go unnoticed and unused.

And what will happen is every library that uses or exposes IAsyncEnumerable with ship it's own implementation of some of these methods or it will ship a transitive dependency on the AsyncEnumerable nuget. Too many transitive nugets lead to versioning hell.

Please reconsider.

[MgSam]

@stephentoub

the existing async LINQ library met all the key needs for it...

Actions speak louder than words. Rx and Ix have been 3rd class citizens for years. For anyone who does the most basic Google search they'll find ancient documentation for Rx v1 much of which is wrong. introtorx.com, one of the top results, was last updated in 2015.

Every .NET Core release you write an enormous blog post detailing the performance and other enhancements going into .NET. When is the last time Rx or Ix was mentioned in that? You have someone from the team itself on this very thread saying they don't have the resources they need to produce a good quality library. Async Linq needs to be properly designed by Microsoft engineers and put in .NET proper or IAsyncEnumerable will remain a throw away feature few know about or use.

It's fine if you guys don't have the resources for this now, but it should at least discussed and triaged honestly instead of pretending that there is already a satisfactory solution available.