From 53d86302fd3f6e89b0b1db9881d33661c82fe4cf Mon Sep 17 00:00:00 2001 From: Paul Louth Date: Sat, 15 Feb 2025 18:12:09 +0000 Subject: [PATCH] Docs --- LanguageExt.Core/Monads/README.md | 6 +- LanguageExt.Pipes/README.md | 163 +++++++++++------------------ LanguageExt.Pipes/README.nuget.md | 165 ++++++++++++------------------ 3 files changed, 129 insertions(+), 205 deletions(-) diff --git a/LanguageExt.Core/Monads/README.md b/LanguageExt.Core/Monads/README.md index c142bbfd3..96a319110 100644 --- a/LanguageExt.Core/Monads/README.md +++ b/LanguageExt.Core/Monads/README.md @@ -1,3 +1,5 @@ -Not all of the monadic types are in this section, but most of them are. The other monadic types are -[the collections](https://louthy.github.io/language-ext/LanguageExt.Core/Immutable%20Collections/index.html), and the [effect-system monads of `IO`, `Eff`, `StreamT`, and `Proxy`](https://louthy.github.io/language-ext/LanguageExt.Core/Effects/index.html). +Not all of the monadic types are in this section, but most of them +are. The other monadic types are [the collections](https://louthy.github.io/language-ext/LanguageExt.Core/Immutable%20Collections/index.html), and the +[effect-system monads of `IO`, `Eff`, `StreamT`](https://louthy.github.io/language-ext/LanguageExt.Core/Effects/index.html), +and [`Pipes`](https://louthy.github.io/language-ext/LanguageExt.Pipes/index.html) diff --git a/LanguageExt.Pipes/README.md b/LanguageExt.Pipes/README.md index 208ef1b11..37b75f64f 100644 --- a/LanguageExt.Pipes/README.md +++ b/LanguageExt.Pipes/README.md @@ -1,10 +1,4 @@ -_This feature of language-ext is based on the wonderful work of -[Gabriella Gonzalez](https://twitter.com/GabriellaG439) on the [Haskell Pipes -library](https://hackage.haskell.org/package/pipes-4.3.16/docs/Pipes.html). I have -had to make some significant changes to make it work in C#, but the essence is the -same, and the core types and composition of the components is exactly the same._ - -* If you find this feature confusing at first, and it wouldn't be surprising as +> If you find this feature confusing at first, and it wouldn't be surprising as it's quite a complex idea, there are some examples in the [EffectsExample sample in the repo](https://github.com/louthy/language-ext/blob/main/Samples/EffectsExamples/Examples/TextFileChunkStreamExample.cs) Conventional stream programming forces you to choose only two of the @@ -18,7 +12,7 @@ If you sacrifice _Effects_ you get `IEnumerable`, which you can transform using composable functions in constant space, but without interleaving effects (other than of the _imperative kind_). -If you sacrifice _Streaming_ you get 'Traverse' and 'Sequence', which are +If you sacrifice _Streaming_ you get `Traverse` and `Sequence`, which are composable and effectful, but do not return a single result until the whole list has first been processed and loaded into memory. @@ -30,18 +24,19 @@ is streaming and effectful, but is not modular or separable. programming. `Pipes` also provides a wide variety of stream programming abstractions which are all subsets of a single unified machinery: -* effectful [`Producer`](Producer), -* effectful [`Consumer`](Consumer), -* effectful [`Pipe`](Pipe) (like Unix pipes) - -On top of that, `Pipes` has more advanced features, including bi-directional -streaming. This comes into play when fusing clients and servers: +* Effectful [`Producer`](Producer) and [`ProducerT`](ProducerT), +* Effectful [`Consumer`](Consumer) and [`ConsumerT`](ConsumerT), +* Effectful [`Pipe`](Pipe) and [`PipeT`](PipeT) (like Unix pipes) +* Effectful [`Effect`](Effect) and [`EffectT`](EffectT) -* effectful [`Client`](Client), -* effectful [`Server`](Server), +> The `T` suffix types (`ProducerT`, `ConsumerT`, `PipeT`, and `EffectT`) are the +> more generalist monad-transformers. They can lift any monad `M` you like into them, +> supplementing the behaviour of `Pipes` with the behaviour of `M`. The non-`T` +> suffix types (`Producer`, `Consumer`, `Pipe`, and `Effect`) only support the lifting +> of the `Eff` type. They're slightly easier to use, just less flexible. All of these are connectable and you can combine them together in clever and -unexpected ways because they all share the same underlying type. +unexpected ways because they all share the same underlying type: [`PipeT`](PipeT). The pipes ecosystem decouples stream processing stages from each other so that you can mix and match diverse stages to produce useful streaming @@ -52,148 +47,114 @@ produce a highly-efficient program that streams data in constant memory. To enforce loose coupling, components can only communicate using two commands: -* [`yield`](#Proxy_0_yield_1): Send output data -* [`awaiting`](#Proxy_0_awaiting_1): Receive input data +* `yield`: Send output data +* `awaiting`: Receive input data Pipes has four types of components built around these two commands: -* [`Producer`](Producer) can only [`yield`](#Proxy_0_yield_1) values and they model streaming sources -* [`Consumer`](Consumer) can only be [`awaiting`](#Proxy_0_awaiting_1) values and they model streaming sinks -* [`Pipe`](Pipe) can both [`yield`](#Proxy_0_yield_1) and be [`awaiting`](#Proxy_0_awaiting_1) values and they model stream transformations -* [`Effect`](Effect) can neither [`yield`](#Proxy_0_yield_1) nor be [`awaiting`](#Proxy_0_awaiting_1) and they model non-streaming components - -Pipes uses parametric polymorphism (i.e. generics) to overload all operations. +* [`Producer`](Producer) and [`ProducerT`](ProducerT) yield values downstream and can only do so using: `Producer.yield` and `ProducerT.yield`. +* [`Consumer`](Consumer) and [`ConsumerT`](ConsumerT) await values from upstream and can only do so using: `Consumer.awaiting` and `ConsumerT.awaiting`. +* [`Pipe`](Pipe) and [`PipeT`](PipeT) can both await and yield, using: `Pipe.awaiting`, `PipeT.awaiting`, `Pipe.yield`, and `PipeT.yield`. +* [`Effect`](Effect) and [`EffectT`](EffectT) can neither yield nor await and they model non-streaming components. -You've probably noticed this overloading already: +Pipes uses parametric polymorphism (i.e. generics) to overload all operations. The operator `|` connects `Producer`, `Consumer`, and `Pipe` by 'fusing' +them together. Eventually they will 'fuse' together into an `Effect` or `EffectT`. This final state can be `.Run()`, you must fuse to an `Effect` or +`EffectT` to be able to invoke any of the pipelines. - * [`yield`](#Proxy_0_yield_1) works within both [`Producer`](Producer) and a [`Pipe`](Pipe) - * [`Consumer`](Consumer) works within both [`Consumer`](Consumer) and [`Pipe`](Pipe) - * The operator `|` connects [`Producer`](Producer), [`Consumer`](Consumer), and [`Pipe`](Pipe) in varying ways - -_This overloading is great when it works, but when connections fail they -produce type errors that appear intimidating at first. This section -explains the underlying types so that you can work through type errors -intelligently._ - -`Producer`, `Consumer`, `Pipe`, and `Effect` are all special cases of a -single underlying type: [`Proxy`](#LanguageExt.Pipes_0_Proxy_6). This overarching type permits fully -bidirectional communication on both an upstream and downstream interface. +`Producer`, `ProducerT`, `Consumer`, `ConsumerT`, `Pipe`, `Effect`, and `EffectT` are all special cases of a +single underlying type: [`PipeT`](PipeT). You can think of it as having the following shape: - Proxy + PipeT Upstream | Downstream +---------+ | | - UOut ◄-- ◄-- DIn -- Information flowing upstream - | | - UIn --► --► DOut -- Information flowing downstream + IN --► --► OUT -- Information flowing downstream | | | +----|----+ | A - The four core types do not use the upstream flow of information. This means - that the `UOut` and `DIn` in the above diagram go unused unless you use the - more advanced features. - Pipes uses type synonyms to hide unused inputs or outputs and clean up type signatures. These type synonyms come in two flavors: * Concrete type synonyms that explicitly close unused inputs and outputs of - the `Proxy` type + the `Proxy` type. * Polymorphic type synonyms that don't explicitly close unused inputs or - outputs + outputs. The concrete type synonyms use `Unit` to close unused inputs and `Void` (the uninhabited type) to close unused outputs: - * `Effect`: explicitly closes both ends, forbidding `awaiting` and `yield` + * `EffectT`: explicitly closes both ends, forbidding `awaiting` and `yield`: - Effect = Proxy + EffectT = PipeT Upstream | Downstream + +---------+ | | - Void ◄-- ◄-- Unit - | | - Unit --► --► Void + Unit --► --► Void | | | +----|----+ | A - * `Producer`: explicitly closes the upstream end, forbidding `awaiting` + * `ProducerT`: explicitly closes the upstream end, forbidding `awaiting`: - Producer = Proxy + ProducerT = PipeT Upstream | Downstream + +---------+ | | - Void ◄-- ◄-- Unit - | | - Unit --► --► OUT + Unit --► --► OUT | | | +----|----+ | A - * `Consumer`: explicitly closes the downstream end, forbidding `yield` + * `ConsumerT`: explicitly closes the downstream end, forbidding `yield`: - Consumer = Proxy + ConsumerT = PipeT Upstream | Downstream - +---------+ - | | - Unit ◄-- ◄-- Unit - | | - IN --► --► Void - | | | - +----|----+ - | - A - - * `Pipe`: marks both ends open, allowing both `awaiting` and `yield` - Pipe = Proxy - - Upstream | Downstream +---------+ | | - Unit ◄-- ◄-- Unit - | | - IN --► --► OUT + IN --► --► Void | | | +----|----+ | A -When you compose `Proxy` using `|` all you are doing is placing them +When you compose `PipeT` using `|` all you are doing is placing them side by side and fusing them laterally. For example, when you compose a -`Producer`, `Pipe`, and a `Consumer`, you can think of information flowing +`ProducerT`, `PipeT`, and a `ConsumerT`, you can think of information flowing like this: - Producer Pipe Consumer - +------------+ +------------+ +-------------+ - | | | | | | - Void ◄-- ◄-- Unit ◄-- ◄-- Unit ◄-- ◄-- Unit - | readLine | | parseInt | | writeLine | - Unit --► --► string --► --► string --► --► Void - | | | | | | | | | - +-----|------+ +----|-------+ +------|------+ - v v v - () () () - - Composition fuses away the intermediate interfaces, leaving behind an `Effect`: - - Effect - +-----------------------------------+ - | | - Void ◄-- ◄-- Unit - | readLine | parseInt | writeLine | - Unit --► --► Void - | | - +----------------|------------------+ - Unit + ProducerT PipeT ConsumerT + +-------------+ +------------+ +-------------+ + | | | | | | + Unit --► readLine --► string --► parseInt --► int --► writeLine --► Void + | | | | | | | | | + +------|------+ +------|-----+ +------|------+ + | | | + A A A + + Composition fuses away the intermediate interfaces, leaving behind an `EffectT`: + + EffectT + +-------------------------------------+ + | | + Unit --► readLine | parseInt | writeLine --► Void + | | + +------------------|------------------+ + | + A + +This `EffectT` can be `Run()` which will return the composed underlying `M` type. Or, +if it's an `Effect` will return the composed underlying `Eff`. \ No newline at end of file diff --git a/LanguageExt.Pipes/README.nuget.md b/LanguageExt.Pipes/README.nuget.md index d3cc53b0d..306063661 100644 --- a/LanguageExt.Pipes/README.nuget.md +++ b/LanguageExt.Pipes/README.nuget.md @@ -1,13 +1,7 @@ # LanguageExt.Pipes -_This feature of language-ext is based on the wonderful work of -[Gabriella Gonzalez](https://twitter.com/GabriellaG439) on the [Haskell Pipes -library](https://hackage.haskell.org/package/pipes-4.3.16/docs/Pipes.html). I have -had to make some significant changes to make it work in C#, but the essence is the -same, and the core types and composition of the components is exactly the same._ - -* If you find this feature confusing at first, and it wouldn't be surprising as - it's quite a complex idea, there are some examples in the [EffectsExample sample in the repo](https://github.com/louthy/language-ext/blob/main/Samples/EffectsExamples/Examples/TextFileChunkStreamExample.cs) +> If you find this feature confusing at first, and it wouldn't be surprising as +it's quite a complex idea, there are some examples in the [EffectsExample sample in the repo](https://github.com/louthy/language-ext/blob/main/Samples/EffectsExamples/Examples/TextFileChunkStreamExample.cs) Conventional stream programming forces you to choose only two of the following three features: @@ -20,7 +14,7 @@ If you sacrifice _Effects_ you get `IEnumerable`, which you can transform using composable functions in constant space, but without interleaving effects (other than of the _imperative kind_). -If you sacrifice _Streaming_ you get 'Traverse' and 'Sequence', which are +If you sacrifice _Streaming_ you get `Traverse` and `Sequence`, which are composable and effectful, but do not return a single result until the whole list has first been processed and loaded into memory. @@ -32,18 +26,19 @@ is streaming and effectful, but is not modular or separable. programming. `Pipes` also provides a wide variety of stream programming abstractions which are all subsets of a single unified machinery: -* effectful [`Producer`](Producer), -* effectful [`Consumer`](Consumer), -* effectful [`Pipe`](Pipe) (like Unix pipes) - -On top of that, `Pipes` has more advanced features, including bi-directional -streaming. This comes into play when fusing clients and servers: +* Effectful [`Producer`](Producer) and [`ProducerT`](ProducerT), +* Effectful [`Consumer`](Consumer) and [`ConsumerT`](ConsumerT), +* Effectful [`Pipe`](Pipe) and [`PipeT`](PipeT) (like Unix pipes) +* Effectful [`Effect`](Effect) and [`EffectT`](EffectT) -* effectful [`Client`](Client), -* effectful [`Server`](Server), +> The `T` suffix types (`ProducerT`, `ConsumerT`, `PipeT`, and `EffectT`) are the +> more generalist monad-transformers. They can lift any monad `M` you like into them, +> supplementing the behaviour of `Pipes` with the behaviour of `M`. The non-`T` +> suffix types (`Producer`, `Consumer`, `Pipe`, and `Effect`) only support the lifting +> of the `Eff` type. They're slightly easier to use, just less flexible. All of these are connectable and you can combine them together in clever and -unexpected ways because they all share the same underlying type. +unexpected ways because they all share the same underlying type: [`PipeT`](PipeT). The pipes ecosystem decouples stream processing stages from each other so that you can mix and match diverse stages to produce useful streaming @@ -54,148 +49,114 @@ produce a highly-efficient program that streams data in constant memory. To enforce loose coupling, components can only communicate using two commands: -* [`yield`](#Proxy_0_yield_1): Send output data -* [`awaiting`](#Proxy_0_awaiting_1): Receive input data +* `yield`: Send output data +* `awaiting`: Receive input data Pipes has four types of components built around these two commands: -* [`Producer`](Producer) can only [`yield`](#Proxy_0_yield_1) values and they model streaming sources -* [`Consumer`](Consumer) can only be [`awaiting`](#Proxy_0_awaiting_1) values and they model streaming sinks -* [`Pipe`](Pipe) can both [`yield`](#Proxy_0_yield_1) and be [`awaiting`](#Proxy_0_awaiting_1) values and they model stream transformations -* [`Effect`](Effect) can neither [`yield`](#Proxy_0_yield_1) nor be [`awaiting`](#Proxy_0_awaiting_1) and they model non-streaming components - -Pipes uses parametric polymorphism (i.e. generics) to overload all operations. +* [`Producer`](Producer) and [`ProducerT`](ProducerT) yield values downstream and can only do so using: `Producer.yield` and `ProducerT.yield`. +* [`Consumer`](Consumer) and [`ConsumerT`](ConsumerT) await values from upstream and can only do so using: `Consumer.awaiting` and `ConsumerT.awaiting`. +* [`Pipe`](Pipe) and [`PipeT`](PipeT) can both await and yield, using: `Pipe.awaiting`, `PipeT.awaiting`, `Pipe.yield`, and `PipeT.yield`. +* [`Effect`](Effect) and [`EffectT`](EffectT) can neither yield nor await and they model non-streaming components. -You've probably noticed this overloading already: +Pipes uses parametric polymorphism (i.e. generics) to overload all operations. The operator `|` connects `Producer`, `Consumer`, and `Pipe` by 'fusing' +them together. Eventually they will 'fuse' together into an `Effect` or `EffectT`. This final state can be `.Run()`, you must fuse to an `Effect` or +`EffectT` to be able to invoke any of the pipelines. -* [`yield`](#Proxy_0_yield_1) works within both [`Producer`](Producer) and a [`Pipe`](Pipe) -* [`Consumer`](Consumer) works within both [`Consumer`](Consumer) and [`Pipe`](Pipe) -* The operator `|` connects [`Producer`](Producer), [`Consumer`](Consumer), and [`Pipe`](Pipe) in varying ways - -_This overloading is great when it works, but when connections fail they -produce type errors that appear intimidating at first. This section -explains the underlying types so that you can work through type errors -intelligently._ - -`Producer`, `Consumer`, `Pipe`, and `Effect` are all special cases of a -single underlying type: [`Proxy`](#LanguageExt.Pipes_0_Proxy_6). This overarching type permits fully -bidirectional communication on both an upstream and downstream interface. +`Producer`, `ProducerT`, `Consumer`, `ConsumerT`, `Pipe`, `Effect`, and `EffectT` are all special cases of a +single underlying type: [`PipeT`](PipeT). You can think of it as having the following shape: - Proxy + PipeT Upstream | Downstream +---------+ | | - UOut ◄-- ◄-- DIn -- Information flowing upstream - | | - UIn --► --► DOut -- Information flowing downstream + IN --► --► OUT -- Information flowing downstream | | | +----|----+ | A -The four core types do not use the upstream flow of information. This means -that the `UOut` and `DIn` in the above diagram go unused unless you use the -more advanced features. - Pipes uses type synonyms to hide unused inputs or outputs and clean up type signatures. These type synonyms come in two flavors: * Concrete type synonyms that explicitly close unused inputs and outputs of - the `Proxy` type + the `Proxy` type. * Polymorphic type synonyms that don't explicitly close unused inputs or - outputs + outputs. The concrete type synonyms use `Unit` to close unused inputs and `Void` (the uninhabited type) to close unused outputs: -* `Effect`: explicitly closes both ends, forbidding `awaiting` and `yield` +* `EffectT`: explicitly closes both ends, forbidding `awaiting` and `yield`: - Effect = Proxy + EffectT = PipeT Upstream | Downstream + +---------+ | | - Void ◄-- ◄-- Unit - | | - Unit --► --► Void + Unit --► --► Void | | | +----|----+ | A -* `Producer`: explicitly closes the upstream end, forbidding `awaiting` +* `ProducerT`: explicitly closes the upstream end, forbidding `awaiting`: - Producer = Proxy + ProducerT = PipeT Upstream | Downstream + +---------+ | | - Void ◄-- ◄-- Unit - | | - Unit --► --► OUT + Unit --► --► OUT | | | +----|----+ | A -* `Consumer`: explicitly closes the downstream end, forbidding `yield` +* `ConsumerT`: explicitly closes the downstream end, forbidding `yield`: - Consumer = Proxy + ConsumerT = PipeT Upstream | Downstream - +---------+ - | | - Unit ◄-- ◄-- Unit - | | - IN --► --► Void - | | | - +----|----+ - | - A - -* `Pipe`: marks both ends open, allowing both `awaiting` and `yield` - Pipe = Proxy - - Upstream | Downstream +---------+ | | - Unit ◄-- ◄-- Unit - | | - IN --► --► OUT + IN --► --► Void | | | +----|----+ | A -When you compose `Proxy` using `|` all you are doing is placing them +When you compose `PipeT` using `|` all you are doing is placing them side by side and fusing them laterally. For example, when you compose a -`Producer`, `Pipe`, and a `Consumer`, you can think of information flowing +`ProducerT`, `PipeT`, and a `ConsumerT`, you can think of information flowing like this: - Producer Pipe Consumer - +------------+ +------------+ +-------------+ - | | | | | | - Void ◄-- ◄-- Unit ◄-- ◄-- Unit ◄-- ◄-- Unit - | readLine | | parseInt | | writeLine | - Unit --► --► string --► --► string --► --► Void - | | | | | | | | | - +-----|------+ +----|-------+ +------|------+ - v v v - () () () - -Composition fuses away the intermediate interfaces, leaving behind an `Effect`: - - Effect - +-----------------------------------+ - | | - Void ◄-- ◄-- Unit - | readLine | parseInt | writeLine | - Unit --► --► Void - | | - +----------------|------------------+ - Unit + ProducerT PipeT ConsumerT + +-------------+ +------------+ +-------------+ + | | | | | | + Unit --► readLine --► string --► parseInt --► int --► writeLine --► Void + | | | | | | | | | + +------|------+ +------|-----+ +------|------+ + | | | + A A A + +Composition fuses away the intermediate interfaces, leaving behind an `EffectT`: + + EffectT + +-------------------------------------+ + | | + Unit --► readLine | parseInt | writeLine --► Void + | | + +------------------|------------------+ + | + A + +This `EffectT` can be `Run()` which will return the composed underlying `M` type. Or, +if it's an `Effect` will return the composed underlying `Eff`. \ No newline at end of file