Skip to content

Commit

Permalink
Docs
Browse files Browse the repository at this point in the history
  • Loading branch information
@louthy
louthy committed Feb 15, 2025
1 parent b6cff1b commit 53d8630
Show file tree
Hide file tree
Showing 3 changed files with 129 additions and 205 deletions.
6 changes: 4 additions & 2 deletions LanguageExt.Core/Monads/README.md
View file
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
Not all of the monadic types are in this section, but most of them are. The other monadic types are
[the collections](https://louthy.github.io/language-ext/LanguageExt.Core/Immutable%20Collections/index.html), and the [effect-system monads of `IO`, `Eff`, `StreamT`, and `Proxy`](https://louthy.github.io/language-ext/LanguageExt.Core/Effects/index.html).
Not all of the monadic types are in this section, but most of them
are. The other monadic types are [the collections](https://louthy.github.io/language-ext/LanguageExt.Core/Immutable%20Collections/index.html), and the
[effect-system monads of `IO`, `Eff`, `StreamT`](https://louthy.github.io/language-ext/LanguageExt.Core/Effects/index.html),
and [`Pipes`](https://louthy.github.io/language-ext/LanguageExt.Pipes/index.html)

163 changes: 62 additions & 101 deletions LanguageExt.Pipes/README.md
View file
Original file line number Diff line number Diff line change
@@ -1,10 +1,4 @@
_This feature of language-ext is based on the wonderful work of
[Gabriella Gonzalez](https://twitter.com/GabriellaG439) on the [Haskell Pipes
library](https://hackage.haskell.org/package/pipes-4.3.16/docs/Pipes.html). I have
had to make some significant changes to make it work in C#, but the essence is the
same, and the core types and composition of the components is exactly the same._

* If you find this feature confusing at first, and it wouldn't be surprising as
> If you find this feature confusing at first, and it wouldn't be surprising as
it's quite a complex idea, there are some examples in the [EffectsExample sample in the repo](https://github.com/louthy/language-ext/blob/main/Samples/EffectsExamples/Examples/TextFileChunkStreamExample.cs)

Conventional stream programming forces you to choose only two of the
Expand All @@ -18,7 +12,7 @@ If you sacrifice _Effects_ you get `IEnumerable`, which you
can transform using composable functions in constant space, but without
interleaving effects (other than of the _imperative kind_).

If you sacrifice _Streaming_ you get 'Traverse' and 'Sequence', which are
If you sacrifice _Streaming_ you get `Traverse` and `Sequence`, which are
composable and effectful, but do not return a single result until the whole
list has first been processed and loaded into memory.

Expand All @@ -30,18 +24,19 @@ is streaming and effectful, but is not modular or separable.
programming. `Pipes` also provides a wide variety of stream programming
abstractions which are all subsets of a single unified machinery:

* effectful [`Producer`](Producer),
* effectful [`Consumer`](Consumer),
* effectful [`Pipe`](Pipe) (like Unix pipes)

On top of that, `Pipes` has more advanced features, including bi-directional
streaming. This comes into play when fusing clients and servers:
* Effectful [`Producer`](Producer) and [`ProducerT`](ProducerT),
* Effectful [`Consumer`](Consumer) and [`ConsumerT`](ConsumerT),
* Effectful [`Pipe`](Pipe) and [`PipeT`](PipeT) (like Unix pipes)
* Effectful [`Effect`](Effect) and [`EffectT`](EffectT)

* effectful [`Client`](Client),
* effectful [`Server`](Server),
> The `T` suffix types (`ProducerT`, `ConsumerT`, `PipeT`, and `EffectT`) are the
> more generalist monad-transformers. They can lift any monad `M` you like into them,
> supplementing the behaviour of `Pipes` with the behaviour of `M`. The non-`T`
> suffix types (`Producer`, `Consumer`, `Pipe`, and `Effect`) only support the lifting
> of the `Eff<RT, A>` type. They're slightly easier to use, just less flexible.
All of these are connectable and you can combine them together in clever and
unexpected ways because they all share the same underlying type.
unexpected ways because they all share the same underlying type: [`PipeT`](PipeT).

The pipes ecosystem decouples stream processing stages from each other so
that you can mix and match diverse stages to produce useful streaming
Expand All @@ -52,148 +47,114 @@ produce a highly-efficient program that streams data in constant memory.

To enforce loose coupling, components can only communicate using two commands:

* [`yield`](#Proxy_0_yield_1): Send output data
* [`awaiting`](#Proxy_0_awaiting_1): Receive input data
* `yield`: Send output data
* `awaiting`: Receive input data

Pipes has four types of components built around these two commands:

* [`Producer`](Producer) can only [`yield`](#Proxy_0_yield_1) values and they model streaming sources
* [`Consumer`](Consumer) can only be [`awaiting`](#Proxy_0_awaiting_1) values and they model streaming sinks
* [`Pipe`](Pipe) can both [`yield`](#Proxy_0_yield_1) and be [`awaiting`](#Proxy_0_awaiting_1) values and they model stream transformations
* [`Effect`](Effect) can neither [`yield`](#Proxy_0_yield_1) nor be [`awaiting`](#Proxy_0_awaiting_1) and they model non-streaming components

Pipes uses parametric polymorphism (i.e. generics) to overload all operations.
* [`Producer`](Producer) and [`ProducerT`](ProducerT) yield values downstream and can only do so using: `Producer.yield` and `ProducerT.yield`.
* [`Consumer`](Consumer) and [`ConsumerT`](ConsumerT) await values from upstream and can only do so using: `Consumer.awaiting` and `ConsumerT.awaiting`.
* [`Pipe`](Pipe) and [`PipeT`](PipeT) can both await and yield, using: `Pipe.awaiting`, `PipeT.awaiting`, `Pipe.yield`, and `PipeT.yield`.
* [`Effect`](Effect) and [`EffectT`](EffectT) can neither yield nor await and they model non-streaming components.

You've probably noticed this overloading already:
Pipes uses parametric polymorphism (i.e. generics) to overload all operations. The operator `|` connects `Producer`, `Consumer`, and `Pipe` by 'fusing'
them together. Eventually they will 'fuse' together into an `Effect` or `EffectT`. This final state can be `.Run()`, you must fuse to an `Effect` or
`EffectT` to be able to invoke any of the pipelines.

* [`yield`](#Proxy_0_yield_1) works within both [`Producer`](Producer) and a [`Pipe`](Pipe)
* [`Consumer`](Consumer) works within both [`Consumer`](Consumer) and [`Pipe`](Pipe)
* The operator `|` connects [`Producer`](Producer), [`Consumer`](Consumer), and [`Pipe`](Pipe) in varying ways

_This overloading is great when it works, but when connections fail they
produce type errors that appear intimidating at first. This section
explains the underlying types so that you can work through type errors
intelligently._

`Producer`, `Consumer`, `Pipe`, and `Effect` are all special cases of a
single underlying type: [`Proxy`](#LanguageExt.Pipes_0_Proxy_6). This overarching type permits fully
bidirectional communication on both an upstream and downstream interface.
`Producer`, `ProducerT`, `Consumer`, `ConsumerT`, `Pipe`, `Effect`, and `EffectT` are all special cases of a
single underlying type: [`PipeT`](PipeT).

You can think of it as having the following shape:

Proxy<RT, UOut, UIn, DIn, DOut, A>
PipeT<IN, OUT, M, A>

Upstream | Downstream
+---------+
| |
UOut ◄-- ◄-- DIn -- Information flowing upstream
| |
UIn --► --► DOut -- Information flowing downstream
IN --► --► OUT -- Information flowing downstream
| | |
+----|----+
|
A
The four core types do not use the upstream flow of information. This means
that the `UOut` and `DIn` in the above diagram go unused unless you use the
more advanced features.

Pipes uses type synonyms to hide unused inputs or outputs and clean up
type signatures. These type synonyms come in two flavors:

* Concrete type synonyms that explicitly close unused inputs and outputs of
the `Proxy` type
the `Proxy` type.

* Polymorphic type synonyms that don't explicitly close unused inputs or
outputs
outputs.

The concrete type synonyms use `Unit` to close unused inputs and `Void` (the
uninhabited type) to close unused outputs:

* `Effect`: explicitly closes both ends, forbidding `awaiting` and `yield`
* `EffectT`: explicitly closes both ends, forbidding `awaiting` and `yield`:

Effect<RT, A> = Proxy<RT, Void, Unit, Unit, Void, A>
EffectT<M, A> = PipeT<Unit, Void, M, A>
Upstream | Downstream

+---------+
| |
Void ◄-- ◄-- Unit
| |
Unit --► --► Void
Unit --► --► Void
| | |
+----|----+
|
A

* `Producer`: explicitly closes the upstream end, forbidding `awaiting`
* `ProducerT`: explicitly closes the upstream end, forbidding `awaiting`:

Producer<RT, OUT, A> = Proxy<RT, Void, Unit, Unit, OUT, A>
ProducerT<OUT, M, A> = PipeT<Unit, OUT, M, A>
Upstream | Downstream

+---------+
| |
Void ◄-- ◄-- Unit
| |
Unit --► --► OUT
Unit --► --► OUT
| | |
+----|----+
|
A

* `Consumer`: explicitly closes the downstream end, forbidding `yield`
* `ConsumerT`: explicitly closes the downstream end, forbidding `yield`:

Consumer<RT, IN, A> = Proxy<RT, Unit, IN, Unit, Void, A>
ConsumerT<IN, M, A> = PipeT<IN, Void, M, A>
Upstream | Downstream
+---------+
| |
Unit ◄-- ◄-- Unit
| |
IN --► --► Void
| | |
+----|----+
|
A
* `Pipe`: marks both ends open, allowing both `awaiting` and `yield`

Pipe<RT, IN, OUT, A> = Proxy<RT, Unit, IN, Unit, OUT, A>
Upstream | Downstream
+---------+
| |
Unit ◄-- ◄-- Unit
| |
IN --► --► OUT
IN --► --► Void
| | |
+----|----+
|
A
When you compose `Proxy` using `|` all you are doing is placing them
When you compose `PipeT` using `|` all you are doing is placing them
side by side and fusing them laterally. For example, when you compose a
`Producer`, `Pipe`, and a `Consumer`, you can think of information flowing
`ProducerT`, `PipeT`, and a `ConsumerT`, you can think of information flowing
like this:

Producer Pipe Consumer
+------------+ +------------+ +-------------+
| | | | | |
Void ◄-- ◄-- Unit ◄-- ◄-- Unit ◄-- ◄-- Unit
| readLine | | parseInt | | writeLine |
Unit --► --► string --► --► string --► --► Void
| | | | | | | | |
+-----|------+ +----|-------+ +------|------+
v v v
() () ()

Composition fuses away the intermediate interfaces, leaving behind an `Effect`:

Effect
+-----------------------------------+
| |
Void ◄-- ◄-- Unit
| readLine | parseInt | writeLine |
Unit --► --► Void
| |
+----------------|------------------+
Unit
ProducerT PipeT ConsumerT
+-------------+ +------------+ +-------------+
| | | | | |
Unit --► readLine --► string --► parseInt --► int --► writeLine --► Void
| | | | | | | | |
+------|------+ +------|-----+ +------|------+
| | |
A A A

Composition fuses away the intermediate interfaces, leaving behind an `EffectT`:

EffectT
+-------------------------------------+
| |
Unit --► readLine | parseInt | writeLine --► Void
| |
+------------------|------------------+
|
A

This `EffectT` can be `Run()` which will return the composed underlying `M` type. Or,
if it's an `Effect` will return the composed underlying `Eff<RT, A>`.
Loading

0 comments on commit 53d8630

Please sign in to comment.