Any future?

[dumblob]

I wonder whether there is some extensive roadmap or at least ambitious ideas how to further develop this neat language?

I have some crazy ideas but I'd first want to know about existing plans 😉.

[dkumor]

My personal plans for Pipescript are mainly towards ease of use at the moment, since I have less and less free time these days. The list of things I am currently planning:

• Syntax sugar for object handling: d['hello']:d['world'] would become d.hello.world, and allowing negative indices d[-1].
• Type conversions, so str, int, float, etc
• More functions that make it easier to use in practice - this will require some time to see pain points in realistic scenarios
• Some way to store variables temporarily for later use in the pipe
• More powerful/robust dataset generation functionality
• Speed/efficiency improvements

Unfortunately, the above might take some time, since my main focus is currently on improving and documenting heedy, as well as building integrations.

I am curious about your ideas! I'd love to hear about them, maybe some could be easily implemented!

[dumblob]

I'm generally impressed about the amount of functionality you managed to implement in pipescript. So take the following crazy brainstorming as a highly uneducated convoluted list of wishes (not necessarily implementable in pipescript; not necessarily good ideas as they're largely unexplored AFAIK).

1. create a comprehensive language for data manipulation (I'm deliberately not saying a programming language because that has some prejudicial scent to it - i.e. no stop-the-world before function entrance and resume after return, but rather focus on data manipulation in time without stopping the world)
2. feedback loops (basically arbitrary cycles - feed a (part of) output back as (part of) input)
3. splitting & muxing (both with and without breaking the full ordering)
4. merging & demuxing (both with and without breaking the full ordering)
5. changing frequency
6. reshaping data samples
7. (recursive) windowing/buffering & stepped/window-delimited processing (to break full ordering into partial ordering and back)
8. synchronous (blocking) joining/merging/demuxing
9. asynchronous (non-blocking) joining/merging/demuxing
10. recoding data samples (useful for integrations, compatibility with outer world, performance optimization, etc.)
11. runtime arbitrary creation & deletion of pipe_instances/pipelines based on runtime inputs
12. full AOT type safety (incl. null/none/nil safety) - this is vague but basically ensuring all inputs of all commands in pipelines have known format at the latest at the very moment of invoking the application
13. pipe-based-feedback-loop error handling (i.e. no "side channels" - no exceptions, no effects, nothing like that but just pipes - any error shall be treated as just another regular pipe)
14. push/pull semantics choice anywhere in the graph
15. pipes in pipes (i.e. recursive embedding of infinite streams)
16. pipes can be heterogeneous (or trunks of homogeneous pipes?)
17. ...

This has many use cases as you know - from easy data exploration through easy writing of true apps up to having no-effort UIs (implicit insertion of | prettyprint in nushell, https://github.com/calebwin/stdg etc.). And the best of all - it inherently supports manycore parallelism (see https://binpa.sh/docs/tutorial/ )! Yeah - a (first ever?) low-effort low-code fully parallel data manipulation (programming?) paradigm!

Crazy, isn't it?

[dkumor]

That looks very ambitious! Based on the points given, it seems like the idea is a bit like creating a streaming data computation graph using a simple syntax, is that correct?

I have a couple of questions for clarification:
2) Can you give an example where only a feedback loop can work? The reason I ask is because once you add cycles to a computation graph, things can get complex really fast!
3 & 4) A basic version of split/merge can already be achieved with objects: {'a': count, 'b': sum} | d('a') + d('b') - the split into several parallel pipelines happens when generating the object, and the merge happens when combining the keys. Were you imagining something more general?
5) I think this one is very approachable - it seems similar to interpolation, and can be directly implemented as for example an interpolate transform. Currently this is achieved with datasets https://heedy.org/analysis/datasets.html - I think this can be implemented without big changes to the current framework.
8&9) Seem similar to 3&4 - if I understand this correctly, whether a computation happens synchronously or asynchronously is a detail of implementation (i.e. the user should just define the computation graph, and the interpreter/VM/compiler should decide whether to do in parallel or sync/async)
11) I think something like this is already achieved in the map and while transforms (e.g. Pipe Args in https://heedy.org/analysis/pipescript/tutorial.html#advanced-pipes). Is that what you had in mind?
12) Pipes and transforms currently define a JSON Schema of their expected inputs and outputs, and Heedy timeseries also define schemas. While IIRC these are not auto-checked currently, I think the machinery already exists to do this in the future.
13) Error handling is an interesting question, for which I currently don't have a solution. Right now PipeScript throws an error and stops the moment a weird datapoint appears that can't be processed in the pipe. Not sure what the best way to approach this is.
15) Multiple separate streams are currently handled by the dataset generation machinery https://heedy.org/analysis/datasets.html, which uses a structure separate from PipeScript itself. Embedding the dataset generation in PipeScript is a really cool idea!
6,10,14,16) What do you mean by these, can you give an example of input/output you'd like to achieve with this?

[dumblob]

That looks very ambitious! Based on the points given, it seems like the idea is a bit like creating a streaming data computation graph using a simple syntax, is that correct?

Basically yes.

2. Can you give an example where only a feedback loop can work? The reason I ask is because once you add cycles to a computation graph, things can get complex really fast!

I want it to have general computational power. Any signal processing needs feedback, any simulation of digital HW (e.g. circuits from logic gates) needs feedback, etc.

I have some very shallow thoughts of how to handle the complexity you talk about. So I'm aware of certain edge cases (most of which should be "solved" by supporting both sync & async computation).

3 & 4) A basic version of split/merge can already be achieved with objects: {'a': count, 'b': sum} | d('a') + d('b') - the split into several parallel pipelines happens when generating the object, and the merge happens when combining the keys. Were you imagining something more general?

Actually yes. I'd like to achieve "partial ordering". Imagine you have an expensive computation you want to run on many cores but you have only one input. Then I'd like to have an option to use the input as a common queue for work-stealing - each core would steal the current sample/item (or batch of those for speed) from the queue. Assume that each core does a chain of operations (not only one) and that these operations might again need to "split" across several other cores (i.e. nested "splitting"). Then I'd like to assemble the output from all these cores again to one queue. Here I'd like to have 2 options - either assemble them as they come (in "random" order) or assemble them in the original order (e.g. based on nested tagging done when splitting).

5. I think this one is very approachable - it seems similar to interpolation, and can be directly implemented as for example an interpolate transform. Currently this is achieved with datasets https://heedy.org/analysis/datasets.html - I think this can be implemented without big changes to the current framework.

Ah, that's an interesting take on the feature. I was originally thinking more in discrete terms (i.e. no continuous/smooth function but rather a step function). And thus defining it as a relative measure to other streams (i.e. closer to how traffic shaping is being done - i.e. by delaying some samples based on given measure to achieve the desired "average" throughput).

8&9) Seem similar to 3&4 - if I understand this correctly, whether a computation happens synchronously or asynchronously is a detail of implementation (i.e. the user should just define the computation graph, and the interpreter/VM/compiler should decide whether to do in parallel or sync/async)

Actually I meant "true" synchronicity and asynchronicity as seen in physical world (though in real world there seems to be everything async if not talking about things like quantum entanglement etc., but usually we define things to happen synchronously if the measured difference of absolute time is less than x - it's easier if the time is discrete (a counter) like in CPUs). This seems one of the major differences from "usual" pipe languages.

I imagine that I could specify a set of subgraphs where each subgraph would either await each other to provide one new sample as a result of their computation (this is synchronous) or to not await each other to provide one new sample and just proceed further (this is asynchronous).

I have some very preliminary syntax ideas for this (unrelated to heedy and thus most probably incompatible with heedy). But the main point is the semantics as described.

11. I think something like this is already achieved in the map and while transforms (e.g. Pipe Args in https://heedy.org/analysis/pipescript/tutorial.html#advanced-pipes). Is that what you had in mind?

Yes. But I'd actually be fine even with more constrained variation - like requiring to know in compile time how the to-be-deleted or to-be-added graph will look like and only care about it's (de)instantiation in run time.

12. Pipes and transforms currently define a JSON Schema of their expected inputs and outputs, and Heedy timeseries also define schemas. While IIRC these are not auto-checked currently, I think the machinery already exists to do this in the future.

Ok.

13. Error handling is an interesting question, for which I currently don't have a solution. Right now PipeScript throws an error and stops the moment a weird datapoint appears that can't be processed in the pipe. Not sure what the best way to approach this is.

My idea was to forget completely about "errors being errors" but simply treat any error from a given instance of a command as yet another full-featured output queue (i.e. the command would be seen as multiple-output "subgraph").

15. Multiple separate streams are currently handled by the dataset generation machinery https://heedy.org/analysis/datasets.html, which uses a structure separate from PipeScript itself. Embedding the dataset generation in PipeScript is a really cool idea!

I didn't think of all the edge cases but my initial idea was more syntax-oriented than semantics. Because I see "streams of streams" as simply a "trunk" or "set" of streams but emphasizing that a command (disregarding whether source or sink or both (aka filter)) has the liberty to "open" and/or "close" any new or existing stream any time. In other words, this point merely holistically reiterates on many of the other points above 😉.

6,10,14,16) What do you mean by these, can you give an example of input/output you'd like to achieve with this?

ad 6) This is one of many potential (conceptual) solutions to the problem of outputting more than one stream from a command - merge of two streams of 64bit ints could e.g. result in a stream of structures containing two 64bit ints (and padding if needed).

Reshaping would then either add or remove or swap or rename "fields" (actually representing streams at one point in time) in such a structure. Reshaping shall ideally encompass both physical representation (imagine a C struct with fields and padding and alignment) and logical representation (names of fields, their order if any, value constraints over all fields together - please do not confuse with types as types do specify constraints only over each field separately and actually are usually very limited).

For synchronous computation, it's clear how to populate such struct. For async I'd simply take the currently newly computed value for a given field and then took old most recent values of all other fields to populate the struct. But again, this is just a crazy not very thought-through idea (it actually emerged from the need to allow to run all commands in parallel - therefore so much copying).

ad 10) Assuming our language will deliberately not specify how the streams will be implemented on inside, there has to be a way to read (as a source) & write (as a sink) data in a specific format. If we had reshaping as suggested in (6), this wouldn't be a problem as reshaping covers the physical representation (endianess, sizes, paddings, signedness, etc.). But as I wrote, it's just an idea and something else (better) might be used. In which case we'll need an explicit generic way how to e.g. read & write serialized data (e.g. in JSON or BSON or FlatBuffers or any other of those gazillions of existing & future formats).

ad 14) This is merely a note, that e.g. in case the graph(s) should be deployed as a distributed computing system (cores connected with some type of unreliable network), then there might be the need to specify which of the two commands with pipe in between shall (re)initiate the communication.

ad 16) This "heterogeneous or not" is partially a philosophical question, partially an implementation detail and partially an important decision affecting how the pipe language shall be structured 😉.

It's just another of the possible solutions to multiple outputs from one command (in additiona to the "struct" concept explained in the context of reshaping in (6) ). We could actually put samples representing different outputs into one stream but in a serial manner instead of "next to each other as described in (6)".

This way any command would first need to read the sample and dynamically in run time determine its "type" (incl. length/size, padding, etc.) because generally it wouldn't be easily possible to determine this with some clever analysis up front in compile time. This might have some performance influence - probably less copying than the "struct" approach but also more processing due to a "switch/branch" inside of nearly every single command. Again, this is not thought-through and all rather preliminary...

Any comments, ideas, thoughts very much welcome and appreciated!

[dkumor]

Overall this looks like the goals here are quite a bit more general than mine. PipeScript is currently used as an interpreted query language for heedy timeseries, where data is guaranteed to be in a specific format, and write operations don't matter, since the program output is the resulting stream itself.

I do think that expanding PipeScript to work over networks and read/write multiple formats of data would be really cool, as well as the idea of being able to open/close arbitrary streams of data.

That being said I, unfortunately, don't have the time to really explore these areas - and I am not sure golang would be the right language to approach the implementation of such a project! I think that adding language semantics for reading/combining multiple heedy timeseries might be something I add to PipeScript eventually if there are issues with the current dataset machinery, but I won't have the time until at least next summer to really dive deep into the code.

Since this repository is more geared towards the needs of heedy, I think that the approach described in your post would be best served by a fork or fully new backend, perhaps sharing a similar language semantics: https://github.com/heedy/pipescript/blob/master/parser.y