Subsystems or components that take as input the whole system?

[vspinu]

The following two patterns keep occurring in my design.

First is the monitoring or reporter functionality which is not strictly part of the system but needs access to the whole system in order to report its state. The way I imagine it currently is to have a nested system, the outer layer contains two components - monitoring and original system as a component. Or, alternatively a special keyword :integrant/system which would indicate that a components accepts the whole system as input (recursive dependency in a sense).

Second pattern is when an application consists of almost identical sub-systems which are loosely connected through some statistics channels. This again calls for some notion of nested systems.

I wonder what are your thoughts on this and if you intend to add some explicit provisions for nested systems in the future. Thanks!

[weavejester]

My solution to these sort of problems has not been nested systems, but functions for querying and transforming the configuration. In Duct, there's the concept of "modules" that provides exactly this.

My intent is to keep modules specific to Duct for now (more specifically, duct/core), as they're a more experimental feature. Once they're more proven, I'll consider adding some equivalent capability to Integrant itself.

[vspinu]

It seems to me that modules are essentially a way to merge second level map into root config with a custom transform :fn. At the moment I don't quite see how this helps with nested systems.

Let's say I have a system ::generic-subsystem and now I want to have two sub-systems that derive from it but differ in configuration:

{[::generic-subsystem ::sys1] {:name "AAA" ... }
 [::generic-subsystem ::sys2] {:name "BBB" ... }

If I understand correctly if I use modules for this task it would merge ::sys1 and ::sys2 into root config resulting into collisions.

What I am really looking for is a simple mechanism to mark a component as a system and then be able to reference components of sub-system A from sub-system B. First part is easy through derived keywords:

(defmethod ig/init-key ::system [_ m]
  (ig/init m))

(defmethod ig/halt-key! ::system [_ s]
  (ig/halt! s))

;; would be still nicer to have a reader `#ig/system` for this

The second part is a bit trickier but it looks to me that allowing ig/ref to accept vectors of keys for nested references is small and straightforward extension of the current mechanism.

In a nutshell:

{:sys1 #ig/system {:comp-a {:param-a 1}
                   :comp-b {:param-b 2}}
 
 :sys2 #ig/system {:comp-c #ig/ref [:sys1 :comp-b]}
 
 :comp-y {:param-y #ig/ref [:sys1 :comp-b]}}

[vspinu]

Nested systems could also be used for grouping similar components into easy-to-reference packs:

{::reporters #ig/system {:my-app.reporters/console {}
                         :my-app.reporters/influx  {:host "localhost"
                                                    :port 8086}
                         ...}
 ::monitor   {:reporters (ig/ref ::reporters)}}

Instead of the flat and redundant:

{
 :my-app.reporters/console {}
 :my-app.reporters/influx  {:host "localhost"
                            :port 8086}
 ...
 ::monitor {:reporters {:console (ig/ref :my-app.reporters/console)
                        :influx  (ig/ref :my-app.reporters/influx)
                        ...}}
 }

[weavejester]

You're describing a solution, rather than the problem you want to solve. Why do you think you need nested systems? We need to put together a clear use-case first, then we can work through potential solutions. Can you describe in more detail one of the monitoring/statistics systems you mentioned previously?

Modules are pure functions that transform the configuration. They're not just ways of merging maps into the configuration; they're far more flexible than that. Any solution we can come up with can be replicated using modules. The question is not so much "can we use modules for this", because the answer is always "yes", but rather "should we use modules for this, or introduce something new"?

[vspinu]

I have a generic system (trading app) which I want to use as a component in a bigger system. Each such component is a replication of the generic trading app for different exchanges/currency pairs. All these app publish statistics to one common core/async channel (separate component).

I just need a intermediate unit between system and component, a grouping of components which I can move around, instantiate and replicate at will. It's a rather obvious modularity requirement - recursively build components as if they are systems.

The monitoring part is as follows. My system is a bunch of components represented by core.async channels. I want to have a separate monitoring component which is able to install listeners on selected core.async channels and report appropriately. Thus, my monitoring component needs to take as an input the whole system after it has been built by integrant. Then inspect each node and keep only core async channels. And finally hook on them and start reporting. I just don't see how this could be done with a flat map besides listing each channel as a dependency to the reporter.

BTW, the concern of #20 is somewhat related. If I have a watch-dog component which monitors a sub-system, then having a notion of a subsystem opens a door for watch-dogs which could alter the system by restarting parts of the syb-system. For instance, if a sub-system is be stored as an atom inside the parent system. Does this make a bit of sense?

[weavejester]

Thanks for the more detailed explanation. I believe I can see where you're coming from. I'm not currently sure what would be the best way to handle this, but let me put forward my thoughts so far. Bear in mind that these are little more than rough notes so far; this isn't intended to be a complete solution.

When it comes to nested maps, there's always a way to transform a nested map into a flat map or vice versa. For example:

{:a {:b 1
     :c {:d 2
         :e 3}}

Could be represented as:

{[:a :b]    1
 [:a :c :d] 2
 [:a :c :e] 3}

Similarly, a nested set of system maps can always be represented as a flat system map. Whether to use nested systems or something else is effectively a question of presentation.

If subsystems are kept in atoms, then the situation changes. I think we need to distinguish between what is a true subsystem, and what is merely a grouping of keys. To my mind, a true subsystem has its own lifecycle. I'm not sure whether this is a good idea or not.

If we're just talking about a grouping of keys, we can achieve that with composite keys. Consider a configuration like:

{:database/sql  {:uri "jdbc:sqlite:db/usd.sqlite"}
 :handler/trade {:currency :usd, :db #ig/ref :database/sql}}

We can both group and make the keys unique using composites:

{[:group/usd :database/sql]  {:uri "jdbc:sqlite:db/usd.sqlite"}
 [:group/usd :handler/trade] {:currency :usd, :db #ig/ref [:group/usd :database/sql]}}

Currently this won't work in Integrant, because references can't be vectors. However, since we have composite keys, having composite references seems like a logical and consistent extension to the syntax.

This syntax is somewhat verbose, but it shouldn't be too hard to come up with some syntax sugar. Perhaps:

{[:group/usd :_]
 {:database/sql  {:uri "jdbc:sqlite:db/usd.sqlite"}
  :handler/trade {:currency :usd, :db #ig/ref :database/sql}}}

In any case, we preserve the flat structure of the configuration map. A flat map is simpler to reason about and allows for easier cross-referencing.

You might well point out that merely grouping keys doesn't solve the problem of repetition. Suppose we want to have several trading systems, each passing data to a central monitor. We might write:

{:global/monitor {}

 [:group/usd :database/sql]
 {:uri "jdbc:sqlite:db/usd.sqlite"}
 [:group/usd :handler/trade]
 {:currency :usd, :monitor #ig/ref :global/monitor, :db #ig/ref [:group/usd :database/sql]}

 [:group/eur :database/sql]
 {:uri "jdbc:sqlite:db/eur.sqlite"}
 [:group/eur :handler/trade]
 {:currency :eur, :monitor #ig/ref :global/monitor, :db #ig/ref [:group/eur :database/sql]}}

Not very concise, but this is where modules come in. Modules in Duct are pure functions that transform the configuration, so any repetition can be factored out:

{:global/monitor {}
 [:group/usd :module/trade] {:currency :usd}
 [:group/eur :module/trade] {:currency :eur}}

Anyway, this is all very rough and by no means a firm solution. However, my current inclination is to avoid nesting subsystems, as it seems like something that could rapidly get complex. Keeping the system a single block, and instead using pure functions and syntax in the configuration feels conceptually simpler, even if the syntax currently feels rather rough.

[vspinu]

Thanks for having a thorough take on this.

When it comes to nested maps, there's always a way to transform a nested map into a flat map or vice versa.

Sure, but it might lead to incompatible semantics as integrant already uses composite keys as a synonym for key inheritance.

I think we need to distinguish between what is a true subsystem, and what is merely a grouping of keys.

I see grouping of keys as a sub-system where components don't necessarily depend on each other. If sub-systems can do grouping why to invent a separate concept for those? I see atom-sub-systems just as an optional feature for cases when components in the parent system need to manage the life cycle of the sub-system. For most cases imutable sub-systems should be just fine.

To my mind, a true subsystem has its own lifecycle. I'm not sure whether this is a good idea or not.

Component, systems and subsystems all have lifecycles and the distinction between them strikes me as being a bit artificial. I think it makes sense to reduce them to a single notion and allow each component to accept both other components and references to other compoenents.

Enforcing components to accept only references leads to unnecessary repetitions. If a component A depends on B and no other component depends on B why am I forced to declare it in the root config and need to duplicate the key in the component A?

We can both group and make the keys unique using composites:

{[:group/usd :database/sql] {:uri "jdbc:sqlite:db/usd.sqlite"}
[:group/usd :handler/trade] {:currency :usd, :db #ig/ref [:group/usd :database/sql]}}

However, since we have composite keys, having composite references seems like a logical and consistent extension to the syntax.

It should then handle grouping key (ig/ref [:group/usd]) as a collection of all keys with that marker. Not sure if this fits into current semantics though. Likely not.

From the user prospective composite keys are less natural than nested maps. From programmer prospective (get config [:group/usd :handler/trader]) is the same as (get-in config [:group/usd :handler/trader]). So IMO there is no real gain by operating on a flat map.

In any case, we preserve the flat structure of the configuration map. A flat map is simpler to reason about and allows for easier cross-referencing.

Easier to reason for the user or the integrant developer? I think that the main aim of integrant (component, mount etc) is to allow each component to access its resources through a flat map. And this is where integrant shines. But it seems to me that flat config map has little usefulness to the end user. Quite the opposite actually. It forces system designer to bent natural 3D designs into a flat 2D paper.

I am afraid transformers are also quite complex beasts actually. The user writes an essentially hierarchical config which is then transformed under the hood into a flat map by who knows which rules and then into system which doesn't resemble the original. So each time I want to understand what a module does, I need to look at the :fn transformer and figure out what extra cookies it adds to the standard integrant semantics.

Also, with arbitrary transformers standardized tooling ecosystem for integrant would be hard to implement. My primary concern is with visualization. If I want to visualize a system I can either make a graph of the config or the instantiated system. Currently both graphs are the same and this is just great. With modules as transformers I won't be able to visualize satisfactory either of these two. Config with modules is a truncated version of the system and there is no way to infer how the modules look without actually running them. The instantiated system is a transformed flat sausage which might or might not resemble the original configuration.

This is what I have right now as a working system:

(def config
  {::exchanges {:ex1 #'config-ex1
                :ex2 #'config-ex2}
   ::reporters {:reporters/console {}                 
                :reporters/influx  {:host "localhost"
                                    :port 8086}}
   :monitor    {:systems   (ig/ref ::exchanges)
                :reporters (ig/ref ::reporters)}})

Where ::exchanges is a map of vars with config of the sub-systems :ex1 and :ex2; ::reporters is a grouping sub-system with components :reporters/console and :reporters/influx.

This example ilustrates 3 concepts. First ::exchanges is a map of sub-systems - a grouping. Second ::reporters is a sub-system which acts as a grouping of homogenous components. Finally, monitor is a stand alone component which reports the status of ::exchanges using reporters. Note that monitor here is not the same as monitor in your example. My sub-systems are not aware of the monitor; the monitor is an external watch dog which knows how to monitor the systems because of their regular design (core.async channels).

The lifesycle of ::systems and ::system are:

(defmethod ig/init-key ::system [_ var]
  (ig/init var))
(defmethod ig/halt-key! ::system [_ s]
  (ig/halt! s))

(defmethod ig/init-key ::systems [_ m]
  (map-vals #(ig/init @%) m))
(defmethod ig/halt-key! ::systems [_ m]
  (map-vals ig/halt! m))

[weavejester]

Sure, but it might lead to incompatible semantics as integrant already uses composite keys as a synonym for key inheritance.

There's no incompatibility here; inheritance is just a way of grouping keys. We're essentially saying "this subsystem is defined by all keys derived from X".

Component, systems and subsystems all have lifecycles and the distinction between them strikes me as being a bit artificial.

The difference is whether their lifecycle is independent. For instance, if a subsystem is defined in an atom, we could halt! and re-init it without bringing the containing system down. This differs from components as they exist in Integrant currently, as their lifecycle is tied to that of the system they're in.

It should then handle grouping key (ig/ref [:group/usd]) as a collection of all keys with that marker. Not sure if this fits into current semantics though. Likely not.

I think we'd want a separate tag for that. Maybe #ig/refset.

So IMO there is no real gain by operating on a flat map.

The more we restrict a data structure, the more we can reason about it. For instance, let's say you're asked "What components are in the system?" With a flat map, the answer is keys. With a nested structure, we need to walk the tree.

If we're using tags, like #ig/system, then we probably need to introduce a new record type to represent a subsystem. Likewise we need new syntax for nested references. I think it's worth asking whether this is necessary, or if we can use the existing tools at our disposal instead.

We already have composites, which allow grouping of keys. If that's not enough we can consider introducing something new, but I'd like to see what we can do with minimal changes first.

I am afraid transformers are also quite complex beasts actually.

Right, that's definitely a concern. Modules do have the advantage of being pure, so we can always run them and see what they produce, but too much transformation and it becomes hard to reason about the configuration.

I'd certainly like to see something a little more restrictive and easier to reason about than transformation functions.

Note that monitor here is not the same as monitor in your example. My sub-systems are not aware of the monitor; the monitor is an external watch dog which knows how to monitor the systems because of their regular design (core.async channels).

This is the sort of design I want to avoid, as I'd rather query the configuration than the system. My reasoning is that the more information the configuration provides, the more we can understand the system without needing to initiate it. For example, we can ask programmatically "What components are being monitored?" without needing a running system.

Integrant is designed to have as little to do with the running system as possible. The more we focus on the configuration, the more we can do using only pure functions before the system is initiated.

Would it be possible for you to give a little info on #'config-ex1 and #'config-ex2? Or else could you put together some example configuration that has the gist of your application, without revealing any proprietary parts?

[vspinu]

My reasoning is that the more information the configuration provides, the more we can understand the system without needing to initiate it.

Very much agreed, but I would like to be able to monitor the running system as well. A graph of the system config is cool, but being able to interactively explore a graph of the running system is much more exiting. This is why having identical config's and system's topology would be quite useful.

Would it be possible for you to give a little info on #'config-ex1 and #'config-ex2? Or else could you put together some example configuration that has the gist of your application, without revealing any proprietary parts?

Those are systems which listen to a web-socket, do some data processing, compute statistics and perform algoritmic trading but I don't think there is anything special about those; any other system listening to an external event stream would do. I cannot share much of it right now because it involves quite some hackery on top of core.async and is very experimental at the moment, but I don't think it matters much. The idea is to be able to easily pick a system or a complex component with its own lifecycle and be able to plug it directly into your system without much ado.

One important thing which I currently cannot do within integrant's framework is to share statistics between those two sub-systems. Allowing for that would probably involve quite some core changes to integrant and introduction of nested references syntax as you pointed out.

I think I will go ahead and do a bit of hackery on top of integrant myself to try this system==subsystem==component idea which I am currently quite exited about. Will be back with it if it proves worthwhile.

[weavejester]

Very much agreed, but I would like to be able to monitor the running system as well.

Sure, but it seems to me that there are two ways of doing this.

1. We can make components adhere to a monitorable interface, and then at runtime have a monitor search the system for these components.

2. We can make components accept a monitor option, and then at compile-time give all the components we want to monitor a reference to the monitor component.

Both these approaches are equivalent in functionality; we could use either approach and get back the same information. However, to my mind option 2 is the better design, because it allows us to know at compile time what is being monitored, and it more cleanly separates functional and side-effectful code.

I think I will go ahead and do a bit of hackery on top of integrant myself to try this system==subsystem==component idea which I am currently quite exited about. Will be back with it if it proves worthwhile.

Good luck! I think I'm going to give this problem some more hammock-time as well. I'd ideally want a solution where the dependencies between components are explicit at compile-time.

[vspinu]

to my mind option 2 is the better design,

Yerh. I just figured that the hard way. I just implemented option 1 thinking that it will save me loads of time later but no, even before being able to monitor anything I started running into huge design issues. Global monitor will never be flexible enough and I would need to also either declare/implement a bunch of stuff at component level or specify those for each component separately in the monitor itself. So I am moving full speed from strategy 1 to 2 right now. Thanks for the input!

[vspinu]

Hi James,

I am back with commix. It solves all my issues and I am reasonably confident that it should be able to provide an elegant solution for duct modules as well.

I have added a wiki page on differences with Integrant for the motivation.

At this stage I am almost happy with it except of one conceptual inconsistency with lifecycle method dispatch which I am still brewing. I will likely not preserve [k v] dispatch as there are more useful things to pass as first argument there.

[weavejester]

Thanks for providing an alternative approach. It goes in a different direction to where I want to take Integrant, but that's the benefit of forks and alternative designs :)

It doesn't seem powerful enough to act as a solution for Duct modules, however. Duct modules are Turing-complete transformations of the configuration; they're not merely groupings or subsystems.

I'll keep this issue open, in case I find another solution that fits in more with my plans for Integrant.

[vspinu]

Duct modules are Turing-complete transformations of the configuration

Do they really need to be transformations? Do you already have an example of a duct module which couldn't be accomplished with a sub-system?

[weavejester]

Sure. A lot of modules involve searching and transforming the configuration. One example is the :duct.module/ataraxy module. A configuration like:

{:duct.core/project-ns foo
 :duct.module/ataraxy {["/bar/" id] [:bar id]}}

Will be transformed into:

{:duct.core/project-ns foo
 :duct.module/ataraxy {["/bar/" id] [:bar id]}
 :duct.router/ataraxy
 {:routes   {["/bar/" id] [:bar id]}
  :handlers {:bar #ig/ref :foo.handler/bar}}}

Note that the result key :bar and the project namespace foo are used to produce a reference :foo.handler/bar. This module needs to be able to parse Ataraxy's routing syntax, derive a collection of results, then perform string concatenation to produce the handler keys. My understanding is that commix subsystems aren't capable of this.

There are other examples that are more based around searching and making decisions based on what already exists in the configuration. For instance, the :duct.module/web module searches the configuration for a web server (i.e. a key that derives from :duct.server/http), and if such a key is missing, a default web server is added.