Application Concept

[hyperthunk]

OTP is heavily dependent on the concept of an application: a top level process that launches the root supervisor and provides one or two other goodies.

Applications are a central unit of reuse, control (e.g., initialisation, upgrade) and dependency in OTP. I suspect we'd do well to borrow this idiom.

What might we do differently in CH to OTP? Can we improve in that model (which isn't without it's flaws and was designed decades ago) and what kind of feature set do we want?

[hyperthunk]

Some of the features OTP applications provide are:

• standardised configuration
• dependencies (between applications)
• loading/starting applications (see below)
• shutdown handling
• distributed applications

The application concept in OTP is quite simple. When you start an application, the application resource file is loaded and any dependent applications need to be started if they're not already running. The application is started by the application controller, which calls a behaviour (callback module) for your application (defined in the .app resource file) that in turn is supposed to return the pid of a fully initialised top-level supervisor.

In practise things are a bit more complex typically. Putting aside OTP features that aren't relevant to CH right now (such as releases and hot code upgrade) there are other parts of starting up an application that bear some consideration. Application's support the concept of start phases (and stop phases too), which allow us to initialise e.g., certain subsystems one at a time. These boot steps can depend on one another and define behaviours that need to be run successfully in order to complete. Sometimes a boot step might return some intermediate response that needs to be provided to subsequent steps.

In OTP you need to either explicitly start dependent applications in your code or build a release which creates a boot script that does this (in order) for you. In OTP applications can include other applications, which is a way of saying that you want to start the top-level supervisor for that application yourself and put it beneath one of your own supervisors. One way of looking at this is to see the included application as providing a sub-system, for example a web server process might be just one subsystem within the overall application hierarchy.

OTP applications have start-types which define how the runtime system should behave if the application fails (i.e., the top level supervisor crashes). You might want to

• shut down
• restart
• do either just for you application
• or for all dependent applications
• or for all applications on the right as it were

So - I think that the application concept can be worked into quite a clean model for CH. Without thinking about distributed (yet) we might extend the OTP definitions up a bit like so.

• an application is a kind of (glorified) root supervision tree
• all applications have a single top-level supervisor process which is managed for them (by the platform)
• an application must define (e.g., export and specify in configuration) one user-defined top-level supervisor
• the top level process might launch this supervisor when the application is started
• an application might want to use functionality provided by other applications
• to use other applications the application specifies them as dependencies - these are referred to as sub-systems
• application dependencies can be started and installed into the top level (managed) supervisor process automatically
• application dependencies can be started (and installed) into any supervision tree within another application or sub-system

That's ignoring boot steps, but it does address the issue of application startup/restart/shutdown policy, simply by defining applications in terms of supervision trees.

The assumption at the moment is that the absolute top level application that starts on a node is the application controller application. This in turn will treat the applications loaded underneath it as sub-systems, even though they appear to be top-level (as they are allowed to be independent of one another).

[hyperthunk]

Nice-2-have stuff that OTP doesn't support:

• distributed applications that aren't just about fail-over
• parallel startup
• parallel distributed startup
• more than one mode of start/stop synchronisation

[hyperthunk]

See also https://github.com/jepst/distributed-process-global

[ericbmerritt]

@hyperthunk I think for clarity we need to boil the concept of an Application down to its most basic definition, then we need to layer the higher level concept of the Release Handler on top of it. There are a lot of special cases that come up once you have this basic application support in place (start phases as you mention, are one of them). However, for the first most basic implementation all of that can be ignored. I am also going to ignore the tooling Erlang has to support releases. I am just going to talk about Applications and Releases in terms of the somewhat concreatized concept not Erlang`s implementation of that concept. Its not that we don't need these things, just that they are built on top of this functionality. So for version 0.0.0.1 its just a distraction.

Releases

In Release is simply a list of Applications that work together to provide a specific functionality. That functionality is usually at the level of a system. This list of Applications is usually described a file that the Release Handler uses to start each Application.

** Note: ** The concept of a Release is what most other systems would call an Application. Unfortunately, in Erlang the concept of an Application is overridden. So we use the term Release. For example the 'Riak' Release specifies a set of OTP Applications that work together to provide a No SQL database. In the same vein, the 'ejabberd' Release is a set of OTP Applications that work together to provide a Chat Server.

Release Handler

The Release Handler is a system process that is responsible for starting and stoping all the Applications described in the Release file in an orderly way. It uses metadata 'exported' by the Applications to figure out what order the Applications should be started in and what modules/functions should be used to start the Application. Once all Applications are started, it then monitors those Applications. If one of those Applications stops for some reason, it attempts to shut down all applications in the Release. It also reacts to external events asking for a shutdown by also attempting to shut down all Applications in the Release.

Applications

An Application, in the OTP sense, is a unit of functionality that is expected to provide a fine grained, long lived set of 'services' to the system. It includes the definition of those long lived services, related support code, and metadata that tells the release handler how to start the Application and what prerequisites must exist for the Application to be started successfully.

** Note 1: ** In Erlang there are Active Applications and Library Applications. That is because, in Erlang, Applications provide both the concept of a Library and a living set of Services. However, Haskell already has a very good module system that completely covers the idea of a 'Library Application' as expressed in Erlang. So Library Applications can be more or less ignored here.

** Note 2: ** I am completely ignoring supervision trees. From a Release Handling/Application Lifecycle perspective they don't matter one bit.

Lifecycle

The Application behavior supports two fundamental lifecycle operations, start and stop.

Start

The Application must export a start function that is responsible for starting a long running process and returning that process id. This process is expected to be long lived. If the process stops for some reason, that is considered an failure and the release handler begins shutting down the entire release.

In OTP Applications, the process id returned is always a Supervisor.

Stop

The Application must export a stop function that is responsible for gracefully stopping the Application.

Metadata

The metadata (in Erlang) is simply a configuration file in a well defined format and a well defined location that the Application uses to describe to the Release Handler how to start and stop it, and what other Applications need to be already running be fore this application starts.

Wrapping Up

This is a really stripped down description of Applications and Releases. However, it is a complete description and, if implemented' would provide a base on which to build the more complex additional features you describe above.

[ericbmerritt]

To comment on my own issues. There is no reason not to create the Release Handler that takes a list of App descriptions as its argument. Doing that you leverage the compiler to handle versioning and you just worry about the mechanics of starting the applications in the right order. With that App descriptions are just records. That way you let the compiler do all the checking for you on the App metadata. If the part of the metadata is the relevant start and stop functions you remove any need at all to do any discovery.

[hyperthunk]

Yes I completely agree that we should leverage the compiler as much as possible. I think we should make some decisions about whether or not leaving it up to the user to return a supervisor pid is the best way to manage an 'application' though.

I also think that you've made a good point about the terminology here: I don't like using application to describe what is a component when you're developing it, yet is (or forms part of) a sub-system when it's being used at runtime. I also think release is not a good word to use for the overall system, as that's overloaded too.

[ericbmerritt]

Releases are not about managing an application at all. They are about providing the application lifecycle. That is starting up and shutting down. Application management is what the supervision structure it is for. I think its somewhat important to keep those concepts distinct.

Granted, 99.99% of the time people are going to use supervisors and thats absolutely what we should encourage in all the documentation. If we place that restriction on the apps it really will be an arbitrary restriction, one we have to go out of our way to enforce. So we both add some (slight) complexity and reduce the potential developers ability to try both new and interesting things, like a new, better implementation of the concept of application supervision.

I agree with you on the terms. Coming up with new ones is going to be painful though from a thinking up useful names perspective.

On a side note. I realize that all we have to worry about is lifecycle here. Cabal actually provides all of the other features of a standard OTP Application. We should encourage one app per cabal package and then just let cabal manage the rest.