-
Notifications
You must be signed in to change notification settings - Fork 125
Cuttlefish for Erlang Developers
Note: If you are looking for information in how to "wire" Cuttlefish to your application, consult Cuttlefish for node_package users or Cuttlefish for non-node_package users.
As an Erlang developer, you're probably used to application:set_env
and app.config files. The good news for you is that you can keep on
coding that way! The... additional news is that people who are using
your application may not understand that syntax so easily. So how can
you help? I'm glad you asked!
You have a new job. You get to choose which knobs you expose to users! You can choose to name these things anything you want, so where you previously might have been confined to including a dependency's application name, you are now not.
You can define datatypes for these settings, and you can explain to Cuttlefish how a simple name-value pair becomes part of a complex hierarchy of Erlang terms!
As the Erlang developer, you are the person responsible for being the ambassador of your setting to the world. Here's how it looks in ASCII art:
┌--------------------┐ ┌--------------------┐
| <app>.conf | | app.config |
|--------------------| {mapping, {translation, | -------------------|
| my.setting = value | ---> "my.setting", ---> "<dependency>.setting", ---> | [{<dependency>, [ |
| ... | "<dependency>.setting", fun(Conf) -> | {setting, value}|
| | []}. %% erlang fun | ]}, ... ]. |
└--------------------┘ end}. └--------------------┘
There are three types of Schema elements in Cuttlefish: mapping, translation
and validator. It's easy to tell the difference! They're all tuples, and the
first element is an atom: mapping, translation, or validator. You're welcome.
Mappings are the only schema element type that support annotations and there are
two available: @doc and @see.
@doc: If you write a multiline @doc it will be included in
your generated .conf file. These docs will be available
programatically. We chose to make it an annotation because as
Erlangers, you already know and love @doc AND we didn't want you to
worry about multiline strings and an array of strings as a member of a
proplist. This just seemed cleaner.
@see: If you already wrote an @doc for a setting, but you were describing
how multiple settings work together, you can just specify an @see to that
mappings name. It's just a pointer to additional documentation.
%% @doc 'a.x' and 'a.y' set the start coordinate
{mapping, "a.x", "app.a.x", []}.
%% @see a.x
{mapping, "a.y", "app.a.y", []}.Note: Cuttlefish has a bit of fun with these when generating your default
.conf file. It will include the documentation inline if you have a @doc.
If you also have a @see, it will include references to those other settings.
If it only has an @see, it will include the @doc of the mapping you
specified in the @see.
Aside from documentation, there is plenty going on with mappings, but the basic form is as follows:
%% {mapping, string(), string(), proplist()}
Mapping = {mapping, ConfKey, ErlangConfMapping, Attributes}.element(1, Mapping) = mapping-
element(2, Mapping) = ConfKey- the string key that you want this setting to have in the.conffile -
element(3, Mapping) = ErlangConfMapping- the nested location of the thing in theapp.configthat this field represents -
element(4, Mapping) = Attributes- other helpful things we'll go into right... about... now!
Attributes is a proplist, and let's assume you know how those work. Here are the keys in that proplist that we work with:
-
default- This is the default value for the setting. When a.conffile is generated, a line will be added to set the setting to this value. Additionally, if no value is specified in the.conffile, the value specified in thedefaultattribute will be used when generating theapp.configfile. -
new_conf_value- If this is defined, then when you generate a.conffile, this value will be used for the default setting. This causes thedefaultattribute to be overridden when generating a.conffile, but has no effect on what default value is used when generatingapp.config. -
commented- If this is defined, then when you generate a.conffile the documentation for this setting appears, along with the setting, but the setting is commented out and that comment includes this value. -
datatype- This is the datatype for the field. For a list of those, check Datatypes. -
hidden- If this atom is present or set to true, this value will be in the generatedapp.config, but not the generated.conffile. It still can be overridden in the.conffile, you just have to know about it. It's a way of adding "undocumented knobs". -
include_default- If there is a substitutable value in the ConfKey, in the generated.conffile, this value is substituted. (don't worry if that last one didn't make so much sense now, I'll explain more below) -
validators- the list of names for validators for this mapping. -
merge- is the only thing included just as an atom. It specifies that this is supplemental to an existing mapping, not a replacement mapping. The default behavior is to replace an existing mapping with this newer one
The best way to get it, is to take a look at some examples. Let's start with Riak's ring_size.
%% example of super basic mapping
%% @doc Default ring creation size. Make sure it is a power of 2,
%% e.g. 16, 32, 64, 128, 256, 512 etc
{mapping, "ring_size", "riak_core.ring_creation_size", [
{datatype, integer},
{commented, 64}
]}.First of all, comments before the @doc annotation are ignored, so
feel free to put Schema specific comments in here as you see fit.
Everything after the @doc in the comments, is part of the
documentation. Cuttlefish will treat this documentation as:
[
"Default ring creation size. Make sure it is a power of 2,",
"e.g. 16, 32, 64, 128, 256, 512 etc"
].Then, we can also see from element(1) that this is a mapping.
element(2) says that it's represented by "ring_size" in the
riak.conf file. element(3) says that it's
"riak_core.ring_creation_size" in the app.config. We also know from
the attibutes that it is an integer, and that it will appear in the
generated riak.conf file with a value of 64. It just so happens that
the default is also 64, but that's specified in riak_core's app.src.
Let's talk about element(3) here for a minute. What that means is
that there's an app.config out there that looks like this:
[
{riak_core, [
{ring_creation_size, X}
]}
].and that we're concerned with X.
Here at cuttlefish, one thing we're really disturbed by is this idea of
a "magic number". A magic number is a setting that we're not actually
sure where the default value comes from. In the above example it comes
from riak_core's app.src file, but they can come from anywhere, even
the dreaded application:get_env/3.
We encourage every mapping in a cuttlefish schema to set an explicit default. In that case your application has a single location for default values, and it generates a complete app.config.
Now, if life were as simple as 1:1 mappings like this, we'd be done. But it's not, and so we need to introduce translations.
Actually, they're pretty easy.
A translation looks like this:
%% {translation, string(), fun((proplist()) -> term())}
Translation = {translation, ErlangConfMapping, TranslationFunction}.Let's break it down:
element(1, Translation) = translation-
element(2, Translation) = ErlangConfMappingthis is the same as the corresponding ErlangConfMapping in the mapping above. This is how we tie back to a mapping -
element(2, Translation) = TranslationFunctionthis is a fun() that takes in a proplist representing the.conffile and returns an erlang term. This erlang term will be the value of the ErlangConfMapping.
Ok, that does sound more confusing than it should. Let's take a look at one, you'll like it better in practice.
%% @doc How Riak will repair out-of-sync keys. Some features require
%% this to be set to 'active', including search.
%%
%% * active: out-of-sync keys will be repaired in the background
%% * passive: out-of-sync keys are only repaired on read
%% * active-debug: like active, but outputs verbose debugging
%% information
{mapping, "anti_entropy", "riak_kv.anti_entropy", [
{datatype, {enum, [active, passive, 'active-debug']}},
{default, active}
]}.
{translation,
"riak_kv.anti_entropy",
fun(Conf) ->
Setting = cuttlefish:conf_get("anti_entropy", Conf),
case Setting of
active -> {on, []};
'active-debug' -> {on, [debug]};
passive -> {off, []};
_Default -> {on, []}
end
end
}.See what's happening? First of all, you need a mapping. If you don't
have one, don't bother writing a translation for it. It will not get run.
It will also not get run if you have a mapping for it, but you don't have a value for it. That value can be a default in the mapping or set in your .conf file. The mapping we defined for "anti_entropy" says that it's an enum with values "active",
"passive", and "active-debug". The configuration in the app.config is more
complicated. Basically, it works like this:
- active - {on, []}
- passive - {off, []}
- active-debug - {on, [debug]}
It's a relatively simple translation, but we want to spare non-Erlangers from this very Erlangy syntax. So, we give them the values "active", "passive", and "active-debug" and the translation "translates" (not just a clever nickname!) them into the erlang value we expect.
You may have noticed that the translation fun takes an argument Conf. Conf is a cool proplist and needs to get its "props". Here are some fun facts about Conf:
By the time Conf gets to this function, a lot has happened since it was a .conf file. First of all, if you omitted a variable in your .conf file, but it had a default, that default value is in Conf.
Conf is a proplist, but its keys are no longer in the form "a.b.c". For "easier" erlang processing (pattern matching, etc), the keys come in the form ["a", "b", "c"]. The values are also different. As you might imagine, in the .conf file, they're all just strings. If you specified a datatype in the mapping, it will be transformed into that erlang type by the time it gets here.
There are convenience functions in the cuttlefish module for accessing these. They allow you to pass variables in with either "a.b.c" or ["a", "b", "c"] notation. For more on that, see the cuttlefish_variable module.
See the Cuttlefish Developer's API page for more.
There are other cases when multiple values turn into a single
app.config complex data structure. Take lager as an example.
%% complex lager example
%% @doc location of the console log
{mapping, "log.console.file", "lager.handlers", [
{default, "./log/console.log"}
]}.
%% *gasp* notice the same @mapping!
%% @doc location of the error log
{mapping, "log.error.file", "lager.handlers", [
{default, "./log/error.log"}
]}.
%% *gasp* notice the same @mapping!
%% @doc turn on syslog
{mapping, "log.syslog", "lager.handlers", [
{default, off},
{datatype, enum},
{enum, [on, off]}
]}.
{ translation,
"lager.handlers",
fun(Conf) ->
SyslogHandler = case cuttlefish:conf_get("log.syslog", Conf) of
on -> {lager_syslog_backend, ["riak", daemon, info]};
_ -> undefined
end,
ErrorHandler = case cuttlefish:conf_get("log.error.file", Conf) of
undefined -> undefined;
ErrorFilename -> {lager_file_backend, [{file, ErrorFilename}, {level, error}]}
end,
ConsoleHandler = case cuttlefish:conf_get("log.console.file", Conf) of
undefined -> undefined;
ConsoleFilename -> {lager_file_backend, [{file, ConsoleFilename}, {level, info}]}
end,
lists:filter(fun(X) -> X =/= undefined end, [SyslogHandler, ErrorHandler, ConsoleHandler])
end
}.We define three mappings here, that have different values in the
riak.conf file, but represent a complex list of lager handlers in
the app.config. The solution is to have them all map to the same
ErlangConfMapping, which references lager.handlers. When we create a
translation for that, we're basically saying that "The return value of
this function will be the value of {lager, [{handers, X}]}". that was
a weird way of saying it, but the generated app.config looks like
this:
{lager,
[
{handlers,
[{lager_syslog_backend,["riak",daemon,info]},
{lager_file_backend,[{file,"/var/log/error.log"},{level,error}]},
{lager_file_backend,[{file,"/var/log/console.log"},{level,info}]}]},
]},Sometimes you'll find yourself needing to map elements of a list or proplist. Consider the way we configure HTTP listeners for Riak.
{riak_core,
[
{http,
[
{"127.0.0.1",8098},
{"10.0.0.1",80}
]
}
]
}We got really aggressive with the line breaks here, to illustrate that
riak_core.http is a list of {IP, Port} tuples. Now, say for some
reason, you wanted 10 of these listeners. We're not here to judge you,
we're here to help. What we didn't want to do was introduce some kind
of list data structure on the right hand side of our .conf file.
Instead we took a "list element per line" approach. We wanted to give
you a syntax that was something like this:
listener.http.internal = 127.0.0.1:8098
listener.http.external = 10.0.0.1:80But wait, what's the deal with this "internal"/"external" business? Well, the mapping is defined with a wildcard. Think of it like a match group in a regex.
%% HTTP Listeners
%% @doc listener.http.<name> is an IP address and TCP port that the Riak
%% HTTP interface will bind.
{mapping, "listener.http.$name", "riak_api.http", [
{default, {"127.0.0.1", 8098}},
{datatype, ip},
{include_default, "internal"}
]}.
{translation,
"riak_api.http",
fun(Conf) ->
HTTP = cuttlefish_variable:filter_by_prefix("listener.http", Conf),
[ IP || {_, IP} <- HTTP]
end
}.See the $name? it can be anything! Then the translation is "smart"
enough to parse all the listner.http.* config keys and create the list
of {IP, Port}s for the "riak_core.http" section. (TODO: in the future,
we'll add the ability to refer back to $name as a variable, but for
now, we didn't need to because in this case, name was a throwaway).
Also, notice the {datatype, ip}, that is smart enough to turn
"IP:Port" into {IP, Port}. Don't worry, it works for IPv6 too. More on
Datatypes.
This is the perfect place to talk about 'include_default'. If there's
a wildcard in the ConfKey, we don't want to include that wildcard in
the default generated .conf file, so we need an example. The value
from include_default provides that sample. So, the generated .conf
looks like this:
## listener.http.<name> is an IP address and TCP port that the Riak
## HTTP interface will bind.
listener.http.internal = 127.0.0.1:8098What's in a $name? That which we call internal by any other name would still bind internally; so HTTP would, were it not HTTP called.
Not so fast, Billy! Sometimes it does matter. Let's look at the userlist in Riak Control:
%% @doc If auth is set to 'userlist' then this is the
%% list of usernames and passwords for access to the
%% admin panel.
{mapping, "riak_control.user.$username.password", "riak_control.userlist", [
{default, "pass"},
{include_default, "user"}
]}.
{translation,
"riak_control.userlist",
fun(Conf) ->
UserList = lists:filter(
fun({K, _V}) ->
cuttlefish_variable:is_fuzzy_match(K, string:tokens("riak_control.user.$username.password", "."))
end,
Conf),
Users = [ {Username, Password} || {[_, _, Username, _], Password} <- UserList ],
case Users of
[] ->
throw(unset);
_ -> Users
end
end}.Right now, we're leaving it up to the translation fun to tokenize the string and extract the username, but it shouldn't have to. We should provide helpers for this, and we will.
You can write validator functions to perform advanced validation for mappings. Let's take the ring size in riak as an example:
{validator, "ring_size", "not a power of 2 greater than 1",
fun(Size) ->
Size > 1 andalso (Size band (Size-1) =:= 0)
end}.What we're saying here is that ring_size has to be an integer that's a power of 2 and greater than 1.
So how does this get triggered?
%% @doc Number of partitions in the cluster (only valid when first
%% creating the cluster). Must be a power of 2, minimum 8 and maximum
%% 1024.
{mapping, "ring_size", "riak_core.ring_creation_size", [
{datatype, integer},
{default, 64},
{validators, ["ring_size^2", "ring_size_max", "ring_size_min"]},
{commented, 64}
]}.
%% ring_size validators
{validator, "ring_size_max", "2048 and larger are supported, but considered advanced config",
fun(Size) ->
Size =< 1024
end}.
{validator, "ring_size^2", "not a power of 2",
fun(Size) ->
(Size band (Size-1) =:= 0)
end}.
{validator, "ring_size_min", "must be at least 8",
fun(Size) ->
Size >= 8
end}.In the end, we went with three different validators. The validators property of a mapping specifies which validators get run for a field.They can be reused on multiple mappings, and you can add as many as you want.
Me too. It'd be pretty crazy to just write a bunch of Erlang and expect it to just work in your application. So we decided to give you a module of unit test helper functions: cuttlefish_unit. Please see the Unit Testing A Schema page.