Impossible to override reference.conf

[mikea]

Because reference.conf is fully resolved before merging with main config, the following doesn't work:

reference.conf:

myapp {
    storageDir = "."
}
myappinternal {
    file1 = ${myapp.storageDir}/file1
    file2 = ${myapp.storageDir}/file2
}

aplication.conf:

myapp {
    storageDir = "mydir"
}

file1 and file2 will not be "mydir/file1" paths but "./file1" instead.

[havocp]

Right, it is specified to work this way for better or worse. https://github.com/typesafehub/config/blob/master/HOCON.md#conventional-configuration-files-for-jvm-apps

It can't change without breaking back compat, which means not anytime soon. Do remember though that ConfigFactory.load is only a convenience API so for a reference.conf that you don't expect to escape "into the wild" you could rely on loading config in custom way instead of using the convention that ConfigFactory.load uses. For a public open source library it'd be best not to rely on loading config in a special way though.

#164 has some loosely-related discussion. The suggestion there to put your above snippet of config in an application.conf instead of a reference.conf may be helpful. You can include an application.conf in a library, though it won't be loaded if someone chooses to use a different config file at the app level.

Another option if your internal section is used only in the library would be to load that internal section by hand and add it in to the app-provided config (so as a final step), then internal would be visible only to your lib and would late-resolve.

In the sample library I'd do that right here, after the checkValid:
https://github.com/typesafehub/config/blob/master/examples/scala/simple-lib/src/main/scala/simplelib/SimpleLib.scala#L19

You could parse an inline string containing your internal config values and then merge it into the app-provided config using withFallback and resolve the result.

[mikea]

I see your point. Unfortunately using custom config looks to be problematic with Play.

You are right, that this change would break compatibility. But let's imagine that we did it anyway. I.e. reference.conf is used as fallback without resolving first.

This will definitely generate different config. But it seems to me that in most cases it would be what users actually wanted, no? Do you have an example when current behavior is better?

Also what do you think about "late binding" variables? Let's imagine that my reference.conf has:

myapp {
    storageDir = "."
}
myappinternal {
    file1 = $${myapp.storageDir}/file1
    file2 = $${myapp.storageDir}/file2
}

(or any other syntax). Do you think this could be useful?

Splitting reference.conf and remembering to include parts of it into application.conf would work, but I would like to use this pattern in several reference.conf files, and I'm afraid it would quickly go out of control.

[havocp]

I honestly don't remember the full rationale for how it works; I know it was discussed and considered (since the spec and implementation spell it out), but it was a while ago.

A trouble with late-binding syntax is specifying what it means. ConfigFactory .load and the reference.conf/application.conf stack are currently an extremely superficial layer, not something that extends down into the syntax/semantics of the file format.

Some related discussion at #100

To keep things organized, perhaps you could establish a convention internal.conf, load all of those on classpath with ConfigFactory.parseResources and merge them in to the config that you get from the app. Or something along those lines, the details really depend on the details of your setup.

[rkuhn]

I’m not sure which kind of “going out of control” you mean: if you just stick to application.conf as name, then all resources with that name will be loaded and merged automatically, will they not? (technically, it would work for any name, but of course they need to be named the same)

[mikea]

@rkuhn Are you sure that default Config factory would load all application.conf from classpath? I was under impression that only one would be loaded.

[havocp]

ConfigFactory.load loads all application.conf - I'm not 100% positive what Play does, it may change things.

[rkuhn]

I am not not exactly sure, but this line seems to indicate that it merges resources.

[rkuhn]

Yes, this is by design: reference.conf is the stand-alone reference, its values should not depend on other configs layered on top of it. The intention is that reference.conf is used by library authors and application.conf by application authors. You should be able to place the myapp.internal values in an application.conf shipped with your “library” instead.

[michaelahlers]

Even though this issue's closed, it's worth noting for others that I independently attempted @mikea's design and that it's not possible. The resolution documentation is clear, but the previous section that states application.conf takes precedence over reference.conf may cause confusion.

To illustrate, I'd intended that reference.conf would provide a template into which application.conf could provide a shared value used by multiple contexts while allowing specific values where needed.

Given reference.conf:

sample {
  foobear.value: ${sample.value}
  fizban.value: ${sample.value}
  zifnab.value: ${sample.value}
  value: 0
}

And application.conf:

// Use 42 everywhere except for zifnab.value.
sample {
  value: 42
  zifnab.value: 24
}

I'd hoped for:

sample {
  foobear.value: 42
  fizban.value: 42
  zifnab.value: 24
  value: 42
}

But the actual is:

sample {
  foobear.value: 0
  fizban.value: 0
  zifnab.value: 24
  value: 42
}

There's a clear workaround that isn't terrible: specify a shared value in application.conf and reference it everywhere that's appropriate save for special cases.

[jroper]

I believe this should be reconsidered. It won't break backwards compatibility if we do. I know this, because Play has implemented this custom config loading behaviour for 4 years now, and nothing has broken. Play and Lagom supply a config parsed in this way to Akka, so Akka has been running on Play using this custom loaded config, and nothing has broken.

The ability for reference.conf to be able to rely on variable substitutions that can be overridden in application.conf is super convenient, and very important for a convention over configuration approach to software. There are many places where the default value for configuration should come from or be based from another configuration option. An example is discussed here:

akka/akka-management#477

Allowing the default to be based on another overridable property that makes getting the configuration right less error prone, resulting in a better user experience.

[havocp]

It’s possible it doesn’t break back compat too much, but I don’t think it’s hard to write programs that would have worked before and no longer would, since attempting to use an unresolved value throws an exception. So any sort of “get values from the defaultReference() config” would presumably now have a chance to throw. I believe checkValid() is a thing which relies on getting values from the reference config, with no access to application.conf in there.

I wasnt aware Play had changed this, though.

I’m not really opposed to changing this though I think ideally it might be in a major version (there are several other issues with a “best done in major version” tag) ... and if it makes checkValid fundamentally not work, perhaps that API would be removed. also there would be API docs and HOCON.md to update. All of the defaultApplication, defaultReference methods would likely be made consistent.

It’s some work

[jroper]

But hang on, what you're describing there forwards compatibility, not backwards compatibility. If today, a reference.conf refers to a property that its expecting to set in application.conf, then today, that program will throw an exception. This change would change the behaviour such that it now doesn't throw an exception. Unless there are apps out there that depend on that exception being thrown, nothing is going to break.

However, if we make this change, then someone writes a reference.conf that depends on the change and sources a config property set in application.conf, and then they downgrade to the current version of config that doesn't support this, then that will break. But that just means it's forwards incompatible, not backwards incompatible. I don't think that's a problem.

As for checking validity, I'm not actually proposing that we allow for use cases where reference.conf references undefined properties, that would be bad practice to do that. What I'm saying is that reference.conf should be able to reference properties that it defines itself, and that application.conf should be able to override them, such that the references in reference.conf to them also pick the overridden value up. In this case, reference.conf is able to be parsed and resolved in isolation, and so checking validity will still work.

[havocp]

I don't have a REPL working on my laptop right now (which is a good indicator of how much maintenance on this library is lacking!) but I believe with the following reference.conf:

foo = "hello"
bar = ${foo} "/world"

Then this code does not throw now and would throw with this change:

ConfigFactory.defaultReference().getString("bar")

and checkValid() effectively does something like that.

I suppose if we did not modify defaultReference() to return something unresolved, and only changed ConfigFactory.load to no longer use defaultReference, then the above code would be OK, but then defaultReference would be broken / deprecated (because it wouldn't be the actual reference config which is used by default).

checkValid could resolve the passed-in reference config instead of expecting it to be resolved already, I suppose.

Anyway all these threads would need pulling if things were going to still be coherent and accurately-documented after this change

[dwijnand]

I'm optimistic we can find a way forward (and 👍 to reopening this) but it would seem to me that part of the issue is that pre-resolved and resolved Configs are the same type, no? I think the defaultReference() situation would be clearer if there were distinct types.

[havocp]

Agreed that could be worth exploring (two separate types).

[jroper]

Ah, I understand. I don't think we should change the behaviour of defaultReference, or in fact change any of the methods that currently return resolved config to return unresolved. Rather, I think just the internals of load should change.

then defaultReference would be broken / deprecated (because it wouldn't be the actual reference config which is used by default).

I disagree. It would be the actual reference config. Let's take a concrete real world example, in Play we have this:

play.server {
  dir = "."
  pidfile.path = "${play.server.dir}/RUNNING_PID"
}

So, the resolved reference configuration returns:

play.server {
  dir = "."
  pidfile.path = "./RUNNING_PID"
}

That is the reference configuration. Setting anything in application.conf wouldn't change it.

Now if I have an application.conf with this:

play.server.dir = "/my/application/dir"

Conceptually, I've only changed my applications configuration, which is:

play.server {
  dir = "/my/application/dir"
  pidfile.path = "/my/application/dir/RUNNING_PID"
}

Just because pidfile.path changed automatically doesn't mean that I changed the reference configuration, that still is ./RUNNING_PID, just as play.server.dir reference configuration is ., putting something in application.conf has no impact on the reference configuration.

checkValid wouldn't be broken, you'd just pass in defaultReference, which returns the resolved reference config. It would verify that play.server.dir and play.server.pidfile.path are both strings. I don't think anything would change there.

There is something weird though in Typesafe config, and that is that defaultReference loads the system properties first, which means variables in reference config can be overridden by system properties. Now, changing that would be breaking, since any libraries that do depend on system properties being set in their reference.conf would now fail to resolve when defaultReference was loaded. But that's weird, the reference conf shouldn't change based on system properties, since system properties are a property of the running application, not of the library, so it should stay the same everywhere. I guess it was only added to work around this very issue, and once again, I think the fact that that hack exists and hasn't caused problems is evidence that there's nothing wrong with changing the loading process.

[havocp]

If defaultReference remains resolved I think we’d want some new method that returns the unresolved reference config - load() is meant to be a fairly trivial stacking of other methods, so people can come up with their own stacking.

I’d have to go digging to see when/why the reference config started to pull in system props. There may be a thread from the past.

I’d like to hear from akka team or other popular library users on this change since it certainly could break someone here and there.

btw I’m not reopening this since I don’t have an active plan to actually make the PR and there’s way too much open stuff already, but I’m not trying to block someone from working on it.

The hard part won’t be the code code change per se but finding all the affected docs and tests and such, I believe. Could be wrong!

[jroper]

I've created a PR in #619. Where this might break is where a user might be overriding something in reference.conf, but then be expecting something that references it to not be overridden.

Looking at the Akka code base, here are some things it could impact, determined by searching for all variable references, ie, occurances of ${ in reference.conf files:

• If you set a name for the cluster singleton actor in application.conf, that name will now also apply to the cluster singleton proxy actor, where previously it wouldn't.
• Changes to top level cluster singleton settings will now impact the cluster singleton actor used by cluster sharding.
• Changes to the top level distributed data settings will now impact the cluster sharding distributed data settings, except where cluster sharding overrides it.
• Changes to the Akka stream materializer will also impact the materializer used by Akka remoting.
• Changes to the Netty TCP transport configuration will impact the SSL and UDP transports too (though, not likely to be an issue because it would be incredibly rare to use more than one transport).

There is a chance that something could break in the above. The most likely to break would be the Akka cluster sharding reference to distributed data, the rest are advanced configuration features that are rarely changed, and if they are changed, would likely be advantageous to override the place that references it too. The configuration element in the Akka cluster sharding reference to distributed data that would have the biggest impact is the list of durable keys, but its already overridden by the cluster sharding configuration, so that won't be a problem. The rest of the keys there, I can't imagine why you would want to change one and not the other, for example, why would you override the default durable store for distributed data, but have cluster sharding use the default one? It's possible that you might want to, but unlikely.

@patriknw fyi

[patriknw]

@jroper @havocp The proposed behavior by James would be very useful. We have run into that "limitation" many times and had to explain to contributors and ourselves how substitution works. I have always thought it was how it was designed for some reasons and have not challenged it.

I agree that it will probably not be any compatibility problems, although I might hesitate dropping it into an Akka 2.5.x patch release. Akka 2.6.0 would be a safe target. It would still be good if this change doesn't require a major release of Typesafe Config because then we might not be able to update in Akka 2.6.

[patriknw]

I should also mention that currently, and for years, we have been using different loading behavior for Akka and Play and that is pretty scary in itself. Would be great to align.

[havocp]

If y’all were to catalog custom behavior and what would have to change in this library for everyone to use the same default behavior that’s probably a useful exercise (not in this same PR obviously)

[jroper]

Here's Play's behaviour:

https://github.com/playframework/playframework/blob/5f67396318a9b5cb08e9f42fb7a97c6ca49ba268/core/play/src/main/scala/play/api/Configuration.scala#L48-L107

Aside from loading reference.conf without resolving it first, Play does two additional unique things:

1. It loads all play/reference-overrides.conf files as a fallback between reference.conf and application.conf. This was primarily done to override some Akka defaults, for example, in dev mode, to make Akka not exit the JVM when a fatal exception is caught because more often than not such errors are recoverable on the next reload of the Play dynamic classloader. It's also now used by Lagom to provide Lagom specific defaults for a number of Akka features.
2. It injects configuration behind in the build file (ie, defined in build.sbt) for use in dev mode, as a fallback between application.conf and system properties.