[doc] Document how to use the dynamic config

[wacban]

Write a nice description on how to use the dynamic configs e.g. expected_shutdown.
For reference you can use the Pagoda's internal documentation here

[MuhtasimTanmoy]

Is the documentation link working?

[wacban]

Ah sorry it's a Pagoda internal doc. Here is the content for anyone without internal access:

Quick how-to (Linux & Mac)

Enable debug logs:
echo '{"verbose_module": ""}' > ~/.near/log_config.json
killall -SIGHUP neard

Disable debug logs:
rm ~/.near/log_config.json
killall -SIGHUP neard
Change log level at runtime

    ssh into the machine with the node you’re debugging

    Create file ~/.near/log_config.json with the following content to enable global debug logging:

        '{"verbose_module": ""}'

    Or the following content to enable more targeted logging:

        '{"rust_log": "warn,stats=info,neard=info"}'

    Send signal SIGHUP to the neard process. Note that this will NOT kill your job :wink:

        killall -s 1 neard

The neard process will instantly apply the new logging configuration.

 

If you remove the file ~/.near/log_config.json and send the signal, the process will revert to the configuration specified in the RUST_LOG environment variable.
LogConfig

Let’s look at how to configure logging in more details.

Here is the config struct:
struct LogConfig {
  pub rust_log: Option<String>,
  pub verbose_module: Option<String>,
}

rust_log is equivalent to setting RUST_LOG environment variable. You can pass a comma-separate list of filtering directives. All of the following are valid examples:

    ““

    “warn”

    “stats=info”

    “stats=info,chain=debug”

    “warn,stats=info,chain=debug”

Note that enabling debug logging globally is not recommended. Use verbose_module instead.

verbose_module has two options:

    "verbose_module": "" - enables debug logging globally, but reduces verbosity of certain 3rd party noisy targets.

    "verbose_module": "some_module" - enables debug logging specifically for target some_module. You can only provide one module at a time. If you need debug logging for several targets, simply use rust_log.

Design choices

This feature is implemented as reloading a file upon receiving a signal, because it is:

    Secure

    Low overhead

One of the alternatives was to provide an HTTP RPC endpoint. As we don’t want to let anyone on the internet to control log levels of our nodes, we decided this needs a robust authentication mechanism. Such mechanism wasn’t available at the moment, and setting it up was considered to be too much work.

Another alternative was to read a logging config file every X seconds. This was rejected due to its less predictable behavior and a lot of unneeded file operations. Logging config changes happen very rarely. At most only a few times per day.
Rollout
The feature will be included in the 1.27 release of neard, and available on canary and is already available on canary nodes and betanet nodes.