Night - file filter (grep, head & tail) - server.

DevOps helper HTTP server.

Install

$ npm install koron/night

   or

$ npm install -g koron/night

Tutorial

Start night on a server.

$ night

Then access by HTTP from client.

$ curl http://127.0.0.1:9280/files/var/log/messages?tail

You will get last 10 lines of /var/log/messages on a server. When open this URL using WEB browser, you will see same content, but refresh in each 5 seconds.

http://127.0.0.1:9280/files/var/log/messages?tail&refresh=5

Sever Usage

$ node [OPTIONS]

Where OPTIONS are:

• -c {FILE} or --config=FILE - server configuration file.
• -v or --verbose - verbose message (not implemented yet).
• -h or --help - show help message

Server configuration file.

Configuration file is JSON format. All properties are optional.

• ssl - Boolean. Use HTTPS instead of HTTP. (default: false; not tested)
  • ssl_key - String. Path of private key of the server in PEM format.
  • ssl_cert - String. Path of certificate key of the server in PEM format.
• port - Number. Listen port number. (default: 9280)
• locations - String array. List of path prefix of accessible resources. Empty means no restriction (default: [] (empty)).
• forbiddens - String array. List of path prefix of resources which forbidden to be accessed. Empty means no restriction (default: [] (empty)). This override locations settings.
• users - String array. List of "{USER}:{PASS}" which allowed to access.
• commands - Object. Table of command consists name, command file and arguments.
• config_visible - Booelan. Accessible config under /config/ path. (default: false, security reason)
• refresh_always - Number. When larger than zero, "Refresh" header is added for all requests with tha value. 0 for disable. default is 0.
• default_filters - String. Applied if any filters are not specified by query string. Please refer below section for details. (default: empty, no default filters)

Example #1:

{
  "port": 8000,
  "locations": [
    "/var/log/",
    "/etc/"
  ],
  "forbiddens": [
    "/etc/ssh",
    "/etc/passwd"
  ],
  "users": [
    "test:1234",
    "root:9999"
  ]
  "commands": {
    "df": ["df", "-h"],
    "lstmp": ["ls", "-l", "/tmp"]
  }
}

Example #2: To enable SSL.

{
  "ssl": true,
  "ssl_key": "/path/to/ssl_key",
  "ssl_cert": "/path/to/ssl_cert"
}

Client Usage

Access URL is consisted from these rule.

http://{host}:{port}/files{/path/to/source}[?{filters}]

• host - host address, IP or host name.
• port - port number, configurable see above.
• /path/to/source - path of the resource to read, file or directory.
• filters - filters spec string, see below. (optional)

Directory Source

When /path/to/source point a directory, you will get lines in this format:

{filename}\t{type}\t{size}\t{time}

Where \t is TAB character, so you can choose by cut filter easily.

• filename - Name of entry.
• type - Type of entry: file, dir, error or unknown.
• size - Size of entry, or 0 when type is error.
• time - Timestamp of entry in UTC format, or error detail when type is error.

Example: get name and size entries in /var/log.

http://127.0.0.1:9280/files/var/log?cut=list:1,3

File Source

*.gz files are decompressed automatically.

*.bz2 files are decompressed automatically when available bzip2 command.

*.lz4 files are decompressed automatically when available lz4 command.

Glob Files Source

Allowed file globs (wildcards) in /path/to/source like this:

• /var/log/httpd/access*.gz?grep=...

Command Source

以下のURLにアクセスすると、予め Configuration file で指定したコマンドを実行しそ の標準出力を取得できます。出力内容にはファイルなどと同じようにフィルタを適用で きます。

http://{host}:{port}/commands/{command_name}[?{filters}]

TODO: translate to English.

TODO: write description and examples.

Config Source

This feature is enabled when config_visible is true.

When access below URL, you'll get config in JSON format.

http://{host}:{port}/config[?{filters}]

Help Source

When access below URL, you'll get help (this file).

http://{host}:{port}/help[?{filters}]

Version Source

When access below URL, you'll get version info like night/0.2.1.

http://{host}:{port}/version[?{filters}]

Filter Spec

Where {filters} is:

{filter}[&{filter}...]

Where {filter} is:

{filter_name}[={options}]

Where {options} is:

{option_name}:{value}[;{option_name}:{value}...]

See other section for detail of each filters.

Example: get last 50 lines except empty lines.

http://127.0.0.1:9280/files/var/log/messages?grep=re:^$;match:false&tail=limit:50

Filters

Currenly support these filters:

• Grep filter
• Head filter
• Tail filter
• Cut filter
• Hash filter
• LTSV filter
• Refresh (pseudo) filter
• All (dummy) filter

Grep filter

Output lines which matches against regular expression.

As default, matching is made for whole line. But when valid option field is given, then matching is made for specified a field, which is splitted by delim character.

grep command equivalent.

• filter_name: grep
• options
  • re - regular expression used for match.
  • match - output when match or not match. default is true.
  • field - a match target N'th field counted from 1. default is none (whole line).
  • delim - field delimiter string (default: TAB character).

Head filter

Output the first N lines.

head command equivalent.

• filter_name: head
• options
  • start - start line number for output. begging 0. default is 0.
  • limit - line number for output. defualt is 10.

Tail filter

Output the last N lines.

tail command equivalent.

• filter_name: tail
• options
  • limit - line number for output. defualt is 10.

Cut filter

Output selected fields of lines.

cut command equivalent.

• filter_name: cut
• options:
  • delim - field delimiter string (default: TAB character).
  • list - selected fields, combinable by comma ,.
    • N - N'th field counted from 1.
    • N-M - from N'th, to M'th field (included).
    • N- - from N'th field, to end of line.
    • N- - from first, to N'th field.

Hash filter

Output hash value.

• filter_name: hash
• options:
  • algorithm - one of md5 (default), sha1, sha256 or sha512
  • encoding - one of hex (default), base64 or binary

Count filter

Count lines.

• filter_name: count
• options: (none)

LTSV filter

Output, match to value of specified label, and output selected labels.

• filter_name: ltsv
• options:
  • grep - match parameter: {label},{pattern}
  • match - output when match or not match. default is true.
  • cut - selected labels, combinable by comma ,.

Refresh (pseudo) filter

Add "Refresh" header with specified time (sec).

• filter_name: refresh
• options: interval seconds to refresh. 0 for disable.

Example: Open below URL using WEB browser, it refresh in each 5 seconds automatically.

http://127.0.0.1:9280/files/var/log/messages?tail&refresh=5

All (dummy) filter

Through all lines as is. This would be used for default_filters enabled environment only.

• filter_name: all
• options: (none)

Default filters

When you specify default_filters in config JSON, night apply that filters as default, when not specified any filters by query string. For its format, please refer Filter Spec section.

To just disable this default_filters temporary, you can use all special filter in query string.

Configuration Example

This configuration will apply "tail" filter (default: last 10 lines) as default_filters.

{
  "default_filters": "tail"
}

If you changed like below, it show last 20 lines as default_filters.

{
  "default_filters": "tail=limit:20"
}

Complex default filters

If giving a string for default_filters, night apply it to resources under /files/ only. But if you give an object, you can specify path which be applied to. Let's see configuration sample:

{
  "default_filters": {
    "/files/var/": "tail",
    "/files/tmp/": "head"
  }
}

This apply tail filter to resources under /files/var/, and head filter to under /files/tmp/.

Examples

Detect I/O error from /var/log/messages

$ curl http://127.0.0.1:9280/files/var/log/messages?grep=re:I%27O

Check whether a program "WATCHDOG" was executed by cron or not.

$ curl http://127.0.0.1:9280/files/var/log/cron?grep=re:WATCHDOG

Found errors of HTTP access, limited recent 100 records.

$ curl http://127.0.0.1:9280/files/var/log/nginx/access.log?grep=re:200,match:false&tail=limit:100