Skip to content

Commit decd3f6

Browse files
jelujelu
committed
man-pages, URLs
- `generator/client_subnet_authority`: - Add option `urlv4` and `urlv6` to specify IANA's IP address space registries CSV URLs - Write map-page - `generator/client_subnet_country`: Write man-page - `output/influxdb`: Write man-page - `transformer/labler`: Write man-page - `transformer/netremap`: Write man-page - `transformer/reranger`: Write man-page
1 parent c85f17d commit decd3f6

7 files changed

+217
-11
lines changed

‎dsc_datatool/generator/client_subnet_authority.py‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -47,15 +47,16 @@ def _read(self, input):
4747
def __init__(self, opts):
4848
self.auth = {}
4949
csvs = opts.get('csv', None)
50+
urlv4 = opts.get('urlv4', 'https://www.iana.org/assignments/ipv4-address-space/ipv4-address-space.csv')
51+
urlv6 = opts.get('urlv6', 'https://www.iana.org/assignments/ipv6-unicast-address-assignments/ipv6-unicast-address-assignments.csv')
5052
if csvs:
5153
if not isinstance(csvs, list):
5254
csvs = [ csvs ]
5355
for file in csvs:
5456
with open(file, newline='') as csvfile:
5557
self._read(csvfile)
5658
elif opts.get('fetch', 'no').lower() == 'yes':
57-
urls = opts.get('url', [ 'https://www.iana.org/assignments/ipv4-address-space/ipv4-address-space.csv',
58-
'https://www.iana.org/assignments/ipv6-unicast-address-assignments/ipv6-unicast-address-assignments.csv' ])
59+
urls = opts.get('url', [ urlv4, urlv6 ])
5960
if urls and not isinstance(urls, list):
6061
urls = [ urls ]
6162
logging.info('bootstrapping client subnet authority using URLs')

‎man/man7/dsc-datatool-generator-client_subnet_authority.7‎

Lines changed: 34 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,40 @@
11
.TH "dsc-datatool-generator client_subnet_authority" "7"
22
.SH NAME
3-
client_subnet_authority \- Generates network authority information based on client subnet.
3+
client_subnet_authority \- Generates network authority information (RIR) based on client subnet.
4+
.SH SYNOPSIS
5+
.SY dsc-datatool
6+
.B \-\-generator
7+
.I """;client_subnet_authority;<options...>"""
8+
.YS
49
.SH DESCRIPTION
5-
TODO
10+
This generator will lookup the network authority (RIR) based network
11+
information in the
12+
.I client_subnet
13+
dataset and generate a new dataset
14+
.IR client_subnet_authority .
15+
16+
The network authority lookup will be done from imports of CSV files or by
17+
fetching the required data from IANA.
18+
.SH OPTIONS
19+
.TP
20+
.BR csv =<files>
21+
Read the network authority information from the given
22+
.IR file(s) .
23+
This option can be a single file, a list of comma separated files or given
24+
multiple times.
25+
.TP
26+
.BR fetch =yes
27+
Fetch the network authority information from IANA IP address space registry
28+
for IPv4 and IPv6.
29+
.TP
30+
.BR urlv4 =url
31+
URL for IANA IPv4 address space registry, default to
32+
.IR https://www.iana.org/assignments/ipv4-address-space/ipv4-address-space.csv .
33+
.TP
34+
.BR urlv6 =url
35+
URL for IANA IPv6 address space registry, default to
36+
.IR https://www.iana.org/assignments/ipv6-unicast-address-assignments/ipv6-unicast-address-assignments.csv .
37+
.LP
638
.SH "SEE ALSO"
739
.BR dsc-datatool (1)
840
.SH AUTHORS

‎man/man7/dsc-datatool-generator-client_subnet_country.7‎

Lines changed: 35 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,10 +1,43 @@
11
.TH "dsc-datatool-generator client_subnet_country" "7"
22
.SH NAME
33
client_subnet_country \- Generates country information based on client subnet using Maxmind database.
4+
.SH SYNOPSIS
5+
.SY dsc-datatool
6+
.B \-\-generator
7+
.I """;client_subnet_country;<options...>"""
8+
.YS
49
.SH DESCRIPTION
5-
TODO
10+
This generator looks up country code in a Maxmind database for the subnets
11+
in the
12+
.I client_subnet
13+
dataset and the dataset
14+
.I client_subnet_country
15+
from that.
16+
.SH OPTIONS
17+
.TP
18+
.BR path =<directory>
19+
Search for the Maxmind database in the specified
20+
.IR directory .
21+
This option can be given multiple times to search in multiple directories.
22+
23+
Default search paths are:
24+
.IR /var/lib/GeoIP ,
25+
.IR /usr/share/GeoIP ,
26+
.IR /usr/local/share/GeoIP .
27+
.TP
28+
.BR filename =filename
29+
The database filename to search for, default to
30+
.IR GeoLite2-Country.mmdb .
31+
.TP
32+
.BR db =/path/to/database.mmdb
33+
Full path to Maxmind database, this option overrides
34+
.I path
35+
and
36+
.IR filename .
37+
.LP
638
.SH "SEE ALSO"
7-
.BR dsc-datatool (1)
39+
.BR dsc-datatool (1),
40+
.BR geoipupdate (1)
841
.SH AUTHORS
942
Jerry Lundström, DNS-OARC
1043
.LP

‎man/man7/dsc-datatool-output-influxdb.7‎

Lines changed: 37 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,10 +1,45 @@
11
.TH "dsc-datatool-output influxdb" "7"
22
.SH NAME
33
InfluxDB \- InfluxDB output.
4+
.SH SYNOPSIS
5+
.SY dsc-datatool
6+
.B \-\-output
7+
.I """;InfluxDB;<options...>"""
8+
.YS
49
.SH DESCRIPTION
5-
TODO
10+
This output generates InfluxDB importable output to stdout or to a specified
11+
file.
12+
.SH OPTIONS
13+
.TP
14+
.BR timestamp =[start|stop]
15+
Choose which timestamp from the dataset to use in the InfluxDB output to
16+
specify when the metrics took place.
17+
18+
Default to
19+
.I start
20+
timestamp.
21+
.TP
22+
.BR file =<filename>
23+
Specify a file to output to instead of stdout.
24+
.TP
25+
.BR append
26+
If given, the output will be appended to the file specified rather then
27+
overwritten.
28+
.TP
29+
.BR dml
30+
Add a
31+
.I "# DML"
32+
header to the output.
33+
.TP
34+
.BR database =name
35+
If
36+
.I dml
37+
is used, this can be used to specify which database the output should be
38+
imported into.
39+
.LP
640
.SH "SEE ALSO"
7-
.BR dsc-datatool (1)
41+
.BR dsc-datatool (1),
42+
.BR influx (1)
843
.SH AUTHORS
944
Jerry Lundström, DNS-OARC
1045
.LP

‎man/man7/dsc-datatool-transformer-labler.7‎

Lines changed: 34 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,41 @@
11
.TH "dsc-datatool-transformer labler" "7"
22
.SH NAME
33
Labler \- Rewrite numeric labels to textual labels using the provided YAML data as lookup tables.
4+
.SH SYNOPSIS
5+
.SY dsc-datatool
6+
.B \-\-transform
7+
.I """;Labler;<dataset>;yaml=<file>"""
8+
.YS
49
.SH DESCRIPTION
5-
TODO
10+
This transformer rewrites labels in datasets based on information provided
11+
in a YAML file.
12+
13+
The YAML structure is as follows:
14+
.EX
15+
---
16+
dataset_name:
17+
DimentionName:
18+
DimentionValue: RewriteTo
19+
.EE
20+
21+
The
22+
.I dataset_name
23+
is the name of the dataset,
24+
.I DimentionName
25+
is the name of the dimension to rewrite and
26+
.I DimensionValue
27+
is the value to change to
28+
.IR RewriteTo .
29+
.SH OPTIONS
30+
.TP
31+
.B dataset
32+
See
33+
.IR dsc-datatool (1)
34+
on how to specify which dataset(s) to run the transformer on.
35+
.TP
36+
.BR yaml =<file>
37+
The YAML file to load rewrite data from (required).
38+
.LP
639
.SH "SEE ALSO"
740
.BR dsc-datatool (1)
841
.SH AUTHORS

‎man/man7/dsc-datatool-transformer-netremap.7‎

Lines changed: 27 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,34 @@
11
.TH "dsc-datatool-transformer netremap" "7"
22
.SH NAME
33
NetRemap \- Remap network addresses to other ranges/subnets.
4+
.SH SYNOPSIS
5+
.SY dsc-datatool
6+
.B \-\-transform
7+
.I """;NetRemap;<dataset>;<options...>"""
8+
.YS
49
.SH DESCRIPTION
5-
TODO
10+
This transformer can remap network ranges on IP addresses and subnets.
11+
.SH OPTIONS
12+
.TP
13+
.B dataset
14+
See
15+
.IR dsc-datatool (1)
16+
on how to specify which dataset(s) to run the transformer on.
17+
.TP
18+
.BR net =<network>
19+
The network to remap both IPv4 and IPv6 addresses and networks, specify
20+
only the number in a short-form network (e.g. "/<network>").
21+
.TP
22+
.BR v4net =<network>
23+
The IPv4 network to use for IPv4 addresses and networks, default to
24+
.I net
25+
if that is specified.
26+
.TP
27+
.BR v6net =<network>
28+
The IPv6 network to use for IPv6 addresses and networks, default to
29+
.I net
30+
if that is specified.
31+
.LP
632
.SH "SEE ALSO"
733
.BR dsc-datatool (1)
834
.SH AUTHORS

‎man/man7/dsc-datatool-transformer-reranger.7‎

Lines changed: 47 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,54 @@
11
.TH "dsc-datatool-transformer reranger" "7"
22
.SH NAME
33
ReRanger \- Rewrite ranged or numerical statistics into other ranges.
4+
.SH SYNOPSIS
5+
.SY dsc-datatool
6+
.B \-\-transform
7+
.I """;ReRanger;<dataset>;<options...>"""
8+
.YS
49
.SH DESCRIPTION
5-
TODO
10+
This transformer can re-range values in datasets to new rangers.
11+
12+
It supports both values that are a single numerical value or one that is
13+
already a range itself
14+
.RI ( num\-num ).
15+
.SH OPTIONS
16+
.TP
17+
.B dataset
18+
See
19+
.IR dsc-datatool (1)
20+
on how to specify which dataset(s) to run the transformer on.
21+
.TP
22+
.BR key =[low|mid|high]
23+
Specify what value to use when re-ranging an already ranged value
24+
.RI (low\-high).
25+
26+
Default to
27+
.IR mid ,
28+
which is high minus low then divided by 2.
29+
.TP
30+
.BR func =<func>
31+
The function to use when aggregating multiple values into a new ranger,
32+
default to
33+
.IR sum .
34+
35+
Currently only one function exists,
36+
.IR sum ,
37+
which adds the sum of all values into the new range.
38+
.TP
39+
.B allow_invalid_keys
40+
If given the "invalid" keys/values that is not numerical or a range will be
41+
passed through unmodified.
42+
.TP
43+
.BR range =</num>
44+
The new range given as
45+
.I /number
46+
(required).
47+
48+
For example
49+
.I range=/64
50+
will re-range all values into buckets of 64, 0-63, 64-127 and so on.
51+
.LP
652
.SH "SEE ALSO"
753
.BR dsc-datatool (1)
854
.SH AUTHORS

0 commit comments

Comments
 (0)