Skip to content

Commit f5ca007

Browse files
jelujelu
committed
man-pages
- Update development environment instructions - Add man-pages: - `dsc-datatool(1)`: Main page with command line options - `dsc-datatool.conf(5)`: Configuration file (placeholder, not implemented) - `dsc-datatool-generator client_subnet_country(7)`: (placeholder, todo) - `dsc-datatool-generator client_subnet_authority(7)`: (placeholder, todo) - `dsc-datatool-transformer reranger(7)`: (placeholder, todo) - `dsc-datatool-transformer labler(7)`: (placeholder, todo) - `dsc-datatool-transformer netremap(7)`: (placeholder, todo) - `dsc-datatool-output influxdb(7)`: (placeholder, todo) - `debian`: Install man-pages - `dsc-datatool`: Fix `--list` command
1 parent fd21ce4 commit f5ca007

11 files changed

+445
-74
lines changed

‎README.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -11,8 +11,8 @@ on how to set this up using Influx DB and Grafana.
1111
## python
1212

1313
```
14-
sudo apt-get install python3-maxminddb python3-venv
14+
sudo apt-get install python3-maxminddb python3-yaml python3-venv
1515
python3 -m venv venv --system-site-packages
1616
. venv/bin/activate
17-
pip install -e .
17+
pip install -e . --no-deps
1818
```

‎debian/manpages‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
man/man7/dsc-datatool-transformer-reranger.7
2+
man/man7/dsc-datatool-generator-client_subnet_country.7
3+
man/man7/dsc-datatool-generator-client_subnet_authority.7
4+
man/man7/dsc-datatool-output-influxdb.7
5+
man/man7/dsc-datatool-transformer-labler.7
6+
man/man7/dsc-datatool-transformer-netremap.7
7+
man/man5/dsc-datatool.conf.5
8+
man/man1/dsc-datatool.1

‎dsc_datatool/__init__.py‎

Lines changed: 47 additions & 72 deletions
Original file line numberDiff line numberDiff line change
@@ -19,78 +19,38 @@
1919
import re
2020

2121

22-
parser = argparse.ArgumentParser(prog='dsc-datatool', description='Export DSC data into various formats and databases.')
23-
parser.add_argument('-c', '--conf', nargs=1)
24-
# Specify the YAML configuration file to use (default to ~/.dsc-datatool.conf),
25-
# any command line option will override the options in the configuration file.
26-
# See B<dsc-datatool.conf(5)> for more information.
27-
parser.add_argument('-s', '--server', nargs=1, required=True)
28-
# Specify the server for where the data comes from.
29-
parser.add_argument('-n', '--node', nargs=1, required=True)
30-
# Specify the node for where the data comes from.
31-
parser.add_argument('-x', '--xml', action='append')
32-
# Read DSC data from the given file or directory, can be specified multiple
33-
# times.
34-
# If a directory is given then all files ending with B<.xml> will be read.
35-
parser.add_argument('-d', '--dat', action='append')
36-
# Read DSC data from the given directory, can be specified multiple
37-
# times.
38-
# Note that the DAT format is depended on the filename to know what type
39-
# of data it is.
40-
parser.add_argument('--dataset', action='append')
41-
# Specify that only the list of datasets will be processed, the list is comma
42-
# separated and the option can be given multiple times.
43-
parser.add_argument('-o', '--output', action='append')
44-
# =item B<[ -o | --output ] <sep><type>[<sep>option=value...]>
45-
#
46-
# Output data to B<type> and use B<separator> as an options separator, example:
47-
#
48-
# --output ;Carbon;host=localhost;port=2003
49-
#
50-
# Can be specified multiple times to output to more then one.
51-
#
52-
# To see a full list of options, check the man-page of the output module:
53-
#
54-
# man App:DSC::DataTool::Output::NAME
55-
parser.add_argument('-t', '--transform', action='append')
56-
# =item B<[ -t | --transform ] <sep><type><sep><datasets>[<sep>option=value...]>
57-
#
58-
# Use the transformer B<type> to change the list of datasets in B<datasets>,
59-
# example:
60-
#
61-
# --transform ;ReRanger;rcode_vs_replylen;type=sum;range=/128
62-
#
63-
# B<datasets> is a comma separated list of datasets to run the tranformer on,
64-
# because of this do not use comma (,) as a separator. B<*> in B<datasets> will
65-
# make the tranformer run on all datasets.
66-
#
67-
# Can be specific multiple times to chain transformation, the chain will be
68-
# executed in the order on command line with one exception. All transformations
69-
# specified for dataset B<*> will be executed before named dataset
70-
# transformations.
71-
#
72-
# To see a full list of options, check the man-page of the transformer module:
73-
#
74-
# man App:DSC::DataTool::Transformer::NAME
75-
#
76-
# For a list of datasets see the DSC configuration that created the data files
77-
# and the documentation for the Presenter.
78-
parser.add_argument('-g', '--generator', action='append')
79-
# =item B<[ -g | --generator ] <list of generators>>
80-
#
81-
# Use the specified generators to generate additional datasets, the list is comma
82-
# separated and the option can be given multiple times.
83-
#
84-
parser.add_argument('--list', action='store_true')
85-
# List the available B<inputs>, B<generators>, B<transformers> and B<outputs>.
86-
parser.add_argument('--skipped-key', nargs=1, default='-:SKIPPED:-')
87-
# Set the special DSC skipped key, default to "-:SKIPPED:-".
88-
parser.add_argument('--skipped-sum-key', nargs=1, default='-:SKIPPED_SUM:-')
89-
# Set the special DSC skipped sum key, default to "-:SKIPPED_SUM:-".
90-
parser.add_argument('-v', '--verbose', action='count', default=0)
91-
# Increase the verbose level, can be given multiple times.
92-
parser.add_argument('-V', '--version', action='version', version='%(prog)s v'+__version__)
93-
# Display version and exit.
22+
parser = argparse.ArgumentParser(prog='dsc-datatool',
23+
description='Export DSC data into various formats and databases.',
24+
epilog='See man-page dsc-datatool(1) and dsc-datatool [generator|transformer|output] <name>(5) for more information')
25+
parser.add_argument('-c', '--conf', nargs=1,
26+
help='Not implemented')
27+
# help='Specify the YAML configuration file to use (default to ~/.dsc-datatool.conf), any command line option will override the options in the configuration file. See dsc-datatool.conf(5)for more information.')
28+
parser.add_argument('-s', '--server', nargs=1,
29+
help='Specify the server for where the data comes from. (required)')
30+
parser.add_argument('-n', '--node', nargs=1,
31+
help='Specify the node for where the data comes from. (required)')
32+
parser.add_argument('-x', '--xml', action='append',
33+
help='Read DSC data from the given file or directory, can be specified multiple times. If a directory is given then all files ending with .xml will be read.')
34+
parser.add_argument('-d', '--dat', action='append',
35+
help='Read DSC data from the given directory, can be specified multiple times. Note that the DAT format is depended on the filename to know what type of data it is.')
36+
parser.add_argument('--dataset', action='append',
37+
help='Specify that only the list of datasets will be processed, the list is comma separated and the option can be given multiple times.')
38+
parser.add_argument('-o', '--output', action='append',
39+
help='"<sep><output>[<sep>option=value...]>" Output data to <output> and use <separator> as an options separator.')
40+
parser.add_argument('-t', '--transform', action='append',
41+
help='"<sep><name><sep><datasets>[<sep>option=value...]>" Use the transformer <name> to change the list of datasets in <datasets>.')
42+
parser.add_argument('-g', '--generator', action='append',
43+
help='"<name>[,<name>,...]" or "<sep><name>[<sep>option=value...]>" Use the specified generators to generate additional datasets.')
44+
parser.add_argument('--list', action='store_true',
45+
help='List the available generators, transformers and outputs then exit.')
46+
parser.add_argument('--skipped-key', nargs=1, default='-:SKIPPED:-',
47+
help='Set the special DSC skipped key. (default to "-:SKIPPED:-")')
48+
parser.add_argument('--skipped-sum-key', nargs=1, default='-:SKIPPED_SUM:-',
49+
help='Set the special DSC skipped sum key. (default to "-:SKIPPED_SUM:-")')
50+
parser.add_argument('-v', '--verbose', action='count', default=0,
51+
help='Increase the verbose level, can be given multiple times.')
52+
parser.add_argument('-V', '--version', action='version', version='%(prog)s v'+__version__,
53+
help='Display version and exit.')
9454

9555
args = parser.parse_args()
9656

@@ -258,6 +218,21 @@ def main():
258218
log_level = 0
259219
logging.basicConfig(format='%(asctime)s %(levelname)s %(module)s: %(message)s', level=log_level, stream=sys.stderr)
260220

221+
if args.list:
222+
print('Generators:')
223+
for name in _generators:
224+
print('',name)
225+
print('Transformers:')
226+
for name in _transformers:
227+
print('',name)
228+
print('Outputs:')
229+
for name in _outputs:
230+
print('',name)
231+
return 0
232+
233+
if not args.server or not args.node:
234+
raise Exception('--server and --node must be given')
235+
261236
if isinstance(args.server, list):
262237
args.server = ' '.join(args.server)
263238
elif not isinstance(args.server, str):

‎man/man1/dsc-datatool.1‎

Lines changed: 185 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,185 @@
1+
.TH "dsc-datatool" "1"
2+
.SH NAME
3+
dsc-datatool \- Tool for converting, exporting, merging and transforming DSC data.
4+
.SH SYNOPSIS
5+
.SY dsc-datatool
6+
.OP \-h
7+
.OP \-c CONF
8+
.OP \-s SERVER
9+
.OP \-n NODE
10+
.OP \-x XML
11+
.OP \-d DAT
12+
.OP \-\-dataset DATASET
13+
.OP \-o OUTPUT
14+
.OP \-t TRANSFORM]
15+
.OP \-g GENERATOR
16+
.OP \-\-list
17+
.OP \-\-skipped\-key SKIPPED_KEY
18+
.OP \-\-skipped\-sum\-key SKIPPED_SUM_KEY
19+
.OP \-v
20+
.OP \-V
21+
.YS
22+
.SH DESCRIPTION
23+
Tool for converting, exporting, merging and transforming DSC data.
24+
25+
Please have a look at the wiki article on how to set this up using
26+
Influx DB and Grafana.
27+
28+
https://github.com/DNS-OARC/dsc-datatool/wiki/Setting-up-a-test-Grafana
29+
.SH OPTIONS
30+
.TP
31+
.B -h, --help
32+
show this help message and exit
33+
.TP
34+
.BI "-c " CONF ", --conf " CONF
35+
Not implemented
36+
.TP
37+
.BI "-s " SERVER ", --server " SERVER
38+
Specify the server for where the data comes from. (required)
39+
.TP
40+
.BI "-n " NODE ", --node " NODE
41+
Specify the node for where the data comes from. (required)
42+
.TP
43+
.BI "-x " XML ", --xml " XML
44+
Read DSC data from the given file or directory, can be specified multiple
45+
times.
46+
If a directory is given then all files ending with .xml will be read.
47+
.TP
48+
.BI "-d " DAT ", --dat " DAT
49+
Read DSC data from the given directory, can be specified multiple times.
50+
Note that the DAT format is depended on the filename to know what type of
51+
data it is.
52+
.TP
53+
.BI "--dataset " DATASET
54+
Specify that only the list of datasets will be processed, the list is
55+
comma separated and the option can be given multiple times.
56+
.TP
57+
.BI "-o " OUTPUT ", --output " OUTPUT
58+
.I OUTPUT
59+
has the following format that uses
60+
.I output
61+
to specify the output module and
62+
.I sep
63+
as an options separator.
64+
65+
.EX
66+
<sep><output>[<sep>option=value...]>
67+
.EE
68+
69+
Can be specified multiple times to output to more then one.
70+
71+
Use
72+
.B dsc-datatool --list
73+
to see a list of modules and the man-page of each output for information
74+
about options.
75+
.TP
76+
.BI "-t " TRANSFORM ", --transform " TRANSFORM
77+
.I TRANSFORM
78+
has the following format that uses
79+
.I name
80+
to specify the transformer module and
81+
.I sep
82+
as an options separator.
83+
The
84+
.I datasets
85+
field can specify which dataset to run the transformer on, or use
86+
.I *
87+
to specify all datasets.
88+
89+
.EX
90+
<sep><name><sep><datasets>[<sep>option=value...]>
91+
.EE
92+
93+
Can be specific multiple times to chain transformation, the chain will be
94+
executed in the order on command line with one exception.
95+
All transformations specified for dataset
96+
.I *
97+
will be executed before named dataset transformations.
98+
99+
Use
100+
.B dsc-datatool --list
101+
to see a list of modules and the man-page of each transformer for
102+
information about options.
103+
For a list of datasets see the DSC configuration that creates the XML files
104+
or the documentation for the Presenter that creates the DAT files.
105+
.TP
106+
.BI "-g " GENERATOR ", --generator " GENERATOR
107+
.I GENERATOR
108+
has two formats, one to specify a comma separated list of generators
109+
and one that uses
110+
.I name
111+
to specify the generator module and
112+
.I sep
113+
as an options separator.
114+
115+
.EX
116+
<name>[,<name>,...]
117+
118+
<sep><name>[<sep>option=value...]>
119+
.EE
120+
121+
This option can be given multiple times.
122+
123+
Use
124+
.B dsc-datatool --list
125+
to see a list of modules and the man-page of each generator for
126+
information about options.
127+
.TP
128+
.B --list
129+
List the available generators, transformers and outputs then exit.
130+
.TP
131+
.BI "--skipped-key " SKIPPED_KEY
132+
Set the special DSC skipped key. (default to "-:SKIPPED:-")
133+
.TP
134+
.BI "--skipped-sum-key " SKIPPED_SUM_KEY
135+
Set the special DSC skipped sum key. (default to "-:SKIPPED_SUM:-")
136+
.TP
137+
.B -v, --verbose
138+
Increase the verbose level, can be given multiple times.
139+
.TP
140+
.B -V, --version
141+
Display version and exit.
142+
.LP
143+
.SH EXAMPLE
144+
.EX
145+
dsc-datatool \\
146+
--server "$SERVER" \\
147+
--node "$NODE" \\
148+
--output ";InfluxDB;file=influx.txt;dml=1;database=dsc" \\
149+
--transform ";Labler;*;yaml=$HOME/labler.yaml" \\
150+
--transform ";ReRanger;rcode_vs_replylen;range=/64;pad_to=5" \\
151+
--transform ";ReRanger;qtype_vs_qnamelen;range=/16;pad_to=3" \\
152+
--transform ";ReRanger;client_port_range;key=low;range=/2048;pad_to=5" \\
153+
--transform ";ReRanger;edns_bufsiz,priming_queries;key=low;range=/512;pad_to=5;allow_invalid_keys=1" \\
154+
--transform ";ReRanger;priming_responses;key=low;range=/128;pad_to=4" \\
155+
--transform ";NetRemap;client_subnet,client_subnet2,client_addr_vs_rcode,ipv6_rsn_abusers;net=8" \\
156+
--generator client_subnet_country \\
157+
--generator ";client_subnet_authority;fetch=yes" \\
158+
--xml "$XML"
159+
.EE
160+
.SH "SEE ALSO"
161+
.BR dsc-datatool.conf (5),
162+
.BI dsc-datatool- [ generator | transformer | output ]
163+
.BR <name> (7)
164+
.SH AUTHORS
165+
Jerry Lundström, DNS-OARC
166+
.LP
167+
Maintained by DNS-OARC
168+
.LP
169+
.RS
170+
.I https://www.dns-oarc.net/tools/dsc
171+
.RE
172+
.LP
173+
.SH BUGS
174+
For issues and feature requests please use:
175+
.LP
176+
.RS
177+
\fIhttps://github.com/DNS-OARC/dsc-datatool/issues\fP
178+
.RE
179+
.LP
180+
For question and help please use:
181+
.LP
182+
.RS
183+
\fIhttps://lists.dns-oarc.net/mailman/listinfo/dsc\fP
184+
.RE
185+
.LP

‎man/man5/dsc-datatool.conf.5‎

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
.TH "dsc-datatool.conf" "5"
2+
.SH NAME
3+
dsc-datatool.conf \- Configuration file for dsc-datatool.
4+
.SH DESCRIPTION
5+
Not implemented.
6+
.SH "SEE ALSO"
7+
.BR dsc-datatool (1)
8+
.SH AUTHORS
9+
Jerry Lundström, DNS-OARC
10+
.LP
11+
Maintained by DNS-OARC
12+
.LP
13+
.RS
14+
.I https://www.dns-oarc.net/tools/dsc
15+
.RE
16+
.LP
17+
.SH BUGS
18+
For issues and feature requests please use:
19+
.LP
20+
.RS
21+
\fIhttps://github.com/DNS-OARC/dsc-datatool/issues\fP
22+
.RE
23+
.LP
24+
For question and help please use:
25+
.LP
26+
.RS
27+
\fIhttps://lists.dns-oarc.net/mailman/listinfo/dsc\fP
28+
.RE
29+
.LP

‎man/man7/dsc-datatool-generator-client_subnet_authority.7‎

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
.TH "dsc-datatool-generator client_subnet_authority" "7"
2+
.SH NAME
3+
client_subnet_authority \- Generates network authority information based on client subnet.
4+
.SH DESCRIPTION
5+
TODO
6+
.SH "SEE ALSO"
7+
.BR dsc-datatool (1)
8+
.SH AUTHORS
9+
Jerry Lundström, DNS-OARC
10+
.LP
11+
Maintained by DNS-OARC
12+
.LP
13+
.RS
14+
.I https://www.dns-oarc.net/tools/dsc
15+
.RE
16+
.LP
17+
.SH BUGS
18+
For issues and feature requests please use:
19+
.LP
20+
.RS
21+
\fIhttps://github.com/DNS-OARC/dsc-datatool/issues\fP
22+
.RE
23+
.LP
24+
For question and help please use:
25+
.LP
26+
.RS
27+
\fIhttps://lists.dns-oarc.net/mailman/listinfo/dsc\fP
28+
.RE
29+
.LP

0 commit comments

Comments
 (0)