Merge pull request #292 from kellyjonbrazil/dev

Dev v1.22.0
kellyjonbrazil · Sep 27, 2022 · b877cc9 · b877cc9
2 parents f1383b5 + 32fddce
commit b877cc9
Show file tree

Hide file tree

Showing 422 changed files with 32,759 additions and 3,766 deletions.
diff --git a/CHANGELOG b/CHANGELOG
@@ -1,5 +1,66 @@
 jc changelog
 
+20220926 v1.22.0
+- Add /proc file parsers for linux. Support for the following files:
+  `/proc/buddyinfo`
+  `/proc/consoles`
+  `/proc/cpuinfo`
+  `/proc/crypto`
+  `/proc/devices`
+  `/proc/diskstats`
+  `/proc/filesystems`
+  `/proc/interrupts`
+  `/proc/iomem`
+  `/proc/ioports`
+  `/proc/loadavg`
+  `/proc/locks`
+  `/proc/meminfo`
+  `/proc/modules`
+  `/proc/mtrr`
+  `/proc/pagetypeinfo`
+  `/proc/partitions`
+  `/proc/slabinfo`
+  `/proc/softirqs`
+  `/proc/stat`
+  `/proc/swaps`
+  `/proc/uptime`
+  `/proc/version`
+  `/proc/vmallocinfo`
+  `/proc/vmstat`
+  `/proc/zoneinfo`
+  `/proc/driver/rtc`
+  `/proc/net/arp`
+  `/proc/net/dev`
+  `/proc/net/dev_mcast`
+  `/proc/net/if_inet6`
+  `/proc/net/igmp`
+  `/proc/net/igmp6`
+  `/proc/net/ipv6_route`
+  `/proc/net/netlink`
+  `/proc/net/netstat`
+  `/proc/net/packet`
+  `/proc/net/protocols`
+  `/proc/net/route`
+  `/proc/net/unix`
+  `/proc/<pid>/fdinfo/<fd>`
+  `/proc/<pid>/io`
+  `/proc/<pid>/maps`
+  `/proc/<pid>/mountinfo`
+  `/proc/<pid>/numa_maps`
+  `/proc/<pid>/smaps`
+  `/proc/<pid>/stat`
+  `/proc/<pid>/statm`
+  `/proc/<pid>/status`
+- Magic syntax support for `/proc` files
+- Enhance bash and zsh autocompletions for `/proc` files
+- Enhance `free` parser to support `-w` option integer conversions
+- Fix `ini` and `kv` parsers so they don't change keynames to lower case
+  NOTE: This can be a breaking change in your scripts
+- Fix `id` command parser to allow usernames and groupnames with spaces
+- Enhance metadata output to output metadata even when results are empty
+- Optimize tests
+- Optimize documentation build script
+
 20220829 v1.21.2
 - Fix IP Address string parser for older python versions that don't cleanly
   accept decimal input format - IPv6 fix (e.g. python 3.6)

diff --git a/EXAMPLES.md b/EXAMPLES.md
@@ -1637,18 +1637,18 @@ cat example.ini | jc --ini -p
 ```json
 {
   "bitbucket.org": {
-    "serveraliveinterval": "45",
-    "compression": "yes",
-    "compressionlevel": "9",
-    "forwardx11": "yes",
-    "user": "hg"
+    "ServeraLiveInterval": "45",
+    "Compression": "yes",
+    "CompressionLevel": "9",
+    "ForwardX11": "yes",
+    "User": "hg"
   },
   "topsecret.server.com": {
-    "serveraliveinterval": "45",
-    "compression": "yes",
-    "compressionlevel": "9",
-    "forwardx11": "no",
-    "port": "50022"
+    "ServeraLiveInterval": "45",
+    "Compression": "yes",
+    "CompressionLevel": "9",
+    "ForwardX11": "no",
+    "Port": "50022"
   }
 }
 ```

diff --git a/README.md b/README.md
@@ -70,7 +70,7 @@ values are converted, and, in some cases, additional semantic context fields are
 added.
 
 To access the raw, pre-processed JSON, use the `-r` cli option or the `raw=True`
-function parameter in `parse()`.
+function parameter in `parse()` when using `jc` as a python library.
 
 Schemas for each parser can be found at the documentation link beside each
 [**Parser**](#parsers) below.
@@ -133,14 +133,18 @@ on Github.
 `jc` accepts piped input from `STDIN` and outputs a JSON representation of the
 previous command's output to `STDOUT`.
 ```bash
-COMMAND | jc PARSER [OPTIONS]
+COMMAND | jc [OPTIONS] PARSER
+cat FILE | jc [OPTIONS] PARSER
+echo STRING | jc [OPTIONS] PARSER
 ```
 
 Alternatively, the "magic" syntax can be used by prepending `jc` to the command
-to be converted. Options can be passed to `jc` immediately before the command is
-given. (Note: command aliases and shell builtins are not supported)
+to be converted or in front of the absolute path for Proc files. Options can be
+passed to `jc` immediately before the command or Proc file path is given.
+(Note: command aliases and shell builtins are not supported)
 ```bash
 jc [OPTIONS] COMMAND
+jc [OPTIONS] /proc/<path-to-procfile>
 ```
 
 The JSON output can be compact (default) or pretty formatted with the `-p`
@@ -217,14 +221,15 @@ option.
 | `        --nmcli` | `nmcli` command parser                                  | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/nmcli)          |
 | `         --ntpq` | `ntpq -p` command parser                                | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/ntpq)           |
 | `       --passwd` | `/etc/passwd` file parser                               | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/passwd)         |
-| `      --pidstat` | `pidstat -h` command parser                             | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/pidstat)        |
-| `    --pidstat-s` | `pidstat -h` command streaming parser                   | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/pidstat_s)      |
+| `      --pidstat` | `pidstat -H` command parser                             | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/pidstat)        |
+| `    --pidstat-s` | `pidstat -H` command streaming parser                   | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/pidstat_s)      |
 | `         --ping` | `ping` and `ping6` command parser                       | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/ping)           |
 | `       --ping-s` | `ping` and `ping6` command streaming parser             | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/ping_s)         |
 | `     --pip-list` | `pip list` command parser                               | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/pip_list)       |
 | `     --pip-show` | `pip show` command parser                               | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/pip_show)       |
 | `        --plist` | PLIST file parser                                       | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/plist)          |
 | `     --postconf` | `postconf -M` command parser                            | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/postconf)       |
+| `         --proc` | `/proc/` file parser                                    | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/proc)           |
 | `           --ps` | `ps` command parser                                     | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/ps)             |
 | `        --route` | `route` command parser                                  | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/route)          |
 | `       --rpm-qi` | `rpm -qi` command parser                                | [details](https://kellyjonbrazil.github.io/jc/docs/parsers/rpm_qi)         |
@@ -278,7 +283,7 @@ option.
 | `-a`  | `--about`       | About `jc`. Prints information about `jc` and the parsers (in JSON or YAML, of course!)                             |
 | `-C`  | `--force-color` | Force color output even when using pipes (overrides `-m` and the `NO_COLOR` env variable)                           |
 | `-d`  | `--debug`       | Debug mode. Prints trace messages if parsing issues are encountered (use`-dd` for verbose debugging)                |
-| `-h`  | `--help`        | Help. Use `jc -h --parser_name` for parser documentation                                                            |
+| `-h`  | `--help`        | Help. Use `jc -h --parser_name` for parser documentation. Use twice to show hidden parsers (e.g. `-hh`)             |
 | `-m`  | `--monochrome`  | Monochrome output                                                                                                   |
 | `-M`  | `--meta-out`    | Add metadata to output including timestamp, parser name, magic command, magic command exit code, etc.               |                                                                        |
 | `-p`  | `--pretty`      | Pretty format the JSON output                                                                                       |
@@ -537,12 +542,12 @@ that case you can suppress the warning message with the `-q` cli option or the
 
 macOS:
 ```bash
-cat lsof.out | jc --lsof -q
+cat lsof.out | jc -q --lsof
 ```
 
 or Windows:
 ```bash
-type lsof.out | jc --lsof -q
+type lsof.out | jc -q --lsof
 ```
 
 Tested on:
@@ -586,7 +591,7 @@ documentation.
 
 ### arp
 ```bash
-arp | jc --arp -p          # or:  jc -p arp
+arp | jc -p --arp          # or:  jc -p arp
 ```
 ```json
 [
@@ -625,7 +630,7 @@ cat homes.csv
 ...
 ```
 ```bash
-cat homes.csv | jc --csv -p
+cat homes.csv | jc -p --csv
 ```
 ```json
 [
@@ -666,7 +671,7 @@ cat homes.csv | jc --csv -p
 ```
 ### /etc/hosts file
 ```bash
-cat /etc/hosts | jc --hosts -p
+cat /etc/hosts | jc -p --hosts
 ```
 ```json
 [
@@ -693,7 +698,7 @@ cat /etc/hosts | jc --hosts -p
 ```
 ### ifconfig
 ```bash
-ifconfig | jc --ifconfig -p          # or:  jc -p ifconfig
+ifconfig | jc -p --ifconfig          # or:  jc -p ifconfig
 ```
 ```json
 [
@@ -751,29 +756,29 @@ Port = 50022
 ForwardX11 = no
 ```
 ```bash
-cat example.ini | jc --ini -p
+cat example.ini | jc -p --ini
 ```
 ```json
 {
   "bitbucket.org": {
-    "serveraliveinterval": "45",
-    "compression": "yes",
-    "compressionlevel": "9",
-    "forwardx11": "yes",
-    "user": "hg"
+    "ServeraLiveInterval": "45",
+    "Compression": "yes",
+    "CompressionLevel": "9",
+    "ForwardX11": "yes",
+    "User": "hg"
   },
   "topsecret.server.com": {
-    "serveraliveinterval": "45",
-    "compression": "yes",
-    "compressionlevel": "9",
-    "forwardx11": "no",
-    "port": "50022"
+    "ServeraLiveInterval": "45",
+    "Compression": "yes",
+    "CompressionLevel": "9",
+    "ForwardX11": "no",
+    "Port": "50022"
   }
 }
 ```
 ### ls
 ```bash
-$ ls -l /usr/bin | jc --ls -p          # or:  jc -p ls -l /usr/bin
+$ ls -l /usr/bin | jc -p --ls          # or:  jc -p ls -l /usr/bin
 ```
 ```json
 [
@@ -809,7 +814,7 @@ $ ls -l /usr/bin | jc --ls -p          # or:  jc -p ls -l /usr/bin
 ```
 ### netstat
 ```bash
-netstat -apee | jc --netstat -p          # or:  jc -p netstat -apee
+netstat -apee | jc -p --netstat          # or:  jc -p netstat -apee
 ```
 ```json
 [
@@ -897,7 +902,7 @@ netstat -apee | jc --netstat -p          # or:  jc -p netstat -apee
 ```
 ### /etc/passwd file
 ```bash
-cat /etc/passwd | jc --passwd -p
+cat /etc/passwd | jc -p --passwd
 ```
 ```json
 [
@@ -923,7 +928,7 @@ cat /etc/passwd | jc --passwd -p
 ```
 ### ping
 ```bash
-ping 8.8.8.8 -c 3 | jc --ping -p          # or:  jc -p ping 8.8.8.8 -c 3
+ping 8.8.8.8 -c 3 | jc -p --ping          # or:  jc -p ping 8.8.8.8 -c 3
 ```
 ```json
 {
@@ -976,7 +981,7 @@ ping 8.8.8.8 -c 3 | jc --ping -p          # or:  jc -p ping 8.8.8.8 -c 3
 ```
 ### ps
 ```bash
-ps axu | jc --ps -p          # or:  jc -p ps axu
+ps axu | jc -p --ps          # or:  jc -p ps axu
 ```
 ```json
 [
@@ -1023,7 +1028,7 @@ ps axu | jc --ps -p          # or:  jc -p ps axu
 ```
 ### traceroute
 ```bash
-traceroute -m 2 8.8.8.8 | jc --traceroute -p
+traceroute -m 2 8.8.8.8 | jc -p --traceroute
 # or:  jc -p traceroute -m 2 8.8.8.8
 ```
 ```json
@@ -1088,7 +1093,7 @@ traceroute -m 2 8.8.8.8 | jc --traceroute -p
 ```
 ### uptime
 ```bash
-uptime | jc --uptime -p          # or:  jc -p uptime
+uptime | jc -p --uptime          # or:  jc -p uptime
 ```
 ```json
 {
@@ -1133,7 +1138,7 @@ cat cd_catalog.xml
   ...
 ```
 ```bash
-cat cd_catalog.xml | jc --xml -p
+cat cd_catalog.xml | jc -p --xml
 ```
 ```json
 {
@@ -1185,7 +1190,7 @@ spec:
       mode: ISTIO_MUTUAL
 ```
 ```bash
-cat istio.yaml | jc --yaml -p
+cat istio.yaml | jc -p --yaml
 ```
 ```json
 [

diff --git a/completions/jc_bash_completion.sh b/completions/jc_bash_completion.sh
@@ -4,7 +4,7 @@ _jc()
           jc_about_options jc_about_mod_options jc_help_options jc_special_options
 
     jc_commands=(acpi airport arp blkid chage cksum crontab date df dig dmidecode dpkg du env file finger free git gpg hciconfig id ifconfig iostat iptables iw jobs last lastb ls lsblk lsmod lsof lsusb md5 md5sum mdadm mount mpstat netstat nmcli ntpq pidstat ping ping6 pip pip3 postconf printenv ps route rpm rsync sfdisk sha1sum sha224sum sha256sum sha384sum sha512sum shasum ss stat sum sysctl systemctl systeminfo timedatectl top tracepath tracepath6 traceroute traceroute6 ufw uname update-alternatives upower uptime vdir vmstat w wc who xrandr zipinfo)
-    jc_parsers=(--acpi --airport --airport-s --arp --asciitable --asciitable-m --blkid --cef --cef-s --chage --cksum --crontab --crontab-u --csv --csv-s --date --df --dig --dir --dmidecode --dpkg-l --du --email-address --env --file --finger --free --fstab --git-log --git-log-s --gpg --group --gshadow --hash --hashsum --hciconfig --history --hosts --id --ifconfig --ini --iostat --iostat-s --ip-address --iptables --iso-datetime --iw-scan --jar-manifest --jobs --jwt --kv --last --ls --ls-s --lsblk --lsmod --lsof --lsusb --m3u --mdadm --mount --mpstat --mpstat-s --netstat --nmcli --ntpq --passwd --pidstat --pidstat-s --ping --ping-s --pip-list --pip-show --plist --postconf --ps --route --rpm-qi --rsync --rsync-s --sfdisk --shadow --ss --stat --stat-s --sysctl --syslog --syslog-s --syslog-bsd --syslog-bsd-s --systemctl --systemctl-lj --systemctl-ls --systemctl-luf --systeminfo --time --timedatectl --timestamp --top --top-s --tracepath --traceroute --ufw --ufw-appinfo --uname --update-alt-gs --update-alt-q --upower --uptime --url --vmstat --vmstat-s --w --wc --who --x509-cert --xml --xrandr --yaml --zipinfo)
+    jc_parsers=(--acpi --airport --airport-s --arp --asciitable --asciitable-m --blkid --cef --cef-s --chage --cksum --crontab --crontab-u --csv --csv-s --date --df --dig --dir --dmidecode --dpkg-l --du --email-address --env --file --finger --free --fstab --git-log --git-log-s --gpg --group --gshadow --hash --hashsum --hciconfig --history --hosts --id --ifconfig --ini --iostat --iostat-s --ip-address --iptables --iso-datetime --iw-scan --jar-manifest --jobs --jwt --kv --last --ls --ls-s --lsblk --lsmod --lsof --lsusb --m3u --mdadm --mount --mpstat --mpstat-s --netstat --nmcli --ntpq --passwd --pidstat --pidstat-s --ping --ping-s --pip-list --pip-show --plist --postconf --proc --proc-buddyinfo --proc-consoles --proc-cpuinfo --proc-crypto --proc-devices --proc-diskstats --proc-filesystems --proc-interrupts --proc-iomem --proc-ioports --proc-loadavg --proc-locks --proc-meminfo --proc-modules --proc-mtrr --proc-pagetypeinfo --proc-partitions --proc-slabinfo --proc-softirqs --proc-stat --proc-swaps --proc-uptime --proc-version --proc-vmallocinfo --proc-vmstat --proc-zoneinfo --proc-driver-rtc --proc-net-arp --proc-net-dev --proc-net-dev-mcast --proc-net-if-inet6 --proc-net-igmp --proc-net-igmp6 --proc-net-ipv6-route --proc-net-netlink --proc-net-netstat --proc-net-packet --proc-net-protocols --proc-net-route --proc-net-unix --proc-pid-fdinfo --proc-pid-io --proc-pid-maps --proc-pid-mountinfo --proc-pid-numa-maps --proc-pid-smaps --proc-pid-stat --proc-pid-statm --proc-pid-status --ps --route --rpm-qi --rsync --rsync-s --sfdisk --shadow --ss --stat --stat-s --sysctl --syslog --syslog-s --syslog-bsd --syslog-bsd-s --systemctl --systemctl-lj --systemctl-ls --systemctl-luf --systeminfo --time --timedatectl --timestamp --top --top-s --tracepath --traceroute --ufw --ufw-appinfo --uname --update-alt-gs --update-alt-q --upower --uptime --url --vmstat --vmstat-s --w --wc --who --x509-cert --xml --xrandr --yaml --zipinfo)
     jc_options=(--force-color -C --debug -d --monochrome -m --meta-out -M --pretty -p --quiet -q --raw -r --unbuffer -u --yaml-out -y)
     jc_about_options=(--about -a)
     jc_about_mod_options=(--pretty -p --yaml-out -y --monochrome -m --force-color -C)
@@ -67,6 +67,12 @@ _jc()
         fi
     done
 
+    # if "/pr[oc]" (magic for Procfile parsers) is in the current word, complete with files/directories in the path
+    if [[ "${cur}" =~ "/pr" ]]; then
+        _filedir
+        return 0
+    fi
+
     # if a parser arg is found anywhere in the line, only show options and help options
     for i in "${words[@]::${#words[@]}-1}"; do
         if [[ " ${jc_parsers[*]} " =~ " ${i} " ]]; then