useful commands

Content

1. Content
2. ddrescue
3. grep
4. Easy table generation for markdown
5. Explore disk usage
6. diff's
7. Find execute
8. Wacom Intuos Dual Monitor Config (Offset)
9. Columnise console/bash output
10. Recursive/partial copy - cp
11. Linux cut
12. Get date/time
13. Password generators
14. Get abs. filepath
15. Tee; split outputs
16. Find path of executables
17. curl
18. wget
19. Get open connections - netstat

---

ddrescue

Recover hard disks, scratched CDs, DVDs, ...

ddrescue -d -r 3 -b 2048 /dev/sr1 /media/sdd1/mystuff.iso /media/sdd1/mystuff.log
          ^    ^    ^^^^^^   ^      ^^^^^^^^^^^^^^^^^^^^^^  ^^^^^^^^^^^^^^^^^^^^^^^
          |    |       |     |       rescue file = output   log file (kind of index
    try 3 times before |  device                            of what is recovered)
    giving up          |  CD/DVD/..
          |            |
          |      Set block size 
          |      2048 = standard 
          |      for CD/DVD
          |  
          |
          |
-d: direct access to the drive without any caching

For more info: https://html5.litten.com/how-to-fix-recover-data-from-a-scratched-or-damaged-cd-or-dvd/

grep

Print lines matching a pattern. If no files are specified, or if - is given, grep searches standard input.

Interesting options

Invert match (exclude)

-v "THISwillBEexcluded"

Treat binary files as text

In case you have binaries with some bits of text in it you wish to search

-e

Extended regular expresions

This is shorter:

find . -type f | grep -vE ".dep|.dla|.bat" 

Instead of:

find . -type f | grep -v ".dep" | grep -v ".dla" | grep -v ".bat"

Print files with prefixed line numbers with grep

# Print files with prefixed line numbers
grep -n "." ./myFile.json

Full-text search

Performs a full text search recursivly in current folder

# Performs a full text search recursivly in current folder 
grep --color=auto -rnw '.' -e "$@"

Easy table generation for markdown

Install csv2md: pip3 install csv2md.

1. Create your table via LibreOffice Calc or MS Excel, ...
2. Save it as .csv
3. Run csv2md -d ';' filename (-d: use ; as delimiter)

Explore disk usage

ncdu is a command line tool to explore the current disk usage:

ncdu -x .
#    ^^ ^---------- path
#    ||
#  Handle only local filesystem

diff's

Of course Kdiff3 and Meld are great tools. When it comes to huge amounts of data and speed other tools might be more beneficial (note: in my case - diff'ing 2 x 1 TiB of data let Kdiff3 hang and Meld crash):

• diff -qr dir1/ dir2/
  • Reads both folders in parallel (Kdiff3 and Meld do sequential reads 👎 )
  • Good if you only want to see if a file differs
• git diff --no-index --summary dir1/ dir2
  • Good if you want to compare file contents
  • You can also just get a summary (--summary (just tells you if a file differs) or --compact-summary (additions/deletions style)) where you can hide the actual content diff's

Find exec

Some examples with find execute:

Replace invalid NTFS characters

Invalid characters

• https://docs.microsoft.com/en-us/windows/win32/intl/character-sets-used-in-file-names -> \, /, ., ?, and *
• https://stackoverflow.com/a/31976060/4773274 -> >, :, ", /, \, |, ? and *.

# Find all invalid file names
find ./  \( -iname "*:*" -o -iname "*|*" -o -iname "*\?*" -o -iname "*\**" \)
#       ^^^----------    ^^-- OR   ------^^---------------^^               ^^
#                   |----------- expressions -------------------------------
#                      (see: https://unix.stackexchange.com/a/50613/116710)

# Replacement example with perl-rename (for some distributions it's just `rename`)
find ./ -iname "*:*" -exec perl-rename -n 's/:/-/' {} \;
#                           dry-run ---^^  ^^^^^^----- perl substitute pattern

For perl substitute patterns see also https://perldoc.perl.org/perlre.html.

Wacom dual monitor config

Say your monitor is right to your other (main) display and you only want to use your graphics tablet on your right monitor:

#!/bin/bash
xsetwacom --set 20 MapToOutput "2560x1440+1920+0"

Columnise console/bash output

$ printf "flah ;klj jk;lj " | column -t -s ";"
flah   klj jk  lj

• -t : creates a table
• -s : separator

Recursive/partial copy - cp

cp -r --parents ./source ./dest

• --parents : Copies only folders needed until file
• -r : recursive - copies every file (found left and right) in this path

cut

Extract/cut information column-wise.

Usage

Input file:

## text.txt ##
root:x:0:0:root:/root:/bin/bash
daemon:x:1:1:daemon:/usr/sbin:/bin/sh
bin:x:2:2:bin:/bin:/bin/sh
…

Example usage:

$ cut -d : -f 1 text.txt
root
daemon
bin
…

Useful options

• -c, --characters=LIST select only these characters
• -d, --delimiter=DELIM use DELIM instead of TAB for field delimiter
• -f, --fields=LIST select only these fields; also print any line that contains no delimiter character, unless the -s option is specified
• -s, --only-delimited do not print lines not containing delimiters
• --output-delimiter=STRING use STRING as the output delimiter the default is to use the input delimiter

Get date/time

$ date +%F\ %T
2018-05-15 16:50:11

Password generators

apg

Generates random and memorisable passwords:

Sample output:

$ apg
Please enter some random data (only first 16 are significant)
(eg. your old password):>
ust1Glayduf (ust-ONE-Glayd-uf)
ojNumhis6 (oj-Num-his-SIX)
RemkibIr4 (Rem-kib-Ir-FOUR)
QuicHap7 (Quic-Hap-SEVEN)
MulOcnowit2 (Mul-Oc-now-it-TWO)
yegEkEc6 (yeg-Ek-Ec-SIX)

pwgen

An alternative ist pwgen:

$ pwgen 40
shee8joM5zeic8iohoosohrahfimi3aer9OazuCh
Food1olei6feeghohHeneechaiTifaiz9Eighaec

readlink

Get an absolute path of a file:

readlink -f file.txt

tee

Double output to file (>> to stdout, .. + file) (name comes from T-Splitter used in plumbing):

ls -l / | tee out.file | grep b
# __>>__________v>__________>>____
#               vv
#               vv
#           out.file

touch

"Touch" file access.

• Changes date to now
• Creates empty file if not existing -> This is a very convenient side-effect

touch -c <filename>

• -c : Do not create new files

watch

Execute commands periodically:

watch [options] command
Options (extract):
    -c, --color                            interpret ANSI color and style sequences
    -d, --differences[=<permanent>]        highlight changes between updates
    -n, --interval <secs>  seconds to wait between updates
    -p, --precise          attempt run command in precise intervals
    -t, --no-title         turn off header

Example 1

Print every second the current second ;)

$ watch -n 1 -pd 'date +%S'        # Clears screen after execution
Every 1.0s: date +%S                                                                                                                 PZI13774: Thu Aug 10 13:16:42 2017

42

Example 2

Monitor files in folder (>> monitor ls). For instance you copy something using rsync and want to check whats already copied.

watch ls

Get path of executable

Find paths of executables

$ whereis gparted
gparted: /usr/bin/gparted /usr/share/man/man8/gparted.8.gz
$ whereis cp
cp: /usr/bin/cp /usr/share/man/man1/cp.1p.gz /usr/share/man/man1/cp.1.gz

or:

$ which cp
/bin/cp

xargs

xargs reads items from the standard input, delimited by blanks and executes the command (default is /bin/echo) one or more times with any initial arguments followed by items read from standard input.

command1 | xargs [options] [command2]

Example

Find files in the current directory, exclude some file endings (using grep) and output each file content using cat:

find . -type f | grep -vE ".dep|.dla|.bat" | xargs cat

curl

Proxy

Configuration if you are behind a proxy server:

# ~/.curlrc
proxy = 192.0.2.111:2222
proxy-user = "<uname>:<password>"

netrc

• Allow different logins for destination/machine/sites
• Save credentials for commands like curl

# ~/.netrc
machine url.or.ip.org
	login <uname>
	password <password>

machine 192.168.0.123
	login <uname>
	password <password>

# This is a comment
# If you have the same credentials for multiple destinations/machines you have to have multiple entries

Notes:

• Use curl -u to use .netrc (-k to disable ssl verify)
  • Alternatively you can use curl -netrc-file /path/to/netrc/file
• Windows
  • On Windows it is _netrc (not .netrc) except you use MSYS2, etc. (then it is .netrc).
  • You must set the HOME environment variable on Windows to your home directory (where .netrc lies) that -u works

wget

Credentials for wget:

# Tell wget that it should use a proxy
use_proxy=yes

http_proxy=http://<uname>:<password>@192.0.1.123:1234
https_proxy=https://<uname>:<password>@192.0.1.123:1234

proxy_user=<uname>
proxy_password=<password>

# Credentials for access
http_user=<uname>
http_password=<password>

Get open connections - netstat

sudo netstat -tulpn

Results in:

Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 127.0.0.1:631           0.0.0.0:*               LISTEN      780/cupsd           
tcp6       0      0 ::1:631                 :::*                    LISTEN      780/cupsd           
udp        0      0 0.0.0.0:5353            0.0.0.0:*                           6267/chromium       
udp        0      0 0.0.0.0:5353            0.0.0.0:*                           296/avahi-daemon: r 
udp        0      0 192.168.1.200:68        0.0.0.0:*                           298/NetworkManager  
udp        0      0 0.0.0.0:33896           0.0.0.0:*                           296/avahi-daemon: r 
udp6       0      0 :::5353                 :::*                                6267/chromium       
udp6       0      0 :::5353                 :::*                                296/avahi-daemon: r 
udp6       0      0 fe80::368:3588:6aa5:546 :::*                                298/NetworkManager  
udp6       0      0 :::58036                :::*                                296/avahi-daemon: r