useful commands
-
Content
- ddrescue
- grep
- Easy table generation for markdown
-
Parallelise with GNU
parallel - Explore disk usage -
ncdu - diff's
- Find exec
- Remove files from file
- Wacom dual monitor config
- Columnise console/bash output
- Recursive/partial copy -
cp -
cut - Get date/time
- Password generators
readlinkteetouch- watch
- Example 2
- Get path of executable
-
xargs - curl
- wget
- Get open connections -
netstat -
socat
Recover hard disks, scratched CDs, DVDs, ...
ddrescue -d -r 3 -b 2048 /dev/sr1 /media/sdd1/mystuff.iso /media/sdd1/mystuff.log
^ ^ ^^^^^^ ^ ^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^^^
| | | | rescue file = output log file (kind of index
try 3 times before | device of what is recovered)
giving up | CD/DVD/..
| |
| Set block size
| 2048 = standard
| for CD/DVD
|
|
|
-d: direct access to the drive without any cachingFor more info: https://html5.litten.com/how-to-fix-recover-data-from-a-scratched-or-damaged-cd-or-dvd/
Print lines matching a pattern. If no files are specified, or if - is given, grep searches standard input.
-v "THISwillBEexcluded"
In case you have binaries with some bits of text in it you wish to search
-e
This is shorter:
find . -type f | grep -vE ".dep|.dla|.bat" Instead of:
find . -type f | grep -v ".dep" | grep -v ".dla" | grep -v ".bat"# Print files with prefixed line numbers
grep -n "." ./myFile.jsonPerforms a full text search recursivly in current folder
# Performs a full text search recursivly in current folder
grep --color=auto -rnw '.' -e "$@"Grep does not show the filename when it matches only one file. Force with /dev/null:
$ find . -name *.tf | grep zis-heart | grep -v version | xargs grep CI_TYPE
CI_TYPE = "dummy" // Define CI type
$ find . -name *.tf | grep zis-heart | grep -v version | xargs grep CI_TYPE /dev/null
./zis-heartbeat/main.tf: CI_TYPE = "dummy" // Define CI typeHide your (own) grep process when you search for a process:
$ ps -e -o pid,args | grep [s]omeprocess
-> [s] will get resolved (TODO check why this is so) as s by grep but not in ps
Install csv2md: pip3 install csv2md.
- Create your table via LibreOffice Calc or MS Excel, ...
- Save it as
.csv - Run
csv2md -d ';' filename(-d: use;as delimiter)
# apt-get install parallel
# pacman -S parallel
parallel -j20 -k 'echo {}; sleep 2' ::: {a..g}-
-j: max number of parallel jobs (j0spawns 252 jobs;j10spawns 2520 jobs) -
-k: Keep order of output sorted like input (if not set it prints the outputs immediately when one finishes (order can be mixed) -> no slow-down but it needs 4 file handles for each job; if it runs out of file handles, it will wait until it can fetch new handles -
--progress: shows progress and duration of each job -
--dry-run: shows the commands which would be executed -
--pipe: splits an input file in (by default 1 MB; change with--block 100m-> 100 MB blocks (basically the maximum possible)) blocks; by default it detects\nas block ends (can be overwritten by using--recend
See here. This allows to distribute tasks on several remote machines considering CPU count and memory just by using ssh and parallel.
See also here: https://www.gnu.org/software/parallel/parallel_examples.html
# Nested for loops
parallel echo {1} {2} ::: red green blue ::: S M L XL XXL | sort# 1| fast job - lots of overhead due to process spawning
seq -w 0 9999 | parallel touch pict{}.jpg
# 2| -X spawns jobs groups -> command must support multiple arguments as input
seq -w 0 9999 | parallel -X touch pict{}.jpg-> more about speed-ups see here: https://www.gnu.org/software/parallel/parallel_examples.html#example-speeding-up-fast-jobs
On a normal GNU/Linux system you can spawn 32000 jobs using this technique with no problems. To raise the 32000 jobs limit raise
/proc/sys/kernel/pid_maxto 4194303.
from https://www.gnu.org/software/parallel/parallel_examples.html#example-speeding-up-fast-jobs
Output of 1 and 2 in a dry run:
$ seq -w 0 3 | parallel --dry-run touch pict{}.jpg
touch pict0.jpg
touch pict1.jpg
touch pict2.jpg
touch pict3.jpg
$ seq -w 0 10 | parallel --dry-run -X touch pict{}.jpg
touch pict00.jpg pict01.jpg
touch pict02.jpg pict03.jpg
touch pict04.jpg pict05.jpg
touch pict06.jpg pict07.jpg
touch pict08.jpg pict09.jpg
touch pict10.jpg
$ seq -w 0 30 | parallel --dry-run -X touch pict{}.jpg
touch pict00.jpg pict01.jpg pict02.jpg pict03.jpg
touch pict04.jpg pict05.jpg pict06.jpg pict07.jpg
touch pict08.jpg pict09.jpg pict10.jpg pict11.jpg
touch pict12.jpg pict13.jpg pict14.jpg pict15.jpg
touch pict16.jpg pict17.jpg pict18.jpg pict19.jpg
touch pict20.jpg pict21.jpg pict22.jpg pict23.jpg
touch pict24.jpg pict25.jpg pict26.jpg pict27.jpg
touch pict28.jpg pict29.jpg pict30.jpg
ncdu (NCurses Disk Usage) is a command line tool to explore the current disk usage:
# Install
sudo apt install ncdu
# Usage
ncdu -x .
# ^^ ^---------- path
# ||
# Handle only local filesystem
Of course Kdiff3 and Meld are great tools. When it comes to huge amounts of data and speed other tools might be more beneficial (note: in my case - diff'ing 2 x 1 TiB of data let Kdiff3 hang and Meld crash):
-
diff -qr dir1/ dir2/- Reads both folders in parallel (Kdiff3 and Meld do sequential reads 👎 )
- Good if you only want to see if a file differs
-
git diff --no-index --summary dir1/ dir2- Good if you want to compare file contents
- You can also just get a summary (
--summary(just tells you if a file differs) or--compact-summary(additions/deletions style)) where you can hide the actual content diff's
Some examples with find execute:
Invalid characters
-
https://docs.microsoft.com/en-us/windows/win32/intl/character-sets-used-in-file-names ->
\,/,.,?, and* -
https://stackoverflow.com/a/31976060/4773274 ->
>,:,",/,\,|,?and*.
# Find all invalid file names
find ./ \( -iname "*:*" -o -iname "*|*" -o -iname "*\?*" -o -iname "*\**" \)
# ^^^---------- ^^-- OR ------^^---------------^^ ^^
# |----------- expressions -------------------------------
# (see: https://unix.stackexchange.com/a/50613/116710)
# Replacement example with perl-rename (for some distributions it's just `rename`)
find ./ -iname "*:*" -exec perl-rename -n 's/:/-/' {} \;
# dry-run ---^^ ^^^^^^----- perl substitute patternFor perl substitute patterns see also https://perldoc.perl.org/perlre.html.
Here we also specify the file type (via -name) to make sure we don't delete anything else:
find . -type f -name "*.jpg" -size -2k -delete-
-2k: Smaller than 2 KiB -
2k: Exactly 2 KiB -
+2k: Greater than 2 KiB
while IFS= read -r file ; do rm -- "$file" ; done < /path/to/file/which/contains/files/to/delete.txtSay your monitor is right to your other (main) display and you only want to use your graphics tablet on your right monitor:
#!/bin/bash
xsetwacom --set 20 MapToOutput "2560x1440+1920+0"$ printf "flah ;klj jk;lj " | column -t -s ";"
flah klj jk lj-
-t: creates a table -
-s: separator
cp -r --parents ./source ./dest
-
--parents: Copies only folders needed until file -
-r: recursive - copies every file (found left and right) in this path
Extract/cut information column-wise.
Input file:
## text.txt ##
root:x:0:0:root:/root:/bin/bash
daemon:x:1:1:daemon:/usr/sbin:/bin/sh
bin:x:2:2:bin:/bin:/bin/sh
…
Example usage:
$ cut -d : -f 1 text.txt
root
daemon
bin
…-
-c,--characters=LISTselect only these characters -
-d,--delimiter=DELIMuse DELIM instead of TAB for field delimiter -
-f,--fields=LISTselect only these fields; also print any line that contains no delimiter character, unless the -s option is specified -
-s,--only-delimiteddo not print lines not containing delimiters -
--output-delimiter=STRINGuse STRING as the output delimiter the default is to use the input delimiter
$ date +%F\ %T
2018-05-15 16:50:11Generates random and memorisable passwords:
Sample output:
$ apg
Please enter some random data (only first 16 are significant)
(eg. your old password):>
ust1Glayduf (ust-ONE-Glayd-uf)
ojNumhis6 (oj-Num-his-SIX)
RemkibIr4 (Rem-kib-Ir-FOUR)
QuicHap7 (Quic-Hap-SEVEN)
MulOcnowit2 (Mul-Oc-now-it-TWO)
yegEkEc6 (yeg-Ek-Ec-SIX)An alternative ist pwgen:
$ pwgen 40
shee8joM5zeic8iohoosohrahfimi3aer9OazuCh
Food1olei6feeghohHeneechaiTifaiz9EighaecGet an absolute path of a file:
readlink -f file.txtDouble output to file (>> to stdout, .. + file) (name comes from T-Splitter used in plumbing):
ls -l / | tee out.file | grep b
# __>>__________v>__________>>____
# vv
# vv
# out.file"Touch" file access.
- Changes date to
now - Creates empty file if not existing -> This is a very convenient side-effect
touch -c <filename>-
-c: Do not create new files
Execute commands periodically:
watch [options] command
Options (extract):
-c, --color interpret ANSI color and style sequences
-d, --differences[=<permanent>] highlight changes between updates
-n, --interval <secs> seconds to wait between updates
-p, --precise attempt run command in precise intervals
-t, --no-title turn off header
Print every second the current second ;)
$ watch -n 1 -pd 'date +%S' # Clears screen after execution
Every 1.0s: date +%S PZI13774: Thu Aug 10 13:16:42 2017
42Monitor files in folder (>> monitor ls). For instance you copy something using rsync and want to check whats already copied.
watch ls
Find paths of executables
$ whereis gparted
gparted: /usr/bin/gparted /usr/share/man/man8/gparted.8.gz
$ whereis cp
cp: /usr/bin/cp /usr/share/man/man1/cp.1p.gz /usr/share/man/man1/cp.1.gzor:
$ which cp
/bin/cpxargs reads items from the standard input, delimited by blanks and executes the command (default is /bin/echo) one or more times with any initial arguments followed by items read from standard input.
command1 | xargs [options] [command2]Find files in the current directory, exclude some file endings (using grep) and output each file content using cat:
find . -type f | grep -vE ".dep|.dla|.bat" | xargs catConfiguration if you are behind a proxy server:
# ~/.curlrc
proxy = 192.0.2.111:2222
proxy-user = "<uname>:<password>"- Allow different logins for destination/machine/sites
- Save credentials for commands like
curl
# ~/.netrc
machine url.or.ip.org
login <uname>
password <password>
machine 192.168.0.123
login <uname>
password <password>
# This is a comment
# If you have the same credentials for multiple destinations/machines you have to have multiple entriesNotes:
- Use
curl -uto use.netrc(-kto disable ssl verify)- Alternatively you can use
curl -netrc-file /path/to/netrc/file
- Alternatively you can use
- Windows
- On Windows it is
_netrc(not.netrc) except you use MSYS2, etc. (then it is.netrc). - You must set the
HOMEenvironment variable on Windows to your home directory (where.netrclies) that-uworks
- On Windows it is
Credentials for wget:
# Tell wget that it should use a proxy
use_proxy=yes
http_proxy=http://<uname>:<password>@192.0.1.123:1234
https_proxy=https://<uname>:<password>@192.0.1.123:1234
proxy_user=<uname>
proxy_password=<password>
# Credentials for access
http_user=<uname>
http_password=<password>sudo netstat -tulpnResults in:
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 127.0.0.1:631 0.0.0.0:* LISTEN 780/cupsd
tcp6 0 0 ::1:631 :::* LISTEN 780/cupsd
udp 0 0 0.0.0.0:5353 0.0.0.0:* 6267/chromium
udp 0 0 0.0.0.0:5353 0.0.0.0:* 296/avahi-daemon: r
udp 0 0 192.168.1.200:68 0.0.0.0:* 298/NetworkManager
udp 0 0 0.0.0.0:33896 0.0.0.0:* 296/avahi-daemon: r
udp6 0 0 :::5353 :::* 6267/chromium
udp6 0 0 :::5353 :::* 296/avahi-daemon: r
udp6 0 0 fe80::368:3588:6aa5:546 :::* 298/NetworkManager
udp6 0 0 :::58036 :::* 296/avahi-daemon: r
To see if it works you can execute the following command on the listener to check on your desired port:
tcpdump -i eth0 -s 1500 port 443Enables data transfers between two addresses using multiple protocols (e.g. TCP, UDP, SOCKS4, RAW IP, OpenSSL, other ifaces, ...).
┌──────────────────────┐ ┌──────────────────────┐
│ <Connector> │ │ <Listener> │
│ PROTO:ADDRESS:PORT │◄──►socat◄──►│ PROTO:ADDRESS:PORT │
│ │ │ │
└──────────────────────┘ └──────────────────────┘
Syntax:
socat [GENERAL_OPTIONS] <PROTO>:<PORT> <AddressType>:<ARGS>, <OPTIONS>
ARGS are usually prefixed by `CW` in the documentation (you need to strip away the `CW` part)
e.g.
socat -d -d TCP-LISTEN:2223 OPEN:inputDump.txt, creat
Create TCP listener to std::out
socat -d -d TCP-LISTEN:2223 --
-d: verbosity (repeat to increase:-d -d: fatal, error, notice) -
-: print to std::out
Connect to listener via IP and port and use std::in
socat -d -d TCP-CONNECT:191.1.1.123:2223 -
Send message to listener: hello listenerCreate a file:
socat -d -d TCP-LISTEN:2223 OPEN:inputDump.txt, creat
# <CMD>:<ARGS>, <OPTION>- flags
-
creat: Creat file -
trunc: Overwrite file if present -
excl: Do not overwrite if file present -
append: Append to file
-
Transfer file to listener
socat -d -d TCP-CONNECT:191.1.1.123:2223 FILE:/path/to/some/file/to/transfer.txtUse-cases:
- Circumvent firewall protocol/port restrictions (e.g. proxy UDP to TPC connection)
- Encryption
Create an openssl key:
openssl req -newkey rsa:2048 -nodes -keyout cert.key -x509 -days 1000 -out cert.crt
# Combine both files to make it usable for socat:
cat cert.key cert.crt > sslkey.pemAdd OpenSSL listener:
socat -d -d OPENSSL-LISTEN:2223, cert=sslkey.pem, verify=0-
verify=0: since it's self-signed
Connector (create a bash shell on the other host):
socat -d -d OPENSSL-CONNECT:191.1.1.123:2223, verify=0 EXEC:/bin/bashLet the connector connect to the listener via ssh:
# Create cert
openssl req -newkey rsa:2048 -nodes -keyout sshsocatcert.key -x509 -days 1000 -out sshsocatcert.crt
# Combine both files to make it usable for socat:
cat sshsocatcert.key sshsocatcert.crt > sshsocatcert.pem
# Listener
sudo socat -ddd OPENSSL-LISTEN:443,fork,cipher=aNULL,verify=0,cert=sshsocatcert.pem,compress=auto TCP-CONNECT:localhost:22
# Connector
ssh -o ProxyCommand='socat -dd STDIO OPENSSL-CONNECT:%h:443,compress=auto,cipher=HIGH,verify=0' 192.1.1.123Alternatively (you must do it if you use sshfs because it does not support the -o option) put the proxy config in the ~/.ssh.config file:
Host 192.1.1.123
ProxyCommand socat STDIO OPENSSL-CONNECT:%h:443,cipher=aNULL,verify=0,compress=autoNow you could mount a folder via sshfs like this (requires that you added the config (-> ~/.ssh/config) options like before):
sshfs 192.1.1.123:/ /tmp/mountFolderNote: As an alternative you could use rsync too; it reads the settings from ~/.ssh/config and applies it accordingly
Verify that port 443 is used:
sudo tcpdump -ni any port 443
# Now do something (like navigating within your mounted folder)
# Assuming you don't have any other traffic on 443 it should be very silent otherwise (in comparison to scanning port 22)No cert is provided; create a cert and add the relevant option. I.e.:
Create openssl key (if you haven't already):
openssl req -newkey rsa:2048 -nodes -keyout socatsshcert.key -x509 -days 100 -out socatsshcert.crt
# Combine both files to make it usable for socat:
cat socatsshcert.key socatsshcert.crt > socatsshcert.pemFor the OpenSsl listener run with cert:
socat OPENSSL-LISTEN:2223, cert=socatsshcert.pem, verify=0
# or similar
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License *.
Code (snippets) are licensed under a MIT License *.
* Unless stated otherwise