Basic Shell Commands.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Linux"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Linux** is an open source version of the **UNIX operating system**. The open source nature of this operating system\n",
    "has led to many different versions of it. These versions are called **distributions**, often abbreviated as **distro**. Some popular Linux distributions are **Debian**, **Red Hat**, **Fedora**, and **Ubuntu**.\n",
    "\n",
    "The portion of the operating system that handles the computer resources (CPU, RAM and devices) is referred to as the **kernel**. When people refer to the Linux operating system they mean the kernel as well as the accompanying set of tools and libraries that is actually called **GNU**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "## Shell"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "The **Command Line Interface** (**CLI**), also known as **Shell**, is one way to interact with our computer. The default shell on Linux systems is usually **Bash**. Other shells include the Bourne shell (**sh**), the Korn shell (**ksh**), **tcsh**, **zsh**, and **fish**.\n",
    "\n",
    "A **shell** is a powerful user interface for **Unix-like operating systems**. It can interpret commands and run other programs. It also enables access to files, utilities, and applications, and is an interactive scripting language. Additionally, we can use a shell to automate tasks. Linux shell commands are used for navigating and working with files and directories. We can also use them for file compression and archiving.\n",
    "\n",
    "\n",
    "The **command line prompt** looks something like this:\n",
    "```bash\n",
    "shaur@JAMWINE:~/workspace$\n",
    "```\n",
    "* The first part of the command prompt `shaur` is the **username** followed by the **@**symbol. \n",
    "\n",
    "* The second part of the command prompt `JAMWINE` is the **server address** (or **machine name**) after the **@** sign. On cloud instances, it is randomly specified using a unique combination of words. \n",
    "\n",
    "* The information after the **colon** is an abbreviation of the current working directory.\n",
    "* The command line prompt ends with a **$**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Basic Linux and Bash Commands"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* The **~** or the **tilde symbol** represents the **home directory**.\n",
    "\n",
    "\n",
    "* **Dots** in bash are useful utilities that help with navigating the file system using the CLI. In an `ls -a` listing:\n",
    "    * **.** refers to the **current directory**.\n",
    "    * **..** refers to the **parent directory**.\n",
    "\n",
    "\n",
    "* A **relative path** is related to the **current working directory**. \n",
    "\n",
    "\n",
    "* An **absolute path** is represented as starting from the **root directory** by leading with the **/**.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "### Getting information"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "#### return your user name\n",
    "`whoami`\n",
    "\n",
    "#### return your user and group id\n",
    "`id`\n",
    "\n",
    "#### return operating system name, username, and other info\n",
    "`uname -a`\n",
    "\n",
    "#### display reference manual for a command\n",
    "`man top`\n",
    "\n",
    "#### get help on a command\n",
    "`curl --help`\n",
    "\n",
    "#### return the current date and time\n",
    "`date`\n",
    "\n",
    "#### find out the path of the command 'bash'\n",
    "`which bash`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "### Navigating and working with directories"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "#### A regular sorted list of files in the directory\n",
    "`ls`\n",
    "\n",
    "#### A more complete list with permissions\n",
    "`ls -l`\n",
    "\n",
    "#### Hidden files were included\n",
    "`ls -a`\n",
    "\n",
    "#### A comma separated list\n",
    "`ls -m`\n",
    "\n",
    "#### Included directories and their contents\n",
    "`ls -R`\n",
    "\n",
    "#### list files and directories by date, newest last\n",
    "`ls -lrt`\n",
    "\n",
    "#### return present working directory\n",
    "`pwd`\n",
    "\n",
    "#### make a new directory\n",
    "`mkdir new_folder`\n",
    "\n",
    "#### change the current directory: up one level\n",
    "`cd ..`\n",
    "\n",
    "#### change the current directory: home\n",
    "`cd ~ or cd`\n",
    "\n",
    "#### change the current directory: some other path\n",
    "`cd another_directory`\n",
    "\n",
    "#### remove directory (recursive)\n",
    "`rmdir -r temp_directory`\n",
    "\n",
    "#### find files in home directory with suffix 'sh'\n",
    "`find ~/ -name '*.sh'`\n",
    "\n",
    "#### find files in home directory that are greater in size than 100 bytes\n",
    "To specify bigger than, we need to put a **+** in front of the 100.\n",
    "\n",
    "`find ~/ -size +100`\n",
    "\n",
    "* We may specify **k** for **kilobytes**, **M** for **megabytes** next to the size (e.g. `find ~/ -size +10k`).\n",
    "* For a range, we can specify both a minimum and maximum file size. \n",
    "\n",
    "#### find files between 10 and 50 bytes\n",
    "`find ~/ -size +10 -size -50`\n",
    "\n",
    "#### Find files or directories in ~/workspace/code that have been modified in the last year\n",
    "`find ~/workspace/code -mtime -365`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Working with files"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### copy a file\n",
    "`cp file.txt new_path/new_name.txt`\n",
    "\n",
    "#### copy a directory\n",
    "`cp -r my_directory/ new_path/new_copied_directory`\n",
    "\n",
    "#### change file name or path\n",
    "`mv this_file.txt that_path/that_file.txt`\n",
    "\n",
    "#### move all our .txt files into the code directory\n",
    "`mv *.txt code`\n",
    "\n",
    "#### remove a file verbosely\n",
    "`rm this_old_file.txt -v`\n",
    "\n",
    "#### create an empty file, or update existing file's timestamp\n",
    "`touch a_new_file.txt`\n",
    "\n",
    "#### change/modify file permissions to 'execute' for all users\n",
    "`chmod +x my_script.sh`\n",
    "\n",
    "####  add write permission for all users for the file sample.txt\n",
    "`sudo chmod ugo+w sample.txt`\n",
    "\n",
    "* `sudo` or **super user do** combined with the `chmod` command allows to change file permissions.\n",
    "* `ugo` stands for `user`, `group` and `other`\n",
    "\n",
    "#### Giving different privileges to user, group, other\n",
    "`chmod g-w,o-x sample.txt`\n",
    "\n",
    "`chmod u=rwx,g=r,o=r sample.txt`\n",
    "\n",
    "`chmod 756 sample.txt`\n",
    "\n",
    "* Permissions can be reduced to a **3 digit number**:\n",
    "\n",
    "<img src='imgs/permissions.png' alt='permissions' width=500 height=500>\n",
    "\n",
    "    * The total value of 7 equals the rwx permissions (sum of the read, write and execute permission)\n",
    "    * The read permission has a value of 4, the write a value of 2 and execute a value of 1 for a total of 7.\n",
    "    * The total value of 5 equals the r-x, 4 for read, - is 0, 1 for execute.\n",
    "    * The total value of 6 equals the rw-, 4 for read, 2 for write and 0 for execute.\n",
    "\n",
    "\n",
    "#### remove the execution permissions\n",
    "`chmod u-x my_script.sh`\n",
    "\n",
    "#### get count of lines in file\n",
    "`wc -l table_of_data.csv`\n",
    "\n",
    "#### get count of words in file\n",
    "`wc -w my_essay.txt`\n",
    "\n",
    "#### get count of characters in file\n",
    "`wc -m some_document.txt`\n",
    "\n",
    "#### return lines matching a pattern from files matching a filename pattern - whole words only\n",
    "`grep -w 'hello' \\*.txt`\n",
    "\n",
    "#### return lines matching a pattern from files matching a filename pattern - case insensitive and whole words only\n",
    "`grep -iw 'hello' \\*.txt`\n",
    "\n",
    "#### return file names with lines matching the pattern 'hello' from files matching a filename pattern\n",
    "`grep -l 'hello' \\*.txt`\n",
    "\n",
    "#### return lines that do not contain printf in the c files in the code/src/ directory\n",
    "`grep -v 'printf' code/src/*.c`\n",
    "\n",
    "#### provide access information about a file\n",
    "`stat sample.txt`\n",
    "\n",
    "#### provide system information about a file\n",
    "`stat -f sample.txt`\n",
    "\n",
    "#### information about the file type\n",
    "`file sample.txt`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "### Printing file and string contents"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "#### print file contents\n",
    "`cat my_shell_script.sh`\n",
    "\n",
    "#### print file contents page-by-page, displays text by one screen at a time\n",
    "`more ReadMe.txt`\n",
    "\n",
    "`more -d ReadMe.txt`\n",
    "\n",
    "* The `-d` parameter instructs the more command to put a prompt at the bottom telling you to press space for more text or q to quit.\n",
    "\n",
    "#### displays text by allowing scrolling\n",
    "`less ReadMe.txt`\n",
    "\n",
    "#### print first N lines of file\n",
    "`head -10 data_table.csv`\n",
    "\n",
    "#### print last N lines of file\n",
    "`tail -10 data_table.csv`\n",
    "\n",
    "#### print string\n",
    "`echo \"I am not a robot\"`\n",
    "\n",
    "#### print variable value\n",
    "`echo \"I am $USERNAME\"`\n",
    "\n",
    "---\n",
    "The **printf** command formats and prints data. Here is a summary of the common parameters used with `printf`:\n",
    "\n",
    "* `%d` integer number printed in decimal \n",
    "* `%f` floating point number \n",
    "* `%c` character \n",
    "* `%s` string \n",
    "\n",
    "Unlike `echo`, `printf` does not send an automatic newline at the end of the output. The printf command allows us to optionally assign the result to a variable rather than outputting to the screen (which is useful in Bash\n",
    "scripts)\n",
    "\n",
    "#### print a formatted string\n",
    "`printf \"%s got %s wrong answer(s)\\n\" \"Jane\" \"1\"`\n",
    "\n",
    "#### Print the result of a math equation\n",
    "`printf \"%d\\n\" $((8+4))`\n",
    "\n",
    "#### Print only the first 4 digits beyond the decimal point of a floating point number\n",
    "`printf \"%.4f\\n\" 3.1415926535`\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "### Compression and archiving"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "#### gzip compression tool\n",
    "* Zip the large text file: `gzip file1.txt`\n",
    "* Get information about the compression using the `-l` option: `gzip -l file1.txt.gz`\n",
    "* Unzip the file: `gzip -d file1.txt.gz`\n",
    "\n",
    "The above commands are similar for **bzip2** and **xz** compression tools.\n",
    "\n",
    "#### archive a set of files\n",
    "`tar -cvf my_archive.tar.gz file1 file2 file3`\n",
    "\n",
    "Some common **tar** flags are:\n",
    "* `-c`\tCreate an archive\n",
    "* `-f`\tUse archive file\n",
    "* `-r`\tAppend to an archive\n",
    "* `-t`\tList contents of an archive\n",
    "* `-v`\tVerbose output\n",
    "* `-x`\tExtract contents of an archive\n",
    "* `-z`\tCompress the archive using gzip\n",
    "\n",
    "#### list the contents of the images.tar file\n",
    "`tar -tvf images.tar`\n",
    "\n",
    "#### Archive all the files in the images directory to a file named imagedir.cpio\n",
    "`cd images`\n",
    "\n",
    "`ls | cpio -o > imagedir.cpio`\n",
    "\n",
    "`ls -l imagedir.cpio`\n",
    "\n",
    "* The **cpio command** is used to copy files into and out of archives. The name refers to its functionality, **copy in copy out**. It is similar to `tar` but there are some distinct differences:\n",
    "    * `cpio` is **not recursive** by default\n",
    "    * `cpio` will **not overwrite newer data file** whereas the default for `tar` is to overwrite\n",
    "    \n",
    "    \n",
    "* Common cpio flags are:\n",
    "    * **-o**:\tcopy out mode copies files into an archive\n",
    "    * **-i**:\tcopy in mode copies files out of an archive\n",
    "    * **-p**:\tcopy pass copies files from one directory tree to another\n",
    "\n",
    "#### Extract a compressed archive to a different directory - the directory must already exist\n",
    "`mkdir extracted`\n",
    "\n",
    "`tar -zxvf images.tar.gz -C extracted`\n",
    "\n",
    "#### compress a set of files\n",
    "`zip my_zipped_files.zip file1 file2`\n",
    "\n",
    "`zip my_zipped_folders.zip directory1 directory2`\n",
    "\n",
    "* The **zip** command archives and compresses a set of files. It is a popular cross platform tool and that makes it ideal for sharing files with people on different systems. `zip` is **not recursive** by default, if we want it to zip everything in a directory, we need to specify the `-r` flag.\n",
    "\n",
    "`zip -r my_zipped_folders.zip directory`\n",
    "\n",
    "#### extract files from a compressed zip archive\n",
    "`unzip my_zipped_file.zip`\n",
    "\n",
    "`unzip my_zipped_file.zip -d extract_to_this_directory`\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "### Performing network operations"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "#### print hostname\n",
    "`hostname`\n",
    "\n",
    "#### send packets to URL and print response\n",
    "`ping www.google.com`\n",
    "\n",
    "#### display or configure system network interfaces\n",
    "`ifconfig`\n",
    "\n",
    "`ip`\n",
    "\n",
    "#### display contents of file at a URL\n",
    "`curl <url>`\n",
    "\n",
    "#### download file from a URL\n",
    "`wget <url>`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "### Pipes and Filters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "#### chain filter commands using the pipe operator\n",
    "`ls | sort -r`\n",
    "\n",
    "#### pipe the output of manual page for ls to head to display the first 20 lines\n",
    "`man ls | head -20`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "### Shell and Environment Variables"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "#### list all shell variables\n",
    "`set`\n",
    "\n",
    "#### define a shell variable called my_planet and assign value Earth to it\n",
    "`my_planet=Earth`\n",
    "\n",
    "#### display shell variable\n",
    "`echo $my_planet`\n",
    "\n",
    "#### list all environment variables\n",
    "`env`\n",
    "\n",
    "#### environment vars: define/extend variable scope to child processes\n",
    "`export my_planet`\n",
    "\n",
    "`export my_galaxy='Milky Way'`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "### Metacharacters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "* **\\*** represents **any number of characters**\n",
    "* **?** represents **any single character**\n",
    "* **\\[ \\]** represents a **range**, can be `[1-3]` or `[1,2,3]`\n",
    "\n",
    "#### comments\n",
    "`#The shell will not respond to this message`\n",
    "\n",
    "#### command separator\n",
    "`echo 'here are some files and folders'; ls`\n",
    "\n",
    "#### file name expansion wildcard\n",
    "`ls *.json`\n",
    "\n",
    "#### single character wildcard\n",
    "`ls file_2021-06-??.json`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "### Quoting"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "#### single quotes - interpret literally\n",
    "`echo 'My home directory can be accessed by entering: echo $HOME'`\n",
    "\n",
    "#### double quotes - interpret literally, but evaluate metacharacters\n",
    "`echo \"My home directory is $HOME\"`\n",
    "\n",
    "#### backslash - escape metacharacter interpretation\n",
    "`echo \"This dollar sign should render: \\$\"`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "### I/O Redirection"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "The **>** character or **stdout redirection operator** changes the **stdout** to be a file with the specified name rather than the console.\n",
    "\n",
    "\n",
    "The **>> redirection operato**r does an append rather than an overwrite. It will create the file if it doesn’t exist but if it does, it will append to it."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "#### redirect output to file\n",
    "`echo 'Write this text to file x' > x.txt`\n",
    "\n",
    "#### append output to file\n",
    "`echo 'Add this line to file x' >> x.txt`\n",
    "\n",
    "#### redirect standard error to file\n",
    "`bad_command_1 2 > error.log`\n",
    "\n",
    "#### append standard error to file\n",
    "`bad_command_2 2 >> error.log`\n",
    "\n",
    "#### redirect file contents to standard input\n",
    "`tr “[a-z]” “[A-Z]” < a_text_file.txt`\n",
    "\n",
    "#### the input redirection above is equivalent to\n",
    "`cat a_text_file.txt | tr “[a-z]” “[A-Z]”`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "### Command Substitution"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "#### capture output of a command and echo its value\n",
    "`THE_PRESENT=$(date)`\n",
    "\n",
    "`echo \"There is no time like $THE_PRESENT\"`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "### Command line arguments"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "`./My_Bash_Script.sh arg1 arg2 arg3`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "### Batch vs. concurrent modes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "#### run commands sequentially\n",
    "`start=$(date);`\n",
    "\n",
    "`./MyBigScript.sh;`\n",
    "\n",
    "`end=$(date)`\n",
    "\n",
    "#### run commands in parallel\n",
    "`./ETL_chunk_one_on_these_nodes.sh & ./ETL_chunk_two_on_those_nodes.sh`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "### Scheduling jobs with Cron"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "The **cron daemon** (**crond**) is a system-managed executable that runs in memory and is used to schedule tasks. The job scheduling syntax is: (**minute, hour, day of month, month, day of week**)\n",
    "`m h dom mon dow command`\n",
    "\n",
    "#### open crontab editor\n",
    "`crontab -e`\n",
    "\n",
    "#### append the date/time to file every Sunday at 6:15 pm\n",
    "`15 18 * * 0 date >> sundays.txt`\n",
    "\n",
    "(**\\*** means any)\n",
    "\n",
    "#### run a shell script on the first minute of the first day of each month\n",
    "`1 0 1 * * ./My_Shell_Script.sh`\n",
    "\n",
    "#### back up your home directory every Monday at 3 am\n",
    "`0 3 * * 1 tar -cvf my_backup_path\\my_archive.tar.gz $HOME\\`\n",
    "\n",
    "#### deploy your cron job\n",
    "- Close the crontab editor and save the file\n",
    "\n",
    "#### list all cron jobs\n",
    "`crontab -l`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "### User Input"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "#### Wait for user to enter a name, and save the entered name into the variable 'name'\n",
    "`echo -n \"Enter your name :\";`\n",
    "\n",
    "`read name;`\n",
    "\n",
    "`echo \"Welcome $name\"`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "### Monitoring performance and status"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "#### list selection of or all running processes and their PIDs\n",
    "`ps`\n",
    "\n",
    "`ps -e`\n",
    "\n",
    "`ps au`\n",
    "\n",
    "#### display resource usage, provides a dynamic real-time view of a running system\n",
    "`top`\n",
    "\n",
    "Running the `top` command provides the following information:\n",
    "- **Tasks**: A list of the number of current processes the computer is running. The top program is the one labeled as **running** while other software such as the bash shell is **paused**. **Zombie programs** are the ones that fail to quit or **crash** because of a bug or a system memory overload.\n",
    "\n",
    "\n",
    "- **PID**: The **process identication number**. Running or Zombie programs can be killed by using this unique id.\n",
    "\n",
    "\n",
    "- **USER**: The user that started the task. For example, the root user tasks, also called **super user** are running in the background. \n",
    "\n",
    "\n",
    "- **%CPU** & **%MEM**: Display the percentage of the computer’s Central Processing Unit usage and the Random Access Memory or RAM usage. These indicators can help us identify which program is making the computer laggy or slow.\n",
    "\n",
    "\n",
    "- **COMMAND**: The name of the program (software)\n",
    "\n",
    "#### estimates and displays the disk space used by files\n",
    "`du-h`\n",
    "\n",
    "#### list mounted file systems and usage\n",
    "`df-Th`\n",
    "\n",
    "* `-T` for type and `h` displays the output in a more human readable form\n",
    "\n",
    "#### displays the total amount of free and used memory\n",
    "`free`\n",
    "\n",
    "#### Display the amount of memory in megabytes\n",
    "`free -m`\n",
    "\n",
    "#### Display the amount of memory in gigabytes\n",
    "`free -g`\n",
    "\n",
    "#### find processes that are an exact match of the search string\n",
    "`pidof abc`\n",
    "\n",
    "(assuming `abc` is the process)\n",
    "\n",
    "#### find processes that contain the string\n",
    "`pgrep xyz`\n",
    "\n",
    "(assuming `xyz` is the process)\n",
    "\n",
    "#### kill processes that contain the string\n",
    "`pkill xyz`\n",
    "\n",
    "(assuming `xyz` is the process)\n",
    "\n",
    "####  find information about all available block devices\n",
    "`lsblk`\n",
    "\n",
    "* In the case of the virtual environment, we mostly see **loop** in this list. These are **loopback devices**, which are **virtual devices** and mean that all our data is stored on a different system. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "### Stream Editor (sed)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "The name **sed** is short for **stream editor**. The `sed` command is most often used for finding and replacing, or searching and deleting. The sed utility in its simplest form works like `grep`, where we can use regular expressions to find a string in a file. It can also provide a substitute string for the string we found, or delete every line that matches a string.\n",
    "\n",
    "#### find a string in a file\n",
    "`sed -n '/text/p' sample.txt`\n",
    "\n",
    "* The `-n` means it will only produce output when explicitly told to via the `p` command (the default is to print each line)\n",
    "* The `'/text/p'` tells it to print out lines that have the word `text` in them.\n",
    "\n",
    "#### delete underscores in a file (replace the `_` with nothing)\n",
    "`sed 's%_%%g' myfile.txt`\n",
    "\n",
    "* `sed` is the command.\n",
    "* `'s%_%%g'` is the pattern where:\n",
    "    * **s** means **substitute**.\n",
    "    * The **%** is the **delimiter** (we can use any characters here).\n",
    "    * The **search pattern** follows the first **%**.\n",
    "    * The **replace pattern** follows the second **%**.\n",
    "    * The **g** stands for **global replace** - replace all occurrences in the file.\n",
    "    \n",
    "#### delete underscores in a file (replace the `_` with nothing) and send the output to a file instead of the console\n",
    "`sed 's%_%%g' myfile.txt > newfile.txt`\n",
    "\n",
    "#### delete underscores in a file (replace the `_` with nothing) and save it in the original file\n",
    "`sed -i 's%_%%g' myfile.txt`\n",
    "\n",
    "#### Change the name of one of the characters\n",
    "`sed 's%Christopher%Chris%g' myfile.txt`\n",
    "\n",
    "#### deleting lines\n",
    "`sed '45,54d' myfile.txt`\n",
    "\n",
    "* The `'45,54'` represents lines from 45 to 54. The comma indicates a range.\n",
    "* The **d** lets `sed` know it will be a deletion\n",
    "\n",
    "#### deleting all blank lines\n",
    "`sed '/^$/d' myfile.txt`\n",
    "\n",
    "* **^** indicates the beginning of the line \n",
    "* **$** indicates the end of the line \n",
    "\n",
    "(Since there is nothing between the two, it means a blank line.)\n",
    "\n",
    "#### deleting from a specific line to the end of the file\n",
    "`sed '/So I tried/,$d' myfile.txt`\n",
    "\n",
    "#### deleting everything after a specific line and save it “in place”\n",
    "`sed -i '1,/So I tried/!d' myfile.txt`\n",
    "\n",
    "* The `-i` command line option means edit in place\n",
    "* The `1,/So I tried/` specifies a range from the first line up to and including “So I tried”.\n",
    "* The **!d** means things in this range will not be deleted but everything else will"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "### awk"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "The **awk** command provides a way to search for a pattern, and perform actions on the found text. `awk` reads lines from the input file one at a time. The line is scanned for each pattern in the program and if there is a match the associated action is executed.\n",
    "\n",
    "Either the pattern or the action can be omitted, but not both.\n",
    "* If the pattern is omitted, then the action is performed for every line.\n",
    "* If the action is omitted, then all lines that match the pattern will be printed out.\n",
    "\n",
    "A few built in AWK variables are:\n",
    "\n",
    "* `$0` The entire line - not including the newline at the end\n",
    "* `$1..$n` The fields in a line (delimited by the field separator)\n",
    "* `FNR` Current line number - just spans the current file\n",
    "* `FS` Field separator - default is space\n",
    "* `NF` Number of fields\n",
    "* `NR` Current line number - spans multiple files\n",
    "* `RS` Record separator - the default is newline\n",
    "\n",
    "\n",
    "AWK special patterns are:\n",
    "* `BEGIN` Startup actions\n",
    "* `END` Cleanup actions\n",
    "\n",
    "#### count the number of occurrences of the word “Pooh” in the text\n",
    "`awk '/Pooh/{x++}END{print x}' myfile.txt`\n",
    "\n",
    "* `awk` is the command\n",
    "* `/Pooh/` is the pattern\n",
    "* `{x++}END{print x}` is the action\n",
    "\n",
    "#### number of words in a file\n",
    "`awk '{ total = total + NF }; END {print total}' myfile.txt`\n",
    "\n",
    "#### number of lines in a file\n",
    "`awk 'END{print NR}' myfile.txt`\n",
    "\n",
    "#### pull out items in field separated data (such as csv)\n",
    "`awk -F\",\" '{if (NR!=1){ print $1 \" wrong answers \" $5 \" out of total \" $8 }}' data.csv`\n",
    "\n",
    "* The option `-F` sets the field separator, the default is space. \n",
    "* `$` specifies which column is required (such as `$5`)\n",
    "\n",
    "#### print out just the second column of the file\n",
    "`awk -F\",\" '{print $2}' data.csv`\n",
    "\n",
    "#### Print out the number of answered assessments for “Jane Smith”\n",
    "`awk -F\",\" '/Jane Smith/{print $6}' data.csv`\n",
    "\n",
    "#### Create a lower case version of the file testupper.txt called testlower.txt\n",
    "`awk '{print tolower($0)}' < testupper.txt > testlower.txt`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "### Soft and Hard links"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "**Hard links** can only be created for regular files, not for directories or special files. They cannot span multiple file systems.\n",
    "\n",
    "#### Create a hard link to myfile.txt using `ln`\n",
    "`ln myfile.txt test/myfile.txt`\n",
    "\n",
    "* This will create a new hard link of myfile.txt to test/myfile.txt. On modifying test/myfile.txt file, we will notice that the changes also appear on the original myfile.txt file as both are the same files by nature.\n",
    "\n",
    "\n",
    "* The same is true for changing permissions on one of the file names, it will change them for both.\n",
    "\n",
    "A **soft link** is a special file that points to an existing file. Soft links are also called symbolic links and they can span multiple file systems. The `ln` command with the `-s` argument will create a symbolic link.\n",
    "\n",
    "#### Create a soft link to myfile.txt\n",
    "`ln -s myfile.txt myfilesym.txt`\n",
    "\n",
    "The **soft links** and **hard links** seemingly work the same. The one drawback with soft links is that if the original file is deleted, the soft link is broken and we are left with a dangling link.\n",
    "\n",
    "A hard link is an additional name for the original file that references to the target file through inode (index node). On the other hand, Soft link is different from the original file and is an alternative for it, but it does not use inode. A hard link remains valid even if the target file is deleted.\n",
    "\n",
    "* The **inode** number refers to the data structure that stores all the file information (metadata) other than the name. The number is assigned when a file is created."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "### Other Commands"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "####  Lists open files\n",
    "`lsof`\n",
    "\n",
    "#### displays all processes running on the system, along with their command line arguments\n",
    "`htop`\n",
    "\n",
    "#### sleep for 30 seconds\n",
    "`sleep 30`\n",
    "\n",
    "#### view a list all the possible signals available with the kill command\n",
    "`kill -l`\n",
    "\n",
    "* All processes should contain code that will handle a kill signal, these are called **signal handlers**. A typical action of a signal handler might be to delete temporary files or prompt to save changes. The exceptions to this are **SIGKILL** and **SIGSTOP** which are immediate and cannot be handled, ignored or blocked.\n",
    "\n",
    "\n",
    "* If we do not specify a signal to the **kill** (or **pkill**) command, the default is **SIGTERM** which allows the application to clean up before quitting.\n",
    "\n",
    "\n",
    "* The most commonly used signals are:\n",
    "    * **SIGINT** - has a value `2` and is similar to pressing `Ctrl+C`, it may be ignored by the process\n",
    "    * **SIGHUP** - has a value `1`. This is a **hang-up signal** and is used to report that the user’s terminal is disconnected\n",
    "    * **SIGQUIT** - has a value `3` and is similar to **SIGINT** but with the ability to produce a core dump\n",
    "    * **SIGKILL** - has a value `9`. This forces the process to terminate immediately and cannot be ignored\n",
    "    * **SIGTERM** - has a value `13`. This gives the application time to shutdown gracefully and may be ignored\n",
    "\n",
    "#### Pause the running jobs\n",
    "Use `CTRL-Z` \n",
    "\n",
    "\n",
    "#### filters out the “Permission Denied” errors\n",
    "For example, on executing this command: `find / -name \"*.conf\" 2>/dev/null`, the **2>/dev/null** at the end of the command filters out the “Permission Denied” errors. The `2` means **stderr** and the `/dev/null` is the **null device** which means throw it away.\n",
    "\n",
    "#### Add a user\n",
    "`sudo useradd testname`\n",
    "\n",
    "* The added new user can be seen in `/etc/passwd` configuration file. Files that are copied over to a new user’s home directory are stored in `/etc/skel` (for skeleton). They are copied over the `/etc/profile` directory for the new user.\n",
    "\n",
    "* The file **.bash_profile** is read and executed when Bash is invoked as an interactive login shell. The **.bashrc** file is executed for an interactive non-login shell.\n",
    "\n",
    "* The **adduser** command wraps the **useradd** functionality with a script and adds other useful features we would want when adding a user such as prompting us for a password and other details and creating a directory for the user in the home directory.\n",
    "\n",
    "\n",
    "#### Setup a password\n",
    "`sudo passwrd testname`\n",
    "\n",
    "* Encrypted passwords are stored in the file `/etc/shadow`. Account expiration information is also stored in that file.\n",
    "\n",
    "#### switch to different user\n",
    "`su testname`\n",
    "\n",
    "#### Delete a user\n",
    "`sudo userdel testname`\n",
    "\n",
    "#### displays a list of currently logged in users and time of login\n",
    "`who`\n",
    "\n",
    "#### displays information about the users currently on a machine, and the processes they are running\n",
    "`w`\n",
    "\n",
    "#### displays user and group ids\n",
    "`id`\n",
    "\n",
    "#### Create a new Group\n",
    "`sudo groupadd mygroup`\n",
    "\n",
    "#### Add users to a group\n",
    "`sudo groupadd mygroup`\n",
    "\n",
    "#### Create a new Group\n",
    "`sudo adduser test1 mygroup`\n",
    "\n",
    "#### View the group information\n",
    "`cat /etc/group`\n",
    "\n",
    "* Group information is stored in the `/etc/group` file. Newest additions are at the bottom.\n",
    "\n",
    "#### Delete a user from a group\n",
    "`sudo deluser test3 mygroup`\n",
    "\n",
    "#### Change a group’s name\n",
    "`sudo groupmod -n mynewgroup mygroup`\n",
    "\n",
    "#### Delete a group\n",
    "`sudo groupdel mynewgroup`\n",
    "\n",
    "#### Default Permissions - umask\n",
    "The **umask** command sets the default file permissions for newly created files in our current session.\n",
    "\n",
    "`umask u=rw,g=r,o=r`\n",
    "\n",
    "#### view the umask value in symbolic form\n",
    "`umask -S`\n",
    "\n",
    "#### view attributes of a file\n",
    "`lsattr`\n",
    "\n",
    "#### change attributes of a file\n",
    "`sudo chattr +i myfile`\n",
    "\n",
    "* `chattr`, or **change attribute** is mostly used to keep files secure and prevent them from being deleted accidentally or overwritten. The `chattr` command can only be run by super users and therefore must be preceded by the command `sudo`. These file attributes are stored in a file’s metadata properties.\n",
    "\n",
    "\n",
    "* Few attributes of `chattr` are:\n",
    "    * `a` - File can only be opened in append mode\n",
    "    * `i` - Make a file immutable (uneditable)\n",
    "    * `S` - File will be synchronously updated on the disk\n",
    "    * `u` - Makes it possible to undelete the file\n",
    "    \n",
    "    \n",
    "* Use a `+` to add to the existing attributes of the files, a `-` to remove an attribute, and an `=` to overwrite the existing attributes.\n",
    "\n",
    "\n",
    "#### setup users and a group\n",
    "```\n",
    "sudo useradd janedoc\n",
    "sudo useradd joedoc\n",
    "sudo groupadd documentation\n",
    "sudo adduser janedoc documentation\n",
    "sudo adduser joedoc documentation\n",
    "```\n",
    "\n",
    "#### view the groups\n",
    "`cat /etc/group`\n",
    "\n",
    "#### get ACL entries for file\n",
    "`getfacl file1.txt`\n",
    "\n",
    "* An **Access Control List** (**ACL**) allows us to apply specific privileges to files and directories without needing to change owners or groups. It is a way to specify privileges for people/groups who are not the owner.\n",
    "\n",
    "\n",
    "#### Give the group documentation all access to file1.txt\n",
    "`setfacl -m \"g:documentation:rwx\" file1.txt`\n",
    "\n",
    "#### Remove the ACL settings \n",
    "`setfacl -b file1.txt`\n",
    "\n",
    "#### Change the owner of file\n",
    "`sudo chown user1 file1.txt`\n",
    "\n",
    "`sudo chown -R user2 mydir`\n",
    "\n",
    "#### Change the group owner of a file or folder\n",
    "`sudo chgrp group1 file1.txt`\n",
    "\n",
    "#### Set the SUID for the file\n",
    "`chmod u+s file1.sh`\n",
    "\n",
    "* **Special permissions** are a level above the user, group, anyone permissions.\n",
    "\n",
    "\n",
    "* **SUID** stands for **set owner user id**. A file with an SUID permission will always execute as if run by the user who owns the file. \n",
    "\n",
    "\n",
    "* For example, `/usr/bin/passwd` has an SUID permission set by default. There's an `s` in place of an `x` in the execute permission of the user and the file name has a `red background`. This is important because users need to be able to change their own passwords, and with this bit set they don’t have to be the root user to do so.\n",
    "\n",
    "\n",
    "#### Set the SGID for the file\n",
    "`chmod g+s file1.sh`\n",
    "\n",
    "* This is similar to the **SUID** - a file with an **SGID permission** will always execute as if run by the group who owns the file. In some versions of UNIX, this setting is called **GUID**. If this permission is set on a directory, any files created in the directory will have their group ownership set to that of the owner of the directory\n",
    "\n",
    "\n",
    "* For example, `/usr/bin/crontab` or the crontab executable file has the **SGID** permission.\n",
    "\n",
    "\n",
    "* There is a capitol `S` in the place of the execution privileges for the group. This is because we set the **SGID** and there were **no execution privileges** for the group in place. To fix this, execute `chmod g+x file1.sh`\n",
    "\n",
    "\n",
    "* There's another special permission called the **sticky bit permission**, which applies to directories only. When it is set, only the owner of a file and the root user can delete a file in that directory. This setting is particularly useful for shared directories. For example, the `/tmp` directory."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "## Process States"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "In Linux, there are five possible states a process may be in:\n",
    "\n",
    "<img src='imgs/process_states.png' alt='process_states' width=500 height=500>\n",
    "\n",
    "Processes always start off in the **Running** or **Runnable state**. After it starts it might change states to one of the **sleeping states** if it needs to wait for resources or signals.\n",
    "\n",
    "### Running or Runnable R\n",
    "When the process is running, it is using the CPU to execute instructions. It can also be in the **runnable state** which means it is in the scheduling queue for using the CPU.\n",
    "\n",
    "### Uninterruptible Sleep D and Interruptable Sleep S\n",
    "A process which needs to wait for resources such as IO or data or network requests will enter sleep mode. This allows other processes to use the CPU while the process waits for the resources. The difference between the two types of sleep states is that the **Interruptable Sleep** state will react to both the availability of the resources it needs or to signals. The **Uninterruptible Sleep** state only reacts to the availability of the resources.\n",
    "\n",
    "### Stopped T\n",
    "A process will enter the **stopped state** when it receives either the **SIGSTOP** or **SIGTSTP** signals. The **SIGSTOP** is not ignorable but a process can choose to ignore the **SIGTSTP** signal. The **SIGCONT** signal returns the process to a running state.\n",
    "\n",
    "### Zombie Z\n",
    "When a process is terminated or has completed, it sends a **SIGCHLD** signal to the parent process and enters a **Zombie state**. The parent process is responsible for clearing the process from the process table."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "## Types of processes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "There are two types of computer processes: \n",
    "* Foreground processes\n",
    "* Background processes\n",
    "\n",
    "#### Foreground processes\n",
    "A **foreground process** is different from a **background process** in 2 ways:\n",
    "1. Some foreground processes show an interface, through which the user can interact with the program. In Linux, the command line interface (CLI) is that user interface.\n",
    "\n",
    "\n",
    "2. The user must wait for one foreground process to complete before running another one.\n",
    "\n",
    "#### Background processes\n",
    "Unlike with a foreground process, the shell does not have to wait for a background process to end before it can run more processes. We can enter as many background commands one after another within the limit of the amount of memory available.\n",
    "\n",
    "#### Example: Using `&` ampersand\n",
    "For instance, after executing `sleep 30`, we cannot type another command in until the `sleep 30` command is\n",
    "complete. \n",
    "\n",
    "However, if we execute `sleep 30 & ps`, the **& ampersand** specifies that the command should run in the background and we have access to the command prompt right away. The `ps` command will list the running processes and we can see that `sleep` is one of them."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "## Job versus Process"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "A **process** is a program that is running on our computer. For example, the command `ls` is a process.\n",
    "\n",
    "Sometimes we may combine two processes in one command line by **piping** the output of one to the input of the next. These combined processes are called a **job**. \n",
    "\n",
    "For example: `grep abc sample.txt | sort &` shows us a sorted list of all the lines that contain the word\n",
    "`abc` in the file `sample.txt`. The `grep` command returns every occurrence of the word `abc` and that is piped to the `sort` command which returns a sorted list. The two processes in this list are `grep` and `sort` but they are run as one job. The `ampersand` at the end causes this job to run in the background and it prints a notification that the job is done at the end.\n",
    "\n",
    "Linux provides the following commands to control running jobs:\n",
    "* **bg** - Resume specified job in the background\n",
    "* **fg** - Resume specified job in the foreground\n",
    "* **jobs** - List all current jobs\n",
    "* **kill** - Can be used to kill a job or a process\n",
    "* **wait** - Wait until the specified job is complete\n",
    "* **disown** - Removes specified job from the table of active jobs\n",
    "\n",
    "#### The `bg` Command\n",
    "This is useful for activities that might be tying up our interface for too long. We can suspend a process using `Ctrl-Z` and then send it to the background. If we don’t specify a job number to the `bg` command, it defaults to working on the most recently stopped process.\n",
    "`(sleep 30; printf \"I am awake\\n\")`\n",
    "* Type `Ctrl- Z` to suspend execution of the job.\n",
    "* Type `jobs` to view the status\n",
    "* Type `bg %1` to resume the job\n",
    "* Type `jobs` to see that it is running again, in the background\n",
    "\n",
    "#### The `fg` Command\n",
    "We can use the `fg` command to move a job to the foreground, it will operate on the most recently backgrounded process if we don’t specify a job number. Start the following job in the background:\n",
    "`COUNTER=0; while true; do printf \"Hello World %d\\n\" $COUNTER >> temp; sleep 1; let COUNTER++; done &`\n",
    "\n",
    "This command first initializes a `COUNTER` to `0`, then sets up a forever loop while `true`, it prints `Hello World` followed by the value of `COUNTER` in the current iteration, then it `sleeps` for a `second` (to slow it down) and increments `COUNTER`. The `done` marks the end of the loop.\n",
    "\n",
    "* Type `cat temp` at any time to see how many iterations it has gone through.\n",
    "* Type `fg` at the command line to see that it ties up the interface when it runs in the command line. At any point, we can type `CTRL-Z` to stop execution and then `cat temp` to see how many more iterations it went through.\n",
    "* Type `jobs` to see that job is stopped in the queue.\n",
    "\n",
    "\n",
    "##### Create a foreground job that prints out numbers to the console in a forever loop\n",
    "`COUNTER=0; while true; do printf $COUNTER; sleep 1; let COUNTER++; done`\n",
    "\n",
    "\n",
    "#### The kill command for jobs\n",
    "The `kill` command can be used to kill a job by using the `%` key before the job number. For example, `kill %1`\n",
    "\n",
    "#### The `wait` command\n",
    "This command will suspend script execution until all jobs running in background have terminated, or if a specific job or process number is specified, it will wait until that terminates. Returns the exit status of waited-for command.\n",
    "\n",
    "We can use the `wait` command to prevent a script from running before a background job finishes executing.\n",
    "* Start this process in the background, it creates a file with the numbers `1` to `20`:\n",
    "```for i in `seq 1 20`; do echo $i >> temp3; sleep 1; done &```\n",
    "* If we enter `cat temp3` right away, we’ll see that the first job hasn’t finished and all the numbers aren’t there.\n",
    "* Now after entering the command `wait %1; cat temp3`, the file won’t list out until the first job is complete. All the numbers will be there.\n",
    "\n",
    "#### The `disown `command\n",
    "The `disown` command removes a job from the table of active jobs. If we don’t specify a job number, it will disown the most recently launched job. Paste the commands below to set up four jobs running in the background: \n",
    "```\n",
    "sleep 100 &\n",
    "sleep 100 &\n",
    "sleep 100 &\n",
    "sleep 100 &\n",
    "jobs\n",
    "```\n",
    "On executing `disown %2` and then `jobs`, we will observe that jobs 1, 3 and 4 still remains and job 2 goes away. Typing `disown` again (this time without a job number), will disown the last job run, thus the remaining jobs are 1 and 3.\n",
    "\n",
    "\n",
    "### Sending a signal to a running job\n",
    "#### The `CTRL-Z` command\n",
    "The `CTRL-Z` command is used to suspend a foreground process. We need to suspend a foreground job in order to have access to the command line. Once we have access to the terminal, we can do other things such as send the job to the background `bg` or kill it. The `CTRL-Z` command does not work on background processes.\n",
    "\n",
    "#### The `CTRL-C` command\n",
    "We can use `CTRL-C` to kill a foreground process. To kill a background job, we must first bring it into the foreground `fg` and then type `CTRL-C`. It is simpler to use the `kill` command on the process. We can get a list of the running jobs with the `jobs` command.\n",
    "\n",
    "#### Example: Compare the outcomes of these commands\n",
    "Run this command: ```for i in `seq 1 100`; do echo $i; sleep 1; done```\n",
    "\n",
    "This sets up a loop that runs from `1` to `100`, then it `sleeps` for a `second` (to slow it down). The `done` marks the end of the loop. While this is running, type `CTRL-C` and then `jobs` to see which jobs are running. We will not see any `jobs` in the list.\n",
    "\n",
    "Now, run it again: ```for i in `seq 1 100`; do echo $i; sleep 1; done```. While this is running, type `CTRL-Z` and then `jobs` to see which jobs are running. With `CTRL-Z`, the job is still there but stopped.\n",
    "\n",
    "#### The `CTRL-D` command\n",
    "The `CTRL-D` command signals the end of input. We can use it to let a command know there will be no more input coming.\n",
    "\n",
    "\n",
    "### Setting process priority\n",
    "\n",
    "The **nice** and **renice** commands are used to set and change the priority of a Linux process. The term `nice` refers to the fact that high priority tasks are less nice because they don’t share resources as well as low priority tasks.\n",
    "Priority values range from **-20** (**highest **) to **19** (**lowest priority**).\n",
    "\n",
    "Using the `-l` flag with the `ps` command will list the priority of jobs, the priority is in the **NI column**. We can also see the `nice` values in the NI column when we use the `top` command.\n",
    "\n",
    "#### set the priority of a process before it is started\n",
    "`nice -n 19 sleep 100 &`\n",
    "\n",
    "* The `renice` is used set the priority of an already running process."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "## Filesystem Hierarchy Standard (FHS)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "The **Filesystem Hierarchy Standard** ensures that software packages running on a Linux system will know where to find essential files and directories.\n",
    "\n",
    "To view the directories of the root type in:\n",
    "```bash\n",
    "cd /\n",
    "ls\n",
    "```\n",
    "The contents of the root directory include:\n",
    "* **/bin** : Binaries or executables that are essential for functionality\n",
    "* **/boot** : Files needed to boot the system such as the Linux kernel\n",
    "* **/dev** : Device files - interface with hardware drivers\n",
    "* **/etc** : Host-specific system configuration - editable text\n",
    "* **/home** : User directories live under here\n",
    "* **/lib** : Common libraries\n",
    "* **/lib64** : Common 64 bit libraries\n",
    "* **/media** : Mount point for removable media\n",
    "* **/mnt** : Mount point for mounting a filesystem temporarily\n",
    "* **/opt** : Optional add on software\n",
    "* **/proc** : Keeps track of running processes\n",
    "* **/root** : Home directory for root user\n",
    "* **/run** : Data relevant to running processes\n",
    "* **/sbin** : System binaries or executables that are essential for functionality\n",
    "* **/srv** : Data for services provided by this system\n",
    "* **/sys** : A symbolic link to the kernel source tree\n",
    "* **/temp** : Temporary files that won't be persistent between reboots\n",
    "* **/var** : Variable files - things that will change as the operating system is being run such as logs and cache files\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "The **/bin directory** contains these commands:\n",
    "\n",
    "| Command | Description |\n",
    "|---|---|\n",
    "|cat |Concatenate files to standard output|\n",
    "|chgrp| Change file group ownership|\n",
    "|chmod| Change file access permissions|\n",
    "|chown| Change file owner and group|\n",
    "|cp| Copy files and directories|\n",
    "|date| Print or set the system data and time|\n",
    "|dd| Convert and copy a file|\n",
    "|df |Report filesystem disk space usage|\n",
    "|dmesg |Print or control the kernel message buffer|\n",
    "|echo |Display a line of text|\n",
    "|false |Do nothing, unsuccessfully|\n",
    "|hostname |Show or set the system’s host name|\n",
    "| kill | Send signals to processes|\n",
    "|ln |Make links between files|\n",
    "|login |Begin a session on the system|\n",
    "|ls |List directory contents|\n",
    "|mkdir |Make directories|\n",
    "|mknod |Make block or character special files|\n",
    "|more |Page through text|\n",
    "|mount |Mount a filesystem|\n",
    "|mv |Move/rename files|\n",
    "|ps |Report process status|\n",
    "|pwd |Print name of current working directory|\n",
    "|rm |Remove files or directories|\n",
    "\n",
    "The **/boot directory** contains everything required for the boot process. The exceptions are **configuration files** not needed at boot time and the **map installer**. The operating system kernel must be located in **/** or **/boot**.\n",
    "\n",
    "The **/dev directory** contains special or device files. These might also contain some **symbolic links** (**symlink**). \n",
    "\n",
    "* A **symlink** is a type of file in Linux that points to a different file or folder. Symlinks allow multiple access points to a file without needing multiple copies.\n",
    "\n",
    "The **/etc directory** contains all system related configuration files. **Configuration files** are editable text files, executable files should not be placed in this directory. The configuration files should be placed in subdirectories of the `/etc` folder grouped by the application they serve.\n",
    "\n",
    "The **/home directory** contains user directories.\n",
    "\n",
    "The **/lib directory** contains essential shared libraries and kernel modules. The `/lib` directory contains shared library images needed to boot the system and run the commands in `/bin` and `/sbin`.\n",
    "\n",
    "The **/media directory** is used for removable media such as USB drives and CD ROMS. This is typically used by the system. \n",
    "The **/mnt directory** is used for temporarily mounted file systems, mostly for user mounted items.\n",
    "External devices are **mounted** to the Linux files system at `/media` or `/mnt`.\n",
    "\n",
    "The **/opt directory** is reserved for the installation of add-on application software packages. \n",
    "\n",
    "The **/proc directory** or more often referred to as filesystem is built every time the system starts and it contains information about currently running processes, hardware and memory management. It represents the current state of the kernel.\n",
    "\n",
    "The **/run directory** contains system information about the system since it was booted. This directory must be cleared at the beginning of the boot process. This directory contains **.pid** files or **Processor identifier files** (**PID**).\n",
    "\n",
    "* A **PID** file consists of a process identifier in ASCII-encoded decimal, followed by a newline character.\n",
    "\n",
    "The **/sbin directory** is for system binaries. Binary files are executable programs and may also be referred to as **commands**. These binary files are essential for booting, restoring, recovering, and repairing the system. The `/sbin` directory must not contain any subdirectories. At the very least the `/sbin` directory must contain the shutdown command.\n",
    "\n",
    "* The **/bin directory** contains binaries (commands) that are for users as well as items needed to bring the system up or repair it. The **/sbin directory** contains binaries that the system uses for booting up. These are generally not run by users, you need sudo privileges to be able to run them.\n",
    "\n",
    "The **/srv directory** is used for data for services provided by the system.\n",
    "\n",
    "The **/sys directory** is a virtual file system where we can find information about devices, drivers, and other kernel components.\n",
    "\n",
    "The **/tmp directory** may be used by applications to store temporary files, files that an application does not expect to remain after it stops running. It is recommended (but not required) that the files in the `/tmp` are deleted whenever the\n",
    "system is rebooted.\n",
    "\n",
    "The data in the **/usr directory** is read only. It is for user runnable programs and user accessible data is located. The required directories, or symbolic links to directories in `/usr` are:\n",
    "\n",
    "| Command | Description |\n",
    "|---|---|\n",
    "|bin | Most user commands|\n",
    "|lib| Libraries|\n",
    "|local| Local hierarchy (empty after main installation)|\n",
    "|sbin| Non-vital system binaries|\n",
    "|share| Architecture-independent|\n",
    "\n",
    "The optional directories, or symbolic links to directories in `/usr` are:\n",
    "\n",
    "| Command | Description |\n",
    "|---|---|\n",
    "|games | Games and educational binaries|\n",
    "|include| Header files included by C programs|\n",
    "|lib <qual>| Alternate Format Libraries|\n",
    "|libexec| Binaries run by other programs|\n",
    "|src| Source code|\n",
    " \n",
    "* The **/bin directory** contains executable commands that are required by the system and **/usr/bin** contains executable files that are not required.\n",
    " \n",
    "* One of the useful commands in the **/usr/bin directory** is **whereis**. Runing `whereis python3` shows all the locations of python3 related files.\n",
    "  \n",
    "* The **/sbin directory** holds commands needed to boot the system. The **/usr/sbin direcory** contains program binaries for system administration which are not essential for the boot process.\n",
    "    \n",
    "\n",
    "The **/usr/local directory** is for use by the system administrator when installing software locally. The following directories, or symbolic links to directories, must be in `/usr/local`:\n",
    "   \n",
    "| Directory| Description |\n",
    "|---|---|\n",
    "| bin | Local binaries |\n",
    "| etc | Host-specific system configuration for local binaries |\n",
    "| games | Local game binaries |\n",
    "| include | Local C header files |\n",
    "| lib | Local libraries |\n",
    "| man | Local online manuals |\n",
    "| sbin | Local system binaries |\n",
    "| share | Local architecture-independent hierarchy |\n",
    "| src | Local source code |\n",
    "    \n",
    "    \n",
    "The **/var** or **variable data** hierarchy contains files to which the system writes data during the course of its operation. The following directories, or symbolic links to directories, are required in `/var`:\n",
    "    \n",
    "| Directory | Description |\n",
    "|---|---|\n",
    "| cache | Application cache data |\n",
    "| lib | Variable state information |\n",
    "| local | Variable data for /usr/local |\n",
    "| lock | Lock files |\n",
    "| log | Log files and directories |\n",
    "| opt | Variable data for /opt |\n",
    "| run | Data relevant to running processes |\n",
    "| spool | Application spool data |\n",
    "| tmp | Temporary files preserved between system reboots |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "## System Services"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "**System services** are processes that continuously run in the background, waiting for requests to come in.\n",
    "\n",
    "In Unix, **init** is the first process that starts, and it starts up other processes.\n",
    "\n",
    "For the most part, Linux is UNIX-like or UNIX-compatible. With only a few exceptions, the Linux and UNIX systems are very similar and it is easy to move between the two. Linux’s use of **systemd** instead of `init` is one of the few exceptions to this.\n",
    "\n",
    "### init\n",
    "**init** starts the machine in one of the **7 run levels** (from `0` to `6`) which indicate machine state.\n",
    "Example of standard Linux run levels:\n",
    "* `0` – Shut down\n",
    "* `1` – Single user mode\n",
    "* `3` – Multiple user mode with command line interface\n",
    "* `5` – Multiple user mode under GUI\n",
    "* `6` – Reboot/Restart\n",
    "* Run level `5` is the standard run level for most of the LINUX based systems.\n",
    "\n",
    "### systemd\n",
    "In Linux, **systemd** also known as **daemon** is the system and service manager for Linux systems. A **daemon** is a process that runs in the background. After the Linux kernel is booted, `systemd` is activated to manage the user space components know as a **unit**. The `systemd` tools are used to start, stop, enable and disable services and retrieve status.\n",
    "\n",
    "This move away from `init` was and is a highly debated move away from the Unix-based approach. Many disliked how bloated and inter-connected `systemd` was which is in direct opposition to the Unix philosophy of do one thing well. The uncharacteristically large and interconnected code base for `systemd` causes concerns for both reliability and security. Others embraced it as a fix to the existing, Unix-inspired `init` solution since it addressed many long-standing issues.\n",
    "\n",
    "Run the `top` command, we will see that `systemd` is the first process and it has a PID of `1`.\n",
    "\n",
    "The **systemctl** program is the tool used to manage `systemd`. The `systemctl` allows us to manage services, check statuses and change system states.\n",
    "\n",
    "### Systemctl commands\n",
    "\n",
    "#### Show system status\n",
    "`systemctl status`\n",
    "\n",
    "#### Show all unit files\n",
    "`systemctl list-unit-files`\n",
    "\n",
    "#### Show the status of a particular service\n",
    "`systemctl status cron.service`\n",
    "\n",
    "### Install a service\n",
    "The **sudo apt update** command updates a list with all the latest versions of packages. For example:\n",
    "```\n",
    "sudo apt update\n",
    "sudo apt install mariadb-server\n",
    "```\n",
    "\n",
    "#### View service status\n",
    "`systemctl status mariadb.service`\n",
    "\n",
    "#### Activate a service\n",
    "`sudo systemctl start mariadb.service`\n",
    "\n",
    "#### Deactivate a service immediately\n",
    "`sudo systemctl stop mariadb.service`\n",
    "\n",
    "#### Restart a service\n",
    "`sudo systemctl restart mariadb.service`\n",
    "\n",
    "#### Enables a service to be started on bootup\n",
    "`sudo systemctl enable mariadb.service`\n",
    "\n",
    "#### Disables a service from starting on bootup\n",
    "`sudo systemctl disable mariadb.service`\n",
    "\n",
    "#### Mask a service so it can’t be started\n",
    "`sudo systemctl mask mariadb.service`\n",
    "\n",
    "### Scheduling Services\n",
    "\n",
    "* The **cron** daemon is a Linux command that can be used to schedule tasks on our computer. The **crontab** (**cron table**) contains information about the date and time cron should run something. We can create a cron table using the command `crontab -e`, and will be presented with options for editors.\n",
    "\n",
    "\n",
    "* The **at** utility can be used for scheduling one off tasks. It does not output to the console when it is run, it is meant for tasks we might run when we are not logged in. The time and date syntax is very flexible but the time must come before the day. The **current day** is the default if day is not specified. The following words are recognized: `now`, `midnight`, `noon`, `teatime (4 PM)`, `AM`, `PM`. We use `24 hour time` as well. We can also use the following words to designate a time relative to `now` (using the **+** sign): `minutes`, `hours`, `days`, `weeks`, `months`, `years`. For example: `at now +1 minute -f timed.sh`\n",
    "\n",
    "#### Schedule a command that runs at 1 minute after the current time\n",
    "`echo \"hello\" > test | at now +1 minutes`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "## Shell Scripting"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "The `.sh` extension is a convention used to indicate that the file is a shell script.\n",
    "\n",
    "A **shell script** is an executable text file in which the first line usually has the form of an interpreter directive. The **interpreter directive** is also known as a `shebang directive`, and has the following form:\n",
    "```linux\n",
    "#!interpreter [optional argument]\n",
    "```\n",
    "**Interpreter** is an absolute path to an `executable program`, and the **optional argument** is a string representing a `single argument`.\n",
    "\n",
    "Shell scripts are scripts that invoke a shell program. For example:\n",
    "* `#!/bin/sh` invokes the Bourne shell or other compatible shell program, from the bin directory.\n",
    "* `#!/bin/bash` aka the **bash shebang** invokes the Bash shell. \n",
    "\n",
    "**Shebang directives** aren't limited to shell programs. For example, we could create a python script with the following directive:\n",
    "\n",
    "*`#!/usr/bin/env python3`\n",
    "\n",
    "**NOTE**: While running a script, we need a **./** before the file name. This ensures that the current directory is included in the **Path** such that the linux finds the executable file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": false,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}