Development

docs/README.md

@@ -1,10 +1,10 @@
-The Python programming language is increasingly popular.  It is a
-versatile language for general purpose programming and accessible
-for novice programmers.  For more than a decade, Python has been
-used for a large number of systems-related programming tasks.
+Python is a very versatile programming language that has a wide range of
+applications.  This training concentrates on interaction with the
+operating system, the file system, other applications and the network.
+This is useful for systems programming, but also when you want to use
+Python as a coordination language for your workflows.
 This training introduces modules that are useful in that context.
 
-
 ## Learning outcomes
 
 When you complete this training you will
@@ -20,7 +20,7 @@ When you complete this training you will
 
 ## Schedule
 
-Total duration: 3.5 hours.
+Total duration: 4 hours.
 
   | Subject                                     | Duration |
   |---------------------------------------------|----------|
@@ -29,6 +29,7 @@ Total duration: 3.5 hours.
   | logging                                                           | 10 min.  |
   | interacting with the file system                                  | 40 min.  |
   | coffee break                                                      | 10 min.  |
+  | data formats                                                      | 30 min.  |
   | templates                                                         | 10 min.  |
   | interacting with processes                                        | 45 min.  |
   | interacting with remote systems using SSH                         | 30 min.  |

environment.yml

@@ -2,7 +2,9 @@ name: python_for_systems_programming
 channels:
   - defaults
 dependencies:
+  - fabric
   - jinja2
+  - jupyter
   - matplotlib
   - numpy
   - paramiko

python_for_systems_programming_linux64_conda_specs.txt

@@ -11,6 +11,7 @@ https://repo.anaconda.com/pkgs/main/linux-64/libstdcxx-ng-9.1.0-hdf63c60_0.conda
 https://repo.anaconda.com/pkgs/main/linux-64/libgcc-ng-9.1.0-hdf63c60_0.conda
 https://repo.anaconda.com/pkgs/main/linux-64/mkl-2019.4-243.conda
 https://repo.anaconda.com/pkgs/main/linux-64/expat-2.2.6-he6710b0_0.conda
+https://repo.anaconda.com/pkgs/main/linux-64/gmp-6.1.2-h6c8ec71_1.conda
 https://repo.anaconda.com/pkgs/main/linux-64/icu-58.2-h9c2bf20_1.conda
 https://repo.anaconda.com/pkgs/main/linux-64/jpeg-9b-h024ee3a_2.conda
 https://repo.continuum.io/pkgs/main/linux-64/libffi-3.2.1-hd88cf55_4.tar.bz2
@@ -26,8 +27,10 @@ https://repo.anaconda.com/pkgs/main/linux-64/glib-2.63.1-h5a9c865_0.tar.bz2
 https://repo.anaconda.com/pkgs/main/linux-64/libedit-3.1.20181209-hc058e9b_0.conda
 https://repo.anaconda.com/pkgs/main/linux-64/libpng-1.6.37-hbc83047_0.conda
 https://repo.anaconda.com/pkgs/main/linux-64/libxml2-2.9.9-hea5a465_1.conda
+https://repo.anaconda.com/pkgs/main/linux-64/pandoc-2.2.3.2-0.conda
 https://repo.anaconda.com/pkgs/main/linux-64/readline-7.0-h7b6447c_5.conda
 https://repo.anaconda.com/pkgs/main/linux-64/tk-8.6.8-hbc83047_0.conda
+https://repo.anaconda.com/pkgs/main/linux-64/zeromq-4.3.1-he6710b0_3.conda
 https://repo.anaconda.com/pkgs/main/linux-64/dbus-1.13.12-h746ee38_0.tar.bz2
 https://repo.anaconda.com/pkgs/main/linux-64/freetype-2.9.1-h8a8886c_1.conda
 https://repo.anaconda.com/pkgs/main/linux-64/gstreamer-1.14.0-hb453b48_1.conda
@@ -36,33 +39,76 @@ https://repo.anaconda.com/pkgs/main/linux-64/fontconfig-2.13.0-h9420a91_0.conda
 https://repo.anaconda.com/pkgs/main/linux-64/gst-plugins-base-1.14.0-hbbd80ab_1.conda
 https://repo.anaconda.com/pkgs/main/linux-64/python-3.7.5-h0371630_0.conda
 https://repo.anaconda.com/pkgs/main/linux-64/asn1crypto-1.2.0-py37_0.tar.bz2
+https://repo.anaconda.com/pkgs/main/noarch/attrs-19.3.0-py_0.tar.bz2
+https://repo.anaconda.com/pkgs/main/linux-64/backcall-0.1.0-py37_0.conda
 https://repo.anaconda.com/pkgs/main/linux-64/certifi-2019.9.11-py37_0.tar.bz2
+https://repo.anaconda.com/pkgs/main/noarch/decorator-4.4.1-py_0.tar.bz2
+https://repo.anaconda.com/pkgs/main/noarch/defusedxml-0.6.0-py_0.tar.bz2
+https://repo.anaconda.com/pkgs/main/linux-64/entrypoints-0.3-py37_0.conda
 https://repo.anaconda.com/pkgs/main/linux-64/idna-2.8-py37_0.conda
+https://repo.anaconda.com/pkgs/main/linux-64/invoke-1.3.0-py37_0.tar.bz2
+https://repo.anaconda.com/pkgs/main/linux-64/ipython_genutils-0.2.0-py37_0.conda
 https://repo.anaconda.com/pkgs/main/linux-64/kiwisolver-1.1.0-py37he6710b0_0.conda
 https://repo.anaconda.com/pkgs/main/linux-64/markupsafe-1.1.1-py37h7b6447c_0.conda
+https://repo.anaconda.com/pkgs/main/linux-64/mistune-0.8.4-py37h7b6447c_0.conda
+https://repo.anaconda.com/pkgs/main/linux-64/more-itertools-7.2.0-py37_0.conda
+https://repo.anaconda.com/pkgs/main/linux-64/pandocfilters-1.4.2-py37_1.conda
+https://repo.anaconda.com/pkgs/main/noarch/parso-0.5.1-py_0.tar.bz2
+https://repo.anaconda.com/pkgs/main/linux-64/pickleshare-0.7.5-py37_0.conda
+https://repo.anaconda.com/pkgs/main/noarch/prometheus_client-0.7.1-py_0.tar.bz2
 https://repo.anaconda.com/pkgs/main/linux-64/psutil-5.6.5-py37h7b6447c_0.tar.bz2
+https://repo.anaconda.com/pkgs/main/linux-64/ptyprocess-0.6.0-py37_0.conda
 https://repo.anaconda.com/pkgs/main/linux-64/pycparser-2.19-py37_0.conda
 https://repo.anaconda.com/pkgs/main/noarch/pyparsing-2.4.4-py_0.tar.bz2
 https://repo.anaconda.com/pkgs/main/noarch/pytz-2019.3-py_0.tar.bz2
+https://repo.anaconda.com/pkgs/main/linux-64/pyzmq-18.1.0-py37he6710b0_0.conda
 https://repo.anaconda.com/pkgs/main/linux-64/qt-5.9.7-h5867ecd_1.conda
+https://repo.anaconda.com/pkgs/main/linux-64/send2trash-1.5.0-py37_0.conda
 https://repo.anaconda.com/pkgs/main/linux-64/sh-1.12.14-py37_0.tar.bz2
 https://repo.anaconda.com/pkgs/main/linux-64/sip-4.19.8-py37hf484d3e_0.conda
 https://repo.anaconda.com/pkgs/main/linux-64/six-1.13.0-py37_0.tar.bz2
+https://repo.anaconda.com/pkgs/main/noarch/testpath-0.4.4-py_0.tar.bz2
 https://repo.anaconda.com/pkgs/main/linux-64/tornado-6.0.3-py37h7b6447c_0.conda
+https://repo.anaconda.com/pkgs/main/linux-64/wcwidth-0.1.7-py37_0.conda
+https://repo.anaconda.com/pkgs/main/linux-64/webencodings-0.5.1-py37_1.conda
 https://repo.anaconda.com/pkgs/main/linux-64/cffi-1.13.2-py37h2e261b9_0.tar.bz2
 https://repo.anaconda.com/pkgs/main/linux-64/cycler-0.10.0-py37_0.conda
+https://repo.anaconda.com/pkgs/main/linux-64/jedi-0.15.1-py37_0.conda
 https://repo.anaconda.com/pkgs/main/linux-64/mkl-service-2.3.0-py37he904b0f_0.conda
+https://repo.anaconda.com/pkgs/main/linux-64/pexpect-4.7.0-py37_0.conda
 https://repo.anaconda.com/pkgs/main/linux-64/pyqt-5.9.2-py37h05f1152_2.conda
+https://repo.anaconda.com/pkgs/main/linux-64/pyrsistent-0.15.5-py37h7b6447c_0.tar.bz2
 https://repo.anaconda.com/pkgs/main/noarch/python-dateutil-2.8.1-py_0.tar.bz2
 https://repo.anaconda.com/pkgs/main/linux-64/setuptools-41.6.0-py37_0.tar.bz2
+https://repo.anaconda.com/pkgs/main/linux-64/terminado-0.8.3-py37_0.tar.bz2
+https://repo.anaconda.com/pkgs/main/linux-64/traitlets-4.3.3-py37_0.tar.bz2
+https://repo.anaconda.com/pkgs/main/noarch/zipp-0.6.0-py_0.tar.bz2
 https://repo.anaconda.com/pkgs/main/linux-64/bcrypt-3.1.7-py37h7b6447c_0.tar.bz2
+https://repo.anaconda.com/pkgs/main/linux-64/bleach-3.1.0-py37_0.conda
 https://repo.anaconda.com/pkgs/main/linux-64/cryptography-2.8-py37h1ba5d50_0.tar.bz2
+https://repo.anaconda.com/pkgs/main/linux-64/importlib_metadata-0.23-py37_0.tar.bz2
 https://repo.anaconda.com/pkgs/main/noarch/jinja2-2.10.3-py_0.tar.bz2
+https://repo.anaconda.com/pkgs/main/linux-64/jupyter_core-4.6.1-py37_0.tar.bz2
 https://repo.anaconda.com/pkgs/main/linux-64/numpy-base-1.17.3-py37hde5b4d6_0.tar.bz2
+https://repo.anaconda.com/pkgs/main/noarch/pygments-2.4.2-py_0.tar.bz2
 https://repo.anaconda.com/pkgs/main/linux-64/pynacl-1.3.0-py37h7b6447c_0.conda
 https://repo.anaconda.com/pkgs/main/linux-64/wheel-0.33.6-py37_0.tar.bz2
+https://repo.anaconda.com/pkgs/main/linux-64/jsonschema-3.2.0-py37_0.tar.bz2
+https://repo.anaconda.com/pkgs/main/linux-64/jupyter_client-5.3.4-py37_0.tar.bz2
 https://repo.anaconda.com/pkgs/main/linux-64/paramiko-2.6.0-py37_0.tar.bz2
 https://repo.anaconda.com/pkgs/main/linux-64/pip-19.3.1-py37_0.tar.bz2
+https://repo.anaconda.com/pkgs/main/noarch/prompt_toolkit-2.0.10-py_0.tar.bz2
+https://repo.anaconda.com/pkgs/main/linux-64/fabric-2.5.0-py37_0.tar.bz2
+https://repo.anaconda.com/pkgs/main/linux-64/ipython-7.9.0-py37h39e3cac_0.tar.bz2
+https://repo.anaconda.com/pkgs/main/linux-64/nbformat-4.4.0-py37_0.conda
+https://repo.anaconda.com/pkgs/main/linux-64/ipykernel-5.1.3-py37h39e3cac_0.tar.bz2
+https://repo.anaconda.com/pkgs/main/linux-64/nbconvert-5.6.1-py37_0.tar.bz2
+https://repo.anaconda.com/pkgs/main/linux-64/jupyter_console-6.0.0-py37_0.conda
+https://repo.anaconda.com/pkgs/main/linux-64/notebook-6.0.2-py37_0.tar.bz2
+https://repo.anaconda.com/pkgs/main/noarch/qtconsole-4.6.0-py_0.tar.bz2
+https://repo.anaconda.com/pkgs/main/linux-64/widgetsnbextension-3.5.1-py37_0.conda
+https://repo.anaconda.com/pkgs/main/noarch/ipywidgets-7.5.1-py_0.tar.bz2
+https://repo.anaconda.com/pkgs/main/linux-64/jupyter-1.0.0-py37_7.conda
 https://repo.anaconda.com/pkgs/main/linux-64/matplotlib-3.1.1-py37h5429711_0.conda
 https://repo.anaconda.com/pkgs/main/linux-64/mkl_fft-1.0.15-py37ha843d7b_0.tar.bz2
 https://repo.anaconda.com/pkgs/main/linux-64/mkl_random-1.1.0-py37hd6b4f25_0.conda

source-code/README.md

@@ -0,0 +1,40 @@
+# Source code
+
+This is source code that is either used in the presentation, or was developed
+to create it.  There is some material not covered in the presentation as well.
+
+## Requirements
+
+* Python version: at least 3.6
+* Packages (names listed that can be used with `pip` or `conda` to install):
+  * jinja2
+  * jupyter
+  * matplotlib
+  * numpy
+  * paramiko
+  * psutil
+  * python=3.7.5
+  * sh
+
+## What is it?
+
+1. `application`: illustration of how to combine `configparser` and `argparse`.
+1. `cmd`: illustration of how to create a repl application.
+1. `code-evaluation`: illustrates how to evaluate a string containing
+    Python code at runtime.
+1. `command-Line-args`: illustration of how to use the argparse and the
+    click module to handle command line arguments.
+1. `config-parser`: illustrates how to use the ConfigParser module to handle
+    configuration files.
+1. `data-formats`: illustrates how to deal with data formats such as CSV
+    files, binary data and XML.
+1. `hydra`: Facebook Hydra application framework illustration.
+1. `logging`: illustration of Python's logging facilities.
+1. `file-system`: illustrations of interacting with the operating system
+    and the file system.
+1. `paramiko`: a few examples of using the Paramiko library for SSH
+    to remote hosts.
+1. `Sched`: scheduled execution of funcitons in Python.
+1. `Subprocess`: illustrates executing a shell command from a Python script
+    using the `subprocess` module.
+1. `xml-generator`: code to generate a random XML documents.

source-code/application/README.md

@@ -0,0 +1,22 @@
+# Application
+
+Example of a very simple application that prints random integers between
+two given values.  It uses argparse and configparser to handle configuration
+files and command line arguments.
+
+The parameters are
+* `n`: the number of random values to generate,
+* `a`: the smallest value to generate, and
+* `b`: the largest value to generate.
+
+The point is to illustrate specifying the parameters via
+1. defaults in the application,  `n = 1`, `a = 0`, `b = 6`,
+1. a "system" configuration file, `etc/my_app.conf`, `n = 2`, `a = 1`, `b = 6`,
+1. optionally, a configurtion file specified as a command line option, i.e.,
+   `--conf my_app.conf`, `n = 3`, `a = 2`,
+1. command line options.
+
+## What is it?
+1. `my_apps.py`: Python script to print random integer values.
+1. `etc/my_app.conf`: "system" level configuration file.
+1. `my_app.conf`: optional "user" level configuration file.

source-code/application/etc/my_app.conf

@@ -0,0 +1,4 @@
+[defaults]
+n = 2
+a = 1
+b = 6

source-code/application/my_app.conf

@@ -0,0 +1,3 @@
+[defaults]
+n = 3
+a = 2

source-code/application/my_app.py

@@ -0,0 +1,55 @@
+#!/usr/bin/env python
+
+from argparse import ArgumentParser
+from configparser import ConfigParser
+from pathlib import Path
+import random
+import sys
+
+
+def generate_data(n, a=0, b=1):
+    return [random.randint(a, b) for _ in range(n)]
+
+def main():
+    arg_parser = ArgumentParser(description='test application')
+    arg_parser.add_argument('--conf',
+                            help='configuration file to use')
+    arg_parser.add_argument('--verbose', action='store_true',
+                            help='verbose output')
+    options, remaining_options = arg_parser.parse_known_args()
+    system_conf = Path.cwd() / 'etc' / 'my_app.conf'
+    cfg = ConfigParser()
+    cfg['defaults'] = {'n': '1', 'a': '0', 'b': '6'}
+    if options.verbose:
+        print('application default values:', file=sys.stderr)
+        cfg.write(sys.stderr)
+    if system_conf.exists():
+        cfg.read('etc/my_app.conf')
+        if options.verbose:
+            print('system configuration file values:', file=sys.stderr)
+            cfg.write(sys.stderr)
+    else:
+        print(f'missing configuration file {system_conf}', file=sys.stderr)
+    if options.conf:
+        cfg.read(options.conf)
+        if options.verbose:
+            print('user configuration file values:', file=sys.stderr)
+            cfg.write(sys.stderr)
+    cfg_opts = dict(cfg['defaults'])
+    arg_parser.set_defaults(**cfg_opts)
+    arg_parser.add_argument('n', type=int, nargs='?',
+                            help='number of random values to generate')
+    arg_parser.add_argument('--a', type=int, help='smallest value')
+    arg_parser.add_argument('--b', type=int, help='largest value')
+    arg_parser.parse_args(remaining_options, options)
+    if options.verbose:
+        print('final options:', file=sys.stderr)
+        print(f'n = {options.n}\na = {options.a}\nb = {options.b}', end='\n\n',
+              file=sys.stderr)
+    values = generate_data(options.n, a=options.a, b=options.b)
+    print('\n'.join(str(value) for value in values))
+    return 0
+
+if __name__ == '__main__':
+    status = main()
+    sys.exit(status)

source-code/cmd/README.md

@@ -1,8 +1,10 @@
 # Cmd
+
 Standard Python library framework to facilitate writing command line
 driven interpreters.
 
 ## What is it?
+
 1. `simple.py`: a very simmple example that keeps track of a list of
     friends.  A friend is added by saying `hi` to her, and is removed by
     saying `bye` to her.  You can't say `hi` to someone who is your friend

source-code/code-evaluation/README.md

@@ -1,12 +1,11 @@
-EvaluateCode
-============
+# Code evaluation
 
 Python allows executino of Python source code at runtime.  Although this
 is a powerful feature, it should be used with extreme care, since it can
 give rise to serious security issues.
 
-What is it?
------------
+## What is it?
+
 1. `evaluate.py`: short example that reads a code fragment, evaluates it,
     and prints all values of variables defined in that code fragment.
     Only variables of type `int`, '`float`, `str`, `bool` and `list` will

source-code/command-line-arguments/README.md

@@ -1,4 +1,4 @@
-# CommandLineArgs
+# Command line arguments
 
 How to handle command line arguments in Python scripts.
 

source-code/config-parser/README.md

@@ -1,11 +1,10 @@
-ConfigParser
-============
+# ConfigParser
 
 ConfigParser is a module in Python's standard library to handle
 configuration files easily.
 
-What is it?
------------
+## What is it?
+
 1. `config_reader.py`: reads a configuration file provided on the command
    line, or `default.conf`.  It prints the sections of the configuration
    files, and the key/value pairs defined therein.

source-code/file-system/README.md

@@ -1,4 +1,5 @@
-# OsFileSystem
+# File system
+
 Examples of code interacting with the operating system or performing
 file system operations.
 

source-code/hydra/README.md

@@ -7,6 +7,7 @@ It support configuration file handling, command line arguments, logging,
 multiruns and so on.
 
 ## What is it?
+
 1. `gen_rand.py`: Python script to write random numbers to standard
    output.
 1. `conf/`: directory containing the configuration files.

source-code/logging/README.md

@@ -1,12 +1,10 @@
-Logging
-=======
+# Logging
 
 Using logging facilities rather than print or sys.stderr.write calls
 adds a lot of flexibility to more complex programs.  These facilities
 are provided through Python's standard library.
 
-What is it?
------------
+## What is it?
+
 `log_it_all.py`: illustration of how to use the default logging mechanisms,
 how to set the log level the destination, and the log format
-

source-code/paramiko/README.md

@@ -1,9 +1,11 @@
 # Paramiko
+
 Paramiko is a non-standard Python library for working with SSH.  It can
 be used to execute command on a remote host, or perform file transfers
 using the SFTP protocol.
 
 ## What is it?
+
 1. `ssh.py`: module with some SSH connection utitlity functions, imported
     in the scripts.
 1. `ls.py`: performs an `ls` command for the specified directory on a

source-code/sched/README.md

@@ -0,0 +1,11 @@
+# Sched
+
+Sometimes it can be useful to run functions at certain moments in time,
+similar to what cron does on Linux operatring systems.
+
+## What is it?
+
+* `cron.py`: example script using `schedule` that runs two tasks, one
+    printing the time, and a computed value, the other updating the
+    value to be printed.  The first is scheduled every 2 seconds, the
+    other every second.

source-code/sched/cron.py

@@ -0,0 +1,21 @@
+#!/usr/bin/env python
+
+import schedule
+import time
+
+
+def print_event(event):
+    time_str = time.strftime('%Y-%m-%d %H:%M:%S', time.localtime())
+    print('{0}: {1}'.format(time_str, event['counter']))
+
+
+def set_event(event):
+    event['counter'] += 1
+
+if __name__ == '__main__':
+    event = {'counter': 0}
+    schedule.every(2).seconds.do(print_event, event)
+    schedule.every().seconds.do(set_event, event)
+    while True:
+        schedule.run_pending()
+        time.sleep(0.5)