terraref · dlebauer · May 1, 2018 · Jan 17, 2018 · Jan 22, 2018 · Jan 22, 2018
diff --git a/README.md b/README.md
@@ -18,5 +18,32 @@ Our objectives are to ensure that the software and data in the reference data an
 
 ## Versions
 
-The first edition will be published in **November 2016**.
+* The first edition (alpha release) was published **November 2016**.
+* The second edition (beta release) will be published **November 2018** 
+* The third edition (version 1.0) will be published **November 2019**
+
+## Building the book
+
+### Dependencies
+
+```sh
+apt-get install -y curl gnupg
+curl -sL https://deb.nodesource.com/setup_6.x | sudo -E bash -
+apt-get install nodejs
+npm install -g gitbook-cli 
+```
+
+### Docker
+
+from: https://github.com/billryan/docker-gitbook
+
+```sh
+# init
+docker run --rm -v "$PWD:/gitbook" -p 4000:4000 billryan/gitbook gitbook init
+# serve
+docker run --rm -v "$PWD:/gitbook" -p 4000:4000 billryan/gitbook gitbook serve
+# build
+docker run --rm -v "$PWD:/gitbook" -p 4000:4000 billryan/gitbook gitbook build
+```
+
 
diff --git a/SUMMARY.md b/SUMMARY.md
@@ -1,73 +1,74 @@
 # Summary
 
 * [Introduction](README.md)
-  * [Data Sources](data-sources.md)
-  * [Software](software.md)
+    * [Data Sources](data-sources.md)
+    * [Software](software.md)
 * [Scientific Objectives and Experimental Design](user/scientific-objectives-and-experimental-design.md)
-  * [Protocols](user/protocols.md)
-    * [Controlled Environment Protocols](user/controlled-environment-protocols.md)
-    * [Manual Field Data Protocols](user/protocols-manual.md)
-    * [Phenotractor Protocols](user/protocols-tractor.md)
-    * [Sensor Calibration](sensor-calibration.md)
-    * [Template Protocol](user/protocol_template.md)
-    * [UAV Protocols](user/protocols-UAV.md)
-  * [Experimental Design](user/experimental-design.md)
-    * [Experimental Design Danforth](user/experimental-design-danforth.md)
-      * [Sorghum Lines Danforth](user/sorghum_danforth_1.md)
-    * [Experimental Design Genomics](user/experimental-design-genomics.md)
-      * [Sorghum Lines Genomics Year 1](user/sorghum-lines-genomics.md)
-      * [Sorghum Lines Genomics Year 2](user/sorghum-lines-genomics-year-2.md)
-    * [Experimental Design MAC](user/experimental-design-mac.md)
+    * [Protocols](user/protocols.md)
+        * [Controlled Environment Protocols](user/controlled-environment-protocols.md)
+        * [Manual Field Data Protocols](user/protocols-manual.md)
+        * [Phenotractor Protocols](user/protocols-tractor.md)
+        * [Sensor Calibration](sensor-calibration.md)
+        * [Template Protocol](user/protocol_template.md)
+        * [UAV Protocols](user/protocols-UAV.md)
+    * [Experimental Design](user/experimental-design.md)
+        * [Experimental Design Danforth](user/experimental-design-danforth.md)
+            * [Sorghum Lines Danforth](user/sorghum_danforth_1.md)
+        * [Experimental Design Genomics](user/experimental-design-genomics.md)
+            * [Sorghum Lines Genomics Year 1](user/sorghum-lines-genomics.md)
+            * [Sorghum Lines Genomics Year 1 (continued)](user/sorghum-lines-genomics-2.md)
+            * [Sorghum Lines Genomics Year 2](user/sorghum-lines-genomics-year-2.md)
+        * [Experimental Design MAC](user/experimental-design-mac.md)
 * [User Manual](user/user-manual.md)
-  * [What Data is Available](user/what-data-is-available.md)
-  * [Data Products](user/data-products.md)
-    * [Environmental conditions](products/environmental-conditions.md)
-    * [Fluorescence intensity imaging](products/fluorescence-intensity-imaging.md)
-    * [Genomics data](products/genomics-data.md)
-    * [Geospatial information](user/geospatial-information.md)
-    * [Hyperspectral imaging data](products/hyperspectral-imaging-data.md)
-    * [Infrared heat imaging data](products/infrared-heat-imaging-data.md)
-    * [Multispectral imaging data](products/multispectral-imaging-data.md)
-    * [Meteorological data](products/meteorological_data.md)
-    * [Phenotype data](products/trait-data.md)
-    * [Point Cloud Data](products/point-cloud-data.md)
-  * [How to Access Data](user/how-to-access-data.md)
-    * [Using Clowder \(Sensor and Genoomics data\)](user/using-clowder.md)
-    * [Using Globus \(Sensor and Genomics data\)](user/using-globus.md)
-    * [Using BETYdb \(trait data, experimental metadata\)](user/using-betydb.md)
-      * [Accessing BETYdb via ArcMap and other GIS software](accessing-betydb-with-arcmap.md)
-    * [Using CoGe \(Genomics\)](user/using-coge.md)
-    * [Using CyVerse \(Genomics\)](user/using-cyverse.md)
-    * [Using Analysis Workbench \(all data\)](user/using-analysis-workbench.md)
-  * [Data Use Policy](user/data_release_policy.md)
-  * [Manuscripts and Authorship Guidelines](manuscripts-and-authorship.md)
-  * [Release / reprocessing schedule](user/release_schedule.md)
+    * [What Data is Available](user/what-data-is-available.md)
+    * [Data Products](user/data-products.md)
+        * [Environmental conditions](products/environmental-conditions.md)
+        * [Fluorescence intensity imaging](products/fluorescence-intensity-imaging.md)
+        * [Genomics data](products/genomics-data.md)
+        * [Geospatial information](user/geospatial-information.md)
+        * [Hyperspectral imaging data](products/hyperspectral-imaging-data.md)
+        * [Infrared heat imaging data](products/infrared-heat-imaging-data.md)
+        * [Multispectral imaging data](products/multispectral-imaging-data.md)
+        * [Meteorological data](products/meteorological_data.md)
+        * [Phenotype data](products/trait-data.md)
+        * [Point Cloud Data](products/point-cloud-data.md)
+    * [How to Access Data](user/how-to-access-data.md)
+        * [Using Clowder (Sensor and Genoomics data)](user/using-clowder.md)
+        * [Using Globus (Sensor and Genomics data)](user/using-globus.md)
+        * [Using BETYdb (trait data, experimental metadata)](user/using-betydb.md)
+            * [Accessing BETYdb via ArcMap and other GIS software](accessing-betydb-with-arcmap.md)
+        * [Using CoGe (Genomics)](user/using-coge.md)
+        * [Using CyVerse (Genomics)](user/using-cyverse.md)
+        * [Using Analysis Workbench (all data)](user/using-analysis-workbench.md)
+    * [Data Use Policy](user/data_release_policy.md)
+    * [Manuscripts and Authorship Guidelines](manuscripts-and-authorship.md)
+    * [Release / reprocessing schedule](user/release_schedule.md)
 * [Technical Documentation](technical-documentation.md)
-  * [Data Standards](data-standards.md)
-    * [Existing Data Standards](existing_data_standards.md)
-    * [Agronomic and Phenotype Data Standards](agronomic-and-phenotype-data-standards.md)
-    * [Genomic Data Standards](genomic-data-standards.md)
-    * [Sensor Data Standards](sensor-data-standards.md)
-    * [Data Standards Committee](data-standards-committee.md)
-  * [Directory Structure](directory-structure.md)
-  * [Data Storage](data-storage.md)
-  * [Data Transfer](data-transfer.md)
-  * [Data Processing Pipeline](data-processing-pipeline.md)
-    * [Geospatial Time Series Structure](geostreams-structure.md)
-  * [Data Backup](data-backup.md)
-  * [Data Collection](data-collection.md)
-  * [Data Product Creation](data-product-creation.md)
-    * [Genomic Data](genomic-data-product.md)
-    * [Hyperspectral Data](hyperspectral-data.md)
-  * [Quality Assurance and Quality Control](qaqc.md)
+    * [Data Standards](data-standards.md)
+        * [Existing Data Standards](existing_data_standards.md)
+        * [Agronomic and Phenotype Data Standards](agronomic-and-phenotype-data-standards.md)
+        * [Genomic Data Standards](genomic-data-standards.md)
+        * [Sensor Data Standards](sensor-data-standards.md)
+        * [Data Standards Committee](data-standards-committee.md)
+    * [Directory Structure](directory-structure.md)
+    * [Data Storage](data-storage.md)
+    * [Data Transfer](data-transfer.md)
+    * [Data Processing Pipeline](data-processing-pipeline.md)
+        * [Geospatial Time Series Structure](geostreams-structure.md)
+    * [Data Backup](data-backup.md)
+    * [Data Collection](data-collection.md)
+    * [Data Product Creation](data-product-creation.md)
+        * [Genomic Data](genomic-data-product.md)
+        * [Hyperspectral Data](hyperspectral-data.md)
+    * [Quality Assurance and Quality Control](qaqc.md)
 * [Developer Manual](developer-manual.md)
-  * [Submitting data to Clowder](submitting-data-to-clowder.md)
-  * [Submitting data to BETYdb](submitting-data-to-betydb.md)
-  * [Submitting Data to CoGe](submitting-data-to-coge.md)
-  * [Developing Clowder Extractors](developing-clowder-extractors.md)
+    * [Submitting data to Clowder](submitting-data-to-clowder.md)
+    * [Submitting data to BETYdb](submitting-data-to-betydb.md)
+    * [Submitting Data to CoGe](submitting-data-to-coge.md)
+    * [Developing Clowder Extractors](developing-clowder-extractors.md)
 * [Tutorials](tutorials.md)
 * [Appendix](appendix.md)
-  * [Code of Conduct](code_of_conduct.md)
-  * [Collaboration Tools](collaboration_tools.md)
-  * [Glossary](glossary.md)
+    * [Code of Conduct](code_of_conduct.md)
+    * [Collaboration Tools](collaboration_tools.md)
+    * [Glossary](glossary.md)
 
diff --git a/accessing-betydb-with-arcmap.md b/accessing-betydb-with-arcmap.md
@@ -11,8 +11,7 @@ A step-by-step process is given below.
 
 ## Configuration used for these instructions
 
-* ArcMap 10.3 or later
-   * ArcMap implies Windows operating system
+* ArcMap 10.3 or later (Requires Windows operating system)
 * Instructions for using QGIS and other GIS software are provided below  
 * PuTTY: ssh client for Windows that can be downloaded here:  [PuTTY](http://www.chiark.greenend.org.uk/~sgtatham/putty/latest.html)
 
@@ -182,9 +181,9 @@ This assumes you have followed instructions for ArcMAP to create a database conn
   * Password: DelchevskoOro
   * Options: select 'Also list tables with no geometry'
 
-### Dumping a Shapefile
+### How to export plots from PostGIS as a Shapefile
 
-This does not require GIS software. While connecting directly to the database within GIS software is handy, it is also straightforward to dump out Shapefiles.
+This does not require GIS software other than the PostGIS traits database. While connecting directly to the database within GIS software is handy, it is also straightforward to export Shapefiles.
 
 After you have connected via ssh to the PostGIS server, the `pgsql2shp` function is available and can be used to dump out all of the plot and site definitions (names and geometries) thus:
 

diff --git a/data-backup.md b/data-backup.md
@@ -15,3 +15,10 @@ Runs every night at 23:59.  [View the script](https://github.com/PecanProject/pe
 This script creates a daily backup every day of the month. On Sundays creates a weekly backup, on the last day of the month it creates a monthly backup and at the last day of the year it will create a yearly backup. This script overwrite existing backups, for example every 1st of the month it will create a backup called bety-d-1 that contains the backup of the 1st of the month. See the script for the rest of the file names.
 
 These backups are copied using crashplan to a central location and should allow recovery in case of a catastrophic failure.
+
+## See Also
+
+* Description of Blue Water's nearline storage system https://bluewaters.ncsa.illinois.edu/data
+* Github issues: 
+  * https://github.com/terraref/computing-pipeline/issues/87
+  * https://github.com/terraref/computing-pipeline/issues/384
diff --git a/data-sources.md b/data-sources.md
@@ -29,8 +29,5 @@ The [Bellwether Foundation Phenotyping Facility](https://www.danforthcenter.org/
 ------------------
 ## Genomics
 
-#### HudsonAlpha Institute for Biotechnology. Alabama
-
-coming 2017
-
+Genomic data includes whole-genome resequencing data from the HudsonAlpha Institute for Biotechnology, Alabama for 384 samples for accessions from the sorghum Bioenergy Association Panel (BAP) and genotyping-by-sequencing (GBS) data from Kansas State University for 768 samples from a population of sorghum recombinant inbred lines (RIL).
 
diff --git a/developing-clowder-extractors.md b/developing-clowder-extractors.md
@@ -13,6 +13,54 @@ If you want to add an algorithm to the TERRAREF pipeline, or use the Clowder sof
 2. Using the Clowder API
 3. Using the pyClowder [pyClowder](https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/pyclowder/browse) [pyClowder](https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/pyclowder/browse) to add an analytical or technical component to the pipeline.
 
-### Contacts:
-  * Extractor development and deployment: Max Burnette via [email, phone](http://www.ncsa.illinois.edu/assets/php/directory/contact.php?contact=mburnet2), on GitHub, or on our [Slack Channel](https://terra-ref.slack.com/).
-  * Development environments: [Craig Willis](http://www.ncsa.illinois.edu/assets/php/directory/contact.php?contact=willis8).
+## TERRA-REF Extractor Resources
+
+### terrautils 
+
+To make working with the TERRA-REF pipeline as easy as possible, the [terrautils](https://github.com/terraref/terrautils) Python library was written. By importing this library in an extractor script, developers can ensure that code duplication is minimized and standard practices are used for common tasks such as GeoTIFF creation and georeferencing. It also provides modules for managing metadata, downloading and uploading, and BETYdb/geostreams API wrapping.
+
+Modules include:
+
+* [betydb](https://github.com/terraref/terrautils/blob/master/terrautils/betydb.py) BETYdb API wrapper
+* [extractors](https://github.com/terraref/terrautils/blob/master/terrautils/extractors.py) General extractor tools e.g. for creating metadata JSON objects and generating folder hierarchies
+* [formats](https://github.com/terraref/terrautils/blob/master/terrautils/formats.py) Standard methods for creating output files e.g. images from numpy arrays
+* [gdal](https://github.com/terraref/terrautils/blob/master/terrautils/gdal.py) GDAL general image tools
+* [geostreams](https://github.com/terraref/terrautils/blob/master/terrautils/geostreams.py) Geostreams API wrapper
+* [influx](https://github.com/terraref/terrautils/blob/master/terrautils/influx.py) InfluxDB logging API wrapper
+* [lemnatec](https://github.com/terraref/terrautils/blob/master/terrautils/lemnatec.py) LemnaTec-specific data management methods
+* [metadata](https://github.com/terraref/terrautils/blob/master/terrautils/metadata.py) Getting and cleaning metadata
+* [products](https://github.com/terraref/terrautils/blob/master/terrautils/products.py) Get file lists
+* [sensors](https://github.com/terraref/terrautils/blob/master/terrautils/sensors.py) Standard sensor information resources
+* [spatial](https://github.com/terraref/terrautils/blob/master/terrautils/spatial.py) Geospatial metadata management
+
+### Science packages
+
+To keep code and algorithms broadly applicable, TERRA-REF is developing a series of science-driven packages to collect methods and algorithms that are generic to an input and output from the pipeline. That is, these packages should not refer to Clowder or extraction pipelines, but instead can be used in applications to manipulate data products. They are organized by sensor.
+
+These packages will also include test suites to verify that any changes are consistent with previous outputs. The test directories can also act as examples on how to instantiate and use the science packages in actual code.
+
+* [stereo_rgb](https://github.com/terraref/stereo_rgb) stereo RGB camera (stereoTop in raw_data, rgb_ prefix elsewhere)
+* [flir_ir](https://github.com/terraref/flir_ir) FLIR infrared camera (flirIrCamera in raw_Data, ir_ prefix elsewhere)
+* [scanner_3d](https://github.com/terraref/scanner_3d) laser 3D scanner (scanner3DTop in raw_Data, laser3d_ elsewhere)
+
+
+### Extractor repositories
+
+Extractors can be considered wrapper scripts that call methods in the science packages to do work, but include the necessary components to communicate with TERRA's RabbitMQ message bus to process incoming data as it arrives and upload outputs to Clowder. There should be no science-oriented code in the extractor repos - this code should be implemented in science packages instead so it is easier for future developers to leverage.
+
+Each repository includes extractors in the workflow chain corresponding to the named sensor.
+
+* [extractors-stereo-rgb](https://github.com/terraref/extractors-stereo-rgb)
+* [extractors-3dscanner](https://github.com/terraref/extractors-3dscanner)
+* [extractors-multispectral](https://github.com/terraref/extractors-multispectral)
+* [extractors-metadata](https://github.com/terraref/extractors-metadata)
+* [extractors-hyperspectral](https://github.com/terraref/extractors-hyperspectral)
+* [extractors-environmental](https://github.com/terraref/extractors-environmental)
+* [extractors-lemnatec-indoor](https://github.com/terraref/extractors-lemnatec-indoor)
+
+### Contact:
+
+* Extractor development and deployment: [Max Burnette](http://www.ncsa.illinois.edu/assets/php/directory/contact.php?contact=mburnet2)
+* Development environments: [Craig Willis](http://www.ncsa.illinois.edu/assets/php/directory/contact.php?contact=willis8)
+* On our [Slack Channel](https://terra-ref.slack.com/)
+* On [GitHub](https://github.com/terraref)
diff --git a/developing-science-packages.md b/developing-science-packages.md
@@ -0,0 +1,73 @@
+## Developing Science Packages for Use in TERRA-REF Pipeline
+
+In 2018 the decision was made to split Clowder-specific extractor code from the actual science code that performs critical conversion and analysis on sensor data.
+
+### General Structure
+
+The general directory structure is as follows:
+
+```
+<Repository Root>
+setup.py
+terraref/
+    __init__.py
+    SCIENCE_PKG_NAME/
+        __init__.py
+        science_functions.py
+```
+
+In order to ensure consistency of structure across packages, terraref/__init__.py contains only one line:
+
+```
+__import__('pkg_resources').declare_namespace(__name__)
+```
+
+Then, within setup.py (example from stereo_rgb package):
+
+```
+from setuptools import find_packages, setup
+
+setup(name='terraref-stereo_rgb',
+      version='1.0.3',
+      packages=find_packages(),
+      namespace_packages=['terraref'],
+      include_package_data=True,
+      url='https://github.com/terraref/stereo_rgb',
+      install_requires=[
+            'numpy',
+            'scipy',
+            'multiprocessing',
+            'matplotlib',
+            'Pillow'
+      ]
+      )
+```
+
+...and within terraref/SCIENCE_PKG_NAME/__init__.py we specify which functions to import when the package is imported:
+
+```
+from .stereo_rgb import \
+    calculate_canopycover,  process_raw,  demosaic, get_image_shape
+```
+
+### Deploying to PyPi
+
+When the package is ready, the following commands can be used to publish:
+
+```
+cd <Repository Root>
+python setup.py sdist
+twine upload dist/<newly tagged version.gz>
+```
+
+### Using Science Package
+
+```
+pip install terraref-stereo-rgb
+
+import terraref.stereo_rgb
+terraref.stereo_rgb.calculate_canopycover()
+
+from terraref.stereo_rgb import calculate_canopycover
+calculate_canopycover()
+```