Skip to content

merge develpment into master #172

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 5 commits into from
Jan 22, 2018
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 5 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,11 +2,11 @@

## About this book

This book describes the TERRA-REF data collection, computing, and analysis pipelines.
This book describes the TERRA-REF data collection, computing, and analysis pipelines. The following links provide quick access to

* [What data is available?](/user/what-data-is-available.md)
* [Where do I get the data?](/user/how-to-access-data.md)
* [User tutorials](/tutorials.md)
* [Available Data](/user/what-data-is-available.md)
* [How to access data](/user/how-to-access-data.md)
* [Hands on tutorials](/tutorials.md)

## About TERRA-REF

Expand All @@ -19,3 +19,4 @@ Our objectives are to ensure that the software and data in the reference data an
## Versions

The first edition will be published in **November 2016**.

16 changes: 8 additions & 8 deletions SUMMARY.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,13 +32,13 @@
* [Phenotype data](products/trait-data.md)
* [Point Cloud Data](products/point-cloud-data.md)
* [How to Access Data](user/how-to-access-data.md)
* [Using Clowder (Sensor and Genoomics data)](user/using-clowder.md)
* [Using Globus (Sensor and Genomics data)](user/using-globus.md)
* [Using BETYdb (trait data, experimental metadata)](user/using-betydb.md)
* [Accessing BETYdb via ArcMap and other GIS software](accessing-betydb-with-arcmap.md)
* [Using CoGe (Genomics)](user/using-coge.md)
* [Using CyVerse (Genomics)](user/using-cyverse.md)
* [Using Analysis Workbench (all data)](user/using-analysis-workbench.md)
* [Using Clowder \(Sensor and Genoomics data\)](user/using-clowder.md)
* [Using Globus \(Sensor and Genomics data\)](user/using-globus.md)
* [Using BETYdb \(trait data, experimental metadata\)](user/using-betydb.md)
* [Accessing BETYdb via ArcMap and other GIS software](accessing-betydb-with-arcmap.md)
* [Using CoGe \(Genomics\)](user/using-coge.md)
* [Using CyVerse \(Genomics\)](user/using-cyverse.md)
* [Using Analysis Workbench \(all data\)](user/using-analysis-workbench.md)
* [Data Use Policy](user/data_release_policy.md)
* [Manuscripts and Authorship Guidelines](manuscripts-and-authorship.md)
* [Release / reprocessing schedule](user/release_schedule.md)
Expand All @@ -53,7 +53,7 @@
* [Data Storage](data-storage.md)
* [Data Transfer](data-transfer.md)
* [Data Processing Pipeline](data-processing-pipeline.md)
* [Geospatial Time Series Structure](geostreams-structure.md)
* [Geospatial Time Series Structure](geostreams-structure.md)
* [Data Backup](data-backup.md)
* [Data Collection](data-collection.md)
* [Data Product Creation](data-product-creation.md)
Expand Down
139 changes: 70 additions & 69 deletions accessing-betydb-with-arcmap.md
Original file line number Diff line number Diff line change
@@ -1,12 +1,12 @@
# Accessing BETYdb from GIS Software
# Accessing BETYdb from GIS Software

Interested researchers can access BETYdb directly from GIS software such as ESRI ArcMap and QGIS.
Interested researchers can access BETYdb directly from GIS software such as ESRI ArcMap and QGIS.
In some cases direct access can simplify the use of spatial data in BETYdb data, but this convenience must be weighed against a more complex setup, limits of GIS software compatibility, and additional complexity of extracting data from a PostGIS SQL database.

## Overview

Accessing the production BETYdb used by the TERRA REF program requires creating a secure shell tunnel (SSH) to a remote server.
After creating the tunnel, the database is accessed as if it were available on the local machine.
Accessing the production BETYdb used by the TERRA REF program requires creating a secure shell tunnel \(SSH\) to a remote server.
After creating the tunnel, the database is accessed as if it were available on the local machine.
A step-by-step process is given below.

## Configuration used for these instructions
Expand All @@ -18,13 +18,13 @@ A step-by-step process is given below.

## Setup

### Request Access
### Request Access

Request access [to the BETYdb server](https://identity.ncsa.illinois.edu/join/TU49BUUEDM) by following the link.
This will take you to the NCSA identity service. If you do not have an NCSA account, you will be asked to create one.
Request access [to the BETYdb server](https://identity.ncsa.illinois.edu/join/TU49BUUEDM) by following the link.
This will take you to the NCSA identity service. If you do not have an NCSA account, you will be asked to create one.
This account and password will be used to login to the database server. Access will generally be granted within 24-hours.

### Confirm Access
### Confirm Access

Use PuTTY or your preferred SSH client and your NCSA account. First open the terminal and then login to bety6.ncsa.illinois.edu using ssh from the command line:

Expand All @@ -34,28 +34,28 @@ ssh <login>@bety6.ncsa.illinois.edu

After confirming access to bety6 logout by typing `exit`.

### Create SSH Tunnel to BETYdb
### Create SSH Tunnel to BETYdb

The following command will create an `SSH tunnel` from your computer to the BETYdb server:
The following command will create an `SSH tunnel` from your computer to the BETYdb server:

_Note_ if have a postgres running on your desktop computer (using the default port 5432), you will need to stop it first.
_Note_ if have a postgres running on your desktop computer \(using the default port 5432\), you will need to stop it first.

```sh
ssh -L 5432:localhost:5432 <login>@bety6.ncsa.illinois.edu
```

The above will bind the local port 5432 (first parameter) to port 5432 (second parameter), the default Postgres listening port, on the remote server. All traffic bound for port 5432 on your local machine will be automatically forwarded to the remote server. As a result, programs such as ArcGIS running on your computer will connect to the remote BETYdb as if it were on your computer.
The above will bind the local port 5432 \(first parameter\) to port 5432 \(second parameter\), the default Postgres listening port, on the remote server. All traffic bound for port 5432 on your local machine will be automatically forwarded to the remote server. As a result, programs such as ArcGIS running on your computer will connect to the remote BETYdb as if it were on your computer.

_Note_ **you will need to create the SSH connection with the tunnel every time you wish to
_Note_ **you will need to create the SSH connection with the tunnel every time you wish to
access BETYdb from your local machine.**

To keep the tunnel open, use
To keep the tunnel open, use

```sh
ssh -Nf -L 5432:localhost:5432 <login>@bety6.ncsa.illinois.edu
```

> _note for PuTTY Users:_ you can configure Putty to remember these settings. In the navigation tree on the left-hand side, click Connection > SSH > Tunnels. Enter '5432' under Source port and 'localhost:5432' in the Destination field. Then click session and save this configuration for future use.
> _note for PuTTY Users:_ you can configure Putty to remember these settings. In the navigation tree on the left-hand side, click Connection &gt; SSH &gt; Tunnels. Enter '5432' under Source port and 'localhost:5432' in the Destination field. Then click session and save this configuration for future use.

The next section of the guide will discuss accessing BETYdb using ArcMap, querying plots and joining these to the traits and experiments tables. The instructions for setting up a SSH tunnel will also work psql, pgAdmin3, QGIS, and other clients. Instructions for connecting via QGIS and ArcGIS Pro are provided below.

Expand All @@ -65,45 +65,45 @@ The next section of the guide will discuss accessing BETYdb using ArcMap, queryi

BETYdb is configured with PostGIS geometry support. This allows ArcGIS Desktop clients to access geometry layers stored within BETYdb.

> _Warning:_ ArcGIS releases prior to 10.3 required you to place the
PostgreSQL libpq files in the ArcGIS client's bin directory. This is no
longer required for the ArcGIS Desktop clients but some ESRI tools may
still require the library be installed.*
> _Warning:_ ArcGIS releases prior to 10.3 required you to place the
> PostgreSQL libpq files in the ArcGIS client's bin directory. This is no
> longer required for the ArcGIS Desktop clients but some ESRI tools may
> still require the library be installed.\*

1. Click on the ArcCatalog icon (on right edge of ArcMap window) to open the ArcCatalog Tree
1. Click on the ArcCatalog icon \(on right edge of ArcMap window\) to open the ArcCatalog Tree
2. In the tree, click on 'Database Connections' and then "Add Database Connnections". A Database Connection dialog window will open.
3. Within the dialog box:
```
Database Platform: PostgreSQL
Instance: localhost
Authentication Type: Database authentication
User name: viewer
Password: DelchevskoOro
Database: select bety (if everything else is correct)
```
```
Database Platform: PostgreSQL
Instance: localhost
Authentication Type: Database authentication
User name: viewer
Password: DelchevskoOro
Database: select bety (if everything else is correct)
```
4. Click OK
4. The connection will be saved as "Connection to localhost.sde", right
click and rename to it to "TERRA REF BETYdb trait database" to allow easy reuse.
5. Click on the Add Layer icon (black cross over yellow diamand) button to open the Add Data dialog window.
6. Under 'Look in' on the second line choose 'Database Connections'.
7. Select the "TERRA REF BETYdb trait database" that created above
8. Select the bety.public.sites table and click 'Add'.
- This 'sites' table is the only table in the database with a geospatial 'geometry' data type.
- Any of the other tables can also be added, as described below.
6. The New Query Layer dialog will be displayed asking for the Unique Identifier Field for the layer. For the bety.public.sites table, the unique identifier is the "sitename" field.
7. Click Finish.

> _Warning_: ArcMap does not support the big integer format used by BETYdb
as primary keys and those fields will not be visible or available for
selection. In most cases you should be able to use other fields as
unique identifiers.*
5. The connection will be saved as "Connection to localhost.sde", right
click and rename to it to "TERRA REF BETYdb trait database" to allow easy reuse.
6. Click on the Add Layer icon \(black cross over yellow diamand\) button to open the Add Data dialog window.
7. Under 'Look in' on the second line choose 'Database Connections'.
8. Select the "TERRA REF BETYdb trait database" that created above
9. Select the bety.public.sites table and click 'Add'.
* This 'sites' table is the only table in the database with a geospatial 'geometry' data type.
* Any of the other tables can also be added, as described below.
10. The New Query Layer dialog will be displayed asking for the Unique Identifier Field for the layer. For the bety.public.sites table, the unique identifier is the "sitename" field.
11. Click Finish.

> _Warning_: ArcMap does not support the big integer format used by BETYdb
> as primary keys and those fields will not be visible or available for
> selection. In most cases you should be able to use other fields as
> unique identifiers.\*

### Modifying the Query Layer

BETYdb contains one geometry table called betydb.public.sites containing
the boundaries for each plot. Because the plot boundaries can change each season, and even within season, different
plot definitions may be used (e.g. to subset plots or exclude boundary rows), there is significant overlap that can cause confusion
when displayed.
BETYdb contains one geometry table called betydb.public.sites containing
the boundaries for each plot. Because the plot boundaries can change each season, and even within season, different
plot definitions may be used \(e.g. to subset plots or exclude boundary rows\), there is significant overlap that can cause confusion
when displayed.
In general, you will want to use the query layer to limit plots to a single season and a single definition.

1. Right click the bety.public.sites layer and choose properties.
Expand All @@ -115,39 +115,39 @@ For more advanced selection of sites by experiment or season, you can join the `

### Joining Additional BETYdb Tables

Additional tables can be added and joined to the sites table. Tables can
be added just like any other layer. In this case, we'll add
bety.public.traits_and_yields_view and join it to the bety.public.sites
Additional tables can be added and joined to the sites table. Tables can
be added just like any other layer. In this case, we'll add
bety.public.traits\_and\_yields\_view and join it to the bety.public.sites
layer.

1. To create a join with other tables, start by adding the desired table.
2. Follow instructions above to add the bety.public.traits_and_yields_view
2. Follow instructions above to add the bety.public.traits\_and\_yields\_view
3. On this table the unique identifier is a group of columns, so select sitename, cultivar, scientificname, trait, date, entity, and method as the unique identifiers.
2. Right click on the bety.public.sites layer.
3. Under 'Joins and Relates' select 'Join'.
3. Choose sitename (from bety.public.sites) in part 1
4. Choose bety.public.traits_and_yields_view in part 2
5. Choose sitename in part 3
6. Click OK
4. Right click on the bety.public.sites layer.
5. Under 'Joins and Relates' select 'Join'.
6. Choose sitename \(from bety.public.sites\) in part 1
7. Choose bety.public.traits\_and\_yields\_view in part 2
8. Choose sitename in part 3
9. Click OK

### Creating a Thematic View

The final section describes how to create a thematic view of the
bety.public.sites layer based on the mean attribute where the trait is
NDVI from the bety.public.traits_and_yields_view. Remove any previous
joins from bety.public.sites (right click bety.public.sites --> joins and relates --> remove join) prior to performing this procedure because we will be selecting the NDVI data by creating a query layer
from bety.public.traits_and_yields_view prior to the join.
The final section describes how to create a thematic view of the
bety.public.sites layer based on the mean attribute where the trait is
NDVI from the bety.public.traits\_and\_yields\_view. Remove any previous
joins from bety.public.sites \(right click bety.public.sites --&gt; joins and relates --&gt; remove join\) prior to performing this procedure because we will be selecting the NDVI data by creating a query layer
from bety.public.traits\_and\_yields\_view prior to the join.

1. Right click bety.public_traits_and_yields_view table and select properties
1. Right click bety.public\_traits\_and\_yields\_view table and select properties
2. Click on the Definition Query tab
3. Add the line "trait = 'NDVI'" to the Definition Query box
4. Click OK
5. Follow the steps defined in Joining Additional BETYdb Tables
6. Right click on the bety.sites layer and choose properties
7. Choose the Symbology tab
8. Under the Show section, choose Quantities --> Graduated Colors
8. Under the Show section, choose Quantities --&gt; Graduated Colors
9. Under the Fields Value selection choose mean
10. Click OK
10. Click OK

## Connecting to Other GIS Software

Expand All @@ -160,11 +160,11 @@ This assumes you have followed instructions for ArcMAP to create a database conn
* Open ArcCatalog
* Under database connections, you will find the connection made above, called 'TERRA REF BETYdb.sde'
* right click this and select 'properties'
* copy the file path (it should look like `C:\Users\<USER NAME>\AppData\Roaming\ESRI\Desktop10.4\ArcCatalog\TERRA REF BETYdb.sde`
* copy the file path \(it should look like `C:\Users\<USER NAME>\AppData\Roaming\ESRI\Desktop10.4\ArcCatalog\TERRA REF BETYdb.sde`
* Open ArcGIS Pro
* Under the Insert tab, select connections --> 'add database'
* paste the path to 'TERRA REF BETYdb.sde' in the directory navigation bar
* select 'TERRA REF BETYdb.sde'
* Under the Insert tab, select connections --&gt; 'add database'
* paste the path to 'TERRA REF BETYdb.sde' in the directory navigation bar
* select 'TERRA REF BETYdb.sde'

### QGIS

Expand Down Expand Up @@ -192,3 +192,4 @@ After you have connected via ssh to the PostGIS server, the `pgsql2shp` function
pgsql2shp -f terra_plots.shp -h localhost -u bety -P bety bety \
"SELECT sitename, geometry FROM sites"
```

14 changes: 8 additions & 6 deletions data-sources.md
Original file line number Diff line number Diff line change
@@ -1,25 +1,27 @@
#Data Sources
# Field Sites and Platforms

## Field phenotyping
####Maricopa Agricultural Center (MAC), Arizona
## Field phenotyping research sites

Over 300 sorghum accessions and recombinant inbred lines were planted at the University of Arizona Maricopa Agricultural Center and USDA Arid Land Research Station in Maricopa, Arizona. Field Scanner
### Maricopa Agricultural Center (MAC), Arizona

The Maricopa field site is located at the the University of Arizona Maricopa Agricultural Center and USDA Arid Land Research Station in Maricopa, Arizona. At this site, we have deployed the following phenotyping platforms.

- The [**Lemnatec Scanalyzer Field System**](http://www.lemnatec.com/products/hardware-solutions/scanalyzer-field/) is the largest field crop analytics robot in the world. This high-throughput phenotyping field-scanning robot has a 30-ton steel gantry that autonomously moves along two 200-meter steel rails while continuously imaging the crops growing below it with a diverse array of [cameras and sensors](http://terraref.org/articles/lemnatec-scanalyzer-field-sensors/).
- The **PhenoTractor** is fitted with a sensor frame that supports a real time kinematic (RTK) satellite navigation antenna, a sonar transducer, an infrared temperature (IRT) scanner, and three [GreenSeeker crop sensing systems](http://www.trimble.com/agriculture/greenseeker.aspx).
- **UAV** (release V1)
- **Manually Collected Field Data** - Data will be collected manually to verify the sensor-collected data.
- **Manually Collected Field Data** - Data will are collected manually using standard field methods. These measurements are used to calibrate and validate phenotypes derived from sensor-collected data.


### Kansas State University


- Tractor - coming 2017

- UAV - coming 2017

------------------
## Controlled-environment phenotyping
####Donald Danforth Plant Science Center, Missouri
#### Donald Danforth Plant Science Center, Missouri

The [Bellwether Foundation Phenotyping Facility](https://www.danforthcenter.org/scientists-research/core-technologies/phenotyping) is a climate controlled 70 m<sup>2</sup> growth house with a conveyor belt system for moving plants to and from fluorescence, color, and near infrared imaging cabinets. This automated, high-throughput platform allows repeated non-destructive time-series image capture and multi-parametric analysis of 1,140 plants in a single experiment.

Expand Down
Empty file added field-sites-and-platforms.md
Empty file.
Loading