List of MAC field measurements to import into BETYdb

[dlebauer]

@NewcombMaria please provide a list of the field measurements that are ready to be added to BETYdb (have a date column and are arranged in 'long' format with one measurement per row.

• Season 1 (Upload 2016 first season field measurements to BETYdb #46)

  • emergence vigor
  • emergence final stand counts
  • plant heights
  • node and tiller counts on marked plants
  • phenology growth stage data
  • leaf desiccation ratings
  • radiation interception
  • managements
  • Incomplete harvest yield and moisture data
  • in-season biomass
  • porometer readings
  • lodging / leaning
  • plant counts
  • radiation interception
  • phenology stages

• Season 2

  • plant heights here
  • managements
  • emergence vigor
  • emergence final stand counts
  • node and tiller counts on marked plants
  • leaf length and width on marked plants, one date
  • yields

We should include also the agronomic metadata and the plot --> genotype mappings.

[NewcombMaria]

Working on it. Will update field measurements list and agronomic metadata as soon as possible. Not certain what is meant by plot to genotype mappings. Unique genotype id??

[dlebauer]

by plot to genotype mappings, I meant, what genotype was planted in each plot?

[NewcombMaria]

• For genotype names, please use the genotype in the column 'name_corrected' in the spreadsheet https://docs.google.com/spreadsheets/d/107aijRIvO5i-HHwVDQTXn0-9sE02cXO_MdZk3DERzQ0/edit#gid=1166373347
  @dlebauer @rachelshekar Let me know if you have questions. Thanks.

[NewcombMaria]

The following are formatted and ready for importing into BETYdb. For all of these, rows that are missing data can be excluded. All data should have 'Newcomb, Maria 2016' as the citation. If you need to create and define new variables or otherwise modify the spreadsheets to allow uploading, that is OK.

• Season1 plant heights: 'Sorghum_ManualData_Maricopa2016_HtsAddedJuly22_LongFormatJuly25' the last sheet (Heights_Full_field_LongFormat)
• Season1 emergence stand counts 'F13_2016_sorghum_whole_field.csv'
• Season1 emergence vigor 'F13_2016_sorghum_emergence.csv'
• managements for sorghum season1 and sorghum season2
• I sorghum season2 plant heights in 'DRAFT_Sorghum2_PlantHeights_AugustPlanting.xlsx'. (ready 2016-12-01) [name changed to "Sorghum2_PlantHeights_AugustPlanting.xlsx"]
• sorghum season2 emergence vigor 'Sorghum2_emergence_SubsetPlots_plusFinalCounts_Nov7.xlsx'.
• sorghum season1 leaf node and tiller count 'Sorghum_ManualData_NodeCounts_BAP_RepeatMeasures.xlsx' (ready 2016-12-01)
• season 2 emergence final stand counts (ready 2016-12-01)

These will require further work to determine how to encode and enter these values into the database.

• Season1 leaf desiccation ratings (see 'Sorghum_ManualData_Maricopa_Leaf_Desiccation_final' at https://docs.google.com/spreadsheets/d/1CDV3dJSoV5F_mFgZI6cYEt2fUpa4BHgv-adbP704bSQ/edit#gid=2112860580 ). The ratings include an incidence and a severity of leaf desiccation.
• Cesar and Nadia (Danforth) reformatted the sorghum season1 phenology data. The long format data are in the csv file 'Sorghum_ManualData_Maricopa_Phenology_Fixed.csv', see https://docs.google.com/spreadsheets/d/1AFwjkUz5lRzR47xc3R2OwjKd-mRJLz-ongAIzD2Hnvs/edit#gid=1209684786 . Please find the definitions in the 2nd sheet/tab in the spreadsheet 'Sorghum_ManualData_Maricopa_Phenology.xlsx', in https://docs.google.com/spreadsheets/d/1BQeFBlyk9y_eE4wiDKYCwQ2wxMqMx1t8rUrplGSqMhI/edit#gid=622354195 .
  Cesar and Nadia (Danforth) also reformatted the sorghum season1 radiation interception data, and I added interpolated incident radiation for each measurement. The long format data set is in the csv file 'Sorghum_ManualData_RadiationInterception_BAP_Fixed_V2.csv' (https://docs.google.com/spreadsheets/d/1aW3NJ98GxF-XIRibvUwBcuqG6CQVqVqFBW1-_SRfLJ0/edit#gid=1280560409 ). Please find the definitions in the 2nd sheet/tab in the spreadsheet 'Sorghum_ManualData_RadiationInterception' (https://docs.google.com/spreadsheets/d/1QnO2yu_njVTdPueQNrI4O2PY_MBqz9iyN8WZWfVvc1g/edit#gid=1238504444).
• It's still incomplete and will need additional info, but the preliminary sorghum season1 harvest info provided by Mike Ottman is in the spreadsheet 'Sorghum1_harvest_data_July2016_INCOMPLETE_SAS - Copy.xlsx' (https://docs.google.com/spreadsheets/d/1iHmSyFeCO7np_lxYyvdiXQQcAwBJpIKpeAOyMXFd-OY/edit#gid=901711246) .

[NewcombMaria]

I added 2 new files of Sorghum season2 data in the Data for uploading google docs directory https://drive.google.com/drive/folders/0B4JvLJOau3GmaFZrZ2Yzc2xSbUk

Sorghum2_emergence_SubsetPlots_plusFinalCounts_Nov7.xlsx, see https://docs.google.com/spreadsheets/d/1xIRwroYObD125I0TDCDp3YPIGPi2Qzeh-vlUCg_b85Q/edit#gid=1432184448 . This has emergence vigor and emergence final stand counts - needs to be formatted.

Sorghum2_NodeTillerCounts_SubsetPlots.xlsx, see https://docs.google.com/spreadsheets/d/10eEcZST1D0_kogUVvK_iG0cFE_I70OTJM6HitYztRBU/edit#gid=707942000 . This has node and tiller counts on marked plants, and leaf length and width on marked plants, one date - needs to be formatted.

@dlebauer @rachelshekar Let me know if you have questions. Thanks.

[NewcombMaria]

I will ask Nadia if someone at Danforth can work on reformatting the following for importing into BETYdb:

Season1 phenology growth stage data, (see 'Sorghum_ManualData_Maricopa_Phenology.xlsx' at https://docs.google.com/spreadsheets/d/1BQeFBlyk9y_eE4wiDKYCwQ2wxMqMx1t8rUrplGSqMhI/edit#gid=194121296 ).

Season1 radiation interception, (see 'Sorghum_ManualData_RadiationInterception' at https://docs.google.com/spreadsheets/d/1QnO2yu_njVTdPueQNrI4O2PY_MBqz9iyN8WZWfVvc1g/edit#gid=482347767 )

@dlebauer I'll do my best to work on reformatting the other data sets and adding management info.

[dlebauer]

@gsrohde could you work on importing these spreadsheets into the bety6 database?

I've updated the cultivars (https://gist.github.com/dlebauer/e42dd56cc7823ff1cf3f3852218a6886)

[dlebauer]

@NewcombMaria is this emergence vigor or emergence count? 'F13_2016_sorghum_emergence.csv',

[NewcombMaria]

Regarding: is this emergence vigor or emergence count? 'F13_2016_sorghum_emergence.csv':
It's emergence count over time, or an emergence rate, which some prefer to call emergence vigor. A repeat measure of emergence over time, which indicates emergence vigor

[gsrohde]

@dlebauer Before I begin work on importing these, I need to know exactly which spreadsheets you mean. For example, do you mean every CSV or spreadsheet file linked to in this issue? Or only those with unchecked checkboxes next to them?

Then I will probably have questions about each one, since they are not yet in the format required for bulk upload. And perhaps some don't even pertain to trait data, so I will have to devise some other method of inserting the data.

[dlebauer]

@gsrohde the links in #61 (comment) go to the relevant worksheets. The first set of 8 should be ready or close enough. Did you see these? All are traits except the managements that can be entered via the web interface or insert statements.

[gsrohde]

@dlebauer Many or most of these files contain several sheets. It's not clear whether I need only pay attention to the one that opens upon clicking the link.

So I'll start with items 2 and 3, which each consist of only a single CSV file. Here are my questions and comments on those:

First of all, I’m assuming all new trait rows connected with this data should refer to citation "Newcomb, Maria, 2016, Field measurements from Maricopa Agricultural Center” and species “Sorghum bicolor”.

Can I assume that date values are the date at the site?

Should access_level be 2 as for the other data?

I’m assuming no treatment, method, or entity unless you tell me otherwise.

File-specific notes:

Item 2 (F13_2016_sorghum_whole_field): I removed every row that didn’t have a date and a value for emergence_count. I’m assuming no specific cultivar for this data. I’m ignoring the Entry column; I’m also ignoring plot_id and Row, these being encapsulated in the site name.

Item 3 (F13_2016_sorghum_emergence): Again, I’m ignoring the Entry, plot_id, and Row columns.

[dlebauer]

Yes, data are in the worksheet that opens from the link. Many I've highlighted columns to upload in green

Yes all of your assumptions are correct. We will insert cultivars and methods separately / later