lysimeter-analysis

Warning

This python module is currently in undergoing BETA testing, and is not ready for full-release until further validation studies are conducted. Please report any issues to the author and community using the discussion board.

A Python module that processes and analyzes weighing lysimeter data for quantifying crop water use (i.e., evapotranspiration).

Created by:
A.J. Brown
Agricultural Data Scientist
Colorado State Univeristy
Ansley.Brown@colostate.edu
5 April 2024

Table of Contents

0. Helpful Definitions
1. Introduction
2. Installation
3. Usage
4. Core Functionalities
5. Workflow
6. Algorithm Validation
7. Data Disclosure
8. References

Helpful Definitions

The following is a glossary of key terms related to evapotranspiration and coefficient terminology provided by DeJonge et al. (2020) which are helpful for understanding the concepts discussed in the lysimeter-analysis documentation.

• Evapotranspiration (ET): The movement of water from land and plant surfaces through two processes: evaporation from soil and water surfaces, and transpiration through plant stomata.

• Reference Evapotranspiration (ETref): The rate of evapotranspiration from a hypothetical reference crop under well-watered conditions. It serves as a climatic index for estimating crop water requirements. Commonly used reference surfaces are:

  • ETos: Short crop (e.g., grass) reference surface.
  • ETrs: Tall crop (e.g., alfalfa) reference surface.

• Crop Coefficient (Kc): A dimensionless factor used to estimate crop evapotranspiration (ETc) from reference evapotranspiration (ETref). The formula is: [ ETc = ETref \times Kc ] This coefficient accounts for the crop's development stage and the biological properties affecting water use.

• Potential Evapotranspiration (ETp): The theoretical maximum rate of evapotranspiration when all soil and plant surfaces are wet. It is not standardized, and its use is often confused with reference evapotranspiration.

• Transpiration Coefficient (Kcb): Represents the potential transpiration component of crop evapotranspiration, which is separate from the evaporation component. It is often adjusted for water stress conditions.

• Evaporation Coefficient (Ke): Represents the potential evaporation from the soil surface, particularly important when the soil is bare or has minimal canopy coverage.

• Water Stress Coefficient (Ks): A factor that adjusts the transpiration coefficient (Kcb) to account for reduced transpiration due to water stress conditions in the crop.

Introduction

Precision weighing lysimeters are instruments used to measure crop water use with high accuracy. They work by continuously weighing a soil-plant system, contained within a monolith enclosed underneat the soil surface, to determine the amount of water lost through evapotranspiration (ET; Figure 1).

Figure 1: Precision weighing lysimeter at the Arkansas Valley Research Center in Rocky Ford, Colorado. The lysimeter is comprised of a monolith enclosed beneath the soil surface, which is continuously weighed to determine the amount of water lost through evapotranspiration (ET). Additionally, researchers may often install a weather station on/near the monolith for characterizing reference ET. Image Credit: Dr. Allan Andales

However, data generated by precision weighing lysimeters are often complex, noisy, and voluminous, posing a significant barrier to their widespread use, reliability, and adoption. The intricate nature of the data requires specialized knowledge and significant time investment to analyze and interpret, which can be a deterrent for many users. One major challenge is identifying and deriving ET during non-standard events (NSEs). An NSE is any event that affects the lysimeter monolith's weight outside of the typical ET process (i.e., water leaving the monolith). Examples of NSEs include rainfall, irrigation, fertilizer application, monolith drainage, and other disturbances that can affect the lysimeter's weight at the same time that normal ET is occuring. The time and other resources needed to merge, clean, and process weighing lysimeter data in lieu of these NSEs can take months, and often requires a dedicated research technician for data quality assurance.

To address this challenge, this python module titled lysimeter-analysis simplifies the data analysis by automating NSE and noise detection, and then calculates resulting actual crop ET values from lysimeter weight data (ETa). lysimeter-analysis can also compare calculated ETc values to ASCE Penman-Monteith (ASCE-PM) reference ET (ETo and ETr for short and tall crops, respectively) derived from the pyfao56 python module, and determine the crop coefficient (Kc) for the given crop growing on the lysimeter. The package includes functionality for data merging, lysimeter calibration, NSE detection, weather data processing, and report generation.

By reducing the complexity and time required for data analysis, this automated process encourages the broader use of precision weighing lysimeters, and lets researchers focus on obtaining meaningful results, ultimately contributing to more efficient and sustainable agricultural practices.

Note

This module was created originally to serve the Arkansas Valley Lysimeter Project, located at the Colorado State University (CSU) Arkansas Valley Research Center (AVRC). For more information, review Andales et al. (2018).

Installation

To use the module, ensure you have Python 3.6 or higher installed. Then, clone the repository and install the module.

git clone https://github.com/ansleybrown1337/lysimeter-analysis.git

Then follow the instructions on the "How-To" guide to install the required packages and run the analysis.

Note

After the authors complete the beta testing phase, the module will be available for installation via PyPI (i.e., pip) instead of forking this repository and using the local installation method described in the how-to guide.

Usage

See the detailed "How-To" guide created for installing and running this script in python via terminal commands.

Alternatively, if you are not interested in coding yourself, you can find an online application here:

This streamlit application allows you to upload your data and run the analysis without writing any code. It is a user-friendly interface that guides you through the process of analyzing lysimeter data, using the run_analysis.py script created in this repository as the backend.

Code Core Functionalities

1. Data Processing (dat_file_merger.py): Merges .dat files containing raw lysimeter (and sometimes weather) data.

2. Calibration (load_cell_calibration.py): Applies calibration to the load cell data using specified coefficients (alpha and beta), converting mV/V to kg.

3. Non-Standard Events Detection (non_standard_events.py): Identifies non-standard events (NSEs) such as rainfall, irrigation, etc. using an automatic algorithm.

4. Weather Data Processing (weather.py): Uses user-provided weather data to calculate ASCE-PM ETr and derive crop coefficients from processed lysimeter data.

5. Water Balance Calculation (water_balance.py): Takes raw lysimeter weight data, calculates ETa from it, and interpolates over NSE events and noisy ET data.

6. Report Generation (report_generator.py): Creates a model run report for users to have record of parameter settings for each model run.

7. Utility Functions (utils.py): General utility functions to support the above modules.

8. Running the Analysis (run_analysis.py): This script ties all the above functionalities together, processing the data end-to-end based on user input parameters.

9. Online Application (lysimeter_app.py): A script that creates the online version of the run_analysis.py script that's user friendly and eliminates the need for python on a local device.

Workflow

graph TD
    %% Define subgraph for data collection
    subgraph 1[Data Collection]
    lysimeter[Raw Data from Lysimeter]
    weather[Weather Data from METS]
    nse[Manual Non-Standard Events]
    end

    %% Define subgraph for utils.py
    subgraph U[utils.py]
    U3[Aggregate Data]
    U1[export_csv]
    U2[AWAT Filter]
    end

    %% Define subgraph for dat_file_merger.py
    subgraph A[dat_file_merger.py]
    A1[Load Data]
    A2[Clean Data]
    A3[Calibrate Data]
    end

    %% Define subgraph for non_standard_events.py
    subgraph B[non_standard_events.py]
    B1[Detect NSEs]
    B2[Plot NSEs]
    end

    %% Define subgraph for load_cell_calibration.py
    subgraph C[load_cell_calibration.py]
    C1[Set Calibration Parameters]
    end

    %% Define subgraph for water_balance.py
    subgraph D[water_balance.py]
    D1[Calculate ETa]
    D2[Interpolate ETa]
    D3[Calculate Cumulative ETa]
    D4[Plot ETa with NSEs]
    D5[Plot Cumulative ETa]
    end

    %% Define subgraph for weather.py
    subgraph W[weather.py]
    W1[Import daily weather data]
    W2[Calculate daily ASCE ETref]
    W3[Compare ETref and lysimeter ETc]
    W4[Calculate Kc crop coefficient]
    W5[Plot results]
    end

    %% Define subgraph for report_generator.py
    subgraph E[report_generator.py]
    E1[Create Analysis Report]
    end

    %% Define the flow between the functions
    lysimeter --> A1
    weather --> A1
    nse --> B1
    A1 --> A2 --> A3 --> B1 --> C1 --> D1 --> D2 --> D3
    D3 --> D4 
    D3 --> D5
    D3 --> U3
    U3 --> W3
    W1 --> W2 --> W3 --> W4 --> W5

    %% Final export of results
    B2 --> F[Export Results]
    E1 --> F
    D3 --> F
    D4 --> F
    D5 --> F
    U1 --> F
    W5 --> F

Loading

Algorithm Validation

To illustrate the accuracy of the lysimeter-analysis package, a comparison was made between the manual data cleaning process used previously and the algorithms found in the lysimeter-analysis package. The comparison was made using 2022 data from the CSU AVRC large lysimeter (LL). The results of this comparison are presented below.

lysimeter-analysis vs. Previous Method

Previous to lysimeter-analysis, lysimeter data was cleaned manually using Excel. This involved the following steps:

1. Raw data files are collected from the lysimeter datalogger
2. Raw data are manually exported into CSV files, then merged into single excel file
3. Weather sensor data from the merged CSV were calibrated using manufacturer-provided coefficients
4. Looking at 48-hour time windows in an excel graph, non-standard events (NSEs) were identified and flagged manually
5. Weight data during an NSE were recorded (e.g., how much water was added during an irrigation event), and ETa was estimated using the ETa rates before and after the NSE, an linearly interpolated between those two datetimes to estimate ETa during the NSE.
6. Weights for each day were recorded at 00:00, and ETa was calculated using the difference in weight between each of these times, accounting for recorded NSE event weight changes.
7. Cumulative ETa was calculated by summing daily ETa values, including linearly interpolated ETa values during NSEs.

This process took weeks to complete, and was prone to human error. The lysimeter-analysis package was developed to automate this process, and to reduce the potential for human error. One especially helpful feature of the package is the ability to automatically detect NSEs, and interpolate over them to estimate ETa during these events, not only at the daily timescale, but also at the hourly and 15-minute timescales. This allows for a more accurate estimation of ETa during NSEs, and a more accurate cumulative ETa calculation.

Here is a timeseries graph showing an example of the lysimeter-analysis package's ability to interpolate over NSEs, avoid noisy ET spikes, and to estimate ETa during these events (Figure 2).

Figure 2: An example of the lysimeter-analysis package's ability to interpolate over NSEs, avoid noisy ET spikes, and to estimate ETa during these events. The orange dotted line is the raw lysimeter ETa data, the blue line is the lysimeter-analysis-derived ETa data, and the red dots are duto-detected NSEs.

Comparison Results

Using 2022 data from the CSU AVRC large lysimeter (LL), lysimeter-derived ETa values were compared from 1) the manual data cleaning process used previously, and 2) the lysimeter-analysis package. Comparison results showed very strong agreement between the two methods, with root mean squared error (RMSE) values and index of agreement values of 0.324 mm and 0.999, respectively. Visually, this can be seen in the one-to-one graph presented in Figure 3.

Figure 3: A one-to-one plot of manually- (x-axis) and python-derived (y-axis) daily cumulative ETa values in millimeters. The black dashed line is the one-to-one line and indicates perfect agreement.

Additionally, a timeseries plot is shown to illustrate how both methods of ETa derivation illustrate an expected trend, and align closely with one another (Figure 4). Daily ETa rates start small early on in the growing season, peaks in late July and early August, then shrinks until harvest on October 15, when the crop on the monolith is removed.

Figure 4: A timeseries line plot showing ETa rates derived from the original, manual NSE water balance method (blue) and from the lysimeter-analysis module (red). ASCE-PM ETr derived from a local weather station placed on a well-watered alfalfa crop is shown for reference (orange).

Overall, goodness of fit between the two methods were good enough that using lysimeter-analysis should be considered a reasonable substitue to the previous method, where NSEs were identified "by-hand" using 48-hr chunks of data in excel to map timeseries lysimeter weights, coupled with a logbook recording irrigations and other NSEs captured when the technician was on the clock. This "Old Method" required many weeks of work to accomplish a full ETa timeseries, whereas lysimeter-analysis was able to accomplish statistically similar results in less than two minutes.

Data Disclosure

The data provided in this Repository is for example purposes only, and has been anonymized to prevent unintentional interpretation or use in other applications. Users are explicitly discouraged from interpreting, sharing, or using this data for applications other than illustrative or educational examples related to the Repository's described purposes.

References

• Thorp, K. R., DeJonge, K. C., Pokoski, T., Gulati, D., Kukal, M., Farag, F., Hashem, A., Erismann, G., Baumgartner, T., & Holzkaemper, A. (2024). Version 1.3.0 - pyfao56: FAO-56 evapotranspiration in Python. SoftwareX, In review.

• DeJonge, K. C., Thorp, K. R., & Marek, G. W. (2020). The apples and oranges of reference and potential evapotranspiration: Implications for agroecosystem models. Agricultural & Environmental Letters, 5(1), e20011. https://doi.org/10.1002/ael2.20011

• Andales, A., Straw, D., Marek, T., Simmons, L., Bartolo, M., & Ley, T. (2018). Design and construction of a precision weighing lysimeter in Southeast Colorado. Transactions of the ASABE, 61, 509-521. https://doi.org/10.13031/trans.12282

• Allen, R. G., Walter, I. A., Elliott, R. L., Howell, T. A., Itenfisu, D., Jensen, M. E., & Snyder, R. L. (Eds.). (2005). The ASCE Standardized Reference Evapotranspiration Equation. American Society of Civil Engineers. https://doi.org/10.1061/9780784408056