[FEATURE] Optimize for running at field level scale

[rohansaw]

Feature description

Avoid running exactly the same APSIM simulation if VeRCYe is run at field level scale and field fall within the same CHIRPS Pixel.

Problem it solves

Currently, if running CHIRPS at the field level scale, it is likely that we run the exact same simulation multiple times. The reason for this is that most APSIM parameters are static in a region and if the fields are close enough to each other, they might share the same weather data. E.g for CHIRPS the pixel resolution is 0.5 degrees, so all fields within one CHIRPS pixel, will have the same precipitation data.

For the met data from NASA Power this applies aswell with a different resolution from CHIRPS.

Suggested solution

Use some form of caching mechanism to save the already executed simulations with their parameters. Then copy these results for the current run instead of rerunning the simulation. Only the simulation-to-remotesensing matching is then run for the fields individually.

Additional context

No response

[rohansaw]

We might be able to use Snakemakes inbuilt caching. With this the hashes of the input apsim files should be computed and if they are the same, the cached results should be reused.

[wronk]

Yep, this is a key feature of snakemake. As you were alluding to in your Suggested solution, we should only need to track the input files needed to generate any specific output. If you define those input file paths in a rule in the Snakemake file, it shouldn't repeat any computation.

Apologies if you're already aware, but noting two tips I learned the hard way

• In case there is a process step/rule that doesn't generate any output file but does need to happen in a particular order, you can output a simple .log file as a hack. Then, identify that .log file as an input to the next step/rule. Even though this simple dummy file isn't actually used, Snakemake will register that as part of the processing chain and respect the processing order.
• There are also some flags you can pass when executing snakemake to run in specialized ways. For example, forcing snakemake to rerun (even when files didn't change) or avoiding re-running specific rules (even if the input files did). There are many special flags in the documentation and the --dry-run flag comes in handy if you're needing to do something special and trying to figure the exact command to accomplish it. That'll spit out the pipeline to be run, reasons for each step, and often useful tips too

[rohansaw]

That sounds great, thanks for the tips! :)

The main reason why we have duplicate computation for APSIM simulations is that even though the met data might be exactly the same and the params in the APSIM template are also the same for different fields (ROIs), the met data always has its own file in the respective ROI.

From my understanding Snakemake uses some hashing to check for already computing results. Since the APSIM file contains a line pointing to the met file with the respective path, the input file for APSIM will always have a different hash, even tough all APSIM inputs are the same.

Some quick thoughts on a possible solutions:

• Save met data globally (ROI independent). Allows us to use the same path in the APSIM file to automatically avoid recomputation.
• We can still copy the actual met files in their current format to the corresponding ROI folder.
• Will need to ensure that if the CHIRPS pixel is the same, the output met filepath is the same.

Will try to find some time in the next days to properly think this through.

[wronk]

The main reason why we have duplicate computation for APSIM simulations is that even though the met data might be exactly the same and the params in the APSIM template are also the same for different fields (ROIs), the met data always has its own file in the respective ROI.
From my understanding Snakemake uses some hashing to check for already computing results. Since the APSIM file contains a line pointing to the met file with the respective path, the input file for APSIM will always have a different hash, even tough all APSIM inputs are the same.

Gotcha, that makes sense. I know the APSIM simulations are pretty expensive from a runtime perspective. Is this causing things to run pretty slow for areas with smaller fields?

Your approach sounds good to me. I like that it lets snakemake handle much of the checking/handling if and just circumvents the APSIM hash check. That way you're not maintaining as much custom logic.

If you used the existing APSIM file substitution rule, you'd hopefully just need to come up with a new file naming system/code for the met_fpath input such that the same CHIRPS/APSIM combinations map to a single met file.