Adversarial Observations in Weather Forecasting

This is the code repository accompanying our paper Adversarial Observations in Weather Forecasting.

Dependencies

We package all dependencies using an apptainer container for ease of reproduction. Our forked variant of the GenCast package is included in this repository under src/graphcast.

Quickstart

Make sure all dependencies are installed on your machine before running the attack pipeline. Once they are satisfied, you can setup the container environments and download necessary data by running the scripts/build.sh script. After successful execution, the scripts/run.sh script can be run to execute the whole evaluation.

We assume that all scripts are executed with an appropriate CUDA-capable GPU. All test were performed on NVIDIA A40 gpus with 48 GB VRAM.

1. scripts/build.sh (Estimated time: 5h)
2. Evaluation of attacks
   1. Full evaluation: scripts/run.sh (Estimated time: 203h)
   2. Partial evaluation: scripts/run_short.sh (Estimated time: 8h)

This will produce the file data/report/report.pdf. Depending on if you have run the full evaluation or the short evaluation, this report should match either expected_results/report_full.pdf or expected_results/report_short.pdf, respectively.

Manual execution

If you want to run through every step manually, you can. Just use the shell scripts as a guide, or this readme for a bit of further description.

You can build the container using

apptainer build container.sif container.def

and then start an interactive session (with support for Nvidia CUDA) using

apptainer run --nv container.sif bash

The instructions in the following sections assume that you are running such an interactive session

Data

Full evaluation requires downloading the ERA5 dataset, available from ECMWF. This can best be accessed as Zarr from Weatherbench2's ERA5 data. For your convenience, you can simply run the following command in the apptainer to download the data required for evaluation.

python3 src/cache_era5.py

The target locations are computed using GHS-UCDB R2024A. To ease exact reproduction, we provide the random locations and times chosen by us at data/weather_evaluation_targets.json. Alternatively, a new list can be generated by

python3 src/generate_weather_targets.py

The variance in the weather variables is estimated by running

python3 src/estimate_error_variance.py

Visualization of the case studies requires data for political administrative boundary data. We use geoBoundaries and the respective file data/geoBoundariesCGAZ_ADM0.geojson can be downloaded manually using

wget https://github.com/wmgeolab/geoBoundaries/raw/main/releaseData/CGAZ/geoBoundariesCGAZ_ADM0.geojson -O data/geoBoundariesCGAZ_ADM0.geojson

Validate installation of dependencies

You can run scripts/check.sh to validate that the container is built correctly and all dependencies are included. The script runs a short end-to-end test (single step, single target). If everything is working as expected, it should complete in 10 compute-minutes and print

Everything seems to be fine.

as the final line.

Evaluation of fabricating extreme weather predictions

We assume that all data from the previous section is set up. In that case, you can run the evaluation using the following commands:

python3 src/evaluate.py --epsilons 0.02 0.03 0.04 0.05 0.07 --target=wind
python3 src/evaluate.py --epsilons 0.02 0.03 0.04 0.05 0.07 --target=temperature
python3 src/evaluate.py --epsilons 0.02 0.03 0.04 0.05 0.07 --target=precipitation

Ablation study

python3 src/evaluate.py --epsilons 0.03 0.05 --offset 0 --locations 34 --target=wind
python3 src/evaluate.py --epsilons 0.03 0.05 --offset 34 --locations 34 --target=temperature
python3 src/evaluate.py --epsilons 0.03 0.05 --offset 68 --locations 32 --target=precipitation

Case Studies on manipulating extreme weather predictions

python3 src/case_study_heat.py
python3 src/case_study_wind.py
python3 src/case_study_rain.py

Report summarizing results

To generate a final report, you can run

python3 src/generate_report.py

which will produce the file data/report/report.pdf. Depending on if you have run the full evaluation or the short evaluation, this report should match either expected_results/report_full.pdf or expected_results/report_short.pdf, respectively.

License and Disclaimers

The used model weights were made available for use under the terms of the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International (CC BY-NC-SA 4.0). You may obtain a copy of the License at: https://creativecommons.org/licenses/by-nc-sa/4.0/.

Unless required by applicable law or agreed to in writing, all software and materials distributed here under the Apache 2.0 or CC-BY-NC-SA 4.0 licenses are distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the licenses for the specific language governing permissions and limitations under those licenses.

The attacks on weather models described here are part of an experimental research project. You are solely responsible for determining the appropriateness of using or distributing the attacks or any outputs generated and assume all risks associated with your use or distribution of this code and outputs and your exercise of rights and permissions granted to you under the relevant License.

The evaluation of the attacks uses ECMWF's ERA5 data. Data and products of the European Centre for Medium-range Weather Forecasts (ECMWF). Modified Copernicus Climate Change Service information 2023. Neither the European Commission nor ECMWF is responsible for any use that may be made of the Copernicus information or data it contains. ECMWF HRES datasets Copyright statement: Copyright "© 2023 European Centre for Medium-Range Weather Forecasts (ECMWF)". Source: www.ecmwf.int License Statement: ECMWF open data is published under a Creative Commons Attribution 4.0 International (CC BY 4.0). https://creativecommons.org/licenses/by/4.0/ Disclaimer: ECMWF does not accept any liability whatsoever for any error or omission in the data, their availability, or for any loss or damage arising from their use.

Use of the third-party materials referred to above may be governed by separate terms and conditions or license provisions. Your use of the third-party materials is subject to any such terms and you should check that you can comply with any applicable restrictions or terms and conditions before use.

The original GenCast and GraphCast code is licensed under the Apache License, Version 2.0. You may obtain a copy of the License at: https://www.apache.org/licenses/LICENSE-2.0.