SEVIRI Wildfire Notebook

[samueljackson92]

Long overdue: a PR for contributing a SEVIRI data demonstrator notebook to the Environmental AI book. This notebook:

• Use satpy to load, visualise, and regrid SEVIRI level 1.5 data.
• Fetch a fire database containing some 880 fires from September 1s5, 2020.
• Visualisation of fire pixels from the database
• Visualisation of the fire pixels alongside bands from the SEVIRI satellite data.

@acocac let me know if there is anything I need to adjust or add.

[review-notebook-app]

Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB

[review-notebook-app]

View / edit / reply to this conversation on ReviewNB

acocac commented on 2021-11-04T09:06:18Z
----------------------------------------------------------------

For the header:

• In highlights, I would expand that you use satpy embedded/framed/called into a custom intake driver?

Also, for the latest version of the EnvAI book, the new sensor notebooks includes further metadata of the datasets explored. Can you please fill the following fields for the SEVIRI and fire pixel data. See below the example of the MODIS wildfire notebook:

Dataset originator/creator

MOD021KM

• MODIS Characterization Support Team (MCST)
• MODIS Adaptive Processing System (MODAPS)

Firms

• University of Maryland

Dataset authors

MOD021KM

• MODIS Science Data Support Team (SDST)

Firms

• NASA’s Applied Sciences Program

Dataset documentation (here just provide me the relevant links e.g. technical report or paper)

• 
  

[review-notebook-app]

View / edit / reply to this conversation on ReviewNB

acocac commented on 2021-11-04T09:06:19Z
----------------------------------------------------------------

It seems there is a missing package, pyorbital for plotting the seviri scene so we have to install it too. Also , I would suggest removing the leading white space

!pip -q install pyspectral
!pip -q install 'satpy==0.26.0'
!pip -q install gdown
!pip -q install pyorbital

[review-notebook-app]

View / edit / reply to this conversation on ReviewNB

acocac commented on 2021-11-04T09:06:20Z
----------------------------------------------------------------

After the import, the latest version of the EnvAI Book includes a new section of setting the project folder structure. This structure facilitates to organise the input/output files a given directory. In your case you only have to add the following cells:

## Set project structure

notebook_folder = '../sensors/wildfires-sensors-seviri'
if not os.path.exists(notebook_folder):
    os.makedirs(notebook_folder)

[review-notebook-app]

View / edit / reply to this conversation on ReviewNB

acocac commented on 2021-11-04T09:06:20Z
----------------------------------------------------------------

For the EnvAI book, the notebook won't render with the lines above as it requires to indicate the credentials while the notebook is rendered. To avoid it, I'm holding a copy of the filename inside the notebook project folder and use the conditional to skip the downloading process. I would also suggest to use os.path.join to combines one or more path names into a single path.

I suggest to change the above cells to:

filename = 'MSG4-SEVI-MSG15-0100-NA-20200901104243.230000000Z-NA.nat'
if not os.path.isfile(os.path.join(notebook_folder, filename)) or os.path.getsize(os.path.join(notebook_folder, filename)) == 0:
    consumer_key = getpass('Consumer Key: ')

    consumer_secret = getpass('Consumer Secret: ')
    token_url = 'https://api.eumetsat.int/token'

    response = requests.post(

            token_url,

            auth=requests.auth.HTTPBasicAuth(consumer_key, consumer_secret),

            data = {'grant_type': 'client_credentials'},

            headers = {"Content-Type" : "application/x-www-form-urlencoded"}

        )
    access_token = response.json()['access_token']
    product_url = 'https://api.eumetsat.int/data/download/collections/EO%3AEUM%3ADAT%3AMSG%3AHRSEVIRI/products/MSG4-SEVI-MSG15-0100-NA-20200901104243.230000000Z-NA/entry'

    product_url += f'?name={filename}'

    product_url += f'&access_token={access_token}'
    urllib.request.urlretrieve(product_url, os.path.join(notebook_folder, filename))

[review-notebook-app]

View / edit / reply to this conversation on ReviewNB

acocac commented on 2021-11-04T09:06:21Z
----------------------------------------------------------------

Have you preprocessed the pixel data? from which source? Maybe, it would be better to upload the sample fire pixel data to a zenodo repository linked to this notebook. We did something similar with the IceNet demo. I preprocessed certain sample data using the IceNet codebase and then uploaded to a Zenodo repository.

[review-notebook-app]

View / edit / reply to this conversation on ReviewNB

acocac commented on 2021-11-04T09:06:21Z
----------------------------------------------------------------

Line #1.    path = Path(tempfile.mkdtemp())

remove

[review-notebook-app]

View / edit / reply to this conversation on ReviewNB

acocac commented on 2021-11-04T09:06:22Z
----------------------------------------------------------------

Line #2.    catalog_file = path / 'catalog.yaml'

We can now save the catalog into the notebook project folder, please replace the line by:

# set catalogue location
catalog_file = os.path.join(notebook_folder, 'catalog.yaml')

[review-notebook-app]

View / edit / reply to this conversation on ReviewNB

acocac commented on 2021-11-04T09:06:23Z
----------------------------------------------------------------

wowwww so nice earth!

[review-notebook-app]

View / edit / reply to this conversation on ReviewNB

acocac commented on 2021-11-04T09:06:23Z
----------------------------------------------------------------

ok, now I got the point of the distortions (:

[review-notebook-app]

View / edit / reply to this conversation on ReviewNB

acocac commented on 2021-11-04T09:06:24Z
----------------------------------------------------------------

I might be good to point key papers or website explaining further details of the FRP concept e.g. Wooster et al. or another research group.

[review-notebook-app]

View / edit / reply to this conversation on ReviewNB

acocac commented on 2021-11-04T09:06:25Z
----------------------------------------------------------------

Nice tip about rasterize=False! very useful for the reader experimenting a local machine or Binder.

[review-notebook-app]

View / edit / reply to this conversation on ReviewNB

acocac commented on 2021-11-04T09:06:25Z
----------------------------------------------------------------

It's worth to mention you create/define a simple intake driver to read satpy objects.

[acocac]

@samueljackson92 Thanks for the contribution, very nice work! I added some suggestions to make the notebook compatible with the latest version of the EnvAI book. Find below some general comments:

• We need to work on the best strategy to keep the fire pixel data in safer and more stable storage e.g. Zenodo instead of your personal Google Drive.
• It would be great to highlight the custom intake driver. Indeed, please mention you successfully integrate satpy and intake to fetch MODIS data by cross-referencing the MODIS notebook.
• I've found great you mention the tip of rasterize=False!

[samueljackson92]

Hi @acocac. Thanks for the comments. I think I agree with all your comments. I'll find some time to make the suggested changes. I agree Zenodo is probably the best option for stability. I also think it might be worth spinning out the intake driver to a separate repo/library.

[acocac]

Hi @samueljackson92, thanks for the reply, no rush. I'll find a way to keep a Zenodo repo linked to a research group, in this case the Environmental AI book. I'll keep you informed about my progress here.

[samueljackson92]

@acocac Updated!

The only thing I have not changed is the requirement for Google Drive. For now, I wait for the Zenodo solution or similar.

[acocac]

@samueljackson92, the notebook is ready to go so we can release it after a test in Binder tomorrow morning.

FYI, I pushed some changes to fully integrate the notebook with the new template describing further metadata, fix some minor typos and replace fetching the fire sample data from gdrive by a zenodo repository hosted by the Environmental Data Science community. I've found this route could lead us to follow FAIR principles to share/curate future sample data for the Environmental AI book.

Finally, as part of the transparency and credits, it would be great if we can add some key metadata to the Zenodo repository i.e. further details of the original source and preprocessing steps would be very helpful e.g. how did you subset the dates? In addition, can you also share your ORCID?