Refine product_volume.ksh script (ASI and GMAG downloads, versus products processed locally)

[jameswilburlewis]

This script produces reports of data volume for a given time period in several different categories: data retrieved from THEMIS probes and GBO stations; processed data (L0/L1/L2 products, orbit and summary plots, ASI images, movies, and mosaics), and data archived at SPDF (probe L0/L1/L2, GMAG CDFs, but not ASI products).

I had a little trouble figuring out which GMAG and ASI data was retrieved from the GBO sites, versus what we processed after it was downloaded.

Nick and Cindy: What directories should I be looking at to find the downloaded ASI and GMAG data for the THEMIS GBO sites?
I need to be able to find all of it going back to the start of the mission -- is it all in the same place, or did some of it go to different directories as our download/processing scripts evolved over time?

[clrussell90404]

Hi Jim, Each GMAG station data is downloaded into GMAG mirrored directories. Those files are located at: /disks/themisdata/thg/mirrors/mag/nnn (where nnn is the network name/type) and includes the networks aari_ascii, lrv_ascii, magstar_csv, uatha_ascii, falcon_rmd, maccs_ascii, mcmac_rmd, uatha_rmd, fmi_ascii, step_ascii, ucalgary_rmd, greenland_ascii, ualaska_netcdf, ucla_rmd, intermagnet_ascii, ualberta_ascii usgs_ascii Downloaded data formats are varied and depend on which network is being downloaded. Some are in ascii, csv, and or rmd formats. All L2 CDF GMAG data (except Greenland) is located at: /disks/themisdata/thg/l2/mag/sss/yyyy (where sss = 3 or 4 character site name, and yyyy = year) The exception to the L2 CDF GMAG location is the Greenland data. It is in its own directory. /disks/themisdata/thg/greenland_gmag/l2/sss (where sss is the 3 character site name) I think Greenland was one of the first networks added to the THEMIS GBO and EPO network (by Pat Cruce and Lydia Philpott). The Greenland code is very old, and it was probably decided at the time it would be kept separate from the THEMIS data. Hope this helps. Cindy

…

________________________________ From: Jim Lewis ***@***.***> Sent: Monday, November 4, 2024 12:07 PM To: spedas/bleeding_edge ***@***.***> Cc: Russell, Cindy ***@***.***>; Assign ***@***.***> Subject: [spedas/bleeding_edge] Refine product_volume.ksh script (ASI and GMAG downloads, versus products processed locally) (Issue #197) This script produces reports of data volume for a given time period in several different categories: data retrieved from THEMIS probes and GBO stations; processed data (L0/L1/L2 products, orbit and summary plots, ASI images, movies, and mosaics), and data archived at SPDF (probe L0/L1/L2, GMAG CDFs, but not ASI products). I had a little trouble figuring out which GMAG and ASI data was retrieved from the GBO sites, versus what we processed after it was downloaded. Nick and Cindy: What directories should I be looking at to find the downloaded ASI and GMAG data for the THEMIS GBO sites? I need to be able to find all of it going back to the start of the mission -- is it all in the same place, or did some of it go to different directories as our download/processing scripts evolved over time? — Reply to this email directly, view it on GitHub<#197>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/A5NNFAT324DW625OHEHH77TZ66ZVBAVCNFSM6AAAAABRE5OPM6VHI2DSMVQWIX3LMV43ASLTON2WKOZSGYZTGNRSGA2TGMQ>. You are receiving this because you were assigned.Message ID: ***@***.***>

[nickssl]

For ASI:

We download three types of images (low res, hi res, rego):
https://themis.ssl.berkeley.edu/data/themis/thg/mirrors/asi/

Everything else is created using these images.

Products:

1. cdfs with low and hi res images (used only for storing info, they are not used for creating other products).
   https://themis.ssl.berkeley.edu/themisdata/thg/l1/asi/
2. summary plots (per day all stations, and also per station, per hour, per minute)
   https://themis.ssl.berkeley.edu/themisdata/thg/l0/asi/2020/11/
3. keograms (per day, all stations)
   https://themis.ssl.berkeley.edu/themisdata/thg/l0/asi/2020/11/
4. keogram cdfs (per day, all stations)
   https://themis.ssl.berkeley.edu/themisdata/thg/l1/asi/ask/
5. average (per day, all stations)
   https://themis.ssl.berkeley.edu/themisdata/thg/l0/asi/2020/11/
6. mosaics (every 3 secs, all stations)
   https://themis.ssl.berkeley.edu/themisdata/thg/l0/asi/2020/11/11/MOSA/
7. movies (every 10 mins, all stations)
   https://themis.ssl.berkeley.edu/themisdata/thg/l0/asi/2020/11/11/MOVI
8. rego keograms (per day, all stations)
   https://themis.ssl.berkeley.edu/themisdata/thg/l0/asi/2020/11/
9. rego cdfs
   https://themis.ssl.berkeley.edu/themisdata/thg/l1/reg/
10. rego keogram cdfs
    https://themis.ssl.berkeley.edu/themisdata/thg/l1/reg/ask/

Most of the above are created with low res images every day, then they are replaced by creating them again with high res images periodically (filenames remain the same). The programs that create the products from hi res images currently do not work correctly, they are giving errors. I am recreating the software for all the above, I expect to finish this in a month. And then I have to run the reprocessing using the full images, this will take a long time (perhaps months), because of the size of the data files.

Sizes:
One day of hi res images is 22GB, it takes about an hour to download it from Calgary using wget.
One day of low res images is 231MB, takes about 3 minutes to download it from Calgary using wget.

The number of images per day varies, depending on the length of the night at a particular station (and the size of the data files also varies, the 22GB is only a typical size, not the average). The number of stations that are alive also varies. The hi res images are not available at Calgary every day for all stations, for some stations the data is available only after they collect the hard disks (usually, twice per year).

[nickssl]

For the GMAG availability, we have this page:
https://themis.ssl.berkeley.edu/gmag/gmag_list.php?selyear=4000&selmonth=13&smap=on&sinfo=on&saelist=on&ae=on

For the ASI availability, we have this page:
https://themis.ssl.berkeley.edu/gmag/asi_list.php?selyear=4000&selmonth=13&smap=on&sinfo=on&seltxt=0

(On the daily ASI availability, the table on the right, we also show how many hours of data per day is available, like 13, 14 etc and clicking on that number opens the directory with the files.)