Skip to content

Latest commit

 

History

History
151 lines (103 loc) · 4.56 KB

Readme.md

File metadata and controls

151 lines (103 loc) · 4.56 KB
Raw

Broad Band LAP

Developed in the Laboratory of Atmospheric Physics of Thessaloniki, Greece.

To process the data from broadband instruments of LAP.

Some plots and reports should be found here.

This is partial used in operational procedures (github.com/thanasisn/CS_id is still in use).

Table of Contents

Data status overview

Name Rows Vars Values Size Fill
BBDB 16737121 71 407158146 2.3 GiB 34.26%
BBDB meta 11629 80 476711 3.4 MiB 51.24%
TrackerDB 8576719 23 77184381 165.6 MiB 39.13%
TrackerDB meta 3139 9 15323 259.0 KiB 54.24%
Raw files hashes 760483 4 3041932 4.0 MiB 100%
Total 26089091 187 487876493 2.4 GiB NA%

Table: Datasets sizes on 2024-11-05 (continued below)

Bytes/Value
6.01
7.44
2.25
17.31
1.38
5.39

What it does

For CHP-1

  • Digest raw data
    • Signal from CHP-1
    • Tracker "async"
    • CHP-1 internal temperature from thermistor
  • Bad data ranges flagging
    • From manual set execution ranges
    • From acquisition signal physical limits
  • Converts signal to radiation
    • Computes temperature correction when possible
  • Plots
    • Overview of Clean/Dirty signal
    • Daily signal with and without dark
    • Overview of Direct radiation measurements
    • Daily Direct radiation measurements

For CM-21

  • Digest raw data
    • Signal from CHP-1
  • Bad data ranges flagging
    • From manual set execution ranges
    • From acquisition signal physical limits
  • Converts signal to radiation
  • Plots
    • Overview of Clean/Dirty signal
    • Daily signal with and without dark

Other processes

  • Quality Check of radiation data (QCRad)
    • Flags data using mainly the algorithm of C. N. Long and Y. Shi (2006)
  • Imports data from github.com/thanasisn/TSI
    • Sun_Dist_Astropy Sun - LAP distance
    • TSI_TOA TSI at TOA at LAP
    • TSI_1au TSI
    • TSI_source TSI data source
  • Imports atmospheric pressure data from proxies
    • Pressure Atmospheric pressure at LAP
    • Pressure_source Data source
  • Keeps an md5sum of all input files to check for bit rot and other data corruption.

Tools

  • inspect_days_DB.R interactive plot of some data in the DB
  • inspect_days_Lap.R interactive plot of some data from source files
  • inspect_days_Lap_sirena.R interactive plot of some data from source files

TODO

  • Fully port all to duckdb
  • Replace and compare processes from "CM_21_GLB"
    • All the major stages have been replaced
    • Secondary processes are to be ported
  • Process more instruments
  • Import libRadtran data
  • May import CSid
  • Import other references

Details

Development and Design

Some aspects on the implementation of this project.

  • We use a dataset of parquet files as a database for all measurements and additional data.
  • We are migrating the original parquet dataset scheme to Duckdb to improve overall efficiency.
  • The parquet dataset use one file for each month, this facilitates:
    • Syncing of the data between different computers.
    • Partial processing when needed without using the dataset function.
  • It should be easy to migrate to a pure database like duckdb or sqlite.
  • There are some files with extra meta data for the data in the database and the analysis performed.
  • We use features of the arrow library, and also data.table when it is more suitable or clear to code.
  • The analysis should be able to be performed with under 8Gb of RAM, but is not assured.
  • There is a trade-of with the disk usage/wearing, especially when starting from scratch.
  • New data should be easy to be added on daily base on all levels.
  • New process and analysis should be easy to added for all data.
  • Goal to become a framework for all broadband instruments data analysis and manipulation.

Documentation and usage

There is no centralized documentation for the project. Although you can refer to:

  • Readme.md or other markdown files for a relevant overview
  • Summary notes on the start of each script
  • Comments inside each script
  • Compiled reports from each script