In this project, we will develop a data pipeline to ingest, process, store it so you can access it through different means.
SEVIR: The Storm EVent ImagRy (SEVIR) dataset is a collection of temporally and spatially aligned images containing weather events captured by satellite and radar.
The dataset contains thousands of samples of 4 hour events captured by one or more of these weather sensors. This loop shows one such event:
-
Start by reading this tutorial on the dataset.
Storm Events Database: The database currently contains data from January 1950 to November 2020, as entered by NOAA's National Weather Service (NWS). Data are available on the Registry of Open Data on AWS. Dataset and the Website
More to read:
- Storm Data Export Format, Field names Event Details File (named storm_data_search_results.csv):
- SEVIR : A Storm Event Imagery Dataset for Deep Learning Applications in Radar and Satellite Meteorology:
- Python 3.7+
- Python IDE
- Code editor
- Amazon S3 Buckets
- Amazon Glue
- Amazon Athena
- Amazon Quicksight
- Google storage buckets
- Google Dataflow
- Google Bigquery
- Data studio
- Snowflake
- Sql-alchemy
- Apache Superset
Clone this repo to your local machine using https://github.com/goyal07nidhi/Data-Pipeline.git
Refer README.md inside the respective directories for setup instructions.
- ✅ AWS S3:
AWS - ✅ GCP - Dataflow, Datalab:
GCP - ✅ SNOWFLAKE:
SNOWFLAKE
- Nidhi Goyal
- Kanika Damodarsingh Negi
- Rishvita Reddy Bhumireddy