Merge branch 'master' into ppival-patch-5

README.md

@@ -2,12 +2,16 @@
 
 [![Deployment](https://github.com/awesomedata/apd-core/actions/workflows/deploy.yaml/badge.svg)](https://github.com/awesomedata/apd-core/actions/workflows/deploy.yaml)
 
-Next iteration of APD project.
+The core meta of awesome-public-datasets.
 
 ## How to contribute
 
 Please refer to the instructions of [CONTRIBUTING](https://github.com/awesomedata/apd-core/blob/master/CONTRIBUTING.md) or latest [Wiki](https://github.com/awesomedata/apd-core/wiki).
 
 ## Contact
 
-Xiaming Chen <chenxm35@gmail.com>
+Xiaming Chen, chenxm35@gmail.com
+
+<a href="https://github.com/Bai-Yu-Lan">
+<img src="https://raw.githubusercontent.com/awesomedata/apd-core/master/logo/baiyulan.PNG" width="300">
+</a>

core/Biology/ANHIR.yml

@@ -0,0 +1,22 @@
+---
+title: ANHIR
+homepage: https://anhir.grand-challenge.org/
+category: Biology
+description: Automatic Non-rigid Histological Image Registration (ANHIR) consists of 2D histological microscopy images.
+version:
+keywords: histology
+image:
+temporal:
+spatial:
+access_level:
+copyrights:
+accrual_periodicity:
+specification:
+data_quality: false
+data_dictionary:
+language:
+license:
+publisher:
+organization:
+issued_time:
+sources: []

core/Biology/CIMA.yml

@@ -0,0 +1,22 @@
+---
+title: CIMA
+homepage: https://cmp.felk.cvut.cz/~borovji3/?page=dataset
+category: Biology
+description: CIMA dataset includes images of 2D histological microscopy tissue slices.
+version:
+keywords: histology
+image:
+temporal:
+spatial:
+access_level:
+copyrights:
+accrual_periodicity:
+specification:
+data_quality: false
+data_dictionary:
+language:
+license:
+publisher:
+organization:
+issued_time:
+sources: []

core/Biology/Complete-Genomics-Public-Data.yml

@@ -1,6 +1,6 @@
 ---
 title: Complete Genomics Public Data
-homepage: http://www.completegenomics.com/public-data/69-genomes/
+homepage: https://completegenomics.mgiamericas.com/demodata
 category: Biology
 description: A diverse data set of whole human genomes are freely available for public use to enhance any genomic study or evaluate Complete Genomics data results and file formats. These include 69 DNA samples sequenced using our Standard Sequencing Service, which includes whole genome sequencing, mapping of the resulting reads to a human reference genome, comprehensive detection of variations, scoring, and informative annotation.
 version:

core/Climate+Weather/Open-Meteo.yml

@@ -0,0 +1,53 @@
+---
+title: Open-Meteo - Open-Source Weather API
+homepage: https://open-meteo.com
+category: Climate+Weather
+description: Open-source weather API with free access for non-commercial use. No API key required.
+version:
+keywords: weather, climate
+image:
+temporal: 80 years
+spatial: 1 km
+access_level: public
+copyrights:
+accrual_periodicity: hourly
+specification:
+data_quality:
+data_dictionary:
+language: en
+license: CC BY 4.0
+publisher:
+organization:
+issued_time:
+sources:
+  - name: NOAA GFS & HRRR Weather Model
+    access_url: https://www.nco.ncep.noaa.gov/pmb/products/gfs/
+  - name: DWD ICON Weather Model
+    access_url: https://opendata.dwd.de/weather/nwp/
+  - name: MeteoFrance AROME & ARPEGE Weather Model
+    access_url: https://mf-models-on-aws.org/en/doc/models/arpege-world/
+  - name: ECMWF IFS Weather Model
+    access_url: https://www.ecmwf.int/en/forecasts/datasets/open-data
+  - name: JMA Weather Model
+    access_url: https://www.jma.go.jp/jma/en/Activities/nwp.html
+  - name: Met Norway Weather Model
+    access_url: https://github.com/metno/NWPdocs/wiki/MET-Nordic-dataset
+  - name: Canadian Meteorological Centre GEM & HRDPS Weather Model
+    access_url: https://weather.gc.ca/grib/grib2_glb_25km_e.html
+  - name: ERA5 Historical Weather Reanalysis 
+    access_url: https://cds.climate.copernicus.eu/cdsapp#!/dataset/reanalysis-era5-single-levels?tab=overview
+  - name: IPCC Climate Models from CMIP6 HighResMIP
+    access_url: https://hrcm.ceda.ac.uk/research/cmip6-highresmip/
+  - name: CAMS global atmospheric composition forecasts
+    access_url: https://ads.atmosphere.copernicus.eu/cdsapp#!/dataset/cams-global-atmospheric-composition-forecasts?tab=overview
+  - name: GeoNames Location database
+    access_url: https://www.geonames.org
+  - name: Copernicus DEM 2021 release GLO-90 Elevation Model
+    access_url: https://spacedata.copernicus.eu/collections/copernicus-digital-elevation-model
+  - name: Global Flood Awareness System (GloFAS)
+    access_url: https://www.globalfloods.eu
+references:
+  - title: Open-Meteo Source Code under AGL-3.0
+    reference: https://github.com/open-meteo/open-meteo
+  - title: Open-Meteo Zenodoo Publication DOI URL
+    reference: https://doi.org/10.5281/zenodo.7970649

core/Government/Brazil.yml

@@ -1,6 +1,6 @@
 ---
 title: Brazil
-homepage: http://dados.gov.br/dataset
+homepage: https://dados.gov.br/dados/conjuntos-dados
 category: Government
 description:
 version:

core/Physics/Quantum.yml

@@ -1,7 +1,7 @@
 ---
 title: Quantum simulations of an electron in a two dimensional potential well
 homepage: http://doi.org/10.4224/PhysRevA.96.042113.data
-category: Quantum simulation
+category: Physics
 description: The data was generated in a numerical simulation of an electron in a 2 dimensional confining potential. It was used as a test case for training a deep neural network to reproduce the results of a partial differential equation (the time independent Schrödinger equation).
 version: 1.0
 keywords: quantum simulation, numerical modeling

core/Psychology+Cognition/Open-Cognitive-Science-Data-Repository.yml

@@ -1,6 +1,6 @@
 ---
 title: Open Cognitive Science Data
-homepage: https://nivlab.github.io/opendata
+homepage: https://nimh-dsst.github.io/OpenCogData/
 category: Psychology+Cognition
 description: Publicly available behavioral datasets from across cognitive science (with a strong focus on learning & decision-making). Maintained by the Niv Lab at Princeton University.
 version: 1.0
@@ -20,8 +20,8 @@ publisher:
   - name:
     web:
 organization:
-  - name: Niv Lab, Princeton University
-    web: https://nivlab.princeton.edu/
+  - name: Data Science and Sharing Team, National Institute of Mental Health
+    web: https://cmn.nimh.nih.gov/dsst
 issued_time:
 sources:
   - name:

core/Transportation/GeoLife-GPS-Trajectory-from-Microsoft-Research.yml

@@ -1,6 +1,6 @@
 ---
 title: GeoLife GPS Trajectory from Microsoft Research
-homepage: http://research.microsoft.com/en-us/downloads/b16d359d-d164-469e-9fd4-daa38f2b2e13/
+homepage: https://www.microsoft.com/en-us/download/details.aspx?id=52367
 category: Transportation
 description:
 version:

core/Transportation/Melbourne-pedestrian-counting.yml

@@ -0,0 +1,31 @@
+---
+title: Melbourne Pedestrian Counting
+homepage: https://data.melbourne.vic.gov.au/explore/dataset/pedestrian-counting-system-monthly-counts-per-hour/
+category: Transportation
+description: This dataset contains hourly pedestrian counts since 2009 from pedestrian sensor devices located across the City of Melbourne. 
+version: 1.0
+keywords: pedestrian count, foot traffic, pedestrian sensors, covid-19,traffic flow
+image: 
+temporal:
+spatial: 
+access_level: public
+copyrights:
+accrual_periodicity: 
+specification:
+data_quality: 
+data_dictionary: 
+language: en
+license: CC BY 4.0
+publisher:
+  - name: 
+    web: 
+organization:
+  - name: City of Melbourne  
+    web: https://www.melbourne.vic.gov.au/
+issued_time: 2014.05
+sources:
+  - name: 
+    access_url: 
+references:
+  - title: 
+    reference: 

deploy/index.mako

@@ -5,25 +5,31 @@ Awesome Public Datasets
    :alt: Awesome
    :target: https://github.com/sindresorhus/awesome
 
+This is a list of `topic-centric public data sources <https://github.com/awesomedata/awesome-public-datasets>`_
+in high quality. They are collected and tidied from blogs, answers, and user responses.
+Most of the data sets listed below are free, however, some are not.
+This project was incubated at `OMNILab <https://github.com/OMNILab>`_, Shanghai Jiao Tong University during Xiaming Chen's Ph.D. studies.
+OMNILab is now part of the `BaiYuLan Open AI community <https://github.com/Bai-Yu-Lan>`_.
+Other amazingly awesome lists can be found in `sindresorhus's awesome <https://github.com/sindresorhus/awesome>`_ list.
 
-.. |OK_ICON| image:: https://raw.githubusercontent.com/awesomedata/apd-core/master/deploy/ok-24.png
-.. |FIXME_ICON| image:: https://raw.githubusercontent.com/awesomedata/apd-core/master/deploy/fixme-24.png
+Special thanks to
 
+.. image:: https://raw.githubusercontent.com/awesomedata/apd-core/master/logo/baiyulan.PNG
+   :alt: BaiYuLanAI
+   :target: https://github.com/Bai-Yu-Lan
 
 **NOTICE**: This repo is automatically generated by `apd-core <https://github.com/awesomedata/apd-core/tree/master/core>`_.
-Please **DO NOT** modify this file directly. We have provided
-`a new way <https://github.com/awesomedata/apd-core/blob/master/CONTRIBUTING.md>`_
-to contribute to Awesome Public Datasets. `Join <https://join.slack.com/t/awesomedataworld/shared_invite/zt-dllew5xy-PJYi~mWUdY3hupohbmVZsA>`_ the `slack community <https://awesomedataworld.slack.com>`_ for more communication.
+Please **DO NOT** modify this file directly. We have provided a new way to `contribute to 
+this repo <https://github.com/awesomedata/apd-core/blob/master/CONTRIBUTING.md>`_. 
+`Join <https://join.slack.com/t/awesomedataworld/shared_invite/zt-dllew5xy-PJYi~mWUdY3hupohbmVZsA>`_ 
+the `slack community <https://awesomedataworld.slack.com>`_ for an instant touch of HQ data updates.
+
+.. |OK_ICON| image:: https://raw.githubusercontent.com/awesomedata/apd-core/master/deploy/ok-24.png
+.. |FIXME_ICON| image:: https://raw.githubusercontent.com/awesomedata/apd-core/master/deploy/fixme-24.png
 
 * |OK_ICON| I am well.
 * |FIXME_ICON| Please fix me.
 
-`This list of a topic-centric public data sources <https://github.com/awesomedata/awesome-public-datasets>`_
-in high quality. They are collected and tidied from blogs, answers, and user responses.
-Most of the data sets listed below are free, however, some are not.
-Other amazingly awesome lists can be found in `sindresorhus's awesome <https://github.com/sindresorhus/awesome>`_ list.
-
-
 .. contents:: **Table of Contents**
 
 % for category, data_entries in categories.items():

tests/validate.py

@@ -26,6 +26,7 @@ def scan_core_data(core_dir):
 def validate_classification(category_map):
     for category, entries in category_map.items():
         for entry in entries:
+            print(entry)
             try:
                 data_obj = yaml.load(open(entry), Loader=yaml.Loader)
             except Exception as e:

tools/requirements.txt

@@ -1,2 +1,2 @@
-requests==2.22.0
+requests==2.31.0
 ruamel.yaml>=0.15.35