ENH: Linked Datasets (RDF)

[westurner]

ENH: Linked Datasets (RDF)

• This is very much a meta ticket.
• There are a number of bare links here.
• They are for documentation

(UPDATE: see westurner/pandasrdf#1)

Use Case

So I:

• retrieved some data
  • from somewhere
  • about a certain #topic
• perfomed analysis
  • with certain transformations and aggregations
  • with certain versions of certain tools
  • confirmed/rejected a [null] hypothesis

and I want to share my findings so that others can find, review, repeat, reproduce, and verify (confirm/reject) a given conclusion.

User Story

As a data analyst, I would like to share or publish Series, DataFrames, Panels, and Panel4Ds as structured, hierarchical, RDF linked data ("DataSet").

Status Quo: Pandas IO

How do I go from a [CSV] to a DataFrame to something shareable with a URL?

http://pandas.pydata.org/pandas-docs/dev/io.html

• http://pandas.pydata.org/pandas-docs/dev/dsintro.html
• https://github.com/pydata/pandas/blob/master/pandas/core/format.py
• https://github.com/pydata/pandas/blob/master/pandas/rpy/common.py

.

• Series (1D)
  • index
  • data
    • NumPy datatypes
• DataFrame (2D)
  • index
  • column(s)
    • NumPy datatypes
• Panel (3D)
• Panel4D (4D)

Read or parse a data format into a DataSet:

• pandas.read_*
  • read_clipboard
  • read_csv
  • read_excel
  • read_fwf
  • read_gbq
  • read_hdf
  • read_html
  • read_json
  • read_msgpack
  • read_pickle
  • read_sql
  • read_stata
  • read_table
• pandas.HDFStore
  • https://pandas.pydata.org/docs/dev/io.html#hdf5-pytables

Add metadata:

• Add RDF metadata (RDFa, JSONLD)

Save or serialize a DataSet into a data format:

• pandas.DataFrame.
  • to_csv
  • to_dict
  • to_excel
  • to_gbq
  • to_html
  • to_latex
  • to_panel
  • to_period
  • to_records
  • to_sparse
  • to_sql
  • to_stata
  • to_string
  • to_timestamp
  • to_wide
• to_ RDF
• to_ CSVW
• to_ HTML + RDFa
• to_ JSONLD
  • create a JSONLD @context

Share or publish a serialized DataSet with the internet:

• Email Attachment (Table in a PDF)
  • opendatahandbook.org
  • project-open-data.github.io
• FTP, SFTP, RSYNC, NFS
• HTML web upload form with metadata form fields
• CLI tool
• Version Control: Git, Hg, Svn
  • challenge: 'large' files ("binary blobs") in VCS systems
• HTTP API: Object Storage (~LDP)
  • GET/POST /container/filename.csv # [.json|.xml|.xls|.rdf|.html]
  • challenge: indexing metadata from a separate document / named graph
    • GET/POST to/container/filename.csv`
• Push to CKAN
  • http://docs.ckan.org/en/latest/api.html
  • http://docs.ckan.org/en/latest/linked-data-and-rdf.html
  • https://github.com/okfn/ckan
• Host DataSet metadata
  • python -m SimpleHTTPServer 8088
  • e.g. http://datasets.schema-labs.appspot.com/about Indexes http://schema.org/Dataset s

Implementation

What changes would be needed for Pandas core to support this workflow?

• .meta schema
• to_rdf for Series, DataFrames, Panels, and Panel4Ds
• read_rdf for Series, DataFrames, Panels, and Panel 4Ds
• ~@datastep process decorators
• ~DataSet
• ~DataCatalog of precomputed aggregations/views/slices.
• Units support (.meta?)

.meta schema

It's easy enough to serialize a dict and a table to naieve RDF.

For interoperability, it would be helpful to standardize with a common
set of terms/symbols/structures/schema for describing
the tabular, hierarchical data which pandas is designed to handle.

There is currently no standard method for storing columnar metadata
within Pandas (e.g. in .meta['columns'][colname]['schema'], or as a JSON-LD @context).

Ontology Resources

• http://www.w3.org/TR/rdf-schema/ (rdfs:)
• http://www.w3.org/TR/owl-overview/ (owl:)
• http://www.w3.org/TR/sparql11-query/#sparqlDefinition
• http://lov.okfn.org
• http://prefix.cc

CSV2RDF (csvw)

• http://www.w3.org/ns/csvw
• https://github.com/w3c/csvw
• https://w3c.github.io/csvw/

W3C PROV (prov:)

• http://www.w3.org/TR/prov-primer/#intuitive-overview-of-prov
• http://www.w3.org/TR/prov-o/
• http://www.w3.org/2011/prov/wiki/ProvImplementations
  • https://pypi.python.org/pypi/prov
  • http://prov.readthedocs.org/en/latest/usage.html

schema.org (schema:)

• http://schema.org
• http://www.w3.org/wiki/WebSchemas
• http://schema.rdfs.org/
• https://schema.org/docs/full.html :
  • schema:Dataset -- A body of structured information describing some topic(s) of interest.
    • [schema:Thing, schema:CreativeWork]
    • distribution -- A downloadable form of this dataset, at a specific location, in a specific format (DataDownload)
    • spatial, temporal
    • catalog -- A data catalog which contains a dataset (DataCatalog)
  • schema:DataCatalog -- collection of Datasets
    • [schema:Thing, schema:CreativeWork]
    • dataset -- A dataset contained in a catalog. (Dataset)
  • schema:DataDownload -- A dataset in downloadable form.
    • [schema:Thing, schema:CreativeWork]
    • contentSize
    • contentURL
    • uploadDate

W3C RDF Data Cube (qb:)

• http://www.w3.org/TR/vocab-data-cube/
• http://www.w3.org/2011/gld/wiki/Data_Cube_Vocabulary#The_history_of_Data_Cube.2C_SDMX-RDF_and_SCOVO
• http://www.w3.org/TR/vocab-data-cube/#vocab-reference :
  • qb:DataSet -- a collection of observations, possibly organized into various slices, conforming to some common dimensional structure
    • qb:Slice -- a subset of a DataSet defined by fixing a subset of the dimensional values.
  • qb:Observation -- a single observation in the cube, may have one or more associated measured values.
    • qb:dataset -- data set of which this observation is a part.
  • qb:ObservationGroup -- a, possibly arbitrary, group of observations.
    • qb:observation -- an observation contained within this slice of the data set.
  • qb:Slice -- a subset of a DataSet defined by fixing a subset of the dimensional values, component properties on the Slice.
  • [Components, Properties, Dimensions, Attributes, Measures]

to_rdf

http://pandas.pydata.org/pandas-docs/dev/io.html

Arguments:

• output fmt
• JSON-LD: compaction

.

• Series.meta
• Series.to_rdf()
• DataFrame.meta
• DataFrame.to_rdf()
• Panel.meta
• Panel.to_rdf()
• Panel4D.meta
• Panel4D.to_rdf()

read_rdf

http://pandas.pydata.org/pandas-docs/dev/remote_data.html

• Series.read_rdf()
• DataFrame.read_rdf()
• Panel.read_rdf()
• Panel4D.read_rdf()

Arguments to read_rdf would need to describe which dimensions of data to
read into 1D/2D/3D/4D form.

@datastep / PROV

• Objective: Additive journal of transformations
• Link to source script(s) URIs
• Decorator for annotating data transformations with metadata.
• Generate PROV metadata for data transformations

Ten Simple Rules for Reproducible Computational Research (3, 4, 5, 7, 8, 10)

DataCatalog

A collection of Datasets.

• DataCatalog = {that=df1, this=df1.group().apply(), also_this=df2]
  • 'this is an aggregation of that'
    • 'this' has a URI
    • 'that' has a URI
• What if there is no metadata for df2?

Units support

• Series.meta
• DataFrame.column.meta
• NumPy [, PyTables]
• http://docs.scipy.org/doc/numpy/reference/arrays.dtypes.html
• https://pint.readthedocs.org/en/latest/
• http://pythonhosted.org/quantities/

RDF Datatypes

• http://en.wikipedia.org/wiki/ISO_8601
• http://www.w3.org/TR/xmlschema-2/#decimal
• http://schema.org/Date
• http://schema.org/DateTime
• http://schema.org/Float
• http://schema.org/Quantity
• https://github.com/RDFLib/rdflib
  • from rdflib.namespace import XSD, RDF, RDFS
  • from rdflib import URIRef, Literal
  • https://github.com/RDFLib/rdflib-sqlalchemy (SQLAlchemy)

JSON-LD (RDF in JSON)

• https://github.com/digitalbazaar/pyld (JSON-LD)
• https://github.com/RDFLib/rdflib-jsonld (JSON-LD)

Linked Data Primer

Linked Data Abstractions

• Graphs are represented as triples of (s,p,o)
• Subject, Predicate, Object
• Queries are patterns with ?references
  • graph.triples((None, None, None))
  • SELECT ?s, ?p, ?o WHERE { ?s ?p ?o };
• subjects are linked to objects by predicates
  • subjects and predicate are identified by URI 'keys'

URIs and URLs

• a URI is like a URL
• usually, we expect URLs to be 'dereferencable` HTTP URIs
  • HTTP GET http://en.wikipedia.org/
• a URI may start with a different URI prefix
  • urn:
  • uuid:

SQL and Linked Data

• there exist standard mappings for whole SQL tablesets
  • rdb2rdf
  • similar to application scaffolding
  • ACL support adds complexity
• virtuoso supports SQL and RDF and SPARQL
  • standard mappings
  • virtuoso powers http://dbpedia.org/
    • dbpedia.org has a high degree of centrality
      • http://lod-cloud.net/
• rdflib-sqlalchemy maps RDF onto SQL tables
  • fairly inefficiently, when compared to native triplestores

Named Graphs

• Quads: (g, s, p, o)
• g: sometimes called the 'context' of a triple
• Metadata about GRAPH ?g
• Multiple named graphs in one file: TriX, TriG

Linked Data Formats

• NTriples
• RDF/XML
  • TriX
• Turtle, N3
  • TriG
• JSON-LD

Choosing Schema

• XSD, RDF, RDFS, DCTERMS
• Which schema is most popular?
• Which schema is a best fit for the data?
• Which schema will search engines index for us?
• What do the queries look like?
• Years Later... What is OWL?
• Why would we start with RDFS now?

Linked Data Process, Provenance, and Schema

DataSets have [implicit] URIs:

http://example.com/datasets/#<key>

Shared or published DataSets have URLs:

http://ckan.example.org/datasets/<key>

DataSets are about certain things:

e.g. URIs for #Tags, Categories, Taxonomy, Ontology

DataSets are derived from somewhere, somehow:

• where and how was it downloaded? (digital sense)
• how was it collected? (process control sense)

Datasets have structure:

• Tabular, Hierarchical
• 1D, 2D, 3D, 4D
• Graph-based
  • Chains
  • Flows
• Schema

5 ★ Open Data
http://5stardata.info/
http://www.w3.org/TR/ld-glossary/#x5-star-linked-open-data

☆ Publish data on the Web in any format (e.g., PDF, JPEG) accompanied by an explicit Open License (expression of rights).
☆☆ Publish structured data on the Web in a machine-readable format (e.g., XML).
☆☆☆ Publish structured data on the Web in a documented, non-proprietary data format (e.g., CSV, KML).
☆☆☆☆ Publish structured data on the Web as RDF (eg Turtle, RDFa, JSON-LD, SPARQL)
☆☆☆☆☆ In your RDF, have the identifiers be links (URLs) to useful data sources.

https://en.wikipedia.org/wiki/Linked_Data

Hi,

Thanks for the thoroughly-researched idea-issue.
Most of the spec links you provided are exploratory in practice, and have yet (to my knowledge)
taken over the world. Controversially probably, I'm including RDF in that statement, which
has certainly gotten a lot of attention and there are real services built on top of it (freebase,
opencalais, semantic search engines and so on).

DataFrame metadata has come up again and again, please read through the (long) metadata
discussion in #2495 to catch up on some the issues already discussed.
#3297 is planned for 0.12, but has nothing to do with RDF and has very limited scope, since
it's intended to answer a different use-case. However, users would be free to
embed their own JSON schemas under .meta, so it's somewhat open-ended.

The next step after that, embedding metadata in axis labels is interesting, but right now isn't
planned for a specific release. Although I'm sure, the `quantities' users would find that useful.

IMO, it's premature to bake these specs into pandas at this point in the life of
the semantic web.
Is there a fundamental reason why all this can't be done in an auxiliary package,
on top of pandas?

That's my opinion, other devs may feel differently.

Bringing over comments made by @westurner in GH3297 :

https://www.google.com/search?q=sdmx+json
http://json-stat.org

[westurner]

Thx.

[westurner]

From https://news.ycombinator.com/item?id=5657935 :

In terms of http://en.wikipedia.org/wiki/Linked_data , there are a number of standard (overlapping) URI-based schema for describing data with structured attributes:

• http://schema.org/docs/full.html
• http://schema.rdfs.org/all.json
• http://schema.rdfs.org/all.ttl (Turtle RDF Triples)
• http://rdfs.org/sioc/spec/
• http://json-ld.org/
• http://json-ld.org/spec/latest/json-ld/
• http://json-ld.org/spec/latest/json-ld-api/
• http://www.w3.org/TR/ldp/ Linked Data Platform TR defines a RESTful API standard
• http://wiki.apache.org/incubator/MarmottaProposal implements LDP 1.0 Draft and SPARQL 1.1

[westurner]

• http://schema.org/Dataset
• http://schema.org/DataCatalog
• http://schema.org/docs/full.html

[westurner]

@y-p

Most of the spec links you provided are exploratory in practice, and have yet (to my knowledge)
taken over the world. Controversially probably, I'm including RDF in that statement, which
has certainly gotten a lot of attention and there are real services built on top of it (freebase,
opencalais, semantic search engines and so on).

• https://en.wikipedia.org/wiki/Linked_data
• https://en.wikipedia.org/wiki/Semantic_web#Standards
• https://en.wikipedia.org/wiki/Semantic_search
• http://www.w3.org/wiki/WebSchemas
• http://blog.schema.org/

[dr-leo]

I stumbled upon this proposal while looking for SDMX tools that might help read economic data from Eurostat, the OECD, IMF, BIS and their likes. So a DataFrame.to_rdf method would need to be complemented by a read_sdmx function. Well, the mentioned data providers offer CSV files as well. But the benefits of working with XML and EDIFact-based formats such as described on http://sdmx.org/ are obvious.

I don't know what level of generality would be appropriate to IO just SDMX. But it might be interesting to look at Eurostat's SDMX Reference Implementation and the other material available at.

https://webgate.ec.europa.eu/fpfis/mwikis/sdmx/index.php.

Starting "small" with SCMX might be appropriate to do within pandas. A more general semantic web focused approach can be studied at http://www.cubicweb.org.

[benjello]

It would definitely be great to be able to read data from Eurostat, the OECD, IMF, BIS using pandas !

[jreback]

if someone is interested, could follow the paradigm of pandas.io.wb.py (The world bank dataset)
basically wrap functions to get the data and return a frame

[westurner]

read_sdmx would be great.

write_rdf would also be great. (to_triples)

TODO: re-topical-cluster globs of links in this thread. Here are three more:

• http://www.reddit.com/r/semanticweb/comments/1dvakc/schemaorgdataset_standard_schema_for_linked_data/ (schema.org: schema:, XSD, semantic web background)
• http://project-open-data.github.io/metadata-resources/ (Data.gov requirements)
• http://docs.ckan.org/en/latest/linked-data-and-rdf.html#schema-mapping (CKAN RDF: Dublin Core, DCAT, VoID, SCOVO)

[westurner]

It may well be easy enough to transform .meta to RDF.

The more challenging part is, IMHO, storing the procedural metadata while/in applying transforms to the Series, DataFrames, and Panels.

From a provenance and reproducibility standpoint: how do downstream users who are not reading the Python source which produced the calculations compare/review the applied data analysis methods (and findings) with RDF metadata?

[EDIT]

There should be a link to the revision id and/or version of the code in the .meta information.

[westurner]

General Ontology Resources:

• http://lov.okfn.org
• http://prefix.cc

[dr-leo]

All this looks very interesting.

Again, I recommend a deeper dive into CubicWeb, a web framework
supporting RDF and other semantic web standards. It also implements a
SparQL-like query language called RQL. Apart from reusing some of its
core components it seems worth exploring whether in the long term
CubicWeb could be used as a web front end for admin and representation
tasks relating to datasets.

There is no doubt a lot of speculation in these statements. But we
should avoid reinventing wheels.

the pandas.io.wb.py module is a children's game compared to teaching
pandas RDF. The latter goal should probably be pursued in a separate
project such as pandas-rdf or pandas-sdmx, as has been suggested before.
That said, I know nothing about the relationship between RDF and SDMX.

Writing pandas.io.eurostat.py, oecd.py bis.py modules along the lines of
wb.py should not be too difficult, especially if one focuses on CSV
formatted data. Still, using SDMX could make the user's life much easier
and richer.

To add both complexity and to the links collection: Some elements of the
SDMX standard build on EDIFACT. Here,

https://pypi.python.org/pypi/bots-open-source-edi-translator/version%203.0.0

could come in handy.

Am 25.07.2013 22:28, schrieb Wes Turner:

General Ontology Resources:

• http://lov.okfn.org
• http://prefix.cc

—
Reply to this email directly or view it on GitHub
#3402 (comment).

[westurner]

• http://pandas.pydata.org/pandas-docs/dev/io.html#google-bigquery-experimental
• http://www.reddit.com/r/bigquery/comments/1mt6r6/pandas_bigquery_io_module_github/ccoj7q4
• http://datasets.schema-labs.appspot.com/about (http://schema.org/Dataset)