PyData APIs notebook

[hammer]

Given the philosophy of sgkit to rely on PyData APIs when possible, I thought it might be nice to have a notebook that spends time introducing and exploring PyData projects using sgkit data structures.

[hammer]

Some thoughts on what to cover, inspired by our GWAS tutorial and the Xarray tutorial at SciPy 2020.

• NumPy basics using ds.call_genotype.values
• Xarray data structures: Dataset and DataArray and especially coordinates
• Xarray operations
  • Creating new data variables, e.g. ds["variant_contig_name"]
  • Creating coords (i.e. indexing), e.g. ds2 = ds.set_index({"variants": ..., ds.reset_index("samples")
  • Positional and labeled selection along dimensions and indexes
  • Split-apply-combine, e.g. ds_annotations.samples.count(), ds_annotations.SuperPopulation.groupby, count variants per sample
  • Combining data
• Zarr (tutorial, spec)
  • sorted(os.listdir('1kg.zarr'))
  • print(open('1kg.zarr/.zmetadata').read()) (cf. consolidating metadata)
  • z1kg = zarr.open_consolidated('1kg.zarr')
  • z1kg.call_genotype.info
  • z1kg.tree() (not working)
• Dask basics
  • Chunking
  • .compute()
  • .visualize()
  • Web UI
  • Performance reports
  • Dask cloud provider
• Dask integration with Xarray
  • .load(), .persist()
  • .apply_ufunc(), .map_blocks()
• Dask internals: video, code
• Numba for making things faster?
• CuPy and other ways to target a GPU?
• Pandas? Cf. Pandas for Everyone, user guide, Pandas 1.x Cookbook
• Pangeo tutorial gallery likely also helpful

[jeromekelleher]

Sounds great - I'd find this useful personally!