Survey: Frontend Language Support

[ax3l]

Dear @openPMD/general-contributors,

this issue is pinned to keep track of your needs for our reference API in terms of minimum supported language features.

So far we support Python 3.6+ and C++14 or newer. But as compilers and languages progress, we would love to drop old language revisions in newer releases to keep maintenance at a minimum and make use of newer language features in public interfaces. Please vote in the following comments according to the minimum requirements you have in your downstream code-bases. This will help us to plan potentially breaking changes with minimum hazzle.

Voting in the C++ and Python frontend will help us to ping and interact with you in case we would like to drop, e.g. C++14 bindings (for C++17 and newer) or older Python bindings in the future.

Voting on new language bindings does not implicate someone will find the time to provide such implementations, but it potentially motivates volunteers to step forward and add further bindings or share their own implementations where needed.

[ax3l]

C++ Frontend

As of today, I (only) need support for

• 👍 C++11 and newer (we supported this up to release 0.12.0)
• 🎉 C++14 and newer (this is the oldest one we currently support)
• 🚀 C++17 and newer
• ❤️ C++20 and newer
• 😄 C++23 or newer

You can update your vote at any time, e.g., if your code-base updated.

Note: C++ compiler defaults and C++ compiler support.

[ax3l]

Python Frontend

As of today, I (only) need support for

• Python 3.5 and newer (we supported this up to release 0.12.0)
• 👍 Python 3.6 and newer (we support this up to release 0.14; this is the oldest one we currently support)
• 🎉 Python 3.7 and newer
• 🚀 Python 3.8 and newer
• ❤️ Python 3.9 and newer
• 👀 Python 3.10 and newer

You can update your vote at any time, e.g., if your code-base updated.

[ax3l]

C Frontend

We do not yet provide a C-frontend. If one would spend time developing one, I (only) need support for

• 😕 ANSI C/C90 or older
• 👍 C99 and newer
• 🎉 C11 and newer (introduces standardized complex numbers)
• 🚀 C18 and newer
• ❤️ C2x and newer

You can update your vote at any time, e.g., if your code-base updated.

Note: C compiler defaults.

Potential library usage: shroud

[ax3l]

Fortran Frontend

Currently, there are at least two implementations of a HDF5-Fortran openPMD library [ref1] which we hightly recommend to check out and use.
We have no plans yet to do so, but just to mention the technical possibilities: one could in principle, based on implementing a C-Frontend first, use the former to implement Fortran bindings herin as well. This would widen the supported backends to include, e.g., ADIOS1 and ADIOS2 for all Fortran users. Alternatively, one could also add these backends via their native Fortran bindings to the aforementioned libraries.

If one would add Fortran bindings to openPMD-api, I (only) need support for

• 😕 older versions of Fortran (e.g. Fortran 2003)
• 👍 Fortran 2008 and newer
• 🎉 Fortran 2018 and newer

Note: Fortran bindings prior to the 2008 standard are extremely cumbersome (read as: not worth spending time developing), since iso-C-bindings were first standardized in version 2008.
Potential library usage: shroud

You can update your vote at any time, e.g., if your code-base updated.

[ax3l]

More Exotic Bindings

It's in principle possible to add bindings for further (modern) languages in case there is a sufficiently large user-base for it. Just tell us what you currently use in production workflows, maybe someone wants to team up with you and build bindings!

I use the following languages in my workflows and I would benefit from adding bindings to

• 👍 Javascript Compile-Target: JS/WebASM|wasm #295
• 🎉 Julia (e.g. via libcxxwrap): Julia bindings #1025
• 🚀 R (e.g. via Rinside or Rcpp)
• 👀 Matlab (non-free) / GNU Octave
• 😄 Mathematica (non-free) (docs)

(Multiple answers are welcome, too.)

[mccoys]

I know I should use the front-end already ... Will do when I have time. I envision only c++11 & py3.5+ for now

[ax3l]

Sounds great, feel free to ping us if you need help with anything when adopting to the API (reviews, more examples, etc.). Since Python 3.5's EOL is in Q3 2020 I do not expect any drop of our supported versions soon until it runs into this.
Conda-Forge already transitioned to ship Python 3.6+ only, but that's just one of our many release channels.

[ax3l]

Announcement: since most user-level distributions already dropped Python 3.5 and it will not receive any updates after September 2019, we anticipate to drop Python 3.5 support in upcoming releases.

This approach is on par with other major projects such as Numpy dropping Python 3.5 in upcoming releases (1.19.0+). We continue to support Python 3.6-3.8 and anticipate addition of Python 3.9 after its October release.

Please see: https://devguide.python.org/#status-of-python-branches

Update: Python 3.5 dropped and 3.9 added in #828 (0.13.0 release). We already ship Python 3.9 via conda-forge for 0.12.0.

[williamfgc]

I see a label for Julia, are you guys invested on it? I'm currently evaluating its use and seems very promising now that it's been stable for a year.

[ax3l]

Unfortunately not, I know exactly one person in our community that does Julia @berceanu

[bussmann]

Hi William, yes, Julia support would be lovely. As ax3l pointed out, community-wise the interest is unfortunately limited, so the manpower is as well. Julia is IMHO quite o.k., but the parallelism does not come out of the box as promised (it never did). Why would you got for Julia? What would be the features you are missing with Python?

[williamfgc]

@ax3l @bussmann thanks for chiming in. Python is great, nothing is missing with the added ecosystem (numpy, scipy, matplotlib, etc.), still dependency versions must be tracked when packaging with conda, pip, etc.. We are mostly exploring Julia for the long run (still young in 2020, I just hit a wall myself with IDE support) as it presents interesting concepts built into the language (type declarations, the parallelism you mentioned, math) targeting the kind of work we do in their roadmap. I just got mostly curious as I saw the label in this issue and I wanted to learn more about your view/experiences. Thanks.

[ax3l]

With our main downstream consumers now upgraded to C++14 code bases, we'll soon migrate to C++14, too. This will simplify the maintenance of the library and we can stop testing some ancient compilers that only came with now deprecated operating systems.

Many compilers already switched to build with C++14 by default and some already set C++17 as their default compilation language. C++ is generally forward-compatible and we make sure in our CI that our C++ standard (e.g. 14) requirement will built with C++17 and 20 flags, too. We will generally aim to continue to adopt new C++ standards, as soon as vendor compilers and HPC systems provide stable support for them; and thus once our main downstream users can make the switch.

Update: transition to C++14 and newer will happen with the 0.13.0 release #825

[eschnett]

@axl3 Incidentally, I started working on a Julia package openPMD_api_jll today.

[eschnett]

C++: C++17
Python, C, Fortran: n/a
Julia: yes please!

[ax3l]

Awesome, thanks for the update & contributions @eschnett! :)

[ax3l]

C++ Frontend Announcement:
With WarpX and PIConGPU both upgrading to C++17, we will likely transition to require C++17 #1164 in openPMD-api with release 0.15 and newer 🎉

[ax3l]

Python Frontend Announcement:
Since Python 3.6 is EOL in three days, the last release line that will support it is 0.14.*. It's likely that major user-level package managers, like conda-forge, and release support software, like cibuildwheel, will drop support soon after (to make room in build resources for newer releases, like Python 3.10).
https://endoflife.date/python

[ax3l]

@skuschel do you still use Python 3.6 in production? Just checking in, feel free to update your current needs here by updating your reaction emoji :)

[berceanu]

Hi William, yes, Julia support would be lovely. As ax3l pointed out, community-wise the interest is unfortunately limited, so the manpower is as well. Julia is IMHO quite o.k., but the parallelism does not come out of the box as promised (it never did). Why would you got for Julia? What would be the features you are missing with Python?

A short update regarding Julia. While the maturity and diversity of the ecosystem still lags somewhat behind that of Python, I feel that the community has progressed a lot in recent years, especially on issues related to parallelism and GPU kernel generation. Here are two recent comparative papers:

• Julia as a unifying end-to-end workflow language on the Frontier exascale system

• Evaluating performance and portability of high-level programming models: Julia, Python/Numba, and Kokkos on exascale nodes

Probably @williamfgc has more details on this, as first author on both studies

[DavidSagan]

@williamfgc Note: There is a Bmad (https://www.classe.cornell.edu/bmad/) port to Julia project in development and part of this project will be a Julia front end for the EXT_BeamPhysics.md extension.