Standard library proposals

[milancurcic]

Edit: Moved to https://github.com/fortran-lang/stdlib

---

I'm opening this meta issue with the goal of a broad and open-ended discussion around specific proposals that aim to add intrinsic procedures to the standard. Recent examples of such proposals are: #96, #100, #101, #103.

While I'm personally in support of most such proposals, I often see them as putting the cart before the horse. Why so? A common argument for such additions is that they are commonly used and often re-implemented over and over again because they're not available as intrinsics. I get this. However, if such intrinsics are really needed, I imagine there'd be an obvious choice of a 3rd party library available. Reusable and open libraries emerge when the pain is high enough in the community. That's how datetime-fortran came about. Lack of open source libraries may be a signal that there just isn't great need for such procedures.

Should we, as the Fortran community, consider working on a non-standard library that would include such procedures that we aim to the propose to the committee for inclusion in the standard? This library, or a subset of it, could over time become a candidate for the Fortran standard library, in addition to built-in intrinsics.

I foresee multiple benefits:

• Near-immediate quality of life improvement for many application developers like myself.
• In the process we'd flesh out what's really useful and what's not so important; we could troubleshoot caveats in the implementations before they're committed in writing in the standard.
• We'd have concrete examples to show to the committee, which would make it easier to argue for their inclusion in the language;
• A maintained general purpose library could ease and increase adoption of Fortran by newcomers to the language.
• Implementations of new procedures could be used to solicit use cases.

There are quite a few projects in this spirit, such as Arjen's flibs and Ondrej's fortran-utils. Such libraries could serve as design examples, and perhaps even contribute implementations toward a dedicated non-stdlib.

[certik]

I think this is a great idea. The fortran-utils project began as a necessity, where I collaborated with as many people as I could, but that was about 10 years ago, and we now have a much better community and a will to collaborate on a "standard library", with the idea that that what is in there are strong candidates for inclusion into the standard. (Sort of like Boost in C++.) I am imagining that we would carefully discuss a design of each function in there. With many eyes and experiences that we have, I think it can be really well designed.

Indeed, currently the committee always puts the carriage in front of the horse. The fix for language features is to first implement them in a compiler. That will come in a few years, hopefully even sooner. But for library additions (this feature), the fix is to first implement them in the "standard library", get it used in projects, gain some experience, and then write up a proposal and get it into the standard.

What should we call this library?

[jacobwilliams]

FYI: @apthorpe had some good thoughts about this a couple years ago:

• http://www.acorvid.com/2017/12/13/what-i-miss-when-writing-fortran/
• http://www.acorvid.com/2017/12/11/what-should-a-fortran-standard-library-contain-part-1/
• http://www.acorvid.com/2017/12/12/what-should-a-fortran-standard-library-contain-part-2/

[marshallward]

An ancillary issue is how much of such a library should be written in Fortran itself. Most libc implementations that I've seen are predominantly written in C, with occasional assembly code (most notably crt0.s). The CPython Standard Library, on the other hand, seems to be a blend of Python and C.

At the risk of repeating myself, I think the answer to this question would help to more strongly define what can and cannot (or ought and ought not) be done within the constraints of the language itself, and perhaps reveal more precisely when an extension of the core language is required.

Another issue is whether particular intrinsics ought to be part of such a library, rather than part of the language itself. I know that my own group would like more control over the behavior of many transcendental functions like exp() for the sake of bit reproducibility, for example.

Overall, I am very supportive of something like this and would love to contribute to its development.

[milancurcic]

@certik I don't know about the name, for the time being we can refer to it as just standard library, and see if it meets adversity. My guess is that it'd be welcome and embraced by most of the community. If a better name candidate comes up, we'll know. More important are the processes that we adopt to elect candidates for stdlib, and how to implement them. Your strong experience with SymPy will come in useful here as well.

I think a good model in the beginning would be for stdlib candidates to evolve from certain proposals in this repo. If it's a request for adding an intrinsic procedure or module to the language, we tag is as stdlib-candidate, and issues that gather strong support by the community can "level up" to experimental implementation in the stdlib. The stdlib could be another repo under github.com/j3-fortran. This repo would also be a place to include unit and regression tests, benchmarks, and of course the stdlib docs.

@marshallward Good points, and I don't see a good reason to limit this to pure Fortran implementation. For example, sorting can be done with pure Fortran, but file system procedures would need quite a bit of C. I think this is OK.

Whether some intrinsics should be part of a 3rd party library or part of the language, we'll learn that along the way. If we find that a procedure or module should be standardized, we'd use this as a stepping stone in that process.

[certik]

@milancurcic I agree. Let's call it the "Fortran Standard Library", or stdlib in short. I would like to try get (informal) support / blessing from the committee for this effort. In SymPy or fortran-utils the bar to get a new feature in is lower --- one just has to send a PR and convince the core developers to merge it. In this stdlib the bar to get a feature in should be much higher --- there needs to be a proposal in this repository, it needs to have support from the wider community, and then also ideally several members of the committee would "approve" it. And the goal of stdlib would be that if the Fortran Standard Committee decides that Fortran should have a standard library specified in the Fortran Standard itself, this stdlib will be it. In particular, we want to prevent a situation that in the future the committee decides to have a standard library, and they would not like some API of stdlib and decide to change it in backwards incompatible way. So for that reason I want to have a buy in of the committee from the very beginning.

@sblionel what do you think?

[FortranFan]

@certik wrote:

.. Let's call it the "Fortran Standard Library", or stdlib in short. ..

@milancurcic , @certik, @marshallward ,

It's great to see your enthusiasm toward this effort, the value of this to the Fortran community is immeasurable and priceless. I really hope the ISO IEC working group WG5 will wholeheartedly support and cooperate with such an initiative.

It will be truly remarkable if stdlib can cover the 4 broad categories noticed with 'standard libraries' of other languages:

1. Containers
   a. string type
   b. bitsets
   c. Enhanced 'array' types such as vectors, singly-linked and doubly-linked lists, etc,
   d. Adapters such as stacks, queues, etc.
   e Associative ones such as dictionaries (maps), hash_sets, etc.

2. Algorithms
   a. Generic methods for sort, findloc, etc. that can work with any type, intrinsic and derived,
   b. Operations and permutations on a range of elements such as merge/union, difference,
   etc.

3. Utilities
   a. Iterator-like facilities which make it easy to work with Containers,
   b. Operator (<, >, ==, etc. ) and assignment(=) overload abstracttions that perhaps make
   the use of standard algorithms more efficient?
   c. Miscellaneous other functions, subroutines (like generic swap), datetime, named
   constants, etc,

4. Special aspects: a placeholder for any fundamental needs that get identified as this initiative progresses:
   a. Any basic facilities (extensions perhaps to ABSTRACT INTERFACE block?) needed toward
   "special" functions such as Variadic ones in the language e.g., MAX, MIN
   b. Ability to "overload" array subsection notation facility with Containers that standard Fortran
   provides with its 2 built-in containers: arrays and CHARACTER intrinsic type.
   c. Any special mechanisms that can help aid with improved constructors of arrays/containers
   and derived types ('classes'). I envision certain fundamental 'computer engineering' aspects
   being pursued here that can enable, say, efficient operation on the diagonal of a matrix or
   initialization to an identity matrix; or efficient 'dynamic' construction of 'classes' in Fortran similar
   to that is achieved near universally using new keyword in other languages.

[FortranFan]

@jacobwilliams wrote:

FYI: @apthorpe had some good thoughts about this a couple years ago:

• http://www.acorvid.com/2017/12/13/what-i-miss-when-writing-fortran/
• http://www.acorvid.com/2017/12/11/what-should-a-fortran-standard-library-contain-part-1/
• http://www.acorvid.com/2017/12/12/what-should-a-fortran-standard-library-contain-part-2/

Agree.

Thoughts by @apthorpe under the "Release and Distribution Model" e.g.

I have a background in nuclear safety and writing code to the ASME NQA-1 standard for
safety-related quality assurance where strict configuration management is the norm.
Having worked in both fast-paced startups and in the glacially-paced nuclear industry,
I have to say I prefer the more thoughtful release model. I feel it’s a better fit for Fortran
culture and it offers a more consistent, predictable, and reliable experience for end users

are particularly important for all the users I work with.

[ivan-pi]

For the past five years I've been programming in Fortran, I have always felt there is a lack of a central library for common things such containers, string handling, IO, etc., but also general numerical routines for interpolation, LAPACK interfaces, ODE solvers, evaluating polynomials and more. It seems like each of us has (re-)implemented these tools at some point in time, wasting lots of human hours. Ironically, many wrapped Fortran libraries are much easier to use from Python (via SciPy) than from Fortran itself.

Would a Fortran stdlib also cover such routines for numerical work or should there be a second library for scientific Fortran routines? I imagine the scientific library would more or less reimplement what the NAG library and the former IMSL library (now RogueWave) already have. I wonder if perhaps these two libraries might be the reason why we have no open-source scientific Fortran library, as many Fortran users (or there employers) in the past simply bought a license.

Finally, I wanted to mention the Flibcpp project which provides Fortran bindings to the C++ standard library using auto-generated code from SWIG-Fortran. Would we allow the Fortran standard library to have different backends (e.g. via submodules)? For example one with "hand-written" Fortran/C routines, and another one which is just a thin wrapper of the C++ one. It would be interesting to see if this library effort gained momentum, would the compiler vendors slowly shift to this standard library interface and provide there own specialized implementations (kind of like the Intel distribution of Python which has linked up Numpy/SciPy with the Intel Math Kernel Library).

[apthorpe]

I'm very heartened to see this pick up again, especially with people I've met since I originally posted my thoughts on the matter. I've been working with C++ and the Boost library for about a year and I like their approach. A number of Boost proposals and libraries have made it into the C++ standard library with minor modifications. This allows early adopters to exercise the code, find limitations, etc., prior to making a proposal to the standards committee. Much of the developmental 'churn' takes place long before standards proposals are made. When Boost libraries are accepted to the standard library, it's usually very easy for end-developers to migrate from Boost to stdlib routines. My preference is to concentrate on areas where traditionally there hasn't been much emphasis, especially data import/export, string processing, containers, and search/sort. Numerical code is covered well (for example, Intel Fortran ships with optimized versions of BLAS/LAPACK). Regular expressions, hashes (dict/map/associative array), and data validation and import utilities would substantially reduce the burden of parsing, processing, and validating input. I've seen a surprisingly large fraction of application code devoted to input processing, much of which could be improved or eliminated with regexes or hashes. As for implementation language, I wouldn't limit it to just Fortran. While one can write a regular expression library in Fortran, one probably shouldn't, especially when C++11 already has a perfectly usable regular expression implementation. My inclination is to stay with mature standardized languages such as C++, C, etc. If Fortran is workable, great; in many cases it isn't. To simplify adoption, it may be wise to restrict implementation languages to C, C++, and Fortran, at least initially.

…

On Sun, Dec 1, 2019 at 6:22 AM Ivan ***@***.***> wrote: For the past five years I've been programming in Fortran, I have always felt there is a lack of a central library for common things such containers, string handling, IO, etc., but also general numerical routines for interpolation, LAPACK interfaces, ODE solvers, evaluating polynomials and more. It seems like each of us has (re-)implemented these tools at some point in time, wasting lots of human hours. Ironically, many wrapped Fortran libraries are much easier to use from Python (via SciPy) than from Fortran itself. Would a Fortran stdlib also cover such routines for numerical work or should there be a second library for scientific Fortran routines? I imagine the scientific library would more or less reimplement what the NAG <https://www.nag.com/numeric/fl/nagdoc_latest/html/frontmatter/manconts.html> library and the former IMSL <https://www.roguewave.com/products-services/imsl-numerical-libraries/fortran-libraries> library (now RogueWave) already have. I wonder if perhaps these two libraries might be the reason why we have no open-source scientific Fortran library, as many Fortran users would simply buy a license. Finally, I wanted to mention the Flibcpp <https://github.com/swig-fortran/flibcpp> project which provides Fortran bindings to the C++ standard library using auto-generated code from SWIG-Fortran. Would we allow the Fortran standard library to have different backends (e.g. via submodules)? For example one with "hand-written" Fortran/C routines, and another one which is just a thin wrapper of the C++ one. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#104?email_source=notifications&email_token=AAIPODKKOFJOIL2PXZBD72LQWOT6ZA5CNFSM4JSZ6FO2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEFRHOEQ#issuecomment-560101138>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAIPODPAE55AXSTR5R3MJCLQWOT6ZANCNFSM4JSZ6FOQ> .

[rweed]

As a first step towards a standard library, i think it would be useful if we had a complete list (as possible) of currently available open source projects/libraries that address the things that most people would agree are of immediate need (for me thats enhanced string handling, sorting, and ADT containers and algorithms (lists, dictionaries/unordered maps, stacks, queues, etc.). If the authors of the projects are ameanable, their code could be forked into a standard repostitory that aggregates the code into one place. Then we could begin a community effort to evaluate the various ways people implemented the needed functionality we envision for a standard fortran library and write reveiws on what we liked about their approach, what we didnt' like and how appropriate the code would be as a candidate for inclusion into the standard library. I think this would also benefit the original authors because it would provide useful feedback for their individual projects. We could also encourage people to consider placing their code on the repository first prior to standing up their own github site.

[cmacmackin]

This is just a small contribution, but I did come up with a way of storing arbitrary data, with run-time type-checking: https://github.com/cmacmackin/PolyCon. At one point I'd planned to use this to build a bunch of ADT containers, but I never got around to it. The code is currently GPL licensed, but I'd be willing to release it under something more permissive if people could find a use for it.

[jacobwilliams]

• I'm all for this!
• I would prefer this activity to be 100% community led. It would be great to have some support or blessing from the Fortran committee. But, I don't wish to have a situation where we have to wait for approval from the committee or some number of members to get code merged in. That would be fatal mistake, I think.

[certik]

@sblionel what do you think of this effort? What would be the best way to get an approval / blessing by the committee for this, per the discussion above?