Add function for run-time parsing

[alhirzel]

The @u_str macro is useful when units are known in advance. In some cases, however, units must be determined at runtime. This requires parsing strings as numeric and Unitful values, which is currently sort of ugly:

val, unit = split(rawValue, " "; limit=2)
return parse(Float64, val)*(@eval @u_str $unit) # ugly

I'd like to propose that Unitful include a function which exposes this kind of behavior (i.e. parse quantity from string). It would be good if it were resilient to having a space (or not) between the numeric and the unit.

[tlnagy]

I use Unitful in OMETIFF.jl and I annotate the axes with units, but I don't know the units beforehand so I have to something like this:

unittype = getfield(Unitful, Symbol(image["TimeIncrementUnit"])) # ugly hack

at runtime. It would be great to have a better, sanctioned way of doing this.

[c42f]

I just ran into this with regards to deserilaizing unit strings from a file. I do think we should have a better story for unit string (de)serialization.

It should be fairly easy to implement by processing the output of replace_value(context_module, Meta.parse(str)) to evaluate the limited set of allowed functions in the resulting Expr. But there's a few wrinkles:

• What do we do about unit extension modules? This is currently dealt with by the macro inferring context_module to be the current module, and using that to infer the set of "active unit extension modules". But the logic for this is a bit convoluted and confusing (cf. @u_str macro hygiene? #272 for example). For a runtime version it might be better just to have the user explicitly pass the allowed set of extension modules (with Unitful being the default).
• What about security in parsing unknown strings? Meta.parse itself is safe in principle, but replace_value looks up arbitrary symbols in the unit definition modules which seems a little iffy. Maybe it would be sensible to restrict the type of such lookups with an extra check.

[Datseris]

Just chiming in to say that we want something exactly like this in NCDatasets, for JuliaGeo/JuliaEarth/JuliaClimate. Actually we have the units as a separate string like "W m-2" and we'd like to use that string to make numeric arrays Unitful arrays. Seems that the only way to do this is at the moment is (@eval @u_str $unit), with unit our string, but I was expecting a normal and well tested function like tounit(str). It would lead to a more robust ecosystem if this had more support.

[c42f]

What's the format of your unit strings? If they're not julia syntax (as your example "W m-2" seems not to be) then u_str won't be able to parse them currently.

What is the desired format for serializing unit strings in general? Currently it's julia syntax which seems like a sensible default for interactive work. But it would also "be nice" to support a cross-language standard for unit strings if a suitable one exists.

I did some research but couldn't find a really obvious consensus. Some things I did find:

• The NIST SP811 is influential but doesn't specify a precise grammar for unit strings.
• The FITS standard is well developed but "overcomplete" allowing flexible parsing of various non-unique representations of a unit string. It's the standard followed in the astropy.units library and seems to have inspired various other standards and proto-standards such as ASDF which defers to FITS.
• The MIXF standard seems to be an elaborated version of SP811 for plain text unit strings provides a formal grammar. This seems promising but is rather opinionated about using . as multiplication which means incompatibility with julia syntax.
• The NetCDF format appears to use the udunits package which has a unit parser with a grammar also derived from SP811 with an extension for affine units.
• The python pint library seems to use python syntax for unit string constructions

The example "W m-2" appears to follow the FITS standard which interprets m-2 to be m^(-2).

[Datseris]

Yeap, in my case "W m-2" means Watt per meter squared. Well this function I am asking here should definitely make at least some standard things like interpreting all numbers as exponents of the previous symbol and also taking into account white space.

Searching for a super duper general standard might not even be feasible as different fields may write something differently. Having a starting point function from Unitful.jl that takes in a string and makes it a unit is good. Then we can improve it as time progresses, or if something is too specific for a specific package like e.g. NCDatasets.jl, the string can be first converted to a more appripriate form in the specialized package and then passed into Unitful.jl.

[c42f]

at least some standard things like interpreting all numbers as exponents

To be honest I found this quite surprising behavior! It's inconsistent with the way u_str currently works. Not to say it's bad or wrong; it appears to be a standard thing for FITS which allows several different representations for exponents. But I think it shows that there can be some subtlety even for "obvious" parsing behaviors.

As a baseline, I'd suggest we just implement the current u_str behavior as a function.

[c42f]

I implemented a baseline version of this in #298

[Datseris]

To be honest I found this quite surprising behavior!

Just so that we are clear, you found surprising that all numbers are exponents in a unit string? Because you would expect that ^ must always be explicitly stated?

[c42f]

Because you would expect that ^ must always be explicitly stated?

Yes.

In fact, u_str currently uses normal julia parsing rules, so u"m2" is looked up as a unit with the symbolic name :m2 in any of the unit extension modules. (No judgement implied about whether that's the ideal behavior, but that's what happens currently.)

[Datseris]

But since we are talking about units, which don't have a numeric value and therefore it does not make sense to multiply them by a number, wouldn't you agree that it seems more reasonable to interpret numbers as exponents? Like, the default behavior?

I am more surprised than you were initially that this is not the default behavior :P

[c42f]

wouldn't you agree that it seems more reasonable to interpret numbers as exponents

Perhaps. But of course m2 is a valid julia variable name. Unitful gives package authors freedom to name their custom units as they wish, according to the usual rules for constructing valid Julia identifiers. So I think there's quite a tension between m2 being a julia identifier, and your desire for "m2" to be parsed as m^2

[Datseris]

You have a good point that I didn't consider. So therefore I want to write a custom formatting function that does what I find "intuitive" formatting at a string and then gives it to parseunit.

Although I will soon open a proposal issue where the argument preformat can be part of parseunit, defaulting to identity, so that there is one "go-to" way to do such transformations.

[c42f]

To be honest I feel like string munging prior to parseunit is unlikely to work robustly for all edge cases. As I said at #298 (comment), you may need a custom NetCDF-specific parser to really get it right.