It's impossible to support multiple versions of defmt in a crate

[jonas-schievink]

If you do this:

[dependencies]
defmt01 = { version = "0.1.0", package = "defmt", optional = true }
defmt02 = { version = "0.2.0", package = "defmt", optional = true }

it results in Cargo complaining:

error: failed to select a version for `defmt`.
    ... required by package `bxcan v0.4.0 (/home/jonas/dev/bxcan)`
versions that meet the requirements `^0.1.0` are: 0.1.3, 0.1.2, 0.1.1, 0.1.0

the package `defmt` links to the native library `defmt`, but it conflicts with a previous package which links to `defmt` as well:
package `defmt v0.2.0`
    ... which is depended on by `bxcan v0.4.0 (/home/jonas/dev/bxcan)`

This is because defmt makes use of the links key in Cargo.toml to prevent linking multiple versions of it into the same executable. It looks like Cargo handles this very poorly and demands that even optional dependencies don't use the same links directive (we should probably check if this is a Cargo bug).

[Lotterleben]

Note that this has the side effect that, given this dependency tree:

        defmt
         |  |
 libuseful  |
         |  |
        my_crate

e.g. my_crate depending on defmt and libuseful, and libuseful using defmt as well, my_crate is forced to update their defmt dep (and potentially rewrite all of their log statements) if libuseful bumps their defmt version

[jonas-schievink]

Found the upstream issue that's causing this bug: rust-lang/cargo#7880

[Dirbaio]

Have you considered excluding the wire format from the semver guarantees?

Minor bumps like 0.2.2 -> 0.2.3 would be guaranteed to keep API compatibility (so existing code will still compile), but are allowed to do breaking changes to the wire format.

Probe-run could support decoding many wire format versions at once (for example by depending on multiple defmt-decoder versions). Doing a minor defmt upgrade would cause the user to get prompted to upgrade probe-run, but no other breakage should happen.

This would allow defmt to evolve faster without splitting the ecosystem. API breaking changes should be very rare.

Also you guys mentioned that extending the format string is a breaking change. Why is that? Is it due to old decoders now being unable to decode the stream? If so, that'd be solved by this too. Extending the format string would break wire format but not API.

[jonas-schievink]

Also you guys mentioned that extending the format string is a breaking change. Why is that? Is it due to old decoders now being unable to decode the stream?

Yeah, but this was mostly solved in 0.2 by allowing us to at least extend/change the display hint syntax (if the decoder doesn't understand a hint it just ignores it)

[Dirbaio]

Adding a new data type {=foo} (not {:foo}) is still breaking, isn't it? The decoder won't know how to decode the next bytes.

[jonas-schievink]

Yup, that's still breaking. However I'm not sure how versioning the wire format separately would solve this, unless you also version all the ELF metadata separately and duplicate the decoder for each version.

[Dirbaio]

When a new wire format version is released, this'd happen:

• Wire format version number is bumped.
• New minor defmt release, not breaking API. All projects can update without breaking build. No lib in the ecosystem needs to explicitly opt-in to support this new version.
• New major defmt-decoder release, which can decode the new wire format (and not the old one)
• New probe-run release, adding a dependency on the new defmt-decoder major version but also still depending on the old major versions. At runtime probe-run checks the wire format from the ELF symbol, then dispatches to the right defmt-decoder version.

To avoid infinite bloat in probe-run, it could drop support for defmt-decoder versions older than a year, for example.

[jonas-schievink]

Yeah, that does sound reasonable

[japaric]

When a new wire format version is released, this'd happen:

In that case, cargo update (updating the firmware lockfile; w/o changing the code) could make it not possible to cargo run the firmware with (the currently installed) probe-run anymore, right? probe-run would refuse to run the firmware and prompt the user to install a newer probe-run in that case?

[Dirbaio]

In that case, cargo update could make it not possible to cargo run the firmware with (the currently installed) probe-run anymore, right?

Yep. This is the main big downside to this approach.

This is already the case with the "latest stable only" MSRV , isn't it?. If a new stable rust is released, a minor defmt release would be allowed to start requiring it, so a cargo update could break the build if you're not on latest stable rust.

probe-run would refuse to run the firmware and prompt the user to install a newer probe-run in that case?

yep

[Urhengulas]

I just discovered #287 which seems to be related to the discussion

[Urhengulas]

@korken89 said in #625 (comment):

Are there any issues with specifying a range of versions for defmt for libraries? E.g.:

[dependencies.defmt]
version = ">=0.2,<0.4"
optional = true

This seems to work in my experiments in heapless, but I'm not sure if this has unintended consequences.