NIO needs to clarify its minor-versioning rules

[helje5]

Introduction

One would think that NEO already states its versioning rules by claiming that the SemVer standard is being followed.

Intentionally leaving out the discussion of what SemVer means by 'backwards compatible' and whether NEO should use the word in the first place. (answer: NO 😬)

The NEO project does not consider 'backwards compatible' to mean that arbitrary valid Swift code build on top of NEO is going to run w/o modifications after a minor version bump.

Because of this I think we need a clarification of what "backwards compatible" means to NEO, so that any software - particularily frameworks wrapping NEO - can write their code so that it is NEO minor version bump resilent.

Goal

The goal is that NEO can reliably be imported using this w/o breaking the compilation of dependent packages:

.package(url: "https://github.com/apple/swift-nio.git", .from("1.5.1"))

instead of

.package(url: "https://github.com/apple/swift-nio.git", .exact("1.5.1"))

And this cross-frameworks: E.g. my NEO based Kitura application may want to import nio-irc or nio-redis (the issue being much more prominent when packages are mixed, but it is not restricted to this).

Issues

There are many issues and pitfalls w.r.t. backwards compatibility in Swift. NEO needs to more precisely document what it will and won't do when bumping minor version numbers, so that code consuming NEO can prepare for it.

Example 1

A nice feature of Swift is being able to extend other types. Unlike categories those are generally safe to use as they are statically dispatched, but they come w/ pitfalls w.r.t. backwards compatibility.

Consider this ByteBuffer extension in swift-nio-irc.

It extends ByteBuffer with two methods, showing just one:

extension ByteBuffer {
  
    public mutating func write<T: SignedInteger>
      (integerAsString integer: T, as: T.Type = T.self) -> Int
    ...
}

While it is wise not to overdo the extension thing, I think everyone can agree that this is a reasonable extension and matches the design of how extensions are supposed to be used in Swift.

This works fine in NEO 1.6.1. However, write(integerAsString:) is generally useful and NEO itself would no doubt want to integrate such a method in NEO itself.

My understanding of the current rules - and this issue is about writing them down - is that NEO reserves the right to add write(integerAsString:) to the main ByteBuffer type w/o a major version bump, e.g in 1.7.1. Like so:

public struct ByteBuffer {
    ...
    public mutating func write<T: SignedInteger>
      (integerAsString integer: T, as: T.Type = T.self) -> Int
    ...
}

If it does it like that, swift-nio-irc and other modules relying on it will fail to compile!

There are two solutions to that:

A) State that NEO doesn't add functions to a public type in minor versions

E.g. instead of doing the above, NEO could also put that into an extension, i.e. the ByteBuffer would be the same, but two extensions with the same method can coexist. That is, swift-nio-irc would indeed continue to compile and operate.

However, there is subtle pitfall to this: If swift-nio-irc would also export the extension (make it public, which it doesn't), a third framework (like Kitura) importing swift-nio-irc and swift-nio might fail to compile, because the selection between the two is now ambiguous.

B) State that NEO reserves the right to add functions to public types in minor versions

This essentially means that swift-nio-irc needs to be rewritten to not use extensions on NEO types. We usually then end up with a beauty like this:

enum IRCNIOHelpers {
    public static func write<T: SignedInteger>
        (integerAsString integer: T, as: T.Type = T.self, 
         to: &ByteBuffer) -> Int
    ...
}

(this also still has potential cross-module naming issues, you just have to love Swift ;-) )

Example 2

Similar to Example 1, but with an entirely different motiviation.

Another valid usage of extension is for forwards compatibility. Let's assume NIO 2.0.0 is going to introduce compactMap on Future replacing flatMap, because you know, that was just wrong and we didn't think about it, etc.

Your current code wants to support both, NIO 1.6 and 2.0, so it just does a

extension Future {
  
    public compactMap(....)  .... { return flatMap(...) }
  
}

And the same issue is hit like in Example 1 when NIO 1.7 has the same clever idea.

ABI

All this becomes particularily important once the Swift ABI hits.

On Linux platforms it must be clear what version to use in soname's, i.e. whether it will be save to replace a libSwiftNIO.1.6.0.so w/ a libSwiftNIO.1.7.0.so - a guarantee usually given by C (or Java) libraries.

(same applies to dylib's of course)

Solution

While it would be nice to have stable minor version upgrades, this is pretty hard in Swift and kinda hard to do.
But for such a low level framework with huge adoption in the server side infrastructure, the minimum should be, that NEO declares its own rules for minor version bumps.

I expect that to be a simple list, which probably won't be quite right from the beginning as we discover more pitfalls.

Something like this:

This we do in major version bumps:

• API may change completely

• Dependend source code needs to be adjusted

This we do in minor version bumps:

• The API is only enhanced

• we do not drop any API

• but we may add new methods or properties to public framework types

• Dependend source code may need to be adjusted if:

  • it uses extensions on types
  • ...

This we do in subminor version bumps:

• The API is fully backwards compatible, compilation will never fail

Closing Notes

I think this actually doesn't really belong here. I think what we need is a "SemVer for Swift" aka SwiftVer, which outlines commonly agreed community rules on what Swift subset to use
cross module.
Not sure whether the ABI effort has that in scope (/CC @jckarter).

[Lukasa]

Mind if I open this in the NIO forums? This issue can track actually adding clarifying documentation, but the wider discussion should probably happen over there: this issue tracker doesn't get enough traffic.

[weissi]

I think a forums thread would be useful as it as @Lukasa says does get more traffic. Regardless, thanks for opening this issue with a useful example @helje5 .

I can take on the effort of documenting our rules. I guess given the Swift pitfalls it needs to include some rules for the consumers of our types such as:

• don't add public conformances of our types to protocols of things we depend on; really you should only conform our types to your own protocols
• only add public extensions to our types with methods that can't clash. Either by using your own types (like say public extension ByteBuffer { func write(myOwnType: MyOwnType) }) or by picking a function name that makes it very unlikely there'll be a clash (public extension ByteBuffer { func myLibraryNameWrite(integer: UInt128) } })

this is not a complete list but two obvious candidates that probably any Swift library that wants to adopt SemVer needs.

[helje5]

Feel free to put it on the forum for discussion (but I won't take part in that).

I considered just creating a SwiftVer project and repository for this. This would essentially just be a README.md, with the rules for Swift framework providers, to which they could just link (instead of SemVer.org). Maybe some examples with common stuff.

But then I wondered, whether the ABI effort produces something wrt this anyways?

[Lukasa]

See: https://forums.swift.org/t/semantic-versioning-in-swift/12720

[helje5]

So, local shadowing of functions actually seems to work just fine for both structs and classes (tested w/ 4.1). One issue less, yay.

[weissi]

this is important, we should do it for 2.0