Retract "Incompatible versions" of this package

[jihoon-seo]

According to https://pkg.go.dev/github.com/mattn/go-sqlite3?tab=versions ,
there are 4 "Incompatible versions" of this package.

because of these,

1. In my project, I go get the latest release of this package
   go get github.com/mattn/go-sqlite3@v1.14.8

2. But when I run some commands like go mod tidy or go mod vendor,
   Go modules automatically updates go.mod like this:
   github.com/mattn/go-sqlite3 v2.0.3+incompatible
   which is what I don't want, because according to the README,

Latest stable version is v1.14 or later not v2.

NOTE: The increase to v2 was an accident. There were no major changes or features.

[mattn]

Unfortunately, no way to remove version on Go module index server. https://index.golang.org/

[rittneje]

The weird thing is, the +incompatible "upgrade" issue was supposed to be fixed in Go 1.14. https://golang.org/doc/go1.14#incompatible-versions Also it is unclear from the description of retract whether you can retract +incompatible versions. Plus it seems odd that go mod tidy and go mod vendor would update your dependencies. It might be a good idea to report this issue to the Go authors to see what they recommend doing.

Curiously, I would have expected issues due to v2.0.4 through v2.0.6, as these have go.mod files and thus are not +incompatible. However, they don't have a v2 folder so are still partially incompatible. That may be why they don't even show up on pkg.go.dev.

[jihoon-seo]

@rittneje

Curiously, I would have expected issues due to v2.0.4 through v2.0.6, as these have go.mod files and thus are not +incompatible. However, they don't have a v2 folder so are still partially incompatible. That may be why they don't even show up on pkg.go.dev.

Yeah, I guess that for v2.0.4 - v2.0.6 to be shown up on pkg.go.dev,
some works will be required: https://blog.golang.org/v2-go-modules

---

@mattn

Unfortunately, no way to remove version on Go module index server. https://index.golang.org/

@rittneje

it is unclear from the description of retract whether you can retract +incompatible versions.

I did some tests on https://play-with-go.dev/retract-module-versions_go116_en/ ,
and it seems that +incompatible versions can be retracted.

---

$ git tag
v1.14.8
v2.0.0
v2.1.0

$ git log
commit bcb92f95c4364d7514ba83942c4a752854891d42 (HEAD -> main, tag: v1.14.8, origin/main)
Author: Random Gopher <gopher@gopher.com>
Date:   Wed Aug 4 06:04:32 2021 +0000

    Fix severe error in Go proverb

commit a39f79d0990a9555f0d628c0a03c3c9e8214c62a (tag: v2.1.0)
Author: Random Gopher <gopher@gopher.com>
Date:   Wed Aug 4 06:01:14 2021 +0000

    Switch Go proverb to something more famous

commit 55c9e7421a031751b8977709be0fdc1c16b905a7 (tag: v2.0.0)
Author: Random Gopher <gopher@gopher.com>
Date:   Wed Aug 4 06:00:01 2021 +0000

    Initial commit

→ In here, you can see that I tagged v2.0.0 first, and then v2.1.0, and then v1.14.8 for the last.

$ cat go.mod
module gopher.live/u9eedb79bfe02/proverb

go 1.16

// Go proverb was totally wrong
retract v2.1.0

→ In here, you can see that I retracted v2.1.0 by writing retract v2.1.0 in go.mod.

$ go list -m -versions gopher.live/u9eedb79bfe02/proverb
gopher.live/u9eedb79bfe02/proverb v1.14.8 v2.0.0+incompatible

$ go list -m -versions -retracted gopher.live/u9eedb79bfe02/proverb
gopher.live/u9eedb79bfe02/proverb v1.14.8 v2.0.0+incompatible v2.1.0+incompatible

→ In here, you can see that v2.1.0+incompatible is retracted.

---

So, my suggestion for the go.mod is:

module github.com/mattn/go-sqlite3

go 1.16

retract (
	v2.0.3
	v2.0.2
	v2.0.1
	v2.0.0
)

[AlekSi]

In addition to retracting a module, it is possible to deprecate it: https://golang.org/ref/mod#go-mod-file-grammar, "Deprecation" sub-header.

I think the ideal solution could be:

1. Release v2.0.7 that both deprecates this major version in favor of v1, and retracts v2.0.0-v2.0.7.
2. Release the next minor or patch release of v1 that retracts v2.0.0-v2.0.7.

[martinrode]

@mattn is there a solution to this problem? I am stuck with this v2.0.3 version because it is referenced by some package deps I am using. But I must upgrade to v1.14.10 which I currently can not.

[rittneje]

@martinrode Does adding a replace directive to your go.mod work?

[martinrode]

@martinrode Does adding a replace directive to your go.mod work?

Yes it does. I am using

// go-sqlite3 must overwrite the version v2.0.3+incompatible which is an old 2
// years old version but considered newer by the go module system.
replace github.com/mattn/go-sqlite3 => github.com/mattn/go-sqlite3 v1.14.10

From #modules in the Gopher Slack, Brian C. Mills added this:

"Given that they appear to be planning to retract the bad version, an exclude seems more appropriate than a replace — in combination with module graph pruning (as of Go 1.17) it helps reduce the impact of the bad version on consumers of your module, if any."

[bcmills]

@AlekSi, v2.0.7 cannot retract v2.0.3+incompatible because it is a different module.

However, v1.14.10 can.

[SVilgelm]

Is there any news on this issue?

[mattn]

Now I retracted v2.0.0 - v2.0.6.

[hajimehoshi]

@bcmills

@AlekSi, v2.0.7 cannot retract v2.0.3+incompatible because it is a different module.
However, v1.14.10 can.

Just out of curiosity, I was wondering why v1.14.10 can retract v2.0.3+incompatible. The document says:

To retract a version, a module author should add a retract directive to go.mod, then publish a new version containing that directive. The new version must be higher than other release or pre-release versions;

Is v1.1.4.10 higher than v2.0.3+incompatibile?

Thanks,

[bcmills]

v1.14.10 is not higher than v2.0.3+incompatible, but if the latest v1 has a go.mod file then the version query @latest prefers that over +incompatible major versions. (That part might be missing from the documentation..? 🤔)

[hajimehoshi]

if the latest v1 has a go.mod file then the version query @latest prefers that over +incompatible major versions.

My understanding is

• @latest prefers a higher version including +incompatible, and that's why go-sqlite3 required retract.
• Searching retract ignores +incompatible (https://cs.opensource.google/go/go/+/master:src/cmd/go/internal/modfetch/coderepo.go;l=977-984;drc=8c8429fe4113b399355c11203e60e6b37bc823ba)

Is that correct?

[bcmills]

• @latest prefers a higher version including +incompatible, and that's why go-sqlite3 required retract.

Mostly yes, but if the highest v1 release has a go.mod file, the +incompatible versions are not considered.
(So, for example, github.com/russross/blackfriday v2.0.0+incompatible still exists and is not retracted, but go list -m github.com/russross/blackfriday@latest resolves to v1.6.0 because that's the highest v1 release and it has a go.mod file.)

However, if the higher +incompatible versions appear explicitly in someone's dependencies, they will still be used and they will still be treated as semantically higher. (So it is still useful to retract them to encourage folks to prune out those dependencies.)

• Searching retract ignores +incompatible (https://cs.opensource.google/go/go/+/master:src/cmd/go/internal/modfetch/coderepo.go;l=977-984;drc=8c8429fe4113b399355c11203e60e6b37bc823ba)

Correct.

[martinrode]

I think something is still off here. If I remove

replace github.com/mattn/go-sqlite3 => github.com/mattn/go-sqlite3 v1.14.15

from my go.mod file, my program is built using SQLITE version 3.31.1 (2 years old).

With that line it build with the freshest SQLITE which is 3.39.2.

So, I don't think this problem is fixed and the ticket should be re-opened.

[bcmills]

@martinrode, retracting erroneous releases does not change your own module's dependencies. To do that, first drop the replace directive and then run:

	go get github.com/mattn/go-sqlite3@v1.14.15

[martinrode]

Ok, now I found the source of the problem. I had another replace in my go.mod

replace github.com/ory/fosite => github.com/ory/fosite v0.40.2

That line caused Go to pull SQLITE 3.31.1 even with the steps performed @bcmills suggested.

Removing that replace however causes Go to downgrade fosite again to an undesired version:

➜  ✗ go get github.com/mattn/go-sqlite3@v1.14.15                          
go: downgraded github.com/mattn/go-sqlite3 v2.0.3+incompatible => v1.14.15
go: downgraded github.com/ory/fosite v0.42.2 => v0.35.1
go: downgraded github.com/ory/x v0.0.214 => v0.0.109

That is somehow unexpected. I must admit I really don't understand what is going on and which module has the right of way here. Its is very confusing to me.

Why can Go not use the modules requested by 3rd party modules the way they require it? Why does a 3rd party module change the module which my program is using? I don't get the idea here.

[bcmills]

@martinrode, when you run go get github.com/mattn/go-sqlite3@v1.14.15, it will downgrade other dependencies that require higher versions of that module until no higher version is required.

If you just want to lop out those parts of the dependency graph, you can use an exclude directive:

go mod edit -exclude=github.com/mattn/go-sqlite3@v2.0.3+incompatible

If you exclude all of the versions that have higher SemVer precedence than the version you do want, then go get will be able to select that version without performing other downgrades.