Ignored Ordered flag

[jabielecki]

Reproduction:

1. At the root of this repo, on branch master (yesterday, commit 9e9e3e5), create file ginkgo_test.go with:

// Use it with:  ginkgo --seed 42 -v . | less -RSI
package ginkgo_test

import (
	"fmt"
	"testing"
	"time"

	. "github.com/onsi/ginkgo/v2"
	. "github.com/onsi/gomega"
)

func TestBuggy(t *testing.T) {
	RegisterFailHandler(Fail)
	RunSpecs(t, t.Name())
}

var _ = Describe("", func() { // the only non-Ordered Describe
	OrderedRun("a")
	OrderedRun("b")
	OrderedRun("c")
})

// OrderedRun creates a sequence of tests, all are intended to be Ordered.
func OrderedRun(description string) {
	Describe(description, Ordered, func() {
		It(fmt.Sprint(description, "01"), func(ctx SpecContext) {
			time.Sleep(time.Millisecond)
		})

		Sub2(description)
		Sub3(description)

		Describe("", Ordered, func() {
			Sub4(description)
		})
	})
}

func Sub2(description string) {
	It(fmt.Sprint(description, "02"), func(ctx SpecContext) {
		time.Sleep(time.Millisecond)
	})
}

func Sub3(description string) {
	It(fmt.Sprint(description, "03"), func(ctx SpecContext) {
		time.Sleep(time.Millisecond)
	})
}

func Sub4(description string) {
	Describe("", Ordered, func() {
		It(fmt.Sprint(description, "04"), func(ctx SpecContext) {
			time.Sleep(time.Millisecond)
		})
	})

	Describe("", Ordered, func() {
		It(fmt.Sprint(description, "05"), func(ctx SpecContext) {
			time.Sleep(time.Millisecond)
		})
	})
}

2. Run:

go run ./ginkgo --seed 42 -v .

3. This executes "a04" before "a03".

4. I expected that "a03" should be before "a04".

5. This is not merely an output artifact. With some sync.Once or even By() it's easily verified that the code is in fact executed in a non-Ordered manner.

These steps also reproduce on ginkgo v2.8.4. The v2.6.1 seems fine.

I've tried to make it even more minimal, but was unable to figure out how. The weird nesting of functions and containers seems to be relevant here.

[onsi]

hey thanks for reporting this with the clear reproducer. it almost certainly is due to this commit:

89dda20

that was introduced in 2.7.0 to solve the issue of non-deterministic order when specs are generated by iterating over a map. i tried to be careful around not breaking the explicit order of Ordered specs but clearly failed. I'll need to take a deeper look at this... will try to do so this weekend/next week.

[onsi]

alrighty - can you try the latest commit on master with your original code? I've identified the issue (it's quite subtle, but Ginkgo was not sorting specs in Ordered containers correctly when there were multiple Ordered containers). Things should be working now.

One other thing. As written, if one of your tests fails, you'll have to go by the name of the test to figure out which one failed. Ideally it would be possible to associate the outermost Describe(description, Ordered,...) generated by OrderedRun with the line that called OrderedRun. That way Ginkgo's output will point you to the right invocation of OrderedRun.

You can actually do this using GinkgoHelper:

func OrderedRun(description string) {
        GinkgoHelper()
	Describe(description, Ordered, func() {
              ...
	})
}

this actually resolves the bug you've discovered as well! As it now generates a different line number for each outer Ordered Describe.

(The bug is still real, and I should have fixed it now... and will cut a release once I hear back from you - but just wanted to point out that GinkgoHelper() is something you might want to use here anyway.)

[jabielecki]

I've pulled the latest master branch 7fa0b6b, I've done a few complex runs and everything is Ordered perfectly! 👍 LGTM

I didn't use the GinkgoHelper suggestion:

• It does not work that way - all the links still point to the failing It()
• More importantly, this is fine by me. It might be specific to this particular suite, but the strings are descriptive enough so I've never been confused about the location. If the It() fails I very much prefer to be taken directly to the source code of It().

I cannot thank you enough for the fix, sir! For me and for my open-source-ish problems, this is literally the interaction of the year! 🎺 🎺

[onsi]

hey @jabielecki - glad that worked for you. Thanks for the super clear reproduction steps, that certainly made this much much easier to debug and fix. I'll cut a release momentarily.

[onsi]

alrighty 2.9.5 is out with this fix.