util/tracing: add ability to backfill Span

[dt]

If it is determined that something waited for a span of time only after it waited, it could still be helpful to reflect that it waited for that span in the trace. As such, the ability to add a span to the trace which reflect that it started at some prior time is introduced. For now this is restricted to adding already-finished spans, as this is the only use-case identified so far for a backdated span.

Release note: none.
Epic: none.

[cockroach-teamcity]

This change is 

[sumeerbhola]

@sumeerbhola reviewed all commit messages and made 1 comment.
Reviewable status: complete! 0 of 0 LGTMs obtained (waiting on @dhartunian, @dt, and @visheshbardia).

---

pkg/util/tracing/context.go line 86 at r1 (raw file):

		return
	}
	// Create a child span with the custom start time, then immediately finish it.

(drive-by)
is there any requirement that spans should be started on the parent, with monotonic start times, that this could violate by starting one in the past?

[dt]

@dt made 1 comment.
Reviewable status: complete! 0 of 0 LGTMs obtained (waiting on @dhartunian, @sumeerbhola, and @visheshbardia).

---

pkg/util/tracing/context.go line 86 at r1 (raw file):

Previously, sumeerbhola wrote…

(drive-by)
is there any requirement that spans should be started on the parent, with monotonic start times, that this could violate by starting one in the past?

I think no: the comments in the existing code already suggest spans starts can be out of order here: https://github.com/cockroachdb/cockroach/blob/master/pkg/util/tracing/tracingpb/recording.go#L201-L205