add statsutil, callback manager and streaming media utilities #83

lukseven · 2025-11-26T16:47:25Z

This PR is a combination of multiple, independent additions:

Statistics Utilities

/util/statsutil package with

Statistics[T Number]: a utility for collecting statistics of a given measurement that calculates count, min, max, sum, mean and standard deviation.
Periodic[T Number]: a utility for collecting periodic statistics, including current period, last period and aggregate

Media Protocol and Format Support:

Parsers, packetizers, pacers & monitors for MPEG-TS, RTP and TLV (type, length, value). Extracted and adapted from content-fabric code.

media
├── io
├── mpegts
├── rtp
└── tlv

A new package /media/io for handling media packet I/O sources and sinks, supporting multiple protocols (UDP, RTP, SRT, and file).

Utility Functions and Testing:

Added ExtractQID utility in format/util.go to extract content IDs from various string formats, with comprehensive tests in format/util_test.go.
Improved test reliability in format/structured/copy_test.go by replacing require.NotSame with a custom requireNotSame to handle non-pointer comparisons, addressing breaking changes in the testify library.

Dependency and Module Updates:

Updated Go version to 1.23 and refreshed several dependencies in go.mod, including testify, log-go, errors-go, and added new dependencies for SRT and related protocols.

Other:

Added a new type Rounded in format/duration/rounded.go for duration formatting.

… token string

…ault * callbacks: go1.23 does not allow type aliases with generics util/callback/callbacks.go:16:6: generic type alias requires GOEXPERIMENT=aliastypeparams * ifutil.FirstOrDefault: do not check array elements for "zero" anymore. That check had unexpected consequences, i.e. FirstOrDefault([]bool{false}, true) returned true, whereas the expected result is false.

plus * include latest errors-go, log-go, stretchr/testify

Copilot

Pull request overview

This PR adds comprehensive media streaming utilities and statistics infrastructure to support MPEG-TS, RTP, and TLV protocols. The changes introduce packet handling, pacing, validation, and monitoring capabilities extracted from content-fabric code.

Key changes:

New statistics collection framework (statsutil) for periodic and aggregate metrics
Generic callback registry system for concurrent-safe event notification
Complete media protocol support for MPEG-TS, RTP, and TLV with packetizers, pacers, and validators
Media I/O abstraction layer supporting UDP, RTP, SRT, and file sources/sinks
Go 1.23 upgrade with updated dependencies (testify, log-go, errors-go)

Reviewed changes

Copilot reviewed 56 out of 57 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
util/statsutil/*	Statistics collection utilities for tracking count, min, max, sum, mean with periodic aggregation
util/callback/*	Thread-safe callback manager with async dispatch using channels
util/byteutil/ringbuffer.go	Ring buffer implementation using double-sized buffer to avoid modulo logic
util/ifutil/ifutil.go	Modified FirstOrDefault to return first element regardless of zero value
util/testutil/assets.go	Test asset path resolution with environment variable support
media/rtp/*	RTP packet handling including packetizer, pacer, gap detector, and timestamp unwrapper
media/mpegts/*	MPEG-TS stream tracking, validation, pacing, and monitoring
media/tlv/*	TLV packetizer and decapsulator for type-length-value encoding
media/io/*	Abstraction for packet sources/sinks supporting UDP, SRT, and file protocols
media/noop.go	No-op implementations of media interfaces for testing
media/transformer.go	Transformer chaining utility
format/util.go	QID extraction from hash/ID/token strings
format/duration/rounded.go	Duration type that formats with single decimal precision
format/structured/copy_test.go	Custom requireNotSame helper for testify compatibility
go.mod	Go 1.23 upgrade and dependency updates

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

* some restructuring in media/io * basic unit tests for media/io

elv-gilles

I think the media package should go into another module: common-go is used from modules that do not deal with media.

elv-gilles · 2025-12-10T09:42:11Z

go.mod


 require (
+	github.com/Comcast/gots/v2 v2.2.1
+	github.com/HdrHistogram/hdrhistogram-go v1.2.0


The fabric is currently using a replacement for this library pointing to a fork in qluvio. I believe the fork was made in order to include some fix.

github.com/qluvio/hdrhistogram-go v1.1.1-0.20210518163530-2bc0df1bbb91

Please clarify which one should be used.

elv-gilles · 2025-12-10T11:07:41Z

util/statsutil/stats.go

+	Max      T             `json:"max"`
+	Sum      T             `json:"sum"`
+	Mean     float64       `json:"mean"`
+	m2       float64       // used for mean calc


m2 would be used for variance calculation, it is updated but never used.

elv-gilles · 2025-12-10T11:14:15Z

media/io/io.go

+	default:
+		err = errors.E("createPacketSink", errors.K.Invalid,
+			"sink", sinkUrl,
+			"reason", "unsupported protocol, expecting udp|rtp|srt",


Suggested change

"reason", "unsupported protocol, expecting udp|rtp|srt",

"reason", "unsupported protocol, expecting udp|rtp|srt|file",

elv-gilles · 2025-12-10T11:16:07Z

media/io/srt.go

+	// ensures message boundaries of the sender are preserved
+	srtConfig.MessageAPI = true
+
+	if false {


please, add comment

elv-gilles · 2025-12-10T11:20:02Z

media/io/srtsink.go

+				// log.Warn("srt connect error", err)
+				wc := io.WriteCloser(&ErrorWriter{err: errors.E("srt connect error", errors.K.Invalid.Default(), err)})
+				dw.writer.Store(&wc)
+				time.Sleep(time.Second)


does this need to be configurable ?

elv-gilles · 2025-12-10T11:48:57Z

media/tlv/tlv_decapsulator.go

+import "github.com/eluv-io/errors-go"
+
+// NewTlvDecapsulator creates a new decapsulator for TLV payloads.
+func NewTlvDecapsulator() *Decapsulator {


This does not seem to be used

lukseven added 9 commits November 12, 2025 14:38

add statsutil

036926d

add callback manager

5f39f81

add Rounded that formats a duration.Spec to a single decimal digit

dbaf48c

add ExtractQID() helper to extract a content ID from an ID/hash/write…

a64ad5c

… token string

new media package for mpeg-ts, rtp & tlv, new byteutil.RingBuffer

5b9f173

plus * include latest errors-go, log-go, stretchr/testify

add media/io package with various source and sink implementations

c38f53d

fix srt config parsing, add multicast for udp source, add debug logs

ec00a88

move rtp/tlv from /media/transport to /media

9aaccfb

lukseven requested review from JanFalkin, Copilot, elv-gilles, elv-serban and jslching November 26, 2025 16:47

Copilot started reviewing on behalf of lukseven November 26, 2025 16:47 View session

Copilot finished reviewing on behalf of lukseven November 26, 2025 16:48

Copilot AI reviewed Nov 26, 2025

View reviewed changes

fix callback manager constructor naming

3c792c7

lukseven force-pushed the luk/media-transport branch from dcae4bf to e0a0e8e Compare November 30, 2025 11:50

add format.ParseQihot & format.ParseQhot

691438f

lukseven force-pushed the luk/media-transport branch from e0a0e8e to 691438f Compare November 30, 2025 11:56

fix srt source/sink when source is listener

e893905

* some restructuring in media/io * basic unit tests for media/io

elv-gilles reviewed Dec 10, 2025

View reviewed changes

lukseven added 2 commits December 13, 2025 03:25

add missed unit test for srt source/sink

5302fc9

add rtp stream tracker, fix mpegts stats logger name

bed49ff

lukseven force-pushed the luk/media-transport branch from b598851 to bed49ff Compare December 19, 2025 13:09

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

add statsutil, callback manager and streaming media utilities #83

add statsutil, callback manager and streaming media utilities #83

Uh oh!

lukseven commented Nov 26, 2025

Uh oh!

Copilot AI left a comment

Uh oh!

elv-gilles left a comment

Uh oh!

elv-gilles Dec 10, 2025

Uh oh!

elv-gilles Dec 10, 2025

Uh oh!

elv-gilles Dec 10, 2025

Uh oh!

elv-gilles Dec 10, 2025

Uh oh!

elv-gilles Dec 10, 2025

Uh oh!

elv-gilles Dec 10, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

	"reason", "unsupported protocol, expecting udp\|rtp\|srt",
	"reason", "unsupported protocol, expecting udp\|rtp\|srt\|file",

add statsutil, callback manager and streaming media utilities #83

Are you sure you want to change the base?

add statsutil, callback manager and streaming media utilities #83

Uh oh!

Conversation

lukseven commented Nov 26, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

elv-gilles left a comment

Choose a reason for hiding this comment

Uh oh!

elv-gilles Dec 10, 2025

Choose a reason for hiding this comment

Uh oh!

elv-gilles Dec 10, 2025

Choose a reason for hiding this comment

Uh oh!

elv-gilles Dec 10, 2025

Choose a reason for hiding this comment

Uh oh!

elv-gilles Dec 10, 2025

Choose a reason for hiding this comment

Uh oh!

elv-gilles Dec 10, 2025

Choose a reason for hiding this comment

Uh oh!

elv-gilles Dec 10, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants