Add: track per-port seq gaps and add redundant GST coverage

doc/st40_validation_updates.md

@@ -9,6 +9,9 @@ This note captures all recent ST40/ST40p feature changes and the accompanying va
 - **Frame-info log surfacing:** When tests pass `log_frame_info=True`, the harness dumps the frame-info file and a summary directly into the test log (no artifact download needed).
 - **ST40P test mutation knobs:** The GStreamer TX plugin accepts `tx-test-mode` with helpers for `no-marker`, `seq-gap`, `bad-parity`, and `paced` (optionally with `tx-test-pkt-count` and `tx-test-pacing-ns`). These drive targeted negative/edge-path tests.
 - **Interlaced & split-mode coverage:** RX/TX can be flagged interlaced independently; mismatch fails fast. Split-mode plus marker handling is now covered in tests.
+- **Interlace auto-detect option:** RX can infer progressive vs interlaced cadence from RTP F bits (`rx-auto-detect-interlaced=true` in GStreamer, `ST40P_RX_FLAG_AUTO_DETECT_INTERLACED` in C API). Use when cadence is unknown; RX updates `interlaced` after first field detection while still logging `second_field`/`interlaced` in frame-info.
+- **Auto-detect reset on discontinuity:** A sequence gap (`seq_discont>0`) clears the interlace detection state; RX re-learns cadence from subsequent F bits and continues logging `second_field`/`interlaced` per field.
+- **Warnings when cadence is unknown:** Pipeline RX and the GStreamer RX plugin emit a warning if neither `rx-interlaced` nor auto-detect is set, to avoid silent progressive defaults.
 - **Integration safety nets:** New noctx integration tests for ST40 interlaced flows, and expanded GStreamer validation tests across single-host and dual-host (VF/VF) paths.
 
 ## Data path (split ANC per RTP)
@@ -44,6 +47,16 @@ flowchart LR
 - **Negative-path observability:** In interlaced mode, a missing marker or sequence gap can occur on just one field. Frame-info shows the gap on that field only, which is why `test_st40i_split_mode_frame_info_logging` and the noctx `st40i_split_seq_gap_reports_loss` exist—to prove per-field accounting is accurate.
 - **Naming clarity:** The `p` in `st40p` refers to the pipeline API, not progressive-only transport. Interlaced support is a first-class, opt-in property (`tx-interlaced` / `rx-interlaced`) and is exercised in both GStreamer and C integration suites.
 
+## Interlace auto-detect: usage and expected signals
+
+- Enable on RX when cadence is unknown: `rx-auto-detect-interlaced=true` in the GStreamer RX pipeline or set `ST40P_RX_FLAG_AUTO_DETECT_INTERLACED` in `st40p_rx_ops.flags` (C API). Keep `rx-interlaced=false` when auto-detecting.
+- TX still needs to emit F bits (set `tx_interlaced=true` in GStreamer or `interlaced=true` in TX ops) so RX can learn cadence.
+- Warnings: RX logs a warning if both `rx-interlaced=false` and auto-detect are disabled. Expect a GST_WARNING from `mtl_st40p_rx` and a pipeline warning from `st40p` if cadence is unknown.
+- Frame-info fields: `second_field` (bool) and `interlaced` are populated once F bits are observed (bit1 set for interlaced; `second_field=true` when F==0x3). These are visible in both C API frame_info and GStreamer frame-info dumps.
+- Expect detection log line when auto-detect flips interlaced on RX (`detected interlaced stream (F=0x2/0x3)`); downstream tests assert on these fields rather than relying on caps only.
+- Discontinuity handling: a session-level sequence hole (e.g., `tx-test-mode=seq-gap` or real loss) clears detection state; expect `seq_discont>0` in frame-info and a subsequent “detected interlaced stream” log once new F bits arrive. This proves re-learn after reset.
+- Build verification: RX attach logs should show `rx_ancillary_session_attach(... flags 0x4 ... auto)` when `rx-auto-detect-interlaced=true` propagates; if flags stay `0x0`, rebuild the GStreamer plugin and ensure the test uses the fresh `--gst-plugin-path`.
+
 ## API and plugin changes (developer-facing)
 
 - **TX (GStreamer st40p):**
@@ -95,6 +108,8 @@ Across all single-host tests, frame-info is dumped and summarized in pytest logs
   - `st40i_split_multi_packet_roundtrip` — sends multiple ANC blocks per field (sizes 8/6/4) in split mode; expects RX accumulation and nonzero frames.
   - `st40i_split_loopback` — interlaced split-mode loopback with custom fps and framebuff; ensures split/interlace coexist.
   - `st40i_split_seq_gap_reports_loss` — crafts a manual RTP sequence gap and validates `seq_discont`/`seq_lost` are reported via the C API frame_info, mirroring the GStreamer gap test at a lower level.
+- [tests/integration_tests/noctx/testcases/st40p_auto_detect_tests.cpp](tests/integration_tests/noctx/testcases/st40p_auto_detect_tests.cpp)
+  - `st40p_rx_auto_detect_interlace` — TX flags interlaced, RX starts progressive with `ST40P_RX_FLAG_AUTO_DETECT_INTERLACED`; asserts `interlaced`/`second_field` populate on RX frame_info.
 
 These integration tests exercise the C pipeline APIs directly (outside the GStreamer harness) to prove split-mode, parity, marker, and sequence reporting work end-to-end without additional context setup.
 
@@ -107,7 +122,8 @@ These integration tests exercise the C pipeline APIs directly (outside the GStre
    - Missing marker: `tx-test-mode=no-marker`
    - Bad parity: `tx-test-mode=bad-parity`
    - Paced burst: `tx-test-mode=paced tx-test-pkt-count=3 tx-test-pacing-ns=200000`
-4. Inspect log: look for `FrameInfo` lines followed by `FrameInfoSummary` to confirm packet counts and seq accounting.
+4. If cadence is unknown on RX, set `rx-auto-detect-interlaced=true` (GStreamer) or `ST40P_RX_FLAG_AUTO_DETECT_INTERLACED` (C API); leave `rx-interlaced=false` so detection can flip it. Expect a warning if you omit both.
+5. Inspect log: look for `FrameInfo` lines followed by `FrameInfoSummary` to confirm packet counts and seq accounting.
 
 ## Signals to watch
 
@@ -119,7 +135,7 @@ These integration tests exercise the C pipeline APIs directly (outside the GStre
 
 - For split-mode validation, set RX `frame_info_path` (for logging) and a power-of-two `rx_rtp_ring_size`; the plugin rejects non power-of-two values.
 - Frame-info summaries rely on the regular expression in the harness; unexpected formats will fall back to raw dump only.
-- Interlaced mismatch between TX/RX fails the test early (before file compare).
+- Interlaced mismatch between TX/RX fails the test early unless RX uses auto-detect; a warning is emitted when neither interlaced nor auto-detect is set.
 
 ## Quick architecture view (tests + logging)
 
@@ -141,4 +157,8 @@ flowchart TD
 - APIs: [include/st40_api.h](include/st40_api.h), [include/st40_pipeline_api.h](include/st40_pipeline_api.h)
 - Plugin: [ecosystem/gstreamer_plugin/gst_mtl_st40p_tx.c](ecosystem/gstreamer_plugin/gst_mtl_st40p_tx.c), [ecosystem/gstreamer_plugin/gst_mtl_st40p_rx.c](ecosystem/gstreamer_plugin/gst_mtl_st40p_rx.c)
 - Pipeline impl: [lib/src/st2110/pipeline/st40_pipeline_tx.c](lib/src/st2110/pipeline/st40_pipeline_tx.c), [lib/src/st2110/pipeline/st40_pipeline_rx.c](lib/src/st2110/pipeline/st40_pipeline_rx.c)
-- Tests: [tests/validation/tests/single/gstreamer/anc_format/test_anc_format.py](tests/validation/tests/single/gstreamer/anc_format/test_anc_format.py), [tests/validation/tests/dual/gstreamer/anc_format/test_anc_format_dual.py](tests/validation/tests/dual/gstreamer/anc_format/test_anc_format_dual.py), [tests/integration_tests/noctx/testcases/st40i_tests.cpp](tests/integration_tests/noctx/testcases/st40i_tests.cpp)
+- Tests:
+  - [tests/validation/tests/single/gstreamer/anc_format/test_anc_format.py](tests/validation/tests/single/gstreamer/anc_format/test_anc_format.py)
+  - [tests/validation/tests/dual/gstreamer/anc_format/test_anc_format_dual.py](tests/validation/tests/dual/gstreamer/anc_format/test_anc_format_dual.py)
+  - [tests/integration_tests/noctx/testcases/st40i_tests.cpp](tests/integration_tests/noctx/testcases/st40i_tests.cpp)
+  - [tests/integration_tests/noctx/testcases/st40p_auto_detect_tests.cpp](tests/integration_tests/noctx/testcases/st40p_auto_detect_tests.cpp)

doc/validation_framework.md

@@ -124,6 +124,8 @@ The `tests/` directory contains test implementations organized by scenario type:
     - Progressive and interlaced GStreamer loops (RFC8331, fps/frame buffer sweeps)
     - Split-mode packetized ANC with frame-info logging (sequence discontinuity, packet totals, RTP marker) and ring-size validation
     - Pacing sanity via RTP sender helpers and ramdisk-backed media fixtures (configure `ramdisk.media` in `configs/test_config.yaml`)
+    - Redundant ST40p/ST40i GStreamer ANC cases with per-port seq-gap scheduling (real payloads, lifted packet caps) and frame-info checks for seq discontinuity/loss logging
+    - Interlace auto-detect on RX (`rx-auto-detect-interlaced` / `ST40P_RX_FLAG_AUTO_DETECT_INTERLACED`) with warnings if neither interlaced nor auto-detect is set; frame-info now includes `second_field` (bool) and `interlaced` for detected cadence, and detection resets on `seq_discont` before re-learning from subsequent F bits
   - Backend-specific tests (DMA, kernel socket, etc.)
   - Integration tests (FFmpeg, GStreamer)
 

ecosystem/gstreamer_plugin/README.md

@@ -492,6 +492,7 @@ The `mtl_st40p_rx` plugin supports all pad capabilities (the data is not checked
 | timeout             | uint    | Timeout in seconds for blocking frame retrieval             | 0 to 300                    | 60            |
 | frame-info-path     | string  | Optional path to append frame info and sequence stats per frame | N/A                      | NULL          |
 | rx-interlaced       | boolean | Whether the incoming ancillary stream is interlaced         | TRUE/FALSE                  | FALSE         |
+| rx-auto-detect-interlaced | boolean | Enable RTP F-bit based interlace detection when cadence is unknown; leaves `rx-interlaced` optional and will warn if both are false. | TRUE/FALSE | FALSE |
 | output-format       | enum    | Serialization format for received ancillary data            | `raw-udw` / `rfc8331`       | `raw-udw`     |
 
 When `output-format` is set to `rfc8331`, each ancillary packet is serialized with an 8-byte
@@ -502,6 +503,7 @@ its user data words so that downstream elements receive complete RFC8331 payload
 > **Note:** `rtp-ring-size` values are validated at runtime. If the supplied number is not a power of
 > two the element fails to initialize, matching the behavior enforced inside
 > `gst_mtl_st40p_rx_start()`.
+> **Interlace auto-detect:** When cadence is unknown, set `rx-auto-detect-interlaced=true` and leave `rx-interlaced=false`. The plugin warns if both are false. Sequence discontinuities reset detection; frame-info shows `seq_discont` when it happens and `interlaced` re-populates after new F bits arrive.
 
 #### 5.2.2. Example GStreamer Pipeline for Reception
 

ecosystem/gstreamer_plugin/gst_mtl_st40p_rx.c

@@ -128,6 +128,7 @@ enum {
   PROP_ST40P_RX_RTP_RING_SIZE,
   PROP_ST40P_RX_TIMEOUT,
   PROP_ST40P_RX_INTERLACED,
+  PROP_ST40P_RX_AUTO_DETECT_INTERLACED,
   PROP_ST40P_RX_OUTPUT_FORMAT,
   PROP_ST40P_RX_FRAME_INFO_PATH,
   PROP_MAX
@@ -346,6 +347,13 @@ static void gst_mtl_st40p_rx_class_init(Gst_Mtl_St40p_RxClass* klass) {
                            "Set to true if ancillary stream is interlaced", FALSE,
                            G_PARAM_READWRITE | G_PARAM_STATIC_STRINGS));
 
+  g_object_class_install_property(
+      gobject_class, PROP_ST40P_RX_AUTO_DETECT_INTERLACED,
+      g_param_spec_boolean(
+          "rx-auto-detect-interlaced", "Auto detect interlaced cadence",
+          "Enable RTP F-bit based interlace auto-detection when cadence is unknown",
+          FALSE, G_PARAM_READWRITE | G_PARAM_STATIC_STRINGS));
+
   g_object_class_install_property(
       gobject_class, PROP_ST40P_RX_OUTPUT_FORMAT,
       g_param_spec_enum("output-format", "Output Format",
@@ -375,6 +383,7 @@ static void gst_mtl_st40p_rx_init(Gst_Mtl_St40p_Rx* src) {
   src->rtp_ring_size = DEFAULT_RTP_RING_SIZE;
   src->timeout_s = 60;
   src->interlaced = FALSE;
+  src->auto_detect_interlaced = FALSE;
   src->output_format = GST_MTL_ST40P_RX_OUTPUT_FORMAT_RAW_UDW;
   src->frame_info_path = NULL;
   src->frame_info_fp = NULL;
@@ -413,6 +422,9 @@ static void gst_mtl_st40p_rx_set_property(GObject* object, guint prop_id,
     case PROP_ST40P_RX_INTERLACED:
       src->interlaced = g_value_get_boolean(value);
       break;
+    case PROP_ST40P_RX_AUTO_DETECT_INTERLACED:
+      src->auto_detect_interlaced = g_value_get_boolean(value);
+      break;
     case PROP_ST40P_RX_OUTPUT_FORMAT:
       src->output_format = g_value_get_enum(value);
       break;
@@ -453,6 +465,9 @@ static void gst_mtl_st40p_rx_get_property(GObject* object, guint prop_id, GValue
     case PROP_ST40P_RX_INTERLACED:
       g_value_set_boolean(value, src->interlaced);
       break;
+    case PROP_ST40P_RX_AUTO_DETECT_INTERLACED:
+      g_value_set_boolean(value, src->auto_detect_interlaced);
+      break;
     case PROP_ST40P_RX_OUTPUT_FORMAT:
       g_value_set_enum(value, src->output_format);
       break;
@@ -483,6 +498,7 @@ static gboolean gst_mtl_st40p_rx_start(GstBaseSrc* basesrc) {
       src->rx_framebuff_cnt ? src->rx_framebuff_cnt : GST_MTL_DEFAULT_FRAMEBUFF_CNT;
   ops_rx.max_udw_buff_size = src->max_udw_size;
   ops_rx.flags = 0; /* Use non-blocking mode - blocking causes preroll timeout */
+  if (src->auto_detect_interlaced) ops_rx.flags |= ST40P_RX_FLAG_AUTO_DETECT_INTERLACED;
 
   GST_DEBUG_OBJECT(src, "RX START: framebuff_cnt=%d, max_udw_buff_size=%d",
                    ops_rx.framebuff_cnt, ops_rx.max_udw_buff_size);
@@ -498,6 +514,11 @@ static gboolean gst_mtl_st40p_rx_start(GstBaseSrc* basesrc) {
   GST_DEBUG_OBJECT(src, "RX START: rtp_ring_size=%d", ops_rx.rtp_ring_size);
 
   ops_rx.interlaced = src->interlaced;
+  if (!src->interlaced && !src->auto_detect_interlaced) {
+    GST_WARNING_OBJECT(src,
+                       "RX interlaced flag not set; use rx-auto-detect-interlaced=true "
+                       "if cadence is unknown");
+  }
 
   /* Optional frame info logging */
   if (src->frame_info_path && !src->frame_info_fp) {

ecosystem/gstreamer_plugin/gst_mtl_st40p_rx.h

@@ -72,6 +72,7 @@ struct _Gst_Mtl_St40p_Rx {
   guint rtp_ring_size;
   guint timeout_s;
   gboolean interlaced;
+  gboolean auto_detect_interlaced;
   gint output_format; /* GstMtlSt40pRxOutputFormat enum value */
   gchar* frame_info_path;
   FILE* frame_info_fp;

ecosystem/gstreamer_plugin/gst_mtl_st40p_tx.c

@@ -385,7 +385,7 @@ static void gst_mtl_st40p_tx_class_init(Gst_Mtl_St40p_TxClass* klass) {
       g_param_spec_uint("tx-test-pkt-count", "Test packet count",
                         "Number of ANC packets to emit when a test mode is active"
                         " (0 uses a mode-specific default)",
-                        0, ST40_MAX_META, 0, G_PARAM_READWRITE | G_PARAM_STATIC_STRINGS));
+                        0, G_MAXUINT, 0, G_PARAM_READWRITE | G_PARAM_STATIC_STRINGS));
 
   g_object_class_install_property(
       gobject_class, PROP_ST40P_TX_TEST_PACING_NS,
@@ -1043,7 +1043,11 @@ static GstFlowReturn gst_mtl_st40p_tx_parse_8331_simple_block(Gst_Mtl_St40p_Tx*
 static GstFlowReturn gst_mtl_st40p_tx_parse_memory_block(Gst_Mtl_St40p_Tx* sink,
                                                          GstMapInfo map_info,
                                                          GstBuffer* buf) {
-  if (sink->test_mode != GST_MTL_ST40P_TX_TEST_MODE_NONE) {
+  /* For seq-gap we still want to mutate RTP sequence numbers but keep the real payload
+   * to preserve frame size for validation. Other test modes keep the synthetic frame
+   * behavior. */
+  if (sink->test_mode != GST_MTL_ST40P_TX_TEST_MODE_NONE &&
+      sink->test_mode != GST_MTL_ST40P_TX_TEST_MODE_SEQ_GAP) {
     return gst_mtl_st40p_tx_prepare_test_frame(sink, map_info, buf);
   }
   struct st40_frame_info* frame_info = NULL;

include/st40_api.h

@@ -106,6 +106,12 @@ struct st40_tx_test_config {
  * If enable the rtcp.
  */
 #define ST40_RX_FLAG_ENABLE_RTCP (MTL_BIT32(1))
+/**
+ * Flag bit in flags of struct st40_rx_ops.
+ * If set, lib will auto-detect progressive vs interlaced based on RTP F bits.
+ * The interlaced boolean in st40_rx_ops is ignored once detection completes.
+ */
+#define ST40_RX_FLAG_AUTO_DETECT_INTERLACED (MTL_BIT32(2))
 
 /**
  * Session type of st2110-40(ancillary) streaming
@@ -336,7 +342,7 @@ struct st40_tx_ops {
   enum st_fps fps;
   /** Mandatory. 7 bits payload type define in RFC3550 */
   uint8_t payload_type;
-  /** Mandatory. interlaced or not */
+  /** Mandatory unless ST40_RX_FLAG_AUTO_DETECT_INTERLACED is set. Interlaced or not. */
   bool interlaced;
 
   /** Optional. Synchronization source defined in RFC3550, if zero the session will assign

include/st40_pipeline_api.h

@@ -45,6 +45,11 @@ struct st40_frame_info {
    * 'pkts_total,' which serves as an indicator of signal quality.  */
   uint32_t pkts_recv[MTL_SESSION_PORT_MAX];
 
+  /** Packet loss per session port based on per-port sequence tracking. */
+  uint32_t port_seq_lost[MTL_SESSION_PORT_MAX];
+  /** True when a per-port sequence discontinuity was detected in this frame. */
+  bool port_seq_discont[MTL_SESSION_PORT_MAX];
+
   /** Whether a marker bit was seen on any RTP packet in this frame. */
   bool rtp_marker;
   /** True if a sequence number discontinuity was observed within this frame. */
@@ -55,6 +60,11 @@ struct st40_frame_info {
   /** TAI timestamp measured right after the RTP packet for this frame was received */
   uint64_t receive_timestamp;
 
+  /** True if this frame represents the second interlaced field (F=0b11). */
+  bool second_field;
+  /** True if the frame was flagged as interlaced (F bits indicate field 1/2). */
+  bool interlaced;
+
   /** priv pointer for lib, do not touch this */
   void* priv;
 };
@@ -187,6 +197,12 @@ enum st40p_rx_flag {
    * Force the numa of the created session, both CPU and memory
    */
   ST40P_RX_FLAG_FORCE_NUMA = (MTL_BIT32(2)),
+  /**
+   * If set, lib will auto-detect progressive vs interlaced using RTP F bits. The
+   * st40p_rx_ops.interlaced field becomes optional and will be updated after
+   * detection.
+   */
+  ST40P_RX_FLAG_AUTO_DETECT_INTERLACED = (MTL_BIT32(3)),
   /** Enable the st40p_rx_get_frame block behavior to wait until a frame becomes
    available or timeout(default: 1s, use st40p_rx_set_block_timeout to customize)*/
   ST40P_RX_FLAG_BLOCK_GET = (MTL_BIT32(15)),
@@ -199,7 +215,7 @@ enum st40p_rx_flag {
 struct st40p_rx_ops {
   /** Mandatory. rx port info */
   struct st_rx_port port;
-  /** Mandatory. interlaced or not */
+  /** Mandatory unless ST40P_RX_FLAG_AUTO_DETECT_INTERLACED is set. interlaced or not */
   bool interlaced;
   /** Mandatory. the frame buffer count. */
   uint16_t framebuff_cnt;