CPU level UDP packet counter

[ykyohei]

Add CPU level UDP packet counter to the HWP encoder.

To identify whether we are dropping the hwp encoder packet at PRU->CPU level or at network level (UDP packet loss), we add the CPU level packet counter. The counter_index is indexed at the PRU level.

This has not been tested yet. We need to use an updated Encoder Agent together.

[ykyohei]

Thanks for the updates, I have one comment.

[ykyohei]

Another comment/question.
It might be fine to have more information, but I'm bit confused by the difference between buffer_reset_counter and packet_counter. They seem almost redundant to me.
Your buffer_reset_counter seems more fundamental, so we can remove the packet_counter?

Then we should have buffer_reset_counter also for the IRIG packet.

[jlashner]

Ya so the main difference between the counters is that the buffer_reset_counter counts when the PRU has finished writing a packet and starts on the next one, while packet_counter increments when the CPU detects a new packet and sends it over the network.

These are related but we still want to track both of them. The failure mode that I am concerned about is when the CPU does not register the packet from the PRU in time to write it to the network before the PRU overwrites it. This will show up as a dropped index in the buffer_reset_counter, while there is no corresponding dropped index in the packet_counter (meaning the packet was never written to the network). Like in this plot below:

I don't think I know enough about how IRIG is handled to know if we need the same kind of book-keeping. Do you know how often the IRIG buffer is swapped / if we're concerned about that dropping?

[ykyohei]

Ah okay thanks.
If I understand correctly, reset_buffer_counter is redundant as the 120 jump in counter, but I agree that it is helpful to have reset_buffer_counter.

I was also less careful about IRIG data. Unfortunately, we have jump in IRIG time as well, and I think the rate is higher than in lab. This might be because of the site timing system. Also, this is less concerning because the interpolation of the dropped IRIG time is much simpler, but I think it is definitely helpful to monitor.

The rate is not negligible. WARNING of deleted wrong irig_time corresponds to this issue.

16774.603: Calculating Angles for obs_1710841702_satp1_1111111 (INFO)
WARNING: sotodlib.core.context: Key "metadata" not present in context.
WARNING: sotodlib.hwp.g3thwp: deleted wrong irig_time, indices: [1199 1842 2410]
WARNING: sotodlib.hwp.g3thwp: Packet drop exists
WARNING: sotodlib.hwp.g3thwp: 6 dropped packets are found, performing fill process
WARNING: sotodlib.hwp.g3thwp: deleted wrong irig_time, indices: [1199]
WARNING: sotodlib.hwp.g3thwp: Packet drop exists
WARNING: sotodlib.hwp.g3thwp: 2 dropped packets are found, performing fill process
17046.032: Calculating Angles for obs_1710848051_satp1_1111111 (INFO)
WARNING: sotodlib.core.context: Key "metadata" not present in context.
WARNING: sotodlib.hwp.g3thwp: Packet drop exists
WARNING: sotodlib.hwp.g3thwp: 1 dropped packets are found, performing fill process
WARNING: sotodlib.hwp.g3thwp: Packet drop exists
WARNING: sotodlib.hwp.g3thwp: 1 dropped packets are found, performing fill process
17159.491: Calculating Angles for obs_1710865890_satp1_1101111 (INFO)
WARNING: sotodlib.core.context: Key "metadata" not present in context.
WARNING: sotodlib.hwp.g3thwp: No HWP data in the specified timestamps.
WARNING: sotodlib.hwp.g3thwp: No HWP data in the specified timestamps.
WARNING: sotodlib.hwp.g3thwp: Offcentering calculation is only available when two encoders are operating. Skipped.
17635.181: Calculating Angles for obs_1710882494_satp1_1101111 (INFO)
WARNING: sotodlib.core.context: Key "metadata" not present in context.
WARNING: sotodlib.hwp.g3thwp: deleted wrong irig_time, indices: [ 643 1239 1826 1831 4022 4027]
WARNING: sotodlib.hwp.g3thwp: cannot find reference points, # of data is less than # of slit
WARNING: sotodlib.hwp.g3thwp: analyzed encoder data is None
WARNING: sotodlib.hwp.g3thwp: deleted wrong irig_time, indices: [ 643 1239 1833 3399]
WARNING: sotodlib.hwp.g3thwp: cannot find reference points, # of data is less than # of slit
WARNING: sotodlib.hwp.g3thwp: analyzed encoder data is None
WARNING: sotodlib.hwp.g3thwp: Offcentering calculation is only available when two encoders are operating. Skipped.
17884.236: Calculating Angles for obs_1710897250_satp1_1111111 (INFO)
WARNING: sotodlib.core.context: Key "metadata" not present in context.
WARNING: sotodlib.hwp.g3thwp: deleted wrong irig_time, indices: [  28 1226]
WARNING: sotodlib.hwp.g3thwp: Packet drop exists
WARNING: sotodlib.hwp.g3thwp: 1 dropped packets are found, performing fill process
WARNING: sotodlib.hwp.g3thwp: deleted wrong irig_time, indices: [624]
WARNING: sotodlib.hwp.g3thwp: Packet drop exists
WARNING: sotodlib.hwp.g3thwp: 1 dropped packets are found, performing fill process

[jlashner]

Yes... kind of. It's redundant with 120-edge frame drops as long as long as samples can only be dropped in complete packets. I think there are also some race conditions in the code which would mean these counters aren't completely redundant, but I believe those are much more unlikely to be triggered.

[bbixler500]

I'm fine merging this code on the beaglebone side. However, I believe the agent code branch needs to be updated as well before we do that. Currently, when I look at the hwp_encoder_packet_count branch on socs the code is not set up currently to handle the version, num_samples, and buffer_reset_counter fields.

[jlashner]

There is currently a PR here that adds that support: simonsobs/socs#647

I think there are possibly some issues remaining, but I was having some trouble emulating irig data packets locally which is why I ended up putting it on hold. I think it would still be good to get this working though.