Sort measurements by another channel timestamps

[pablocael]

Is it possible to reorder measurements according to some other channel? The goal is to be able to write in log time and read in publish time.

[ihedvall]

I'm not sure what you want to do but I assume that you want to plot (or similar) the values using the absolute time. This is typical when merging measurements. We need some common naming standard.

• DG block = A measurement and holds the blob of data sample.
• CG block = A group of values/channels with an optional master channel.
• CN block = A channel/value with an optional conversion CC block.
• Channel value is stored value while the engineering value is the scaled value.
• Record is a byte array holding sample values for all channels in a CG block. The bytes are stored in blob of the DG block. This is a sample.

Back to your original question.

You can setup a remote master CG block. This means that you configure one CG block that stores the time channel and the other CG blocks stores the data values. This is a new feature in MDF 4.2 and I don't see any use of it. You still have to order your sample in time order.

You can however setup a default X channel for each CN block. Normally the master channel is what viewer uses on X-axis but you can redirect this to point to another channel in another DG/CG/CN block. This default X channel doesn't need to be ordered. I think this is more what you want.

Assume that your application get samples with absolute time unordered. A sample is equal to a CG block not at channel. You should skip the master channel on the CG block and store the absolute time as data channel together with the other sample data channels. The data channels should now set the default X-axis to the absolute time channel.

To get this working, the current pre-trig cache might cause a problem as it do store the samples with the absolute time but there a need of twist to fix the pre-trig issue. Maybe the pre-trig function is not needed?

I have not tested the default x channel functionality in MDF viewers so we might need to take a rain check that some changes are needed.

[pablocael]

Thanks for the reply.

My uses case is like the following. The application will write events as they are available to be logged, and this is the natural chronological order of logging. But in another channel, I have the "publish" time, which is the timestmaps in which the events actually happened. Since we cannot ingest all events, its often the case that publish time and log time are different.
Its ok to use logtime for many things, but for some operations we need the actual publish time.

So I actually need to have a way of sorting the events or records that are originally in the order of the master record, according to another channel of publish timestamps.

Is there any thing like this possible?

[ihedvall]

Your use cases are very similar to the event log server application (github: ihedvall/eventlog). I'm using a SQLite database file instead of an MDF file. This might be a better solution.

The basic problem with MDF is that it appends the events at the end of the file. The only solution I recommend is to do the above or redesign the cache so it sorts the events before inserting them. This implies that the cache has some time window before storage.

Maybe it's time for a MS Teams meeting?

[pablocael]

Thanks! Well the good news is that we are not yet using this file format for this purpose, we are in evaluation phase with several other formats. So my question is exactly to validate if we should go for this format. We are also evaluating several other requirements. This specific one seems its not supported. Thanks again for all the help.

[ihedvall]

The sorting requirement indicate that you need some sort of indexing. A database file is a natural selection here. Let me know if you need some help.

Best Regards
Ingemar Hedvall

[pablocael]

Thanks for all the replies... It seems something is still not right in my tests.

I have wrote a trivial example:

    dataGroup->Description("the main data group container");

    auto* channel = createChannel(writer.get(), dataGroup, "/topic/one");

    writer->InitMeasurement();
    writer->StartMeasurement(0);
    for(uint64_t i = 0; i < 10; i++) 
    {
      std::string payload = "chan 1:: test " + std::to_string(i);
      std::cout << "Writing sample to channel " << i << std::endl;
      channel->SetChannelValue(payload);
      writer->SaveSample(*channel->ChannelGroup(), i);

    }
    writer->StopMeasurement(10);
    writer->FinalizeMeasurement();

When trying to read it back, all times are 0 and the payloads read back are just two samples, 8 an 9... I am missing something?

This is the channel creation code:

IChannel *createChannel(MdfWriter *writer, IDataGroup *dataGroup, const std::string &channelName) const
  {
    auto *channelGroup = writer->CreateChannelGroup(dataGroup);
    channelGroup->Name(channelName);

    // Create a master channel
    auto *master = channelGroup->CreateChannel();
    master->Name("Time");
    master->Type(ChannelType::Master);
    master->DataType(ChannelDataType::FloatLe);
    master->DataBytes(8);

    auto *channel = channelGroup->CreateChannel();
    channel->Name(channelName);
    // For string payloads
    channel->Type(ChannelType::VariableLength);
    channel->Sync(ChannelSyncType::None);
    channel->DataType(ChannelDataType::StringUTF8);
    return channel;
  }

This is the code for reading the file:

 std::cout << "Scanning metadata" << std::endl;
    size_t result = 0;
    uint64_t startTime;
    MdfReader reader(kFilePath);
    ChannelObserverList observerList;
    // this one will create the observers list, and also get the start time from the reader.
    readFileChannelObservers(reader, observerList, startTime);
    CHECK(observerList.size() > 0);

    for (auto &observer : observerList)
    {
        auto* channel = &observer->Channel();
        std::cout << "Channel name: " << channel->Name() << std::endl;
        std::cout.flush();
        if(channel->Type() != ChannelType::Master)
        {
          continue;
        }
        auto nofSamples = observer->NofSamples();
        for (size_t sample = 0; sample < nofSamples; sample++)
        {
          double channel_value;
          const auto valid = observer->GetChannelValue(sample, channel_value);
          EXPECT_TRUE(valid);
          auto restoredTime = startTime + SECONDS_TO_NANOSECONDS(channel_value);
          std::cout << "Restored time: " << restoredTime << std::endl;
          std::cout << "Raw      time: " << channel_value << std::endl;
    }

[ihedvall]

I have not tested your code but I suspect that the problem is in the StartMeasurement(0) call. The input argument is the start time of the measurement in absolute time (ns since 1970). Unfortunately, I'm using the start time equal to zero as an initial value, indicating that the measurement is not started. The effect in your example is that the measurement is stopped but never started.

I have not tested this but I recommend that you define an uint64_t start_time = 1 (or some valid time) and use this as an offset in the StartMeasurent(start_time), SaveSample(..., start_time + i) and last StopMeasurement(start_time + 10). I think you get 10 samples after that. You can return if it doesn't work.