Data coupling

[higgx2]

Hello,

I'm working with mf4 files and want to do some data cleaning. For this purpose I would like to read the timestamps and sample values of some channel, do the magic and set them back to the respective channel and rewrite the mf4 file. But the internal structure of the mf4 file should kept unchanged.
I thougt the easiest way would be not to copy the arrays but have all changes to the signal immediately also in the file and so I'm struggling with something like MDF(filename, copy_on_get=False) which I supposed to return something like a pointer instead of a copy to the actual data. But I cannot figure out any different behaviour if I use True or False for this keyword.
Could please anybody give me a hint what I am missing to get it right?

thx, Roland

[danielhrisca]

The data extracted from the file using get, select, to_dataframe is always decoupled from the file.

The only way to have your changes preserved is to create a new file and append the altered signals to it.

copy_on_get does not really matter since version 7.0.0 (I should update the documentation)

[higgx2]

Too sad, because the input file is quite large (between 500MB and 3-4GB) and I have to substitue only very few values in the samples of 1 or channels. The overhead to copy the whole file except the one or two modified channels and then to append the respective channels seems way too high to me for this task. That's why I prefered to get just something like a pointer to the channel data, modify the content and then rewrite the file.
But if the needed method is deprecated, I guess I have to take the road that you suggested.

Btw. the option to select certain channels at reading time like MDF(filename, channels=["channel_name"]) doesnt seem to work correctly in my case or I am doing it not the right way? Meaning that the signal is selected and the only channel inside the mdf object, but I have all the time channels from all channel groups of the original input file.

Thx, Roland

[higgx2]

I just tried something like this to follow you suggestion:

in_mdf = MDF(input_file)

cdb = copy.deepcopy(in_mdf.channels_db)
cdb.pop('time')
cdb.pop(some_signal_name)      **# this is the channel to omit**

 new_mdf = in_mdf.select(cdb)
 new.mdf.append(some_modified_signal)  **# this is the modified channel to replace the omitted one**

But now it complains, that there are more than one channel with the same name during the select statement. That's why I wanted to preserve the original mdf structure as explained before.

[danielhrisca]

The way the samples are stored make this very complicated, especially if the data blocks are compressed.

[higgx2]

Ok, I see your point.

But is it actually possible to rebuild a new mf4 object from scratch with the same internal structure as a mf4 file loaded from file system?
or is there any way to preserve the same channel groups and transfer the signals from input file to new mf4 object?

"I hope I explained clear enough what my initial task was and now I would like some hints how to proceed."
ps: sorry for this last sentence. I apologize, I am not a native speaker, but reviewing this line made me think, that this wording was quite unpolite.

Actually I wanted to ask, if my english was good enough to let you know, what I am acutally trying to do and how I could proceed. The way I described above, seems to work, except that I exchanged the select() method with the filter() method. But still it is quite slow, since I have to do an internal copy of the whole file and then rewrite it. I was just thinking if I was missing a better way to do such small modifications.

thx in advance again for your efforts

[danielhrisca]

Don't worry about your English, it is clear enough. I also did not take offense on what you wrote.

I know that writing a new file is not so fast but it is the only way to do it. You can have a look at the convert method that preserves the internal structure

asammdf/asammdf/mdf.py

Line 734 in 1507f10

def convert(self, version: str) -> MDF:

[higgx2]

Hello, thanks for pointing me out, but unfortunately it seems, that I do not really understand what happens here :-(
I was trying to follow your suggestion, but now I am facing a weird behavior (at least to me it is weird)...

` #get one channel
odo = in_mdf.get('Odo_Abs')

# walk through all groups and get all channels
for i, virtual_group in enumerate(in_mdf.virtual_groups):

    for idx, sigs in enumerate(
            in_mdf._yield_selected_signals(virtual_group, version=version)
    ):
        if idx == 0:
            if sigs:
                timestamps = sigs[0].timestamps
                if len(timestamps) > 0:
                    timestamps, changed_timestamps = repair_timestamps(timestamps)
                for j, s in enumerate(sigs):
                    s.timestamps = timestamps
                    if s.name == 'Odo_Abs':
                        samples = repair_channel_samples(sigs[j].samples, timestamps)[0]
                        sigs[j].samples = copy.copy(samples)
                        pass`

Input object to my function is an MDF object named in_mdf.
In line 1 I get some channel using the get method. The length of the timestamps and samples arrays are 267890 values each.
In line 14 I am selecting the exact same channel and the size of s or sigs[j] respectively are only 35394 values each.
How could this behaviour be explained?

thx for your help