Is there a way to overwrite or delete a single variable or step in a .bp4 file?

[sergeylesnik]

Let's say a user wrote data (a variable) to disk using BP4 format with steps 0, 1 and 2. Now, he wants to restart the calculation from step 1 (e.g. to test a different calculation method). Using the BP4 engine in Mode::Write would overwrite all the data and only step 2 would be present. The Mode::Append would recognize that the variable is already present in the .bp4 file and append a new step (step 3).

The problem is that in our case the user expects that the original data from step 2 is overwritten with the results from the new calculation. The final .bp4 should then contain steps 0 and 1 from the first calculation and step 2 from the second one and nothing else.

I assume that manipulating stored data is not a straightforward task. But is there still a way to this with ADIOS2?

[pnorbert]

duplicate of #2775

[pnorbert]

The BP file formats are not designed to be modifiable, so variables cannot be modified. However, that restart capability at a given step is something many users want and should be enabled in the future. Currently however there is no way to do this.

[williamfgc]

@sergeylesnik your best shot is to start "fresh" and copy to a new dataset removing failing trailing steps. But I don't know how feasible that is for your case given that data sizes can be prohibitive.

[sergeylesnik]

Thank you for the responses. I saw that a BP5 format is being developed. Is it planed to include the restart feature therein?

Another rather small question: do I understand correctly that ADIOS will always copy data to its own buffer before writing to the file system (either on PerformPuts() or EndStep() or Close())? Does any method exist which enables to write the data directly without copying?

[pnorbert]

None of BP4 or BP5 supports the restart feature yet, but when we will do it, we will probably add to both. Primarily we will focus on BP5.

The other question is a different topic. Do you have issues with memory consumption? The ultimate performance is provided by BP4 by buffering all data and performing bulk I/O. BP5 with Put(..., adios2::Mode::Deferred) for large variables (>4MB I think) are not copied, but if you have several such variables the write performance will suffer compared to BP4 (multiple I/O requests per each processes vs single I/O request).

Aggregation in BP4 is very memory intensive, BP5 can save you a lot of memory with its new aggregation implementation. This is where you need to look first if there are memory consumption problems, not the buffering strategy.

Technically, if your output variables are temporary arrays created just for writing, you can use the Span feature to preallocate memory in the adios buffer and fill it with the output values directly.

auto var = io.DefineVariable<double>("r64", shape, start, count,adios2::ConstantDims);
engine.BeginStep();
adios2::Variable<double>::Span myOutputVar = engine.Put(var);
...
   fill myOutputVar.data() pointer with output values
...
engine.EndStep()

Span is like a vector, so it has .data(), .begin() and [] operations. But I doubt you need to use it, since BP5 aggregation probably can solve the memory issues, if not then BP5 Put(Deferred) can help even more.

[williamfgc]

This is where you need to look first if there are memory consumption problems, not the buffering strategy.

+1 for @pnorbert comment. Span should only be used if you're memory bound and/or if you have very large non-continuous memory and want to cut the intermediate temporary memory space (by reusing the adios buffer) when making it contiguous for I/O. See how it's used for MFEM+adios2 since it's probably related to the nature of OpenFOAM's mesh solver.

[sergeylesnik]

@pnorbert

Do you have issues with memory consumption?

No, memory consumption is not a problem. I was wondering whether the copying is just waste of resources and may be avoided.

The ultimate performance is provided by BP4 by buffering all data and performing bulk I/O.

I suspected that there is a performance reason behind it. Thanks for clarifying.

Technically, if your output variables are temporary arrays created just for writing, you can use the Span feature

These are not temporary in our case, but good to now about the feature.

@williamfgc

See how it's used for MFEM+adios2

Good to know. We might learn from this.

Thank you both for the detailed explanations. It's a pleasure to open and close issues here.

[pnorbert]

@sergeylesnik BP5 in the master branch now has a parameter AppendAfterSteps. It enables overwriting some steps as you requested. If you have 10 steps in the file,

• value 0 means starting from the beginning, truncating all existing data
• value 1 means appending after the first step, so overwrite 2,3...10
• value 10 means appending after all existing steps
• value >10 means the same, append after all existing steps (gaps in steps are impossible)
• -1 means appending after the last step, i.e. same as 10 or higher
• -2 means removing the last step, i.e. starting from the 10th
• -11 (and <-11) means truncating all existing data

Default value is MAX_INT, so it always appends after the last step. -1 would be another good default value.

We won't add this feature to BP4 because it would require implementing the entire metadata parsing process of the read side in the writing engine.

Can you please play with this new engine and new parameter and let us know your opinion about it?

[sergeylesnik]

@pnorbert Glad to hear! I'll try it out.