Particle Patches: Fallback

[ax3l]

Hi @KseniaBastrakova,

just a minor issue discovered by @NeilZaim: "particle patches" are an optional feature in openPMD:

particle_reduction/reduction_main.py

Lines 649 to 651 in 8283f3f

	SCALAR = openpmd_api.Mesh_Record_Component.SCALAR
	num_particles = particle_species.particle_patches["numParticles"][SCALAR].load()
	series_hdf.flush()

Although PIConGPU always writes them, these patches might not exist in other data sets such as the examples here:
https://github.com/openPMD/openPMD-example-datasets/
This currently leads to a segfault when using the scripts with those examples.

Can we please add the following simple fallback logic? If particle species does not declare particle patches, just chunk the particle read by a fixed number of particles, e.g. 1e7 particles at a time.

[KseniaBastrakova]

Thanks for pointing out, I will try to fix this now

[KseniaBastrakova]

Fixed. Tested on openPMD example datasets. Now it seems to work correctly

[ax3l]

Thanks a lot, this is great!
@NeilZaim does this work for you, too? :)

[NeilZaim]

Thanks a lot! It goes further in the code but I still observe a couple of issues.

I'm trying to use the Voronoi algorithm using the following command:
python reduction_main.py -algorithm=voronoi -hdf=../simData_0008000.h5 -hdf_re=../reduced.h5 -momentum_tol=0.1 -momentum_pos=0.1 (these are random values for the tolerances)

The first issue I encounter is that I think the dimensions attribute of the algorithm is never set, which, for the Voronoi algorithm, results in a failure of the check_needs_subdivision function.

Since I'm testing the code with data from a 2D simulation, I guess that I can temporarily fix this by adding algorithm.dimensions = Dimensions(2,3) approximately anywhere in the code right?

Now if I do this and try the above command line, I obtain the following error:

Traceback (most recent call last):
File "reduction_main.py", line 891, in
base_reduction_function(args.hdf, args.hdf_re, "voronoi", parameters)
File "reduction_main.py", line 133, in base_reduction_function
process_iteration_group(algorithm, current_iteration, series_hdf, series_hdf_reduction, reduction_iteration)
File "reduction_main.py", line 160, in process_iteration_group
process_patches_in_group_v2(iteration.particles[name_group], series_hdf,
File "reduction_main.py", line 737, in process_patches_in_group_v2
write_draft_copy(reduced_weight, relative_result,
File "reduction_main.py", line 537, in write_draft_copy
particle_species["weighting_copy"][SCALAR][previos_idx:current_idx] = reduced_weight
TypeError: setitem(): incompatible function arguments. The following argument types are supported:
1. (self: openpmd_api.Record_Component, tuple of index slices: tuple, array with values to assign: array) -> None
2. (self: openpmd_api.Record_Component, slice: slice, array with values to assign: array) -> None
3. (self: openpmd_api.Record_Component, axis index: int_, array with values to assign: array) -> None

Invoked with: <openPMD.Record_Component of dimensionality '1'>, slice(0, 1, None), [9120000000.0]

Do you know where this is coming from? I can run more tests if needed.

Thanks again!

[KseniaBastrakova]

@NeilZaim thanks. i reproduced it
working on a fix

[KseniaBastrakova]

@NeilZaim now it seems to work. I also add some information about algorithms to a command arguments description

[NeilZaim]

Thanks a lot @KseniaBastrakova !

I am now able to produce h5 output that I can read (tested with the Voronoi algorithm) so I'll start toying around and see what I can get :) .

I've encountered a few issues, listed below:

• On the reduced dataset, the mass and charge still have the shape of the old dataset. (If I had 1000 particles before and 10 particles now, reading the mass on the reduced dataset still results in a vector of size 1000, at least using openPMDviewer). This causes openPMDviewer to crash when reading the momentum during the renormalization step. This is however easy to work around by removing the momentum normalization in openPMDviewer and doing it manually afterwards.

• I still encounter some random segmentation faults (core dumped) during some openpmd-api calls. More specifically this can occur during the line reduction_series.flush() at the very end of copy_meshes in reduction_main.py. If I comment out copy_meshes (which is ok for me, I only need particle data in the reduced dataset, not grid data) I can still have a segfault during the call series_hdf_reduction.flush() in line 536 of reduction_main.py (in the function write_draft_copy). These segfaults do not occur everytime so I guess it's harder to debug but I can provide more info if needed. When there is no segfault the code runs until the end.

• I think that we can remove line 895 in reduction_main.py (base_reduction_function(args.hdf, args.hdf_re, "voronoi", parameters)). The base_reduction_function is already called in the function voronoi_algorithm so this makes the code run twice.

Thanks again!

[KseniaBastrakova]

@NeilZaim Thank you very much for the detailed description of the errors. Working on reproduction and fix

[KseniaBastrakova]

@NeilZaim, I fixed the first issue and, probably, the the secound one(I found a possible problem that could cause it), so now it seems work for all algorithms

[sbastrakov]

And I assume the third one was fixed with PR by @NeilZaim ?

[KseniaBastrakova]

And I assume the third one was fixed with PR by @NeilZaim ?

yes, I merged it already

[NeilZaim]

Thanks a lot! I've just given it a try but I get the following error message

Traceback (most recent call last):
File "reduction_main.py", line 746, in
voronoi_algorithm(args.hdf, args.hdf_re, args.momentum_tol, args.position_lol)
File "reduction_main.py", line 708, in voronoi_algorithm
base_reduction_function(hdf_file_name, hdf_file_reduction_name, "voronoi", parameters)
File "reduction_main.py", line 203, in base_reduction_function
process_iteration_group(algorithm, current_iteration, series_hdf, series_hdf_reduction, reduction_iteration)
File "reduction_main.py", line 222, in process_iteration_group
process_patches_in_group_v2(iteration.particles[name_group], series_hdf,
File "reduction_main.py", line 654, in process_patches_in_group_v2
write_draft_copy(relative_data, dict_data_indexes, series_hdf_reduction, particle_species_reduction,
File "reduction_main.py", line 463, in write_draft_copy
write_record_copy(current_record, values, series_hdf_reduction, previos_idx, current_idx)
File "reduction_main.py", line 435, in write_record_copy
current_reduced_data = values_to_write[:, pos_vector_in_reduction_data].astype(current_type)
IndexError: index 1 is out of bounds for axis 1 with size 1

I don't have time to look at it in more details today but if you want I can run more tests/provide more details tomorrow.

[KseniaBastrakova]

@ NeilZaim Do you use 1-d coordinates? I think my problem in improper handling this dataset. (only testing on 2-d and 3-d)
working on a fix