Add Nnpe adapter class

[elseml]

Introduces a new adapter class, Nnpe, which implements Noisy Neural Posterior Estimation (NNPE) as described in Ward et al. (2022). NNPE augments training data with additive noise from a spike-and-slab distribution (mixture of Cauchy and Normal noise), aiming to improve robustness for noisy real world data by extending the scope of the training data. While the specific noise distributions seem to be mostly motivated by their usage in the non-amortized RNPE method proposed in the same paper, it proved to be a quick-and-easy improvement for situations where one expects the presence of similarly distributed noise in our recent benchmarks.

I tried to incorporate all best practices from existing adapter classes, but would be thankful for a thorough check of the implemented changes and the clarity of the provided explanations. I used a seed argument instead of passing an rng for easier serialization, but there might be a more elegant way.

[codecov]

Codecov Report

Attention: Patch coverage is 89.65517% with 6 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
bayesflow/adapters/transforms/nnpe.py	88.23%	6 Missing ⚠️

Files with missing lines	Coverage Δ
bayesflow/adapters/adapter.py	85.06% <100.00%> (+0.38%)	⬆️
bayesflow/adapters/transforms/__init__.py	100.00% <100.00%> (ø)
bayesflow/adapters/transforms/nnpe.py	88.23% <88.23%> (ø)

... and 5 files with indirect coverage changes

[stefanradev93]

Thanks for the PR, Lasse! I suggest going for NNPE (all caps for the class name, since it's an abbreviation). Also, can you please add a docstring for the forward() method?

[elseml]

Sure, integrated both suggestions with a brief docstring for the forward() method since most information is already contained in the class docstring.

One more thing we might want to discuss: I followed the Standardize class in setting the stage="inference" default in the forward() method but we could also leave out the default arg since the forward() method is only activated when stage="training" is explicitly passed. What is our current stance here?

[elseml]

Since standardization will be moved into approximators with #486, I added automatic adjustment of the default scales of Ward et al. (2022) (which are designed for standardized data) by multiplication with the standard deviation of the data, which removes the reliance of the NNPE transform on previous data standardization.

[stefanradev93]

Great, thanks Lasse! Looks ready to merge. Last question:

Is the fixed p=0.5 for the binomial always desirable or should this he a hyperparameter?

[vpratz]

This will lead to different scales for each batch, I'm not sure if this is desirable. If we choose to do this, we should state it more explicitly in the docstring.

[stefanradev93]

There is an alternative solution:

• Standardize data with batch_mean and batch_std
• Add unscaled spikes and slabs
• Re-scale back with batch batch_mean and batch_std

[elseml]

Thanks for checking! Yes, this is the drawback of that solution. I still think it is preferable to more complex solutions, since the method is more about adding some noise at all rather than adding a very specific amount of noise (the default scales by Ward et al. also seem quite heuristically chosen to me). If you agree on this, I can add some more info in the docstring.

[vpratz]

Would we want to do the automatic scaling globally or per dimension? I think this would be the main difference in what @stefanradev93 proposed and how it is implemented now, right?

[stefanradev93]

The way Lasse explained to me, the approach explicitly wants that scale(original) < scale(transformed). In that case, I think fluctuations between batches are fine, as the downstream Standardize layer (which will be part of approximators) will take care of that.

[vpratz]

Then the question still is how we want to automatically determine the scale, globally or per dimension? If dimensions don't have equal magnitude, we might accidentally erase some of them completely. On the other hand, some dimensions might have zero variation (e.g. in image datasets like MNIST), so we would have to decide how to deal with those...

[stefanradev93]

Good question. I would scale dimensionwise.

[elseml]

I implemented it globally following the original NNPE implementation, but agree that dimensionwise scaling would be valuable in many situations and will implement it as an option. I think dimensions with zero variation are not problematic since in that case nothings breaks, there will simply be no noise added. Dimensionwise instead of global scaling will increase the variability of the std calculation between batches though.

[vpratz]

Thanks! Only to make sure, please set dimensionwise as the default, and make global scaling the option.

[elseml]

@stefanradev93: Ward et al. (2022) chose p=0.5 in the context of the non-amortized RNPE method and ported this to NNPE, so in the context of NNPE it is simply one of many possible noise-generating mechanisms with the advantage of being tested in at least two papers so far. I would stay with the fixed value here to reduce complexity.

[vpratz]

Please use more specific exceptions here, I think ValueError and TypeError are the relevant ones.

[elseml]

Thanks for the careful review! The changes added quite some complexity, since the expected shape of passed scales now changes depending on global (expects float) or dimensionwise (expects 1D array of length data.shape[-1]) noise application, and both passed and automatically determined scales should be possible. But the transform should be quite flexible now to handle unstandardized and heterogeneous input data.