feat: simple filter transformer nc

[LauJohansson]

Tranformer for making a simple filtering on one column for one value.

[MartinBoge]

We could also have a transformer that accepts a custom filter statement to make it more flexible.

[LauJohansson]

We could also have a transformer that accepts a custom filter statement to make it more flexible.

Good point. I think it is a balance between how much customization is needed as input to a transformer.

Should we have a simple filtertransformer like this + a transformer that can have a filter-string as input?

Or, should they be merged?

[MartinBoge]

We could also have a transformer that accepts a custom filter statement to make it more flexible.

Good point. I think it is a balance between how much customization is needed as input to a transformer.

Should we have a simple filtertransformer like this + a transformer that can have a filter-string as input?

Or, should they be merged?

My 2c is that we should have a simple transformer to accept an argument "query: str" because it gives more flexibility. Maybe it would be less confusing if there is only one.
I.e.:

from typing import List, Union

from pyspark.sql import DataFrame
from spetlr.etl import TransformerNC


class QueryFilterTransformer(TransformerNC):
    def __init__(
        self,
        query: str,
        dataset_input_keys: Union[str, List[str]] = None,
        dataset_output_key: str = None,
    ):
        super().__init__(
            dataset_input_keys=dataset_input_keys,
            dataset_output_key=dataset_output_key,
        )
        self.query = query

    def process(self, df: DataFrame) -> DataFrame:
        print("Filter transformer on sql query")
        return df.filter(self.query)

On the other hand, this might be harder to test.