Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update names & functionality for handling missing values #362

Closed
npatki opened this issue Jan 26, 2022 · 2 comments
Closed

Update names & functionality for handling missing values #362

npatki opened this issue Jan 26, 2022 · 2 comments
Labels
feature request Request for a new feature
Milestone

Comments

@npatki
Copy link
Contributor

npatki commented Jan 26, 2022

Problem Description

Let's update the usage of missing value handling to make it more user friendly. The update would apply to all transformers that handle missing values. Currently, these are:

  • BooleanTransformer
  • DateTimeTransformer (+ variants)
  • NumericalTransformer (+ variants)
  • NullTransformer

Expected behavior

  1. nan becomes missing_value_replacement, with updated defaults
# (default) do not replace missing values
nt = NumericalTransformer()

# replace with a static number
nt = NumericalTransformer(missing_value_replacement=0.00)

# replace using a method
nt = NumericalTransformer(missing_value_replacement='mode')
  1. null_column becomes model_missing_values that can only accept True/False values; updated functionality & defaults
# (default = False) do not create a new column
nt = NumericalTransformer()
nt = NumericalTransformer(model_missing_values=False)

# True creates a missing column if there is missing data
nt = NumericalTransformer(model_missing_values=True)
nt.fit(data, columns=['column_with_missing_values'])

# otherwise do not create a column and warn the user
nt = NumericalTransformer(model_missing_values=True)
nt.fit(data, columns=['never_null_column'])
Guidance: There are no missing values in column 'never_null_column'. Extra column not created.
@amontanez24
Copy link
Contributor

A couple points:

  1. This issue will probably have to be completed at the same as Update NullTransformer to make it user friendly #372 since all these transformers call the NullTransformer behind the scenes, so they will crash if the parameters for the NullTransformer change and theirs don't
  2. Since each transformer will be using the same parameters to define how the NullTransformer will be used, perhaps it makes sense to move them to the BaseTransformer class. On top of that, we may want to call the NullTransformer from within the BaseTransformer's fit, transform and reverse_transform methods. Something like
def fit(self, data, columns):
    ...
    self.null_Transformer = NullTransformer(missing_value_replacement=self.missing_value_replacement, model_missing_values = self.model_missing_values)
    self._fit(columns_data)
    self.null_transformer.fit(columns_data)
    
def transform(self, data):
    ...
    transformed_data = self._transform(columns_data)
    transformed_data = self.null_transformer.transform(data)

def reverse_transform(self, data):
    ...
    columns_data = self.null_transformer.reverse_transform(columns_data)
    reversed_data = self._reverse_transform(columns_data)

@pvk-developer
Copy link
Member

Closed by #384

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature request Request for a new feature
Projects
None yet
Development

No branches or pull requests

3 participants