Dataset Improvements #278

hoffmansc · 2021-11-23T23:39:14Z

Major improvements:

Changed how prot_attr arguments are handled. Now, when processing a dataset and running metrics, an explicit array (or list of arrays) containing protected attribute values per sample may be passed instead of requiring an index name.
Added MEPS, COMPAS violent datasets (Port micellaneous items to sklearn-compatible API #150)

Small changes:

Added cache=False option to dataset fetching functions to skip caching
Removed unused categories from DataFrame resulting from dropped rows

Signed-off-by: Samuel Hoffman <hoffman.sc@gmail.com>

* minor change to usecols/dropcols usage ([] -> None) * use fetch_openml `as_frame=True` option * binary_race only affects protected attribute unless numeric_only Signed-off-by: Samuel Hoffman <hoffman.sc@gmail.com>

Signed-off-by: Samuel Hoffman <hoffman.sc@gmail.com>

lgtm-com · 2021-11-24T00:34:36Z

This pull request introduces 2 alerts and fixes 3 when merging 37f3345 into 963df2e - view on LGTM.com

new alerts:

1 for Unused local variable
1 for Unused import

fixed alerts:

3 for 'import *' may pollute namespace

Signed-off-by: Samuel Hoffman <hoffman.sc@gmail.com>

lgtm-com · 2021-11-24T02:20:38Z

This pull request fixes 3 alerts when merging 78a5c3d into 963df2e - view on LGTM.com

fixed alerts:

3 for 'import *' may pollute namespace

nrkarthikeyan · 2021-11-24T15:50:29Z

@monindersingh - FYI, this may address some of the questions you had.

Signed-off-by: Samuel Hoffman <hoffman.sc@gmail.com>

lgtm-com · 2021-12-03T03:59:58Z

This pull request fixes 3 alerts when merging da6d549 into 963df2e - view on LGTM.com

fixed alerts:

3 for 'import *' may pollute namespace

Signed-off-by: Samuel Hoffman <hoffman.sc@gmail.com>

lgtm-com · 2021-12-03T17:43:02Z

This pull request fixes 3 alerts when merging cf0c6c3 into 963df2e - view on LGTM.com

fixed alerts:

3 for 'import *' may pollute namespace

Signed-off-by: Samuel Hoffman <hoffman.sc@gmail.com>

hoffmansc · 2022-01-05T23:21:04Z

@monindersingh I have a question about MEPS. In the pre-processing, you included 'EDUCYR' and 'HIDEG':

AIF360/aif360/datasets/meps_dataset_panel19_fy2015.py

Line 45 in 963df2e

    
           df = df[(df[['FTSTU','ACTDTY','HONRDC','RTHLTH','MNHLTH','HIBPDX','CHDDX','ANGIDX','EDUCYR','HIDEG',

but in the features_to_keep arg of __init__ they're not there:

AIF360/aif360/datasets/meps_dataset_panel19_fy2015.py

Lines 80 to 85 in 963df2e

    
           features_to_keep=['REGION','AGE','SEX','RACE','MARRY', 
        
                           'FTSTU','ACTDTY','HONRDC','RTHLTH','MNHLTH','HIBPDX','CHDDX','ANGIDX', 
        
                           'MIDX','OHRTDX','STRKDX','EMPHDX','CHBRON','CHOLDX','CANCERDX','DIABDX', 
        
                           'JTPAIN','ARTHDX','ARTHTYPE','ASTHDX','ADHDADDX','PREGNT','WLKLIM', 
        
                           'ACTLIM','SOCLIM','COGLIM','DFHEAR42','DFSEE42','ADSMOK42','PCS42', 
        
                           'MCS42','K6SUM42','PHQ242','EMPST','POVCAT','INSCOV','UTILIZATION','PERWT15F'],

Essentially this means those columns are included when we drop NAs but not in the final dataset. This only affects a few rows but what's the reasoning for this? Was it intentional?

lgtm-com · 2022-01-06T00:05:35Z

This pull request fixes 3 alerts when merging 6a7bbef into 963df2e - view on LGTM.com

fixed alerts:

3 for 'import *' may pollute namespace

Signed-off-by: Samuel Hoffman <hoffman.sc@gmail.com>

lgtm-com · 2022-01-14T00:51:48Z

This pull request fixes 3 alerts when merging 88d2e1c into 963df2e - view on LGTM.com

fixed alerts:

3 for 'import *' may pollute namespace

Signed-off-by: Samuel Hoffman <hoffman.sc@gmail.com>

lgtm-com · 2022-07-20T16:11:45Z

This pull request fixes 3 alerts when merging 9d5a8dd into faa75ee - view on LGTM.com

fixed alerts:

3 for 'import *' may pollute namespace

Signed-off-by: Samuel Hoffman <hoffman.sc@gmail.com>

lgtm-com · 2022-07-21T02:05:54Z

This pull request fixes 3 alerts when merging 2e93e9c into faa75ee - view on LGTM.com

fixed alerts:

3 for 'import *' may pollute namespace

Signed-off-by: Samuel Hoffman <hoffman.sc@gmail.com>

lgtm-com · 2022-07-27T21:46:54Z

This pull request fixes 3 alerts when merging 28986da into faa75ee - view on LGTM.com

fixed alerts:

3 for 'import *' may pollute namespace

Signed-off-by: Samuel Hoffman <hoffman.sc@gmail.com>

lgtm-com · 2022-07-28T13:33:57Z

This pull request fixes 3 alerts when merging 230a93b into faa75ee - view on LGTM.com

fixed alerts:

3 for 'import *' may pollute namespace

* allow explicit arrays for prot_attr, target * add MEPS and violent recidivism datasets * option to skip cache * binary_race only affects protected attribute unless numeric_only * remove unused categories after dropping * minimum python version >= 3.7; scikit-learn >= 1.0

hoffmansc added this to the sklearn-compat milestone Nov 23, 2021

hoffmansc added 11 commits November 23, 2021 18:40

allow explicit arrays for prot_attr, target

7423c5e

Signed-off-by: Samuel Hoffman <hoffman.sc@gmail.com>

tweaks to datasets

12b2ee2

* minor change to usecols/dropcols usage ([] -> None) * use fetch_openml `as_frame=True` option * binary_race only affects protected attribute unless numeric_only Signed-off-by: Samuel Hoffman <hoffman.sc@gmail.com>

additional tests

e059683

Signed-off-by: Samuel Hoffman <hoffman.sc@gmail.com>

better categorical column handling

c253f24

Signed-off-by: Samuel Hoffman <hoffman.sc@gmail.com>

allow explicit prot_attr arrays in metrics

d79df84

Signed-off-by: Samuel Hoffman <hoffman.sc@gmail.com>

option to skip cache

6cf67ba

Signed-off-by: Samuel Hoffman <hoffman.sc@gmail.com>

add violent recidivism dataset

fb33edb

Signed-off-by: Samuel Hoffman <hoffman.sc@gmail.com>

default dropna=True

c2e5add

Signed-off-by: Samuel Hoffman <hoffman.sc@gmail.com>

remove unused categories after dropping

c2c6788

Signed-off-by: Samuel Hoffman <hoffman.sc@gmail.com>

initial MEPS dataset

3a47da5

Signed-off-by: Samuel Hoffman <hoffman.sc@gmail.com>

add MEPS to docs

37f3345

Signed-off-by: Samuel Hoffman <hoffman.sc@gmail.com>

hoffmansc force-pushed the datasets branch from 16742d5 to 37f3345 Compare November 23, 2021 23:40

remove unused lines

78a5c3d

Signed-off-by: Samuel Hoffman <hoffman.sc@gmail.com>

hoffmansc added 4 commits December 2, 2021 22:02

fix tests

7102f78

Signed-off-by: Samuel Hoffman <hoffman.sc@gmail.com>

additional tests

708c02a

Signed-off-by: Samuel Hoffman <hoffman.sc@gmail.com>

support get_feature_names_out

591fb1c

Signed-off-by: Samuel Hoffman <hoffman.sc@gmail.com>

fix SettingWithCopyWarning

da6d549

Signed-off-by: Samuel Hoffman <hoffman.sc@gmail.com>

hoffmansc force-pushed the datasets branch from c621d42 to da6d549 Compare December 3, 2021 03:02

hoffmansc added 3 commits December 3, 2021 11:01

fix tests

78fae11

Signed-off-by: Samuel Hoffman <hoffman.sc@gmail.com>

download meps

c8adedb

Signed-off-by: Samuel Hoffman <hoffman.sc@gmail.com>

python version >= 3.7

36a347b

Signed-off-by: Samuel Hoffman <hoffman.sc@gmail.com>

hoffmansc force-pushed the datasets branch from 306ed4b to 36a347b Compare December 3, 2021 16:14

run script from correct dir

cf0c6c3

Signed-off-by: Samuel Hoffman <hoffman.sc@gmail.com>

check german and lawschool match old

6a7bbef

Signed-off-by: Samuel Hoffman <hoffman.sc@gmail.com>

remove numpy 1.19.5 req and upgrade ubuntu to 18.04

88d2e1c

Signed-off-by: Samuel Hoffman <hoffman.sc@gmail.com>

hoffmansc marked this pull request as ready for review February 15, 2022 21:21

hoffmansc added 3 commits July 1, 2022 16:50

remove_unused_categories no longer required

b181571

Signed-off-by: Samuel Hoffman <hoffman.sc@gmail.com>

Merge branch 'master' into datasets

3506698

fix merge issues

9d5a8dd

Signed-off-by: Samuel Hoffman <hoffman.sc@gmail.com>

fix tests

2e93e9c

Signed-off-by: Samuel Hoffman <hoffman.sc@gmail.com>

clarify MEPS education features

28986da

Signed-off-by: Samuel Hoffman <hoffman.sc@gmail.com>

fix test

230a93b

Signed-off-by: Samuel Hoffman <hoffman.sc@gmail.com>

hoffmansc merged commit 3df4fa9 into master Aug 25, 2022

hoffmansc deleted the datasets branch August 25, 2022 20:43

hoffmansc linked an issue Aug 29, 2022 that may be closed by this pull request

assert statement fails in exponentiated gradient reduction (sklearn compatible) notebook #225

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Dataset Improvements #278

Dataset Improvements #278

hoffmansc commented Nov 23, 2021

lgtm-com bot commented Nov 24, 2021

lgtm-com bot commented Nov 24, 2021

nrkarthikeyan commented Nov 24, 2021

lgtm-com bot commented Dec 3, 2021

lgtm-com bot commented Dec 3, 2021

hoffmansc commented Jan 5, 2022

lgtm-com bot commented Jan 6, 2022

lgtm-com bot commented Jan 14, 2022

lgtm-com bot commented Jul 20, 2022

lgtm-com bot commented Jul 21, 2022

lgtm-com bot commented Jul 27, 2022

lgtm-com bot commented Jul 28, 2022

Dataset Improvements #278

Dataset Improvements #278

Conversation

hoffmansc commented Nov 23, 2021

lgtm-com bot commented Nov 24, 2021

lgtm-com bot commented Nov 24, 2021

nrkarthikeyan commented Nov 24, 2021

lgtm-com bot commented Dec 3, 2021

lgtm-com bot commented Dec 3, 2021

hoffmansc commented Jan 5, 2022

lgtm-com bot commented Jan 6, 2022

lgtm-com bot commented Jan 14, 2022

lgtm-com bot commented Jul 20, 2022

lgtm-com bot commented Jul 21, 2022

lgtm-com bot commented Jul 27, 2022

lgtm-com bot commented Jul 28, 2022