Feature selection on dataset_view instead of forward_index

[siddshuk]

feature_selector requires a forward_index to be passed in to the constructor. But the forward_index is built on the entire dataset, which includes training and test data.

Feature selection should be done on training data separately without touching the test data to avoid optimistically biased performance estimates during cross-validation.

Is it possible to allow feature selection to be done on a dataset_view (view for the training portion of the dataset in this case)?

Thanks.

Best Regards,
Sid

[smassung]

Great question. Unfortunately, getting feature selection integrated completely with indexes/datasets is still on the to do list as issue #111

Right now the feature selection can really only show stats for a particular corpus, e.g. by using./feature-summary (https://github.com/meta-toolkit/meta/blob/master/src/features/tools/feature_summary.cpp)

As always though, if anyone is interested in implementing this, we would gladly accept a pull request 😃

[smassung]

If you need this functionality immediately (and are just using unigram words), a super hacky way to accomplish this would be to:

• separate out your training data and test data as different corpora
• run feature selection on the training data to get a list of the top k words
• learn a classifier on the training data with a list filter on the k words using method=accept (only analyze words from that list of top k)
• save your classifier model that was built on the training corpus
• load your classifier model and run it on the test corpus

I think that should work...

[siddshuk]

Thanks for the quick response. I'll give it a try.

[skystrife]

Fixed by PR #150.