Skip to content

Merge Preprocessor into DataSet. #91

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 10 commits into from
Oct 1, 2018
Merged

Merge Preprocessor into DataSet. #91

merged 10 commits into from
Oct 1, 2018

Conversation

FengZiYjun
Copy link
Contributor

Before

loader = DataSetLoader()
train_data = loader.load()
pre = Preprocessor()
train_set, dev_set = pre.run(train_data, split=0.3)

After

data_set = SeqLabelDataSet()
data_set.load(data_path)
train_set, dev_set = data_set.split(0.3, shuffle=True)

DataSet takes care of data loading, indexing, and transformation into tensors.

How to add a new task ?

  • prepare data sets
  • write a function that loads the data set and return multi-level lists
  • define your own data structure, which is a sub-class of DataSet, where convert method handles how your multi-level lists are transformed into Fields.
  • define your PyTorch model
  • define an Evaluator, to compute metrics
  • run and debug

Model, DataSet, and Evaluator are the only three things you need to program.

And there are a lot of reusable sample codes.

FengZiYjun and others added 7 commits September 23, 2018 16:03
@FengZiYjun
- analyze codes for language model, unable to run yet
cb11a1f 
- add character vocab in preprocessor
- add dataset loader for language model dataset
- other minor adjustments
- preserve only a little example data for language model
@FengZiYjun
Merge remote-tracking branch 'origin/master'
2697efc
@FengZiYjun
Merge Preprocessor and DataSet
0b86d7c
@xuyige
Update to new version of framework
91f3d97
@FengZiYjun
Merge pull request #3 from xuyige/final
1d4e406 
Final
@FengZiYjun
Merge Preprocessor into DataSet.
5be4cb7 
- DataSet's __init__ takes a function as argument, rather than class object
- Preprocessor is about to remove. Don't use anymore.
- Remove cross_validate in trainer, because it is rarely used and wired
- Loader.load is expected to be a static method
- Delete sth. in other_modules.py
- Add more tests
- Delete extra sample data
@FengZiYjun
Merge branch 'master' into master
811629d
@codecov-io
Copy link

codecov-io commented Oct 1, 2018 •
edited
Loading

Codecov Report

Merging #91 into master will increase coverage by 6.41%.
The diff coverage is 91.76%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master      #91      +/-   ##
==========================================
+ Coverage   73.05%   79.47%   +6.41%     
==========================================
  Files          67       68       +1     
  Lines        3437     3620     +183     
==========================================
+ Hits         2511     2877     +366     
+ Misses        926      743     -183
Impacted Files Coverage Δ
fastNLP/core/batch.py 97.05% <ø> (+1.93%) ⬆️
fastNLP/modules/other_modules.py 97.02% <ø> (+35.55%) ⬆️
setup.py 0% <ø> (ø) ⬆️
fastNLP/modules/aggregator/self_attention.py 0% <0%> (ø) ⬆️
fastNLP/models/char_language_model.py 0% <0%> (ø) ⬆️
test/model/test_cws.py 96.82% <100%> (+0.15%) ⬆️
test/core/test_tester.py 100% <100%> (ø) ⬆️
test/modules/test_other_modules.py 100% <100%> (ø) ⬆️
test/model/test_seq_label.py 100% <100%> (ø) ⬆️
fastNLP/core/predictor.py 94.33% <100%> (-1.05%) ⬇️
... and 34 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 281b567...81790d7. Read the comment docs.

xuyige
xuyige approved these changes Oct 1, 2018
View reviewed changes
Copy link
Member

@xuyige xuyige left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good
the framework has been updated, i think we should merge it as soon as possible

@FengZiYjun
- add progress bar for data set loading
cc15588 
- improve metrics codes
- fix validator bugs in trainer; remove early saving
- run CWS codes
- improve README.md
@FengZiYjun
Merge branch 'master' of https://github.com/FengZiYjun/fastNLP into t…
da065b4 
…rain_models
@FengZiYjun
Copy link
Contributor Author

  • add progress bar for data set loading
  • improve metrics codes
  • fix validator bugs in trainer; remove early saving
  • run CWS codes
  • improve README.md

@FengZiYjun
Copy link
Contributor Author

FengZiYjun commented Oct 1, 2018 •
edited
Loading

Ready to merge now. This is the latest release V0.1.0.

@FengZiYjun FengZiYjun merged commit 8b6d082 into fastnlp:master Oct 1, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@xuyige xuyige xuyige approved these changes

@choosewhatulike choosewhatulike Awaiting requested review from choosewhatulike

@keezen keezen Awaiting requested review from keezen

@lyhuang18 lyhuang18 Awaiting requested review from lyhuang18

@xpqiu xpqiu Awaiting requested review from xpqiu

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@FengZiYjun @codecov-io @xuyige