FEAT-#1067: skiprows support added for read_csv
#2607
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Codecov Report
@@ Coverage Diff @@
## master #2607 +/- ##
===========================================
- Coverage 85.34% 26.13% -59.22%
===========================================
Files 151 145 -6
Lines 15916 14946 -970
===========================================
- Hits 13584 3906 -9678
- Misses 2332 11040 +8708
Continue to review full report at Codecov.
|
|
|
|
amyskov
force-pushed
the
issue-1067
branch
5 times, most recently
from
8bd2638
to
46aff86
Compare
amyskov
force-pushed
the
issue-1067
branch
3 times, most recently
from
ae42db1
to
6407fe5
Compare
|
Performance numbers obtained by According to the above table, performance of |
|
@anmyachev @dchigarev please review this PR. |
anmyachev
reviewed
Feb 4, 2021
modin/pandas/test/utils.py
Outdated
Comment on lines
733
to
753
| if isinstance(result.index, pandas.MultiIndex): | ||
| levels_number = len(result.index.levels) | ||
| new_index_data = [ | ||
| result.index.get_level_values(lev_id).astype(str) | ||
| for lev_id in range(levels_number) | ||
| ] | ||
| new_index = pandas.MultiIndex.from_arrays(new_index_data) | ||
| new_index.names = result.index.names | ||
| result.index = new_index |
There was a problem hiding this comment.
why not just result.index = result.index.astype(str)?
There was a problem hiding this comment.
If we apply astytpe to the Multiindex object directly we will obtain error
TypeError: Setting <class 'pandas.core.indexes.multi.MultiIndex'> dtype to anything other than object is not supported
But actually in this PR Multiindex conversion is not used, so these changes were removed as unrelated.
There was a problem hiding this comment.
So are you going to delete this?
anmyachev
requested changes
Feb 4, 2021
anmyachev
requested changes
Feb 4, 2021
|
@anmyachev @dchigarev do you have any comments on this PR? |
anmyachev
requested changes
Feb 8, 2021
amyskov
force-pushed
the
issue-1067
branch
2 times, most recently
from
a2f73a4
to
2da042f
Compare
|
@anmyachev PR is ready for review. |
anmyachev
requested changes
Feb 9, 2021
|
@dchigarev please review this |
dchigarev
reviewed
Feb 9, 2021
dchigarev
reviewed
Feb 9, 2021
dchigarev
reviewed
Feb 9, 2021
You are right, we can drop |
There was a problem hiding this comment.
@amyskov it makes sense, maybe we should test the Cython addition first?
There are some test failures here as well. Can you take a look?
amyskov
force-pushed
the
issue-1067
branch
2 times, most recently
from
5dcff16
to
0878d06
Compare
|
@devin-petersohn , this PR was reworked in the way it was discussed offline - now array-like or callable As it can be seen |
|
Hi @devin-petersohn please take a look at this PR. |
|
@amyskov you got some conflicts here |
@devin-petersohn this PR was rebased and now ready for review. |
Unrelated change was moved into separated PR. Yes, it was really long way to get it done) |
Signed-off-by: Dmitry Chigarev <dmitry.chigarev@intel.com> FEAT-modin-project#1067: skiprows support for read_csv added Signed-off-by: Dmitry Chigarev <dmitry.chigarev@intel.com> FEAT-modin-project#1067: make 'skiprows' and 'nrows' work together Signed-off-by: Dmitry Chigarev <dmitry.chigarev@intel.com> FEAT-modin-project#1067: range like 'skiprows' optimizations Signed-off-by: Dmitry Chigarev <dmitry.chigarev@intel.com> Signed-off-by: Alexander Myskov <alexander.myskov@intel.com> FEAT-modin-project#1067: rework read_csv part Signed-off-by: Alexander Myskov <alexander.myskov@intel.com> FEAT-modin-project#1067: rework json and fwf dispatchers Signed-off-by: Alexander Myskov <alexander.myskov@intel.com> FEAT-modin-project#1067: add description Signed-off-by: Alexander Myskov <alexander.myskov@intel.com> FEAT-modin-project#1067: add tests Signed-off-by: Alexander Myskov <alexander.myskov@intel.com> FEAT-modin-project#1067: mark xfailed tests Signed-off-by: Alexander Myskov <alexander.myskov@intel.com> FEAT-modin-project#1067: fix skiprows when encoding is passed Signed-off-by: Alexander Myskov <alexander.myskov@intel.com> FEAT-modin-project#1067: handle partitions with empty DataFrames Signed-off-by: Alexander Myskov <alexander.myskov@intel.com> FEAT-modin-project#1067: mark xfailed tests Signed-off-by: Alexander Myskov <alexander.myskov@intel.com> FEAT-modin-project#1067: remove unused code Signed-off-by: Alexander Myskov <alexander.myskov@intel.com> FEAT-modin-project#1067: add asv tests Signed-off-by: Alexander Myskov <alexander.myskov@intel.com> FEAT-modin-project#1067: handle narrow dataframes Signed-off-by: Alexander Myskov <alexander.myskov@intel.com> FEAT-modin-project#1067: fix asv tests Signed-off-by: Alexander Myskov <alexander.myskov@intel.com> FEAT-modin-project#1067: fix Signed-off-by: Alexander Myskov <alexander.myskov@intel.com> FEAT-modin-project#1067: remove code duplication Signed-off-by: Alexander Myskov <alexander.myskov@intel.com> FEAT-modin-project#1067: consolidate some code into functions Signed-off-by: Alexander Myskov <alexander.myskov@intel.com> FEAT-modin-project#1067: correct io test to check pre_reading part Signed-off-by: Alexander Myskov <alexander.myskov@intel.com> FEAT-modin-project#1067: remove unrelated changes Signed-off-by: Alexander Myskov <alexander.myskov@intel.com> FEAT-modin-project#1067: Apply suggestions from code review Co-authored-by: Anatoly Myachev <45976948+anmyachev@users.noreply.github.com> Signed-off-by: Alexander Myskov <alexander.myskov@intel.com> FEAT-modin-project#1067: correct asv tests Signed-off-by: Alexander Myskov <alexander.myskov@intel.com> FEAT-modin-project#1067: addressing review comments Signed-off-by: Alexander Myskov <alexander.myskov@intel.com> FEAT-modin-project#1067: Update modin/engines/base/io/text/text_file_dispatcher.py Co-authored-by: Anatoly Myachev <45976948+anmyachev@users.noreply.github.com> Signed-off-by: Alexander Myskov <alexander.myskov@intel.com> FEAT-modin-project#1067: Update modin/engines/base/io/text/csv_dispatcher.py Co-authored-by: Dmitry Chigarev <62142979+dchigarev@users.noreply.github.com> Signed-off-by: Alexander Myskov <alexander.myskov@intel.com> FEAT-modin-project#1067: rework asv tests Signed-off-by: Alexander Myskov <alexander.myskov@intel.com> FEAT-modin-project#1067: rollback parser changes Signed-off-by: Alexander Myskov <alexander.myskov@intel.com> FEAT-modin-project#1067: Update modin/engines/base/io/text/text_file_dispatcher.py Co-authored-by: Anatoly Myachev <45976948+anmyachev@users.noreply.github.com> Signed-off-by: Alexander Myskov <alexander.myskov@intel.com> FEAT-modin-project#1067: update tests Signed-off-by: Alexander Myskov <alexander.myskov@intel.com> FEAT-modin-project#1067: fix for cloud tests Signed-off-by: Alexander Myskov <alexander.myskov@intel.com> FEAT-modin-project#1067: Update modin/engines/base/io/text/text_file_dispatcher.py Co-authored-by: Dmitry Chigarev <62142979+dchigarev@users.noreply.github.com> Signed-off-by: Alexander Myskov <alexander.myskov@intel.com> FEAT-modin-project#1067: remove asv tests Signed-off-by: Alexander Myskov <alexander.myskov@intel.com> FEAT-modin-project#1067: addressing review comments Signed-off-by: Alexander Myskov <alexander.myskov@intel.com> FEAT-modin-project#1067: addressing review comments Signed-off-by: Alexander Myskov <alexander.myskov@intel.com> FEAT-modin-project#1067: correct getting dummy_df Signed-off-by: Alexander Myskov <alexander.myskov@intel.com> FEAT-modin-project#1067: remove duplicated code Signed-off-by: Alexander Myskov <alexander.myskov@intel.com> FEAT-modin-project#1067: addressing review comments Signed-off-by: Alexander Myskov <alexander.myskov@intel.com> FEAT-modin-project#1067: addressing review comments Signed-off-by: Alexander Myskov <alexander.myskov@intel.com> FEAT-modin-project#1067: handle corner case with callable skiprows Signed-off-by: Alexander Myskov <alexander.myskov@intel.com> FEAT-modin-project#1067: localize the work with rows_considered Signed-off-by: Alexander Myskov <alexander.myskov@intel.com> FEAT-modin-project#1067: remove unnecessary changes Signed-off-by: Alexander Myskov <alexander.myskov@intel.com> FEAT-modin-project#1067: correct docstrings Signed-off-by: Alexander Myskov <alexander.myskov@intel.com> FEAT-modin-project#1067: drop complex `skiprows` after data import Signed-off-by: Alexander Myskov <alexander.myskov@intel.com> FEAT-modin-project#1067: remove fallback to pandas case Signed-off-by: Alexander Myskov <alexander.myskov@intel.com> FEAT-modin-project#1067: optimize rows skipping Signed-off-by: Alexander Myskov <alexander.myskov@intel.com> FEAT-modin-project#1067: rollback unrelated changes, formatting Signed-off-by: Alexander Myskov <alexander.myskov@intel.com> FEAT-modin-project#1067: fix incorrect rebase Signed-off-by: Alexander Myskov <alexander.myskov@intel.com>
Signed-off-by: Alexander Myskov <alexander.myskov@intel.com>
Signed-off-by: Alexander Myskov <alexander.myskov@intel.com>
Signed-off-by: Alexander Myskov <alexander.myskov@intel.com>
|
@devin-petersohn CI is green now. |
devin-petersohn
approved these changes
Aug 31, 2021
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR introduces parallel implementation for
read_csvfunction with array-like and callableskiprowsparameter.skiprowsis handled differently depending on the parameter type:skiprowsis integer - rows will be eliminated from partitioned data in thepartitioned_filefunction and wouldn't be actually read.skiprowsis uniformly distributed array (for example [1, 2, 3]) - in this case rows within header and the first row fromskiprowswill be read in the first partition (if there are any) andskiprowswill be handled as integer.skiprowsis array, but not uniformly distributed, or callable - in this caseskiprowswill be dropped only after full dataset is imported (this is done because handling of such kind ofskiprowsis inefficient inpartitioned_filefunction).Note 1: this PR is continuation of #1932.
What do these changes do?
flake8 modinblack --check modingit commit -s