Skip to content

Speed up tokenizing of a row in csv and xstrtod parsing #25784

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 7 commits into from
Mar 20, 2019
Merged

Speed up tokenizing of a row in csv and xstrtod parsing #25784

Changes from 1 commit
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
3e0cd7f
Try to speed up COLITER_NEXT
vnlitvinov Feb 26, 2019
c157716
speed up parser's functions by register key word
anmyachev Feb 25, 2019
73edece
Speed up comparisons for special symbols during tokenizing csv row
vnlitvinov Mar 19, 2019
b4c2ca8
Speed up xstrtod by first computing integer part of a float number in…
vnlitvinov Mar 19, 2019
41d1747
Added whatsnew entry
vnlitvinov Mar 19, 2019
12a6df9
Fixed whatsnew entry per @jreback comment
vnlitvinov Mar 20, 2019
72c7570
Fine-tuning whatsnew entry for improved readability
vnlitvinov Mar 20, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Next Next commit
Try to speed up COLITER_NEXT
  • Loading branch information
@vnlitvinov
vnlitvinov committed Mar 20, 2019
commit 3e0cd7fad84f4f530ea99483020a419d86341f46
8 changes: 4 additions & 4 deletions pandas/_libs/src/parser/tokenizer.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -214,10 +214,10 @@ typedef struct coliter_t {
void coliter_setup(coliter_t *self, parser_t *parser, int i, int start);
coliter_t *coliter_new(parser_t *self, int i);

#define COLITER_NEXT(iter, word) \
do { \
const int64_t i = *iter.line_start++ + iter.col; \
word = i < *iter.line_start ? iter.words[i] : ""; \
#define COLITER_NEXT(iter, word) \
do { \
const int64_t i = *iter.line_start++ + iter.col; \
word = i >= *iter.line_start ? "" : iter.words[i]; \
} while (0)

parser_t *parser_new(void);
Expand Down