Skip to content

[READY] perf improvements for strftime #51298

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 154 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
154 commits
Select commit Hold shift + click to select a range
a9ba5de
`period_format` now has a faster default formatter leveraging python …
Feb 6, 2023
a6c06c9
class `_Period`: new method `fast_strftime`
Feb 6, 2023
34113d4
class `Timestamp`: new method `fast_strftime`
Feb 6, 2023
5dd7ab4
New module in tslibs: `strftime.py`. New function in this module: `co…
Feb 6, 2023
8a7c039
`format_array_from_datetime`: new boolean argument `fast_strftime` to…
Feb 6, 2023
f2d2fb1
datetimelike `strftime`: new boolean argument `fast_strftime` to use …
Feb 6, 2023
fac90d7
`DatetimeIndexOpsMixin.format` and `_format_with_header`: new boolean…
Feb 6, 2023
2fc70a6
`NDFrame.to_csv` and `DataFrameRenderer.to_csv` and `CSVFormatter.__i…
Feb 6, 2023
6edda53
Added tests for the `to_csv` dataframe method to cover the new fast_s…
Feb 7, 2023
b4c815d
`TestCategoricalRepr`: added a test for dates without time, with time…
Feb 7, 2023
eaa1dc9
Fixed `test_nat` and `test_api` with the new symbols added
Feb 7, 2023
3254b54
New `test_strftime` module to cover the `strftime.py` module in tslib.
Feb 7, 2023
0f69286
`convert_strftime_format` argument `target` is now mandatory to avoid…
Feb 7, 2023
72fe379
`convert_strftime_format`: Completed unsupported directives for datet…
Feb 7, 2023
eda4243
Fixed bug in tslib `format_array_from_datetime`
Feb 7, 2023
442732f
Fixed issue in `format_array_from_datetime` when tz was not None
Feb 7, 2023
5ae3707
Added 2 todos
Feb 7, 2023
62aca61
`test_format`: Added various tests for the new feature
Feb 7, 2023
86cc1c8
Merge branch 'main' of https://github.com/pandas-dev/pandas into feat…
Feb 7, 2023
6412911
Merge branch 'main' of https://github.com/pandas-dev/pandas into feat…
Feb 8, 2023
2ec16ba
Fixed Datetime64TZFormatter issue due to arg renaming in recent commits
Feb 8, 2023
c325431
Added 2 asv benchs for strftime with iso8601 format, and a variant fo…
Feb 8, 2023
6c1188a
blackened, flake8, and removed asv main
Feb 8, 2023
1fad7b6
Minor improvement
Feb 10, 2023
e33ad65
Added new ASVs for strftime
Feb 10, 2023
6456694
Added asvs for period
Feb 10, 2023
91ad194
Added asvs for datetime and period indexes .format
Feb 14, 2023
80ebc82
`convert_strftime_format` is now part of the public API as it is requ…
Feb 14, 2023
087a949
Merge branch 'main' of https://github.com/pandas-dev/pandas into feat…
Feb 16, 2023
bf170ba
Updated whats new 1.5.4
Feb 16, 2023
1f3db30
Merge branch 'main' of https://github.com/pandas-dev/pandas into feat…
May 9, 2023
4b34639
Fixed issues following the merge from latest main. Introduced new fun…
May 9, 2023
6e189ad
Improved ASV bench slightly (added datetime index formatting tests)
May 9, 2023
9152dbc
Merged whatsnew
May 9, 2023
35a39a0
pre-commit and docstring checks
May 9, 2023
d13895b
blackened
May 10, 2023
35e4c34
Merge branch 'main' of https://github.com/pandas-dev/pandas into feat…
May 10, 2023
4007330
Added two asvs
May 10, 2023
3d8d469
Fixed mypy error
May 10, 2023
1fc3d48
Fixed ASV bench
May 10, 2023
6928941
Improved ASV benchs
May 10, 2023
7c87a2f
Fixed RST format in whatsnew
May 11, 2023
21b7c9d
Hopefully fixed the ASV bench for the case when the format is the def…
May 11, 2023
fdb7309
Fixed issue in period.pyx
May 11, 2023
636f27f
Made hooks happy
May 11, 2023
1773a0d
Merge branch 'main' into feature/44764_perf_issue_new
smarie May 11, 2023
3b96d0b
Fixed variables used before initialization
May 11, 2023
3433c16
pre-commit hook upgrade
May 12, 2023
234685c
Merge branch 'main' of https://github.com/pandas-dev/pandas into feat…
May 12, 2023
227f188
Fixed error in period.pyx
May 12, 2023
9c03fe4
Fixed issue: there was no need to encode to bytes to apply string for…
May 14, 2023
d44a4c9
Fixed issue: there was no need to encode to bytes to apply string for…
May 14, 2023
9a94960
Merge remote-tracking branch 'origin/feature/44764_perf_issue_new' in…
May 14, 2023
f9565ff
Revert "Fixed issue: there was no need to encode to bytes to apply st…
May 14, 2023
3a87c64
Fixed issue: there was no need to encode to bytes to apply string for…
May 14, 2023
bc5cb66
Removed todo in timedeltas.pyx
May 14, 2023
812aa2a
Fixed asv bench
May 14, 2023
69a9b67
Fixed ASV
May 14, 2023
abaf0cb
Fixed indentation issue
May 14, 2023
7c32a44
Merge branch 'main' of https://github.com/pandas-dev/pandas into feat…
May 14, 2023
537b9f7
Doc fix whatsnew entries sorted
May 14, 2023
f7307f4
Improved speed of csv formatting of datetimeindex
May 15, 2023
f9e3b5f
pre-commit fixes
May 15, 2023
9cf6d79
Removed invalid comment
smarie May 16, 2023
f42fe43
Merge branch 'main' of https://github.com/pandas-dev/pandas into feat…
May 31, 2023
9e51d82
Fixed typo
May 31, 2023
c2e150b
Merge branch 'main' of https://github.com/pandas-dev/pandas into feat…
Jun 1, 2023
8339062
Fixed whatsnew
Jun 1, 2023
72df87c
Fixed changelog indentation error: now a one-liner
Jun 5, 2023
0851d55
Merge branch 'main' of https://github.com/pandas-dev/pandas into feat…
Jun 5, 2023
ebb22e8
Fixed mypy error
Jun 5, 2023
a5869c4
Fixed mypy error 2
Jun 5, 2023
14c9cfe
Fixed whatsnew
Jun 5, 2023
e1ed22c
Fixed whatsnew ?
Jun 6, 2023
6a502fc
Fixed whatsnew ?
Jun 6, 2023
d340635
Merge branch 'main' into feature/44764_perf_issue_new
smarie Jun 6, 2023
14b8c37
Merge branch 'main' into feature/44764_perf_issue_new
smarie Jun 13, 2023
cb5bfd9
Merge branch 'main' of https://github.com/pandas-dev/pandas into feat…
Nov 11, 2023
022c185
Moved whatsnew items
Nov 11, 2023
21b0a88
Re applied mods to `TestPeriodIndexFormat`, since it had moved somewh…
Nov 11, 2023
ec8036c
Re applied mods to `TestDatetimeIndexFormat`, since it had moved some…
Nov 11, 2023
85c65a6
Re applied mods from `TestFastStrfTimeScalars` in the right places
Nov 11, 2023
6c35bad
Merge remote-tracking branch 'origin/feature/44764_perf_issue_new' in…
Nov 11, 2023
bac97f9
Ruff: Fixed invalid character in comment
Nov 16, 2023
324052b
black+isort+fixed tests
Nov 16, 2023
20e38ef
Implemented conservative fallback as suggested per code review
Nov 16, 2023
9602eb5
black
Nov 16, 2023
9223451
Implemented hypothesis tests as suggested per code review
Nov 16, 2023
5ffc88d
Merge branch 'main' of https://github.com/pandas-dev/pandas into feat…
Nov 16, 2023
fb0e30c
Removed the `fast_strftime` argument everywhere as suggested per code…
Nov 16, 2023
4ef8ec1
Fixed docstring
Nov 18, 2023
3c50a76
Trying to have meson understand that there is a py file in the tslib
Nov 18, 2023
3e4cd1e
Merge branch 'main' of https://github.com/pandas-dev/pandas into feat…
Nov 18, 2023
e41df33
Fixed ruff error
Nov 18, 2023
6e0e092
Fixed tests
Nov 18, 2023
83d5539
Fixed test
Nov 18, 2023
cd4d4f0
Fixed test
Nov 18, 2023
4198ea6
Fixed test
Nov 18, 2023
8e2ef29
Fixed test for musl linux
Nov 18, 2023
d0c3845
Fixed mypy errors
Nov 18, 2023
a3fcd6c
Fixed mypy error
Nov 18, 2023
0390344
Fixed mypy error
Nov 19, 2023
f43e511
Merge branch 'main' of https://github.com/pandas-dev/pandas into feat…
Nov 19, 2023
18fa4d5
Merge branch 'main' into feature/44764_perf_issue_new
smarie Dec 15, 2023
4521a1e
Merge branch 'main' into feature/44764_perf_issue_new
smarie Dec 20, 2023
cc1a4d2
Merge branch 'main' into feature/44764_perf_issue_new
smarie Dec 31, 2023
9cff856
Merge branch 'main' into feature/44764_perf_issue_new
smarie Jan 9, 2024
42e87c6
Removed `convert_strftime_format` from top-level API and moved it to …
Jan 13, 2024
404ab84
Merge remote-tracking branch 'origin/feature/44764_perf_issue_new' in…
Jan 13, 2024
1726096
Merge branch 'main' into feature/44764_perf_issue_new
smarie Jan 13, 2024
3dd707c
isort
Jan 13, 2024
ea28669
Merge remote-tracking branch 'origin/feature/44764_perf_issue_new' in…
Jan 13, 2024
a94aec0
Merge branch 'main' into feature/44764_perf_issue_new
smarie Jan 13, 2024
aa4b0f6
Merge branch 'main' of https://github.com/pandas-dev/pandas into feat…
Mar 3, 2024
a382325
As per code review: instance level `fast_strftime` are now private `_…
Mar 3, 2024
d2ef85e
Added maintenance comment
Mar 3, 2024
b0471a7
black+isort
Mar 3, 2024
bbbf721
Moved changelog to 3.0.0 and improved it slightly
Mar 3, 2024
5ea765c
Merge branch 'feature/44764_perf_issue_new' of https://github.com/sma…
Mar 3, 2024
691b127
Fixed test: fast_strftime not part of api anymore (private)
Mar 3, 2024
c41e8c9
Merge branch 'main' of https://github.com/pandas-dev/pandas into feat…
Mar 8, 2024
e0ddfcb
formatting
Mar 8, 2024
b09c641
Fixed ASV benchmark: format method disappeared from DateTimeIndex
Mar 8, 2024
a519427
Improved what's new
Mar 8, 2024
c58d5eb
Merge branch 'main' of https://github.com/pandas-dev/pandas into feat…
Mar 8, 2024
098e53e
Fixed ASV error
Mar 8, 2024
9ad010c
Merge branch 'main' into feature/44764_perf_issue_new
smarie Mar 25, 2024
e719599
Code review: renamed `loc_s` into `locale_dt_strings`
Apr 2, 2024
ffc661d
Code review: moved whats new entry to perf
Apr 2, 2024
b6442d3
Code review: removed commented out code
Apr 2, 2024
40a5c48
Code review: added support for negative and small years and added cor…
Apr 2, 2024
b0f73f5
Merge remote-tracking branch 'origin/feature/44764_perf_issue_new' in…
Apr 2, 2024
084f124
Merge branch 'main' of https://github.com/pandas-dev/pandas into feat…
Apr 2, 2024
b4983f8
Improved test for small years
Apr 2, 2024
159f1bd
Fixed failing test
Apr 3, 2024
76fb52e
Fixed failing test
Apr 3, 2024
c88e52d
Merge branch 'main' into feature/44764_perf_issue_new
smarie Apr 3, 2024
103d9cf
Fixed test on linux
Apr 3, 2024
478ea4e
Trying to debug datetime.strftime on linux
Apr 3, 2024
51431b3
Fixed tests on linux and windows
Apr 3, 2024
847a9f3
Removed useless comment
Apr 3, 2024
338404c
Merge branch 'main' of https://github.com/pandas-dev/pandas into feat…
Apr 3, 2024
a21837c
Fixed tests on linux musl
Apr 4, 2024
fc03926
Attempt to fix on MUSL
Apr 4, 2024
03dfc52
Fixed test for linux musl
Apr 4, 2024
2b98619
Merge branch 'main' of https://github.com/pandas-dev/pandas into feat…
Jun 29, 2024
514e785
Simplified `convert_strftime_format` as per code review
Jun 30, 2024
d130c22
Renamed `fast_strftime` with `strftime_pystr` as per code review
Jun 30, 2024
d6d6905
Merge branch 'main' of https://github.com/pandas-dev/pandas into feat…
Jun 30, 2024
66b5bbe
Fixed test on WASM/Pyodide
Jun 30, 2024
25ad4fb
Renamed internal argument ``strftime_pystr`` into ``_use_pystr_engine…
Jul 1, 2024
5001569
Fixed test on pyodide
Jul 1, 2024
48a0096
Fixed cython-lint error
Jul 1, 2024
3e86c1f
Merge branch 'main' into feature/44764_perf_issue_new
smarie Jul 5, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
convert_strftime_format argument target is now mandatory to avoid…
… mistakes
  • Loading branch information
Sylvain MARIE committed Feb 7, 2023
commit 0f6928662aff162c8a5526575dbfbf37cec054b1
2 changes: 1 addition & 1 deletion pandas/_libs/tslib.pyx
Original file line number Diff line number Diff line change
Expand Up @@ -189,7 +189,7 @@ def format_array_from_datetime(
else:
try:
# Try to get the string formatting template for this format
str_format, loc_s = convert_strftime_format(format)
str_format, loc_s = convert_strftime_format(format, target="datetime")
except UnsupportedStrFmtDirective:
# Unsupported directive: fallback to standard `strftime`
fast_strftime = False
Expand Down
3 changes: 2 additions & 1 deletion pandas/_libs/tslibs/period.pyx
Original file line number Diff line number Diff line change
Expand Up @@ -2360,7 +2360,8 @@ cdef class _Period(PeriodMixin):
def fast_strftime(self, fmt_str: str, loc_s: object) -> str:
"""A faster alternative to `strftime` using string formatting.

`fmt_str` and `loc_s` should be created using `convert_strftime_format(fmt)`.
`fmt_str` and `loc_s` should be created using
`convert_strftime_format(fmt, target="period")`.

See also `self.strftime`, that relies on `period_format`.

Expand Down
2 changes: 1 addition & 1 deletion pandas/_libs/tslibs/strftime.py
Original file line number Diff line number Diff line change
Expand Up @@ -127,7 +127,7 @@ def get_current_locale_specific_string() -> LocaleSpecificDtStrings:

def convert_strftime_format(
strftime_fmt: str,
target: str = "datetime",
target: str,
new_style_fmt: bool = False,
) -> Tuple[str, LocaleSpecificDtStrings]:
"""Convert a strftime formatting string into a formatting template string.
Expand Down
5 changes: 3 additions & 2 deletions pandas/_libs/tslibs/timestamps.pyx
Original file line number Diff line number Diff line change
Expand Up @@ -1435,15 +1435,16 @@ class Timestamp(_Timestamp):
def fast_strftime(self, fmt_str: str, loc_s: object) -> str:
"""A faster alternative to `strftime` using string formatting.

`fmt_str` and `loc_s` should be created using `convert_strftime_format(fmt)`.
`fmt_str` and `loc_s` should be created using
`convert_strftime_format(fmt, target="datetime")`.

See also `self.strftime`, that relies on `datetime.strftime`.

Examples
--------
>>> from pandas._libs.tslibs import convert_strftime_format
>>> ts = pd.Timestamp('2020-03-14T15:32:52.192548651')
>>> fmt, loc_s = convert_strftime_format('%Y-%m-%dT%H:%M:%S')
>>> fmt, loc_s = convert_strftime_format('%Y-%m-%dT%H:%M:%S', target="datetime")
>>> ts.fast_strftime(fmt, loc_s)
'2020-03-14T15:32:52'
"""
Expand Down
3 changes: 2 additions & 1 deletion pandas/io/formats/format.py
Original file line number Diff line number Diff line change
Expand Up @@ -1822,7 +1822,8 @@ def get_format_datetime64(
if date_format is not None and fast_strftime:
try:
# Try to get the string formatting template for this format
str_date_fmt, loc_s = convert_strftime_format(date_format)
str_date_fmt, loc_s = convert_strftime_format(date_format,
target="datetime")
except UnsupportedStrFmtDirective:
# Unsupported directive: fallback to standard `strftime`
pass
Expand Down