Skip to content

PERF: Datetime/Timestamp.normalize for timezone naive datetimes #23634

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 19 commits into from
Nov 18, 2018
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Add performance for UTC timestamps
  • Loading branch information
Matt Roeschke committed Nov 16, 2018
commit bc5571acf61c2e1a8806f44438514f1ceb884b17
2 changes: 1 addition & 1 deletion asv_bench/benchmarks/timestamp.py
Original file line number Diff line number Diff line change
Expand Up @@ -87,7 +87,7 @@ def time_microsecond(self, tz, freq):


class TimestampOps(object):
params = [None, 'US/Eastern']
params = [None, 'US/Eastern', 'UTC']
param_names = ['tz']

def setup(self, tz):
Expand Down
2 changes: 1 addition & 1 deletion doc/source/whatsnew/v0.24.0.rst
Original file line number Diff line number Diff line change
Expand Up @@ -1134,7 +1134,7 @@ Performance Improvements
- Improved performance of :func:`IndexEngine.get_indexer_non_unique` for sorted, non-unique indexes (:issue:`9466`)
- Improved performance of :func:`PeriodIndex.unique` (:issue:`23083`)
- Improved performance of :func:`pd.concat` for `Series` objects (:issue:`23404`)
- Improved performance of :meth:`DatetimeIndex.normalize` and :meth:`Timestamp.normalize` for timezone naive datetimes (:issue:`23634`)
- Improved performance of :meth:`DatetimeIndex.normalize` and :meth:`Timestamp.normalize` for timezone naive or UTC datetimes (:issue:`23634`)


.. _whatsnew_0240.docs:
Expand Down
5 changes: 3 additions & 2 deletions pandas/_libs/tslibs/timestamps.pyx
Original file line number Diff line number Diff line change
Expand Up @@ -1286,9 +1286,10 @@ class Timestamp(_Timestamp):
Normalize Timestamp to midnight, preserving
tz information.
"""
if self.tz is None:
if self.tz is None or is_utc(self.tz):
DAY_NS = DAY_SECONDS * 1000000000
return Timestamp(self.value - (self.value % DAY_NS))
normalized_value = self.value - (self.value % DAY_NS)
return Timestamp(normalized_value).tz_localize(self.tz)
normalized_value = normalize_i8_timestamps(
np.array([self.value], dtype='i8'), tz=self.tz)[0]
return Timestamp(normalized_value).tz_localize(self.tz)
Expand Down
2 changes: 1 addition & 1 deletion pandas/_libs/tslibs/timezones.pxd
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# -*- coding: utf-8 -*-

cdef bint is_utc(object tz)
cpdef bint is_utc(object tz)
cdef bint is_tzlocal(object tz)

cdef bint treat_tz_as_pytz(object tz)
Expand Down
2 changes: 1 addition & 1 deletion pandas/_libs/tslibs/timezones.pyx
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ cdef int64_t NPY_NAT = get_nat()

# ----------------------------------------------------------------------

cdef inline bint is_utc(object tz):
cpdef inline bint is_utc(object tz):
return tz is UTC or isinstance(tz, _dateutil_tzutc)


Expand Down
2 changes: 1 addition & 1 deletion pandas/core/arrays/datetimes.py
Original file line number Diff line number Diff line change
Expand Up @@ -853,7 +853,7 @@ def normalize(self):
'2014-08-01 00:00:00+05:30'],
dtype='datetime64[ns, Asia/Calcutta]', freq=None)
"""
if self.tz is None:
if self.tz is None or timezones.is_utc(self.tz):
not_null = self.notnull()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should this be notna? (does DatetimeArray even have notna or notnull?)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It (DatetimeIndex) apparently has notnull, but not sure if i should be using notna or notnull

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you use notna

DAY_NS = ccalendar.DAY_SECONDS * 1000000000
new_values = self.asi8.copy()
Expand Down