Skip to content

API/ENH: tz_localize handling of nonexistent times: rename keyword + add shift option #22644

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 64 commits into from
Oct 25, 2018
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
64 commits
Select commit Hold shift + click to select a range
bf5e7bf
ENH: Add handling of nonexistent times
Sep 4, 2018
36d13c7
Merge remote-tracking branch 'upstream/master' into normalize_tz
Sep 5, 2018
a5ea445
Merge remote-tracking branch 'upstream/master' into normalize_tz
Sep 7, 2018
8753d00
correct misspelling
Sep 7, 2018
e1a6c6a
Merge remote-tracking branch 'upstream/master' into normalize_tz
Sep 7, 2018
a7c86c8
Merge remote-tracking branch 'upstream/master' into normalize_tz
Sep 8, 2018
1884c7b
Merge remote-tracking branch 'upstream/master' into normalize_tz
Sep 8, 2018
a6a05df
change method of handling nonexistent times
Sep 8, 2018
c4dc8aa
Add another comment
Sep 8, 2018
1bc81db
Add tests for timestamps
Sep 9, 2018
c81d58c
Add tests for datetimeindex
Sep 9, 2018
b2c8429
Add series test and entry in timeseries.rst
Sep 9, 2018
a65987d
Add whatsnew
Sep 9, 2018
710014c
Clean up docstring
Sep 9, 2018
93159e5
Fix nat doc
Sep 9, 2018
a0ffcdd
Merge remote-tracking branch 'upstream/master' into normalize_tz
Sep 11, 2018
219256f
Merge remote-tracking branch 'upstream/master' into normalize_tz
Sep 11, 2018
d435481
Merge remote-tracking branch 'upstream/master' into normalize_tz
Sep 14, 2018
7c849b6
Merge remote-tracking branch 'upstream/master' into normalize_tz
Sep 14, 2018
56ac4fe
Merge remote-tracking branch 'upstream/master' into normalize_tz
Sep 14, 2018
b7b09bd
Merge remote-tracking branch 'upstream/master' into normalize_tz
Sep 19, 2018
94a72a5
add versionadded
Sep 19, 2018
39b769e
Remove whitespace
Sep 19, 2018
18664d8
Merge remote-tracking branch 'upstream/master' into normalize_tz
Sep 24, 2018
8852d43
Depreciate errors and see what needs warning captures
Sep 24, 2018
38b95e9
Correct NaT docstring
Sep 24, 2018
c88b0d8
edit whatsnew and check for raised DeprecationWarning
Sep 24, 2018
1bae682
Address review
Sep 26, 2018
d30f891
change default errors argument to None
Sep 26, 2018
f337692
Map depreciation correctly and test
Sep 26, 2018
6a12a7e
Merge remote-tracking branch 'upstream/master' into normalize_tz
Sep 26, 2018
a7b8357
Try to correctly test for FutureWarning
Sep 26, 2018
7ad87ec
Try adjusting catching FutureWarning
Sep 27, 2018
abad726
Merge remote-tracking branch 'upstream/master' into normalize_tz
Sep 27, 2018
6be1c25
Reorder context managers
Sep 27, 2018
f8be4b6
clear previously seen FutureWarning
Sep 28, 2018
c192c9f
separate test
Sep 28, 2018
8909f38
Merge remote-tracking branch 'upstream/master' into normalize_tz
Sep 28, 2018
49f203f
Merge remote-tracking branch 'upstream/master' into normalize_tz
Sep 30, 2018
01678c7
adjust test
Sep 30, 2018
707fdde
Merge remote-tracking branch 'upstream/master' into normalize_tz
Sep 30, 2018
ae27a50
Remove errors argument to tz_localize_to_utc
Sep 30, 2018
85ed25e
Merge remote-tracking branch 'upstream/master' into normalize_tz
Oct 3, 2018
9041ebe
Add nonexistent assert
Oct 4, 2018
a4cdac2
Handle default None arg
Oct 5, 2018
0a9c1db
Merge remote-tracking branch 'upstream/master' into normalize_tz
Oct 5, 2018
efb382e
Address review
Oct 6, 2018
61c73ca
Catch another warning
Oct 6, 2018
20cc925
Merge remote-tracking branch 'upstream/master' into normalize_tz
Oct 7, 2018
394a0db
Add extra docstring
Oct 7, 2018
a5253ee
Merge remote-tracking branch 'upstream/master' into normalize_tz
Oct 8, 2018
5185683
Edit whatsnew
Oct 8, 2018
ba1bfed
Merge remote-tracking branch 'upstream/master' into normalize_tz
Oct 11, 2018
8b06c96
Address comments
Oct 11, 2018
42ae923
Remove stacklevel
Oct 12, 2018
fe575fe
Add back check_stacklevel
Oct 12, 2018
3482f92
Add blank line for rendering
Oct 17, 2018
f0e43e2
Merge remote-tracking branch 'upstream/master' into normalize_tz
Oct 18, 2018
b98d4cf
Merge remote-tracking branch 'upstream/master' into normalize_tz
Oct 18, 2018
e6c5b2d
Validate nonexistent argument
Oct 18, 2018
83423ad
Merge remote-tracking branch 'upstream/master' into normalize_tz
Oct 19, 2018
1ca0ab2
Fix type
Oct 19, 2018
5bcc977
Merge remote-tracking branch 'upstream/master' into normalize_tz
Oct 24, 2018
8cf16e2
Merge remote-tracking branch 'upstream/master' into normalize_tz
Oct 24, 2018
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Address review
  • Loading branch information
Matt Roeschke committed Sep 26, 2018
commit 1bae682d42e38c0295ed6299ba0f9ac439427fe6
2 changes: 1 addition & 1 deletion doc/source/timeseries.rst
Original file line number Diff line number Diff line change
Expand Up @@ -2353,7 +2353,7 @@ constructor as well as ``tz_localize``.
.. _timeseries.timezone_nonexsistent:

Nonexistent Times when Localizing
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

A DST transition may also shift the local time ahead by 1 hour creating nonexistent
local times. The behavior of localizing a timeseries with nonexistent times
Expand Down
2 changes: 1 addition & 1 deletion doc/source/whatsnew/v0.24.0.txt
Original file line number Diff line number Diff line change
Expand Up @@ -191,7 +191,7 @@ Other Enhancements
- :meth:`Series.resample` and :meth:`DataFrame.resample` have gained the :meth:`Resampler.quantile` (:issue:`15023`).
- :meth:`Index.to_frame` now supports overriding column name(s) (:issue:`22580`).
- New attribute :attr:`__git_version__` will return git commit sha of current build (:issue:`21295`).
- :meth:`Timestamp.tz_localize`, :meth:`DatetimeIndex.tz_localize`, and :meth:`Series.tz_localize` have gained the ``nonexistent`` argument for alternative handling of nonexistent times (:issue:`8917`)
- :meth:`Timestamp.tz_localize`, :meth:`DatetimeIndex.tz_localize`, and :meth:`Series.tz_localize` have gained the ``nonexistent`` argument for alternative handling of nonexistent times. See :ref:`timeseries.timezone_nonexsistent` (:issue:`8917`)

.. _whatsnew_0240.api_breaking:

Expand Down
12 changes: 5 additions & 7 deletions pandas/_libs/tslibs/conversion.pyx
Original file line number Diff line number Diff line change
@@ -1,6 +1,4 @@
# -*- coding: utf-8 -*-
import warnings

import cython
from cython import Py_ssize_t

Expand Down Expand Up @@ -841,8 +839,13 @@ def tz_localize_to_utc(ndarray[int64_t] vals, object tz, object ambiguous=None,
If arraylike, must have the same length as vals
nonexistent : str, bool, or arraylike
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

versionadded

If arraylike, must have the same length as vals

.. versionadded:: 0.24.0

errors : {"raise", "coerce"}, default "raise"

.. depreciated:: 0.24.0
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

deprecated


Returns
-------
localized : ndarray[int64_t]
Expand All @@ -863,11 +866,6 @@ def tz_localize_to_utc(ndarray[int64_t] vals, object tz, object ambiguous=None,
# Vectorized version of DstTzInfo.localize
assert is_coerce or is_raise

if is_coerce:
warnings.warn("the errors argument is deprecated, will be removed "
"in a future release. Use the ambiguous or nonexistent "
"argument instead.", DeprecationWarning)

if tz == UTC or tz is None:
return vals

Expand Down
6 changes: 6 additions & 0 deletions pandas/_libs/tslibs/timestamps.pyx
Original file line number Diff line number Diff line change
Expand Up @@ -944,6 +944,12 @@ class Timestamp(_Timestamp):
if ambiguous == 'infer':
raise ValueError('Cannot infer offset with only one time.')

if errors != 'raise':
warnings.warn("The errors argument is deprecated and will be "
"removed in a future release. Use the ambiguous or "
"nonexistent argument instead.", FutureWarning,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we can be more explicit here about how to change your code (basically the replacement that you do the lines below in code)
(and same as above, ambiguous is not relevant I think?)

stacklevel=2)

if self.tzinfo is None:
# tz naive, localize
tz = maybe_get_tz(tz)
Expand Down
5 changes: 5 additions & 0 deletions pandas/core/arrays/datetimes.py
Original file line number Diff line number Diff line change
Expand Up @@ -703,6 +703,11 @@ def tz_localize(self, tz, ambiguous='raise', nonexistent='raise',
'2018-03-03 09:00:00'],
dtype='datetime64[ns]', freq='D')
"""
if errors != 'raise':
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what does errors = 'coerce' actually imply in terms of nonexistent / ambiguous (see my comment above). simply translate this.

Also instead of leaving the default, make errors=None, then check if errors is not None for the warning. Then if its None, set to the appropriate action. IOW if its explicity passed at all you want to show the warning, whether its the default or not.

warnings.warn("The errors argument is deprecated and will be "
"removed in a future release. Use the ambiguous or "
"nonexistent argument instead.", FutureWarning,
stacklevel=2)
if self.tz is not None:
if tz is None:
new_dates = conversion.tz_convert(self.asi8, 'UTC', self.tz)
Expand Down
8 changes: 8 additions & 0 deletions pandas/tests/indexes/datetimes/test_timezones.py
Original file line number Diff line number Diff line change
Expand Up @@ -592,6 +592,14 @@ def test_dti_tz_localize_nonexistent(self, tz, method, exp):
expected = DatetimeIndex([exp] * n, tz=tz)
tm.assert_index_equal(result, expected)

@pytest.mark.filterwarnings('ignore::FutureWarning')
def test_dti_tz_localize_errors_deprecation(self):
# GH 22644
n = 60
dti = date_range(start='2015-03-29 02:00:00', periods=n, freq='min')
with tm.assert_produces_warning(FutureWarning, check_stacklevel=False):
dti.tz_localize('UTC', errors='coerce')

# -------------------------------------------------------------
# DatetimeIndex.normalize

Expand Down
4 changes: 3 additions & 1 deletion pandas/tests/scalar/timestamp/test_timezones.py
Original file line number Diff line number Diff line change
Expand Up @@ -79,14 +79,16 @@ def test_tz_localize_ambiguous(self):
('2015-03-08 02:30', 'US/Pacific'),
('2015-03-29 02:00', 'Europe/Paris'),
('2015-03-29 02:30', 'Europe/Belgrade')])
@pytest.mark.filterwarnings('ignore::FutureWarning')
def test_tz_localize_nonexistent(self, stamp, tz):
# GH#13057
ts = Timestamp(stamp)
with pytest.raises(NonExistentTimeError):
ts.tz_localize(tz)
with pytest.raises(NonExistentTimeError):
ts.tz_localize(tz, errors='raise')
with tm.assert_produces_warning(DeprecationWarning):
# GH 22644
with tm.assert_produces_warning(FutureWarning, check_stacklevel=False):
assert ts.tz_localize(tz, errors='coerce') is NaT

def test_tz_localize_errors_ambiguous(self):
Expand Down
8 changes: 8 additions & 0 deletions pandas/tests/series/test_timezones.py
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,14 @@ def test_series_tz_localize(self):
tm.assert_raises_regex(TypeError, 'Already tz-aware',
ts.tz_localize, 'US/Eastern')

@pytest.mark.filterwarnings('ignore::FutureWarning')
def test_tz_localize_errors_deprecation(self):
rng = date_range('1/1/2011', periods=100, freq='H')
ts = Series(rng)
with tm.assert_produces_warning(FutureWarning, check_stacklevel=False):
# GH 22644
ts.dt.tz_localize('UTC', errors='coerce')

def test_series_tz_localize_ambiguous_bool(self):
# make sure that we are correctly accepting bool values as ambiguous

Expand Down