Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Replace usage of total_seconds compat func with timedelta method #17289

Merged
merged 4 commits into from
Aug 25, 2017
Merged

Replace usage of total_seconds compat func with timedelta method #17289

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
32c8153
Replace usage of total_seconds compat func with timedelta method
jbrockmendel Aug 18, 2017
252dad9
Move implementation of total_seconds to objToJSON.c; remove datetime_…
jbrockmendel Aug 19, 2017
9c0e824
Comment TODO to remove workaround
jbrockmendel Aug 20, 2017
2d2c549
lint fixup?
jbrockmendel Aug 20, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 2 additions & 5 deletions pandas/_libs/period.pyx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,9 +10,6 @@ from numpy cimport (int8_t, int32_t, int64_t, import_array, ndarray,
NPY_INT64, NPY_DATETIME, NPY_TIMEDELTA)
import numpy as np

cdef extern from "datetime_helper.h":
double total_seconds(object)

from libc.stdlib cimport free

from pandas import compat
Expand Down Expand Up @@ -570,7 +567,7 @@ cdef _reso_local(ndarray[int64_t] stamps, object tz):
&dts)
dt = datetime(dts.year, dts.month, dts.day, dts.hour,
dts.min, dts.sec, dts.us, tz)
delta = int(total_seconds(_get_utcoffset(tz, dt))) * 1000000000
delta = int(_get_utcoffset(tz, dt).total_seconds()) * 1000000000
pandas_datetime_to_datetimestruct(stamps[i] + delta,
PANDAS_FR_ns, &dts)
curr_reso = _reso_stamp(&dts)
Expand Down Expand Up @@ -637,7 +634,7 @@ cdef ndarray[int64_t] localize_dt64arr_to_period(ndarray[int64_t] stamps,
&dts)
dt = datetime(dts.year, dts.month, dts.day, dts.hour,
dts.min, dts.sec, dts.us, tz)
delta = int(total_seconds(_get_utcoffset(tz, dt))) * 1000000000
delta = int(_get_utcoffset(tz, dt).total_seconds()) * 1000000000
pandas_datetime_to_datetimestruct(stamps[i] + delta,
PANDAS_FR_ns, &dts)
result[i] = get_period_ordinal(dts.year, dts.month, dts.day,
Expand Down
36 changes: 0 additions & 36 deletions pandas/_libs/src/datetime_helper.h
View file
Open in desktop

This file was deleted.

22 changes: 21 additions & 1 deletion pandas/_libs/src/ujson/python/objToJSON.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -47,9 +47,9 @@ Numeric decoder derived from from TCL library
#include <numpy_helper.h> // NOLINT(build/include_order)
#include <stdio.h> // NOLINT(build/include_order)
#include <ultrajson.h> // NOLINT(build/include_order)
#include <datetime_helper.h> // NOLINT(build/include_order)
#include <np_datetime.h> // NOLINT(build/include_order)
#include <np_datetime_strings.h> // NOLINT(build/include_order)
#include "datetime.h"

static PyObject *type_decimal;

Expand Down Expand Up @@ -329,6 +329,26 @@ static Py_ssize_t get_attr_length(PyObject *obj, char *attr) {
return ret;
}

npy_int64 get_long_attr(PyObject *o, const char *attr) {
npy_int64 long_val;
PyObject *value = PyObject_GetAttrString(o, attr);
long_val = (PyLong_Check(value) ?
PyLong_AsLongLong(value) : PyInt_AS_LONG(value));
Py_DECREF(value);
return long_val;
}

npy_float64 total_seconds(PyObject *td) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

so you shouldn't need this here either

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Getting rid of it would be great. Neither @jorisvandenbossche nor I know how to call obj.total_seconds() in C, and are not inclined to guess. If there's a quick answer I'll edit.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

look in the CPython source code

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

look in the CPython source code

My attempts so far have resulted in test failures and segfaults. Having me spend more time futzing with C benefits no one. If you don't like the PR as it is, feel free to close.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

well this is reducing some code which is nice. but leaving a loose end. If you want to make an issue for this loose end I would accept. (and add a comment in the code as well).

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is no public c api for getting total seconds from a timedelta object (only for days, seconds, microseconds -> https://docs.python.org/3.3/c-api/datetime.html#PyDateTime_DELTA_GET_DAYS, which also is only new in python 3). So through the c-api this is not possible.

But, I suppose we can just get the python attribute total_seconds without using the c-api ? (or is that only possible in cython ?).
@jbrockmendel does npy_int64 days = get_long_attr(td, "total_seconds"); work?

Otherwise, I would just merge this as it is a net improvement

Copy link
Member Author

@jbrockmendel jbrockmendel Aug 20, 2017 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just pushed a comment suggesting it be removed.

does npy_int64 days = get_long_attr(td, "total_seconds"); work?

@jorisvandenbossche I didn't try that specifically because total_seconds is a nullary method. What I tried (and failed) was several variants of PyObject_CallMethod

// Python 2.6 compat
// TODO(anyone): remove this legacy workaround with a more
// direct td.total_seconds()
npy_int64 microseconds = get_long_attr(td, "microseconds");
npy_int64 seconds = get_long_attr(td, "seconds");
npy_int64 days = get_long_attr(td, "days");
npy_int64 days_in_seconds = days * 24LL * 3600LL;
return (microseconds + (seconds + days_in_seconds) * 1000000.0) / 1000000.0;
}

static PyObject *get_item(PyObject *obj, Py_ssize_t i) {
PyObject *tmp = PyInt_FromSsize_t(i);
PyObject *ret;
Expand Down
28 changes: 11 additions & 17 deletions pandas/_libs/tslib.pyx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,9 +26,6 @@ from cpython cimport (
cdef extern from "Python.h":
cdef PyTypeObject *Py_TYPE(object)

cdef extern from "datetime_helper.h":
double total_seconds(object)

# this is our datetime.pxd
from libc.stdlib cimport free

Expand Down Expand Up @@ -1644,7 +1641,7 @@ cdef inline void _localize_tso(_TSObject obj, object tz):
pandas_datetime_to_datetimestruct(obj.value, PANDAS_FR_ns, &obj.dts)
dt = datetime(obj.dts.year, obj.dts.month, obj.dts.day, obj.dts.hour,
obj.dts.min, obj.dts.sec, obj.dts.us, tz)
delta = int(total_seconds(_get_utcoffset(tz, dt))) * 1000000000
delta = int(_get_utcoffset(tz, dt).total_seconds()) * 1000000000
if obj.value != NPY_NAT:
pandas_datetime_to_datetimestruct(obj.value + delta,
PANDAS_FR_ns, &obj.dts)
Expand Down Expand Up @@ -4141,7 +4138,7 @@ def tz_convert(ndarray[int64_t] vals, object tz1, object tz2):
pandas_datetime_to_datetimestruct(v, PANDAS_FR_ns, &dts)
dt = datetime(dts.year, dts.month, dts.day, dts.hour,
dts.min, dts.sec, dts.us, tz1)
delta = (int(total_seconds(_get_utcoffset(tz1, dt)))
delta = (int(_get_utcoffset(tz1, dt).total_seconds())
* 1000000000)
utc_dates[i] = v - delta
else:
Expand Down Expand Up @@ -4181,8 +4178,8 @@ def tz_convert(ndarray[int64_t] vals, object tz1, object tz2):
pandas_datetime_to_datetimestruct(v, PANDAS_FR_ns, &dts)
dt = datetime(dts.year, dts.month, dts.day, dts.hour,
dts.min, dts.sec, dts.us, tz2)
delta = int(total_seconds(
_get_utcoffset(tz2, dt))) * 1000000000
delta = (int(_get_utcoffset(tz2, dt).total_seconds())
* 1000000000)
result[i] = v + delta
return result

Expand Down Expand Up @@ -4248,7 +4245,7 @@ def tz_convert_single(int64_t val, object tz1, object tz2):
pandas_datetime_to_datetimestruct(val, PANDAS_FR_ns, &dts)
dt = datetime(dts.year, dts.month, dts.day, dts.hour,
dts.min, dts.sec, dts.us, tz1)
delta = int(total_seconds(_get_utcoffset(tz1, dt))) * 1000000000
delta = int(_get_utcoffset(tz1, dt).total_seconds()) * 1000000000
utc_date = val - delta
elif _get_zone(tz1) != 'UTC':
trans, deltas, typ = _get_dst_info(tz1)
Expand All @@ -4266,7 +4263,7 @@ def tz_convert_single(int64_t val, object tz1, object tz2):
pandas_datetime_to_datetimestruct(val, PANDAS_FR_ns, &dts)
dt = datetime(dts.year, dts.month, dts.day, dts.hour,
dts.min, dts.sec, dts.us, tz2)
delta = int(total_seconds(_get_utcoffset(tz2, dt))) * 1000000000
delta = int(_get_utcoffset(tz2, dt).total_seconds()) * 1000000000
return utc_date + delta

# Convert UTC to other timezone
Expand Down Expand Up @@ -4338,7 +4335,7 @@ cdef object _get_dst_info(object tz):
"""
cache_key = _tz_cache_key(tz)
if cache_key is None:
num = int(total_seconds(_get_utcoffset(tz, None))) * 1000000000
num = int(_get_utcoffset(tz, None).total_seconds()) * 1000000000
return (np.array([NPY_NAT + 1], dtype=np.int64),
np.array([num], dtype=np.int64),
None)
Expand Down Expand Up @@ -4385,7 +4382,7 @@ cdef object _get_dst_info(object tz):
else:
# static tzinfo
trans = np.array([NPY_NAT + 1], dtype=np.int64)
num = int(total_seconds(_get_utcoffset(tz, None))) * 1000000000
num = int(_get_utcoffset(tz, None).total_seconds()) * 1000000000
deltas = np.array([num], dtype=np.int64)
typ = 'static'

Expand All @@ -4408,9 +4405,6 @@ cdef object _get_utc_trans_times_from_dateutil_tz(object tz):
return new_trans


def tot_seconds(td):
return total_seconds(td)

cpdef ndarray _unbox_utcoffsets(object transinfo):
cdef:
Py_ssize_t i, sz
Expand All @@ -4420,7 +4414,7 @@ cpdef ndarray _unbox_utcoffsets(object transinfo):
arr = np.empty(sz, dtype='i8')

for i in range(sz):
arr[i] = int(total_seconds(transinfo[i][0])) * 1000000000
arr[i] = int(transinfo[i][0].total_seconds()) * 1000000000

return arr

Expand Down Expand Up @@ -4463,7 +4457,7 @@ def tz_localize_to_utc(ndarray[int64_t] vals, object tz, object ambiguous=None,
pandas_datetime_to_datetimestruct(v, PANDAS_FR_ns, &dts)
dt = datetime(dts.year, dts.month, dts.day, dts.hour,
dts.min, dts.sec, dts.us, tz)
delta = int(total_seconds(_get_utcoffset(tz, dt))) * 1000000000
delta = int(_get_utcoffset(tz, dt).total_seconds()) * 1000000000
result[i] = v - delta
return result

Expand Down Expand Up @@ -5202,7 +5196,7 @@ cdef _normalize_local(ndarray[int64_t] stamps, object tz):
pandas_datetime_to_datetimestruct(stamps[i], PANDAS_FR_ns, &dts)
dt = datetime(dts.year, dts.month, dts.day, dts.hour,
dts.min, dts.sec, dts.us, tz)
delta = int(total_seconds(_get_utcoffset(tz, dt))) * 1000000000
delta = int(_get_utcoffset(tz, dt).total_seconds()) * 1000000000
pandas_datetime_to_datetimestruct(stamps[i] + delta,
PANDAS_FR_ns, &dts)
result[i] = _normalized_stamp(&dts)
Expand Down
2 changes: 1 addition & 1 deletion pandas/io/pytables.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4381,7 +4381,7 @@ def _get_tz(tz):
""" for a tz-aware type, return an encoded zone """
zone = tslib.get_timezone(tz)
if zone is None:
zone = tslib.tot_seconds(tz.utcoffset())
zone = tz.utcoffset().total_seconds()
return zone


Expand Down
6 changes: 3 additions & 3 deletions pandas/tseries/offsets.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -777,12 +777,12 @@ def _get_business_hours_by_sec(self):
# create dummy datetime to calcurate businesshours in a day
dtstart = datetime(2014, 4, 1, self.start.hour, self.start.minute)
until = datetime(2014, 4, 1, self.end.hour, self.end.minute)
return tslib.tot_seconds(until - dtstart)
return (until - dtstart).total_seconds()
else:
self.daytime = False
dtstart = datetime(2014, 4, 1, self.start.hour, self.start.minute)
until = datetime(2014, 4, 2, self.end.hour, self.end.minute)
return tslib.tot_seconds(until - dtstart)
return (until - dtstart).total_seconds()

@apply_wraps
def rollback(self, dt):
Expand Down Expand Up @@ -906,7 +906,7 @@ def _onOffset(self, dt, businesshours):
op = self._prev_opening_time(dt)
else:
op = self._next_opening_time(dt)
span = tslib.tot_seconds(dt - op)
span = (dt - op).total_seconds()
if span <= businesshours:
return True
else:
Expand Down
2 changes: 0 additions & 2 deletions setup.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -467,7 +467,6 @@ def pxd(name):

tseries_depends = ['pandas/_libs/src/datetime/np_datetime.h',
'pandas/_libs/src/datetime/np_datetime_strings.h',
'pandas/_libs/src/datetime_helper.h',
'pandas/_libs/src/period_helper.h',
'pandas/_libs/src/datetime.pxd']

Expand Down Expand Up @@ -597,7 +596,6 @@ def pxd(name):

ujson_ext = Extension('pandas._libs.json',
depends=['pandas/_libs/src/ujson/lib/ultrajson.h',
'pandas/_libs/src/datetime_helper.h',
'pandas/_libs/src/numpy_helper.h'],
sources=['pandas/_libs/src/ujson/python/ujson.c',
'pandas/_libs/src/ujson/python/objToJSON.c',
Expand Down