Merge branch 'master' into weighted-roll-var

Author: ihsansecer

ci/deps/azure-37-locale.yaml

@@ -17,7 +17,7 @@ dependencies:
   - openpyxl
   - pytables
   - python-dateutil
-  - python=3.7.3
+  - python=3.7.*
   - pytz
   - s3fs
   - scipy

ci/deps/azure-37-numpydev.yaml

@@ -2,7 +2,7 @@ name: pandas-dev
 channels:
   - defaults
 dependencies:
-  - python=3.7.3
+  - python=3.7.*
   - pytz
   - Cython>=0.28.2
   # universal

ci/deps/travis-37.yaml

@@ -4,7 +4,7 @@ channels:
   - conda-forge
   - c3i_test
 dependencies:
-  - python=3.7.3
+  - python=3.7.*
   - botocore>=1.11
   - cython>=0.28.2
   - numpy

doc/source/user_guide/io.rst

@@ -3572,7 +3572,7 @@ Closing a Store and using a context manager:
 Read/write API
 ''''''''''''''
 
-``HDFStore`` supports an top-level API using  ``read_hdf`` for reading and ``to_hdf`` for writing,
+``HDFStore`` supports a top-level API using  ``read_hdf`` for reading and ``to_hdf`` for writing,
 similar to how ``read_csv`` and ``to_csv`` work.
 
 .. ipython:: python
@@ -3687,7 +3687,7 @@ Hierarchical keys
 Keys to a store can be specified as a string. These can be in a
 hierarchical path-name like format (e.g. ``foo/bar/bah``), which will
 generate a hierarchy of sub-stores (or ``Groups`` in PyTables
-parlance). Keys can be specified with out the leading '/' and are **always**
+parlance). Keys can be specified without the leading '/' and are **always**
 absolute (e.g. 'foo' refers to '/foo'). Removal operations can remove
 everything in the sub-store and **below**, so be *careful*.
 
@@ -3825,7 +3825,7 @@ data.
 
 A query is specified using the ``Term`` class under the hood, as a boolean expression.
 
-* ``index`` and ``columns`` are supported indexers of a ``DataFrames``.
+* ``index`` and ``columns`` are supported indexers of ``DataFrames``.
 * if ``data_columns`` are specified, these can be used as additional indexers.
 
 Valid comparison operators are:
@@ -3917,7 +3917,7 @@ Use boolean expressions, with in-line function evaluation.
 
     store.select('dfq', "index>pd.Timestamp('20130104') & columns=['A', 'B']")
 
-Use and inline column reference
+Use inline column reference.
 
 .. ipython:: python
 
@@ -4593,8 +4593,8 @@ Performance
   write chunksize (default is 50000). This will significantly lower
   your memory usage on writing.
 * You can pass ``expectedrows=<int>`` to the first ``append``,
-  to set the TOTAL number of expected rows that ``PyTables`` will
-  expected. This will optimize read/write performance.
+  to set the TOTAL number of rows that ``PyTables`` will expect.
+  This will optimize read/write performance.
 * Duplicate rows can be written to tables, but are filtered out in
   selection (with the last items being selected; thus a table is
   unique on major, minor pairs)

doc/source/whatsnew/v0.25.1.rst

@@ -25,8 +25,7 @@ Bug fixes
 Categorical
 ^^^^^^^^^^^
 
--
--
+- Bug in :meth:`Categorical.fillna` would replace all values, not just those that are ``NaN`` (:issue:`26215`)
 -
 
 Datetimelike
@@ -83,7 +82,7 @@ Indexing
 ^^^^^^^^
 
 - Bug in partial-string indexing returning a NumPy array rather than a ``Series`` when indexing with a scalar like ``.loc['2015']`` (:issue:`27516`)
-- Break reference cycle involving :class:`Index` to allow garbage collection of :class:`Index` objects without running the GC. (:issue:`27585`)
+- Break reference cycle involving :class:`Index` and other index classes to allow garbage collection of index objects without running the GC. (:issue:`27585`, :issue:`27840`)
 - Fix regression in assigning values to a single column of a DataFrame with a ``MultiIndex`` columns (:issue:`27841`).
 -
 
@@ -105,7 +104,7 @@ I/O
 ^^^
 
 - Avoid calling ``S3File.s3`` when reading parquet, as this was removed in s3fs version 0.3.0 (:issue:`27756`)
--
+- Better error message when a negative header is passed in :func:`pandas.read_csv` (:issue:`27779`)
 -
 
 Plotting
@@ -127,9 +126,9 @@ Reshaping
 ^^^^^^^^^
 
 - A ``KeyError`` is now raised if ``.unstack()`` is called on a :class:`Series` or :class:`DataFrame` with a flat :class:`Index` passing a name which is not the correct one (:issue:`18303`)
--  Bug in :meth:`DataFrame.crosstab` when ``margins`` set to ``True`` and ``normalize`` is not ``False``, an error is raised. (:issue:`27500`)
+- Bug in :meth:`DataFrame.crosstab` when ``margins`` set to ``True`` and ``normalize`` is not ``False``, an error is raised. (:issue:`27500`)
 - :meth:`DataFrame.join` now suppresses the ``FutureWarning`` when the sort parameter is specified (:issue:`21952`)
--
+- Bug in :meth:`DataFrame.join` raising with readonly arrays (:issue:`27943`)
 
 Sparse
 ^^^^^^

doc/source/whatsnew/v1.0.0.rst

@@ -21,27 +21,27 @@ including other versions of pandas.
 Enhancements
 ~~~~~~~~~~~~
 
-.. _whatsnew_1000.enhancements.other:
-
 -
 -
 
+.. _whatsnew_1000.enhancements.other:
+
 Other enhancements
 ^^^^^^^^^^^^^^^^^^
 
-.. _whatsnew_1000.api_breaking:
-
 - Implemented :meth:`pandas.core.window.Window.var` and :meth:`pandas.core.window.Window.std` functions (:issue:`26597`)
 -
 
+.. _whatsnew_1000.api_breaking:
+
 Backwards incompatible API changes
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-.. _whatsnew_1000.api.other:
-
 - :class:`pandas.core.groupby.GroupBy.transform` now raises on invalid operation names (:issue:`27489`).
 -
 
+.. _whatsnew_1000.api.other:
+
 Other API changes
 ^^^^^^^^^^^^^^^^^
 
@@ -87,6 +87,7 @@ Bug fixes
 Categorical
 ^^^^^^^^^^^
 
+- Added test to assert the :func:`fillna` raises the correct ValueError message when the value isn't a value from categories (:issue:`13628`)
 -
 -
 
@@ -165,6 +166,7 @@ Plotting
 
 - Bug in :meth:`Series.plot` not able to plot boolean values (:issue:`23719`)
 -
+- Bug in :meth:`DataFrame.plot` producing incorrect legend markers when plotting multiple series on the same axis (:issue:`18222`)
 - Bug in :meth:`DataFrame.plot` when ``kind='box'`` and data contains datetime or timedelta data. These types are now automatically dropped (:issue:`22799`)
 
 Groupby/resample/rolling

environment.yml

@@ -5,7 +5,7 @@ channels:
 dependencies:
   # required
   - numpy>=1.15
-  - python=3.7.3
+  - python=3
   - python-dateutil>=2.6.1
   - pytz
 

pandas/_libs/hashtable.pyx

@@ -108,7 +108,7 @@ cdef class Int64Factorizer:
     def get_count(self):
         return self.count
 
-    def factorize(self, int64_t[:] values, sort=False,
+    def factorize(self, const int64_t[:] values, sort=False,
                   na_sentinel=-1, na_value=None):
         """
         Factorize values with nans replaced by na_sentinel

pandas/_libs/tslibs/nattype.pyx

@@ -92,6 +92,9 @@ cdef class _NaT(datetime):
     #    int64_t value
     #    object freq
 
+    # higher than np.ndarray and np.matrix
+    __array_priority__ = 100
+
     def __hash__(_NaT self):
         # py3k needs this defined here
         return hash(self.value)
@@ -103,61 +106,102 @@ cdef class _NaT(datetime):
         if ndim == -1:
             return _nat_scalar_rules[op]
 
-        if ndim == 0:
+        elif util.is_array(other):
+            result = np.empty(other.shape, dtype=np.bool_)
+            result.fill(_nat_scalar_rules[op])
+            return result
+
+        elif ndim == 0:
             if is_datetime64_object(other):
                 return _nat_scalar_rules[op]
             else:
                 raise TypeError('Cannot compare type %r with type %r' %
                                 (type(self).__name__, type(other).__name__))
+
         # Note: instead of passing "other, self, _reverse_ops[op]", we observe
         # that `_nat_scalar_rules` is invariant under `_reverse_ops`,
         # rendering it unnecessary.
         return PyObject_RichCompare(other, self, op)
 
     def __add__(self, other):
+        if self is not c_NaT:
+            # cython __radd__ semantics
+            self, other = other, self
+
         if PyDateTime_Check(other):
             return c_NaT
-
+        elif PyDelta_Check(other):
+            return c_NaT
+        elif is_datetime64_object(other) or is_timedelta64_object(other):
+            return c_NaT
         elif hasattr(other, 'delta'):
             # Timedelta, offsets.Tick, offsets.Week
             return c_NaT
-        elif getattr(other, '_typ', None) in ['dateoffset', 'series',
-                                              'period', 'datetimeindex',
-                                              'datetimearray',
-                                              'timedeltaindex',
-                                              'timedeltaarray']:
-            # Duplicate logic in _Timestamp.__add__ to avoid needing
-            # to subclass; allows us to @final(_Timestamp.__add__)
-            return NotImplemented
-        return c_NaT
+
+        elif is_integer_object(other) or util.is_period_object(other):
+            # For Period compat
+            # TODO: the integer behavior is deprecated, remove it
+            return c_NaT
+
+        elif util.is_array(other):
+            if other.dtype.kind in 'mM':
+                # If we are adding to datetime64, we treat NaT as timedelta
+                #  Either way, result dtype is datetime64
+                result = np.empty(other.shape, dtype="datetime64[ns]")
+                result.fill("NaT")
+                return result
+
+        return NotImplemented
 
     def __sub__(self, other):
         # Duplicate some logic from _Timestamp.__sub__ to avoid needing
         # to subclass; allows us to @final(_Timestamp.__sub__)
+        cdef:
+            bint is_rsub = False
+
+        if self is not c_NaT:
+            # cython __rsub__ semantics
+            self, other = other, self
+            is_rsub = True
+
         if PyDateTime_Check(other):
-            return NaT
+            return c_NaT
         elif PyDelta_Check(other):
-            return NaT
+            return c_NaT
+        elif is_datetime64_object(other) or is_timedelta64_object(other):
+            return c_NaT
+        elif hasattr(other, 'delta'):
+            # offsets.Tick, offsets.Week
+            return c_NaT
 
-        elif getattr(other, '_typ', None) == 'datetimeindex':
-            # a Timestamp-DatetimeIndex -> yields a negative TimedeltaIndex
-            return -other.__sub__(self)
+        elif is_integer_object(other) or util.is_period_object(other):
+            # For Period compat
+            # TODO: the integer behavior is deprecated, remove it
+            return c_NaT
 
-        elif getattr(other, '_typ', None) == 'timedeltaindex':
-            # a Timestamp-TimedeltaIndex -> yields a negative TimedeltaIndex
-            return (-other).__add__(self)
+        elif util.is_array(other):
+            if other.dtype.kind == 'm':
+                if not is_rsub:
+                    # NaT - timedelta64 we treat NaT as datetime64, so result
+                    #  is datetime64
+                    result = np.empty(other.shape, dtype="datetime64[ns]")
+                    result.fill("NaT")
+                    return result
+
+                # timedelta64 - NaT we have to treat NaT as timedelta64
+                #  for this to be meaningful, and the result is timedelta64
+                result = np.empty(other.shape, dtype="timedelta64[ns]")
+                result.fill("NaT")
+                return result
+
+            elif other.dtype.kind == 'M':
+                # We treat NaT as a datetime, so regardless of whether this is
+                #  NaT - other or other - NaT, the result is timedelta64
+                result = np.empty(other.shape, dtype="timedelta64[ns]")
+                result.fill("NaT")
+                return result
 
-        elif hasattr(other, 'delta'):
-            # offsets.Tick, offsets.Week
-            neg_other = -other
-            return self + neg_other
-
-        elif getattr(other, '_typ', None) in ['period', 'series',
-                                              'periodindex', 'dateoffset',
-                                              'datetimearray',
-                                              'timedeltaarray']:
-            return NotImplemented
-        return NaT
+        return NotImplemented
 
     def __pos__(self):
         return NaT

pandas/compat/chainmap.py

@@ -15,9 +15,3 @@ def __delitem__(self, key):
                 del mapping[key]
                 return
         raise KeyError(key)
-
-    # override because the m parameter is introduced in Python 3.4
-    def new_child(self, m=None):
-        if m is None:
-            m = {}
-        return self.__class__(m, *self.maps)

pandas/core/algorithms.py

@@ -28,13 +28,11 @@
     is_complex_dtype,
     is_datetime64_any_dtype,
     is_datetime64_ns_dtype,
-    is_datetime64tz_dtype,
     is_datetimelike,
     is_extension_array_dtype,
     is_float_dtype,
     is_integer,
     is_integer_dtype,
-    is_interval_dtype,
     is_list_like,
     is_numeric_dtype,
     is_object_dtype,
@@ -183,8 +181,6 @@ def _reconstruct_data(values, dtype, original):
 
     if is_extension_array_dtype(dtype):
         values = dtype.construct_array_type()._from_sequence(values)
-    elif is_datetime64tz_dtype(dtype) or is_period_dtype(dtype):
-        values = Index(original)._shallow_copy(values, name=None)
     elif is_bool_dtype(dtype):
         values = values.astype(dtype)
 
@@ -1645,19 +1641,13 @@ def take_nd(
         May be the same type as the input, or cast to an ndarray.
     """
 
-    # TODO(EA): Remove these if / elifs as datetimeTZ, interval, become EAs
-    # dispatch to internal type takes
     if is_extension_array_dtype(arr):
         return arr.take(indexer, fill_value=fill_value, allow_fill=allow_fill)
-    elif is_datetime64tz_dtype(arr):
-        return arr.take(indexer, fill_value=fill_value, allow_fill=allow_fill)
-    elif is_interval_dtype(arr):
-        return arr.take(indexer, fill_value=fill_value, allow_fill=allow_fill)
 
     if is_sparse(arr):
         arr = arr.to_dense()
     elif isinstance(arr, (ABCIndexClass, ABCSeries)):
-        arr = arr.values
+        arr = arr._values
 
     arr = np.asarray(arr)
 

pandas/core/arrays/categorical.py

@@ -1840,8 +1840,8 @@ def fillna(self, value=None, method=None, limit=None):
                     raise ValueError("fill value must be in categories")
 
                 values_codes = _get_codes_for_values(value, self.categories)
-                indexer = np.where(values_codes != -1)
-                codes[indexer] = values_codes[values_codes != -1]
+                indexer = np.where(codes == -1)
+                codes[indexer] = values_codes[indexer]
 
             # If value is not a dict or Series it should be a scalar
             elif is_hashable(value):