See https://github.com/pandas-dev/pandas/blob/fdbc6b8f4b36f07da62fc901b19754f922ae3952/pandas/core/categorical.py#L319-L332