Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add more dtypes #122

Merged
merged 12 commits into from
Mar 16, 2023
Prev Previous commit
Next Next commit
updated precommit
  • Loading branch information
melvinfolkers committed Mar 16, 2023
commit 487d9301c1348589ed5f678a67f6bd5a3a5f7e91
21 changes: 18 additions & 3 deletions .pre-commit-config.yaml
Original file line number Diff line number Diff line change
@@ -1,22 +1,37 @@
repos:
- repo: https://github.com/pre-commit/pre-commit-hooks
rev: v3.2.0
rev: v4.3.0
hooks:
- id: trailing-whitespace
- id: end-of-file-fixer
- id: check-yaml
- id: check-added-large-files
- id: check-merge-conflict
- id: debug-statements
- id: detect-private-key
- id: name-tests-test
args: [--pytest-test-first]
- id: requirements-txt-fixer
- repo: https://github.com/pycqa/flake8
rev: 3.9.2
rev: 5.0.4
hooks:
- id: flake8
args: ["--statistics", "--count", "--max-complexity=10", "--max-line-length=120", "--per-file-ignore=__init__.py: F401"]
- repo: https://github.com/psf/black
rev: 22.3.0
hooks:
- id: black
args: [--line-length=120]
- repo: https://github.com/PyCQA/isort
rev: 5.9.1
rev: 5.12.0
hooks:
- id: isort
args: ["--profile", "black", --line-length=120]
- repo: local
hooks:
- id: check-requirements
name: Check requirements
description: Check if requirements in setup.cfg and requirements.txt are equal
language: python
entry: python scripts/check_setupcfg_and_requirements_equal.py
pass_filenames: false
13 changes: 12 additions & 1 deletion df_to_azure/export.py
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,17 @@
import pandas as pd
from azure.storage.blob import BlobServiceClient
from numpy import dtype
from pandas import BooleanDtype, DataFrame, Float64Dtype, Int8Dtype, Int16Dtype, Int32Dtype, Int64Dtype, StringDtype
from pandas import (
BooleanDtype,
DataFrame,
DatetimeTZDtype,
Float64Dtype,
Int8Dtype,
Int16Dtype,
Int32Dtype,
Int64Dtype,
StringDtype,
)
from sqlalchemy.sql.visitors import VisitableType
from sqlalchemy.types import BigInteger, Boolean, DateTime, Integer, Numeric, String

Expand Down Expand Up @@ -241,6 +251,7 @@ def column_types(self) -> dict:
dtype("<M8[ns]"): DateTime(),
dtype("bool"): Boolean(),
BooleanDtype(): Boolean(),
DatetimeTZDtype(tz="utc"): DateTime(),
}

col_types = {col_name: type_conversion[col_type] for col_name, col_type in self.df.dtypes.to_dict().items()}
Expand Down