Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

🎉 Source Hubspot: update user-defined custom field schema generation #4913

Merged
merged 28 commits into from
Aug 3, 2021
Merged

🎉 Source Hubspot: update user-defined custom field schema generation #4913

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
28 commits
Select commit Hold shift + click to select a range
be5ded7
Handle field types, convert unknown to string with warning
gaart Jul 22, 2021
82a66d7
Merge branch 'master' into gaart/4779-bug-sync-hubspot-bigquery
gaart Jul 22, 2021
9729bbe
Bump version
gaart Jul 22, 2021
16d61ee
Upd version in source definitions
gaart Jul 23, 2021
e1fbd88
Add json field format for date fields
gaart Jul 23, 2021
87d671e
Add a stub integration test to bypass CI/CD test stage
gaart Jul 23, 2021
5fec26b
Replace cached_property with lru_cache to support Python < v3.8
gaart Jul 23, 2021
c9afde7
Extract the field format converting logic, cover it with a unit test
gaart Jul 23, 2021
ff59e12
Set default field type to `string`
gaart Jul 27, 2021
727fe3b
Upd changelog
gaart Jul 27, 2021
0982bd5
Merge branch 'master' of github.com:airbytehq/airbyte into gaart/4779…
gaart Jul 27, 2021
e76d7c8
Cleanup
gaart Jul 27, 2021
fb055c7
Fix tests
gaart Jul 27, 2021
e2f9299
Merge branch 'master' of github.com:airbytehq/airbyte into gaart/4779…
gaart Jul 27, 2021
ba51f8b
Merge branch 'master' of github.com:airbytehq/airbyte into gaart/4779…
gaart Jul 28, 2021
c76fed2
Downgrade version tag
gaart Jul 28, 2021
69f6f62
Merge branch 'master' of github.com:airbytehq/airbyte into gaart/4779…
gaart Jul 28, 2021
5354ad5
Remove caching for the custom properties field
gaart Jul 28, 2021
c31420d
Bump version tag
gaart Jul 28, 2021
0fbb4ff
Merge branch 'master' of github.com:airbytehq/airbyte into gaart/4779…
gaart Jul 28, 2021
418b794
Add type casting for malformed fields
gaart Jul 29, 2021
25fff4f
Upd schemas
gaart Jul 29, 2021
d85d7e9
Disable connection tests
gaart Jul 29, 2021
2d51b7a
Fix docs, add custom properties cache
gaart Aug 3, 2021
8b35353
Upd tests, check failed state
gaart Aug 3, 2021
b57c6ba
Disable invalid config test
gaart Aug 3, 2021
d856d8a
Merge remote-tracking branch 'origin' into gaart/4779-bug-sync-hubspo…
gaart Aug 3, 2021
41230f8
Add starte_date param into config
gaart Aug 3, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion airbyte-integrations/connectors/source-hubspot/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,5 +14,5 @@ RUN pip install .

ENV AIRBYTE_ENTRYPOINT "/airbyte/base.sh"

LABEL io.airbyte.version=0.1.5
LABEL io.airbyte.version=0.1.6
LABEL io.airbyte.name=airbyte/source-hubspot
19 changes: 17 additions & 2 deletions airbyte-integrations/connectors/source-hubspot/source_hubspot/api.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,18 @@
# we got this when provided API Token has incorrect format
CLOUDFLARE_ORIGIN_DNS_ERROR = 530

VALID_JSON_SCHEMA_TYPES = {"string", "integer", "number", "boolean", "object", "array", "bit"}
gaart marked this conversation as resolved.
Show resolved Hide resolved

KNOWN_CONVERTIBLE_SCHEMA_TYPES = {
"bool": "boolean",
"enumeration": "string",
"date": "string",
"date-time": "string",
"datetime": "string",
"json": "object",
gaart marked this conversation as resolved.
Show resolved Hide resolved
"phone_number": "string",
}


def retry_connection_handler(**kwargs):
"""Retry helper, log each attempt"""
Expand Down Expand Up @@ -303,8 +315,11 @@ def properties(self) -> Mapping[str, Any]:
data = self._api.get(f"/properties/v2/{self.entity}/properties")
for row in data:
field_type = row["type"]
if field_type in ["enumeration", "date", "date-time"]:
field_type = "string"
if field_type not in VALID_JSON_SCHEMA_TYPES:
field_type = KNOWN_CONVERTIBLE_SCHEMA_TYPES.get(field_type) or None
if not field_type:
gaart marked this conversation as resolved.
Show resolved Hide resolved
logger.warn(f"Unsupported type {field_type} converted to string")
gaart marked this conversation as resolved.
Show resolved Hide resolved
field_type = "string"
gaart marked this conversation as resolved.
Show resolved Hide resolved
props[row["name"]] = {"type": field_type}

return props
Expand Down