🎉 New Source: Recharge #3466

yevhenii-ldv · 2021-05-18T13:11:10Z

What

closes #2308

How

Describe the solution

Pre-merge Checklist

Run integration tests
Publish Docker images

Recommended reading order

schemas.py
streams.py
source.py

…/new-connector-recharge

yevhenii-ldv · 2021-05-18T13:16:43Z

/test connector=source-recharge

🕑 source-recharge https://github.com/airbytehq/airbyte/actions/runs/853435223
❌ source-recharge https://github.com/airbytehq/airbyte/actions/runs/853435223

yevhenii-ldv · 2021-05-18T13:24:54Z

/test connector=source-recharge

🕑 source-recharge https://github.com/airbytehq/airbyte/actions/runs/853458429
✅ source-recharge https://github.com/airbytehq/airbyte/actions/runs/853458429

keu · 2021-05-18T13:35:46Z

airbyte-integrations/connectors/source-recharge/.dockerignore

@@ -0,0 +1,7 @@
+*
+!Dockerfile
+!Dockerfile.test


do we still need this?

keu · 2021-05-18T13:37:04Z

airbyte-integrations/connectors/source-recharge/build.gradle

+
+dependencies {
+    implementation files(project(':airbyte-integrations:bases:source-acceptance-test').airbyteDocker.outputs)
+    implementation files(project(':airbyte-integrations:bases:base-python').airbyteDocker.outputs)


I believe we don't need this anymore

keu · 2021-05-18T13:37:47Z

airbyte-integrations/connectors/source-recharge/integration_tests/abnormal_state.json

@@ -0,0 +1,21 @@
+{
+  "addresses": {
+    "created_at": "2024-05-18T00:00:00"


I assume you are not going to be around at that time?)

Set other dates

keu · 2021-05-18T13:38:37Z

airbyte-integrations/connectors/source-recharge/setup.py

+from setuptools import find_packages, setup
+
+MAIN_REQUIREMENTS = [
+    "airbyte-cdk",


lets pin version, for example ~=0.1

keu · 2021-05-18T13:39:03Z

airbyte-integrations/connectors/source-recharge/source_recharge/__init__.py

@@ -0,0 +1,27 @@
+"""


this one has old header

keu · 2021-05-18T13:42:30Z

airbyte-integrations/connectors/source-recharge/source_recharge/api.py

+        super().__init__(**kwargs)
+        self._start_date = pendulum.parse(start_date)
+
+    cursor_field = "created_at"


lets keep all class attributes on top of the Class

keu · 2021-05-18T13:56:38Z

airbyte-integrations/connectors/source-recharge/source_recharge/api.py

+
+    def read_records(self, stream_slice: Optional[Mapping[str, Any]] = None, **kwargs) -> Iterable[Mapping[str, Any]]:
+        owner_resources = ["customer", "store", "subscription"]
+        for owner in owner_resources:


are you sure that logic that saves state works correctly here?

It is Not Incremental stream

keu · 2021-05-18T14:11:20Z

airbyte-integrations/connectors/source-recharge/source_recharge/api.py

+
+    primary_key = ["shop", "store"]
+
+    def get_stream_data(self, response_data: Any) -> List[dict]:


lets move this logic inside parent class:

declare data_path property, default return self.name

if data_path is not none response_data = response_data.get(self.name, [])

set data_path to None here
WDYT?

airbyte-integrations/connectors/source-recharge/source_recharge/source.py

keu

looks good in general

yevhenii-ldv · 2021-05-19T14:47:59Z

/test connector=source-recharge

🕑 source-recharge https://github.com/airbytehq/airbyte/actions/runs/857252220
❌ source-recharge https://github.com/airbytehq/airbyte/actions/runs/857252220

sherifnada · 2021-05-19T17:11:40Z

airbyte-integrations/connectors/source-recharge/source_recharge/api.py

+from airbyte_cdk.sources.streams.http import HttpStream
+
+
+class RechargeStream(HttpStream, ABC):


Really clean implementation. Well done!

sherifnada · 2021-05-19T17:12:33Z

airbyte-integrations/connectors/source-recharge/source_recharge/source.py

+
+class RechargeTokenAuthenticator(TokenAuthenticator):
+    def get_auth_header(self) -> Mapping[str, Any]:
+        return {"X-Recharge-Access-Token": self._token}


BTW you can use TokenAuthenticator and pass the auth header instead of creating a custom authenticator

I tried it but doesn't work in this case. I only need the token in the value, and if I specify auth_method as an empty string, then I get the following value:
{"X-Recharge-Access-Token": f" self._token"}
And the requests library swears for the presence of a space before the token itself.

sherifnada · 2021-05-19T17:12:52Z

airbyte-integrations/connectors/source-recharge/source_recharge/source.py

+            list(Shop(authenticator=auth).read_records(SyncMode.full_refresh))
+            return True, None
+        except Exception as error:
+            return False, f"Unable to connect to Recharge API with the provided credentials - {error}"


better to use repr(error)

sherifnada · 2021-05-19T17:13:47Z

docs/integrations/sources/recharge.md

+
+### Performance considerations
+
+The Recharge connector should not run into Recharge API limitations under normal usage. Please [create an issue](https://github.com/airbytehq/airbyte/issues) if you see any rate limit issues that are not automatically retried successfully.


Suggested change

The Recharge connector should not run into Recharge API limitations under normal usage. Please [create an issue](https://github.com/airbytehq/airbyte/issues) if you see any rate limit issues that are not automatically retried successfully.

The Recharge connector should gracefully handle Recharge API limitations under normal usage. Please [create an issue](https://github.com/airbytehq/airbyte/issues) if you see any rate limit issues that are not automatically retried successfully.

yevhenii-ldv · 2021-05-20T11:01:47Z

/test connector=source-recharge

🕑 source-recharge https://github.com/airbytehq/airbyte/actions/runs/860148818
✅ source-recharge https://github.com/airbytehq/airbyte/actions/runs/860148818

…/new-connector-recharge

yevhenii-ldv · 2021-05-20T11:29:08Z

/publish connector=connectors/source-recharge

🕑 connectors/source-recharge https://github.com/airbytehq/airbyte/actions/runs/860220716
✅ connectors/source-recharge https://github.com/airbytehq/airbyte/actions/runs/860220716

ykurochkin added 3 commits May 13, 2021 17:39

Recharge Connector: draft variant

6839b85

add Incremental sync, Acceptance tests, docs

12a147c

Merge branch 'master' of github.com:airbytehq/airbyte into ykurochkin…

e1605fc

…/new-connector-recharge

auto-assign bot requested review from masonwheeler and sherifnada May 18, 2021 13:11

yevhenii-ldv requested review from vitaliizazmic, htrueman, keu, po3na4skld, vovavovavovavova and Zirochkaa and removed request for sherifnada and masonwheeler May 18, 2021 13:11

ykurochkin added 2 commits May 18, 2021 16:18

format

904e269

update setup.py

95bb20a

keu reviewed May 18, 2021

View reviewed changes

airbyte-integrations/connectors/source-recharge/source_recharge/source.py Show resolved Hide resolved

keu suggested changes May 18, 2021

View reviewed changes

update after review

520a267

yevhenii-ldv requested a review from keu May 19, 2021 14:43

yevhenii-ldv requested a review from sherifnada May 19, 2021 14:53

keu approved these changes May 19, 2021

View reviewed changes

sherifnada approved these changes May 19, 2021

View reviewed changes

po3na4skld approved these changes May 19, 2021

View reviewed changes

update code after review

51dcd8d

ykurochkin added 2 commits May 20, 2021 14:22

Merge branch 'master' of github.com:airbytehq/airbyte into ykurochkin…

b5a84ed

…/new-connector-recharge

bump version

de0637a

yevhenii-ldv merged commit 3647c3c into master May 20, 2021

yevhenii-ldv deleted the ykurochkin/new-connector-recharge branch May 20, 2021 11:45

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🎉 New Source: Recharge #3466

🎉 New Source: Recharge #3466

yevhenii-ldv commented May 18, 2021

yevhenii-ldv commented May 18, 2021 •

edited by github-actions bot

Loading

yevhenii-ldv commented May 18, 2021 •

edited by github-actions bot

Loading

keu May 18, 2021

yevhenii-ldv May 19, 2021

keu May 18, 2021

keu May 18, 2021

yevhenii-ldv May 19, 2021

keu May 18, 2021

keu May 18, 2021

keu May 18, 2021 •

edited

Loading

keu May 18, 2021

yevhenii-ldv May 18, 2021

keu May 18, 2021

keu May 18, 2021

keu left a comment

yevhenii-ldv commented May 19, 2021 •

edited by github-actions bot

Loading

sherifnada May 19, 2021

sherifnada May 19, 2021

yevhenii-ldv May 20, 2021

sherifnada May 19, 2021

sherifnada May 19, 2021

yevhenii-ldv commented May 20, 2021 •

edited by github-actions bot

Loading

yevhenii-ldv commented May 20, 2021 •

edited by github-actions bot

Loading


		primary_key = ["shop", "store"]

		def get_stream_data(self, response_data: Any) -> List[dict]:

		from airbyte_cdk.sources.streams.http import HttpStream


		class RechargeStream(HttpStream, ABC):


		### Performance considerations

		The Recharge connector should not run into Recharge API limitations under normal usage. Please [create an issue](https://github.com/airbytehq/airbyte/issues) if you see any rate limit issues that are not automatically retried successfully.

	The Recharge connector should not run into Recharge API limitations under normal usage. Please [create an issue](https://github.com/airbytehq/airbyte/issues) if you see any rate limit issues that are not automatically retried successfully.
	The Recharge connector should gracefully handle Recharge API limitations under normal usage. Please [create an issue](https://github.com/airbytehq/airbyte/issues) if you see any rate limit issues that are not automatically retried successfully.

🎉 New Source: Recharge #3466

🎉 New Source: Recharge #3466

Conversation

yevhenii-ldv commented May 18, 2021

What

How

Pre-merge Checklist

Recommended reading order

yevhenii-ldv commented May 18, 2021 • edited by github-actions bot Loading

yevhenii-ldv commented May 18, 2021 • edited by github-actions bot Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

keu May 18, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

keu left a comment

Choose a reason for hiding this comment

yevhenii-ldv commented May 19, 2021 • edited by github-actions bot Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yevhenii-ldv commented May 20, 2021 • edited by github-actions bot Loading

yevhenii-ldv commented May 20, 2021 • edited by github-actions bot Loading

yevhenii-ldv commented May 18, 2021 •

edited by github-actions bot

Loading

yevhenii-ldv commented May 18, 2021 •

edited by github-actions bot

Loading

keu May 18, 2021 •

edited

Loading

yevhenii-ldv commented May 19, 2021 •

edited by github-actions bot

Loading

yevhenii-ldv commented May 20, 2021 •

edited by github-actions bot

Loading

yevhenii-ldv commented May 20, 2021 •

edited by github-actions bot

Loading