Source Twilio: add conversations stream (airbytehq#24320)

* add conversations stream * add conversations schema * add docs * fix connector version * fix tests and add conversation to streams * fix schema and abnormal state * remove incremental from conversations * fix source * fix doc and remove date_field * add eof * add path for conversations * auto-bump connector version --------- Co-authored-by: Haithem Souala <haithem.souala@woopit.fr> Co-authored-by: Augustin <augustin@airbyte.io> Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
natalia-miinto · Mar 21, 2023 · ca4ca68 · ca4ca68
1 parent 100cc60
commit ca4ca68
Show file tree

Hide file tree

Showing 12 changed files with 77 additions and 5 deletions.
diff --git a/airbyte-config/init/src/main/resources/seed/source_definitions.yaml b/airbyte-config/init/src/main/resources/seed/source_definitions.yaml
@@ -2106,7 +2106,7 @@
 - name: Twilio
   sourceDefinitionId: b9dc6155-672e-42ea-b10d-9f1f1fb95ab1
   dockerRepository: airbyte/source-twilio
-  dockerImageTag: 0.1.16
+  dockerImageTag: 0.2.0
   documentationUrl: https://docs.airbyte.com/integrations/sources/twilio
   icon: twilio.svg
   sourceType: api

diff --git a/airbyte-config/init/src/main/resources/seed/source_specs.yaml b/airbyte-config/init/src/main/resources/seed/source_specs.yaml
@@ -16029,7 +16029,7 @@
     supportsNormalization: false
     supportsDBT: false
     supported_destination_sync_modes: []
-- dockerImage: "airbyte/source-twilio:0.1.16"
+- dockerImage: "airbyte/source-twilio:0.2.0"
   spec:
     documentationUrl: "https://docs.airbyte.com/integrations/sources/twilio"
     connectionSpecification:

diff --git a/airbyte-integrations/connectors/source-twilio/Dockerfile b/airbyte-integrations/connectors/source-twilio/Dockerfile
@@ -12,5 +12,5 @@ COPY main.py ./
 ENV AIRBYTE_ENTRYPOINT "python /airbyte/integration_code/main.py"
 ENTRYPOINT ["python", "/airbyte/integration_code/main.py"]
 
-LABEL io.airbyte.version=0.1.16
+LABEL io.airbyte.version=0.2.0
 LABEL io.airbyte.name=airbyte/source-twilio
diff --git a/airbyte-integrations/connectors/source-twilio/acceptance-test-config.yml b/airbyte-integrations/connectors/source-twilio/acceptance-test-config.yml
@@ -24,7 +24,6 @@ acceptance_tests:
           # reference: https://github.com/airbytehq/airbyte/pull/20430
           - name: "accounts"
             bypass_reason: "expected records could not be provided without field bypassing, skipping for now. Read #TODO above."
-            #
           - name: "alerts"
             bypass_reason: "alerts are transient - they emerge and disappear from time to time"
           - name: "dependent_phone_numbers"

diff --git a/...yte-integrations/connectors/source-twilio/integration_tests/constant_records_catalog.json b/...yte-integrations/connectors/source-twilio/integration_tests/constant_records_catalog.json
@@ -155,6 +155,15 @@
       "cursor_field": ["date_created"],
       "sync_mode": "incremental",
       "destination_sync_mode": "append"
+    },
+    {
+      "stream": {
+        "name": "conversations",
+        "json_schema": {},
+        "supported_sync_modes": ["full_refresh"]
+      },
+      "sync_mode": "full_refresh",
+      "destination_sync_mode": "overwrite"
     }
   ]
 }
diff --git a/airbyte-integrations/connectors/source-twilio/integration_tests/expected_records.jsonl b/airbyte-integrations/connectors/source-twilio/integration_tests/expected_records.jsonl
@@ -522,3 +522,4 @@
 {"stream": "usage_triggers", "data": {"sid": "UT33bd2bf238d94863a609133da897d676", "account_sid": "ACdade166c12e160e9ed0a6088226718fb", "date_created": "2020-11-25T10:02:19Z", "date_updated": "2020-11-25T10:02:19Z", "date_fired": null, "friendly_name": null, "usage_category": "sms", "trigger_by": "usage", "recurring": "", "trigger_value": 1000.0, "current_value": 146.0, "callback_url": "http://www.example.com/", "callback_method": "POST", "usage_record_uri": "/2010-04-01/Accounts/ACdade166c12e160e9ed0a6088226718fb/Usage/Records.json?Category=sms", "uri": "/2010-04-01/Accounts/ACdade166c12e160e9ed0a6088226718fb/Usage/Triggers/UT33bd2bf238d94863a609133da897d676.json", "api_version": "2010-04-01"}, "emitted_at": 1655893322691}
 {"stream": "usage_triggers", "data": {"sid": "UT3c3c157dcaf347829d5a0f75e97b572e", "account_sid": "ACdade166c12e160e9ed0a6088226718fb", "date_created": "2020-11-25T10:02:34Z", "date_updated": "2020-11-25T10:02:34Z", "date_fired": null, "friendly_name": null, "usage_category": "sms", "trigger_by": "usage", "recurring": "", "trigger_value": 999.0, "current_value": 146.0, "callback_url": "http://www.example.com/", "callback_method": "POST", "usage_record_uri": "/2010-04-01/Accounts/ACdade166c12e160e9ed0a6088226718fb/Usage/Records.json?Category=sms", "uri": "/2010-04-01/Accounts/ACdade166c12e160e9ed0a6088226718fb/Usage/Triggers/UT3c3c157dcaf347829d5a0f75e97b572e.json", "api_version": "2010-04-01"}, "emitted_at": 1655893322698}
 {"stream": "usage_triggers", "data": {"sid": "UT7170996eff504647ac9f215222ee296f", "account_sid": "ACdade166c12e160e9ed0a6088226718fb", "date_created": "2020-11-25T10:02:41Z", "date_updated": "2020-11-25T10:02:41Z", "date_fired": null, "friendly_name": null, "usage_category": "sms", "trigger_by": "usage", "recurring": "", "trigger_value": 943.0, "current_value": 146.0, "callback_url": "http://www.example.com/", "callback_method": "POST", "usage_record_uri": "/2010-04-01/Accounts/ACdade166c12e160e9ed0a6088226718fb/Usage/Records.json?Category=sms", "uri": "/2010-04-01/Accounts/ACdade166c12e160e9ed0a6088226718fb/Usage/Triggers/UT7170996eff504647ac9f215222ee296f.json", "api_version": "2010-04-01"}, "emitted_at": 1655893322701}
+{"stream": "conversations", "data": {"unique_name": null, "date_updated": "2023-03-21T13:39:44Z", "friendly_name": "Friendly Conversation", "timers": {}, "account_sid": "ACdade166c12e160e9ed0a6088226718fb", "url": "https://conversations.twilio.com/v1/Conversations/CH0ed7b4c3498e455a96fa09fcccee720e", "state": "active", "date_created": "2023-03-21T13:39:44Z", "messaging_service_sid": "MGfdf707ca9a7e03496ad79dc64e5e543e", "sid": "CH0ed7b4c3498e455a96fa09fcccee720e", "attributes": "{}", "bindings": null, "chat_service_sid": "IS5fcc074f7ead44c99a0a24a374a7e19f", "links": {"participants": "https://conversations.twilio.com/v1/Conversations/CH0ed7b4c3498e455a96fa09fcccee720e/Participants", "messages": "https://conversations.twilio.com/v1/Conversations/CH0ed7b4c3498e455a96fa09fcccee720e/Messages", "webhooks": "https://conversations.twilio.com/v1/Conversations/CH0ed7b4c3498e455a96fa09fcccee720e/Webhooks"}}, "emitted_at": 1679415128580}
diff --git a/airbyte-integrations/connectors/source-twilio/source_twilio/schemas/conversations.json b/airbyte-integrations/connectors/source-twilio/source_twilio/schemas/conversations.json
@@ -0,0 +1,47 @@
+{
+  "properties": {
+    "sid": {
+      "type": ["null", "string"]
+    },
+    "account_sid": {
+      "type": ["null", "string"]
+    },
+    "chat_service_sid": {
+      "type": ["null", "string"]
+    },
+    "messaging_service_sid": {
+      "type": ["null", "string"]
+    },
+    "friendly_name": {
+      "type": ["null", "string"]
+    },
+    "unique_name": {
+      "type": ["null", "string"]
+    },
+    "attributes": {
+      "type": ["null", "string"]
+    },
+    "date_created": {
+      "format": "date-time",
+      "type": ["null", "string"]
+    },
+    "date_updated": {
+      "format": "date-time",
+      "type": ["null", "string"]
+    },
+    "state": {
+      "type": ["null", "string"]
+    },
+    "timers": {
+      "type": ["null", "object"]
+    },
+    "bindings": {
+      "type": ["null", "object"]
+    },
+    "url": {
+      "type": ["null", "string"]
+    }
+  },
+  "type": "object",
+  "additionalProperties": true
+}
diff --git a/airbyte-integrations/connectors/source-twilio/source_twilio/source.py b/airbyte-integrations/connectors/source-twilio/source_twilio/source.py
@@ -23,6 +23,7 @@
     Calls,
     ConferenceParticipants,
     Conferences,
+    Conversations,
     DependentPhoneNumbers,
     IncomingPhoneNumbers,
     Keys,
@@ -92,6 +93,7 @@ def streams(self, config: Mapping[str, Any]) -> List[Stream]:
             Calls(**incremental_stream_kwargs),
             ConferenceParticipants(**full_refresh_stream_kwargs),
             Conferences(**incremental_stream_kwargs),
+            Conversations(**full_refresh_stream_kwargs),
             DependentPhoneNumbers(**full_refresh_stream_kwargs),
             IncomingPhoneNumbers(**full_refresh_stream_kwargs),
             Keys(**full_refresh_stream_kwargs),

diff --git a/airbyte-integrations/connectors/source-twilio/source_twilio/streams.py b/airbyte-integrations/connectors/source-twilio/source_twilio/streams.py
@@ -22,6 +22,7 @@
 TWILIO_API_URL_BASE = "https://api.twilio.com"
 TWILIO_API_URL_BASE_VERSIONED = f"{TWILIO_API_URL_BASE}/2010-04-01/"
 TWILIO_MONITOR_URL_BASE = "https://monitor.twilio.com/v1/"
+TWILIO_CONVERSATIONS_URL_BASE = "https://conversations.twilio.com/v1/"
 
 
 class TwilioStream(HttpStream, ABC):
@@ -502,3 +503,12 @@ class Alerts(IncrementalTwilioStream):
 
     def path(self, **kwargs):
         return self.name.title()
+
+
+class Conversations(TwilioStream):
+    """https://www.twilio.com/docs/conversations/api/conversation-resource#read-multiple-conversation-resources"""
+
+    url_base = TWILIO_CONVERSATIONS_URL_BASE
+
+    def path(self, **kwargs):
+        return self.name.title()
diff --git a/airbyte-integrations/connectors/source-twilio/unit_tests/test_source.py b/airbyte-integrations/connectors/source-twilio/unit_tests/test_source.py
@@ -19,6 +19,7 @@
     Calls,
     ConferenceParticipants,
     Conferences,
+    Conversations,
     DependentPhoneNumbers,
     IncomingPhoneNumbers,
     Keys,
@@ -96,6 +97,7 @@ def test_check_connection_handles_exceptions(mocker, config, exception, expected
         (Transcriptions),
         (UsageRecords),
         (UsageTriggers),
+        (Conversations),
     ],
 )
 def test_streams(stream_cls, config):

diff --git a/connectors.md b/connectors.md
@@ -242,7 +242,7 @@
 | **Toggl** | <img alt="Toggl icon" src="https://raw.githubusercontent.com/airbytehq/airbyte/master/airbyte-config/init/src/main/resources/icons/toggl.svg" height="30" height="30"/> | Source | airbyte/source-toggl:0.1.0 | alpha | [link](https://docs.airbyte.com/integrations/sources/toggl) | [code](https://github.com/airbytehq/airbyte/tree/master/airbyte-integrations/connectors/source-toggl) | <small>`7e7c844f-2300-4342-b7d3-6dd7992593cd`</small> |
 | **Trello** | <img alt="Trello icon" src="https://raw.githubusercontent.com/airbytehq/airbyte/master/airbyte-config/init/src/main/resources/icons/trello.svg" height="30" height="30"/> | Source | airbyte/source-trello:0.3.1 | alpha | [link](https://docs.airbyte.com/integrations/sources/trello) | [code](https://github.com/airbytehq/airbyte/tree/master/airbyte-integrations/connectors/source-trello) | <small>`8da67652-004c-11ec-9a03-0242ac130003`</small> |
 | **TrustPilot** | <img alt="TrustPilot icon" src="https://raw.githubusercontent.com/airbytehq/airbyte/master/airbyte-config/init/src/main/resources/icons/trustpilot.svg" height="30" height="30"/> | Source | airbyte/source-trustpilot:0.1.0 | alpha | [link](https://docs.airbyte.com/integrations/sources/trustpilot) | [code](https://github.com/airbytehq/airbyte/tree/master/airbyte-integrations/connectors/source-trustpilot) | <small>`d7e23ea6-d741-4314-9209-a33c91a2e945`</small> |
-| **Twilio** | <img alt="Twilio icon" src="https://raw.githubusercontent.com/airbytehq/airbyte/master/airbyte-config/init/src/main/resources/icons/twilio.svg" height="30" height="30"/> | Source | airbyte/source-twilio:0.1.16 | generally_available | [link](https://docs.airbyte.com/integrations/sources/twilio) | [code](https://github.com/airbytehq/airbyte/tree/master/airbyte-integrations/connectors/source-twilio) | <small>`b9dc6155-672e-42ea-b10d-9f1f1fb95ab1`</small> |
+| **Twilio** | <img alt="Twilio icon" src="https://raw.githubusercontent.com/airbytehq/airbyte/master/airbyte-config/init/src/main/resources/icons/twilio.svg" height="30" height="30"/> | Source | airbyte/source-twilio:0.2.0 | generally_available | [link](https://docs.airbyte.com/integrations/sources/twilio) | [code](https://github.com/airbytehq/airbyte/tree/master/airbyte-integrations/connectors/source-twilio) | <small>`b9dc6155-672e-42ea-b10d-9f1f1fb95ab1`</small> |
 | **Twilio Taskrouter** | <img alt="Twilio Taskrouter icon" src="https://raw.githubusercontent.com/airbytehq/airbyte/master/airbyte-config/init/src/main/resources/icons/twilio.svg" height="30" height="30"/> | Source | airbyte/source-twilio-taskrouter:0.1.0 | alpha | [link](https://docs.airbyte.com/integrations/sources/twilio-taskrouter) | [code](https://github.com/airbytehq/airbyte/tree/master/airbyte-integrations/connectors/source-twilio-taskrouter) | <small>`2446953b-b794-429b-a9b3-c821ba992a48`</small> |
 | **Twitter** | <img alt="Twitter icon" src="https://raw.githubusercontent.com/airbytehq/airbyte/master/airbyte-config/init/src/main/resources/icons/twitter.svg" height="30" height="30"/> | Source | airbyte/source-twitter:0.1.2 | beta | [link](https://docs.airbyte.com/integrations/sources/twitter) | [code](https://github.com/airbytehq/airbyte/tree/master/airbyte-integrations/connectors/source-twitter) | <small>`d7fd4f40-5e5a-4b8b-918f-a73077f8c131`</small> |
 | **Tyntec SMS** | <img alt="Tyntec SMS icon" src="https://raw.githubusercontent.com/airbytehq/airbyte/master/airbyte-config/init/src/main/resources/icons/tyntec.svg" height="30" height="30"/> | Source | airbyte/source-tyntec-sms:0.1.0 | alpha | [link](https://docs.airbyte.com/integrations/sources/tyntec-sms) | [code](https://github.com/airbytehq/airbyte/tree/master/airbyte-integrations/connectors/source-tyntec-sms) | <small>`3c0c3cd1-b3e0-464a-9090-d3ceb5f92346`</small> |

diff --git a/docs/integrations/sources/twilio.md b/docs/integrations/sources/twilio.md
@@ -62,6 +62,7 @@ The Twilio source connector supports the following [sync modes](https://docs.air
 * [Calls](https://www.twilio.com/docs/voice/api/call-resource#create-a-call-resource) \(Incremental\)
 * [Conference Participants](https://www.twilio.com/docs/voice/api/conference-participant-resource#read-multiple-participant-resources) \(Incremental\)
 * [Conferences](https://www.twilio.com/docs/voice/api/conference-resource#read-multiple-conference-resources) \(Incremental\)
+* [Conversations](https://www.twilio.com/docs/conversations/api/conversation-resource#read-multiple-conversation-resources) 
 * [Dependent Phone Numbers](https://www.twilio.com/docs/usage/api/address?code-sample=code-list-dependent-pns-subresources&code-language=curl&code-sdk-version=json#instance-subresources) \(Incremental\)
 * [Incoming Phone Numbers](https://www.twilio.com/docs/phone-numbers/api/incomingphonenumber-resource#read-multiple-incomingphonenumber-resources) \(Incremental\)
 * [Keys](https://www.twilio.com/docs/usage/api/keys#read-a-key-resource)
@@ -83,6 +84,7 @@ For more information, see [the Twilio docs for rate limitations](https://support
 
 | Version | Date       | Pull Request                                             | Subject                                                                                                 |
 |:--------|:-----------|:---------------------------------------------------------|:--------------------------------------------------------------------------------------------------------|
+| 0.2.0  | 2023-03-16 | [24114](https://github.com/airbytehq/airbyte/pull/24114) | Add `Conversations` stream 
 | 0.1.16  | 2023-02-10 | [22825](https://github.com/airbytehq/airbyte/pull/22825) | Specified date formatting in specification                                                     |
 | 0.1.15  | 2023-01-27 | [22025](https://github.com/airbytehq/airbyte/pull/22025) | Set `AvailabilityStrategy` for streams explicitly to `None`                                                     |
 | 0.1.14  | 2022-11-16 | [19479](https://github.com/airbytehq/airbyte/pull/19479) | Fix date range slicing                                                                                  |