🎉 Source Slack: Implement OAuth support with OAuth authenticator #6570
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,12 @@ | ||
| { | ||
| "credentials": { | ||
| "option_title": "Default OAuth2.0 authorization", | ||
| "client_id": "fake-client-id", | ||
| "client_secret": "fake-client-secret", | ||
| "refresh_token": "fake-refresh-token", | ||
| "access_token": "fake-access-token" | ||
| }, | ||
| "start_date": "2021-07-22T20:00:00Z", | ||
| "lookback_window": 2, | ||
| "join_channels": true | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -13,9 +13,8 @@ | |
| from airbyte_cdk.sources import AbstractSource | ||
| from airbyte_cdk.sources.streams import Stream | ||
| from airbyte_cdk.sources.streams.http import HttpStream | ||
| from airbyte_cdk.sources.streams.http.requests_native_auth import Oauth2Authenticator, TokenAuthenticator | ||
| from pendulum import DateTime, Period | ||
|
|
||
|
|
||
| class SlackStream(HttpStream, ABC): | ||
|
|
@@ -107,7 +106,7 @@ def parse_response(self, response: requests.Response, stream_slice: Mapping[str, | |
| yield {"member_id": member_id, "channel_id": stream_slice["channel_id"]} | ||
|
|
||
| def stream_slices(self, **kwargs) -> Iterable[Optional[Mapping[str, any]]]: | ||
| channels_stream = Channels(authenticator=self._session.auth) | ||
| for channel_record in channels_stream.read_records(sync_mode=SyncMode.full_refresh): | ||
| yield {"channel_id": channel_record["id"]} | ||
|
|
||
|
|
@@ -188,7 +187,7 @@ def read_records(self, stream_slice: Optional[Mapping[str, Any]] = None, **kwarg | |
| yield from super().read_records(stream_slice=stream_slice, **kwargs) | ||
| else: | ||
| # if channel is not provided, then get channels and read accordingly | ||
| channels = Channels(authenticator=self._session.auth) | ||
| for channel_record in channels.read_records(sync_mode=SyncMode.full_refresh): | ||
| stream_slice["channel"] = channel_record["id"] | ||
| yield from super().read_records(stream_slice=stream_slice, **kwargs) | ||
|
|
@@ -221,7 +220,7 @@ def stream_slices(self, stream_state: Mapping[str, Any] = None, **kwargs) -> Ite | |
| """ | ||
|
|
||
| stream_state = stream_state or {} | ||
| channels_stream = Channels(authenticator=self._session.auth) | ||
|
|
||
| if self.cursor_field in stream_state: | ||
| # Since new messages can be posted to threads continuously after the parent message has been posted, we get messages from the latest date | ||
|
|
@@ -232,7 +231,7 @@ def stream_slices(self, stream_state: Mapping[str, Any] = None, **kwargs) -> Ite | |
| # If there is no state i.e: this is the first sync then there is no use for lookback, just get messages from the default start date | ||
| messages_start_date = pendulum.from_timestamp(self._start_ts) | ||
|
|
||
| messages_stream = ChannelMessages(authenticator=self._session.auth, default_start_date=messages_start_date) | ||
|
|
||
| for message_chunk in messages_stream.stream_slices(stream_state={self.cursor_field: messages_start_date.timestamp()}): | ||
| self.logger.info(f"Syncing replies {message_chunk}") | ||
|
|
@@ -276,7 +275,7 @@ def path(self, **kwargs) -> str: | |
| return "conversations.join" | ||
|
|
||
| def stream_slices(self, **kwargs) -> Iterable[Optional[Mapping[str, any]]]: | ||
| channels_stream = Channels(authenticator=self._session.auth) | ||
| for channel in channels_stream.read_records(sync_mode=SyncMode.full_refresh): | ||
| yield {"channel": channel["id"], "channel_name": channel["name"]} | ||
|
|
||
|
|
@@ -285,16 +284,41 @@ def request_body_json(self, stream_slice: Mapping = None, **kwargs) -> Optional[ | |
|
|
||
|
|
||
| class SourceSlack(AbstractSource): | ||
| def _get_authenticator(self, config: Mapping[str, Any]): | ||
| # Added to maintain backward compatibility with previous versions | ||
| if "api_token" in config: | ||
| return TokenAuthenticator(config["api_token"]) | ||
|
|
||
| credentials = config.get("credentials") | ||
| credentials_title = credentials.get("option_title") | ||
| if credentials_title == "Default OAuth2.0 authorization": | ||
yevhenii-ldv marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| # We can get `refresh_token` only if the token rotation function is enabled for the Slack Oauth Application. | ||
| # If it is disabled, then we use the generated `access_token`, which acts without expiration. | ||
| # https://api.slack.com/authentication/rotation | ||
| if credentials.get("refresh_token", "").strip(): | ||
|
There was a problem hiding this comment. we should use the option name, as that is the only thing guaranteed to be uniquely present There was a problem hiding this comment. There is a fork here that needs to be handled. If Token rotation is enabled for the application, then in the response to the |
||
| return Oauth2Authenticator( | ||
| token_refresh_endpoint="https://slack.com/api/oauth.v2.access", | ||
| client_id=credentials["client_id"], | ||
| client_secret=credentials["client_secret"], | ||
| refresh_token=credentials["refresh_token"], | ||
| ) | ||
| return TokenAuthenticator(credentials["access_token"]) | ||
| elif credentials_title == "API Token Credentials": | ||
| return TokenAuthenticator(credentials["api_token"]) | ||
| else: | ||
| raise Exception(f"No supported option_title: {credentials_title} specified. See spec.json for references") | ||
|
|
||
| def check_connection(self, logger: AirbyteLogger, config: Mapping[str, Any]) -> Tuple[bool, Optional[Any]]: | ||
| try: | ||
| authenticator = self._get_authenticator(config) | ||
| users_stream = Users(authenticator=authenticator) | ||
| next(users_stream.read_records(SyncMode.full_refresh)) | ||
| return True, None | ||
| except Exception: | ||
| return False, "There are no users in the given Slack instance or your token is incorrect" | ||
|
|
||
| def streams(self, config: Mapping[str, Any]) -> List[Stream]: | ||
| authenticator = self._get_authenticator(config) | ||
| default_start_date = pendulum.parse(config["start_date"]) | ||
| threads_lookback_window = pendulum.Duration(days=config["lookback_window"]) | ||
|
|
||
|
|
||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -4,8 +4,6 @@ | |
|
|
||
| This source can sync data for the [Slack API](https://api.slack.com/). It supports both Full Refresh and Incremental syncs. You can choose if this connector will copy only the new or updated data, or all rows in the tables and columns you set up for replication, every time a sync is run. | ||
|
|
||
| ### Output schema | ||
|
|
||
| This Source is capable of syncing the following core Streams: | ||
|
|
@@ -46,6 +44,16 @@ The Slack connector should not run into Slack API limitations under normal usage | |
|
|
||
| ### Requirements | ||
|
|
||
| #### Slack connector can be connected using two types of authentication: OAuth2.0 or API Token | ||
|
|
||
| #### Using OAuth2.0 authenticator | ||
| * Client ID - issued when you created your app | ||
| * Client Secret - issued when you created your app | ||
| * Refresh Token - a special kind of token used to obtain a renewed access token | ||
|
|
||
yevhenii-ldv marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| You can get more detailed information about this type of authentication by reading [Slack's documentation about OAuth2.0](https://api.slack.com/authentication/oauth-v2). | ||
|
|
||
| #### Using API Token | ||
| * Slack API Token | ||
|
|
||
| ### Setup guide | ||
|
|
@@ -101,6 +109,7 @@ We recommend creating a restricted, read-only key specifically for Airbyte acces | |
|
|
||
| | Version | Date | Pull Request | Subject | | ||
| | :------ | :-------- | :----- | :------ | | ||
| | 0.1.12 | 2021-10-07 | [6570](https://github.com/airbytehq/airbyte/pull/6570) | Implement OAuth support with OAuth authenticator | | ||
| | 0.1.11 | 2021-08-27 | [5830](https://github.com/airbytehq/airbyte/pull/5830) | Fixed sync operations hang forever issue | | ||
| | 0.1.10 | 2021-08-27 | [5697](https://github.com/airbytehq/airbyte/pull/5697) | Fixed max retries issue | | ||
| | 0.1.9 | 2021-07-20 | [4860](https://github.com/airbytehq/airbyte/pull/4860) | Fixed reading threads issue | | ||
|
|
||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -118,6 +118,7 @@ write_standard_creds source-sendgrid "$SENDGRID_INTEGRATION_TEST_CREDS" | |
| write_standard_creds source-shopify "$SHOPIFY_INTEGRATION_TEST_CREDS" | ||
| write_standard_creds source-shortio "$SOURCE_SHORTIO_TEST_CREDS" | ||
| write_standard_creds source-slack "$SOURCE_SLACK_TEST_CREDS" | ||
| write_standard_creds source-slack "$SOURCE_SLACK_OAUTH_TEST_CREDS" "config_oauth.json" | ||
| write_standard_creds source-smartsheets "$SMARTSHEETS_TEST_CREDS" | ||
| write_standard_creds source-snapchat-marketing "$SOURCE_SNAPCHAT_MARKETING_CREDS" | ||
| write_standard_creds source-snowflake "$SNOWFLAKE_INTEGRATION_TEST_CREDS" "config.json" | ||
|
|
||
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this should be backwards compatible with the previous config format to prevent a breaking change for current users
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Added backward compatibility, but for this it was necessary to remove
credentialsfromrequired.@avida described this issue in more detail in this comment.