Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add namespaceDefinitionType to standardSync #3813

Merged
merged 2 commits into from
Jun 4, 2021

Conversation

ChristopheDuong
Copy link
Contributor

@ChristopheDuong ChristopheDuong commented Jun 2, 2021

What

Implements backend part of #3481

How

Expose an option on how to use the source namespaces in the destination:

  1. Ignore it: (Always use default namespace from destination settings)
  2. Use it: (Always use namespace from source when available, fallback to default destination namespace if unavailable)

Pre-merge Checklist

  • Run integration tests
  • Publish Docker images

Recommended reading order

  1. airbyte-workers/src/main/java/io/airbyte/workers/protocols/airbyte/NamespacingMapper.java
  2. airbyte-api/src/main/openapi/config.yaml
  3. the rest

see https://docs.google.com/document/d/1qFk4YqnwxE4MCGeJ9M2scGOYej6JnDy9A0zbICP_zjI/edit# for more context

@auto-assign auto-assign bot requested review from davinchia and michel-tricot June 2, 2021 13:05
@ChristopheDuong ChristopheDuong marked this pull request as draft June 2, 2021 13:12
@ChristopheDuong ChristopheDuong changed the title Add namespaceUsageType Add namespaceDefinitionType Jun 2, 2021
@ChristopheDuong ChristopheDuong force-pushed the chris/namespace-change branch from ed5bb6c to 13b34e0 Compare June 2, 2021 13:56
@ChristopheDuong ChristopheDuong marked this pull request as ready for review June 2, 2021 14:22
@auto-assign auto-assign bot requested review from masonwheeler and sherifnada June 2, 2021 14:22
@ChristopheDuong ChristopheDuong changed the title Add namespaceDefinitionType Add namespaceDefinitionType to standardSync Jun 2, 2021

// This migration does the following:
// 1. Add fields to StandardSync.
public class MigrationV0_25_0 extends BaseMigration implements Migration {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you preempted my migration question!

@@ -2096,6 +2102,13 @@ components:
- days
- weeks
- months
NamespaceDefinitionType:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

discussed offline: we should leave a comment here explaining the default behaviour

return catalog;
}

@Override
public AirbyteMessage mapMessage(final AirbyteMessage inputMessage) {
if (inputMessage.getType() == Type.RECORD) {
final AirbyteMessage message = Jsons.clone(inputMessage);
if (namespaceDefinition != null && namespaceDefinition.equals(NamespaceDefinitionType.DESTINATION)) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

spoke offline: same comment here about default behaviour possibly linking to the api spec file.

Copy link
Contributor

@davinchia davinchia left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great!

One comment about better explanation around the default behaviour. Feel free to merge whenever.

Copy link
Contributor

@cgardens cgardens left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants