Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ElasticSearch, GCS, Keen, Kafka, Kinesis destinations : Enable DAT tests #11971

Merged
merged 25 commits into from
Apr 19, 2022
Merged

ElasticSearch, GCS, Keen, Kafka, Kinesis destinations : Enable DAT tests #11971

merged 25 commits into from
Apr 19, 2022

Conversation

DoNotPanicUA
Copy link
Contributor

@DoNotPanicUA DoNotPanicUA commented Apr 13, 2022 •
edited
Loading

What

Enable DAT tests #9443

  • Destination ElasticSearch
  • Destination GCS
  • Destination Keen
  • Destination Kafka
  • Destination Kinesis

Recommended reading order

  1. x.java
  2. y.python

@DoNotPanicUA
add Boolean, Number, DateTimeWithTZ compare methods
44a6d81
@DoNotPanicUA
Merge remote-tracking branch 'origin/master' into aleonets/add-dat-te…
656148a 
…sts-ph1
@DoNotPanicUA
Improve value comparison
46c3a1b
@DoNotPanicUA
Merge remote-tracking branch 'origin/master' into aleonets/add-dat-te…
2a4b384 
…sts-ph1
@DoNotPanicUA
Compare inherit objects element by element
cd53764
@DoNotPanicUA
Merge remote-tracking branch 'origin/aleonets/add-dat-tests-ph1' into…
4cd50de 
… aleonets/add-dat-tests-ph1
@DoNotPanicUA
Move comparison methods to a new class not to overload the test class.
3c911c9 
Basic - old implementation which provides backward compatibility of this PR.
Advanced - comparator required for destinations with enabled DAT tests.
@DoNotPanicUA
format
f28b9af
@DoNotPanicUA
Merge remote-tracking branch 'origin/master' into aleonets/add-dat-te…
af1cff8 
…sts-ph1
@DoNotPanicUA
Merge remote-tracking branch 'origin/master' into aleonets/add-dat-te…
06764b2 
…sts-ph1
@DoNotPanicUA
Move common method to ComparatorUtils. + Review update
4ddee6c
@DoNotPanicUA
format
127b4ba
@DoNotPanicUA
review
ed890d3
@DoNotPanicUA
Merge remote-tracking branch 'origin/master' into aleonets/add-dat-te…
4007837 
…sts-ph1
@DoNotPanicUA
review
f1e860b
@DoNotPanicUA
mark resolveIdentifier method as deprecated
5b59eb1
@DoNotPanicUA
enable DAT tests for destination-Elasticsearch, destination-gcs, dest…
4803514 
…ination-kafka, destination-keen, destination-kinesis
@github-actions github-actions bot added the area/connectors Connector related issues label Apr 13, 2022
@DoNotPanicUA
Copy link
Contributor Author

DoNotPanicUA commented Apr 13, 2022 •
edited by github-actions bot
Loading

/test connector=connectors/destination-elasticsearch

🕑 connectors/destination-elasticsearch https://github.com/airbytehq/airbyte/actions/runs/2161989055

@DoNotPanicUA
Copy link
Contributor Author

DoNotPanicUA commented Apr 13, 2022 •
edited by github-actions bot
Loading

/test connector=connectors/destination-gcs

🕑 connectors/destination-gcs https://github.com/airbytehq/airbyte/actions/runs/2161990039
❌ connectors/destination-gcs https://github.com/airbytehq/airbyte/actions/runs/2161990039
🐛

@DoNotPanicUA
Copy link
Contributor Author

DoNotPanicUA commented Apr 13, 2022 •
edited by github-actions bot
Loading

/test connector=connectors/destination-kafka

🕑 connectors/destination-kafka https://github.com/airbytehq/airbyte/actions/runs/2161991358
✅ connectors/destination-kafka https://github.com/airbytehq/airbyte/actions/runs/2161991358
No Python unittests run

@DoNotPanicUA
Copy link
Contributor Author

DoNotPanicUA commented Apr 13, 2022 •
edited by github-actions bot
Loading

/test connector=connectors/destination-keen

🕑 connectors/destination-keen https://github.com/airbytehq/airbyte/actions/runs/2161992338
✅ connectors/destination-keen https://github.com/airbytehq/airbyte/actions/runs/2161992338
No Python unittests run

@DoNotPanicUA
Copy link
Contributor Author

DoNotPanicUA commented Apr 13, 2022 •
edited by github-actions bot
Loading

/test connector=connectors/destination-kinesis

🕑 connectors/destination-kinesis https://github.com/airbytehq/airbyte/actions/runs/2161993791
✅ connectors/destination-kinesis https://github.com/airbytehq/airbyte/actions/runs/2161993791
No Python unittests run

@DoNotPanicUA
remove size objects validation. We iterate expected elements and comp…
c2106a4 
…are it with actualObject elements. Expected might have empty elements and they are equal missing element in the actual Object. Also, actualObject might contain additional elements which is not an exception.
@DoNotPanicUA
Merge remote-tracking branch 'origin/aleonets/add-dat-tests-ph1' into…
b3e8a31 
… aleonets/dat-EGK
@DoNotPanicUA
Copy link
Contributor Author

DoNotPanicUA commented Apr 13, 2022 •
edited by github-actions bot
Loading

/test connector=connectors/destination-gcs

🕑 connectors/destination-gcs https://github.com/airbytehq/airbyte/actions/runs/2162268090
❌ connectors/destination-gcs https://github.com/airbytehq/airbyte/actions/runs/2162268090
🐛

@DoNotPanicUA
Copy link
Contributor Author

DoNotPanicUA commented Apr 13, 2022 •
edited by github-actions bot
Loading

/test connector=connectors/destination-kafka

🕑 connectors/destination-kafka https://github.com/airbytehq/airbyte/actions/runs/2162269115
✅ connectors/destination-kafka https://github.com/airbytehq/airbyte/actions/runs/2162269115
No Python unittests run

@DoNotPanicUA
Copy link
Contributor Author

DoNotPanicUA commented Apr 13, 2022 •
edited by github-actions bot
Loading

/test connector=connectors/destination-kinesis

🕑 connectors/destination-kinesis https://github.com/airbytehq/airbyte/actions/runs/2162270248
✅ connectors/destination-kinesis https://github.com/airbytehq/airbyte/actions/runs/2162270248
No Python unittests run

@DoNotPanicUA

This comment was marked as outdated.

Sign in to view
@DoNotPanicUA DoNotPanicUA self-assigned this Apr 13, 2022
@DoNotPanicUA
Copy link
Contributor Author

DoNotPanicUA commented Apr 13, 2022 •
edited by github-actions bot
Loading

/test connector=connectors/destination-gcs

🕑 connectors/destination-gcs https://github.com/airbytehq/airbyte/actions/runs/2163783253
✅ connectors/destination-gcs https://github.com/airbytehq/airbyte/actions/runs/2163783253
No Python unittests run

@DoNotPanicUA
Copy link
Contributor Author

DoNotPanicUA commented Apr 13, 2022 •
edited
Loading

/test connector=connectors/destination-keen

🕑 connectors/destination-keen https://github.com/airbytehq/airbyte/actions/runs/2163783594
❌ connectors/destination-keen https://github.com/airbytehq/airbyte/actions/runs/2163783594
🐛 https://gradle.com/s/7pkhckv24akxe

Test is not stable and has change of random fail

@DoNotPanicUA
Copy link
Contributor Author

DoNotPanicUA commented Apr 14, 2022 •
edited by github-actions bot
Loading

/test connector=connectors/destination-keen

🕑 connectors/destination-keen https://github.com/airbytehq/airbyte/actions/runs/2166288443
✅ connectors/destination-keen https://github.com/airbytehq/airbyte/actions/runs/2166288443
No Python unittests run

@DoNotPanicUA DoNotPanicUA linked an issue Apr 14, 2022 that may be closed by this pull request
Run the new DAT(s) against all destinations and find destinations that are not compliant #9444
Closed
@DoNotPanicUA DoNotPanicUA changed the title ... destinations : Enable DAT tests ElasticSearch, GCS, Keen, Kafka, Kinesis destinations : Enable DAT tests Apr 14, 2022
@DoNotPanicUA
Copy link
Contributor Author

DoNotPanicUA commented Apr 19, 2022 •
edited by github-actions bot
Loading

/test connector=connectors/destination-keen

🕑 connectors/destination-keen https://github.com/airbytehq/airbyte/actions/runs/2189040400
✅ connectors/destination-keen https://github.com/airbytehq/airbyte/actions/runs/2189040400
No Python unittests run

@DoNotPanicUA
Copy link
Contributor Author

DoNotPanicUA commented Apr 19, 2022 •
edited by github-actions bot
Loading

/test connector=connectors/destination-gcs

🕑 connectors/destination-gcs https://github.com/airbytehq/airbyte/actions/runs/2189041058
✅ connectors/destination-gcs https://github.com/airbytehq/airbyte/actions/runs/2189041058
No Python unittests run

@DoNotPanicUA
Copy link
Contributor Author

DoNotPanicUA commented Apr 19, 2022 •
edited by github-actions bot
Loading

/test connector=connectors/destination-kinesis

🕑 connectors/destination-kinesis https://github.com/airbytehq/airbyte/actions/runs/2189041078
✅ connectors/destination-kinesis https://github.com/airbytehq/airbyte/actions/runs/2189041078
No Python unittests run

@DoNotPanicUA
Copy link
Contributor Author

DoNotPanicUA commented Apr 19, 2022 •
edited by github-actions bot
Loading

/test connector=connectors/destination-kafka

🕑 connectors/destination-kafka https://github.com/airbytehq/airbyte/actions/runs/2189041873
✅ connectors/destination-kafka https://github.com/airbytehq/airbyte/actions/runs/2189041873
No Python unittests run

@DoNotPanicUA DoNotPanicUA marked this pull request as ready for review April 19, 2022 10:27
@DoNotPanicUA DoNotPanicUA merged commit 57d059f into master Apr 19, 2022
@DoNotPanicUA DoNotPanicUA deleted the aleonets/dat-EGK branch April 19, 2022 10:56
@octavia-squidington-iii octavia-squidington-iii mentioned this pull request Apr 19, 2022
Merged
suhomud pushed a commit that referenced this pull request May 23, 2022
@DoNotPanicUA @suhomud
ElasticSearch, GCS, Keen, Kafka, Kinesis destinations : Enable DAT te…
3fadaa1 
…sts (#11971)

* add Boolean, Number, DateTimeWithTZ compare methods

* Improve value comparison

* Compare inherit objects element by element

* Move comparison methods to a new class not to overload the test class.
Basic - old implementation which provides backward compatibility of this PR.
Advanced - comparator required for destinations with enabled DAT tests.

* format

* Move common method to ComparatorUtils. + Review update

* format

* review

* review

* mark resolveIdentifier method as deprecated

* enable DAT tests for destination-Elasticsearch, destination-gcs, destination-kafka, destination-keen, destination-kinesis

* remove size objects validation. We iterate expected elements and compare it with actualObject elements. Expected might have empty elements and they are equal missing element in the actual Object. Also, actualObject might contain additional elements which is not an exception.

* fix Csv actual data read. + use proper comparator for different file types

* format
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@sashaNeshcheret sashaNeshcheret sashaNeshcheret approved these changes

@alexandr-shegeda alexandr-shegeda Awaiting requested review from alexandr-shegeda

Assignees

@DoNotPanicUA DoNotPanicUA

Labels
area/connectors Connector related issues
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Run the new DAT(s) against all destinations and find destinations that are not compliant
2 participants
@DoNotPanicUA @sashaNeshcheret