Support running Airbyte on Mac M1 Docker

[jrhizor]

Current status

• Server can build and run.
• Can sync data among some connectors.
• Temporal can fail transiently and may require restarting the server to fix it.
• Some connectors do not work.
• Huge shout-out to @h7kanna.

Todos

• Fix temporal
• Fix connectors
  • source-mssql
  • source-oracle
  • source-file: stuck at installTestReqs
  • source-mysql & destination-mysql
    • MySQL connectors can compile after switching to MariaDB
    • There are weird unit test failures for source-mysql.

Relevant PRs

• Update M1 doc and fix doc links #7205
• Update Mockito for running tests on Mac M1 #7208

[kination]

@jrhizor I'm also failing on Mac M1, with ./gradlew clean build

sorry, seems this is not related with this one...please ignore.

[jrhizor]

Docker on M1 is now generally available https://docs.docker.com/docker-for-mac/apple-silicon/

[ashishgupta-97065]

Having the same issue on M1. Would be great to have a solution

[takan55]

Tried running airbyte with Kind on my M1 chip macbook pro but it got stuck saying "cannot reach server. The server may still be starting up.". This is critical to me to stick with airbyte. Since more and more developers are switching to the M1 macs, I'd love to see the solution asap.

[tuliren]

M1 support reference: https://docs.docker.com/desktop/multi-arch/

[caiopedroso]

Hey guys! We are deciding on our primary connector, I'm advocating on airbyte, but all of our engineers have an m1 at this moment, a possible solution to this would help a lot, do we have an eta?

[tuliren]

Hey @caiopedroso, thank you for letting us know. There is an increasing number of requests for making Airbyte work on M1. We are evaluating if we can prioritize this issue any time soon. Will provide an update this week (probably this Thursday).

[hbd]

Just confirming that running Airbyte on my M1 is not working. Short version of my experience:

1. The primary issue is uptime/reliability of the services. I have seen the Airbyte services successfully run a couple times, where I could access the dashboard or use the API. However, it always crashes within a few minutes of starting.
2. The airbyte-temporal service times out while trying Waiting for default namespace to be initialized in temporal. This happens about 10% of the time I start the services:
   
3. I am also seeing airbyte-temporal fail with context deadline exceeded errors when trying to access the webapp

airbyte-temporal   | {"level":"error","ts":"2021-07-27T21:49:54.515Z","msg":"Operation failed with internal error.","service":"matching","error":"UpdateTaskQueue failed. Failed to start t
ransaction. Error: context deadline exceeded","metric-scope":38,"logging-call-at":"persistenceMetricClients.go:804","stacktrace":"go.temporal.io/server/common/log/loggerimpl.(*loggerImpl).Error\n\t/temporal/common/log/loggerimpl/logger.go:138\ngo.temporal.io/server/common/persistence.(*taskPersistenceClient).updateErrorMetric\n\t/temporal/common/persistence/persistenceMetricClients.go:804\ngo.temporal.io/server/common/persistence.(*taskPersistenceClient).UpdateTaskQueue\n\t/temporal/common/persistence/persistenceMetricClients.go:787\ngo.temporal.io/server/service/matching.(*taskQueueDB).UpdateState\n\t/temporal/service/matching/db.go:109\ngo.temporal.io/server/service/matching.(*taskReader).persistAckLevel\n\t/temporal/service/matching/ta
skReader.go:272\ngo.temporal.io/server/service/matching.(*taskReader).getTasksPump\n\t/temporal/service/matching/taskReader.go:160"}
airbyte-temporal   | {"level":"error","ts":"2021-07-27T21:49:54.519Z","msg":"Persistent store operation failure","service":"matching","component":"matching-engine","wf-task-queue-name":"3218103eeed6:d20106e3-3a07-460c-8f98-6b1d259c7bd2","wf-task-queue-type":"Workflow","store-operation":"update-task-queue","error":"UpdateTaskQueue failed. Failed to start transaction. Error: context deadline exceeded","logging-call-at":"taskReader.go:166","stacktrace":"go.temporal.io/server/common/log/loggerimpl.(*loggerImpl).Error\n\t/temporal/common/log/loggerimpl/logger.
go:138\ngo.temporal.io/server/service/matching.(*taskReader).getTasksPump\n\t/temporal/service/matching/taskReader.go:166"}
airbyte-temporal   | {"level":"error","ts":"2021-07-27T21:50:00.405Z","msg":"Operation failed with internal error.","service":"matching","error":"GetMetadata operation failed. Error: cont
ext deadline exceeded","metric-scope":51,"logging-call-at":"persistenceMetricClients.go:932","stacktrace":"go.temporal.io/server/common/log/loggerimpl.(*loggerImpl).Error\n\t/temporal/common/log/loggerimpl/logger.go:138\ngo.temporal.io/server/common/persistence.(*metadataPersistenceClient).updateErrorMetric\n\t/temporal/common/persistence/persistenceMetricClients.go:932\ngo.temporal.io/server/common/persistence.(*metadataPersistenceClient).GetMetadata\n\t/temporal/common/persistence/persistenceMetricClients.go:910\ngo.temporal.io/server/common/cache.(*namespaceCache).refreshNamespacesLocked\n\t/temporal/common/cache/namespaceCache.go:435\ngo.temporal.io/server/common/cache.(*namespaceCache).refreshNamespaces\n\t/temporal/common/cache/name
spaceCache.go:425\ngo.temporal.io/server/common/cache.(*namespaceCache).refreshLoop\n\t/temporal/common/cache/namespaceCache.go:409"}
airbyte-temporal   | {"level":"error","ts":"2021-07-27T21:50:00.405Z","msg":"Error refreshing namespace cache","service":"matching","error":"GetMetadata operation failed. Error: context deadline exceeded","logging-call-at":"namespaceCache.go:414","stacktrace":"go.temporal.io/server/common/log/loggerimpl.(*loggerImpl).Error\n\t/temporal/common/log/loggerimpl/logger.go:138\ngo.temporal.io/server/common/cache.(*namespaceCache).refreshLoop\n\t/temporal/common/cache/namespaceCache.go:414"}
airbyte-temporal   | {"level":"error","ts":"2021-07-27T21:50:09.409Z","msg":"Operation failed with internal error.","service":"matching","error":"GetMetadata operation failed. Error: cont
ext deadline exceeded","metric-scope":51,"logging-call-at":"persistenceMetricClients.go:932","stacktrace":"go.temporal.io/server/common/log/loggerimpl.(*loggerImpl).Error\n\t/temporal/common/log/loggerimpl/logger.go:138\ngo.temporal.io/server/common/persistence.(*metadataPersistenceClient).updateErrorMetric\n\t/temporal/common/persistence/persistenceMetricClients.go:932\ngo.temporal.io/server/common/persistence.(*metadataPersistenceClient).GetMetadata\n\t/temporal/common/persistence/persistenceMetricClients.go:910\ngo.temporal.io/server/common/cache.(*namespaceCache).refreshNamespacesLocked\n\t/temporal/common/cache/namespaceCache.go:435\ngo.temporal.io/server/common/cache.(*namespaceCache).refreshNamespaces\n\t/temporal/common/cache/name
spaceCache.go:425\ngo.temporal.io/server/common/cache.(*namespaceCache).refreshLoop\n\t/temporal/common/cache/namespaceCache.go:409"}
airbyte-temporal   | {"level":"error","ts":"2021-07-27T21:50:09.410Z","msg":"Error refreshing namespace cache","service":"matching","error":"GetMetadata operation failed. Error: context deadline exceeded","logging-call-at":"namespaceCache.go:414","stacktrace":"go.temporal.io/server/common/log/loggerimpl.(*loggerImpl).Error\n\t/temporal/common/log/loggerimpl/logger.go:138\ngo.temporal.io/server/common/cache.(*namespaceCache).refreshLoop\n\t/temporal/common/cache/namespaceCache.go:414"}

I am moving on to workarounds like using a deployed version and my Intel. Happy to help however possible to get this up and running as soon as possible.

[tuliren]

Hey @caiopedroso, thank you for letting us know. There is an increasing number of requests for making Airbyte work on M1. We are evaluating if we can prioritize this issue any time soon. Will provide an update this week (probably this Thursday).

Update on this. We have allocated one week for this issue in August, and will likely solve it by the end of the month.