Error on cadence-system domain when clean start

[vancexu]

Start cadence with clean db setup will sometimes having following errors:
This will cause serious ServiceBusyError, and break user experience who use docker for integration tests (such as cadence-client integration tests).

{"level":"error","ts":"2019-09-27T13:24:48.361-0700","msg":"error starting cadence-sys-history-scanner-workflow workflow","service":"cadence-worker","error":"EntityNotExistsError{Message: Domain: cadence-system}","logging-call-at":"scanner.go:164","stacktrace":"github.com/uber/cadence/common/log/loggerimpl.(*loggerImpl).Error\n\t/Users/boweixu/gocode/src/github.com/uber/cadence/common/log/loggerimpl/logger.go:133\ngithub.com/uber/cadence/service/worker/scanner.(*Scanner).startWorkflow\n\t/Users/boweixu/gocode/src/github.com/uber/cadence/service/worker/scanner/scanner.go:164\ngithub.com/uber/cadence/service/worker/scanner.(*Scanner).startWorkflowWithRetry.func1\n\t/Users/boweixu/gocode/src/github.com/uber/cadence/service/worker/scanner/scanner.go:150\ngithub.com/uber/cadence/common/backoff.Retry\n\t/Users/boweixu/gocode/src/github.com/uber/cadence/common/backoff/retry.go:99\ngithub.com/uber/cadence/service/worker/scanner.(*Scanner).startWorkflowWithRetry\n\t/Users/boweixu/gocode/src/github.com/uber/cadence/service/worker/scanner/scanner.go:149"}
{"level":"error","ts":"2019-09-27T13:24:54.470-0700","msg":"Error refreshing domain cache","service":"cadence-frontend","error":"ServiceBusyError{Message: Persistence Max QPS Reached.}","logging-call-at":"domainCache.go:359","stacktrace":"github.com/uber/cadence/common/log/loggerimpl.(*loggerImpl).Error\n\t/Users/boweixu/gocode/src/github.com/uber/cadence/common/log/loggerimpl/logger.go:133\ngithub.com/uber/cadence/common/cache.(*domainCache).refreshLoop\n\t/Users/boweixu/gocode/src/github.com/uber/cadence/common/cache/domainCache.go:359"}
{"level":"error","ts":"2019-09-27T13:24:54.472-0700","msg":"Error refreshing domain cache","service":"cadence-frontend","error":"ServiceBusyError{Message: Persistence Max QPS Reached.}","logging-call-at":"domainCache.go:359","stacktrace":"github.com/uber/cadence/common/log/loggerimpl.(*loggerImpl).Error\n\t/Users/boweixu/gocode/src/github.com/uber/cadence/common/log/loggerimpl/logger.go:133\ngithub.com/uber/cadence/common/cache.(*domainCache).refreshLoop\n\t/Users/boweixu/gocode/src/github.com/uber/cadence/common/cache/domainCache.go:359"}
{"level":"info","ts":"2019-09-27T13:24:54.521-0700","msg":"Get dynamic config","name":"history.timerProcessorCompleteTimerFailureRetryCount","value":"10","default-value":"10","logging-call-at":"config.go:58"}
{"level":"info","ts":"2019-09-27T13:24:54.521-0700","msg":"Get dynamic config","name":"history.transferProcessorCompleteTransferFailureRetryCount","value":"10","default-value":"10","logging-call-at":"config.go:58"}
{"level":"error","ts":"2019-09-27T13:25:04.475-0700","msg":"Error refreshing domain cache","service":"cadence-frontend","error":"ServiceBusyError{Message: Persistence Max QPS Reached.}","logging-call-at":"domainCache.go:359","stacktrace":"github.com/uber/cadence/common/log/loggerimpl.(*loggerImpl).Error\n\t/Users/boweixu/gocode/src/github.com/uber/cadence/common/log/loggerimpl/logger.go:133\ngithub.com/uber/cadence/common/cache.(*domainCache).refreshLoop\n\t/Users/boweixu/gocode/src/github.com/uber/cadence/common/cache/domainCache.go:359"}
{"level":"error","ts":"2019-09-27T13:25:14.475-0700","msg":"Error refreshing domain cache","service":"cadence-frontend","error":"ServiceBusyError{Message: Persistence Max QPS Reached.}","logging-call-at":"domainCache.go:359","stacktrace":"github.com/uber/cadence/common/log/loggerimpl.(*loggerImpl).Error\n\t/Users/boweixu/gocode/src/github.com/uber/cadence/common/log/loggerimpl/logger.go:133\ngithub.com/uber/cadence/common/cache.(*domainCache).refreshLoop\n\t/Users/boweixu/gocode/src/github.com/uber/cadence/common/cache/domainCache.go:359"}
{"level":"error","ts":"2019-09-27T13:25:24.475-0700","msg":"Error refreshing domain cache","service":"cadence-frontend","error":"ServiceBusyError{Message: Persistence Max QPS Reached.}","logging-call-at":"domainCache.go:359","stacktrace":"github.com/uber/cadence/common/log/loggerimpl.(*loggerImpl).Error\n\t/Users/boweixu/gocode/src/github.com/uber/cadence/common/log/loggerimpl/logger.go:133\ngithub.com/uber/cadence/common/cache.(*domainCache).refreshLoop\n\t/Users/boweixu/gocode/src/github.com/uber/cadence/common/cache/domainCache.go:359"}
{"level":"error","ts":"2019-09-27T13:25:34.475-0700","msg":"Error refreshing domain cache","service":"cadence-frontend","error":"ServiceBusyError{Message: Persistence Max QPS Reached.}","logging-call-at":"domainCache.go:359","stacktrace":"github.com/uber/cadence/common/log/loggerimpl.(*loggerImpl).Error\n\t/Users/boweixu/gocode/src/github.com/uber/cadence/common/log/loggerimpl/logger.go:133\ngithub.com/uber/cadence/common/cache.(*domainCache).refreshLoop\n\t/Users/boweixu/gocode/src/github.com/uber/cadence/common/cache/domainCache.go:359"}
{"level":"error","ts":"2019-09-27T13:25:44.475-0700","msg":"Error refreshing domain cache","service":"cadence-frontend","error":"ServiceBusyError{Message: Persistence Max QPS Reached.}","logging-call-at":"domainCache.go:359","stacktrace":"github.com/uber/cadence/common/log/loggerimpl.(*loggerImpl).Error\n\t/Users/boweixu/gocode/src/github.com/uber/cadence/common/log/loggerimpl/logger.go:133\ngithub.com/uber/cadence/common/cache.(*domainCache).refreshLoop\n\t/Users/boweixu/gocode/src/github.com/uber/cadence/common/cache/domainCache.go:359"}

[mfateev]

I'm not sure if it is related. But this is what I've got running docker-compose for the first time:

cadence_1      | {"level":"info","ts":"2019-10-05T03:31:44.579Z","msg":"cadence-system domain does not exist, attempting to register domain","service":"cadence-worker","logging-call-at":"service.go:339"}
cadence_1      | {"level":"error","ts":"2019-10-05T03:31:44.590Z","msg":"Operation failed with internal error.","service":"cadence-worker","error":"InternalServiceError{Message: CreateDomain operation failed because of uuid collision.}","metric-scope":31,"logging-call-at":"persistenceMetricClients.go:835","stacktrace":"github.com/uber/cadence/common/log/loggerimpl.(*loggerImpl).Error\n\t/cadence/common/log/loggerimpl/logger.go:133\ngithub.com/uber/cadence/common/persistence.(*metadataPersistenceClient).updateErrorMetric\n\t/cadence/common/persistence/persistenceMetricClients.go:835\ngithub.com/uber/cadence/common/persistence.(*metadataPersistenceClient).CreateDomain\n\t/cadence/common/persistence/persistenceMetricClients.go:729\ngithub.com/uber/cadence/service/worker.(*Service).registerSystemDomain\n\t/cadence/service/worker/service.go:374\ngithub.com/uber/cadence/service/worker.(*Service).ensureSystemDomainExists\n\t/cadence/service/worker/service.go:340\ngithub.com/uber/cadence/service/worker.(*Service).Start\n\t/cadence/service/worker/service.go:163\nmain.execute\n\t/cadence/cmd/server/server.go:225"}
cadence_1      | {"level":"fatal","ts":"2019-10-05T03:31:44.590Z","msg":"failed to register system domain","service":"cadence-worker","error":"InternalServiceError{Message: CreateDomain operation failed because of uuid collision.}","logging-call-at":"service.go:395","stacktrace":"github.com/uber/cadence/common/log/loggerimpl.(*loggerImpl).Fatal\n\t/cadence/common/log/loggerimpl/logger.go:139\ngithub.com/uber/cadence/service/worker.(*Service).registerSystemDomain\n\t/cadence/service/worker/service.go:395\ngithub.com/uber/cadence/service/worker.(*Service).ensureSystemDomainExists\n\t/cadence/service/worker/service.go:340\ngithub.com/uber/cadence/service/worker.(*Service).Start\n\t/cadence/service/worker/service.go:163\nmain.execute\n\t/cadence/cmd/server/server.go:225"}
cadence_1      | 2019/10/05 03:31:44 Command exited with error: exit status 1
docker_cadence_1 exited with code 1

[BrendanBall]

I'm busy evaluating cadence and was happy to see docker-compose files, but they don't actually work :/ . fyi I got similar errors with mysql.

[vancexu]

"CreateDomain operation failed because of uuid collision" is caused by a recent change. It was fixed by #2639.