Fixed deployment of Elasticsearch via its operator #234
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
This change is |
pavolloffay
approved these changes
Feb 26, 2019
jpkrohling
force-pushed
the
Fix-Elasticsearch-Objects
branch
from
2717099
to
5087bb9
Compare
|
Running locally with this PR causes this when deploying
|
jpkrohling
force-pushed
the
Fix-Elasticsearch-Objects
branch
from
5087bb9
to
3a58472
Compare
Codecov Report
@@ Coverage Diff @@
## master #234 +/- ##
=========================================
- Coverage 90.48% 90.19% -0.3%
=========================================
Files 59 61 +2
Lines 2680 2753 +73
=========================================
+ Hits 2425 2483 +58
- Misses 164 172 +8
- Partials 91 98 +7
Continue to review full report at Codecov.
|
pavolloffay
requested changes
Feb 27, 2019
There was a problem hiding this comment.
I have tested this now.
Once I got:
ploffay ~/projects/golang/src/github.com/jaegertracing/jaeger-operator PR234 make run WATCH_NAMESPACE=myproject 10:27
customresourcedefinition.apiextensions.k8s.io/jaegers.io.jaegertracing created
INFO[0000] Versions arch=amd64 operator-sdk=v0.4.1 os=linux version=go1.11.1
INFO[0000] Auto-detected the platform platform=openshift
INFO[0000] Starting the Cmd.
ERRO[0113] failed to apply the changes error="no matches for kind \"Elasticsearch\" in version \"logging.openshift.io/v1alpha1\"" instance=simple-prod namespace=myproject
ERRO[0114] failed to apply the changes error="no matches for kind \"Elasticsearch\" in version \"logging.openshift.io/v1alpha1\"" instance=simple-prod namespace=myproject
ERRO[0115] failed to apply the changes error="no matches for kind \"Elasticsearch\" in version \"logging.openshift.io/v1alpha1\"" instance=simple-prod namespace=myproject
ERRO[0116] failed to apply the changes error="no matches for kind \"Elasticsearch\" in version \"logging.openshift.io/v1alpha1\"" instance=simple-prod namespace=myproject
ERRO[0117] failed to apply the changes error="no matches for kind \"Elasticsearch\" in version \"logging.openshift.io/v1alpha1\"" instance=simple-prod namespace=myproject
ERRO[0118] failed to apply the changes error="no matches for kind \"Elasticsearch\" in version \"logging.openshift.io/v1alpha1\"" instance=simple-prod namespace=myproject
ERRO[0120] failed to apply the changes error="no matches for kind \"Elasticsearch\" in version \"logging.openshift.io/v1alpha1\"" instance=simple-prod namespace=myproject
ERRO[0121] failed to apply the changes error="no matches for kind \"Elasticsearch\" in version \"logging.openshift.io/v1alpha1\"" instance=simple-prod namespace=myproject
ERRO[0122] failed to apply the changes error="no matches for kind \"Elasticsearch\" in version \"logging.openshift.io/v1alpha1\"" instance=simple-prod namespace=myproject
ERRO[0123] failed to apply the changes error="no matches for kind \"Elasticsearch\" in version \"logging.openshift.io/v1alpha1\"" instance=simple-prod namespace=myproject
ERRO[0126] failed to apply the changes error="no matches for kind \"Elasticsearch\" in version \"logging.openshift.io/v1alpha1\"" instance=simple-prod namespace=myproject
ERRO[0131] failed to apply the changes error="no matches for kind \"Elasticsearch\" in version \"logging.openshift.io/v1alpha1\"" instance=simple-prod namespace=myproject
Second time
INFO[0000] Versions arch=amd64 operator-sdk=v0.4.1 os=linux version=go1.11.1
INFO[0000] Auto-detected the platform platform=openshift
INFO[0000] Starting the Cmd.
ERRO[0003] failed to apply the changes error="CronJob.batch \"simple-prod-es-index-cleaner\" is invalid: spec.jobTemplate.spec.template.spec.containers[0].volumeMounts[0].name: Not found: \"certs\"" instance=simple-prod namespace=myproject
ERRO[0005] failed to apply the changes error="CronJob.batch \"simple-prod-es-index-cleaner\" is invalid: spec.jobTemplate.spec.template.spec.containers[0].volumeMounts[0].name: Not found: \"certs\"" instance=simple-prod namespace=myproject
ERRO[0006] failed to apply the changes error="CronJob.batch \"simple-prod-es-index-cleaner\" is invalid: spec.jobTemplate.spec.template.spec.containers[0].volumeMounts[0].name: Not found: \"certs\"" instance=simple-prod namespace=myproject
ERRO[0007] failed to apply the changes error="CronJob.batch \"simple-prod-es-index-cleaner\" is invalid: spec.jobTemplate.spec.template.spec.containers[0].volumeMounts[0].name: Not found: \"certs\"" instance=simple-prod namespace=myproject
ERRO[0008] failed to apply the changes error="CronJob.batch \"simple-prod-es-index-cleaner\" is invalid: spec.jobTemplate.spec.template.spec.containers[0].volumeMounts[0].name: Not found: \"certs\"" instance=simple-prod namespace=myproject
|
To test it make sure that |
jpkrohling
force-pushed
the
Fix-Elasticsearch-Objects
branch
from
1b25167
to
576bc87
Compare
|
I got it working now: The collector and query were in a failed state for quite some time, because they don't reconnect to Elasticsearch upon failure, so, I had to kill the pods manually. Kubernetes then created a new pod for the deployments, which then made them work. This feature should be marked as experimental, as it's not really well polished, especially because the collector/query should wait for ES to be ready before they start. If Jaeger could reconnect to ES upon failure like we do with Cassandra, then it wouldn't be a big issue, but right now, we can't do anything else from the Operator's perspective... |
Signed-off-by: Juraci Paixão Kröhling <juraci@kroehling.de>
jpkrohling
force-pushed
the
Fix-Elasticsearch-Objects
branch
from
576bc87
to
5a8d228
Compare
K8s should reschedule pods once they fail. At least I was this behavior before update PR merge. I didn't had to kill pods manually - they restarted 2-3 times until ES was in the ready state. There is an issue #216 which handles the initialization properly. |
That PR shouldn't change this behavior. The pod was in a "healthy" state, which is why Kubernetes didn't kill it. This is how the logs look like: |
|
The query and collector exit with 1 if there are no ES nodes available. Maybe you should wait longer? I have tested on minishift and it worked like before. However, I get the following errors when I edit jaeger e.g. (oc edit jaeger simple-prod) |
Signed-off-by: Juraci Paixão Kröhling <juraci@kroehling.de>
Not sure: it does get into a failed state when the initial connection cannot be made, with logs like this: The case I got into was later on ( |
|
By the way, the PR has been updated, to fix the error you reported before when doing |
pavolloffay
approved these changes
Feb 27, 2019
|
thanks @jpkrohling for looking into this! The #235 seems to be a side effect |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes #233 by adding the ES type to the controller's reconcile loop and inventory.
Signed-off-by: Juraci Paixão Kröhling juraci@kroehling.de