New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Enterprise] Operations manual #1567

Merged

schultyy merged 16 commits into master from js-operations-manual

Dec 1, 2017

Contributor

schultyy commented Nov 28, 2017

This PR adds a new document which we call the operations manual. The purpose of this document is to provide guidance for our customers in case something goes wrong.

This PR ships with a very small version of the manual. Our plan is to enhance it over the time with more and more topics.
The document has multiple entry points, for each problem (like "My builds don't get worked off") we provide a section with strategies to resolve the problem.

If these strategies don't lead to any results the user gets pointed to the Contact support section explaining how they best get in touch with us (What we need from them etc.).

Jan Schulte added 6 commits

November 28, 2017 15:38


          First version of the operations manual with multiple entries

e730cea


          The manual explains what needs to be included in a backup

43264b0


          Add terminology explanation

fe5c4d3


          Improve wording in the introduction

c7fca20


          Improve wording in worker debugging strategies

b23f897


          Better wording for "Get in touch with support"

e193c18

acnagy reviewed

View reviewed changes

user/enterprise/operations-manual.md Outdated


		---

		The Operations Manual is a guideline which helps you to resolve problems with your Travis CI Enterprise instance. Our plan is to enhance this document with frequently occuring support topics over the time. If you would like to see a specific problem case covered here as well, please get in touch with us via [enterprise@travis-ci.com](mailto:enterprise@travis-ci.com).

This comment was marked as spam.

Sign in to view

acnagy reviewed

View reviewed changes

user/enterprise/operations-manual.md Outdated


		The Operations Manual is a guideline which helps you to resolve problems with your Travis CI Enterprise instance. Our plan is to enhance this document with frequently occuring support topics over the time. If you would like to see a specific problem case covered here as well, please get in touch with us via [enterprise@travis-ci.com](mailto:enterprise@travis-ci.com).

		This document has multiple entrypoints. Each entrypoint is a common problem which we've seen ocurring on a regular basis over the time. The section will guide you through it helping you to resolve it. If the problem couldn't be resolved you'll find instructions on how to proceed at the bottom of the document.

This comment was marked as spam.

Sign in to view

This comment was marked as spam.

Sign in to view

This comment was marked as spam.

Sign in to view

Jan Schulte added 2 commits

November 28, 2017 17:45


          Better introduction

e1e0335


          Replace entrypoint with topic

225e709

acnagy reviewed

View reviewed changes

user/enterprise/operations-manual.md Outdated


		This document has multiple entrypoints. Each entrypoint is a common problem which we've seen ocurring on a regular basis over the time. The section will guide you through it helping you to resolve it. If the problem couldn't be resolved you'll find instructions on how to proceed at the bottom of the document.

		Throughout this document we'll be using the following terms to refer to the two components of your Travis CI Enterprise installation:

This comment was marked as spam.

Sign in to view

This comment was marked as spam.

Sign in to view

acnagy reviewed

View reviewed changes

user/enterprise/operations-manual.md Outdated

+. run `travis bash`. This will open a bash session with `root` privileges into the Travis container.
+. Then run `cat /usr/local/travis/etc/travis/config/travis.yml | grep -A1 encryption:`. Create a backup of the value returned by that command by either writing it down on a piece of paper or storing it on a different computer.
+              > Without this key the information in the database is not recoverable.

This comment was marked as spam.

Sign in to view

This comment was marked as spam.

Sign in to view

This comment was marked as spam.

Sign in to view

This comment was marked as spam.

Sign in to view

acnagy reviewed

View reviewed changes

user/enterprise/operations-manual.md Outdated


		The files are located at `/var/travis` on the platform machine. Please run `sudo tar -czvf travis-enterprise-data-backup.tar.gz /var/travis` to create compressed archive from this folder. After this has finished, copy this file off the machine to a secure location.

		## Builds don't get worked off

This comment was marked as spam.

Sign in to view

acnagy reviewed

View reviewed changes

user/enterprise/operations-manual.md Outdated


		### Symptoms

		In the Travis CI Web UI you see none of the builds getting worked off. They're either in no or the `queued` state. Cancelling and restarting them doesn't make any difference.

This comment was marked as spam.

Sign in to view

acnagy reviewed

View reviewed changes

user/enterprise/operations-manual.md Outdated


		### Strategies

		Below you will find different strategies to resolve the problem. They're meant to be followed in order. After you've completed the steps for a strategy please restart a build in the Travis CI Web UI to see if it gets picked up. If that's not happening, please advance to next strategy.

This comment was marked as spam.

Sign in to view

acnagy reviewed

View reviewed changes

user/enterprise/operations-manual.md Outdated


		#### Connection to RabbitMQ got lost

		We're using RabbitMQ to schedule builds for the worker machine(s). Sometimes it can happen that the worker machine(s) lose the connection to RabbitMQ and therefore don't run any new builds anymore. This is a known problem on our side and we're working on resolving this. To get everything back to normal, restarting the machines usually suffices. To do that, connect via `ssh` and run the following command:

This comment was marked as spam.

Sign in to view

acnagy reviewed

View reviewed changes

user/enterprise/operations-manual.md Outdated


		#### Ports are not open Security groups / firewall

		A source for the problem could be that the worker machine is not able to talk to the platform machine.

This comment was marked as spam.

Sign in to view

acnagy reviewed

View reviewed changes

user/enterprise/operations-manual.md Outdated


		## Contact support

		To get in touch with us, please write a message to [enterprise@travis-ci.com](mailto:enterprise@travis-ci.com). In your message we'd like to ask you to answer the questions below.

This comment was marked as spam.

Sign in to view

acnagy reviewed

View reviewed changes

user/enterprise/operations-manual.md Outdated

+              - A support bundle (You can get it from https://yourdomain:8800/support)
+              - Worker log files (They can be found at `/var/log/upstart/travis-worker.log`) - If you're using multiple worker machines, we need the log files from all of them.
+              Is anything special with your setup? There are certain information we can already see, such as the hostname, which IaaS provider you're using, but there are lots of other things we can't see which could lead to something not working. Therefore we'd like to ask you to also answer the questions below in your support request (if applicable):

This comment was marked as spam.

Sign in to view

acnagy suggested changes

View reviewed changes

Contributor

acnagy left a comment

This is looking awesome! I had a bunch of tone suggestions, but the direction and content are great

Jan Schulte and others added 5 commits

November 29, 2017 10:41


          Add note about HA setups

4a6b496


          Wording fixes

5b833fa


          try relative links

4157b91


          Fix links

aa16b7a


          Merge branch 'master' into js-operations-manual

41c0fbc

plaindocs suggested changes

View reviewed changes

user/enterprise/operations-manual.md Outdated

+              layout: en_enterprise
+              ---
+              Welcome to the Travis CI Enterprise Operations Manual! This a living document which provides guidelines and suggestions for troubleshooting your Travis CI Enterprise instance. If you have questions about a specific situation, please get in touch with us via [enterprise@travis-ci.com](mailto:enterprise@travis-ci.com).

This comment was marked as spam.

Sign in to view

user/enterprise/operations-manual.md Outdated

+              ---
+              Welcome to the Travis CI Enterprise Operations Manual! This a living document which provides guidelines and suggestions for troubleshooting your Travis CI Enterprise instance. If you have questions about a specific situation, please get in touch with us via [enterprise@travis-ci.com](mailto:enterprise@travis-ci.com).
+              This document is made of multiple topics. Each topic is a common problem which we've seen ocurring on a regular basis over the time. The section will guide you through it helping you to resolve it. If the problem couldn't be resolved you'll find instructions on how to proceed at the bottom of the document.

This comment was marked as spam.

Sign in to view

user/enterprise/operations-manual.md Outdated

+              - `Platform machine`: The virtual machine that runs most of the Travis web components. This is the machine your domain is pointing to.
+              - `Worker machine`: The worker machine(s) run your builds.
+              > Please note that this guide is geared towards non-HA setups right now.

This comment was marked as spam.

Sign in to view

This comment was marked as spam.

Sign in to view

user/enterprise/operations-manual.md Outdated

+. run `travis bash`. This will open a bash session with `root` privileges into the Travis container.
+. Then run `cat /usr/local/travis/etc/travis/config/travis.yml | grep -A1 encryption:`. Create a backup of the value returned by that command by either writing it down on a piece of paper or storing it on a different computer.
+              > Without this key the information in the database is not recoverable.

This comment was marked as spam.

Sign in to view

user/enterprise/operations-manual.md Outdated


		## Builds are not starting

		### Symptoms

This comment was marked as spam.

Sign in to view

user/enterprise/operations-manual.md Outdated


		### Symptoms

		In the Travis CI Web UI you see none of the builds are starting. They're either in no or the `queued` state. Cancelling and restarting them doesn't make any difference.

This comment was marked as spam.

Sign in to view

user/enterprise/operations-manual.md Outdated


		### Strategies

		Below you will find different strategies to resolve the problem. They're meant to be followed in order. After you've completed the steps for a strategy please restart a build in the Travis CI Web UI to see if it gets picked up. If that's not happening, please try the next strategy.

This comment was marked as spam.

Sign in to view

user/enterprise/operations-manual.md Outdated


		#### Connection to RabbitMQ got lost

		We're using RabbitMQ to schedule builds for the worker machine(s). Sometimes it happens that the worker machine(s) lose the connection to RabbitMQ and therefore don't run any new builds anymore. This is a known problem on our side and we're working on resolving this. To get everything back to normal, restarting the machines usually suffices. To do that, connect via `ssh` and run the following command:

This comment was marked as spam.

Sign in to view

This comment was marked as spam.

Sign in to view

user/enterprise/operations-manual.md


		## Contact support

		To get in touch with us, please write a message to [enterprise@travis-ci.com](mailto:enterprise@travis-ci.com). It would be very helpful for Support if you could include the following:

This comment was marked as spam.

Sign in to view

plaindocs and others added 3 commits

November 30, 2017 14:36


          Merge branch 'master' into js-operations-manual

b167b70


          Spelling fixes and rephrasing for easier understanding

e8561c6


          "Follow these steps"

ef3ed3e

schultyy merged commit 84a59d6 into master

schultyy deleted the js-operations-manual branch

December 1, 2017 15:27

acnagy added the enterprise label

acnagy added enterprise and removed enterprise labels

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels