Skip to content

Commit e3e67e3

Browse files
polyfractalpolyfractal
committed
start post-deployment
1 parent 763b31f commit e3e67e3

File tree

7 files changed

+155
-38
lines changed

7 files changed

+155
-38
lines changed

‎510_Deployment.asciidoc‎

Lines changed: 2 additions & 29 deletions
Original file line numberDiff line numberDiff line change
@@ -9,36 +9,9 @@ include::510_Deployment/30_other.asciidoc[]
99

1010
include::510_Deployment/40_config.asciidoc[]
1111

12+
include::510_Deployment/45_dont_touch.asciidoc[]
13+
1214
include::510_Deployment/50_heap.asciidoc[]
1315

1416
include::510_Deployment/60_file_descriptors.asciidoc[]
1517

16-
=== Post-Deployment
17-
18-
-Prereqs
19-
20-
- Hardware
21-
-memory, cpu, etc
22-
- disable swap on node and VM host
23-
- ssd noop
24-
25-
- Talking to the cluster
26-
- language clients
27-
- TransportClient vs NodeClient
28-
- Proxies, load balancers, etc
29-
- JVM
30-
31-
- Configs to change before production
32-
- elasticsearch.yml
33-
- cluster settings api
34-
- changing logging dynamically
35-
- don't touch these:
36-
- GC, threadpools
37-
38-
- index performance tips
39-
- Mike's blog
40-
- security
41-
- none lololol
42-
43-
-snapshot/restore
44-
- fs must be accessible from all nodes

‎510_Deployment/45_dont_touch.asciidoc‎

Lines changed: 9 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -68,17 +68,17 @@ will happen on the same core. If you are unlucky, the switch may migrate to a
6868
different core and require transport on inter-core communication bus.
6969

7070
This context switching eats up cycles simply doing administrative housekeeping
71-
-- estimates can peg it as high as 30us on modern CPUs. So unless the thread
72-
will be blocked for longer than 30us, it is highly likely that that time would
73-
have been better spent just processing and finishing early.
71+
-- estimates can peg it as high as 30μs on modern CPUs. So unless the thread
72+
will be blocked for longer than 30μs, it is highly likely that that time would
73+
have been better spent just processing and finishing early.
7474

75-
People routinely set threadpools to silly values. On 8 core machines, we have
76-
run across configs with 60, 100 or even 1000 threads. These settings will simply
77-
thrash the CPU more than getting real work done.
75+
People routinely set threadpools to silly values. On 8 core machines, we have
76+
run across configs with 60, 100 or even 1000 threads. These settings will simply
77+
thrash the CPU more than getting real work done.
7878

79-
So. Next time you want to tweak a threadpool...please don't. And if you
80-
_absolutely cannot resist_, please keep your core count in mind and perhaps set
81-
the count to double. More than that is just a waste.
79+
So. Next time you want to tweak a threadpool...please don't. And if you
80+
_absolutely cannot resist_, please keep your core count in mind and perhaps set
81+
the count to double. More than that is just a waste.
8282

8383

8484

‎510_Deployment/80_cluster_settings.asciidoc‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
2+
===

‎520_Post_Deployment.asciidoc‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
[[post_deploy]]
2+
== Post-Deployment
3+
4+
Once you have deployed your cluster in production, there are some tools and
5+
best practices to keep your cluster running in top shape. In this short
6+
section, we'll talk about configuring settings dynamically, how to tweak
7+
logging levels, indexing performance tips and how to backup your cluster.
8+
9+
include::520_Post_Deployment/10_dynamic_settings.asciidoc[]
10+
11+
include::520_Post_Deployment/20_logging.asciidoc[]
12+
13+
14+
15+
- index performance tips
16+
- Mike's blog
17+
- security
18+
- none lololol
19+
20+
-snapshot/restore
21+
- fs must be accessible from all nodes
22+
23+
rolling restarts

‎520_Post_Deployment/10_dynamic_settings.asciidoc‎

Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,37 @@
1+
2+
=== Changing settings dynamically
3+
4+
Many settings in Elasticsearch are dynamic, and modifiable through the API.
5+
Configuration changes that force a node (or cluster) restart are strenuously avoided.
6+
And while it's possible to make the changes through the static configs, we
7+
recommend that you use the API instead.
8+
9+
The _Cluster Update_ API operates in two modes:
10+
11+
- Transient: these changes are in effect until the cluster restarts. Once
12+
a full cluster restart takes place, these settings are erased
13+
14+
- Persistent: these changes are permanently in place unless explicitly changed.
15+
They will survive full cluster restarts and override the static configuration files.
16+
17+
Transient vs Persistent settings are supplied in the JSON body:
18+
19+
[source,js]
20+
----
21+
PUT /_cluster/settings
22+
{
23+
"persistent" : {
24+
"discovery.zen.minimum_master_nodes" : 2 <1>
25+
},
26+
"transient" : {
27+
"indices.store.throttle.max_bytes_per_sec" : "50mb" <2>
28+
}
29+
}
30+
----
31+
<1> This persistent setting will survive full cluster restarts
32+
<2> While this transient setting will be removed after the first full cluster
33+
restart
34+
35+
A complete list of settings that are dynamically updateable can be found in the
36+
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/cluster-update-settings.html[online reference docs].
37+

‎520_Post_Deployment/20_logging.asciidoc‎

Lines changed: 80 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,80 @@
1+
2+
=== Logging
3+
4+
Elasticsearch emits a number of logs, which by are placed in `ES_HOME/logs`.
5+
The default logging level is INFO. It provides a moderate amount of information,
6+
but is designed to be rather light so that your logs are not enormous.
7+
8+
When debugging problems, particularly problems with node discovery (since this
9+
often depends on finicky network configurations), it can be helpful to bump
10+
up the logging level to DEBUG.
11+
12+
You _could_ modify the `logging.yml` file and restart your nodes...but that is
13+
both tedious and leads to unnecessary downtime. Instead, you can update logging
14+
levels through the Cluster Settings API that we just learned about.
15+
16+
To do so, take the logger you are interested in and prepend `logger.` to it.
17+
Let's turn up the discovery logging:
18+
19+
[source,js]
20+
----
21+
PUT /_cluster/settings
22+
{
23+
"transient" : {
24+
"logger.discovery" : "DEBUG"
25+
}
26+
}
27+
----
28+
29+
While this setting is in effect, Elasticsearch will begin to emit DEBUG-level
30+
logs for the `discovery` module.
31+
32+
INFORMATION: Avoid TRACE, it is extremely verbose, to the point where the logs
33+
are no longer useful.
34+
35+
==== Slowlog
36+
37+
There is another log called the _Slowlog_. The purpose of this log is to catch
38+
queries and indexing requests that take over a certain threshold of time.
39+
It is useful for hunting down user-generated queries that are particularly slow.
40+
41+
By default, the slowlog is not enabled. It can be enabled by defining the action
42+
(query, fetch or index), the level that you want the event logged at (WARN, DEBUG,
43+
etc) and a time threshold.
44+
45+
This is an index-level setting, which means it is applied to individual indices:
46+
47+
[source,js]
48+
----
49+
PUT /my_index/_settings
50+
{
51+
"index.search.slowlog.threshold.query.warn" : "10s", <1>
52+
"index.search.slowlog.threshold.fetch.debug": 500ms", <2>
53+
"index.indexing.slowlog.threshold.index.info": 5s" <3>
54+
}
55+
----
56+
<1> Emit a WARN log when queries are slower than 10s
57+
<2> Emit a DEBUG log when fetches are slower than 500ms
58+
<3> Emit an INFO log when indexing takes longer than 5s
59+
60+
You can also define these thresholds in your `elasticsearch.yml` file. Indices
61+
that do not have a threshold set will inherit whatever is configured in the
62+
static config.
63+
64+
Once the thresholds are set, you can toggle the logging level like any other
65+
logger:
66+
67+
[source,js]
68+
----
69+
PUT /_cluster/settings
70+
{
71+
"transient" : {
72+
"logger.index.search.slowlog" : "DEBUG", <1>
73+
"logger.index.indexing.slowlog" : WARN <2>
74+
}
75+
}
76+
----
77+
<1> Set the search slowlog to DEBUG level
78+
<2> Set the indexing slowlog to WARN level
79+
80+

‎book.asciidoc‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -107,6 +107,8 @@ include::500_Cluster_Admin.asciidoc[]
107107

108108
include::510_Deployment.asciidoc[]
109109

110+
include::520_Post_Deployment.asciidoc[]
111+
110112
[[TODO]]
111113
[appendix]
112114
= TODO

0 commit comments

Comments
 (0)