Updated docs

Author: Docs User

html/branches.yaml

@@ -642,7 +642,16 @@ <h2 class="page-title">Guide template</h2>
 you might need to refresh the index to make the fields available in Kibana. To refresh
 the index, use the <a class="ulink" href="https://www.elastic.co/guide/en/elasticsearch/reference/master/indices-refresh.html" target="_top">refresh API</a>. For example:</p><div class="pre_wrapper"><pre class="programlisting prettyprint lang-sh">curl -XPOST 'http://localhost:9200/filebeat-2016.08.09/_refresh'</pre></div><h3><a id="newline-character-required-eof"></a>Filebeat isn’t shipping the last line of a file?<a href="https://github.com/elastic/beats/edit/master/filebeat/docs/faq.asciidoc" class="edit_me" title="Edit this page on GitHub" rel="nofollow">edit</a></h3><p>Filebeat uses a newline character to detect the end of an event. If lines are added incrementally to a file that’s being
 harvested, a newline character is required after the last line, or Filebeat will not read the last line of
-the file.</p><h3><a id="bandwidth-throttling"></a>Need to limit bandwidth used by Filebeat?<a href="https://github.com/elastic/beats/edit/master/libbeat/docs/faq-limit-bandwidth.asciidoc" class="edit_me" title="Edit this page on GitHub" rel="nofollow">edit</a></h3><p>If you need to limit bandwidth usage, we recommend that you configure the network stack on your OS to perform
+the file.</p><h3><a id="faq-deleted-files-are-not-freed"></a>Filebeat keeps open file handlers of deleted files for a long time?<a href="https://github.com/elastic/beats/edit/master/filebeat/docs/faq.asciidoc" class="edit_me" title="Edit this page on GitHub" rel="nofollow">edit</a></h3><p>In the default behaviour, Filebeat opens the files and keeps them open until it
+reaches the end of them.  In situations when the configured output is blocked
+(e.g. Elasticsearch or Logstash is unavailable) for a long time, this can cause
+Filebeat to keep file handlers to files that were deleted from the file system
+in the mean time. As long as Filebeat keeps the deleted files open, the
+operating system doesn’t free up the space on disk, which can lead to increase
+disk utilisation or even out of disk situations.</p><p>To mitigate this issue, you can set the <a class="xref" href="configuration-filebeat-options.html#close-timeout" title="close_timeoutedit"><code class="literal">close_timeout</code><a href="https://github.com/elastic/beats/edit/master/filebeat/docs/filebeat-options.asciidoc" class="edit_me" title="Edit this page on GitHub" rel="nofollow">edit</a></a> setting to <code class="literal">5m</code>. This
+will ensure every file handler is closed once every 5 minutes, regardless of
+whether it reached EOF or not. Note that this option can lead to data loss if the
+file is deleted before Filebeat reaches the end of the file.</p><h3><a id="bandwidth-throttling"></a>Need to limit bandwidth used by Filebeat?<a href="https://github.com/elastic/beats/edit/master/libbeat/docs/faq-limit-bandwidth.asciidoc" class="edit_me" title="Edit this page on GitHub" rel="nofollow">edit</a></h3><p>If you need to limit bandwidth usage, we recommend that you configure the network stack on your OS to perform
 bandwidth throttling.</p><p>For example, the following Linux commands cap the connection between Filebeat and Logstash by setting a
 limit of 50 kbps on TCP connections over port 5044:</p><div class="pre_wrapper"><pre class="programlisting prettyprint lang-shell">tc qdisc add dev $DEV root handle 1: htb
 tc class add dev $DEV parent 1:1 classid 1:10 htb rate 50kbps ceil 50kbps

html/en/beats/filebeat/master/configuration-filebeat-options.html

@@ -679,6 +679,29 @@ <h2 class="page-title">Guide template</h2>
 </span></dt><dd>
 An array of paths that specify where to look for the log files. If left empty,
 Filebeat will choose the paths depending on your operating systems.
+</dd><dt><span class="term">
+<span class="strong strong"><strong><code class="literal">var.convert_timezone</code></strong></span>
+</span></dt><dd>
+If this option is enabled, Filebeat reads the local timezone and uses it at log
+parsing time to convert the timestamp to UTC. The local timezone is also added
+in each event in a dedicated field (<code class="literal">beat.timezone</code>). The conversion is only
+possible in Elasticsearch &gt;= 6.1. If the Elasticsearch version is less than 6.1,
+the <code class="literal">beat.timezone</code> field is added, but the conversion to UTC is not made.  The
+default is <code class="literal">false</code>.
+</dd></dl></div><h4><a id="_literal_auth_literal_fileset_settings"></a><code class="literal">auth</code> fileset settings<a href="https://github.com/elastic/beats/edit/master/filebeat/docs/modules/system.asciidoc" class="edit_me" title="Edit this page on GitHub" rel="nofollow">edit</a></h4><div class="variablelist"><dl class="variablelist"><dt><span class="term">
+<span class="strong strong"><strong><code class="literal">var.paths</code></strong></span>
+</span></dt><dd>
+An array of paths that specify where to look for the log files. If left empty,
+Filebeat will choose the paths depending on your operating systems.
+</dd><dt><span class="term">
+<span class="strong strong"><strong><code class="literal">var.convert_timezone</code></strong></span>
+</span></dt><dd>
+If this option is enabled, Filebeat reads the local timezone and uses it at log
+parsing time to convert the timestamp to UTC. The local timezone is also added
+in each event in a dedicated field (<code class="literal">beat.timezone</code>). The conversion is only
+possible in Elasticsearch &gt;= 6.1. If the Elasticsearch version is less than 6.1,
+the <code class="literal">beat.timezone</code> field is added, but the conversion to UTC is not made.  The
+default is <code class="literal">false</code>.
 </dd></dl></div><h3><a id="_fields_9"></a>Fields<a href="https://github.com/elastic/beats/edit/master/filebeat/docs/modules/system.asciidoc" class="edit_me" title="Edit this page on GitHub" rel="nofollow">edit</a></h3><p>For a description of each field in the module, see the
 <a class="link" href="exported-fields-system.html" title="System fields">exported fields</a> section.</p></div><div class="navfooter"><span class="prev"><a href="filebeat-module-redis.html">
               « 

html/en/beats/filebeat/master/faq.html

@@ -641,6 +641,9 @@ <h2 class="page-title">Guide template</h2>
     # Filebeat will choose the paths depending on your OS.
     #var.paths:
 
+    # Convert the timestamp to UTC. Requires Elasticsearch &gt;= 6.1.
+    #convert_timezone: false
+
     # Prospector configuration (advanced). Any prospector configuration option
     # can be added under this section.
     #prospector:
@@ -653,6 +656,9 @@ <h2 class="page-title">Guide template</h2>
     # Filebeat will choose the paths depending on your OS.
     #var.paths:
 
+    # Convert the timestamp to UTC. Requires Elasticsearch &gt;= 6.1.
+    #convert_timezone: false
+
     # Prospector configuration (advanced). Any prospector configuration option
     # can be added under this section.
     #prospector:

html/en/beats/filebeat/master/filebeat-module-system.html

@@ -613,7 +613,7 @@ <h2 class="page-title">Guide template</h2>
         </span><span class="next">
 
           </span></div><div class="section"><div class="titlepage"><div><div><h2 class="title"><a id="release-notes-1.0.0-beta4"></a>Beats version 1.0.0-beta4<a href="https://github.com/elastic/beats/edit/master/CHANGELOG.asciidoc" class="edit_me" title="Edit this page on GitHub" rel="nofollow">edit</a></h2></div></div></div><p><a class="ulink" href="https://github.com/elastic/beats/compare/1.0.0-beta3…1.0.0-beta4" target="_top">Check
-1.0.0-beta4 diff</a></p><div class="section"><div class="titlepage"><div><div><h3 class="title"><a id="_breaking_changes_22"></a>Breaking changes<a href="https://github.com/elastic/beats/edit/master/CHANGELOG.asciidoc" class="edit_me" title="Edit this page on GitHub" rel="nofollow">edit</a></h3></div></div></div><p><span class="strong strong"><strong>Affecting all Beats</strong></span></p><div class="itemizedlist"><ul class="itemizedlist" type="disc"><li class="listitem">
+1.0.0-beta4 diff</a></p><div class="section"><div class="titlepage"><div><div><h3 class="title"><a id="_breaking_changes_24"></a>Breaking changes<a href="https://github.com/elastic/beats/edit/master/CHANGELOG.asciidoc" class="edit_me" title="Edit this page on GitHub" rel="nofollow">edit</a></h3></div></div></div><p><span class="strong strong"><strong>Affecting all Beats</strong></span></p><div class="itemizedlist"><ul class="itemizedlist" type="disc"><li class="listitem">
 Update tls config options naming from dash to underline #162
 </li><li class="listitem">
 Feature/output modes: Introduction of PublishEvent(s) to be used by beats #118 #115
@@ -625,7 +625,7 @@ <h2 class="page-title">Guide template</h2>
 Rename elasticsearch index to [packetbeat-]YYYY.MM.DD
 </li></ul></div><p><span class="strong strong"><strong>Topbeat</strong></span></p><div class="itemizedlist"><ul class="itemizedlist" type="disc"><li class="listitem">
 Percentage fields (e.g user_p) are exported as a float between 0 and 1 #34
-</li></ul></div></div><div class="section"><div class="titlepage"><div><div><h3 class="title"><a id="_bugfixes_42"></a>Bugfixes<a href="https://github.com/elastic/beats/edit/master/CHANGELOG.asciidoc" class="edit_me" title="Edit this page on GitHub" rel="nofollow">edit</a></h3></div></div></div><p><span class="strong strong"><strong>Affecting all Beats</strong></span></p><div class="itemizedlist"><ul class="itemizedlist" type="disc"><li class="listitem">
+</li></ul></div></div><div class="section"><div class="titlepage"><div><div><h3 class="title"><a id="_bugfixes_45"></a>Bugfixes<a href="https://github.com/elastic/beats/edit/master/CHANGELOG.asciidoc" class="edit_me" title="Edit this page on GitHub" rel="nofollow">edit</a></h3></div></div></div><p><span class="strong strong"><strong>Affecting all Beats</strong></span></p><div class="itemizedlist"><ul class="itemizedlist" type="disc"><li class="listitem">
 Determine Elasticsearch index for an event based on UTC time #81
 </li><li class="listitem">
 Fixing ES output’s defaultDeadTimeout so that it is 60 seconds #103
@@ -649,7 +649,7 @@ <h2 class="page-title">Guide template</h2>
 Add missing DNS protocol configuration to documentation #269
 </li></ul></div><p><span class="strong strong"><strong>Topbeat</strong></span></p><div class="itemizedlist"><ul class="itemizedlist" type="disc"><li class="listitem">
 Don’t divide the reported memory by an extra 1024 #60
-</li></ul></div></div><div class="section"><div class="titlepage"><div><div><h3 class="title"><a id="_added_28"></a>Added<a href="https://github.com/elastic/beats/edit/master/CHANGELOG.asciidoc" class="edit_me" title="Edit this page on GitHub" rel="nofollow">edit</a></h3></div></div></div><p><span class="strong strong"><strong>Affecting all Beats</strong></span></p><div class="itemizedlist"><ul class="itemizedlist" type="disc"><li class="listitem">
+</li></ul></div></div><div class="section"><div class="titlepage"><div><div><h3 class="title"><a id="_added_31"></a>Added<a href="https://github.com/elastic/beats/edit/master/CHANGELOG.asciidoc" class="edit_me" title="Edit this page on GitHub" rel="nofollow">edit</a></h3></div></div></div><p><span class="strong strong"><strong>Affecting all Beats</strong></span></p><div class="itemizedlist"><ul class="itemizedlist" type="disc"><li class="listitem">
 Add logstash output plugin #151
 </li><li class="listitem">
 Integration tests for Beat → Logstash → Elasticsearch added #195 #188 #168 #137 #128 #112
@@ -712,7 +712,7 @@ <h2 class="page-title">Guide template</h2>
 Updated configuration files with comments for all options #65
 </li><li class="listitem">
 Documentation improvements
-</li></ul></div></div><div class="section"><div class="titlepage"><div><div><h3 class="title"><a id="_deprecated_12"></a>Deprecated<a href="https://github.com/elastic/beats/edit/master/CHANGELOG.asciidoc" class="edit_me" title="Edit this page on GitHub" rel="nofollow">edit</a></h3></div></div></div><p><span class="strong strong"><strong>Affecting all Beats</strong></span></p><div class="itemizedlist"><ul class="itemizedlist" type="disc"><li class="listitem">
+</li></ul></div></div><div class="section"><div class="titlepage"><div><div><h3 class="title"><a id="_deprecated_14"></a>Deprecated<a href="https://github.com/elastic/beats/edit/master/CHANGELOG.asciidoc" class="edit_me" title="Edit this page on GitHub" rel="nofollow">edit</a></h3></div></div></div><p><span class="strong strong"><strong>Affecting all Beats</strong></span></p><div class="itemizedlist"><ul class="itemizedlist" type="disc"><li class="listitem">
 Redis output was deprecated #169 #145
 </li><li class="listitem">
 Host and port configuration options are deprecated. They are replaced by the hosts

html/en/beats/filebeat/master/filebeat-reference-yml.html

@@ -615,7 +615,7 @@ <h2 class="page-title">Guide template</h2>
           <a href="release-notes-1.0.0-beta4.html">Beats version 1.0.0-beta4
                »
             </a></span></div><div class="section"><div class="titlepage"><div><div><h2 class="title"><a id="release-notes-1.0.0-rc1"></a>Beats version 1.0.0-rc1<a href="https://github.com/elastic/beats/edit/master/CHANGELOG.asciidoc" class="edit_me" title="Edit this page on GitHub" rel="nofollow">edit</a></h2></div></div></div><p><a class="ulink" href="https://github.com/elastic/beats/compare/1.0.0-beta4…1.0.0-rc1" target="_top">Check
-1.0.0-rc1 diff</a></p><div class="section"><div class="titlepage"><div><div><h3 class="title"><a id="_breaking_changes_21"></a>Breaking changes<a href="https://github.com/elastic/beats/edit/master/CHANGELOG.asciidoc" class="edit_me" title="Edit this page on GitHub" rel="nofollow">edit</a></h3></div></div></div><p><span class="strong strong"><strong>Affecting all Beats</strong></span></p><div class="itemizedlist"><ul class="itemizedlist" type="disc"><li class="listitem">
+1.0.0-rc1 diff</a></p><div class="section"><div class="titlepage"><div><div><h3 class="title"><a id="_breaking_changes_23"></a>Breaking changes<a href="https://github.com/elastic/beats/edit/master/CHANGELOG.asciidoc" class="edit_me" title="Edit this page on GitHub" rel="nofollow">edit</a></h3></div></div></div><p><span class="strong strong"><strong>Affecting all Beats</strong></span></p><div class="itemizedlist"><ul class="itemizedlist" type="disc"><li class="listitem">
 Rename timestamp field with @timestamp. #237
 </li></ul></div><p><span class="strong strong"><strong>Packetbeat</strong></span></p><div class="itemizedlist"><ul class="itemizedlist" type="disc"><li class="listitem">
 Rename timestamp field with @timestamp. #343
@@ -628,7 +628,7 @@ <h2 class="page-title">Guide template</h2>
 Rename tail_on_rotate prospector config to tail_files
 </li><li class="listitem">
 Removal of line field in event. Line number was not correct and does not add value. #217
-</li></ul></div></div><div class="section"><div class="titlepage"><div><div><h3 class="title"><a id="_bugfixes_41"></a>Bugfixes<a href="https://github.com/elastic/beats/edit/master/CHANGELOG.asciidoc" class="edit_me" title="Edit this page on GitHub" rel="nofollow">edit</a></h3></div></div></div><p><span class="strong strong"><strong>Affecting all Beats</strong></span></p><div class="itemizedlist"><ul class="itemizedlist" type="disc"><li class="listitem">
+</li></ul></div></div><div class="section"><div class="titlepage"><div><div><h3 class="title"><a id="_bugfixes_44"></a>Bugfixes<a href="https://github.com/elastic/beats/edit/master/CHANGELOG.asciidoc" class="edit_me" title="Edit this page on GitHub" rel="nofollow">edit</a></h3></div></div></div><p><span class="strong strong"><strong>Affecting all Beats</strong></span></p><div class="itemizedlist"><ul class="itemizedlist" type="disc"><li class="listitem">
 Use stderr for console log output. #219
 </li><li class="listitem">
 Handle empty event array in publisher. #207
@@ -673,7 +673,7 @@ <h2 class="page-title">Guide template</h2>
 Fix high CPU usage when not connected to Elasticsearch or Logstash. #144
 </li><li class="listitem">
 Fix issue that files were not crawled anymore when encoding was set to something other then plain. #182
-</li></ul></div></div><div class="section"><div class="titlepage"><div><div><h3 class="title"><a id="_added_27"></a>Added<a href="https://github.com/elastic/beats/edit/master/CHANGELOG.asciidoc" class="edit_me" title="Edit this page on GitHub" rel="nofollow">edit</a></h3></div></div></div><p><span class="strong strong"><strong>Affecting all Beats</strong></span></p><div class="itemizedlist"><ul class="itemizedlist" type="disc"><li class="listitem">
+</li></ul></div></div><div class="section"><div class="titlepage"><div><div><h3 class="title"><a id="_added_30"></a>Added<a href="https://github.com/elastic/beats/edit/master/CHANGELOG.asciidoc" class="edit_me" title="Edit this page on GitHub" rel="nofollow">edit</a></h3></div></div></div><p><span class="strong strong"><strong>Affecting all Beats</strong></span></p><div class="itemizedlist"><ul class="itemizedlist" type="disc"><li class="listitem">
 Add Console output plugin. #218
 </li><li class="listitem">
 Add timestamp to log messages #245