Merge branch 'master' into quic-latest

* master:
  Introduce NetEvent to split UnixNetVConnection and NetHandler
  Reduce process duration of test_Http2FrequencyCounter
  Make sure shutdown_cont_event isn't holding any garbage references.
  Add in_destroy to Http2ConnectionState to prevent double delete (similar to Http2ClientSession)
  Fixes a corner case where the NextHop consistent hash ring may not be searched in it's entirety for an available host due to a premature wrapped ring indication.
  Adjust debug tag for ssl_sni_whitelist plugin to match plugin
  Add invalid config warning when cache open write fail and read retry are inconsistent
  Add links to RWW sections
  Fix rst errors for collapsed forwarding plugin
  Set wrap after checking all the parents
  Perf: replace dynamic_cast with static_cast in this_thread()
  Fixes a few Sphinx build warnings (#6290)
  Moving / adding the Roadmap to the release notes (#6257)
  These features are in 9.0.x release notes (#6286)
  Removes the remaining references to TSQA
  LGTM: Fix unused imports
  Assure no SM survives plugin factory deactivation.
  Remove remnants of obsolete remap thread.
  Update the admin-guide hierachical caching and remap.config documentation to include the NextHop strategies feature.
  For combo_handler plugin, add an optional whitelist of allowed values for Content-Type.
  Copy the Client SNI Server Name out of the openssl SSL object and ensure it is null-terminated.
  For per-transaction config override, crossing the const-correctness event horizon.
  Remove -j from lgtm build (#6274)
  LGTM: Add header guards
  LGTM: Fix wrong type of arguments to formatting function
  Change HTTP/2 error rate log to warning
  LGTM: fixes a shadowed variable in ParentSelection::PreprocessParents().
  LGTM: fix a comparison that is always false.
  Fixes various crashers loading/reloading parent.config
  TCL: cleanup in HostLookup.cc, make sure keys are stable. (#6263)
  Fix TS_USE_DIAGS usage for --disable-diags option
  Initial revision of .lgtm.yml config file (#6258)
  auto delete rolled log file fixes
  remap_stats: restore handling of remap/hostname to remove memory leak
  Avoid unnecesarry copy on POST request over HTTP/2
  Add some stats collections to cache_promote.
  Some tweaks to reloading-plugins.en.rst (#6251)
  Fix sni.yaml fqdn to match complete name string
  Adding verify plugin TS maintenance commands
  Removing traffic_cop reference in Admin Guide introduction.

Author: maskit

.gitignore

@@ -1,4 +1,6 @@
 *.o
+*.c_o
+*.cc_o
 *.a
 *.so
 *.pyc

.lgtm.yml

@@ -0,0 +1,32 @@
+#  Licensed to the Apache Software Foundation (ASF) under one
+#  or more contributor license agreements.  See the NOTICE file
+#  distributed with this work for additional information
+#  regarding copyright ownership.  The ASF licenses this file
+#  to you under the Apache License, Version 2.0 (the
+#  "License"); you may not use this file except in compliance
+#  with the License.  You may obtain a copy of the License at
+#
+#  http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+
+
+extraction:
+  cpp:
+    prepare:
+      packages:
+      - "libcjose-dev"
+      - "libcjose0"
+      - "libjansson-dev"
+      - "libluajit-5.1-dev"
+      - "libpcre3-dev"
+      - "zlib1g-dev"
+    after_prepare: "autoreconf -fiv"
+    configure:
+      command: "./configure --enable-experimental-plugins"
+    index:
+      build_command: "make"

contrib/python/compare_RecordsConfigcc.py

@@ -18,7 +18,6 @@
 #
 import re
 import sys
-import string
 
 try:
     src_dir = sys.argv[1]

doc/admin-guide/configuration/hierarchical-caching.en.rst

@@ -106,6 +106,11 @@ host name. This is true regardless of pristine host headers
 (:ts:cv:`proxy.config.url_remap.pristine_host_hdr`) being enabled or not. The parent node
 will receive the translated request (and thus needs to be configured to accept it).
 
+:file:`remap.config` now also allows an alternative configuration that supports all the
+**Parent Selection** policies and failover mentioned here using a new :file:`remap.config` tag,
+**@strategy**. This eliminates the need for a second lookup against the remapped host name
+required when using :file:`parent.config`.  See using NextHop strategies with :file:`remap.config`,
+:doc:`../../admin-guide/files/strategies.yaml.en`
 
 Example
 ~~~~~~~
@@ -165,7 +170,11 @@ the configuration adjustments detailed below.
    cache.
 
 #. Edit :file:`parent.config` to set parent proxy rules which will specify the
-   parent cache to which you want missed requests to be forwarded.
+   parent cache to which you want missed requests to be forwarded.  Or as an
+   alternative to :file:`parent.config`, edit :file:`strategies.yaml` to
+   specify next hop parent proxy rules and hosts.  Then in :file:`remap.config`
+   use the **@strategy** tag to select the next hop parent proxy rules. See
+   :doc:`../../admin-guide/files/strategies.yaml.en`
 
 The following example configures Traffic Server to route all requests
 containing the regular expression ``politics`` and the path

doc/admin-guide/files/records.config.en.rst

@@ -2329,11 +2329,11 @@ Dynamic Content & Content Negotiation
          origin server.
    ``4`` Return a ``502`` error on either a cache miss or on a revalidation.
    ``5`` Retry Cache Read on a Cache Write Lock failure. This option together
-         with `proxy.config.cache.enable_read_while_writer` configuration
+         with :ts:cv:`proxy.config.cache.enable_read_while_writer` configuration
          allows to collapse concurrent requests without a need for any plugin.
-         Make sure to configure Read While Writer feature correctly following
-         the docs in Cache Basics section. Note that this option may result in
-         CACHE_LOOKUP_COMPLETE HOOK being called back more than once.
+         Make sure to configure the :ref:`admin-config-read-while-writer` feature
+         correctly. Note that this option may result in CACHE_LOOKUP_COMPLETE HOOK
+         being called back more than once.
    ===== ======================================================================
 
 Customizable User Response Pages
@@ -3970,15 +3970,6 @@ Plug-in Configuration
 
    Specifies the location of |TS| plugins.
 
-.. ts:cv:: CONFIG proxy.config.remap.num_remap_threads INT 0
-
-   When this variable is set to ``0``, plugin remap callbacks are
-   executed in line on network threads. If remap processing takes
-   significant time, this can be cause additional request latency.
-   Setting this variable to causes remap processing to take place
-   on a dedicated thread pool, freeing the network threads to service
-   additional requests.
-
 SOCKS Processor
 ===============
 

doc/admin-guide/files/remap.config.en.rst

@@ -480,6 +480,13 @@ mapping rules. (It is activated before any mappings and is never
 deactivated.) The filter `local_only` will only be applied to the
 second mapping.
 
+NextHop Selection Strategies
+============================
+
+You may configure Nexthop or Parent hierarchical caching rules by remap using the
+**@strategy** tag.  See :doc:`../configuration/hierarchical-caching.en` and :doc:`strategies.yaml.en`
+for configuration details and examples.
+
 Including Additional Remap Files
 ================================
 

doc/admin-guide/introduction.en.rst

@@ -165,15 +165,15 @@ bindings in memory, DNS traffic is reduced.
 |TS| Processes
 --------------
 
-|TS| contains three processes that work together to serve
+|TS| contains two processes that work together to serve
 requests and manage, control, and monitor the health of the system.
 
--  The :program:`traffic_server` process is the transaction processing engine
+#. The :program:`traffic_server` process is the transaction processing engine
    of |TS|. It is responsible for accepting connections,
    processing protocol requests, and serving documents from the cache or
    origin server.
 
--  The :program:`traffic_manager` process is the command and control facility
+#. The :program:`traffic_manager` process is the command and control facility
    of the |TS|, responsible for launching, monitoring, and
    reconfiguring the :program:`traffic_server` process. The :program:`traffic_manager`
    process is also responsible for the proxy autoconfiguration port, the
@@ -187,14 +187,6 @@ requests and manage, control, and monitor the health of the system.
    first-served order. This connection queueing shields users from any
    server restart downtime.
 
-The figure below illustrates the |TS| processes.
-
-.. figure:: ../static/images/admin/process.jpg
-   :align: center
-   :alt: Illustration of the |TS| Processes
-
-   Illustration of the |TS| Processes
-
 Administration Tools
 --------------------
 

doc/admin-guide/plugins/cache_promote.en.rst

@@ -49,6 +49,21 @@ If :option:`--policy` is set to ``lru`` the following options are also available
 
    The size (number of entries) of the LRU.
 
+.. option:: --stats-enable-with-id
+
+   Enables collecting statistics.  The option requires an argument, the
+   remap-identifier.  The remap-identifier is a string that is concatenated
+   to the stat name.  The following stats are collected.
+
+*  **plugin.cache_promote.${remap-identifier}.cache_hits** - Cache hit total, available for all policies.
+*  **plugin.cache_promote.${remap-identifier}.freelist_size** - Size of the freelist when using the LRU policy.
+*  **plugin.cache_promote.${remap-identifier}.lru_size** - Size of the LRU when using the LRU policy.
+*  **plugin.cache_promote.${remap-identifier}.lru_hit** - LRU hit count when using the LRU policy.
+*  **plugin.cache_promote.${remap-identifier}.lru_miss** - LRU miss count when using the LRU policy.
+*  **plugin.cache_promote.${remap-identifier}.lru_vacated** - count of LRU entries removed to make room for a new request.
+*  **plugin.cache_promote.${remap-identifier}.promoted** - count requests promoted, available in all policies.
+*  **plugin.cache_promote.${remap-identifier}.total_requests** - count of all requests.
+
 These two options combined with your usage patterns will control how likely a
 URL is to become promoted to enter the cache.
 

doc/admin-guide/plugins/collapsed_forwarding.en.rst

@@ -79,14 +79,14 @@ on the error by using an internal redirect follow back to itself, essentially
 blocking the request until a response arrives, at which point, relies on
 read-while-writer feature to start downloading the object to all waiting
 clients. The following config parameters are assumed to be set for this
-plugin to work::
+plugin to work:
 
-:ts:cv:`proxy.config.http.cache.open_write_fail_action`        1
-:ts:cv:`proxy.config.cache.enable_read_while_writer`           1
-:ts:cv:`proxy.config.http.number_of_redirections`             10
-:ts:cv:`proxy.config.http.redirect_use_orig_cache_key`         1
-:ts:cv:`proxy.config.http.background_fill_active_timeout`      0
-:ts:cv:`proxy.config.http.background_fill_completed_threshold` 0
+- :ts:cv:`proxy.config.http.cache.open_write_fail_action`        ``1``
+- :ts:cv:`proxy.config.cache.enable_read_while_writer`           ``1``
+- :ts:cv:`proxy.config.http.number_of_redirections`             ``10``
+- :ts:cv:`proxy.config.http.redirect_use_orig_cache_key`         ``1``
+- :ts:cv:`proxy.config.http.background_fill_active_timeout`      ``0``
+- :ts:cv:`proxy.config.http.background_fill_completed_threshold` ``0``
 
 Additionally, given that collapsed forwarding works based on cache write
 lock failure detection, the plugin requires cache to be enabled and ready.
@@ -119,7 +119,7 @@ For a large scale Video Streaming scenario, there’s a combination of a
 large number of revalidations (e.g. media playlists) and cache misses
 (e.g. media segments) that occur for the same file. Traffic Server’s
 RWW works great in collapsing the concurrent requests in such a scenario,
-however, as described in ``_admin-configuration-reducing-origin-requests``,
+however, as described in :ref:`admin-configuration-reducing-origin-requests`,
 Traffic Server’s implementation of RWW has a significant limitation, which
 restricts its ability to invoke RWW only when the response headers are
 already received. This means that any number of concurrent requests for
@@ -185,6 +185,4 @@ retries, allowing to be able to initiate RWW, whenever the response headers
 are received for the request that was allowed to go to the Origin.
 
 
-More details are available at
-
-https://docs.trafficserver.apache.org/en/6.0.x/admin/http-proxy-caching.en.html#reducing-origin-server-requests-avoiding-the-thundering-herd
+More details are available at :ref:`admin-configuration-reducing-origin-requests`

doc/admin-guide/plugins/combo_handler.en.rst

@@ -47,6 +47,16 @@ The arguments in the :file:`plugin.config` line in order represent
 2. The name of the key used for signature verification (disabled by
    default)
 
+3. A colon separated list of headers which, if present on at least one response, will be
+   added to the combo response.
+
+4. The path of a config file with allowed content types of objects to be combined, one per
+   line, without parameters. (Blank lines and comments starting with "#" are ignored.)
+   Parameters in the Content-Type field value will be ignored when
+   checking if they appear in the allowed types.  If the path does not start with "/", the
+   config file must be located in the ATS config directory.  By default, all content types
+   are allowed, but if this file is specified, it must contain at least one content type.
+
 A "-" can be supplied as a value for any of these arguments to request
 default value be applied.