Skip to content

Conversation

@gtenev
Copy link
Contributor

@gtenev gtenev commented Jul 10, 2020

Fixed problems with initialization of cache volumes when at least
one volume is being forced to a specific "exclusive" span.

Problem description

Disks are cleared in the following configuration where volume sizes are
specified using percentages and also one of the volumes is forced to a
specific span (disk):

storage.config:
  /dev/disk1
  /dev/disk2 volume=3 # <- exclusive span forced to a specific volume

volume.config:
  volume=1 scheme=http size=50%
  volume=2 scheme=http size=50%
  volume=3 scheme=http size=512 # <- volume forced to an exclusive span

During the first start ATS identifies the cleared disks and does the following:
(1) creates and spreads new volume 1 and 2 blocks across disk1 and disk2
(2) deletes all volume 1 and 2 blocks from disk2 to make space for volume 3
(3) creates new volume 3 that takes over the whole disk2.

In step (1) volumes are caclulated larger and spread to disk2 only to be
deleted in step (2) to make space for the forced volume 3.

During the initial start the global volume list cp_list would end up
containing "zombie" CacheVol instances which correspond to the volume 1
and 2 blocks deleted from disk2 to make space for the volume 3 and the
mapping of domains to volumes (hosting.config) could end up mapping
to any of the deleted volume blocks.

This problem disappears after restart since cp_list is initialized
from the disks and cp_list will contain only valid CacheVol instances.

The fix

This fix prevents this from happening by making sure all volumes meant
to have "exclusive" disks are created first to make sure span free
spaces are updated correctly and by excluding the size of
the "exclusive" disks from the total cache size used for volume size
calculations when sizes are specified in percentages (volume.config).

Fixed problems with initialization of cache volumes when at least
one volume is being forced to a specific "exclusive" span.

Problem description:
====================
Disks are cleared in the following configuration where volume sizes are
specified using percentages and also one of the volumes is forced to a
specific span (disk):

storage.config:
  /dev/disk1
  /dev/disk2 volume=3 # <- exclusive span forced to a specific volume

volume.config:
  volume=1 scheme=http size=50%
  volume=2 scheme=http size=50%
  volume=3 scheme=http size=512 # <- volume forced to an exclusive span

During the first start ATS identifies the clears disks and does the following:
1. creates and spreads new volume 1 and 2 blocks across disk1 and disk2
2. deletes all volume 1 and 2 blocks from disk2 to make space for volume 3
3. creates new volume 3 that takes over the whole disk2.

In step (1) volumes are caclulated larger and spread to disk2 only to be
deleted in step (2) to make space for the forced volume 3.

During the initial start the global volume list cp_list would end up
containing "zombie" CacheVol instances which corespond to the volume 1
and 2 blocks deleted from disk2 to make space for the volume 3 and the
mapping of domains to volumes (hosting.config) could end up mapping
to any of the deleted volume blocks.

This problem disappears after restart since cp_list will be initialized
from the disks and cp_list will contain only valid CacheVol instances.

The fix:
========
This fix prevents this from happening by making sure all volumes meant
to have "exclusive" disks are created first to make sure span free
spaces are updated correctly and by excluding the size of
the "exclusive" disks from the total cache size used for volume size
calculations when sizes are specified in percentages (volume.config).
@gtenev gtenev added this to the 10.0.0 milestone Jul 10, 2020
@gtenev gtenev self-assigned this Jul 10, 2020
@vmamidi
Copy link
Contributor

vmamidi commented Jul 13, 2020

@scw00 @SolidWallOfCode did you guys get a chance to look at this PR?


/* There should be a single "forced" volume and no other volumes should exist on this "exclusive" disk (span) */
bool found_nonforced_volumes = false;
for (int j = 0; j < (int)gdisks[i]->header->num_volumes; j++) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

static_cast please

@scw00
Copy link
Member

scw00 commented Jul 14, 2020

Is this a feature that adding exclusively disk without clearing all disk ?

@zwoop zwoop added the OnDocs This is for PR currently running, or will run, on the Docs ATS server label Jul 14, 2020
if (gdisks[d_no]->forced_volume_num == config_vol->number) {
CacheVol *new_cp = new CacheVol();
if (nullptr != new_cp) {
new_cp->disk_vols = (DiskVol **)ats_malloc(gndisks * sizeof(DiskVol *));
Copy link
Member

@SolidWallOfCode SolidWallOfCode Jul 14, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No C-style casts. Because it's a void *, static_cast should work. You could try

new_cp->disk_vols = static_cast<decltype(new_cp->disk_vols)>(ats_malloc(...));

to be extra fancy :-P

continue;
}

/* Now, volumes have been either deleted or did not existing to begin with so we need to create them. */
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"or did not exist to begin with"

Copy link
Member

@SolidWallOfCode SolidWallOfCode left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks reasonable, I only have some stylistic requests.

@zwoop zwoop merged commit 17ee97a into apache:master Jul 14, 2020
zwoop pushed a commit that referenced this pull request Jul 14, 2020
Fixed problems with initialization of cache volumes when at least
one volume is being forced to a specific "exclusive" span.

Problem description:
====================
Disks are cleared in the following configuration where volume sizes are
specified using percentages and also one of the volumes is forced to a
specific span (disk):

storage.config:
  /dev/disk1
  /dev/disk2 volume=3 # <- exclusive span forced to a specific volume

volume.config:
  volume=1 scheme=http size=50%
  volume=2 scheme=http size=50%
  volume=3 scheme=http size=512 # <- volume forced to an exclusive span

During the first start ATS identifies the clears disks and does the following:
1. creates and spreads new volume 1 and 2 blocks across disk1 and disk2
2. deletes all volume 1 and 2 blocks from disk2 to make space for volume 3
3. creates new volume 3 that takes over the whole disk2.

In step (1) volumes are caclulated larger and spread to disk2 only to be
deleted in step (2) to make space for the forced volume 3.

During the initial start the global volume list cp_list would end up
containing "zombie" CacheVol instances which corespond to the volume 1
and 2 blocks deleted from disk2 to make space for the volume 3 and the
mapping of domains to volumes (hosting.config) could end up mapping
to any of the deleted volume blocks.

This problem disappears after restart since cp_list will be initialized
from the disks and cp_list will contain only valid CacheVol instances.

The fix:
========
This fix prevents this from happening by making sure all volumes meant
to have "exclusive" disks are created first to make sure span free
spaces are updated correctly and by excluding the size of
the "exclusive" disks from the total cache size used for volume size
calculations when sizes are specified in percentages (volume.config).

(cherry picked from commit 17ee97a)
@zwoop
Copy link
Contributor

zwoop commented Jul 14, 2020

Cherry-picked to v9.0.x branch.

@zwoop zwoop modified the milestones: 10.0.0, 9.0.0 Jul 14, 2020
@gtenev
Copy link
Contributor Author

gtenev commented Jul 14, 2020

@scw00 and @SolidWallOfCode thanks for the review!

The C-style casts were remnants of old code, C-style casts are all over the place in Cache.cc. Since this PR got merged before I pushed the fix, in the interest of time I may address your "stylistic request" in a following PR.

@gtenev
Copy link
Contributor Author

gtenev commented Jul 14, 2020

@zwoop here is the 8.1.x back port PR #7001

zwoop pushed a commit that referenced this pull request Jul 15, 2020
Fixed problems with initialization of cache volumes when at least
one volume is being forced to a specific "exclusive" span.

Problem description:
====================
Disks are cleared in the following configuration where volume sizes are
specified using percentages and also one of the volumes is forced to a
specific span (disk):

storage.config:
  /dev/disk1
  /dev/disk2 volume=3 # <- exclusive span forced to a specific volume

volume.config:
  volume=1 scheme=http size=50%
  volume=2 scheme=http size=50%
  volume=3 scheme=http size=512 # <- volume forced to an exclusive span

During the first start ATS identifies the clears disks and does the following:
1. creates and spreads new volume 1 and 2 blocks across disk1 and disk2
2. deletes all volume 1 and 2 blocks from disk2 to make space for volume 3
3. creates new volume 3 that takes over the whole disk2.

In step (1) volumes are caclulated larger and spread to disk2 only to be
deleted in step (2) to make space for the forced volume 3.

During the initial start the global volume list cp_list would end up
containing "zombie" CacheVol instances which corespond to the volume 1
and 2 blocks deleted from disk2 to make space for the volume 3 and the
mapping of domains to volumes (hosting.config) could end up mapping
to any of the deleted volume blocks.

This problem disappears after restart since cp_list will be initialized
from the disks and cp_list will contain only valid CacheVol instances.

The fix:
========
This fix prevents this from happening by making sure all volumes meant
to have "exclusive" disks are created first to make sure span free
spaces are updated correctly and by excluding the size of
the "exclusive" disks from the total cache size used for volume size
calculations when sizes are specified in percentages (volume.config).

(cherry picked from commit 17ee97a)
@zwoop zwoop modified the milestones: 9.0.0, 8.1.0 Jul 15, 2020
masaori335 pushed a commit to masaori335/trafficserver that referenced this pull request Jul 21, 2020
* asf/9.0.x:
  Fix volume/stripe calcs when using forced volumes (apache#6995)
maskit added a commit that referenced this pull request Jul 27, 2020
* master:
  Make tls_conn_timeout test more reliable in CI (#7018)
  Remove deprecated verify.server for 9.0 (#7040)
  Updated GitHub description and homepage URL to be https (#7019)
  Add virtual destructor to QUICTPConfig. (#7036)
  Fix code to eliminate warning and enable feature (#7031)
  add a null check to avoid crashing (#7035)
  Squashed commit of the following: (#7000)
  Fixed problem with all "forced" volumes cache (#7028)
  Spacing tweaks to acl_filter_rule::print (#7026)
  Removes dead code from iocore/dns (#7025)
  Removes TODO (#7027)
  Add logic to resolve content-length transfer-encoding conflicts on response (#6992)
  Add memory_profile plugin (#7014)
  Fix typos relating to tls_bridge (#7011)
  slice: clean up of created 502 response header (#6919)
  Add new API / TSPluginDSOReloadEnable that overrides the configuration variable `proxy.config.plugin.dynamic_reload_mode` for a particular plugin. (#6880)
  Remove incorrect assert in inactivity timeout handling (#7012)
  Removes use of SPLIT_DNS macro (#7010)
  Fixed core when sending back a redirect and having an invalid server response (#7004)
  slice: fix throttle not work (#7008)
  Updates to thread scale factor (#7007)
  Added tasks and launch files for vscode, to configure, build and debug (#7005)
  NextHop Strategy Refactor and Fixes (#6782)
  Make the setting of the continuation handler safer. (#6996)
  ProtocolStack n -> count (#7006)
  Fix volume/stripe calcs when using forced volumes (#6995)
  Cleanup: Write error message on diags output instead of stderr (#6997)

 Conflicts:
	iocore/net/P_QUICNetVConnection.h
	iocore/net/P_QUICPacketHandler.h
	iocore/net/QUICNetProcessor.cc
	iocore/net/QUICNetVConnection.cc
	iocore/net/QUICPacketHandler.cc
	iocore/net/quic/Mock.h
	iocore/net/quic/QUICCongestionController.h
	iocore/net/quic/QUICContext.cc
	iocore/net/quic/QUICContext.h
	iocore/net/quic/QUICDebugNames.cc
	iocore/net/quic/QUICFrame.cc
	iocore/net/quic/QUICHandshake.cc
	iocore/net/quic/QUICKeyGenerator.h
	iocore/net/quic/QUICLossDetector.cc
	iocore/net/quic/QUICLossDetector.h
	iocore/net/quic/QUICNewRenoCongestionController.cc
	iocore/net/quic/QUICPacket.cc
	iocore/net/quic/QUICPacket.h
	iocore/net/quic/QUICPacketFactory.cc
	iocore/net/quic/QUICPacketFactory.h
	iocore/net/quic/QUICRetryIntegrityTag.cc
	iocore/net/quic/QUICRetryIntegrityTag.h
	iocore/net/quic/QUICTLS.h
	iocore/net/quic/QUICTLS_boringssl.cc
	iocore/net/quic/QUICTLS_openssl.cc
	iocore/net/quic/QUICTransportParameters.cc
	iocore/net/quic/QUICTransportParameters.h
	iocore/net/quic/QUICTypes.cc
	iocore/net/quic/QUICTypes.h
	iocore/net/quic/qlog/QLogListener.h
	iocore/net/quic/test/test_QUICHandshakeProtocol.cc
	iocore/net/quic/test/test_QUICLossDetector.cc
	iocore/net/quic/test/test_QUICPacket.cc
	iocore/net/quic/test/test_QUICPacketFactory.cc
	iocore/net/quic/test/test_QUICPacketHeaderProtector.cc
	iocore/net/quic/test/test_QUICStreamManager.cc
	iocore/net/quic/test/test_QUICVersionNegotiator.cc
	proxy/http/HttpProxyServerMain.cc
	src/traffic_quic/quic_client.cc
	src/tscore/ink_inet.cc
maskit added a commit to maskit/trafficserver that referenced this pull request Aug 12, 2020
commit 8c114c0
Merge: f39e397 b527464
Author: Masakazu Kitajo <maskit@apache.org>
Date:   Thu Aug 13 07:31:56 2020 +0900

    Merge branch 'master' into quic-latest

    * master:
      Adding autopep8 as a pre-commit hook. (apache#7071)
      Refresh proxy protocol diagram (apache#7095)
      Skip docs builds if there are no changes in the doc dir and files it includes (apache#7088)
      Remove more deadcode (apache#7098)
      destroy threads after job done (apache#7083)
      Fix compilation error - missing '&' operator (apache#7093)
      Adds description for ssl_ticket_number in ssl_multicert docs (apache#7091)
      Ran clang-tidy over the source tree (apache#7077)
      Move the direct self loop check later to HttpSM::do_http_server_open just before connection upstream. (apache#7069)

commit f39e397
Merge: d09a757 2219cee
Author: Masakazu Kitajo <maskit@apache.org>
Date:   Thu Aug 6 09:08:46 2020 +0900

    Merge branch 'master' into quic-latest

    * master:
      Signal WRITE_COMPLETE regardless of transmission progress (apache#7062)
      Converts files to #pragma once (apache#7089)
      Fix eval_ip compile - missing const. (apache#7087)
      Fix a crash on connection migration to the advertised preferred address (apache#7080)
      Update and run the autopep8 make target (apache#7070)
      Fixes no_content_length status code description in docs (apache#7086)
      const-ify quic/http3 code (apache#7084)
      Fixes build warnings in maxmind_acl (apache#7085)
      Add TS_USE_QUIC to traffic_layout info (apache#7074)
      Added support for out of tree builds with vscode (apache#7072)
      constify Print() methods and other low hanging fruit (apache#7068)
      Updating to AuTest 1.8.1. (apache#7065)
      Use system include-style for STL and OpenSSL headers (apache#7066)
      tests: gitignore ssl-delay-server binary (apache#7067)

     Conflicts:
    	iocore/net/quic/QUICLossDetector.cc
    	iocore/net/quic/QUICLossDetector.h

commit d09a757
Author: Randall Meyer <rrm@apache.org>
Date:   Tue Aug 4 13:52:53 2020 -0700

    const-ify quic/http3 code (apache#7079)

commit 84e4c8e
Merge: 50937c0 3087f16
Author: Masakazu Kitajo <maskit@apache.org>
Date:   Fri Jul 31 10:24:13 2020 +0900

    Merge branch 'master' into quic-latest

    * master:
      Fix a crash on active timeout on QUIC connections (apache#7059)
      Don't make an error on receiving retransmitted handshake data (apache#7061)
      Document proxy.config.http.cache.post_method. (apache#7060)
      Quote out lists of servers and domains in splitdns.config example (apache#7057)
      Fix proxy.process.http.current_client_connections (apache#7056)
      Fixed CLIENT-URL to use the pristine client URL (apache#7050)
      Removes FIXME that is unlikely to be fixed at this point in the project history (apache#7058)
      Move to denylists and allowlists (apache#7034)
      Avoid unnecessary copying of STL map for QUICTPConfigQCP class. (apache#7039)

commit 50937c0
Merge: f6e1744 9467a2c
Author: Masakazu Kitajo <maskit@apache.org>
Date:   Tue Jul 28 10:08:26 2020 +0900

    Merge branch 'master' into quic-latest

    * master:
      Fixes spelling/license formatting in traffic_dump plugin (apache#7047)
      Fixes spelling in docs (apache#7048)
      Fixes spelling H3-related code (apache#7046)
      Cleans up various versions checks (apache#7049)
      Fix a typo (apache#7043)

     Conflicts:
    	iocore/net/quic/QUICHandshake.cc
    	iocore/net/quic/QUICLossDetector.cc
    	iocore/net/quic/QUICNewRenoCongestionController.cc
    	iocore/net/quic/test/test_QUICPacketFactory.cc

commit f6e1744
Merge: 83e1da7 549c626
Author: Masakazu Kitajo <maskit@apache.org>
Date:   Mon Jul 27 11:11:58 2020 +0900

    Merge branch 'master' into quic-latest

    * master:
      Make tls_conn_timeout test more reliable in CI (apache#7018)
      Remove deprecated verify.server for 9.0 (apache#7040)
      Updated GitHub description and homepage URL to be https (apache#7019)
      Add virtual destructor to QUICTPConfig. (apache#7036)
      Fix code to eliminate warning and enable feature (apache#7031)
      add a null check to avoid crashing (apache#7035)
      Squashed commit of the following: (apache#7000)
      Fixed problem with all "forced" volumes cache (apache#7028)
      Spacing tweaks to acl_filter_rule::print (apache#7026)
      Removes dead code from iocore/dns (apache#7025)
      Removes TODO (apache#7027)
      Add logic to resolve content-length transfer-encoding conflicts on response (apache#6992)
      Add memory_profile plugin (apache#7014)
      Fix typos relating to tls_bridge (apache#7011)
      slice: clean up of created 502 response header (apache#6919)
      Add new API / TSPluginDSOReloadEnable that overrides the configuration variable `proxy.config.plugin.dynamic_reload_mode` for a particular plugin. (apache#6880)
      Remove incorrect assert in inactivity timeout handling (apache#7012)
      Removes use of SPLIT_DNS macro (apache#7010)
      Fixed core when sending back a redirect and having an invalid server response (apache#7004)
      slice: fix throttle not work (apache#7008)
      Updates to thread scale factor (apache#7007)
      Added tasks and launch files for vscode, to configure, build and debug (apache#7005)
      NextHop Strategy Refactor and Fixes (apache#6782)
      Make the setting of the continuation handler safer. (apache#6996)
      ProtocolStack n -> count (apache#7006)
      Fix volume/stripe calcs when using forced volumes (apache#6995)
      Cleanup: Write error message on diags output instead of stderr (apache#6997)

     Conflicts:
    	iocore/net/P_QUICNetVConnection.h
    	iocore/net/P_QUICPacketHandler.h
    	iocore/net/QUICNetProcessor.cc
    	iocore/net/QUICNetVConnection.cc
    	iocore/net/QUICPacketHandler.cc
    	iocore/net/quic/Mock.h
    	iocore/net/quic/QUICCongestionController.h
    	iocore/net/quic/QUICContext.cc
    	iocore/net/quic/QUICContext.h
    	iocore/net/quic/QUICDebugNames.cc
    	iocore/net/quic/QUICFrame.cc
    	iocore/net/quic/QUICHandshake.cc
    	iocore/net/quic/QUICKeyGenerator.h
    	iocore/net/quic/QUICLossDetector.cc
    	iocore/net/quic/QUICLossDetector.h
    	iocore/net/quic/QUICNewRenoCongestionController.cc
    	iocore/net/quic/QUICPacket.cc
    	iocore/net/quic/QUICPacket.h
    	iocore/net/quic/QUICPacketFactory.cc
    	iocore/net/quic/QUICPacketFactory.h
    	iocore/net/quic/QUICRetryIntegrityTag.cc
    	iocore/net/quic/QUICRetryIntegrityTag.h
    	iocore/net/quic/QUICTLS.h
    	iocore/net/quic/QUICTLS_boringssl.cc
    	iocore/net/quic/QUICTLS_openssl.cc
    	iocore/net/quic/QUICTransportParameters.cc
    	iocore/net/quic/QUICTransportParameters.h
    	iocore/net/quic/QUICTypes.cc
    	iocore/net/quic/QUICTypes.h
    	iocore/net/quic/qlog/QLogListener.h
    	iocore/net/quic/test/test_QUICHandshakeProtocol.cc
    	iocore/net/quic/test/test_QUICLossDetector.cc
    	iocore/net/quic/test/test_QUICPacket.cc
    	iocore/net/quic/test/test_QUICPacketFactory.cc
    	iocore/net/quic/test/test_QUICPacketHeaderProtector.cc
    	iocore/net/quic/test/test_QUICStreamManager.cc
    	iocore/net/quic/test/test_QUICVersionNegotiator.cc
    	proxy/http/HttpProxyServerMain.cc
    	src/traffic_quic/quic_client.cc
    	src/tscore/ink_inet.cc

commit 83e1da7
Author: Masakazu Kitajo <maskit@apache.org>
Date:   Wed Jul 22 23:26:56 2020 +0900

    Fix a crash on path validation

commit c74bd89
Author: Masakazu Kitajo <maskit@apache.org>
Date:   Wed Jul 22 23:19:05 2020 +0900

    Fix a crash on traffic_quic

commit b4a0c8c
Author: Masakazu Kitajo <maskit@apache.org>
Date:   Wed Jul 22 16:37:02 2020 +0900

    Update Congestion Control logic to draft-29

commit 53da240
Author: Masakazu Kitajo <maskit@apache.org>
Date:   Wed Jul 22 11:40:18 2020 +0900

    Adjust debug log verbosity

commit 92a34b0
Author: Masakazu Kitajo <maskit@apache.org>
Date:   Tue Jul 21 12:04:00 2020 +0900

    Fix infinite PING

commit 42c4054
Author: Masakazu Kitajo <maskit@apache.org>
Date:   Thu Jul 9 10:16:04 2020 +0900

    Update Loss Detection logic to draft-29

commit e8109c0
Author: Masakazu Kitajo <maskit@apache.org>
Date:   Fri Jun 26 10:57:42 2020 +0900

    Update tests

commit 3a8cb4a
Author: Masakazu Kitajo <maskit@apache.org>
Date:   Thu Jun 25 14:58:47 2020 +0900

    Use different keys and nonces for Draft-27 and Draft-29

commit 37af625
Author: Masakazu Kitajo <maskit@apache.org>
Date:   Thu Jun 25 14:14:57 2020 +0900

    Use different salts for Draft-27 and Draft-29

commit 52e57fe
Author: Masakazu Kitajo <maskit@apache.org>
Date:   Mon Jun 22 11:53:43 2020 +0900

    Rename SERVER_BUSY to CONNECTION_REFUSED

commit e5fb638
Author: Masakazu Kitajo <maskit@apache.org>
Date:   Mon Jun 22 11:50:38 2020 +0900

    Update QUIC draft version numbers to 29

commit a9ef9b0
Author: Masakazu Kitajo <maskit@apache.org>
Date:   Fri Jul 10 17:21:46 2020 +0900

    Update code for BoringSSL

commit 58cac98
Author: Masakazu Kitajo <maskit@apache.org>
Date:   Thu Jun 4 11:24:16 2020 +0900

    Don't include exercise version number provided by a client into VN packet

commit 47e1eb7
Author: Masakazu Kitajo <maskit@apache.org>
Date:   Thu Jun 4 10:45:14 2020 +0900

    Updates tests and fixes a couple of typoes

commit f41ca7b
Author: Masakazu Kitajo <maskit@apache.org>
Date:   Mon Jun 1 12:46:32 2020 +0900

    Support both QUIC draft-27 and draft-28

commit 3a5b3c7
Author: Masakazu Kitajo <maskit@apache.org>
Date:   Thu May 28 13:57:57 2020 +0900

    Add QUIC APPLICATION_ERROR error code

commit 0457e74
Author: Masakazu Kitajo <maskit@apache.org>
Date:   Thu May 28 13:33:02 2020 +0900

    Add support for new Transport Parameters on draft-28

commit 2717ecc
Author: Masakazu Kitajo <maskit@apache.org>
Date:   Mon May 25 11:46:49 2020 +0900

    Rename QUIC Transport Parameters

    ORIGINAL_CONNECTION_ID -> ORIGINAL_DESTINATION_CONNECTION_ID
    MAX_PACKET_SIZE -> MAX_UDP_PAYLOAD_SIZE

commit 9ef2167
Author: Masakazu Kitajo <maskit@apache.org>
Date:   Mon May 25 10:34:54 2020 +0900

    Update QUIC draft version numbers to 28
maskit added a commit to maskit/trafficserver that referenced this pull request Aug 12, 2020
commit 4d579f4
Author: Masakazu Kitajo <maskit@apache.org>
Date:   Thu Aug 13 08:32:58 2020 +0900

    Fix docs

commit 8c114c0
Merge: f39e397 b527464
Author: Masakazu Kitajo <maskit@apache.org>
Date:   Thu Aug 13 07:31:56 2020 +0900

    Merge branch 'master' into quic-latest

    * master:
      Adding autopep8 as a pre-commit hook. (apache#7071)
      Refresh proxy protocol diagram (apache#7095)
      Skip docs builds if there are no changes in the doc dir and files it includes (apache#7088)
      Remove more deadcode (apache#7098)
      destroy threads after job done (apache#7083)
      Fix compilation error - missing '&' operator (apache#7093)
      Adds description for ssl_ticket_number in ssl_multicert docs (apache#7091)
      Ran clang-tidy over the source tree (apache#7077)
      Move the direct self loop check later to HttpSM::do_http_server_open just before connection upstream. (apache#7069)

commit f39e397
Merge: d09a757 2219cee
Author: Masakazu Kitajo <maskit@apache.org>
Date:   Thu Aug 6 09:08:46 2020 +0900

    Merge branch 'master' into quic-latest

    * master:
      Signal WRITE_COMPLETE regardless of transmission progress (apache#7062)
      Converts files to #pragma once (apache#7089)
      Fix eval_ip compile - missing const. (apache#7087)
      Fix a crash on connection migration to the advertised preferred address (apache#7080)
      Update and run the autopep8 make target (apache#7070)
      Fixes no_content_length status code description in docs (apache#7086)
      const-ify quic/http3 code (apache#7084)
      Fixes build warnings in maxmind_acl (apache#7085)
      Add TS_USE_QUIC to traffic_layout info (apache#7074)
      Added support for out of tree builds with vscode (apache#7072)
      constify Print() methods and other low hanging fruit (apache#7068)
      Updating to AuTest 1.8.1. (apache#7065)
      Use system include-style for STL and OpenSSL headers (apache#7066)
      tests: gitignore ssl-delay-server binary (apache#7067)

     Conflicts:
    	iocore/net/quic/QUICLossDetector.cc
    	iocore/net/quic/QUICLossDetector.h

commit d09a757
Author: Randall Meyer <rrm@apache.org>
Date:   Tue Aug 4 13:52:53 2020 -0700

    const-ify quic/http3 code (apache#7079)

commit 84e4c8e
Merge: 50937c0 3087f16
Author: Masakazu Kitajo <maskit@apache.org>
Date:   Fri Jul 31 10:24:13 2020 +0900

    Merge branch 'master' into quic-latest

    * master:
      Fix a crash on active timeout on QUIC connections (apache#7059)
      Don't make an error on receiving retransmitted handshake data (apache#7061)
      Document proxy.config.http.cache.post_method. (apache#7060)
      Quote out lists of servers and domains in splitdns.config example (apache#7057)
      Fix proxy.process.http.current_client_connections (apache#7056)
      Fixed CLIENT-URL to use the pristine client URL (apache#7050)
      Removes FIXME that is unlikely to be fixed at this point in the project history (apache#7058)
      Move to denylists and allowlists (apache#7034)
      Avoid unnecessary copying of STL map for QUICTPConfigQCP class. (apache#7039)

commit 50937c0
Merge: f6e1744 9467a2c
Author: Masakazu Kitajo <maskit@apache.org>
Date:   Tue Jul 28 10:08:26 2020 +0900

    Merge branch 'master' into quic-latest

    * master:
      Fixes spelling/license formatting in traffic_dump plugin (apache#7047)
      Fixes spelling in docs (apache#7048)
      Fixes spelling H3-related code (apache#7046)
      Cleans up various versions checks (apache#7049)
      Fix a typo (apache#7043)

     Conflicts:
    	iocore/net/quic/QUICHandshake.cc
    	iocore/net/quic/QUICLossDetector.cc
    	iocore/net/quic/QUICNewRenoCongestionController.cc
    	iocore/net/quic/test/test_QUICPacketFactory.cc

commit f6e1744
Merge: 83e1da7 549c626
Author: Masakazu Kitajo <maskit@apache.org>
Date:   Mon Jul 27 11:11:58 2020 +0900

    Merge branch 'master' into quic-latest

    * master:
      Make tls_conn_timeout test more reliable in CI (apache#7018)
      Remove deprecated verify.server for 9.0 (apache#7040)
      Updated GitHub description and homepage URL to be https (apache#7019)
      Add virtual destructor to QUICTPConfig. (apache#7036)
      Fix code to eliminate warning and enable feature (apache#7031)
      add a null check to avoid crashing (apache#7035)
      Squashed commit of the following: (apache#7000)
      Fixed problem with all "forced" volumes cache (apache#7028)
      Spacing tweaks to acl_filter_rule::print (apache#7026)
      Removes dead code from iocore/dns (apache#7025)
      Removes TODO (apache#7027)
      Add logic to resolve content-length transfer-encoding conflicts on response (apache#6992)
      Add memory_profile plugin (apache#7014)
      Fix typos relating to tls_bridge (apache#7011)
      slice: clean up of created 502 response header (apache#6919)
      Add new API / TSPluginDSOReloadEnable that overrides the configuration variable `proxy.config.plugin.dynamic_reload_mode` for a particular plugin. (apache#6880)
      Remove incorrect assert in inactivity timeout handling (apache#7012)
      Removes use of SPLIT_DNS macro (apache#7010)
      Fixed core when sending back a redirect and having an invalid server response (apache#7004)
      slice: fix throttle not work (apache#7008)
      Updates to thread scale factor (apache#7007)
      Added tasks and launch files for vscode, to configure, build and debug (apache#7005)
      NextHop Strategy Refactor and Fixes (apache#6782)
      Make the setting of the continuation handler safer. (apache#6996)
      ProtocolStack n -> count (apache#7006)
      Fix volume/stripe calcs when using forced volumes (apache#6995)
      Cleanup: Write error message on diags output instead of stderr (apache#6997)

     Conflicts:
    	iocore/net/P_QUICNetVConnection.h
    	iocore/net/P_QUICPacketHandler.h
    	iocore/net/QUICNetProcessor.cc
    	iocore/net/QUICNetVConnection.cc
    	iocore/net/QUICPacketHandler.cc
    	iocore/net/quic/Mock.h
    	iocore/net/quic/QUICCongestionController.h
    	iocore/net/quic/QUICContext.cc
    	iocore/net/quic/QUICContext.h
    	iocore/net/quic/QUICDebugNames.cc
    	iocore/net/quic/QUICFrame.cc
    	iocore/net/quic/QUICHandshake.cc
    	iocore/net/quic/QUICKeyGenerator.h
    	iocore/net/quic/QUICLossDetector.cc
    	iocore/net/quic/QUICLossDetector.h
    	iocore/net/quic/QUICNewRenoCongestionController.cc
    	iocore/net/quic/QUICPacket.cc
    	iocore/net/quic/QUICPacket.h
    	iocore/net/quic/QUICPacketFactory.cc
    	iocore/net/quic/QUICPacketFactory.h
    	iocore/net/quic/QUICRetryIntegrityTag.cc
    	iocore/net/quic/QUICRetryIntegrityTag.h
    	iocore/net/quic/QUICTLS.h
    	iocore/net/quic/QUICTLS_boringssl.cc
    	iocore/net/quic/QUICTLS_openssl.cc
    	iocore/net/quic/QUICTransportParameters.cc
    	iocore/net/quic/QUICTransportParameters.h
    	iocore/net/quic/QUICTypes.cc
    	iocore/net/quic/QUICTypes.h
    	iocore/net/quic/qlog/QLogListener.h
    	iocore/net/quic/test/test_QUICHandshakeProtocol.cc
    	iocore/net/quic/test/test_QUICLossDetector.cc
    	iocore/net/quic/test/test_QUICPacket.cc
    	iocore/net/quic/test/test_QUICPacketFactory.cc
    	iocore/net/quic/test/test_QUICPacketHeaderProtector.cc
    	iocore/net/quic/test/test_QUICStreamManager.cc
    	iocore/net/quic/test/test_QUICVersionNegotiator.cc
    	proxy/http/HttpProxyServerMain.cc
    	src/traffic_quic/quic_client.cc
    	src/tscore/ink_inet.cc

commit 83e1da7
Author: Masakazu Kitajo <maskit@apache.org>
Date:   Wed Jul 22 23:26:56 2020 +0900

    Fix a crash on path validation

commit c74bd89
Author: Masakazu Kitajo <maskit@apache.org>
Date:   Wed Jul 22 23:19:05 2020 +0900

    Fix a crash on traffic_quic

commit b4a0c8c
Author: Masakazu Kitajo <maskit@apache.org>
Date:   Wed Jul 22 16:37:02 2020 +0900

    Update Congestion Control logic to draft-29

commit 53da240
Author: Masakazu Kitajo <maskit@apache.org>
Date:   Wed Jul 22 11:40:18 2020 +0900

    Adjust debug log verbosity

commit 92a34b0
Author: Masakazu Kitajo <maskit@apache.org>
Date:   Tue Jul 21 12:04:00 2020 +0900

    Fix infinite PING

commit 42c4054
Author: Masakazu Kitajo <maskit@apache.org>
Date:   Thu Jul 9 10:16:04 2020 +0900

    Update Loss Detection logic to draft-29

commit e8109c0
Author: Masakazu Kitajo <maskit@apache.org>
Date:   Fri Jun 26 10:57:42 2020 +0900

    Update tests

commit 3a8cb4a
Author: Masakazu Kitajo <maskit@apache.org>
Date:   Thu Jun 25 14:58:47 2020 +0900

    Use different keys and nonces for Draft-27 and Draft-29

commit 37af625
Author: Masakazu Kitajo <maskit@apache.org>
Date:   Thu Jun 25 14:14:57 2020 +0900

    Use different salts for Draft-27 and Draft-29

commit 52e57fe
Author: Masakazu Kitajo <maskit@apache.org>
Date:   Mon Jun 22 11:53:43 2020 +0900

    Rename SERVER_BUSY to CONNECTION_REFUSED

commit e5fb638
Author: Masakazu Kitajo <maskit@apache.org>
Date:   Mon Jun 22 11:50:38 2020 +0900

    Update QUIC draft version numbers to 29

commit a9ef9b0
Author: Masakazu Kitajo <maskit@apache.org>
Date:   Fri Jul 10 17:21:46 2020 +0900

    Update code for BoringSSL

commit 58cac98
Author: Masakazu Kitajo <maskit@apache.org>
Date:   Thu Jun 4 11:24:16 2020 +0900

    Don't include exercise version number provided by a client into VN packet

commit 47e1eb7
Author: Masakazu Kitajo <maskit@apache.org>
Date:   Thu Jun 4 10:45:14 2020 +0900

    Updates tests and fixes a couple of typoes

commit f41ca7b
Author: Masakazu Kitajo <maskit@apache.org>
Date:   Mon Jun 1 12:46:32 2020 +0900

    Support both QUIC draft-27 and draft-28

commit 3a5b3c7
Author: Masakazu Kitajo <maskit@apache.org>
Date:   Thu May 28 13:57:57 2020 +0900

    Add QUIC APPLICATION_ERROR error code

commit 0457e74
Author: Masakazu Kitajo <maskit@apache.org>
Date:   Thu May 28 13:33:02 2020 +0900

    Add support for new Transport Parameters on draft-28

commit 2717ecc
Author: Masakazu Kitajo <maskit@apache.org>
Date:   Mon May 25 11:46:49 2020 +0900

    Rename QUIC Transport Parameters

    ORIGINAL_CONNECTION_ID -> ORIGINAL_DESTINATION_CONNECTION_ID
    MAX_PACKET_SIZE -> MAX_UDP_PAYLOAD_SIZE

commit 9ef2167
Author: Masakazu Kitajo <maskit@apache.org>
Date:   Mon May 25 10:34:54 2020 +0900

    Update QUIC draft version numbers to 28
brbzull0 pushed a commit to brbzull0/trafficserver that referenced this pull request Sep 2, 2020
Fixed problems with initialization of cache volumes when at least
one volume is being forced to a specific "exclusive" span.

Problem description:
====================
Disks are cleared in the following configuration where volume sizes are
specified using percentages and also one of the volumes is forced to a
specific span (disk):

storage.config:
  /dev/disk1
  /dev/disk2 volume=3 # <- exclusive span forced to a specific volume

volume.config:
  volume=1 scheme=http size=50%
  volume=2 scheme=http size=50%
  volume=3 scheme=http size=512 # <- volume forced to an exclusive span

During the first start ATS identifies the clears disks and does the following:
1. creates and spreads new volume 1 and 2 blocks across disk1 and disk2
2. deletes all volume 1 and 2 blocks from disk2 to make space for volume 3
3. creates new volume 3 that takes over the whole disk2.

In step (1) volumes are caclulated larger and spread to disk2 only to be
deleted in step (2) to make space for the forced volume 3.

During the initial start the global volume list cp_list would end up
containing "zombie" CacheVol instances which corespond to the volume 1
and 2 blocks deleted from disk2 to make space for the volume 3 and the
mapping of domains to volumes (hosting.config) could end up mapping
to any of the deleted volume blocks.

This problem disappears after restart since cp_list will be initialized
from the disks and cp_list will contain only valid CacheVol instances.

The fix:
========
This fix prevents this from happening by making sure all volumes meant
to have "exclusive" disks are created first to make sure span free
spaces are updated correctly and by excluding the size of
the "exclusive" disks from the total cache size used for volume size
calculations when sizes are specified in percentages (volume.config).
whutwhu pushed a commit to whutwhu/trafficserver that referenced this pull request Nov 1, 2020
Fixed problems with initialization of cache volumes when at least
one volume is being forced to a specific "exclusive" span.

Problem description:
====================
Disks are cleared in the following configuration where volume sizes are
specified using percentages and also one of the volumes is forced to a
specific span (disk):

storage.config:
  /dev/disk1
  /dev/disk2 volume=3 # <- exclusive span forced to a specific volume

volume.config:
  volume=1 scheme=http size=50%
  volume=2 scheme=http size=50%
  volume=3 scheme=http size=512 # <- volume forced to an exclusive span

During the first start ATS identifies the clears disks and does the following:
1. creates and spreads new volume 1 and 2 blocks across disk1 and disk2
2. deletes all volume 1 and 2 blocks from disk2 to make space for volume 3
3. creates new volume 3 that takes over the whole disk2.

In step (1) volumes are caclulated larger and spread to disk2 only to be
deleted in step (2) to make space for the forced volume 3.

During the initial start the global volume list cp_list would end up
containing "zombie" CacheVol instances which corespond to the volume 1
and 2 blocks deleted from disk2 to make space for the volume 3 and the
mapping of domains to volumes (hosting.config) could end up mapping
to any of the deleted volume blocks.

This problem disappears after restart since cp_list will be initialized
from the disks and cp_list will contain only valid CacheVol instances.

The fix:
========
This fix prevents this from happening by making sure all volumes meant
to have "exclusive" disks are created first to make sure span free
spaces are updated correctly and by excluding the size of
the "exclusive" disks from the total cache size used for volume size
calculations when sizes are specified in percentages (volume.config).
masaori335 pushed a commit to masaori335/trafficserver that referenced this pull request Mar 31, 2021
* asf/8.1.x:
  Updated Changes
  Fix volume/stripe calcs when using forced volumes (apache#6995) (apache#7001)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Bug Cache Core OnDocs This is for PR currently running, or will run, on the Docs ATS server

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants