Description
Preconditions
I've been reading about people using Magento2 and the size of the X-Magento-Tags header.
Steps to reproduce
For example http://www.maxbucknell.com/blog/2016/2/10/troubleshooting-varnish-with-magento-2 writes:
The rub here is that Magento adds a tag for every product in a category. And the tag is of the form "catalog_product_{{PRODUCT_ID}}".
Actual result
That's an integer and 20 bytes of string per product, which gives some users headers several kB in size, when many products are in one category. In that case, the author describes hitting Varnish's limit at 400 products.
Expected result
I work for a CDN where some of our users use Magento, and we cache their content with Varnish. Increasing Varnish's http_resp_hdr_len size may suit that one person's situation, but it is infeasible for us, because it would affect all of our users.
Proposed solution
I have a suggestion to offer, for how to use header space more efficiently:
Each product ID is an integer. The X-Magento-Tags header describes the presence or absence of products only. That information can be done with a single bit per product. The index of the bit would give the product ID.
That's usually known as a bit vector, or a bitfield, or a bitmap. My suggestion is to switch the header to use a bitmap, encoded in hex. That'd take one bit per product, instead of over 20 bytes.
More on bitmaps: https://en.wikipedia.org/wiki/Bit_array