Add session affinity to custom load balancing

[zrdaley]

What this PR does / why we need it:

This PR adds session affinity through cookies to the lua custom load balancing module when dynamic configuration is enabled.

This implementation mimics the current implementation used for sticky sessions in kubernetes/ingress-nginx when dynamic configuration is not enabled (k8 documentation on sticky sessions)

Which issue this PR fixes:

N/A

Special notes for your reviewer:

Test coverage

Two new unit tests were added in test/e2e/lua/dynamic_configuration.go. These tests cover dynamically enabled session affinity when an ingress is defined in one two ways (See A and B below). Additionally, the pre-existing tests in test/e2e/annotations/affinity.go ensure that session affinity still works when dynamic configuration is disabled.

A. Default backend

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: test-ingress
spec:
  backend:
    serviceName: testsvc
    servicePort: 80

Note: Session affinity will not be enabled when using method A. This is because the session affinity annotation being appended to an upstream is dependent on the location. (see code here)

B. Using Rules

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: test
  annotations:
    ingress.kubernetes.io/rewrite-target: /
spec:
  rules:
  - host: foo.bar.com
    http:
      paths:
      - path: /foo
        backend:
          serviceName: s1
          servicePort: 80

Low Level Implementation Details

kubernetes/ingress-nginx implementation

1. If the affinity annotation is set to "cookie" on an ingress then a sticky-{{ $upstream.name }} upstream is defined in the nginx template (see code)
2. The nginx-sticky-module-ng is used to handle sticky upstreams

A nginx module to add a sticky cookie to be always forwarded to the same upstream server...
When the sticky module can't apply, it switchs back to the classic Round Robin Upstream or returns a "Bad Gateway" (depending on the no_fallback flag).

3. The nginx template proxy_pass is built in template.go; if the backend is sticky then it will return sticky-{{ $upstream.name }} and use the upstream defined in step i.

New custom Lua implementation

1. In the lua balancer module the session affinity annotation with the value "cookie" is checked for
2. If it exists, then a cookie named by the session-cookie-name annotation value (default is "route") is checked for
3. If the cookie:
   • Does NOT exist:
     1. The round robin algorithm is used to determine the upstream to forward to *
     2. The upstream is converted to an <IP>:<Port> string and hashed using the session-cookie-hash annotation value (default is "md5")
     3. A new cookie is set with session-cookie-name annotation as the name and the hash as the value
     4. The upstream is stored in the shared lua dictionary named sticky-sessions
        • sticky-sessions stores the following key value pairs: **
          • key: The hash of the cookie
          • value: The raw <IP>:<Port> string of the upstream to be forwarded to
     5. The IP and port of the upstream are set as the current peer
   • DOES exist
     1. The stored cookie is looked up in the shared lua dictionary, sticky-sessions**
     2. The IP and port returned from the dictionary lookup are checked to determine if they're still routable
     3. If the IP and port:
        • ARE routable
          • The IP and port of the upstream are set as the current peer
        • Are NOT routable
          • Follow procedure in Step 3: Does NOT exist

*Any of custom load balancing algorithms can be to balance with session affinity. Defaulting to round robin is just to be consistent with nginx-sticky-module-ng used for session affinity when dynamic configuration is not enabled.

[zrdaley]

/assign @aledbf

[codecov-io]

Codecov Report

Merging #2341 into master will increase coverage by 0.16%.
The diff coverage is 84.61%.

@@            Coverage Diff             @@
##           master    #2341      +/-   ##
==========================================
+ Coverage   37.81%   37.97%   +0.16%     
==========================================
  Files          72       72              
  Lines        5144     5158      +14     
==========================================
+ Hits         1945     1959      +14     
  Misses       2902     2902              
  Partials      297      297

Impacted Files	Coverage Δ
internal/ingress/controller/template/template.go	65.52% <100%> (+0.57%)	⬆️
internal/file/bindata.go	57.42% <83.33%> (+1.63%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 353c631...32ea682. Read the comment docs.

[aledbf]

/approve

[aledbf]

/lgtm

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: aledbf, zrdaley

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [aledbf]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[aledbf]

@zrdaley thanks!