Skip to content

Speed up CheckIngress in admission hook by eliminating the deep copy #7297

@cgorbit

Description

@cgorbit

Hi,

In the company where I working we have several k8s clusters and one dedicated to developers personal stands.
k8s is working in aws ec2 nodes.

Developer's cluster's nginx ingress controller config contains 120 server sections, 5271 location sections and have 19MB size as text config.

On such big config amission hook for each ingress update becomes too slow, it lasts about 7 seconds now. And it was even slower before I speed it up by rewriting bash scripts (!) which were used inside CheckIngress (#7076).

But it's still very slow. So we disabled admission hooks, because when somebody creates new developer stand admission hooks must be evaluated many times and sometimes we got error from master that admission hook finished with time out (of 30s).

I found next place which take now about 3s on my config:

// copy location before any change
el, err := copystructure.Copy(location)
if err != nil {
klog.ErrorS(err, "copying location")
}

As far as I understand deep copy is redundant in this case. Shallow copy will be enough.

e2e test: https://gist.github.com/cgorbit/431086b0e78f1a8b75cd497235ae8d51

diff --git a/internal/ingress/controller/location.go b/internal/ingress/controller/location.go
index a6b3d2b14..d40e88960 100644
--- a/internal/ingress/controller/location.go
+++ b/internal/ingress/controller/location.go
@@ -20,10 +20,8 @@ import (
        "fmt"
        "strings"
 
-       "github.com/mitchellh/copystructure"
        networking "k8s.io/api/networking/v1"
        "k8s.io/ingress-nginx/internal/ingress"
-       "k8s.io/klog/v2"
 )
 
 var (
@@ -73,18 +71,14 @@ func updateServerLocations(locations []*ingress.Location) []*ingress.Location {
                        continue
                }
 
-               // copy location before any change
-               el, err := copystructure.Copy(location)
-               if err != nil {
-                       klog.ErrorS(err, "copying location")
-               }
+               var el ingress.Location = *location
 
                // normalize path. Must end in /
                location.Path = normalizePrefixPath(location.Path)
                newLocations = append(newLocations, location)
 
                // add exact location
-               exactLocation := el.(*ingress.Location)
+               exactLocation := &el
                exactLocation.PathType = &pathTypeExact
 
                newLocations = append(newLocations, exactLocation)

Metadata

Metadata

Assignees

No one assigned

    Labels

    kind/cleanupCategorizes issue or PR as related to cleaning up code, process, or technical debt.priority/important-longtermImportant over the long term, but may not be staffed and/or may need multiple releases to complete.triage/acceptedIndicates an issue or PR is ready to be actively worked on.

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions