Skip to content

Enable sloppy_tcp when DSR and Maglev is enabled #1888

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 4 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 23 additions & 0 deletions pkg/controllers/proxy/service_endpoints_sync.go
Original file line number Diff line number Diff line change
Expand Up @@ -400,10 +400,33 @@ func (nsc *NetworkServicesController) setupExternalIPServices(serviceInfoMap ser
}
}
}
nsc.setupSloppyTCP(serviceInfoMap)

return nil
}

func (nsc *NetworkServicesController) setupSloppyTCP(serviceInfoMap serviceInfoMap) {
var sloppyTCPVal int8 = 0
for _, svc := range serviceInfoMap {
// Enable sloppy TCP if any DSR service with Maglev hashing is configured
if svc.directServerReturn && svc.scheduler == IpvsMaglevHashing {
sloppyTCPVal = 1
break
}
}

// enable/disable sloppy_tcp sysctl based on sloppyTCPVal
sloppyTCP := nsc.krNode.SloppyTCP()
if sloppyTCP.CachedVal() != sloppyTCPVal {
sysctlErr := sloppyTCP.WriteVal(sloppyTCPVal)
if sysctlErr != nil {
klog.Errorf("Failed to set IPVS sloppy TCP to %d: %s", sloppyTCPVal, sysctlErr.Error())
return
}
klog.Infof("IPVS sloppy TCP set to %d", sloppyTCPVal)
}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should handle disabling Sloppy TCP in the case that it is enabled and there are no DSR services with maglev hashing found.

Can you add an else if here to handle !enableSloppyTCP && sloppyTCP.CachedVal() == 1? (I think that one of the linters in this project may ask you to change the if ... else if to a switch statement when making this change)

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was thinking about the scenario when an user had already set sloppy_tcp to 1 and if we disable sloppy_tcp then we are going against the user even though kube-router can still function with sloppy_tcp being set.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that its a pretty obscure option, so we're probably safe here. And since it potentially makes services less secure, if we set it, then we should unset it if its not needed.

I see what you're saying, but I think that we can probably just explain this a bit by adding something to the documentation saying that we expect to control this option when the service controller is enabled.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

okay.
I have updated the code to disable sloppy_tcp if there is no service with DSR or Maglev enabled.

}

// setupExternalIPForService does the basic work to setup a non-DSR based external IP for service. This includes adding
// the IPVS service to the host if it is missing, and setting up the dummy interface to be able to receive traffic on
// the node.
Expand Down
16 changes: 16 additions & 0 deletions pkg/utils/node.go
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,7 @@ type LocalKRNode struct {
KRNode
NodeInterfaceName string
linkQ LocalLinkQuerier
sloppyTCP SysctlConfig
}

// NodeIPAware is an interface that provides methods to get the node's IP addresses in various data structures.
Expand Down Expand Up @@ -73,9 +74,14 @@ type NodeIPAndFamilyAware interface {
NodeFamilyAware
}

type NodeConfigAware interface {
SloppyTCP() *SysctlConfig
}

// NodeAware is an interface that combines the NodeIPAware, NodeInterfaceAware, NodeFamilyAware, and NodeNameAware
// interfaces.
type NodeAware interface {
NodeConfigAware
NodeIPAware
NodeInterfaceAware
NodeFamilyAware
Expand Down Expand Up @@ -127,6 +133,10 @@ func (n *KRNode) GetNodeName() string {
return n.NodeName
}

func (n *LocalKRNode) SloppyTCP() *SysctlConfig {
return &n.sloppyTCP
}

// FindBestIPv6NodeAddress returns the best available IPv6 address for the node. If the primary IP is already an IPv6
// address, it will return that. Otherwise, it will return the first internal or external IPv6 address defined in the
// Kubernetes Node Object.
Expand Down Expand Up @@ -241,6 +251,12 @@ func NewKRNode(node *apiv1.Node, linkQ LocalLinkQuerier, enableIPv4, enableIPv6
},
linkQ: linkQ,
NodeInterfaceName: nodeInterfaceName,
// Purposefully set the value of sloppyTCP to 0. This ensures the machine's sloppy_tcp setting remains
// unchanged when there are no services with both Maglev and DSR enabled.
sloppyTCP: SysctlConfig{
name: IPv4IPVSSloppyTCP,
value: 0,
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

At first I was sceptical about hard-coding this this value, but I think that it makes sense, especially when handling the reverse condition that I asked for above. Essentially assuming that this is 0 from a kube-router standpoint, means that if the user has set this value for whatever reason manually, we won't revert it above unless we specifically set it to 1. Which means that we have less of a chance of reverting manual changes to the system that the admin may have made.

Do you mind writing something like that in a comment above this setting so that we remember that we purposefully didn't look up the live setting with GetSysctl()?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sure I will do.

},
}

return krNode, nil
Expand Down
19 changes: 19 additions & 0 deletions pkg/utils/sysctl.go
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@ const (
IPv4IPVSExpireNodestConn = "net/ipv4/vs/expire_nodest_conn"
IPv4IPVSExpireQuiescent = "net/ipv4/vs/expire_quiescent_template"
IPv4IPVSConnReuseMode = "net/ipv4/vs/conn_reuse_mode"
IPv4IPVSSloppyTCP = "net/ipv4/vs/sloppy_tcp"
IPv4ConfAllArpIgnore = "net/ipv4/conf/all/arp_ignore"
IPv4ConfAllArpAnnounce = "net/ipv4/conf/all/arp_announce"
IPv6ConfAllDisableIPv6 = "net/ipv6/conf/all/disable_ipv6"
Expand Down Expand Up @@ -53,6 +54,24 @@ func (e *SysctlError) Unwrap() error {
return e.err
}

type SysctlConfig struct {
name string
value int8
}

func (n *SysctlConfig) CachedVal() int8 {
return n.value
}

func (n *SysctlConfig) WriteVal(val int8) *SysctlError {
err := SetSysctl(n.name, int(val))
if err != nil {
return err
}
n.value = val
return nil
}

func sysctlStat(path string, hasValue bool, value int) (string, *SysctlError) {
sysctlPath := fmt.Sprintf("/proc/sys/%s", path)
if _, err := os.Stat(sysctlPath); err != nil {
Expand Down