Inappropriate use of RFC6598 addresses

[rbtcollins]

I'm not sure if you consider this a bug or not - but I was very surprised when I saw the IP range we got by default with a new kops deployed cluster.

kops/upup/pkg/fi/cloudup/defaults.go

Line 70 in 2c93bff

c.Spec.NonMasqueradeCIDR = "100.64.0.0/10"

defaults the IP range used for the pods and cluster addresses to 100.64 - but this range is specifically reserved for CGN <-> CP interconnect.

At least as I read https://tools.ietf.org/html/rfc6598#section-4 kops fails per " Because CGN service requires non-overlapping address space on each
side of the home NAT and CGN, entities using Shared Address Space for
purposes other than for CGN service, as described in this document,
are likely to experience problems implementing or connecting to CGN
service at such time as they exhaust their supply of public IPv4
addresses."

It would be better - as well as delivering more IP space - to use RFC1918 addresses such as 10/8. I've been trying to think of a technical reason you might prefer RFC6598 addresses, but I'm drawing a blank :). If this isn't actually a strategic choice, I'd be happy to put forward a patch fixing the default.

[justinsb]

10/8 is a conflict with AWS classic, so breaks ClassicLink.

The thinking behind 100.64:

• 100.64 is unused in AWS unless you specify it
• 100.64 won't escape the VPC, so can't conflict with anything outside AWS
• 100.64 is not very widely used, so most people won't specify it - unlike 172 or 192.168, where we're likely to break VPC peering.

[justinsb]

PS and you can change it if you disagree - just kops edit cluster before your first create. But 10/8 as a default causes people problems, 100.64 seems not to :-)

[rbtcollins]

Ah thats interesting about classic - I guess some folk still use it :) That said AWS could go crazy and decide to use 100.64 for interconnects to their nat gw's ;).

W.r.t. VPC conflicts - yes, that can and will happen, but once you're running multiple VPC's you are expecting to do IP range design : its not that I discount it as being important, its that it becomes a fact of life - the VPC CIDR for k8s itself also needs to be non-overlapping.

Perhaps a note in the docs about the reason for this choice would reduce surprise for other folk?

FWIW the reason we pay attention to this is we're working on cross k8s cluster gossip clusters - where we need pod to pod connectivity, so we're doing IP range allocations somewhat carefully :).

[yissachar]

Marking this as a docs issue - I think this is just a matter of grabbing @justinsb's two comments and throwing them in docs.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or @fejta.
/lifecycle stale

[rbtcollins]

/remove-lifecycle stale

[activeshadow]

Interestingly enough, I just got multiple abuse reports from AWS stating some of my K8S node instances were observed "scanning" some external hosts with IPs in the 100.64.0.0/10 block. So either the packets are escaping the VPC or AWS abuse monitoring happens within each VPC.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten
/remove-lifecycle stale