Fix code to handle delayed binding volumes #73863

gnufied · 2019-02-08T22:29:48Z

This fixes counting of volumes that are not immediate binded.

/sig storage

Count PVCs that are unbound towards attach limit

msau42 · 2019-02-08T22:46:04Z

/assign

gnufied · 2019-02-11T16:11:43Z

/retest

bsalamat · 2019-02-11T19:21:12Z

Please follow instructions here to post scheduler benchmarks before and after this PR.

gnufied · 2019-02-11T19:49:30Z

/kind bug

gnufied · 2019-02-11T20:30:43Z

Something is wrong with my setup of scheduler integration tests:

I0211 15:18:29.083115   22229 retry.go:149] clientv3/auth-retry: error "context canceled" on pinned endpoint "127.0.0.1:2379"                                                                                       
E0211 15:18:29.083160   22229 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"context canceled"}                                                                     
E0211 15:18:29.083204   22229 writers.go:172] apiserver was unable to write a JSON response: http: Handler timeout                                                                                                  
I0211 15:18:29.083304   22229 wrap.go:47] GET /apis/rbac.authorization.k8s.io/v1/clusterroles: (1m0.000470325s) 504                                                                                                 
goroutine 1560647 [running]:                      
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/httplog.(*respLogger).recordStatus(0xc01b901c70, 0x1f8)                                                                                                        
        /home/shared/go_projects/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/httplog/httplog.go:204 +0xd2                                                       
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/httplog.(*respLogger).WriteHeader(0xc01b901c70, 0x1f8)                                                                                                         
        /home/shared/go_projects/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/httplog/httplog.go:183 +0x35                                                       
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.(*baseTimeoutWriter).timeout(0xc03fab6aa0, 0xc020a2ef30)                                                                                               
        /home/shared/go_projects/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:218 +0xb5                                                       
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP(0xc04d746040, 0x5db9ce0, 0xc01b901c70, 0xc01482fa00)                                                                       
        /home/shared/go_projects/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:118 +0x2aa                                                      
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.WithWaitGroup.func1(0x5db9ce0, 0xc01b901c70, 0xc01482f900)                                                                                             
        /home/shared/go_projects/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/waitgroup.go:47 +0xd6                                                      
net/http.HandlerFunc.ServeHTTP(0xc04d774b10, 0x5db9ce0, 0xc01b901c70, 0xc01482f900)                
        /data/go/src/net/http/server.go:1964 +0x44                                                                                  
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithRequestInfo.func1(0x5db9ce0, 0xc01b901c70, 0xc01482f800)                                                                                        
        /home/shared/go_projects/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters/requestinfo.go:39 +0x217                                                
net/http.HandlerFunc.ServeHTTP(0xc04d774b40, 0x5db9ce0, 0xc01b901c70, 0xc01482f800)  
        /data/go/src/net/http/server.go:1964 +0x44
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.withPanicRecovery.func1(0x5db9ce0, 0xc01b901c70, 0xc01482f800)                                                                                         
        /home/shared/go_projects/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/wrap.go:46 +0x127                                                          
net/http.HandlerFunc.ServeHTTP(0xc04d746060, 0x5dbada0, 0xc033987420, 0xc01482f800)
        /data/go/src/net/http/server.go:1964 +0x44 
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server.(*APIServerHandler).ServeHTTP(0xc04d774b70, 0x5dbada0, 0xc033987420, 0xc01482f800)                                                                             
        /home/shared/go_projects/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/handler.go:189 +0x51                                                               
k8s.io/kubernetes/test/integration/util.StartApiserver.func1(0x5dbada0, 0xc033987420, 0xc01482f800)
        /home/shared/go_projects/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/integration/util/util.go:44 +0x72                                                                                
net/http.HandlerFunc.ServeHTTP(0xc01b95c690, 0x5dbada0, 0xc033987420, 0xc01482f800)                                                            
        /data/go/src/net/http/server.go:1964 +0x44                                                                
net/http.serverHandler.ServeHTTP(0xc01d1a8000, 0x5dbada0, 0xc033987420, 0xc01482f800)                              
        /data/go/src/net/http/server.go:2741 +0xab
net/http.(*conn).serve(0xc013255220, 0x5dc4ea0, 0xc04f058980)                                               
        /data/go/src/net/http/server.go:1847 +0x646                                                                                                          
created by net/http.(*Server).Serve                                                                        
        /data/go/src/net/http/server.go:2851 +0x2f5                                                                                                          
                                                                                                                     
logging error output: "{\"metadata\":{},\"status\":\"Failure\",\"message\":\"Timeout: request did not complete within 1m0s\",\"reason\":\"Timeout\",\"details\":{},\"code\":504}\n"                                 
 [scheduler_perf.test/v0.0.0 (linux/amd64) kubernetes/$Format 127.0.0.1:55076]                                                               
E0211 15:18:29.083507   22229 master_utils.go:234] last health content: ""                                                                                    
F0211 15:18:29.083530   22229 master_utils.go:235] timed out waiting for the condition                                 
E0211 15:18:29.440434   22229 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}                                                                
W0211 15:18:29.083562   22229 storage_scheduling.go:95] unable to get PriorityClass system-node-critical: the server was unable to return a response in the time allotted, but may still be processing the request (get priorityclasses.scheduling.k8s.io system-node-critical). Retrying...
exit status 255                                                                                                             
FAIL    k8s.io/kubernetes/test/integration/scheduler_perf       203.972s

@bsalamat fwiw - this PR should not introduce any regression in scheduler performance because all it does is - it ensures we are counting unbound but scheduled PVCs towards volume limit.

gnufied · 2019-02-11T20:53:59Z

@bsalamat so I ran../test-performance.sh which gave me..

Before:

Scheduled 3000 Pods in 3 seconds (1000 per second on average). min QPS was 207

After:

Scheduled 3000 Pods in 3 seconds (1000 per second on average). min QPS was 206

pkg/scheduler/algorithm/predicates/csi_volume_predicate.go

pkg/scheduler/algorithm/predicates/csi_volume_predicate_test.go

gnufied · 2019-02-15T20:44:57Z

@msau42 addressed review comments PTAL.

pkg/scheduler/algorithm/predicates/csi_volume_predicate.go

nikopen · 2019-03-12T13:35:48Z

@childsb @gnufied @msau42
we're in code freeze, release is in 2 weeks from now. Is this bug urgent and release-blocking?

spiffxp · 2019-03-13T21:23:17Z

/milestone clear
v1.14 release lead here, given the lack of activity on this PR, I'm removing this from the milestone. Please discuss in #sig-release in slack if it's urgent this land as part of v1.14

gnufied · 2019-03-14T21:08:24Z

I guess we can merge this after code freeze.

@msau42 addressed review comments. PTAL.

msau42 · 2019-03-26T21:23:30Z

/lgtm

spiffxp · 2019-03-26T21:39:28Z

/priority important-soon

bsalamat

/approve

k8s-ci-robot · 2019-04-01T18:03:00Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: bsalamat, gnufied

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~pkg/scheduler/OWNERS~~ [bsalamat]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

matthias50 · 2019-08-23T18:49:46Z

@gnufied , It looks like earliest release where this is present is 1.15. Can this be back-ported to 1.14 and 1.13?

Fix code to handle delayed binding volumes

b3cc469

k8s-ci-robot requested review from bsalamat and misterikkit February 8, 2019 22:32

k8s-ci-robot assigned msau42 Feb 8, 2019

k8s-ci-robot added kind/bug Categorizes issue or PR as related to a bug. and removed needs-kind Indicates a PR lacks a `kind/foo` label and requires one. labels Feb 11, 2019

k8s-ci-robot added release-note Denotes a PR that will be considered when it comes time to generate release notes. and removed do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. labels Feb 14, 2019

msau42 reviewed Feb 15, 2019

View reviewed changes

gnufied force-pushed the fix-attach-limit-delayed branch from ce3e1d0 to 2385eaf Compare February 15, 2019 20:45

k8s-ci-robot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Feb 15, 2019

gnufied force-pushed the fix-attach-limit-delayed branch from 2385eaf to d1fada4 Compare February 22, 2019 03:24

msau42 reviewed Feb 22, 2019

View reviewed changes

pkg/scheduler/algorithm/predicates/csi_volume_predicate.go Show resolved Hide resolved

pkg/scheduler/algorithm/predicates/csi_volume_predicate.go Show resolved Hide resolved

pkg/scheduler/algorithm/predicates/csi_volume_predicate.go Show resolved Hide resolved

childsb added this to the v1.14 milestone Mar 6, 2019

k8s-ci-robot removed this from the v1.14 milestone Mar 13, 2019

Add more volume types and unbound pvcs etc

a3a93c5

gnufied force-pushed the fix-attach-limit-delayed branch from d1fada4 to a3a93c5 Compare March 14, 2019 21:07

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Mar 26, 2019

k8s-ci-robot added priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. and removed needs-priority Indicates a PR lacks a `priority/foo` label and requires one. labels Mar 26, 2019

bsalamat reviewed Apr 1, 2019

View reviewed changes

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Apr 1, 2019

k8s-ci-robot merged commit 76b1d83 into kubernetes:master Apr 1, 2019

gnufied mentioned this pull request Apr 8, 2019

Add KEP for volume scheduling limits kubernetes/enhancements#942

Merged

This was referenced Sep 3, 2019

Bug 1747377: UPSTREAM: 73863: Fix code to handle delayed binding volumes openshift/origin#23718

Merged

Bug 1747377: UPSTREAM: 73863: Fix code to handle delayed binding volumes openshift/origin#23756

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix code to handle delayed binding volumes #73863

Fix code to handle delayed binding volumes #73863

gnufied commented Feb 8, 2019 •

edited

Loading

msau42 commented Feb 8, 2019

gnufied commented Feb 11, 2019

bsalamat commented Feb 11, 2019

gnufied commented Feb 11, 2019

gnufied commented Feb 11, 2019

gnufied commented Feb 11, 2019

gnufied commented Feb 15, 2019

nikopen commented Mar 12, 2019

spiffxp commented Mar 13, 2019

gnufied commented Mar 14, 2019

msau42 commented Mar 26, 2019

spiffxp commented Mar 26, 2019

bsalamat left a comment

k8s-ci-robot commented Apr 1, 2019

matthias50 commented Aug 23, 2019

Fix code to handle delayed binding volumes #73863

Fix code to handle delayed binding volumes #73863

Conversation

gnufied commented Feb 8, 2019 • edited Loading

msau42 commented Feb 8, 2019

gnufied commented Feb 11, 2019

bsalamat commented Feb 11, 2019

gnufied commented Feb 11, 2019

gnufied commented Feb 11, 2019

gnufied commented Feb 11, 2019

gnufied commented Feb 15, 2019

nikopen commented Mar 12, 2019

spiffxp commented Mar 13, 2019

gnufied commented Mar 14, 2019

msau42 commented Mar 26, 2019

spiffxp commented Mar 26, 2019

bsalamat left a comment

Choose a reason for hiding this comment

k8s-ci-robot commented Apr 1, 2019

matthias50 commented Aug 23, 2019

gnufied commented Feb 8, 2019 •

edited

Loading