feat: service restart provides now per node status #4827

harshavardhana · 2024-01-19T05:33:46Z

Community Contribution License

All community contributions in this pull request are licensed to the project maintainers
under the terms of the Apache 2 license.
By creating this pull request I represent that I have the right to license the
contributions to the project maintainers under the Apache 2 license.

Description

Motivation and Context

How to test this PR?

Types of changes

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Optimization (provides speedup with no functional changes)
Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

Fixes a regression (If yes, please add commit-id or PR # here)
Unit tests added/updated
Internal documentation updated
Create a documentation update request here

harshavardhana · 2024-01-19T05:34:10Z

This PR is not ready until minio/madmin-go#263 is merged

klauspost · 2024-01-19T11:43:17Z

Server PR: minio/minio#18826 for reference.

harshavardhana · 2024-01-26T04:43:15Z

This PR is ready with all the changes both restart/update so it is good to be reviewed and merged right away.

shtripat · 2024-01-26T15:13:03Z

In a local single node single disk setup started as minio server ./test the output for mc admin service restart ALIAS shows as below

./mc admin service restart myminio 
Restart command successfully sent to `myminio`. Type Ctrl-C to quit or wait to follow the status of the restart process.
┌────────────────┬────────┐
│ HOST           │ STATUS │
├────────────────┼────────┤
│ 127.0.0.1:9000 │ ✔      │
│                │ ✔      │
└────────────────┴────────┘
...
Restarted `myminio` successfully in 1 seconds

The second tick mark seems unwanted for this scenario.

whereas for a multi node setup looks fine as below

./mc admin service restart myminio
Restart command successfully sent to `myminio`. Type Ctrl-C to quit or wait to follow the status of the restart process.
┌────────────────┬────────┐
│ HOST           │ STATUS │
├────────────────┼────────┤
│ 127.0.0.1:9000 │ ✔      │
│ 127.0.0.1:9001 │ ✔      │
│ 127.0.0.1:9002 │ ✔      │
│ 127.0.0.1:9003 │ ✔      │
└────────────────┴────────┘

...!!
Restarted `myminio` successfully in 3 seconds

Feature wise code looks good to me.

harshavardhana · 2024-01-26T16:43:36Z

Gonna fix the single node output.

shtripat · 2024-01-26T17:13:26Z

Gonna fix the single node output.

Also with SNSD mode ./mc admin update myminio "https://dl.min.io/server/minio/release/linux-amd64/minio.RELEASE.2024-01-18T22-51-28Z.sha256sum" failed with error on MinIO side as below

panic: "POST /minio/admin/v3/update?dry-run=false&type=2&updateURL=https%3A%2F%2Fdl.min.io%2Fserver%2Fminio%2Frelease%2Flinux-amd64%2Fminio.RELEASE.2024-01-18T22-51-28Z.sha256sum": runtime error: invalid memory address or nil pointer dereference
goroutine 738 [running]:
runtime/debug.Stack()
	runtime/debug/stack.go:24 +0x5e
github.com/minio/minio/cmd.serverMain.func8.setCriticalErrorHandler.func2.1()
	github.com/minio/minio/cmd/generic-handlers.go:555 +0x97
panic({0x28080e0?, 0x62c2cf0?})
	runtime/panic.go:914 +0x21f
github.com/minio/pkg/v2/workers.(*Workers).Wait(...)
	github.com/minio/pkg/v2@v2.0.8/workers/workers.go:62
github.com/minio/minio/cmd.(*NotificationGroup).Wait(...)
	github.com/minio/minio/cmd/notification.go:101
github.com/minio/minio/cmd.(*NotificationSys).VerifyBinary(0xc001317740, {0x4c51ec0?, 0xc000ecd860}, 0xc08ff0e240, {0xc08ff01e40, 0x20, 0x40}, {0xc08ff195b1, 0x22}, {0xc09c042000, ...})
	github.com/minio/minio/cmd/notification.go:390 +0x38d
github.com/minio/minio/cmd.adminAPIHandlers.ServerUpdateV2Handler({}, {0x4c4c5e0, 0xc000cd2280}, 0xc08ff14300)
	github.com/minio/minio/cmd/admin-handlers.go:148 +0x645
net/http.HandlerFunc.ServeHTTP(0x52f303b?, {0x4c4c5e0?, 0xc000cd2280?}, 0x4c7f108?)
	net/http/server.go:2136 +0x29
github.com/minio/minio/cmd.adminMiddleware.func1.httpTraceAll.httpTrace.func1({0x4c4c5e0, 0xc000cd2280}, 0xc000cd2280?)
	github.com/minio/minio/cmd/http-tracer.go:189 +0xef
github.com/minio/minio/cmd.adminMiddleware.func1({0x4c4c5e0, 0xc000cd2280}, 0xc08ff14200)
	github.com/minio/minio/cmd/admin-router.go:132 +0x2df
net/http.HandlerFunc.ServeHTTP(0x62d40a0?, {0x4c4c5e0?, 0xc000cd2280?}, 0x4?)
	net/http/server.go:2136 +0x29
github.com/klauspost/compress/gzhttp.NewWrapper.func1.1({0x4c4bc50, 0xc08ff30000}, 0x1b?)
	github.com/klauspost/compress@v1.17.4/gzhttp/compress.go:497 +0x37d
net/http.HandlerFunc.ServeHTTP(0xc000ecd1a0?, {0x4c4bc50?, 0xc08ff30000?}, 0x0?)
	net/http/server.go:2136 +0x29
github.com/minio/minio/cmd.setBucketForwardingMiddleware.func1({0x4c4bc50, 0xc08ff30000}, 0xc08ff14200)
	github.com/minio/minio/cmd/generic-handlers.go:460 +0x2fc
net/http.HandlerFunc.ServeHTTP(0x7f193e5eeee0?, {0x4c4bc50?, 0xc08ff30000?}, 0xc000100000?)
	net/http/server.go:2136 +0x29
github.com/minio/minio/cmd.setUploadForwardingMiddleware.func1({0x4c4bc50, 0xc08ff30000}, 0xc08ff14200)
	github.com/minio/minio/cmd/generic-handlers.go:573 +0x1e5
net/http.HandlerFunc.ServeHTTP(0x2b3ee40?, {0x4c4bc50?, 0xc08ff30000?}, 0x39?)
	net/http/server.go:2136 +0x29
github.com/minio/minio/cmd.setRequestValidityMiddleware.func1({0x4c4bc50?, 0xc08ff30000}, 0xc08ff14200)
	github.com/minio/minio/cmd/generic-handlers.go:443 +0x168d
net/http.HandlerFunc.ServeHTTP(0xc000ecd500?, {0x4c4bc50?, 0xc08ff30000?}, 0xc000fe4cf0?)
	net/http/server.go:2136 +0x29
github.com/minio/minio/cmd.setRequestLimitMiddleware.func1({0x4c4bc50?, 0xc08ff30000}, 0xc08ff14200)
	github.com/minio/minio/cmd/generic-handlers.go:134 +0x662
net/http.HandlerFunc.ServeHTTP(0x2f?, {0x4c4bc50?, 0xc08ff30000?}, 0x2253fab?)
	net/http/server.go:2136 +0x29
github.com/minio/minio/cmd.setCrossDomainPolicyMiddleware.func1({0x4c4bc50?, 0xc08ff30000?}, 0x6?)
	github.com/minio/minio/cmd/crossdomain-xml-handler.go:42 +0xf1
net/http.HandlerFunc.ServeHTTP(0xc08ff14200?, {0x4c4bc50?, 0xc08ff30000?}, 0xc0017e2110?)
	net/http/server.go:2136 +0x29
github.com/minio/minio/cmd.setBrowserRedirectMiddleware.func1({0x4c4bc50, 0xc08ff30000}, 0xc08ff14200)
	github.com/minio/minio/cmd/generic-handlers.go:164 +0x19c
net/http.HandlerFunc.ServeHTTP(0x0?, {0x4c4bc50?, 0xc08ff30000?}, 0x871c46f?)
	net/http/server.go:2136 +0x29
github.com/minio/minio/cmd.setAuthMiddleware.func1({0x4c4bc50, 0xc08ff30000}, 0xc08ff14200)
	github.com/minio/minio/cmd/auth-handler.go:636 +0x6a2
net/http.HandlerFunc.ServeHTTP(0x4c51ec0?, {0x4c4bc50?, 0xc08ff30000?}, 0x4c2a490?)
	net/http/server.go:2136 +0x29
github.com/minio/minio/cmd.httpTracerMiddleware.func1({0x4c4c1f0, 0xc08ff1a000}, 0xc08ff14100)
	github.com/minio/minio/cmd/http-tracer.go:89 +0x3b4
net/http.HandlerFunc.ServeHTTP(0x29faa00?, {0x4c4c1f0?, 0xc08ff1a000?}, 0xa?)
	net/http/server.go:2136 +0x29
github.com/minio/minio/cmd.addCustomHeadersMiddleware.func1({0x4c4c1f0, 0xc08ff1a000}, 0xc000ecd3e0?)
	github.com/minio/minio/cmd/generic-handlers.go:538 +0x438
net/http.HandlerFunc.ServeHTTP(0xc08ff14000?, {0x4c4c1f0?, 0xc08ff1a000?}, 0x69d73f?)
	net/http/server.go:2136 +0x29
github.com/minio/mux.(*Router).ServeHTTP(0xc000dcaa80, {0x4c4c1f0, 0xc08ff1a000}, 0xc08ff14000)
	github.com/minio/mux@v1.9.0/mux.go:214 +0x1f7
github.com/minio/minio/cmd.corsHandler.(*Cors).Handler.func2({0x4c4c1f0, 0xc08ff1a000}, 0xc08ff14000)
	github.com/rs/cors@v1.10.1/cors.go:281 +0x184
net/http.HandlerFunc.ServeHTTP(0x4186e8?, {0x4c4c1f0?, 0xc08ff1a000?}, 0xf?)
	net/http/server.go:2136 +0x29
github.com/minio/minio/cmd.serverMain.func8.setCriticalErrorHandler.func2({0x4c4c1f0?, 0xc08ff1a000?}, 0x63474e0?)
	github.com/minio/minio/cmd/generic-handlers.go:562 +0x78
net/http.HandlerFunc.ServeHTTP(0xc001646ac8?, {0x4c4c1f0?, 0xc08ff1a000?}, 0x727385?)
	net/http/server.go:2136 +0x29
github.com/minio/minio/internal/http.(*Server).Init.func1({0x4c4c1f0?, 0xc08ff1a000?}, 0xc00131e300?)
	github.com/minio/minio/internal/http/server.go:123 +0x19c
net/http.HandlerFunc.ServeHTTP(0x4109a5?, {0x4c4c1f0?, 0xc08ff1a000?}, 0xc08ff1a001?)
	net/http/server.go:2136 +0x29
net/http.serverHandler.ServeHTTP({0x4c44f38?}, {0x4c4c1f0?, 0xc08ff1a000?}, 0x6?)
	net/http/server.go:2938 +0x8e
net/http.(*conn).serve(0xc08ff0e000, {0x4c51ec0, 0xc000e72570})
	net/http/server.go:2009 +0x5f4
created by net/http.(*Server).Serve in goroutine 94
	net/http/server.go:3086 +0x5cb

whereas multi node showed message

Server update request sent successfully `myminio`
┌────────────────┬────────────────────────────────────────────────────────────────────┐
│ HOST           │ STATUS                                                             │
├────────────────┼────────────────────────────────────────────────────────────────────┤
│ 127.0.0.1:9002 │ server is already running the latest version: 2023-08-21T19-40-07Z │
│ 127.0.0.1:9003 │ server is already running the latest version: 2023-08-21T19-40-07Z │
│ 127.0.0.1:9000 │ server is already running the latest version: 2023-08-21T19-40-07Z │
│ 127.0.0.1:9001 │ server is already running the latest version: 2023-08-21T19-40-07Z │
└────────────────┴────────────────────────────────────────────────────────────────────┘

harshavardhana · 2024-01-26T20:32:07Z

Yes server side is fixed @shtripat

commit c88308cf0e6401ca034561357b6e7c932b11a516 (HEAD -> master, origin/master)
Author: Harshavardhana <harsha@minio.io>
Date:   Fri Jan 26 12:07:03 2024 -0800

    avoid 'panic' on mc admin update for single drive setup (#18876)

harshavardhana force-pushed the per-node branch 2 times, most recently from bdd3f07 to 0478e4d Compare January 24, 2024 23:04

bh4t assigned harshavardhana Jan 24, 2024

harshavardhana force-pushed the per-node branch from 0478e4d to 453dba6 Compare January 25, 2024 23:09

harshavardhana mentioned this pull request Jan 25, 2024

add new update v2 that updates per node, allows idempotent behavior minio/minio#18859

Merged

8 tasks

harshavardhana force-pushed the per-node branch from 453dba6 to 85d6fee Compare January 26, 2024 00:12

feat: service restart provides now per node status

c13b1c8

harshavardhana requested review from vadmeste, jiuker and shtripat January 26, 2024 04:43

jiuker approved these changes Jan 26, 2024

View reviewed changes

klauspost approved these changes Jan 26, 2024

View reviewed changes

harshavardhana force-pushed the per-node branch from 63a3fcb to 76ce414 Compare January 26, 2024 19:31

feat: service update provides now per node status

e11167d

harshavardhana force-pushed the per-node branch from 76ce414 to e11167d Compare January 26, 2024 20:17

harshavardhana merged commit 9a5ef7c into minio:master Jan 26, 2024
5 checks passed

harshavardhana deleted the per-node branch January 26, 2024 20:32

djwfyi mentioned this pull request Feb 5, 2024

[RELEASE] mc RELEASE.2024-01-28T16-23-14Z minio/docs#1120

Closed

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: service restart provides now per node status #4827

feat: service restart provides now per node status #4827

harshavardhana commented Jan 19, 2024

harshavardhana commented Jan 19, 2024

klauspost commented Jan 19, 2024

harshavardhana commented Jan 26, 2024

shtripat commented Jan 26, 2024 •

edited

Loading

harshavardhana commented Jan 26, 2024

shtripat commented Jan 26, 2024

harshavardhana commented Jan 26, 2024

feat: service restart provides now per node status #4827

feat: service restart provides now per node status #4827

Conversation

harshavardhana commented Jan 19, 2024

Community Contribution License

Description

Motivation and Context

How to test this PR?

Types of changes

Checklist:

harshavardhana commented Jan 19, 2024

klauspost commented Jan 19, 2024

harshavardhana commented Jan 26, 2024

shtripat commented Jan 26, 2024 • edited Loading

harshavardhana commented Jan 26, 2024

shtripat commented Jan 26, 2024

harshavardhana commented Jan 26, 2024

shtripat commented Jan 26, 2024 •

edited

Loading