Fix garnet persistence #5087

Alirexaa · 2024-07-26T12:44:42Z

Microsoft Reviewers: Open in CodeFlow

…aspire into fix-garnet-persistence

Alirexaa · 2024-07-26T14:21:47Z

WithDataVolume failed to start, related issue in the Garnet repo: microsoft/garnet#536.
also interval is not working, related issue in the Garnet repo: microsoft/garnet#537.
for now only WithDataBindMount works

Alirexaa · 2024-07-30T15:58:25Z

WithPersistence now works, so WithDataVolume and WithDataBindMount
will work.

keysChangedThreshold is unused now. we could mark this method as Obsolete and introduce a new public API.

aspire/src/Aspire.Hosting.Garnet/GarnetBuilderExtensions.cs

Lines 137 to 139 in f42201f

    
           /// <returns>The <see cref="IResourceBuilder{T}"/>.</returns> 
        
           public static IResourceBuilder<GarnetResource> WithPersistence(this IResourceBuilder<GarnetResource> builder, 
        
               TimeSpan? interval = null, long keysChangedThreshold = 1)

also despite Aspire.Garnet.Hosting shipped in 8.1, PublicAPI.Shipped.txt is untouched.

I want to write functional tests for WithDataVolume and WithDataBindMount but we should decide about public API first.

cc @eerhardt @mitchdenny @radical

eerhardt

Marking the existing API as Obsolete and introducing a new API sounds like the right path to me.

playground/Garnet/Garnet.AppHost/Directory.Build.props

Alirexaa · 2024-07-31T21:42:17Z

@eerhardt, do you agree with the last change?
I want to fix tests but I need your confirmation about the public API signature first.

src/Aspire.Hosting.Garnet/GarnetBuilderExtensions.cs

eerhardt · 2024-07-31T21:53:12Z

src/Aspire.Hosting.Garnet/GarnetBuilderExtensions.cs

@@ -83,7 +83,7 @@ public static class GarnetBuilderExtensions
            isReadOnly);
        if (!isReadOnly)
        {
-            builder.WithPersistence();
+            builder.WithPersistence(interval: null);


This is unfortunate that the overloads are conflicting. Can we solve this somehow? Potentially by making the Obsolete overload not have default parameters?

@eerhardt, do you mean something like this?

[Obsolete("This method is obsolete and will be removed in a future version. Use the overload without the keysChangedThreshold parameter.")] public static IResourceBuilder<GarnetResource> WithPersistence(this IResourceBuilder<GarnetResource> builder, TimeSpan? interval, long keysChangedThreshold) => WithPersistence(builder, interval);

Yes, I think removing the optional parameters on the Obsolete overload is a good compromise here.

@mitchdenny - thoughts?

Agree with the strategy.

eerhardt · 2024-08-01T15:25:18Z

Looks like the new test is currently failing.

playground/Garnet/Garnet.AppHost/aspire-manifest.json

mitchdenny · 2024-08-07T05:30:19Z

So, if I am following all of this correctly the following happened:

Bind mounts were not working.
Garnet fixed the issue, but they also broke their args.
we updated the args we use and started using their 1.0
Bind mounts now work?

I think I'm OK with the breaking change here - it seems necessary.

Alirexaa · 2024-08-08T17:08:33Z

So, if I am following all of this correctly the following happened:

Bind mounts were not working.

Garnet fixed the issue, but they also broke their args.

we updated the args we use and started using their 1.0

Bind mounts now work?

I think I'm OK with the breaking change here - it seems necessary.

Actually, we used the wrong args plus they also had issues in persistance.

Alirexaa · 2024-09-03T15:38:09Z

Looks like the test is still failing.

see microsoft/garnet#536 (comment)

eerhardt · 2024-09-04T13:55:19Z

Looks like the non-volume test is broken:

[xUnit.net 00:03:32.05]     Aspire.Hosting.Garnet.Tests.GarnetFunctionalTests.WithDataShouldPersistStateBetweenUsages(useVolume: False) [FAIL]
[xUnit.net 00:03:32.05]       StackExchange.Redis.RedisConnectionException : The message timed out in the backlog attempting to send because no connection became available (5000ms) - Last Connection Exception: UnableToConnect (None, 0-read, last-recv: 0) on localhost:33993/Interactive, Flushed/ReadAsync, last: ECHO, origin: ResetNonConnected, outstanding: 12, last-read: 5s ago, last-write: 5s ago, unanswered-write: 5s ago, keep-alive: 60s, state: ConnectedEstablishing, mgr: 10 of 10 available, last-heartbeat: never, global: 17s ago, v: 2.8.0.27420, command=SET, timeout: 5000, inst: 0, qu: 0, qs: 12, aw: False, bw: CheckingForTimeout, rs: ReadAsync, ws: Idle, in: 0, in-pipe: 0, out-pipe: 0, last-in: 0, cur-in: 0, sync-ops: 0, async-ops: 11, serverEndpoint: localhost:33993, conn-sec: 40, aoc: 0, mc: 1/1/0, mgr: 10 of 10 available, clientName: a006N32(SE.Redis-v2.8.0.27420), IOCP: (Busy=0,Free=1000,Min=1,Max=1000), WORKER: (Busy=2,Free=32765,Min=2,Max=32767), POOL: (Threads=5,QueuedItems=0,CompletedItems=4179,Timers=8), v: 2.8.0.27420 (Please take a look at this article for some common client-side issues that can cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts)
[xUnit.net 00:03:32.05]       ---- StackExchange.Redis.RedisConnectionException : UnableToConnect (None, 0-read, last-recv: 0) on localhost:33993/Interactive, Flushed/ReadAsync, last: ECHO, origin: ResetNonConnected, outstanding: 12, last-read: 5s ago, last-write: 5s ago, unanswered-write: 5s ago, keep-alive: 60s, state: ConnectedEstablishing, mgr: 10 of 10 available, last-heartbeat: never, global: 17s ago, v: 2.8.0.27420
[xUnit.net 00:03:32.05]       Stack Trace:
[xUnit.net 00:03:32.05]         /_/tests/Aspire.Hosting.Garnet.Tests/GarnetFunctionalTests.cs(125,0): at Aspire.Hosting.Garnet.Tests.GarnetFunctionalTests.<>c__DisplayClass3_0.<<WithDataShouldPersistStateBetweenUsages>b__0>d.MoveNext()
[xUnit.net 00:03:32.05]         --- End of stack trace from previous location ---
[xUnit.net 00:03:32.05]            at Polly.ResiliencePipeline.<>c.<<ExecuteAsync>b__3_0>d.MoveNext()
[xUnit.net 00:03:32.05]         --- End of stack trace from previous location ---
[xUnit.net 00:03:32.05]            at Polly.Outcome`1.GetResultOrRethrow()
[xUnit.net 00:03:32.05]            at Polly.ResiliencePipeline.ExecuteAsync(Func`2 callback, CancellationToken cancellationToken)
[xUnit.net 00:03:32.05]         /_/tests/Aspire.Hosting.Garnet.Tests/GarnetFunctionalTests.cs(119,0): at Aspire.Hosting.Garnet.Tests.GarnetFunctionalTests.WithDataShouldPersistStateBetweenUsages(Boolean useVolume)
[xUnit.net 00:03:32.05]         /_/tests/Aspire.Hosting.Garnet.Tests/GarnetFunctionalTests.cs(139,0): at Aspire.Hosting.Garnet.Tests.GarnetFunctionalTests.WithDataShouldPersistStateBetweenUsages(Boolean useVolume)
[xUnit.net 00:03:32.05]         --- End of stack trace from previous location ---
[xUnit.net 00:03:32.05]         ----- Inner Stack Trace -----

Alirexaa · 2024-09-04T14:58:47Z

Looks like the non-volume test is broken:

[xUnit.net 00:03:32.05]     Aspire.Hosting.Garnet.Tests.GarnetFunctionalTests.WithDataShouldPersistStateBetweenUsages(useVolume: False) [FAIL]
[xUnit.net 00:03:32.05]       StackExchange.Redis.RedisConnectionException : The message timed out in the backlog attempting to send because no connection became available (5000ms) - Last Connection Exception: UnableToConnect (None, 0-read, last-recv: 0) on localhost:33993/Interactive, Flushed/ReadAsync, last: ECHO, origin: ResetNonConnected, outstanding: 12, last-read: 5s ago, last-write: 5s ago, unanswered-write: 5s ago, keep-alive: 60s, state: ConnectedEstablishing, mgr: 10 of 10 available, last-heartbeat: never, global: 17s ago, v: 2.8.0.27420, command=SET, timeout: 5000, inst: 0, qu: 0, qs: 12, aw: False, bw: CheckingForTimeout, rs: ReadAsync, ws: Idle, in: 0, in-pipe: 0, out-pipe: 0, last-in: 0, cur-in: 0, sync-ops: 0, async-ops: 11, serverEndpoint: localhost:33993, conn-sec: 40, aoc: 0, mc: 1/1/0, mgr: 10 of 10 available, clientName: a006N32(SE.Redis-v2.8.0.27420), IOCP: (Busy=0,Free=1000,Min=1,Max=1000), WORKER: (Busy=2,Free=32765,Min=2,Max=32767), POOL: (Threads=5,QueuedItems=0,CompletedItems=4179,Timers=8), v: 2.8.0.27420 (Please take a look at this article for some common client-side issues that can cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts)
[xUnit.net 00:03:32.05]       ---- StackExchange.Redis.RedisConnectionException : UnableToConnect (None, 0-read, last-recv: 0) on localhost:33993/Interactive, Flushed/ReadAsync, last: ECHO, origin: ResetNonConnected, outstanding: 12, last-read: 5s ago, last-write: 5s ago, unanswered-write: 5s ago, keep-alive: 60s, state: ConnectedEstablishing, mgr: 10 of 10 available, last-heartbeat: never, global: 17s ago, v: 2.8.0.27420
[xUnit.net 00:03:32.05]       Stack Trace:
[xUnit.net 00:03:32.05]         /_/tests/Aspire.Hosting.Garnet.Tests/GarnetFunctionalTests.cs(125,0): at Aspire.Hosting.Garnet.Tests.GarnetFunctionalTests.<>c__DisplayClass3_0.<<WithDataShouldPersistStateBetweenUsages>b__0>d.MoveNext()
[xUnit.net 00:03:32.05]         --- End of stack trace from previous location ---
[xUnit.net 00:03:32.05]            at Polly.ResiliencePipeline.<>c.<<ExecuteAsync>b__3_0>d.MoveNext()
[xUnit.net 00:03:32.05]         --- End of stack trace from previous location ---
[xUnit.net 00:03:32.05]            at Polly.Outcome`1.GetResultOrRethrow()
[xUnit.net 00:03:32.05]            at Polly.ResiliencePipeline.ExecuteAsync(Func`2 callback, CancellationToken cancellationToken)
[xUnit.net 00:03:32.05]         /_/tests/Aspire.Hosting.Garnet.Tests/GarnetFunctionalTests.cs(119,0): at Aspire.Hosting.Garnet.Tests.GarnetFunctionalTests.WithDataShouldPersistStateBetweenUsages(Boolean useVolume)
[xUnit.net 00:03:32.05]         /_/tests/Aspire.Hosting.Garnet.Tests/GarnetFunctionalTests.cs(139,0): at Aspire.Hosting.Garnet.Tests.GarnetFunctionalTests.WithDataShouldPersistStateBetweenUsages(Boolean useVolume)
[xUnit.net 00:03:32.05]         --- End of stack trace from previous location ---
[xUnit.net 00:03:32.05]         ----- Inner Stack Trace -----

There is issue in the Linux host, this test works in my local (Windows host)

  Standard Output Messages:
 | [2024-09-03T20:24:37] Aspire.Hosting.DistributedApplication Information: Aspire version: 9.0.0-ci
 | [2024-09-03T20:24:37] Aspire.Hosting.DistributedApplication Information: Distributed application starting.
 | [2024-09-03T20:24:37] Aspire.Hosting.DistributedApplication Information: Application host directory is: /mnt/vss/_work/1/s/tests/Aspire.Hosting.Tests
 | [2024-09-03T20:24:37] Aspire.Hosting.Dcp.DcpHostService Information: Starting DCP with arguments: start-apiserver --monitor 22066 --detach --kubeconfig "/datadisks/disk1/work/AEF2095F/t/aspire.3grUsK/kubeconfig"
 | [2024-09-03T20:24:38] Aspire.Hosting.Dcp.start-apiserver.api-server Information: Starting API server...
 | [2024-09-03T20:24:38] Aspire.Hosting.Dcp.start-apiserver.api-server Information: API server started	{"Address": "127.0.0.1", "Port": 41847}
 | [2024-09-03T20:24:38] Aspire.Hosting.Dcp.start-apiserver.dcp-host Information: Starting DCP controller host
 | [2024-09-03T20:24:38] Aspire.Hosting.Dcp.start-apiserver.dcp-host Information: Started all services	{"count": 1}
 | [2024-09-03T20:24:38] Aspire.Hosting.Dcp.dcpctrl.IdeExecutableRunner Information: Executables cannot be started via IDE: missing required environment variable 'DEBUG_SESSION_PORT'
 | [2024-09-03T20:24:38] Aspire.Hosting.Dcp.dcpctrl Information: starting controller manager
 | [2024-09-03T20:24:38] Aspire.Hosting.DistributedApplication Information: Distributed application started. Press Ctrl+C to shut down.
 | [2024-09-03T20:24:40] Aspire.Hosting.Dcp.dcpctrl.ContainerReconciler Error: could not determine host address and port for container port	{"Container": {"name":"garnet-kvsehqje-2ba90115"}, "Reconciliation": 5, "error": "container '/garnet-kvsehqje-2ba90115' is not running: exited"}
 | [2024-09-03T20:24:40] Aspire.Hosting.Dcp.dcpctrl.ContainerReconciler Error: could not create Endpoint object	{"Container": {"name":"garnet-kvsehqje-2ba90115"}, "Reconciliation": 5, "ServiceName": "garnet-2ba90115", "Workload": "/garnet-kvsehqje-2ba90115", "error": "container '/garnet-kvsehqje-2ba90115' is not running: exited"}
 | [2024-09-03T20:24:40] Aspire.Hosting.Tests.Resources.garnet Information: 1: 2024-09-03T20:24:39.7686067Z \x1b[31m    _________
 | [2024-09-03T20:24:40] Aspire.Hosting.Tests.Resources.garnet Information: 2: 2024-09-03T20:24:39.7686762Z    /_||___||_\      \x1b[0mGarnet 1.0.19 64 bit; standalone mode\x1b[31m
 | [2024-09-03T20:24:40] Aspire.Hosting.Tests.Resources.garnet Information: 3: 2024-09-03T20:24:39.7686832Z    '. \   / .'      \x1b[0mPort: 6379\x1b[31m
 | [2024-09-03T20:24:40] Aspire.Hosting.Tests.Resources.garnet Information: 4: 2024-09-03T20:24:39.7686883Z      '.\ /.'        \x1b[35mhttps://aka.ms/GetGarnet\x1b[31m
 | [2024-09-03T20:24:40] Aspire.Hosting.Tests.Resources.garnet Information: 5: 2024-09-03T20:24:39.7686935Z        '.'
 | [2024-09-03T20:24:40] Aspire.Hosting.Tests.Resources.garnet Information: 6: 2024-09-03T20:24:39.7686983Z \x1b[0m
 | [2024-09-03T20:24:40] Aspire.Hosting.Tests.Resources.garnet Information: 7: 2024-09-03T20:24:39.8603021Z Unable to initialize server due to exception: Access to the path '/data/checkpoints/AOF' is denied.
 | [2024-09-03T20:27:31] Aspire.Hosting.Dcp.dcpctrl.ContainerReconciler Information: no more Container resources are being watched, cancelling container watch	{"Container": {"name":"garnet-kvsehqje-2ba90115"}, "Reconciliation": 8}

see microsoft/garnet#536 (comment)

Alirexaa · 2024-09-05T08:59:22Z

@eerhardt the test still failing

eerhardt · 2024-09-05T17:40:12Z

@eerhardt the test still failing

I fixed it locally on my linux machine. It should be passing now.

davidfowl · 2024-09-16T20:39:46Z

I don't think we need a garnet sample until we have a garnet client. Can we add garnet and valkey to the same sample?

Alirexaa · 2024-09-16T21:39:01Z

I don't think we need a garnet sample until we have a garnet client. Can we add garnet and valkey to the same sample?

I could add garnet to the redis sample in this PR and valkey in another PR.

Alirexaa · 2024-09-18T21:43:38Z

@davidfowl, This PR is ready to merge. If you merge it, I can add the Valkey sample to the playground app.

mitchdenny · 2024-09-19T04:53:17Z

Thanks @Alirexaa

WIP: Fix garnet persistence

6fdc65e

dotnet-issue-labeler bot added the area-integrations Issues pertaining to Aspire Integrations packages label Jul 26, 2024

dotnet-policy-service bot added the community-contribution Indicates that the PR has been added by a community member label Jul 26, 2024

Alirexaa added 3 commits July 26, 2024 16:15

Merge branch 'main' into fix-garnet-persistence

d4508ae

remove --aof-commit-freq arg

c9889d3

Merge branch 'fix-garnet-persistence' of https://github.com/Alirexaa/…

f06bf89

…aspire into fix-garnet-persistence

Alirexaa mentioned this pull request Jul 29, 2024

Extract Aspire.Hosting.Garnet.Tests project #4871

Merged

Alirexaa added 2 commits July 30, 2024 10:05

Merge branch 'main' into fix-garnet-persistence

a721a37

fix WithPersistence, using new args

f42201f

Alirexaa marked this pull request as ready for review July 30, 2024 15:49

revert image tag

e629c81

build-analysis bot mentioned this pull request Jul 30, 2024

The Operation will be canceled. The next steps may not contain expected logs. dotnet/dnceng#3008

Open

3 tasks

eerhardt reviewed Jul 31, 2024

View reviewed changes

playground/Garnet/Garnet.AppHost/Directory.Build.props Outdated Show resolved Hide resolved

Alirexaa added 3 commits August 1, 2024 00:44

Merge branch 'dotnet:main' into fix-garnet-persistence

5863341

change playground app to new formet

6474dee

mark method as Obsolete

ed15673

eerhardt reviewed Jul 31, 2024

View reviewed changes

Alirexaa added 2 commits August 1, 2024 13:06

move shipped api from unshipped to shipped

489a57b

fix AddGarnetTests and add functional tests

c23641d

Add ResourceLoggerForwarderService to builder and increase delay

46bd31a

Alirexaa changed the title ~~WIP: Fix garnet persistence~~ Fix garnet persistence Aug 1, 2024

mitchdenny reviewed Aug 7, 2024

View reviewed changes

playground/Garnet/Garnet.AppHost/aspire-manifest.json Outdated Show resolved Hide resolved

Merge branch 'main' into fix-garnet-persistence

3e3dfff

minor change

7d87138

Address PR feedback

a2a7006

Alirexaa added 3 commits September 3, 2024 19:29

Add playground tests

18a3a95

fix build

2dc8f79

fix playground tests

10ca7e0

eerhardt and others added 2 commits September 4, 2024 12:00

Use "other" read/write permissions.

9af6c93

Update appsettings.json

91d8981

Fix bind mount directory permissions

214cdb8

Alirexaa and others added 3 commits September 13, 2024 20:24

Merge branch 'main' into fix-garnet-persistence

08ef349

Merge branch 'dotnet:main' into fix-garnet-persistence

90b42fe

Merge branch 'dotnet:main' into fix-garnet-persistence

9eeb9b3

build-analysis bot mentioned this pull request Sep 14, 2024

The active test run was aborted. Reason: Test host process crashed dotnet/dnceng#451

Open

3 tasks

Merge branch 'dotnet:main' into fix-garnet-persistence

3aaf95d

Alirexaa mentioned this pull request Sep 15, 2024

Redis, Garent and Valkey doesn't have playground app #5723

Closed

3 tasks

Merge branch 'dotnet:main' into fix-garnet-persistence

3adddab

Alirexaa added 4 commits September 17, 2024 12:27

Merge branch 'main' into fix-garnet-persistence

e13427e

Address PR feedback

0c4e3a5

Merge branch 'dotnet:main' into fix-garnet-persistence

8edbd70

Merge branch 'dotnet:main' into fix-garnet-persistence

ad4bc74

mitchdenny approved these changes Sep 19, 2024

View reviewed changes

mitchdenny merged commit dc43825 into dotnet:main Sep 19, 2024
11 checks passed

Alirexaa deleted the fix-garnet-persistence branch September 19, 2024 07:46

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix garnet persistence #5087

Fix garnet persistence #5087

Alirexaa commented Jul 26, 2024 •

edited by dotnet-policy-service bot

Loading

Alirexaa commented Jul 26, 2024

Alirexaa commented Jul 30, 2024

eerhardt left a comment

Alirexaa commented Jul 31, 2024

eerhardt Jul 31, 2024

Alirexaa Jul 31, 2024

eerhardt Aug 1, 2024

mitchdenny Aug 12, 2024

eerhardt commented Aug 1, 2024

mitchdenny commented Aug 7, 2024

Alirexaa commented Aug 8, 2024

Alirexaa commented Sep 3, 2024

eerhardt commented Sep 4, 2024

Alirexaa commented Sep 4, 2024

Alirexaa commented Sep 5, 2024

eerhardt commented Sep 5, 2024

davidfowl commented Sep 16, 2024

Alirexaa commented Sep 16, 2024

Alirexaa commented Sep 18, 2024

mitchdenny commented Sep 19, 2024

Fix garnet persistence #5087

Fix garnet persistence #5087

Conversation

Alirexaa commented Jul 26, 2024 • edited by dotnet-policy-service bot Loading

Microsoft Reviewers: Open in CodeFlow

Alirexaa commented Jul 26, 2024

Alirexaa commented Jul 30, 2024

eerhardt left a comment

Choose a reason for hiding this comment

Alirexaa commented Jul 31, 2024

eerhardt Jul 31, 2024

Choose a reason for hiding this comment

Alirexaa Jul 31, 2024

Choose a reason for hiding this comment

eerhardt Aug 1, 2024

Choose a reason for hiding this comment

mitchdenny Aug 12, 2024

Choose a reason for hiding this comment

eerhardt commented Aug 1, 2024

mitchdenny commented Aug 7, 2024

Alirexaa commented Aug 8, 2024

Alirexaa commented Sep 3, 2024

eerhardt commented Sep 4, 2024

Alirexaa commented Sep 4, 2024

Alirexaa commented Sep 5, 2024

eerhardt commented Sep 5, 2024

davidfowl commented Sep 16, 2024

Alirexaa commented Sep 16, 2024

Alirexaa commented Sep 18, 2024

mitchdenny commented Sep 19, 2024

Alirexaa commented Jul 26, 2024 •

edited by dotnet-policy-service bot

Loading