Backup error

[navidhaghighi]

Hello , we have an issue in our production cockroachDB.
When i try to backup a table from our Database i run into this issue:
Node liveness error

This is the command that i use:
BACKUP public.message INTO 'nodelocal://1/1' AS OF SYSTEM TIME '-10s';

BTW: our daily backups fail as well , this is the Command we use to schedule our daily backups:

CREATE SCHEDULE bkp_schedule FOR BACKUP INTO 'nodelocal://1/dailybackups' RECURRING '0 7 * * *'
FULL BACKUP ALWAYS
WITH SCHEDULE OPTIONS first_run = 'now';

---

Our setup:
CPU

Intel(R) Xeon(R) CPU E5-2697 v4 @ 2.30GHz

Base speed:	2.30 GHz
Sockets:	18
Virtual processors:	36
Virtual machine:	Yes
L1 cache:	N/A

Utilization	21%
Speed	2.30 GHz
Up time	5:05:47:36
Processes	138
Threads	13931
Handles	135384

Memory : 160 GB
Windows :10 - 64 bit architecture
CockroachDB Version : 23.1.8
Running Command:
%~dp0cockroach.exe start-single-node --insecure --cache=.25 --max-sql-memory=.25 && (echo 'didn't fail') || (start cockroach.bat)

---

Also we are using CockroachDB for the purpose of Nakama chat and messaging System.
Our backup has been recently running into this issue.
I can't tell you how to reproduce this issue specifically because we have around 1 million+ users and millions of rows in the messages table of our database , but i can provide logs if necessary.
Thank you in advance

Jira issue: CRDB-43662

[blathers-crl]

Hi @navidhaghighi, please add branch-* labels to identify which branch(es) this C-bug affects.

_{🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.}

[blathers-crl]

Hello, I am Blathers. I am here to help you get the issue triaged.

It looks like you have not filled out the issue in the format of any of our templates. To best assist you, we advise you to use one of these templates.

I have CC'd a few people who may be able to assist you:

• @cockroachdb/disaster-recovery (found keywords: backup, found keywords: CREATE SCHEDULE)
• @cockroachdb/kv (found keywords: liveness)

If we have not gotten back to your issue within a few business days, you can try the following:

• Join our community slack channel and ask on #cockroachdb.
• Try find someone from here if you know they worked closely on the area and CC them.

_{🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.}

[blathers-crl]

cc @cockroachdb/disaster-recovery

[alicia-l2]

Hello, do you mind sharing your cluster config? You may be running into a node liveness issue due to insufficient CPU for running a backup job, or other reasons -- see here for more information on how to diagnose the node liveness issue.

[navidhaghighi]

@alicia-l2
I have attached my cluster settings , i got it by using this command : SHOW CLUSTER SETTINGS ALL;
if this is not the file you asked for , please let me know.
after a shutdown and an immediate backup the issue was fixed but it started happening again.
Cluster setting.txt