roachtest: cdc/frontier-persistence-benchmark/interval=30s/tables=10000/ranges=1 failed

[cockroach-teamcity]

roachtest.cdc/frontier-persistence-benchmark/interval=30s/tables=10000/ranges=1 failed with artifacts on master @ 2eeff2e7ccd8a50ea573e2c7c42a4829cde1e398:

Failed with:

(monitor.go:267).Wait: monitor failure: changefeed[default]: max latency was more than allowed: 3m35.918418171s vs 2m0s
test artifacts and logs in: /artifacts/cdc/frontier-persistence-benchmark/interval=30s/tables=10000/ranges=1/run_1

Cluster Node to Ip Mapping:

Node	Public IP	Private IP
teamcity-20971545-1768374662-16-n4cpu16-0001	18.116.10.29	10.12.1.122
teamcity-20971545-1768374662-16-n4cpu16-0002	3.15.137.10	10.12.5.249
teamcity-20971545-1768374662-16-n4cpu16-0003	18.118.141.64	10.12.3.166
teamcity-20971545-1768374662-16-n4cpu16-0004	3.148.207.29	10.12.1.157

Parameters:

• arch=amd64
• cloud=aws
• coverageBuild=false
• cpu=16
• encrypted=false
• fs=ext4
• localSSD=true
• runtimeAssertionsBuild=false
• ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Grafana is not yet available for aws clusters

/cc @cockroachdb/cdc

This test on roachdash | Improve this report!

Jira issue: CRDB-58672

[cockroach-teamcity]

roachtest.cdc/frontier-persistence-benchmark/interval=30s/tables=10000/ranges=1 failed with artifacts on master @ 2eeff2e7ccd8a50ea573e2c7c42a4829cde1e398:

Failed with:

(latency_verifier.go:208).assertValid: changefeed[default]: latency never dropped to acceptable steady level: 2m0s
(monitor.go:267).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/cdc/frontier-persistence-benchmark/interval=30s/tables=10000/ranges=1/run_1

Cluster Node to Ip Mapping:

Node	Public IP	Private IP
teamcity-20971566-1768373782-32-n4cpu16-0001	40.76.99.146	10.1.0.14
teamcity-20971566-1768373782-32-n4cpu16-0002	20.55.83.21	10.1.0.138
teamcity-20971566-1768373782-32-n4cpu16-0003	20.51.236.239	10.1.0.140
teamcity-20971566-1768373782-32-n4cpu16-0004	20.62.251.133	10.1.0.160

Parameters:

• arch=amd64
• cloud=azure
• coverageBuild=false
• cpu=16
• encrypted=false
• fs=ext4
• localSSD=true
• runtimeAssertionsBuild=false
• ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Grafana is not yet available for azure clusters

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.cdc/frontier-persistence-benchmark/interval=30s/tables=10000/ranges=1 failed with artifacts on master @ 2eeff2e7ccd8a50ea573e2c7c42a4829cde1e398:

Failed with:

(latency_verifier.go:208).assertValid: changefeed[default]: latency never dropped to acceptable steady level: 2m0s
(monitor.go:267).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/cdc/frontier-persistence-benchmark/interval=30s/tables=10000/ranges=1/run_1

Cluster Node to Ip Mapping:

Node	Public IP	Private IP
teamcity-20971544-1768375480-12-n4cpu16-0001	34.74.197.19	10.142.15.219
teamcity-20971544-1768375480-12-n4cpu16-0002	34.139.58.202	10.142.15.218
teamcity-20971544-1768375480-12-n4cpu16-0003	104.196.116.104	10.142.15.224
teamcity-20971544-1768375480-12-n4cpu16-0004	34.139.207.129	10.142.0.178

Parameters:

• arch=amd64
• cloud=gce
• coverageBuild=false
• cpu=16
• encrypted=false
• runtimeAssertionsBuild=false
• ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.cdc/frontier-persistence-benchmark/interval=30s/tables=10000/ranges=1 failed with artifacts on master @ cb94db961b8f55e3473f279d98ae90f0eeb0adcb:

Failed with:

(latency_verifier.go:208).assertValid: changefeed[default]: latency never dropped to acceptable steady level: 2m0s
(monitor.go:267).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/cdc/frontier-persistence-benchmark/interval=30s/tables=10000/ranges=1/run_1

Cluster Node to Ip Mapping:

Node	Public IP	Private IP
teamcity-20978886-1768459579-34-n4cpu16-0001	20.106.172.77	10.1.0.137
teamcity-20978886-1768459579-34-n4cpu16-0002	20.127.171.241	10.1.0.138
teamcity-20978886-1768459579-34-n4cpu16-0003	74.235.140.189	10.1.0.139
teamcity-20978886-1768459579-34-n4cpu16-0004	20.127.232.55	10.1.0.116

Parameters:

• arch=amd64
• cloud=azure
• coverageBuild=false
• cpu=16
• diskCount=0
• encrypted=false
• fs=ext4
• localSSD=true
• runtimeAssertionsBuild=false

Help

See: roachtest README

See: How To Investigate (internal)

Grafana is not yet available for azure clusters

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.cdc/frontier-persistence-benchmark/interval=30s/tables=10000/ranges=1 failed with artifacts on master @ cb94db961b8f55e3473f279d98ae90f0eeb0adcb:

Failed with:

(latency_verifier.go:208).assertValid: changefeed[default]: latency never dropped to acceptable steady level: 2m0s
(monitor.go:267).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/cdc/frontier-persistence-benchmark/interval=30s/tables=10000/ranges=1/cpu_arch=arm64/run_1

Cluster Node to Ip Mapping:

Node	Public IP	Private IP
teamcity-20978876-1768461121-26-n4cpu16-0001	3.129.69.64	10.12.9.2
teamcity-20978876-1768461121-26-n4cpu16-0002	3.144.232.58	10.12.13.120
teamcity-20978876-1768461121-26-n4cpu16-0003	18.118.95.69	10.12.7.248
teamcity-20978876-1768461121-26-n4cpu16-0004	3.141.193.112	10.12.15.102

Parameters:

• arch=arm64
• cloud=aws
• coverageBuild=false
• cpu=16
• diskCount=0
• encrypted=false
• fs=ext4
• localSSD=true
• runtimeAssertionsBuild=false

Help

See: roachtest README

See: How To Investigate (internal)

Grafana is not yet available for aws clusters

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.cdc/frontier-persistence-benchmark/interval=30s/tables=10000/ranges=1 failed with artifacts on master @ cb94db961b8f55e3473f279d98ae90f0eeb0adcb:

Failed with:

(monitor.go:267).Wait: monitor failure: changefeed[default]: max latency was more than allowed: 3m33.143108599s vs 2m0s
test artifacts and logs in: /artifacts/cdc/frontier-persistence-benchmark/interval=30s/tables=10000/ranges=1/cpu_arch=arm64/run_1

Cluster Node to Ip Mapping:

Node	Public IP	Private IP
teamcity-20978875-1768461135-52-n4cpu16-0001	34.72.252.1	10.128.0.126
teamcity-20978875-1768461135-52-n4cpu16-0002	34.46.116.72	10.128.0.111
teamcity-20978875-1768461135-52-n4cpu16-0003	104.197.231.214	10.128.0.119
teamcity-20978875-1768461135-52-n4cpu16-0004	34.171.142.66	10.128.0.168

Parameters:

• arch=arm64
• cloud=gce
• coverageBuild=false
• cpu=16
• diskCount=0
• encrypted=false
• fs=ext4
• localSSD=false
• runtimeAssertionsBuild=false

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.cdc/frontier-persistence-benchmark/interval=30s/tables=10000/ranges=1 failed with artifacts on master @ fd8129a1c152c527c43c94d5cb8355ab74e9dd4d:

Failed with:

(monitor.go:267).Wait: monitor failure: changefeed[default]: max latency was more than allowed: 3m50.302144514s vs 2m0s
test artifacts and logs in: /artifacts/cdc/frontier-persistence-benchmark/interval=30s/tables=10000/ranges=1/run_1

Cluster Node to Ip Mapping:

Node	Public IP	Private IP
teamcity-20984888-1768546406-32-n4cpu16-0001	20.102.98.145	10.1.0.125
teamcity-20984888-1768546406-32-n4cpu16-0002	57.151.100.14	10.1.0.128
teamcity-20984888-1768546406-32-n4cpu16-0003	74.235.141.17	10.1.0.129
teamcity-20984888-1768546406-32-n4cpu16-0004	20.55.84.198	10.1.0.106

Parameters:

• arch=amd64
• cloud=azure
• coverageBuild=false
• cpu=16
• diskCount=0
• encrypted=false
• fs=ext4
• localSSD=true
• runtimeAssertionsBuild=false

Help

See: roachtest README

See: How To Investigate (internal)

Grafana is not yet available for azure clusters

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.cdc/frontier-persistence-benchmark/interval=30s/tables=10000/ranges=1 failed with artifacts on master @ fd8129a1c152c527c43c94d5cb8355ab74e9dd4d:

Failed with:

(latency_verifier.go:208).assertValid: changefeed[default]: latency never dropped to acceptable steady level: 2m0s
(monitor.go:267).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/cdc/frontier-persistence-benchmark/interval=30s/tables=10000/ranges=1/run_1

Cluster Node to Ip Mapping:

Node	Public IP	Private IP
teamcity-20984877-1768548238-49-n4cpu16-0001	35.185.39.101	10.142.0.74
teamcity-20984877-1768548238-49-n4cpu16-0002	35.196.111.138	10.142.0.14
teamcity-20984877-1768548238-49-n4cpu16-0003	34.74.189.131	10.142.0.82
teamcity-20984877-1768548238-49-n4cpu16-0004	35.231.30.83	10.142.3.128

Parameters:

• arch=amd64
• cloud=gce
• coverageBuild=false
• cpu=16
• diskCount=0
• encrypted=false
• fs=ext4
• localSSD=true
• runtimeAssertionsBuild=false

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.cdc/frontier-persistence-benchmark/interval=30s/tables=10000/ranges=1 failed with artifacts on master @ 377adba1a7814b2b18b3ad19462efbd9c8226b98:

Failed with:

(monitor.go:267).Wait: monitor failure: changefeed[default]: max latency was more than allowed: 3m24.216272199s vs 2m0s
test artifacts and logs in: /artifacts/cdc/frontier-persistence-benchmark/interval=30s/tables=10000/ranges=1/run_1

Cluster Node to Ip Mapping:

Node	Public IP	Private IP
teamcity-20992783-1768633008-33-n4cpu16-0001	40.90.250.125	10.1.0.139
teamcity-20992783-1768633008-33-n4cpu16-0002	20.127.184.37	10.1.0.138
teamcity-20992783-1768633008-33-n4cpu16-0003	20.127.168.13	10.1.0.137
teamcity-20992783-1768633008-33-n4cpu16-0004	52.186.179.164	10.1.0.143

Parameters:

• arch=amd64
• cloud=azure
• coverageBuild=false
• cpu=16
• diskCount=0
• encrypted=false
• fs=ext4
• localSSD=true
• runtimeAssertionsBuild=false

Help

See: roachtest README

See: How To Investigate (internal)

Grafana is not yet available for azure clusters

This test on roachdash | Improve this report!