dendrite keeps crashing #3148
Comments
S7evinK
added
T-Defect
|
As a quick fix for this, you should be able to rm -rf '/var/lib/dendrite/jetstream/jetstream/$G/streams/DendriteOutputRoomEvent'(where I'll be looking into this. |
|
Hi @S7evinK thanks for the quick response. |
|
Error seems back and is crashing again: |
|
Unsure if related, but mysellf I have been seeing the following panic on attempted startup since upgrading: |
|
@NyaaaWhatsUpDoc This is unrelated and should already be fixed by #3144 |
|
@clandmeter Would you mind trying the |
|
Does not seem to change much. |
|
Just noticed the same error message in the dendrite test suite: FAIL github.com/matrix-org/dendrite/clientapi 100.071stime="2023-08-02T11:16:00Z" level=info msg="Enabling server notices at /_synapse/admin/v1/send_server_notice"
time="2023-08-02T11:16:01Z" level=info msg=PerformDeviceCreation context=missing device_id="" display_name=0xc00021f2a8 localpart=_server
time="2023-08-02T11:16:01Z" level=info msg=PerformDeviceCreation device_id="" display_name="" localpart=1 req.id=8t08xqu4WQ0j req.method=POST req.path=/_matrix/client/v3/login
time="2023-08-02T11:16:03Z" level=info msg=PerformDeviceCreation device_id="" display_name="" localpart=2 req.id=GhE0kqNGfSpI req.method=POST req.path=/_matrix/client/v3/login
time="2023-08-02T11:16:18Z" level=info msg="Enabling server notices at /_synapse/admin/v1/send_server_notice"
time="2023-08-02T11:16:18Z" level=info msg=PerformDeviceCreation context=missing device_id="" display_name=0xc0003ca8a8 localpart=_server
time="2023-08-02T11:16:19Z" level=info msg=PerformDeviceCreation device_id="" display_name="" localpart=3 req.id=JFB3uOlJr5A1 req.method=POST req.path=/_matrix/client/v3/login
time="2023-08-02T11:16:20Z" level=info msg=PerformDeviceCreation device_id="" display_name="" localpart=4 req.id=UXvZU6b8zgIi req.method=POST req.path=/_matrix/client/v3/login
time="2023-08-02T11:16:44Z" level=info msg="Enabling server notices at /_synapse/admin/v1/send_server_notice"
time="2023-08-02T11:16:44Z" level=info msg=PerformDeviceCreation context=missing device_id="" display_name=0xc000586fa8 localpart=_server
time="2023-08-02T11:16:45Z" level=info msg=PerformDeviceCreation device_id="" display_name="" localpart=5 req.id=QqFsmyFvlDP4 req.method=POST req.path=/_matrix/client/v3/login
time="2023-08-02T11:16:46Z" level=info msg=PerformDeviceCreation device_id="" display_name="" localpart=6 req.id=9HTM4D2U1NdP req.method=POST req.path=/_matrix/client/v3/login
time="2023-08-02T11:16:50Z" level=info msg="Enabling server notices at /_synapse/admin/v1/send_server_notice"
time="2023-08-02T11:16:51Z" level=info msg=PerformDeviceCreation context=missing device_id="" display_name=0xc000587da8 localpart=_server
time="2023-08-02T11:16:52Z" level=info msg=PerformDeviceCreation device_id="" display_name="" localpart=7 req.id=Kxat8g6ei5Il req.method=POST req.path=/_matrix/client/v3/login
time="2023-08-02T11:16:52Z" level=info msg=PerformDeviceCreation device_id="" display_name="" localpart=8 req.id=dTBVMta5mUPj req.method=POST req.path=/_matrix/client/v3/login
time="2023-08-02T11:16:58Z" level=info msg="Enabling server notices at /_synapse/admin/v1/send_server_notice"
time="2023-08-02T11:16:58Z" level=info msg=PerformDeviceCreation context=missing device_id="" display_name=0xc0019901a8 localpart=_server
time="2023-08-02T11:16:59Z" level=info msg=PerformDeviceCreation device_id="" display_name="" localpart=9 req.id=HXFHGja5Gyuz req.method=POST req.path=/_matrix/client/v3/login
time="2023-08-02T11:17:00Z" level=info msg=PerformDeviceCreation device_id="" display_name="" localpart=10 req.id=brAyljywILxQ req.method=POST req.path=/_matrix/client/v3/login
time="2023-08-02T11:17:00Z" level=info msg="Cleaning old notifications"
time="2023-08-02T11:17:00Z" level=error msg="Failed to clean old notifications" error="sqlutil.WithTransaction.Begin: sql: database is closed"
time="2023-08-02T11:17:11Z" level=info msg="Enabling server notices at /_synapse/admin/v1/send_server_notice"
time="2023-08-02T11:17:11Z" level=info msg=PerformDeviceCreation context=missing device_id="" display_name=0xc00040a4a8 localpart=_server
time="2023-08-02T11:17:11Z" level=info msg=PerformDeviceCreation device_id="" display_name="" localpart=11 req.id=eSA0IdJkFFJt req.method=POST req.path=/_matrix/client/v3/login
time="2023-08-02T11:17:13Z" level=info msg=PerformDeviceCreation device_id="" display_name="" localpart=12 req.id=JeIh2BN1UKM6 req.method=POST req.path=/_matrix/client/v3/login
time="2023-08-02T11:17:14Z" level=info msg=PerformDeviceCreation device_id="" display_name="" localpart=vhuser req.id=ztLxI74nVT1e req.method=POST req.path=/_matrix/client/v3/login
time="2023-08-02T11:17:18Z" level=info msg="Cleaning old notifications"
time="2023-08-02T11:17:18Z" level=error msg="Failed to clean old notifications" error="sqlutil.WithTransaction.Begin: sql: database is closed"
time="2023-08-02T11:17:29Z" level=info msg="Enabled perspective key fetcher" num_public_keys=2 server_name=matrix.org
time="2023-08-02T11:17:32Z" level=info msg="Enabling server notices at /_synapse/admin/v1/send_server_notice"
time="2023-08-02T11:17:32Z" level=info msg=PerformDeviceCreation context=missing device_id="" display_name=0xc0005876a8 localpart=_server
time="2023-08-02T11:17:33Z" level=info msg=PerformDeviceCreation device_id="" display_name="" localpart=13 req.id=SBhYO9O3ZnHn req.method=POST req.path=/_matrix/client/v3/login
time="2023-08-02T11:17:34Z" level=warning msg="Evacuated room, purging room from roomserver now" evacuated_users=1 room_id="!1:test"
time="2023-08-02T11:17:34Z" level=warning msg="Purging room from roomserver" room_id="!1:test"
time="2023-08-02T11:17:34Z" level=warning msg="Room purged from roomserver, informing other components" room_id="!1:test"
time="2023-08-02T11:17:34Z" level=warning msg="Failed to evacuate room before purging" error="room does not exist" room_id="!doesnotexist:localhost"
time="2023-08-02T11:17:34Z" level=warning msg="Purging room from federation API" room_id="!1:test"
time="2023-08-02T11:17:34Z" level=warning msg="Room purged from federation API" room_id="!1:test"
time="2023-08-02T11:17:34Z" level=panic msg="roomserver output log: write new event failure" add="[$uaHByYO0K1SQlIVpEXSLNdYjjqkRr76Y1KPxgt3FZo4]" del="[$XhiKUU8FMg1tjoimJ0dbHgats5gpBJKmuXliiiSvwqY]" error="disk I/O error" event="{\"auth_events\":[\"$1mJkXNxeyQcmoNd7js4maIqgmrD5-5hruuqenV7VEbs\",\"$VRSXPuIdgAbLZGQroZcW-hAXVVYmfNdfVx14VlBbtE4\",\"$XhiKUU8FMg1tjoimJ0dbHgats5gpBJKmuXliiiSvwqY\"],\"content\":{\"membership\":\"leave\"},\"depth\":7,\"hashes\":{\"sha256\":\"cqjqdooNZ7pznErxHFEyhydZ63ZvIYE2cKN9Gg2STqQ\"},\"origin\":\"test\",\"origin_server_ts\":1690975054095,\"prev_events\":[\"$Ywxd1AjLmqWqR6eHTRS39JVRROiECyxdjFhOGwkMh20\"],\"prev_state\":[],\"room_id\":\"!1:test\",\"sender\":\"@13:test\",\"signatures\":{\"test\":{\"ed25519:auto\":\"0B5CxYOsFFqbnXNYPQui5WE1ZzQmlaVdhKS/shr3mjWBUgyaiCTUSFERx9iF7UwVflvkGfjKee1JTTZRwAM1Cw\"}},\"state_key\":\"@13:test\",\"type\":\"m.room.member\",\"unsigned\":{\"prev_content\":{\"membership\":\"join\"},\"prev_sender\":\"@13:test\",\"replaces_state\":\"$XhiKUU8FMg1tjoimJ0dbHgats5gpBJKmuXliiiSvwqY\"}}" event_id="$uaHByYO0K1SQlIVpEXSLNdYjjqkRr76Y1KPxgt3FZo4"
panic: (*logrus.Entry) 0xc003c7c070
|
|
@S7evinK sorry to bump this issue, but we are currently stuck with a broken instance. |
|
Can this issue be closed? it was fixed in i believe this commit 3e314e0. Any release after v0.13.1 should not have this issue. |
|
It's still crashing for us: # dendrite --version
0.13.2+a721294Is there anything we need or can do to fix it with the latest version? stacktrace |
|
I ran into this issue, not in #3228, but later on. After crashing the first time, repeated crashes were a segfault: |
just like @clandmeter, removing this file prevents outbound federated messages from being sent. I've gone from a potentially temporarily corrupted crashing instance to a half functional permanently broken instance. Update: After waiting 20 minutes, outbound messages suddenly started working again. |
When I was having this issue, removing the |
In my case, it hasn't done that yet. I'll keep this message updated when/if it does recur. |
|
For us it crashes again after a couple of minutes, just as with @array-in-a-matrix |
After removing the DendriteOutputRoomEvent directory a couple of times and restarting it, it does not seem to crash anymore, though there are still some other issues left. |
I would like to reply that I am getting the same error on a fresh install using the docker-compose file. Really sucks since I was hoping to try setting up a server. How would one attempt the manual patch in docker? |
|
Same issue here |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
We are running dendrite monolith from official docker repo with version 0.13.1+49d75d3 with PG.
We suddenly are experiencing server crashes with the following in our logs:
The server keeps crashing and we are unable to start it anymore.
The text was updated successfully, but these errors were encountered: