Flatpak OSMScoutServer on PureOS: server stopped #419

omhaal · 2023-02-09T08:13:36Z

I have a Purism Librem 5 smartphone that runs PureOS (a Debian derivative). I use Flatpak to install OSMScoutServer. This worked perfectly fine on version 2.2.2. However, after upgrading to version 3.0.0, when I start OSMScoutServer, I see the following messages appear in sequence:

Activating server via DBus
Starting server
Server stopped

It won't progress beyond "Server stopped".

Because I was sure I somehow borked the upgrade from 2.2.2, I flatpak uninstall --delete-data io.github.rinigus.OSMScoutServer and rm -rf ~/Maps.OSM (directory where map data is stored). After a fresh install I can run the app, select the map data directory, download "Mapbox GL World overlay", "Mapbox GL fonts" and "Address parsing language support".

However, when subscribing to the map data of a specific country, I run into the problem described above.

How to reproduce:

Do clean installation of version 3.0.0 using Flatpak on PureOS
Download the suggested world overlay, fonts and address parsing
Subscribe to some country (let's say Fiji because of small size) and download the maps
See error appear in GUI: "OSM Scout Server is not reachable. Waiting for it to appear."

Expected behavior would be to not see this error and be able to query the server.

This is the output of the flatpak run command:

kf.kirigami: Failed to find a Kirigami platform plugin
file:///usr/lib/qml/org/kde/kirigami.2/private/PrivateActionToolButton.qml:76:5: QML Binding: Binding loop detected for property "value"

The session log found in ~/.var/app/io.github.rinigus.OSMScoutServer/cache/osmscout-server/osmscout-server/session.log although it seems of little help debugging this:

INFO: 08:59:53 Storage folder changed to /home/purism/Maps.OSM
INFO: 08:59:53 Add country or feature to requested list: Address parsing language support
WARNING: 08:59:53 No maps loaded: Missing Mapbox GL World overlay
INFO: 08:59:53 Missing data: Australia and Oceania / Fiji (70.150.595)
INFO: 08:59:53 Missing data: Mapbox GL World overlay (96.862.208)
INFO: 08:59:53 Missing data: Mapbox GL fonts (75.730.947)
INFO: 08:59:53 Missing data: Address parsing language support (106.527.869)
WARNING: 08:59:53 No maps loaded: Missing Mapbox GL World overlay
INFO: 08:59:53 Missing data: Australia and Oceania / Fiji (70.150.595)
INFO: 08:59:53 Missing data: Mapbox GL World overlay (96.862.208)
INFO: 08:59:53 Missing data: Mapbox GL fonts (75.730.947)
INFO: 08:59:53 Missing data: Address parsing language support (106.527.869)
INFO: 08:59:53 Add country or feature to requested list: Mapbox GL World overlay
WARNING: 08:59:53 No maps loaded: Missing Mapbox GL World overlay
INFO: 08:59:53 Missing data: Australia and Oceania / Fiji (70.150.595)
INFO: 08:59:53 Missing data: Mapbox GL World overlay (96.862.208)
INFO: 08:59:53 Missing data: Mapbox GL fonts (75.730.947)
INFO: 08:59:53 Missing data: Address parsing language support (106.527.869)
INFO: 08:59:53 Add country or feature to requested list: Mapbox GL fonts
WARNING: 08:59:53 No maps loaded: Missing Mapbox GL World overlay
INFO: 08:59:53 Missing data: Australia and Oceania / Fiji (70.150.595)
INFO: 08:59:53 Missing data: Mapbox GL World overlay (96.862.208)
INFO: 08:59:53 Missing data: Mapbox GL fonts (75.730.947)
INFO: 08:59:53 Missing data: Address parsing language support (106.527.869)
INFO: 08:59:53 Available country or territory: Address parsing language support
INFO: 08:59:53 Available country or territory: Mapbox GL World overlay
INFO: 08:59:53 Available country or territory: Mapbox GL fonts
INFO: 08:59:53 Available country or territory: Australia and Oceania / Fiji
INFO: 08:59:53 Countries used in search: australia-oceania/fiji

Output from journalctl -b -g osm:

feb 09 08:26:11 pureos systemd[840]: Started app-flatpak-io.github.rinigus.OSMScoutServer-2486.scope.
feb 09 08:26:12 pureos dbus-daemon[877]: [session uid=1000 pid=877] Activating service name='io.github.rinigus.OSMScoutServer' requested by ':1.109' (uid=1000 pid=2502 comm="xdg-dbus-proxy --args=42 ")
feb 09 08:26:12 pureos systemd[840]: Started app-flatpak-io.github.rinigus.OSMScoutServer-2515.scope.
feb 09 08:26:14 pureos dbus-daemon[877]: [session uid=1000 pid=877] Activated service 'io.github.rinigus.OSMScoutServer' failed: Process io.github.rinigus.OSMScoutServer exited with status 132
feb 09 08:26:14 pureos systemd[840]: app-flatpak-io.github.rinigus.OSMScoutServer-2515.scope: Succeeded.

The text was updated successfully, but these errors were encountered:

omhaal · 2023-02-09T08:28:36Z

Also, I tried downgrading to 2.2.2, but apparently the new maps don't work with the old server.

rinigus · 2023-02-09T16:56:36Z

Yes, it is impossible to run old server with the new maps - hence release version change.

Try to run

flatpak run --command=/app/bin/osmscout-server io.github.rinigus.OSMScoutServer

That would run the server without GUI. Would that work? Try to use it with Pure Maps, for example

omhaal · 2023-02-09T18:58:42Z

When I run

flatpak run --command=/app/bin/osmscout-server io.github.rinigus.OSMScoutServer

I get the same output I pasted above from the session.log and then it exists with exit code 132.

rinigus · 2023-02-09T19:03:58Z

OK, so it is a crash. now would have to think how to debug it

rinigus · 2023-02-09T19:49:38Z

To debug, we have to follow the guide https://docs.flatpak.org/en/latest/debugging.html :

flatpak install --user io.github.rinigus.OSMScoutServer.Debug

# choose runtime/org.kde.Sdk/x86_64/5.15-22.08 when prompted
flatpak install --user org.kde.Sdk

Then enter container environment:

flatpak run --command=sh --devel io.github.rinigus.OSMScoutServer

and inside it:

gdb /app/bin/osmscout-server

# in gdb
(gdb) run

omhaal · 2023-02-10T14:32:55Z

You'll have to bear with me, I don't use gdb often:

(gdb) run
Starting program: /app/bin/osmscout-server 
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/usr/lib/aarch64-linux-gnu/libthread_db.so.1".
[New Thread 0xfffff1bfedc0 (LWP 18)]
[New Thread 0xfffff13eedc0 (LWP 19)]
[Detaching after fork from child process 20]
INFO: 15:31:06 Storage folder changed to /home/purism/Maps.OSM
INFO: 15:31:06 Add country or feature to requested list: Address parsing language support
WARNING: 15:31:06 No maps loaded: Missing Mapbox GL World overlay
INFO: 15:31:06 Missing data: Australia and Oceania / Fiji (70.150.595)
INFO: 15:31:06 Missing data: Mapbox GL World overlay (96.862.208)
INFO: 15:31:06 Missing data: Mapbox GL fonts (75.730.947)
INFO: 15:31:06 Missing data: Address parsing language support (106.527.869)
WARNING: 15:31:06 No maps loaded: Missing Mapbox GL World overlay
INFO: 15:31:06 Missing data: Australia and Oceania / Fiji (70.150.595)
INFO: 15:31:06 Missing data: Mapbox GL World overlay (96.862.208)
INFO: 15:31:06 Missing data: Mapbox GL fonts (75.730.947)
INFO: 15:31:06 Missing data: Address parsing language support (106.527.869)
INFO: 15:31:06 Add country or feature to requested list: Mapbox GL World overlay
WARNING: 15:31:06 No maps loaded: Missing Mapbox GL World overlay
INFO: 15:31:06 Missing data: Australia and Oceania / Fiji (70.150.595)
INFO: 15:31:06 Missing data: Mapbox GL World overlay (96.862.208)
INFO: 15:31:06 Missing data: Mapbox GL fonts (75.730.947)
INFO: 15:31:06 Missing data: Address parsing language support (106.527.869)
INFO: 15:31:06 Add country or feature to requested list: Mapbox GL fonts
WARNING: 15:31:06 No maps loaded: Missing Mapbox GL World overlay
INFO: 15:31:06 Missing data: Australia and Oceania / Fiji (70.150.595)
INFO: 15:31:06 Missing data: Mapbox GL World overlay (96.862.208)
INFO: 15:31:06 Missing data: Mapbox GL fonts (75.730.947)
INFO: 15:31:06 Missing data: Address parsing language support (106.527.869)
INFO: 15:31:07 Available country or territory: Address parsing language support
INFO: 15:31:07 Available country or territory: Mapbox GL World overlay
INFO: 15:31:07 Available country or territory: Mapbox GL fonts
INFO: 15:31:07 Available country or territory: Australia and Oceania / Fiji
INFO: 15:31:07 Countries used in search: australia-oceania/fiji 

Thread 1 "osmscout-server" received signal SIGILL, Illegal instruction.
0x0000fffff6a894f0 in kyotocabinet::AtomicInt64::set(long) () from /app/lib/libkyotocabinet.so.16

rinigus · 2023-02-10T15:31:51Z

Try to run bt in gdb after it crashes.

Looks like something is broken in geocoder loading (that's the one using kyotocabinet). I checked over here - Fiji loads just fine.

omhaal · 2023-02-10T15:36:54Z

(gdb) bt
#0  0x0000fffff6a894f0 in kyotocabinet::AtomicInt64::set(long) ()
   from /app/lib/libkyotocabinet.so.16
#1  0x0000aaaaaab7db3c in kyotocabinet::HashDB::load_meta() ()
#2  0x0000aaaaaab87b64 in kyotocabinet::HashDB::open(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int) ()
#3  0x0000aaaaaabd99c8 in ?? ()
#4  0x0000aaaaaab77014 in ?? ()
#5  0x0000aaaaaab55604 in ?? ()
#6  0x0000fffff5e5bf74 in ?? () from /usr/lib/aarch64-linux-gnu/libQt5Core.so.5
#7  0x0000aaaaaabe652c in ?? ()
#8  0x0000aaaaaab98b7c in ?? ()
#9  0x0000aaaaaab9bfd0 in ?? ()
#10 0x0000aaaaaaae43ec in ?? ()
#11 0x0000fffff5927400 in __libc_start_call_main ()
   from /usr/lib/aarch64-linux-gnu/libc.so.6
#12 0x0000fffff59274d8 in __libc_start_main ()
   from /usr/lib/aarch64-linux-gnu/libc.so.6
#13 0x0000aaaaaab532f0 in ?? ()

rinigus · 2023-02-10T15:39:48Z

Thanks! Will try to look into it - at least we know a bit more where to dig

omhaal · 2023-02-10T15:48:05Z

Thanks @rinigus! Since switching to the Librem 5 I've been using your app quite extensively for offline navigation during my hikes, so your help is much appreciated! Let me know if I need to do/test something else. Cheers!

rinigus · 2023-02-13T15:23:49Z

The issue has been reproduced in other L5 as well. Let's see if it is L5 specific, will try to test on aarch64 device over here (other OS and CPU, though)

rinigus · 2023-02-13T17:44:06Z

Couldn't reproduce it using flatpak version on Sailfish OS with QCom sdm845. So, it is not aarch64 generic issue...

AgentSmith0 · 2023-02-13T18:14:11Z

I have the same problem using the flatpak version (PinePhone Pro, Manjaro ARM).

rinigus · 2023-02-13T18:20:46Z

@AgentSmith0 - thanks for feedback.

Josh-SMU · 2023-02-13T19:39:22Z

Same issue with Flatpak on Pinephone Pro, Mobian

rinigus · 2023-02-13T19:52:49Z

Those of you on DEB based distros, try to install kyotocabinet-utils. Check if it installs kchashmgr. If it is there, navigate in terminal to your maps, then subfolder geocoder-nlp and choose country in it. After that, run a check on the corresponding database. See example:

> cd geocoder-nlp/europe-estonia
> kchashmgr check geonlp-normalized-id.kch
.................................................. (50000)
........................................... (end)
93578 records were checked successfully
>

Let's see if that fails as well.

Josh-SMU · 2023-02-13T20:05:16Z

I don't see those directories in ~/Maps.OSM which is what is created on first run. It is also not within the flatpak app directory.

rinigus · 2023-02-13T20:12:00Z

But has it managed to download any maps? If it did, there should be subfolder geocoder-nlp there...

kop316 · 2023-02-13T20:13:08Z

it looks like kchashmgr doesn't exist in PureOS.

purism@pureos:~$ sudo apt install kyotocabinet-utils
[sudo] password for purism: 
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following additional packages will be installed:
  kyotocabinet-doc libkyotocabinet16v5
Suggested packages:
  libkyotocabinet-dev
The following NEW packages will be installed:
  kyotocabinet-doc kyotocabinet-utils libkyotocabinet16v5
0 upgraded, 3 newly installed, 0 to remove and 0 not upgraded.
Need to get 1,493 kB of archives.
After this operation, 10.6 MB of additional disk space will be used.
Do you want to continue? [Y/n] n
Abort.
purism@pureos:~$ apt-cache search kchashmgr
purism@pureos:~$

Josh-SMU · 2023-02-13T20:14:11Z

Install kyotocabinet-utils and you should have kchashmgr as a command in $PATH. Installed properly for me on Mobian

Josh-SMU · 2023-02-13T20:18:52Z

But has it managed to download any maps? If it did, there should be subfolder geocoder-nlp there...

My bad, I let it download the basic worldmap but didn't realize that was stored differently from a subscribed map region... Downloading now, will post output of kchashmgr on it once finished.

edit: The .kch file checks out properly during download, not sure if this changes when the download finishes.

rinigus · 2023-02-13T20:24:51Z

@Josh-SMU, thanks! Although, without regional maps it was not supposed to crash if my idea is correct.

OK, if .kch file checks out properly (let's see if it is the same after download is finished). As far as I have seen, debian has newer kyotocabinet than used in my flatpak.

Rebuilding Flatpak with the latest kyotocabinet - maybe that will help. Build https://buildbot.flathub.org/#/builders/6/builds/23019

In addition, building one flatpak version with Kyoto tools: https://buildbot.flathub.org/#/builders/6/builds/23024

The builds would take some time, though

Josh-SMU · 2023-02-13T20:25:56Z

Ok, as soon as it finishes unpacking the regional maps, the screen changes to the error: OSM Scout Server is not reachable. Waiting for it to appear. The .kch file still checks out properly.

rinigus · 2023-02-13T20:27:44Z

That's the same bug.

Below, flatpak is with the bumped kyotocabinet version. Please uninstall released OSM Scout Server before installing this one.

(Add --user if you prefer to have it in home, I usually do):

flatpak install https://dl.flathub.org/build-repo/5661/io.github.rinigus.OSMScoutServer.flatpakref

Let's see if it still crashes

Josh-SMU · 2023-02-13T20:28:24Z

Should I also delete Maps.OSM?

rinigus · 2023-02-13T20:28:56Z

No need, should work with them

rinigus · 2023-02-13T20:29:21Z

... I hope.

Josh-SMU · 2023-02-13T20:30:14Z

Sadly the issue is the same. What I do notice is that the first line in this screen on launch is attempting to register with DBus

Josh-SMU · 2023-02-13T20:33:27Z

This may be irrelevant though, and I have confirmed via Flatseal that it has org.freedesktop.DBus as a permission granted

rinigus · 2023-02-13T20:33:58Z

Yes, DBus is needed for communication between GUI and server behind it, as well as for map matching to work with Pure Maps.

Cleaner test is

flatpak run --command=/app/bin/osmscout-server io.github.rinigus.OSMScoutServer

This starts the server only and shows logs on terminal. Has been done above at #419 (comment) . You could just check if you get crash at the same place.

PS: DBus permission should be there in flatpak manifest

Josh-SMU · 2023-02-13T20:35:02Z

Ok I'm one step ahead of you haha:

mobian@mobian:~$ flatpak run --command=sh --devel io.github.rinigus.OSMScoutServer
[📦 io.github.rinigus.OSMScoutServer ~]$ gdb /app/bin/osmscout-server
GNU gdb (GDB) 12.1
Copyright (C) 2022 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "aarch64-unknown-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /app/bin/osmscout-server...
(No debugging symbols found in /app/bin/osmscout-server)
(gdb) run
Starting program: /app/bin/osmscout-server 
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/usr/lib/aarch64-linux-gnu/libthread_db.so.1".
[New Thread 0xfffff1bfedc0 (LWP 13)]
[New Thread 0xfffff13eedc0 (LWP 14)]
[Detaching after fork from child process 15]
INFO: 14:34:24 Storage folder changed to /home/mobian/Maps.OSM
INFO: 14:34:25 Add country or feature to requested list: Address parsing language support
WARNING: 14:34:25 No maps loaded: Missing Mapbox GL World overlay
INFO: 14:34:25 Missing data: Mapbox GL World overlay (96,862,208)
INFO: 14:34:25 Missing data: Mapbox GL fonts (75,730,947)
INFO: 14:34:25 Missing data: North America / US / Wisconsin (1,243,875,933)
INFO: 14:34:25 Missing data: Address parsing language support (106,527,869)
WARNING: 14:34:25 No maps loaded: Missing Mapbox GL World overlay
INFO: 14:34:25 Missing data: Mapbox GL World overlay (96,862,208)
INFO: 14:34:25 Missing data: Mapbox GL fonts (75,730,947)
INFO: 14:34:25 Missing data: North America / US / Wisconsin (1,243,875,933)
INFO: 14:34:25 Missing data: Address parsing language support (106,527,869)
INFO: 14:34:25 Add country or feature to requested list: Mapbox GL World overlay
WARNING: 14:34:25 No maps loaded: Missing Mapbox GL World overlay
INFO: 14:34:25 Missing data: Mapbox GL World overlay (96,862,208)
INFO: 14:34:25 Missing data: Mapbox GL fonts (75,730,947)
INFO: 14:34:25 Missing data: North America / US / Wisconsin (1,243,875,933)
INFO: 14:34:25 Missing data: Address parsing language support (106,527,869)
INFO: 14:34:25 Add country or feature to requested list: Mapbox GL fonts
WARNING: 14:34:25 No maps loaded: Missing Mapbox GL World overlay
INFO: 14:34:25 Missing data: Mapbox GL World overlay (96,862,208)
INFO: 14:34:25 Missing data: Mapbox GL fonts (75,730,947)
INFO: 14:34:25 Missing data: North America / US / Wisconsin (1,243,875,933)
INFO: 14:34:25 Missing data: Address parsing language support (106,527,869)
INFO: 14:34:25 Available country or territory: Address parsing language support
INFO: 14:34:25 Available country or territory: Mapbox GL World overlay
INFO: 14:34:25 Available country or territory: Mapbox GL fonts
INFO: 14:34:25 Available country or territory: North America / US / Wisconsin
INFO: 14:34:25 Countries used in search: north-america/us/wisconsin 

Thread 1 "osmscout-server" received signal SIGILL, Illegal instruction.
0x0000fffff6a894f0 in kyotocabinet::AtomicInt64::set(long) () from /app/lib/libkyotocabinet.so.16

Josh-SMU · 2023-02-13T20:36:09Z

The weird thing is the overlay did indeed download on first run... This seems like it may somehow be a pathing issue?

rinigus · 2023-02-13T20:39:51Z

Excellent! Then I'll be off to sleep (its late over here) and leave https://buildbot.flathub.org/#/builders/6/builds/23024 this to be built. It should contain kyotocabinet tools (removed one line in flathub/io.github.rinigus.OSMScoutServer@8980a7f). That way maybe you could check that .kch from inside Flatpak environment. Let see if that will work.

As for being weird, not if it fails on geocoder loading. Lines that are supposed to come after in logs are

INFO: 22:34:58 Opened geocoder database /Maps/geocoder-nlp/europe-estonia
INFO: 22:34:58 libpostal using languages: en et fr 
ERROR: 22:34:58 Valhalla: (stat): /data/valhalla/tiles.tar No such file or directory
WARNING: 22:34:58 Valhalla: (stat): /data/valhalla/traffic.tar No such file or directory
WARNING: 22:34:58 Valhalla: Traffic tile extract could not be loaded
WARNING: 22:34:58 Valhalla: /data/valhalla/elevation/ currently has no elevation tiles
INFO: 22:34:58 Valhalla routing engine started
INFO: 22:34:58 Number of parallel worker threads: 8
INFO: 22:34:58 Countries used in search: europe/estonia europe/latvia 
INFO: 22:34:58 Opened geocoder database /Maps/geocoder-nlp/europe-estonia
INFO: 22:34:58 libpostal using languages: en et fr

Sorry, I have to stop for tonight and would be able to engage only tomorrow evening, after work.

Edit: relatively interesting function is that kyotocabinet::AtomicInt64::set

rinigus · 2023-02-13T20:48:36Z

If not mistaken then this function is causing a crash:

https://github.com/rinigus/pkg-kyotocabinet/blob/master/kcthread.cc#L2055

int64_t AtomicInt64::set(int64_t val) {
#if (defined(_SYS_MSVC_) || defined(_SYS_MINGW_)) && defined(_SYS_WIN64_)
  _assert_(true);
  return ::InterlockedExchange((uint64_t*)&value_, val);
#elif _KC_GCCATOMIC
  _assert_(true);
  int64_t oval = __sync_lock_test_and_set(&value_, val);
  __sync_synchronize();
  return oval;
#else
  _assert_(true);
  lock_.lock();
  int64_t oval = value_;
  value_ = val;
  lock_.unlock();
  return oval;
#endif
}

Will continue from here tomorrow, maybe get feedback from latest flatpak build with Kyoto tools:

https://buildbot.flathub.org/#/builders/6/builds/23024

Josh-SMU · 2023-02-13T21:22:02Z

I may be doing this wrong but running flatpak run --command=sh --devel io.github.rinigus.OSMScoutServer and then /app/bin/kchashmgr check geonlp-normalized-id.kch returns Illegal instruction. I installed the build and verified that /app/bin/kchashmgr exists.

rinigus · 2023-02-14T16:01:14Z

Excellent! Now I have to find which patches are applied in Debian and use the same ones. Will take a bit of time...

rinigus · 2023-02-14T17:41:41Z

Build without atomic operations allowed in kyotocabinet is available at

https://buildbot.flathub.org/#/builders/6/builds/23193

Can be installed with

flatpak install --user https://dl.flathub.org/build-repo/5841/io.github.rinigus.OSMScoutServer.flatpakref

Edit: don't forget to uninstall previous server flatpak

Please test:

does OSM Scout Server work now
if it doesn't, please check whether /app/bin/kchashmgr now works

Josh-SMU · 2023-02-14T17:53:04Z

Yes this works! I can set the profile to Offline in Pure Maps and use the subscribed maps.

rinigus · 2023-02-14T17:53:39Z

Excellent! Then I will cleanup the build and publish at Flathub. Thank you very much!

rinigus · 2023-02-14T19:50:44Z

Build is published and should be available via flatpak update.

Thank you all for active feedback and testing!

AgentSmith0 · 2023-02-14T20:29:45Z

Thank you very much, the update is working fine here too!

omhaal · 2023-02-15T07:02:46Z

Yes, confirmed, the flatpak update works! Thanks for your responsiveness and quick fix! As a token of my appreciation I've made a small donation through https://rinigus.github.io/donate/ :) Thanks again!

omhaal changed the title ~~Flatpak OSMScoutServer: server stopped~~ Flatpak OSMScoutServer on PureOS: server stopped Feb 9, 2023

rinigus closed this as completed Feb 14, 2023

rinigus mentioned this issue Feb 18, 2023

Database format revision rinigus/geocoder-nlp#71

Open

Flatpak OSMScoutServer on PureOS: server stopped #419

Flatpak OSMScoutServer on PureOS: server stopped #419

Comments

omhaal commented Feb 9, 2023 • edited Loading

omhaal commented Feb 9, 2023

rinigus commented Feb 9, 2023

omhaal commented Feb 9, 2023

rinigus commented Feb 9, 2023

rinigus commented Feb 9, 2023

omhaal commented Feb 10, 2023

rinigus commented Feb 10, 2023

omhaal commented Feb 10, 2023

rinigus commented Feb 10, 2023

omhaal commented Feb 10, 2023

rinigus commented Feb 13, 2023

rinigus commented Feb 13, 2023

AgentSmith0 commented Feb 13, 2023

rinigus commented Feb 13, 2023

Josh-SMU commented Feb 13, 2023

rinigus commented Feb 13, 2023

Josh-SMU commented Feb 13, 2023

rinigus commented Feb 13, 2023

kop316 commented Feb 13, 2023

Josh-SMU commented Feb 13, 2023

Josh-SMU commented Feb 13, 2023 • edited Loading

rinigus commented Feb 13, 2023

Josh-SMU commented Feb 13, 2023

rinigus commented Feb 13, 2023 • edited Loading

Josh-SMU commented Feb 13, 2023

rinigus commented Feb 13, 2023

rinigus commented Feb 13, 2023

Josh-SMU commented Feb 13, 2023

Josh-SMU commented Feb 13, 2023

rinigus commented Feb 13, 2023 • edited Loading

Josh-SMU commented Feb 13, 2023

Josh-SMU commented Feb 13, 2023 • edited Loading

rinigus commented Feb 13, 2023 • edited Loading

rinigus commented Feb 13, 2023

Josh-SMU commented Feb 13, 2023

rinigus commented Feb 14, 2023

rinigus commented Feb 14, 2023 • edited Loading

Josh-SMU commented Feb 14, 2023

rinigus commented Feb 14, 2023

rinigus commented Feb 14, 2023

AgentSmith0 commented Feb 14, 2023

omhaal commented Feb 15, 2023

omhaal commented Feb 9, 2023 •

edited

Loading

Josh-SMU commented Feb 13, 2023 •

edited

Loading

rinigus commented Feb 13, 2023 •

edited

Loading

rinigus commented Feb 13, 2023 •

edited

Loading

Josh-SMU commented Feb 13, 2023 •

edited

Loading

rinigus commented Feb 13, 2023 •

edited

Loading

rinigus commented Feb 14, 2023 •

edited

Loading