see what it takes to support KOReader

[raisjn]

either through IPC model or using LD_PRELOAD

see koreader/koreader#6792

[raisjn]

for rM2, it seems there is a software thread that acts like the EPDC/TCON (whatever those mean). there's discussion in KOReader and reStream#28 about it a little bit and other places.

the gist seems to be: the framebuffer is now a memory location inside a process (and not /dev/fb, any longer) and there is a software thread that accepts the equivalent commands to update_display(rect, waveform, ...). we are discussing whether to make it possible for each process to instantiate their own SWTCON or if we should have a server that provides an API for interacting with the framebuffer

@NiLuJe: i'm curious if you've seen display drivers that use a software thread for driving the display and what KOReader has done in those cases. specifically: is it reasonable to create a "framebuffer server" that would be required to be running for KOReader to run (and the framebuffer server exposes IPC API for interacting with framebuffer).

[NiLuJe]

Nope, that's entirely new territory for me ;). Would this rely on xochitl in any way? (I'm fairly sure we kill it right now?).

On eInk devices (if you forget about Android, where things are... messy), we generally expect to have full control over the framebuffer device (i.e., either we neuter the stock system (Kindle), we try to play nice with it while still mostly neutering it (PocketBook), or we murder it with fire (Kobo)).

Given the recent discussions about the launcher(s) situation on rM, a centralized server possibly makes more sense if you want to help stuff interoperate. But, technically, both model would work just fine, I think?

[NiLuJe]

On a mildly related note, as far as xochitl is concerned, you might want to take a look at what @pgaskin came up with to avoid having to setup a global LD_PRELOAD and still hook into the Qt5 Kobo app, by piggybacking on the Qt5 image plugin system: https://github.com/pgaskin/NickelHook/

[raisjn]

The server process will rely on using functions from either xochitl or remarkable-shutdown but those processes don't actually run. We use LD_PRELOAD to take over the process, like LD_PRELOAD=rm2fb.so xochitl and then we use our own main() func but call into the APIs we need to 1) start the SWTCON threads and 2) send updates. If we go server model, this only needs to happen once (instead of each application doing this dance)

https://github.com/ddvk/remarkable2-framebuffer/blob/master/src/loader/main.cpp#L100

NickelHook looks interesting, thanks! we will need to read on it and see if we can re-use those ideas.

[ddvk]

all this now is a stopgap solution, until we can drive the framebuffer ourselves (in which case everyone can write their own implementstion) or use the propietary lib from the SDK when/if it is available.

[pgaskin]

If you need help with injection or hooking (both plain C apps, Qt5 apps, and the kernel itself) and other things like that, I can answer any questions you have about my kobo mods and related topics. I'm not very familiar with e-ink fb quirks, though (that's @NiLuJe's area of expertise).

---

Regarding the client-server idea, I have a few concerns about that approach (I was almost going to take a similar approach with NickelHook et al, but abandoned it after some discussion):

• Latency (but this can be avoided with shared memory).
• Handling timing issues between clients (double or triple buffering may help at the expense of CPU).
• IPC compatibility between versions (the severity of this issue could be reduced by having a dynamic library which abstracts the communication method and exposes a very low-level interface).
• Screen orientation changes.
• Upgrades.
• Compatibility with future alternatives.
• Scope creep (you probably wouldn't want this ending being essentially a clone of X11).

[raisjn]

@pgaskin thanks! one major question is: we know the API of the func we want (and have its current memory address and can find it again by hand in other binaries if we need to), but is there a way to search memory for that func's address so that its more future proof? our first idea is to grab X bytes from the current func and then search for those same bytes in the other binaries.

looking at nickelhook, it seems like plugins are shared libraries that are opened with dlopen/dlsym and invoked? this is something i want to explore here, too (i see you commented on #3, thank you!)

Handling timing issues between clients (double or triple buffering may help at the expense of CPU).

what is timing issues? the SWTCON does have double buffering, afaict. they have a "vsync and flip thread" which moves data from one area to another

IPC compatibility between versions (the severity of this issue could be reduced by having a dynamic library which abstracts the communication method and exposes a very low-level interface).

agree. if we start by being compatible with the mxcfb_update struct, i think that should mostly cover the update API but not sure if missing stuff since we don't understand SWTCON yet.

[pgaskin]

we know the API of the func we want (and have its current memory address and can find it again by hand in other binaries if we need to), but is there a way to search memory for that func's address so that its more future proof? our first idea is to grab X bytes from the current func and then search for those same bytes in the other binaries.

It depends. Based on what you said, I'm assuming the remarkable binaries have hidden symbols? If so, you'll have to find the function. A few possible ideas on how this can be done:

• If you don't need to hook it before the first call, and it makes syscalls or libc calls in a specific pattern, you can trace those and then identify the caller based on that, then perform additional checks to ensure there aren't false positives.
• If you know other calls it makes, you can trace or scan for those.
• If it's a QObject, look into the Qt metadata.
• As you said, you can also scan the memory. I'd recommend looking into the ARM ABI and GCC's implementation of it.

it seems like plugins are shared libraries that are opened with dlopen/dlsym and invoked

Not for NickelHook. NickelHook plugins are the libraries themselves. Qt's plugin system does all the required dlopening. This should work here too if it's a proper Qt5 application.

My suggestion in #3 is slightly different.

what is timing issues? the SWTCON does have double buffering, afaict. they have a "vsync and flip thread" which moves data from one area to another

In that case, you should be OK in general. I'm not familiar with the reMarkable tablets.

If you end up supporting multiple clients, you'll need to make sure the writes don't end up overlapping accidentally.

... but not sure if missing stuff since we don't understand SWTCON yet.

Yes, you'll need to look into it more before designing the API if you choose to go that direction.

[raisjn]

using LD_PRELOAD shim for clients: https://imgur.com/a/ZSKw0vi

[raisjn]

Circling back on this, there are a few things that need to be done before it feels like KOReader is finished.

• testing: try KOReader with more books and images
• inputs: input is wrong on the x axis - needs to invert X (or uninvert, as the case may be). is there a setting for this or is it done in code? https://github.com/ddvk/koreader/tree/rm2
• packaging: the current rm2fb setup requires shipping server.so, client.so and remarkable-shutdown binary of a specific version. we'd like to not require a specific version and be more general purpose. master branch no longer relies on a specific binary version
• ???

i'll need help with all of the above and am going to be unavailable from 11/11 for a week or two, so if anyone wants to pick these up, will be very appreciated! @ddvk can help / give guidance

as it is now: KOReader can be run through ssh and it works.

[NiLuJe]

* inputs: input is wrong on the x axis - needs to invert X (or uninvert, as the case may be). is there a setting for this or is it done in code?

There's probably already something available for that. c.f., the Kobo device module for usage examples.

* ???

The rM device module will also need a simple detection to discriminate the two (e.g., https://github.com/NiLuJe/FBInk/blob/d8a9e81335fa57eaf31c228456e3b5e4c5f1ffb5/fbink_device_id.c#L956-L1005).

And, ideally, with a check for the shim's env var, so we can crash gracefully and point to it if it's not available ;).

[raisjn]

ddvk's work has been merged into koreader and nightly works on rm2 (with rm2fb)