Idea/Question: How difficult would select → handwriting recognition be?

[jdkruzr]

And, jumping off that question re: handwriting rec: there is one feature in Boox Notes that is almost painfully incomplete: the "Scribble" search stars and boxes system. This being the thing where you go into "search" in Notes, pick the pen for "Scribbles" and then tap either the square or the asterisk, and it will search through your pages for either character existing on a line, recognize the line, and then show you every time that's happened in a list. Nice! Except then you... can't do anything else with the recognized text. All you can do is click it and it will take you where it found it. Okay, but what if you could click it and it instead gave you a "share to" dialog so you could create a to-do item in another app?

I realize I'm going far down a rabbit hole here, but it can't hurt to get the idea out (and for me, anyway, this would be absolutely transformational in meetings).

[Ethran]

It shouldn't be difficult, its been done already:

https://github.com/goblincontent/notable

And see branch on my repo:
https://github.com/Ethran/notable/tree/goblincontent

But I just tested it, and decided that its not good enough to be useful. It made a lot of mistakes with my handwriting.

[jdkruzr]

hm. what engine is it using?

[Ethran]

I think it's Google mlkit

https://developers.google.com/ml-kit/vision/digital-ink-recognition

[Ethran]

I would also like to see handwriting recognition for mathematical formulas (directly to LaTeX), but it might be very difficult to achieve. I suspect that there aren't any good solutions available.

[jdkruzr]

so I just came back to this and WOW have you made a lot of progress. 1) is there somewhere we can go to contribute to the project, like Ko-Fi or something, 2) is there any way to get an APK with the recognition feature turned on? even if it isn't perfect I'd like to give it a try.

[Ethran]

1. is there somewhere we can go to contribute to the project, like Ko-Fi or something,

I did not setup anything, if there is interest in app I might to it.

2. is there any way to get an APK with the recognition feature turned on? even if it isn't perfect I'd like to give it a try.

For now, not. After some research I think only reasonable option is google mlkit, and I want to introduce it to notable.

[jdkruzr]

I tried messing around with this a little myself. It seems like the goblincontent fork does not launch. But as far as I can tell, MLKit isn't too difficult to use.

[Ethran]

Yeah, getting it to work won’t be the hard part—I think the real challenge will be storing the text and making it actually useful. I’ll need to find more time to start working on it; it’s one of the biggest tasks ahead. But first, bookmarks are probably the next goal (issue #52 ). I’m planning to use OCR to generate smart tags for them, which should make them a lot more helpful.

Also , huge thanks for donation!

[jdkruzr]

was thinking about the "where to store it" question as well. in some cases you might want to put it in an explicitly placed text box, in some ("recognize whole document") you might want a sort of "background" or "invisible" text page which can be called to the front for reading or indexed for searching. it would be really great if we could turn on an option for "lazy recognition of whole documents for searching later," for example only when the device is plugged in.

I couldn't find a way to improve the quality of the model by correcting its results, but that might also just be asking too much from these devices.