Skip to content

feat(mlkit): add open-source document scanner (fixes Google Drive scan) - #3843

Open
grenadin wants to merge 11 commits into
microg:masterfrom
grenadin:feature/mlkit-document-scanner
Open

grenadin wants to merge 11 commits into
microg:masterfrom
grenadin:feature/mlkit-document-scanner

Conversation

@grenadin

@grenadin grenadin commented Oct 1, 2026

Copy link
Copy Markdown
Contributor

Apps using the ML Kit Document Scanner API (GmsDocumentScanning), such as Google Drive, currently show
"Update Google Play services – This update is needed to use scanner" on microG.

Screenshot_20260930_081623_com google android apps docs

Fixes #3228, fixes #3587. Related: #3078, #2018.

Cause

  1. The client calls ModuleInstall.installModules for mlkit.docscan.ui, mlkit.docscan.shadow and
    mlkit.docscan.stain; microG answered Status.CANCELED, so the client shows the update dialog.
  2. The client then starts com.google.android.gms.mlkit.ACTION_SCAN_DOCUMENT in com.google.android.gms,
    for which microG had no activity.
  3. The result is read from plain extras (uri_array_extra_result_image_uris, uri_extra_result_pdf_uri,
    int_extra_result_page_count); the client copies the files and builds GmsDocumentScanningResult itself.
Logcat before (Google Drive 2.26.387, microG 0.3.17.252432-hw)
09-30 08:34:49.468 W/LibraryVersion(22373): Failed to get app version for libraryName: play-services-mlkit-document-scanner
09-30 08:34:49.481 D/ModuleInstall(13790): bound by: GetServiceRequest[MODULE_INSTALL, packageName="com.google.android.apps.docs", gmsVersion=263830000, scopes=[], extras=Bundle[{}], defaultFeatures=[], apiFeatures=[Feature[moduleinstall,7]], supportsConnectionInfo=true]
09-30 08:34:51.801 I/ActivityTaskManager( 1831): START u0 {act=android.intent.action.SEND flg=0x1 cmp=com.google.android.apps.docs/.drive.capture.DocScannerActivity (has extras)}
09-30 08:34:51.863 D/ModuleInstall(13790): Not yet implemented: installModules ApiFeatureRequest[features=[Feature[mlkit.docscan.shadow,1], Feature[mlkit.docscan.stain,1], Feature[mlkit.docscan.ui,1]], urgent=true]
09-30 08:34:51.864 D/ClientTelemetryService(13790): log: TelemetryData[... methodKey=24335, resultStatusCode=602 ...]
09-30 08:34:51.879 I/Dialog  (22373): show dialog,width:-2,height:-2
09-30 08:34:53.980 I/ActivityTaskManager( 1831): START u0 {act=android.intent.action.VIEW cmp=com.huawei.appmarket/.service.externalapi.view.ThirdApiWithAffinityActivity}

Changes

Open-source implementation building on @DaVinci9196's earlier work: it follows the activity pattern and uses
the CameraX dependency from BarcodeScanningActivity (#2787), and the OpenCV dependency from the face
detection module (#2793). No proprietary modules or ML models are downloaded and no new dependencies are added.

The activity implements the public ML Kit Document Scanner contract rather than anything specific to Drive, so
any app using play-services-mlkit-document-scanner works without changes (the test client below is one such
app). The edge detection, perspective correction, filters and PDF writer in org.microg.gms.vision.document
do not depend on the activity and can be reused by other features.

  • ModuleInstall: report the docscan features as installed only if an activity for
    ACTION_SCAN_DOCUMENT resolves; other requests are unchanged.
  • MLKit: DocumentScanningActivity handles the request and returns JPEG pages and a PDF via
    FileProvider with read grants. Cancel returns RESULT_CANCELED.
  • Vision: camera view, document edge detection, perspective correction and page filters in
    org.microg.gms.vision.document.

Scanner features

The scanner UI and camera are implemented in microG itself. Options sent by the caller are respected:

  • Live document outline in the camera preview (OpenCV on the analysis stream), also for colored
    documents and documents taking up only a small part of the frame
  • Automatic capture when the outline is stable, with a Manual/Auto toggle; the default follows
    int_extra_default_capture_mode
  • Flash toggle (torch) when the device has a flash and boolean_extra_flash_mode_change_allowed is set
  • Camera selection via string_extra_camera_id, falling back to the default back camera
  • Review screen after each capture: the outline shown at capture time is kept, corners can be dragged
    with a magnifier, plus Automatic crop / No crop / Rotate, then keep or retake
  • Preview screen with page thumbnails: add pages, crop & rotate, filter, retake or delete a page
  • Filters (boolean_extra_filter_allowed): Original, Auto, Grayscale, Black & white. Auto is the default
    when boolean_extra_enable_auto_enhancements is set
  • Shadow removal (boolean_extra_shadow_removal_allowed / boolean_extra_stain_removal_allowed):
    background estimation and normalization with classical OpenCV operations, not Google's ML models, so
    it evens out shadows and light stains but not heavy stains
  • Multiple pages, respecting int_extra_page_limit_max
  • Gallery import when boolean_extra_gallery_import_allowed is set, with automatic edge detection
    and EXIF orientation applied
  • Results as requested in int_array_extra_result_formats: JPEG per page and a PDF with the JPEG pages
    embedded directly (about the size of the JPEGs combined)

Not implemented: the page edit callback (boolean_extra_page_edit_listener_enabled /
bundle_binder_extra_callbacks), as its binder interface is unknown, and image hashes for gallery imports
(Drive sets boolean_extra_enable_compute_hash_for_gallery_image to false).

Testing

A debug-signed build cannot use signature spoofing on EMUI, so I could not sign in to Drive with it.
I tested with a minimal client: https://github.com/grenadin/mlkit-docscan-test
It uses the official ML Kit library play-services-mlkit-document-scanner as described in the
ML Kit documentation and the official sample
googlesamples/mlkit/android/documentscanner.

Device: Huawei MatePad (MLR-W09), Android 12, huawei flavor

Case Result
Update dialog No longer shown
Live outline: black and white / colored document Works
Live outline while moving the camera closer and further away Works
Low light, with and without flash Works
Automatic capture, Manual/Auto toggle Works
Outline kept between preview and review screen Works
Capture → perspective corrected Works
Gallery import → edges detected automatically Works
Adjust corners with magnifier / Automatic crop / No crop / Rotate Works
Preview: add, crop & rotate, retake, delete pages Works
Filters: Original / Auto / Grayscale / Black & white, shadow removal Works
string_extra_camera_id = 1 → front camera; unknown id → back camera Works
5 pages → 5 JPEGs + 5-page PDF (1.5 MB) read by client Works
Cancel Client receives RESULT_CANCELED

Known limitations: edge detection may fail on white or grey backgrounds and when a corner is outside the
frame (corners can be adjusted manually); stain removal is weaker than Google's ML-based one; not tested
inside Google Drive itself.

Note

#3713 also handles ACTION_SCAN_DOCUMENT, using Google's scanner module. Both PRs register the same action,
so only one entry point can be active if both are merged. I'm happy to adjust this PR to fit whichever
direction is preferred.

This is my largest code contribution to microG so far. I've been following and learning from the project for
about four years, but I'm sure there are conventions I haven't picked up yet, so any feedback on the code or on
how things are usually done here is very welcome.

@mar-v-in

mar-v-in commented Oct 7, 2026

Copy link
Copy Markdown
Member

Thanks for this, I had this on my agenda as an alternative for #3713 (which is why I stopped considering it for 0.3.17) as I think it is much better to bundle our own implementation rather than loading Google's proprietary one as a module.

I'm currently rather busy, so I'll do a more thorough review at a later point

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Google Drive scanner requires Google Play Services update [com.google.android.apps.docs] Google Drive Scan function not working

2 participants