Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Camera] Add lens type information (iOS) #7653

Open
wants to merge 9 commits into
base: main
Choose a base branch
from

Conversation

lenzpaul
Copy link

@lenzpaul lenzpaul commented Sep 16, 2024

This PR adds lens type information. The goal is to identify whether the lens type is wide, ultra-wide, telephoto, etc., as discussed in flutter/flutter#119908. The iOS implementation is complete, so lens data will now be populated on iOS.

  • Introduces a new CameraLensType enum to provide lens type information about the camera (e.g.: ultra-wide, telephoto, ...)
  • Adds lensType in the PlatformCameraDescription and CameraDescription classes
  • Implements utility functions to convert between PlatformCameraLensType and CameraLensType.
  • Updates auto-generated code (using Pigeon) to reflect these changes.

Current CameraDescription for iPhone 11

[
    CameraDescription(
        com.apple.avfoundation.avcapturedevice.built-in_video:0,
        CameraLensDirection.back,
        90
    ),
    CameraDescription(
        com.apple.avfoundation.avcapturedevice.built-in_video:1,
        CameraLensDirection.front,
        90
    ),
    CameraDescription(
        com.apple.avfoundation.avcapturedevice.built-in_video:5,
        CameraLensDirection.back,
        90
    )
]

New CameraDescription for iPhone 11

[
    CameraDescription(
        com.apple.avfoundation.avcapturedevice.built-in_video:0,
        CameraLensDirection.back,
        90,
        CameraLensType.wide
    ),
    CameraDescription(
        com.apple.avfoundation.avcapturedevice.built-in_video:1,
        CameraLensDirection.front,
        90,
        CameraLensType.wide
    ),
    CameraDescription(
        com.apple.avfoundation.avcapturedevice.built-in_video:5,
        CameraLensDirection.back,
        90,
        CameraLensType.ultraWide
    )
]

Pre-launch Checklist

If you need help, consider asking for advice on the #hackers-new channel on Discord.

@lenzpaul lenzpaul changed the title [Camera] Add lens type information [Camera] Add lens type information [ios] Sep 17, 2024
@lenzpaul lenzpaul changed the title [Camera] Add lens type information [ios] [Camera] Add lens type information (iOS) Sep 17, 2024
@lenzpaul lenzpaul marked this pull request as ready for review September 17, 2024 15:01
@stuartmorgan
Copy link
Contributor

Please see https://github.com/flutter/flutter/blob/master/docs/ecosystem/contributing/README.md#changing-federated-plugins for information on how to make this PR pass CI checks, which is important for review.

@lenzpaul
Copy link
Author

Please see https://github.com/flutter/flutter/blob/master/docs/ecosystem/contributing/README.md#changing-federated-plugins for information on how to make this PR pass CI checks, which is important for review.

I made changes as specified by the docs, please let me if there's anything else I need to do

@stuartmorgan
Copy link
Contributor

  • All existing and new tests are passing.

This PR is currently failing a substantial number of our CI checks, so this checkbox state is not accurate. If you click through the details of each CI task failure you can see the specific issues that will need to be fixed before this would be ready for review.

@lenzpaul
Copy link
Author

lenzpaul commented Sep 21, 2024

Hi @stuartmorgan, I fixed tests and most CI errors.

One of the remaining errors is related to versions and the changelog not being updated. But I think that some packages don't require version bumps since there are no changes. Could you confirm if this assumption is correct?

  • camera_avfoundation, camera_platform_interface: Version already bumped and CHANGELOG.md updated.
  • camera, camera_android, camera_android_camerax, camera_web, camera_windows: No changes, no version bump needed.

The other issue is with formatting. I ran the format command using Flutter Plugin Tools, but there's still an error in camera/camera_avfoundation/ios/camera_avfoundation/Sources/camera_avfoundation/include/camera_avfoundation/QueueUtils.h. Should I manually update the formatting for this file?

@stuartmorgan
Copy link
Contributor

But I think that some packages don't require version bumps since there are no changes. Could you confirm if this assumption is correct?

Yes; for packages that don't require any changes at all, you can just revert the dependency_override changes in those packages, at which point the CI will stop flagging them as issue.

The other issue is with formatting. I ran the format command using Flutter Plugin Tools, but there's still an error in camera/camera_avfoundation/ios/camera_avfoundation/Sources/camera_avfoundation/include/camera_avfoundation/QueueUtils.h. Should I manually update the formatting for this file?

This is probably due to a clang-format difference between your local tools and the CI; the CI is the authoritative version, so you'll need to match it by hand in this case, yes.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As a high level review note: as discussed here, we don't model platform interface APIs directly on one platform's APIs. A comparison of lens type information across at least iOS, Android, and web will be a necessary part of evaluating whether the values of this new enum are the ones we want.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is the best way to discuss this more in-depth? Is discord the best channel ?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Discord or the GitHub issue are both options for discussing API design. For something like this, I think the best place to start would be a comparison table showing how each platform's lens information does or doesn't align with other platforms.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here is a comparison table @stuartmorgan

Lens Type iOS Android Web Windows macOS Linux
General Support Available via AVCaptureDeviceType API Lens type info can be determined using Camera2 API. LENS_FACING for front/back/external, and field of view calculated using focal length (LENS_INFO_AVAILABLE_FOCAL_LENGTHS) and sensor size (SENSOR_INFO_PHYSICAL_SIZE). CameraX does not expose this level of detail. Lens type info not available. Web APIs (MediaDevices) allow access to cameras but without detailed hardware info on lens types. Lens type info not available. No native API for detailed lens types, standard webcam info available via DirectShow/MediaFoundation APIs Lens type info available via AVCaptureDevice API. If the attached camera (e.g., with telephoto or wide lens) supports detailed lens info, the API can retrieve it. However, most built-in or external webcams do not expose multiple lens types Lens type info not available. No native API for detailed lens types, standard webcam info available via v4l2 (Video4Linux2)
Ultra-Wide Supported Supported No lens type info No lens type info Supported No lens type info
Telephoto Supported Supported No lens type info No lens type info Supported No lens type info
Wide Supported (default) Supported (default) No lens type info No lens type info Supported (default) No lens type info

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks; what about the other values in your enum? Are dual, dual-wide, and triple supported on Android? What does "continuity" mean conceptually as a lens type, and is there some more abstract definition of it that could apply to other platforms?

(The issue might be a better place for this, as the table is pretty hard to read in the review comment UI.)

@@ -1,6 +1,7 @@
## NEXT

* Updates minimum supported SDK version to Flutter 3.19/Dart 3.3.
* Adds lensType in the PlatformCameraDescription
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We have to bump the version in order to publish the change.

[reply addObject:[FCPPlatformCameraDescription makeWithName:device.uniqueID
lensDirection:lensFacing]];
lensDirection:lensFacing
lensType:lensType]];
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: I know you are following the prior convention of FCPPlatformCameraLensDirection here, but can you move both FCPPlatformCameraLensDirection and FCPPlatformCameraLensType to a helper function since it's getting long?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants