Bug 252560

Summary: REGRESSION (iOS 16.4 Public Beta) getUserMedia ignores facingMode constraint
Product: WebKit Reporter: connell
Component: WebRTCAssignee: youenn fablet <youennf>
Status: RESOLVED FIXED    
Severity: Major CC: connell, datchu, eric.carlson, simontaylor1, webkit-bug-importer, youennf
Priority: P2 Keywords: InRadar
Version: Safari 16   
Hardware: iPhone / iPad   
OS: iOS 16   
Attachments:
Description Flags
A minimal test case showing the issue none

Description connell 2023-02-20 06:12:39 PST 
Created attachment 465085 [details]
A minimal test case showing the issue

In iOS 16.4 Beta (20E5212f), a call to `navigator.mediaDevices.getUserMedia` passing a `facingMode: "environment"` video constraint (as below) starts a stream using the user-facing camera (rather than the rear-facing camera).

```
navigator.mediaDevices.getUserMedia({
	audio: false,
	video: {
		facingMode: "environment"
	}
}).then(stream => {
	vid.srcObject = stream;
	vid.play();
})

```

This looks to be a significant regression - our testing with a device running iOS 16.3.1 shows that it starts the correct camera.

I've attached a minimal example of the issue - index.html - to this report. I'm also hosting the index.html file here:
https://bhqq.zappar.io/1962332228841157357/1.0.0/

Just tap the 'Start' button. On the affected iOS Safari version the user-facing camera appears. On 16.3.1 and earlier the rear-facing camera appears.

This is an important bug for us - we have large number of sites in production that make use of this constraint in order to select the correct device.

Let me know if I can help in any way :-)
Comment 1 youenn fablet 2023-02-20 06:40:59 PST 
Thanks, we will investigate.
Comment 2 Radar WebKit Bug Importer 2023-02-20 06:41:01 PST 
<rdar://problem/105677398>
Comment 3 youenn fablet 2023-02-20 08:10:20 PST 
@connell, which device have you used to reproduce the bug?
On my iPhone 12 Pro, it does not repro.
Comment 4 connell 2023-02-20 08:14:33 PST 
Hi @youenn! It's an iPhone XR (MRY82B/A). Thanks for taking a look so quickly :-)
Comment 5 Dat Chu 2023-02-24 17:01:33 PST 
I can confirm that this issue is repro on my iPhone 14 Plus running iOS 16.4.

It doesn't repro on my iPhone 13 Pro running iOS 16.4.
Comment 6 youenn fablet 2023-02-28 00:17:22 PST 
Pull request: https://github.com/WebKit/WebKit/pull/10784
Comment 7 Simon Taylor 2023-02-28 02:45:02 PST 
Thanks for the link to the pull request @youenn. Capping the fitness distance increase to a max of 0.5 sounds sensible.

However worth noting that Connell initially saw this on iPhone XR which only has a single environment-facing camera (no telephoto ones) so it seems unexpected that this camera would be penalised at all with a fitness distance increase?
Comment 8 EWS 2023-02-28 10:47:48 PST 
Committed 260953@main (c9e35ddc3cab): <https://commits.webkit.org/260953@main>

Reviewed commits have been landed. Closing PR #10784 and removing active labels.
Comment 9 youenn fablet 2023-02-28 10:53:54 PST 
(In reply to Simon Taylor from comment #7)
> Thanks for the link to the pull request @youenn. Capping the fitness
> distance increase to a max of 0.5 sounds sensible.
> 
> However worth noting that Connell initially saw this on iPhone XR which only
> has a single environment-facing camera (no telephoto ones) so it seems
> unexpected that this camera would be penalised at all with a fitness
> distance increase?

The issue is that the fitness score would be the same for the front camera and the back camera even if facing mode is set to environment. The PR fixes that to guarantee that the fitness score would be higher for back cameras if facing mode is set to environment
Comment 10 connell 2023-02-28 12:31:23 PST 
Great news @youenn! Thank you for moving so quickly on it. Do you know if this will make it into the next iOS 16.4 beta? Will make sure to give it a good test if so :-)
Comment 11 Simon Taylor 2023-03-01 02:33:24 PST 
(In reply to youenn fablet from comment #9)
> (In reply to Simon Taylor from comment #7)
> > Thanks for the link to the pull request @youenn. Capping the fitness
> > distance increase to a max of 0.5 sounds sensible.
> > 
> > However worth noting that Connell initially saw this on iPhone XR which only
> > has a single environment-facing camera (no telephoto ones) so it seems
> > unexpected that this camera would be penalised at all with a fitness
> > distance increase?
> 
> The issue is that the fitness score would be the same for the front camera
> and the back camera even if facing mode is set to environment. The PR fixes
> that to guarantee that the fitness score would be higher for back cameras if
> facing mode is set to environment

Yup I was confused by why when there was only one camera on the device it would have been getting a penalty before. I've read through the landed PR now, so I see the priority is a hardcoded ordering regardless of which devices a camera actually has.

The change to fitness score looks reasonable though.

The current priority list in AVVideoCaptureSource::facingModeFitnessScoreAdjustment() looks like this:
 1. AVCaptureDeviceTypeBuiltInTripleCamera
 2. AVCaptureDeviceTypeBuiltInDualWideCamera
 3. AVCaptureDeviceTypeBuiltInUltraWideCamera
 4. AVCaptureDeviceTypeBuiltInDualCamera
 5. AVCaptureDeviceTypeBuiltInWideAngleCamera
 6. AVCaptureDeviceTypeBuiltInTelephotoCamera
 7. AVCaptureDeviceTypeDeskViewCamera

ie the virtual "compound" cameras are favoured if available, but then ultra wide is preferred to the standard "1x" camera.

We make some approximate camera model assumptions in our sites assuming that the 1x camera will be the default for `facingMode: environment` scenarios.

I assume the compound cameras start up by default in a 1x mode, so I wouldn't mind those being preferred but in my view AVCaptureDeviceTypeBuiltInWideAngleCamera should be the next highest priority to limit behaviour change on other sites that don't have a UI for an explicit camera selection.
Comment 12 Simon Taylor 2023-03-01 07:52:26 PST 
(In reply to Simon Taylor from comment #11)
> I assume the compound cameras start up by default in a 1x mode, so I
> wouldn't mind those being preferred but in my view
> AVCaptureDeviceTypeBuiltInWideAngleCamera should be the next highest
> priority to limit behaviour change on other sites that don't have a UI for
> an explicit camera selection.

@youenn Actually I think the virtual triple camera looks like it will be equivalent to the ultra-wide mode by default. Natively you need to set videoZoomFactor to 2 in order to match the for of the standard camera and automatically switch between standard and ultrawide based on focus.

Can you confirm if the default back camera now looks like the ultrawide on an iPhone that has that camera? I'm trying to confirm myself on a (freshly reset) iPhone 14 Pro here but am getting errors downloading the new 16.4 Beta 2 build.

https://bugs.webkit.org/show_bug.cgi?id=250142 has a bit more discussion.

Personally I think Safari should follow the native iOS API here and use just the standard front / back cameras by default, and only expose the other ones via enumerateDevices.

At least for our use cases the rear camera being ultrawide by default would likely break some of our assumptions in our computer vision code on hundreds of live sites.
Comment 13 Simon Taylor 2023-03-03 09:45:15 PST 
I've added Bug 253186 for more discussion of the concerns around changing the default environment facing camera.
Comment 14 connell 2023-03-09 05:12:11 PST 
Just thought I'd add a note that this bug still appears to be present in iOS 16.4 Beta 3. It's making me nervous that it might end up in the shipping version. I know this isn't always possible but an insight into when a beta would contain the fix would be greatly appreciated.