如何使用Swift scenekit在IOS上绘制摄像机视频作为背景？

I am trying to develop an augmented reality app using swift and scenekit on ios. Is there a way to draw the video captured by the device camera as a background of the scene?

我正在尝试使用ios上的swift和scenekit开发增强现实应用程序。有没有办法将设备摄像头捕获的视频绘制为场景背景？

3 个解决方案

#1

This worked for me,

这对我有用，

I used AVFoundation to capture the video input of the device camera:

我使用AVFoundation捕获设备摄像头的视频输入：

   let captureSession = AVCaptureSession()
   let previewLayer = AVCaptureVideoPreviewLayer(session: captureSession)
    previewLayer.videoGravity = AVLayerVideoGravityResizeAspectFill

    if let videoDevice = AVCaptureDevice.defaultDeviceWithMediaType(AVMediaTypeVideo) {
        var err: NSError? = nil
        if let videoIn : AVCaptureDeviceInput = AVCaptureDeviceInput.deviceInputWithDevice(videoDevice, error: &err) as? AVCaptureDeviceInput {
            if(err == nil){
                if (captureSession.canAddInput(videoIn as AVCaptureInput)){
                    captureSession.addInput(videoIn as AVCaptureDeviceInput)
                }
                else {
                    println("Failed to add video input.")
                }
            }
            else {
                println("Failed to create video input.")
            }
        }
        else {
            println("Failed to create video capture device.")
        }
    }
    captureSession.startRunning()

At this point, based on Apple's documentation of the background property of SCNScene, I expected to add the instance of AVCaptureVideoPreviewLayer to an SCNScene's background.contents, with something like:

此时，基于Apple的SCNScene背景属性文档，我希望将AVCaptureVideoPreviewLayer的实例添加到SCNScene的background.contents中，例如：

   previewLayer.frame = sceneView.bounds
   sceneView.scene.background.contents = previewLayer

This changed the background color of my scene from default white to black, but offered no video input. This may be an iOS bug?. So Plan B. Instead, I added the 'AVCaptureVideoPreviewLayer' as a sublayer of a UIView's layer:

这会将场景的背景颜色从默认白色更改为黑色，但不提供视频输入。这可能是一个iOS错误？所以计划B.相反，我添加了'AVCaptureVideoPreviewLayer'作为UIView层的子层：

 previewLayer.frame = self.view.bounds
 self.view.layer.addSublayer(previewLayer)

I then set an SCNView as a subview of that same UIView, setting the SCNView's background color to clear:

然后我将SCNView设置为同一个UIView的子视图，将SCNView的背景颜色设置为清除：

    let sceneView = SCNView()
    sceneView.frame = self.view.bounds
    sceneView.backgroundColor = UIColor.clearColor()
    self.view.addSubview(sceneView)

The device camera's video is now visible as the background of the scene.

设备摄像机的视频现在可以看作场景的背景。

I've created a small demo.

我创建了一个小型演示。

#2

Yes, you can use a AVCaptureVideoPreviewLayer as the contents of a material property (just make sure that you give the layer a bounds).

是的，您可以使用AVCaptureVideoPreviewLayer作为材质属性的内容（只需确保为图层指定边界）。

The scene view has a material property for the background that you can assign the video preview to (assign the layer to the background's contents).

场景视图具有背景的材质属性，您可以将视频预览分配给（将图层指定给背景的内容）。

var background: SCNMaterialProperty! { get }

#3

As of iOS 11, you can now use an AVCaptureDevice as a material on an object or the scene's background (see "Using Animated Content" here)

从iOS 11开始，您现在可以使用AVCaptureDevice作为对象或场景背景的材质（请参阅此处的“使用动画内容”）

Example:

例：

 
    let captureDevice = AVCaptureDevice.default(.builtInWideAngleCamera, for: .video, position: .back)!
    scnScene.background.contents = captureDevice

#1