support filter effect

[fukemy]

Hi, i want to add some filter effect like tiktok, and I just make a demo, it succesfull show my custom image in other view, but logo i wrong orientation, did u ever try to work with this feature?

The step:

1. Convert to UIImage
2. Using VNFaceDetector to detect face boundingBox
3. Add my filter image into camera image
4. Convert back to CMSampleBuffer and pass to delegate

Here is my sample code:

extension WebRTCClient: AVCaptureVideoDataOutputSampleBufferDelegate{
   func captureOutput(_ output: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection) {
       
       guard let imageBuffer = CMSampleBufferGetImageBuffer(sampleBuffer) else {
           return
       }
       let ciimage = CIImage(cvPixelBuffer: imageBuffer)
//        let image = UIImage(ciImage: ciimage, scale: 1.0, orientation: imageOrientation())
       let image = convert(cmage: ciimage)
       print("image: \(image.size)")
       
       let faceDetectionRequest = VNDetectFaceLandmarksRequest(completionHandler: { (request: VNRequest, error: Error?) in
           DispatchQueue.main.async {[weak self] in
               if let observations = request.results as? [VNFaceObservation], !observations.isEmpty {
                   for observation in observations {
                       let box = observation.boundingBox
                       let boxFrame = CGRect(x: box.origin.x * image.size.width, y: box.origin.y * image.size.height, width: box.width * image.size.width, height: box.height * image.size.height)
                       print("box: \(boxFrame)")
                       let logo = UIImage(named: "dog_nose")!.rotate(radians: .pi * 2)
                       if let newImage = self?.drawImageIn(image, logo, inRect: boxFrame){
                           if let pxBuffer = self?.convertImageToBuffer(from: newImage){
                               var newSampleBuffer: CMSampleBuffer? = nil
                               var timimgInfo: CMSampleTimingInfo = .invalid
                               var videoInfo: CMVideoFormatDescription? = nil
                               CMVideoFormatDescriptionCreateForImageBuffer(allocator: nil, imageBuffer: pxBuffer, formatDescriptionOut: &videoInfo)
                               if videoInfo != nil{
                                   CMSampleBufferCreateForImageBuffer(allocator: kCFAllocatorDefault, imageBuffer: pxBuffer, dataReady: true, makeDataReadyCallback: nil, refcon: nil, formatDescription: videoInfo!, sampleTiming: &timimgInfo, sampleBufferOut: &newSampleBuffer)
                                   if newSampleBuffer != nil{
                                       self?.outputCaptureDelegate?.captureOutput!(output, didOutput: newSampleBuffer!, from: connection)
                                   }
                               }

                           }
                       }
                   }
               }
               self?.outputCaptureDelegate?.captureOutput!(output, didOutput: sampleBuffer, from: connection)
           }
       })
       let imageRequestHandler = VNImageRequestHandler(cvPixelBuffer: imageBuffer, orientation: exifOrientationForCurrentDeviceOrientation(), options: [:])
       do {
           try imageRequestHandler.perform([faceDetectionRequest])
       } catch {
           print(error.localizedDescription)
       }
   }
   
   func imageOrientation() -> UIImage.Orientation {
       let curDeviceOrientation = UIDevice.current.orientation
       var exifOrientation: UIImage.Orientation
       switch curDeviceOrientation {
           case UIDeviceOrientation.portraitUpsideDown:  // Device oriented vertically, Home button on the top
               exifOrientation = .left
           case UIDeviceOrientation.landscapeLeft:       // Device oriented horizontally, Home button on the right
               exifOrientation = .upMirrored
           case UIDeviceOrientation.landscapeRight:      // Device oriented horizontally, Home button on the left
               exifOrientation = .down
           case UIDeviceOrientation.portrait:            // Device oriented vertically, Home button on the bottom
               exifOrientation = .up
           default:
               exifOrientation = .up
       }
       return exifOrientation
   }
   
   func exifOrientationForCurrentDeviceOrientation() -> CGImagePropertyOrientation {
       return exifOrientationForDeviceOrientation(UIDevice.current.orientation)
   }
   
   func exifOrientationForDeviceOrientation(_ deviceOrientation: UIDeviceOrientation) -> CGImagePropertyOrientation {
       
       switch deviceOrientation {
       case .portraitUpsideDown:
           return .rightMirrored
           
       case .landscapeLeft:
           return .downMirrored
           
       case .landscapeRight:
           return .upMirrored
           
       default:
           return .leftMirrored
       }
   }
   
   func convert(cmage: CIImage) -> UIImage {
       let context = CIContext(options: nil)
       let cgImage = context.createCGImage(cmage, from: cmage.extent)!
       let image = UIImage(cgImage: cgImage)
       return image
   }
   
   func drawImageIn(_ image: UIImage, _ logo: UIImage, inRect: CGRect) -> UIImage {
       let renderer = UIGraphicsImageRenderer(size: image.size)
       return renderer.image { context in
           image.draw(in: CGRect(origin: CGPoint.zero, size: image.size))
           logo.draw(in: inRect)
       }
   }
   
   func convertImageToBuffer(from image: UIImage) -> CVPixelBuffer? {
       let attrs = [
           String(kCVPixelBufferCGImageCompatibilityKey) : true,
           String(kCVPixelBufferCGBitmapContextCompatibilityKey) : true
       ] as [String : Any]
       var buffer : CVPixelBuffer?
       let status = CVPixelBufferCreate(kCFAllocatorDefault, Int(image.size.width), Int(image.size.height), kCVPixelFormatType_32ARGB, attrs as CFDictionary, &buffer)
       guard (status == kCVReturnSuccess) else {
           return nil
       }
       
       CVPixelBufferLockBaseAddress(buffer!, CVPixelBufferLockFlags(rawValue: 0))
       let pixelData = CVPixelBufferGetBaseAddress(buffer!)
       
       let rgbColorSpace = CGColorSpaceCreateDeviceRGB()
       let context = CGContext(data: pixelData, width: Int(image.size.width), height: Int(image.size.height), bitsPerComponent: 8, bytesPerRow: CVPixelBufferGetBytesPerRow(buffer!), space: rgbColorSpace, bitmapInfo: CGImageAlphaInfo.noneSkipFirst.rawValue)
       
       context?.translateBy(x: 0, y: image.size.height)
       context?.scaleBy(x: 1.0, y: -1.0)
       
       UIGraphicsPushContext(context!)
       image.draw(in: CGRect(x: 0, y: 0, width: image.size.width, height: image.size.height))
       UIGraphicsPopContext()
       CVPixelBufferUnlockBaseAddress(buffer!, CVPixelBufferLockFlags(rawValue: 0))
       
       return buffer
   }
}
extension UIImage {
   func rotate(radians: CGFloat) -> UIImage {
       let rotatedSize = CGRect(origin: .zero, size: size)
           .applying(CGAffineTransform(rotationAngle: CGFloat(radians)))
           .integral.size
       UIGraphicsBeginImageContext(rotatedSize)
       if let context = UIGraphicsGetCurrentContext() {
           let origin = CGPoint(x: rotatedSize.width / 2.0,
                                y: rotatedSize.height / 2.0)
           context.translateBy(x: origin.x, y: origin.y)
           context.rotate(by: radians)
           draw(in: CGRect(x: -origin.y, y: -origin.x,
                           width: size.width, height: size.height))
           let rotatedImage = UIGraphicsGetImageFromCurrentImageContext()
           UIGraphicsEndImageContext()

           return rotatedImage ?? self
       }

       return self
   }
}

if you have any experience on this, please tech me the problem

[Meonardo]

This is great about what you are doing! I was going to use GPUImage to interact with some filters to the camera publisher, like you I am newbie to this, sorry I am not able to create a demo about this feature.

[fukemy]

After longtime searching, i found we have almost 2 solution:

1 . Modify CmsampleBuffer and send it back to capture delegate
-> it's will make publisher video slow, and lost many package nack. i have to downsize camera and framerate + bitrate to using this solution, the result is not good as expected( may be im not good developer)

2. Add a overlay view into publisher and subscriber, when get bounding box using data channel to pass this bounding box to subscriber, from bounding box we draw into overlayview, that mean this does not effect into camera data, no need to modify samplebuffer here.

But the problem is: Im not tried to using this in a room that contain a lot of people who join this call. May be it make problem? When using data channel and send A LOT OF data to every subscriber....

[fukemy]

the sample of solution 2 is below here:
https://github.com/m10117013/WebRTC_MLKit