Merge pull request #1011 from luxonis/tof_docs

Updated ToF docs
luxonis · May 20, 2024 · e0726e1 · e0726e1
2 parents 535364d + 11f8a9d
commit e0726e1
Show file tree

Hide file tree

Showing 7 changed files with 262 additions and 60 deletions.
diff --git a/docs/source/_static/images/components/tof-diagram.png b/docs/source/_static/images/components/tof-diagram.png
diff --git a/docs/source/_static/images/components/tof-optical-correction.png b/docs/source/_static/images/components/tof-optical-correction.png
diff --git a/docs/source/components/nodes/color_camera.rst b/docs/source/components/nodes/color_camera.rst
@@ -57,7 +57,7 @@ It interacts with the 3A algorithms: **auto-focus**, **auto-exposure**, and **au
 adjustments such as exposure time, sensitivity (ISO), and lens position (if the camera module has a motorized lens) at runtime.
 Click `here <https://en.wikipedia.org/wiki/Image_processor>`__ for more information.
 
-**Image Post-Processing** converts YUV420 planar frames from the **ISP** into :code:`video`/:code:`preview`/:code:`still` frames.
+**Image Post-Processing** converts YUV420 planar frames from the **ISP** into ``video``/ ``preview``/ ``still`` frames.
 
 ``still`` (when a capture is triggered) and ``isp`` work at the max camera resolution, while ``video`` and ``preview`` are
 limited to max 4K (3840 x 2160) resolution, which is cropped from ``isp``.
@@ -69,13 +69,42 @@ For IMX378 (12MP), the **post-processing** works like this:
     │ ISP ├────────────────►│  video  ├───────────────►│ preview  │
     └─────┘  max 3840x2160  └─────────┘  and cropping  └──────────┘
 
-.. image:: /_static/images/tutorials/isp.jpg
-
-The image above is the ``isp`` output from the ColorCamera (12MP resolution from IMX378). If you aren't downscaling ISP,
-the ``video`` output is cropped to 4k (max 3840x2160 due to the limitation of the ``video`` output) as represented by
-the blue rectangle. The Yellow rectangle represents a cropped ``preview`` output when the preview size is set to a 1:1 aspect
-ratio (eg. when using a 300x300 preview size for the MobileNet-SSD NN model) because the ``preview`` output is derived from
-the ``video`` output.
+If the resolution is set to 12MP and video mode is used, a 4K frame (3840x2160) will be cropped from the center of the 12MP frame.
+
+Full FOV
+########
+
+Some sensors, such as the IXM378, default to a 1080P resolution, which is a crop from the full sensor resolution. You can print sensor features to see how the field of view (FOV) is affected by the selected sensor resolution:
+
+.. code-block:: python
+
+    import depthai as dai
+
+    with dai.Device() as device:
+        for cam in dev.getConnectedCameraFeatures():
+            print(cam)
+            #continue  # uncomment for less verbosity
+            for cfg in cam.configs:
+                print("   ", cfg)
+
+    # Running on OAK-D-S2 will print:
+    # {socket: CAM_A, sensorName: IMX378, width: 4056, height: 3040, orientation: AUTO, supportedTypes: [COLOR], hasAutofocus: 1, hasAutofocusIC: 1, name: color}
+    #     {width: 1920, height: 1080, minFps: 2.03, maxFps: 60, type: COLOR, fov: {x:108, y: 440, width: 3840, height: 2160}}
+    #     {width: 3840, height: 2160, minFps: 1.42, maxFps: 42, type: COLOR, fov: {x:108, y: 440, width: 3840, height: 2160}}
+    #     {width: 4056, height: 3040, minFps: 1.42, maxFps: 30, type: COLOR, fov: {x:0, y: 0, width: 4056, height: 3040}}
+    #     {width: 1352, height: 1012, minFps: 1.25, maxFps: 52, type: COLOR, fov: {x:0, y: 0, width: 4056, height: 3036}}
+    #     {width: 2024, height: 1520, minFps: 2.03, maxFps: 85, type: COLOR, fov: {x:4, y: 0, width: 4048, height: 3040}}
+    # {socket: CAM_B, sensorName: OV9282, width: 1280, height: 800, orientation: AUTO, supportedTypes: [MONO], hasAutofocus: 0, hasAutofocusIC: 0, name: left}
+    #     {width: 1280, height: 720, minFps: 1.687, maxFps: 143.1, type: MONO, fov: {x:0, y: 40, width: 1280, height: 720}}
+    #     {width: 1280, height: 800, minFps: 1.687, maxFps: 129.6, type: MONO, fov: {x:0, y: 0, width: 1280, height: 800}}
+    #     {width: 640, height: 400, minFps: 1.687, maxFps: 255.7, type: MONO, fov: {x:0, y: 0, width: 1280, height: 800}}
+    # {socket: CAM_C, sensorName: OV9282, width: 1280, height: 800, orientation: AUTO, supportedTypes: [MONO], hasAutofocus: 0, hasAutofocusIC: 0, name: right}
+    #     {width: 1280, height: 720, minFps: 1.687, maxFps: 143.1, type: MONO, fov: {x:0, y: 40, width: 1280, height: 720}}
+    #     {width: 1280, height: 800, minFps: 1.687, maxFps: 129.6, type: MONO, fov: {x:0, y: 0, width: 1280, height: 800}}
+    #     {width: 640, height: 400, minFps: 1.687, maxFps: 255.7, type: MONO, fov: {x:0, y: 0, width: 1280, height: 800}}
+
+For the IMX378 sensor, selecting a 4K or 1080P resolution results in approximately 5% horizontal and 29% vertical FOV cropping.
+To utilize the full sensor FOV, you should choose one of the following resolutions: ``THE_12_MP``, ``THE_1352X1012``, or ``THE_2024X1520``.
 
 Usage
 #####

diff --git a/docs/source/components/nodes/tof.rst b/docs/source/components/nodes/tof.rst
@@ -30,20 +30,88 @@ Inputs and Outputs
 .. code-block::
 
               ┌───────────┐    depth
-  inputConfig |           ├────────►
-  ───────────►│           | amplitude
-     input    |    ToF    ├────────►
-  ───────────►│           │    error
+              │           ├────────►
+  inputConfig |           | amplitude
+  ───────────►│           ├────────►
+              │    ToF    │    phase
+     input    |           ├────────►
+  ───────────►│           │ intensity
               │           ├────────►
               └───────────┘
 
+
 **Message types**
 
 - ``inputConfig`` - :ref:`ToFConfig`
 - ``input`` - :ref:`ImgFrame`
-- ``depth`` - :ref:`ImgFrame`
+- ``depth`` - :ref:`ImgFrame` - Decoded depth map
 - ``amplitude`` - :ref:`ImgFrame`
-- ``error`` - :ref:`ImgFrame`
+- ``phase`` - :ref:`ImgFrame` Phase image, useful for debugging (FP32)
+- ``intensity`` - :ref:`ImgFrame`
+
+ToF Settings
+############
+
+In :ref:`ToF depth` example we allow users to quickly configure ToF settings.
+
+Here are the most important settings:
+
+* Optical Correction: It's a process that corrects the optical effect. When enabled, the ToF returns depth map (represented by Green Line on graph below) instead of distance, so it matches :ref:`StereoDepth` depth reporting. It does rectification and distance to depth conversion (Z-map).
+* Phase Unwrapping - Process that corrects the phase wrapping effect of the ToF sensor. You can set it to [0..5 are optimized]. The higher the number, the longer the ToF range, but it also increases the noise. Approximate max distance (for exact value, see :ref:`Max distance` below):
+
+  * `0` - Disabled, up to ~1.87 meters (utilizing 80MHz modulation frequency)
+  * `1` - Up to ~3 meters
+  * `2` - Up to ~4.5 meters
+  * `3` - Up to ~6 meters
+  * `4` - Up to ~7.5 meters
+
+* Burst mode: When enabled, ToF node won't reuse frames, as shown on the graph below. It's related to post-processing of the ToF frames, not the actual sensor/projector. It's disabled by default.
+* Phase shuffle Temporal filter: Averages shuffled and non-shuffled frames of the same modulation frequency to reduce noise. It's enabled by default. You can disable it to reduce :ref:`ToF motion blur` and system load.
+
+.. image:: /_static/images/components/tof-optical-correction.png
+
+Here's the time diagram which showcases how ToF decoding gets done based on the settings.
+
+.. image:: /_static/images/components/tof-diagram.png
+
+Phase unwrapping
+################
+
+If the time it takes for the light to travel from ToF sensor and back exceeds the period of the emitted wave (1.5m or 1.87m), the resulting measurement will "wrap" back to a lower value. This is called phase wrapping.
+It's similar to how a clock resets after 12 hours. Phase unwrapping is possible as our ToF has two different modulation frequencies (80Mhz and 100MHz).
+
+Phase unwrapping aims to correct this by allowing the sensor to interpret longer distances without confusion. It uses algorithms to keep track of how many cycles (round trips of the wave) have occurred,
+thus correcting the "wrapped" phases. The downside is that the more cycles the sensor has to keep track of, the more noise it introduces into the measurement.
+
+ToF motion blur
+###############
+
+To reduce motion blur, we recommend these settings:
+
+- Increase camera FPS. It goes up to 160 FPS, which causes frame capture to be the fastest (6.25ms between frames). This will reduce motion blur as ToF combines multiple frames to get the depth. Note that 160FPS will increase system load significantly (see :ref:`Debugging <Debugging DepthAI pipeline>`). Note also that higher FPS -> lower exposure times, which can increase noise.
+- Disable phase shuffle temporal filter. This will introduce more noise.
+- Disable phase unwrapping. This will reduce max distance to 1.87 meters (utilizing 80MHz modulation frequency), so about 1 cubic meter of space will be visible (very limited use-cases).
+- Enable burst mode. This is irrelevant if shuffle filter and phase unwrapping are disabled (see diagram above). When enabled, ToF node won't reuse frames (lower FPS).
+
+In the diagram above, the less frames are combined (bottom of the diagram), the less motion blur there is. The more frames are combined (top of the diagram), there's more filtering (better accuracy) but it results in more motion blur.
+
+Max distance
+############
+
+Maximum ToF distance depends on the modulation frequency and the phase unwrapping level. If phase unwrapping is enabled,
+max distance is the shorter of both modulation frequencies (so max distance at 100MHz). Here's the formula:
+
+.. math::
+  :nowrap:
+
+  \begin{align*}
+  c & = 299792458.0 \quad \text{// speed of light in m/s} \\
+  MAX\_80MHZ\_M & = \frac{c}{80000000 \times 2} = 1.873 \, \text{m} \\
+  MAX\_100MHZ\_M & = \frac{c}{100000000 \times 2} = 1.498 \, \text{m} \\
+  MAX\_DIST\_80MHZ_M & = (\text{phaseUnwrappingLevel} + 1) \times 1.873 + \frac{\text{phaseUnwrapErrorThreshold}}{2} \\
+  MAX\_DIST\_100MHZ_M & = (\text{phaseUnwrappingLevel} + 1) \times 1.498 + \frac{\text{phaseUnwrapErrorThreshold}}{2} \\
+  MAX\_DIST\_PHASE\_UNWRAPPING\_M & = MAX\_DIST\_100MHZ\_M
+  \end{align*}
 
 Usage
 #####
@@ -55,10 +123,26 @@ Usage
     pipeline = dai.Pipeline()
 
     tof_cam = pipeline.create(dai.node.Camera)
+    tof_cam.setFps(30)
     # We assume the ToF camera sensor is on port CAM_A
     tof_cam.setBoardSocket(dai.CameraBoardSocket.CAM_A)
 
     tof = pipeline.create(dai.node.ToF)
+
+    # Higher number => faster processing. 1 shave core can do 30FPS.
+    tof.setNumShaves(1)
+
+    # Median filter, kernel size 5x5
+    tof.initialConfig.setMedianFilter(dai.MedianFilter.KERNEL_5x5)
+
+    tofConfig = tof.initialConfig.get()
+    # Temporal filter averages shuffle/non-shuffle frequencies
+    tofConfig.enablePhaseShuffleTemporalFilter = True
+    # Phase unwrapping, for longer range.
+    tofConfig.phaseUnwrappingLevel = 4 # Up to 7.5 meters
+    tofConfig.phaseUnwrapErrorThreshold = 300
+    tof.initialConfig.set(tofConfig)
+
     # ToF node converts raw sensor frames into depth
     tof_cam.raw.link(tof.input)
 

diff --git a/docs/source/samples/StereoDepth/rgb_depth_aligned.rst b/docs/source/samples/StereoDepth/rgb_depth_aligned.rst
@@ -16,6 +16,12 @@ To align depth with **higher resolution color stream** (eg. 12MP), you need to l
 do that with ``stereo.setOutputSize(w,h)``. Code `example here <https://gist.github.com/Erol444/25f374fa18efa7939ec9bb848b39249a>`__.
 
 
+Host alignment
+--------------
+
+StereoDepth node aligns depth map to selected sensor (in this case, color sensor), on the OAK device itself. One can also do the same
+on the host side. We have developed a simple `demo script here <https://github.com/luxonis/depthai-python/commit/94b3177f5f4b28e562f637dcf77baa47826ef643>`__.
+
 Demo
 ####
 

diff --git a/docs/source/samples/ToF/tof_depth.rst b/docs/source/samples/ToF/tof_depth.rst
@@ -15,6 +15,22 @@ on the ToF sensor.
         <iframe src="https://www.youtube.com/embed/D2MnnyxdsMA" frameborder="0" allowfullscreen style="position: absolute; top: 0; left: 0; width: 100%; height: 100%;"></iframe>
     </div>
 
+With keyboard you can configure ToF settings:
+
+* *FPPN Correction*; Turn on/off with `f`. It's a process that corrects the fixed pattern noise (FPN) of the ToF sensor. Should be enabled.
+* *Wiggle Correction*: Turn on/off with `w`. It's a process that corrects the wiggle effect of the ToF sensor. Should be enabled.
+* *Temperature Correction*: Turn on/off with `t`. It's a process that corrects the temperature effect of the ToF sensor. Should be enabled.
+* *Optical Correction*: Turn on/off with `o`. It's a process that corrects the optical effect (On -> ToF returns distance represented by Green Line), so it matches stereo depth reporting.
+* *Phase Unwrapping* - Process that corrects the phase wrapping effect of the ToF sensor. The higher the number, the longer the ToF range, but it also increases the noise.
+
+  * `0` - Disabled, up to ~1.87 meters
+  * `1` - Up to ~3 meters
+  * `2` - Up to ~4.5 meters
+  * `3` - Up to ~6 meters
+  * `4` - Up to ~7.5 meters
+
+.. image:: /_static/images/components/tof-optical-correction.png
+
 Setup
 #####
 

diff --git a/examples/ToF/tof_depth.py b/examples/ToF/tof_depth.py
@@ -1,52 +1,119 @@
 #!/usr/bin/env python3
 
+import time
 import cv2
 import depthai as dai
 import numpy as np
 
-pipeline = dai.Pipeline()
-
-cam_a = pipeline.create(dai.node.Camera)
-# We assume the ToF camera sensor is on port CAM_A
-cam_a.setBoardSocket(dai.CameraBoardSocket.CAM_A)
-
-tof = pipeline.create(dai.node.ToF)
-
-# Configure the ToF node
-tofConfig = tof.initialConfig.get()
-# tofConfig.depthParams.freqModUsed = dai.RawToFConfig.DepthParams.TypeFMod.MIN
-tofConfig.depthParams.freqModUsed = dai.RawToFConfig.DepthParams.TypeFMod.MAX
-tofConfig.depthParams.avgPhaseShuffle = False
-tofConfig.depthParams.minimumAmplitude = 3.0
-tof.initialConfig.set(tofConfig)
-# Link the ToF sensor to the ToF node
-cam_a.raw.link(tof.input)
-
-xout = pipeline.create(dai.node.XLinkOut)
-xout.setStreamName("depth")
-tof.depth.link(xout.input)
-
-# Connect to device and start pipeline
-with dai.Device(pipeline) as device:
-    print('Connected cameras:', device.getConnectedCameraFeatures())
-    q = device.getOutputQueue(name="depth")
-
-    while True:
-        imgFrame = q.get()  # blocking call, will wait until a new data has arrived
-        depth_map = imgFrame.getFrame()
-
-        # Colorize the depth frame to jet colormap
-        depth_downscaled = depth_map[::4]
-        non_zero_depth = depth_downscaled[depth_downscaled != 0] # Remove invalid depth values
-        if len(non_zero_depth) == 0:
-            min_depth, max_depth = 0, 0
-        else:
-            min_depth = np.percentile(non_zero_depth, 3)
-            max_depth = np.percentile(non_zero_depth, 97)
-        depth_colorized = np.interp(depth_map, (min_depth, max_depth), (0, 255)).astype(np.uint8)
-        depth_colorized = cv2.applyColorMap(depth_colorized, cv2.COLORMAP_JET)
-
-        cv2.imshow("Colorized depth", depth_colorized)
-
-        if cv2.waitKey(1) == ord('q'):
-            break
+print(dai.__version__)
+
+cvColorMap = cv2.applyColorMap(np.arange(256, dtype=np.uint8), cv2.COLORMAP_JET)
+cvColorMap[0] = [0, 0, 0]
+
+def create_pipeline():
+    pipeline = dai.Pipeline()
+
+    tof = pipeline.create(dai.node.ToF)
+
+    # Configure the ToF node
+    tofConfig = tof.initialConfig.get()
+
+    # Optional. Best accuracy, but adds motion blur.
+    # see ToF node docs on how to reduce/eliminate motion blur.
+    tofConfig.enableOpticalCorrection = True
+    tofConfig.enablePhaseShuffleTemporalFilter = True
+    tofConfig.phaseUnwrappingLevel = 4
+    tofConfig.phaseUnwrapErrorThreshold = 300
+
+    tofConfig.enableTemperatureCorrection = False # Not yet supported
+
+    xinTofConfig = pipeline.create(dai.node.XLinkIn)
+    xinTofConfig.setStreamName("tofConfig")
+    xinTofConfig.out.link(tof.inputConfig)
+
+    tof.initialConfig.set(tofConfig)
+
+    cam_tof = pipeline.create(dai.node.Camera)
+    cam_tof.setFps(60) # ToF node will produce depth frames at /2 of this rate
+    cam_tof.setBoardSocket(dai.CameraBoardSocket.CAM_A)
+    cam_tof.raw.link(tof.input)
+
+    xout = pipeline.create(dai.node.XLinkOut)
+    xout.setStreamName("depth")
+    tof.depth.link(xout.input)
+
+    tofConfig = tof.initialConfig.get()
+
+    return pipeline, tofConfig
+
+
+if __name__ == '__main__':
+    pipeline, tofConfig = create_pipeline()
+
+    with dai.Device(pipeline) as device:
+        print('Connected cameras:', device.getConnectedCameraFeatures())
+        qDepth = device.getOutputQueue(name="depth")
+
+        tofConfigInQueue = device.getInputQueue("tofConfig")
+
+        counter = 0
+        while True:
+            start = time.time()
+            key = cv2.waitKey(1)
+            if key == ord('f'):
+                tofConfig.enableFPPNCorrection = not tofConfig.enableFPPNCorrection
+                tofConfigInQueue.send(tofConfig)
+            elif key == ord('o'):
+                tofConfig.enableOpticalCorrection = not tofConfig.enableOpticalCorrection
+                tofConfigInQueue.send(tofConfig)
+            elif key == ord('w'):
+                tofConfig.enableWiggleCorrection = not tofConfig.enableWiggleCorrection
+                tofConfigInQueue.send(tofConfig)
+            elif key == ord('t'):
+                tofConfig.enableTemperatureCorrection = not tofConfig.enableTemperatureCorrection
+                tofConfigInQueue.send(tofConfig)
+            elif key == ord('q'):
+                break
+            elif key == ord('0'):
+                tofConfig.enablePhaseUnwrapping = False
+                tofConfig.phaseUnwrappingLevel = 0
+                tofConfigInQueue.send(tofConfig)
+            elif key == ord('1'):
+                tofConfig.enablePhaseUnwrapping = True
+                tofConfig.phaseUnwrappingLevel = 1
+                tofConfigInQueue.send(tofConfig)
+            elif key == ord('2'):
+                tofConfig.enablePhaseUnwrapping = True
+                tofConfig.phaseUnwrappingLevel = 2
+                tofConfigInQueue.send(tofConfig)
+            elif key == ord('3'):
+                tofConfig.enablePhaseUnwrapping = True
+                tofConfig.phaseUnwrappingLevel = 3
+                tofConfigInQueue.send(tofConfig)
+            elif key == ord('4'):
+                tofConfig.enablePhaseUnwrapping = True
+                tofConfig.phaseUnwrappingLevel = 4
+                tofConfigInQueue.send(tofConfig)
+            elif key == ord('5'):
+                tofConfig.enablePhaseUnwrapping = True
+                tofConfig.phaseUnwrappingLevel = 5
+                tofConfigInQueue.send(tofConfig)
+            elif key == ord('m'):
+                medianSettings = [dai.MedianFilter.MEDIAN_OFF, dai.MedianFilter.KERNEL_3x3, dai.MedianFilter.KERNEL_5x5,
+                                  dai.MedianFilter.KERNEL_7x7]
+                currentMedian = tofConfig.median
+                nextMedian = medianSettings[(medianSettings.index(currentMedian) + 1) % len(medianSettings)]
+                print(f"Changing median to {nextMedian.name} from {currentMedian.name}")
+                tofConfig.median = nextMedian
+                tofConfigInQueue.send(tofConfig)
+
+            imgFrame = qDepth.get()  # blocking call, will wait until a new data has arrived
+            depth_map = imgFrame.getFrame()
+            max_depth = (tofConfig.phaseUnwrappingLevel + 1) * 1500 # 100MHz modulation freq.
+            depth_colorized = np.interp(depth_map, (0, max_depth), (0, 255)).astype(np.uint8)
+            depth_colorized = cv2.applyColorMap(depth_colorized, cvColorMap)
+
+            cv2.imshow("Colorized depth", depth_colorized)
+            counter += 1
+
+    device.close()