Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fixes #2815: Non-linear bias corrupts SurfaceNormals when requested through AirSim #2845

Merged
merged 3 commits into from
Jul 24, 2020
Merged

Fixes #2815: Non-linear bias corrupts SurfaceNormals when requested through AirSim #2845

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
63be6fc
Mark post process material for normals as unlit for some speed up
LSBOSS Jul 10, 2020
c3b461f
Fixes (#2815): SurfaceNormals and Segmentation getting non-linear bias
LSBOSS Jul 10, 2020
ba331d3
Replacing magic numbers with enum values
LSBOSS Jul 13, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Fixes (#2815): SurfaceNormals and Segmentation getting non-linear bias
  • Loading branch information
@LSBOSS
LSBOSS committed Jul 10, 2020
commit c3b461f113be94709d1e8c5ec2b209b040fb4bce
26 changes: 18 additions & 8 deletions Unreal/Plugins/AirSim/Source/PIPCamera.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -286,12 +286,21 @@ void APIPCamera::setupCameraFromSettings(const APIPCamera::CameraSetting& camera
const auto& noise_setting = camera_setting.noise_settings.at(image_type);

if (image_type >= 0) { //scene capture components
if (image_type==0 || image_type==5 || image_type==6 || image_type==7)
updateCaptureComponentSetting(captures_[image_type], render_targets_[image_type], false,
image_type_to_pixel_format_map_[image_type], capture_setting, ned_transform);
else
updateCaptureComponentSetting(captures_[image_type], render_targets_[image_type], true,
image_type_to_pixel_format_map_[image_type], capture_setting, ned_transform);
switch (image_type) {
case 0:
case 7:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Having direct int values makes it more difficult to understand, might be better to use enum fields such as ImageType::Scene etc.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Makes absolute sense - I just bluntly copied what was there before :) Let me fix this in the next commit to this pull request.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

DONE 👍

updateCaptureComponentSetting(captures_[image_type], render_targets_[image_type], false, image_type_to_pixel_format_map_[image_type], capture_setting, ned_transform, false);
break;

case 5:
case 6:
updateCaptureComponentSetting(captures_[image_type], render_targets_[image_type], true, image_type_to_pixel_format_map_[image_type], capture_setting, ned_transform, true);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So does forcing Linear Gamma Space change the values in segmentation image like you mentioned in the issue?
(I have no idea what this does though 😅 )

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It absolutely does. Now the values of the segmentation output correspond to those of mentioned seg_rgbs.txt.

The ForeLinearGamma flag basically tells UE4 to not apply any transformation of input colors (which UE4 is usally doing) when running through gamma correction. The background is that UE4 internally stores color information in sRGB space wherever it can for a better color reproduction. That's a non linear space that, if displayed on your monitor directly, seems to be too dark for the human eye. Here come's the gamma correction into action and is "correcting" things back to normal.

That's all fine for ImageType::Scene but not for textures not in sRGB space and where exact colors are important. E.g. Segmentation and SurfaceNormals.

Copy link
Contributor

@madratman madratman Jul 24, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for another good PR.

Here's a segmentation test. Use this settings.json

{
    "SettingsVersion": 1.2,
    "SimMode": "ComputerVision",
    "Vehicles": {
        "MyVehicle": {
            "VehicleType": "ComputerVision",
            "Cameras": {
                "my_camera": {
                    "CaptureSettings": [
                        {
                            "ImageType": 5,
                            "FOV_Degrees": 90,
                            "Width": 1024,
                            "Height": 1024
                        }
                    ],
                    "X": 0.0,
                    "Y": 0.0,
                    "Z": 0.0,
                    "Pitch": 0.0,
                    "Roll": 0.0,
                    "Yaw": 0.0
                }
            }
        }
    }
}

with the following:

import airsim
import numpy as np

client = airsim.VehicleClient(timeout_value = 7200)
client.confirmConnection()

responses = client.simGetImages([airsim.ImageRequest("my_camera", airsim.ImageType.Segmentation, False, False)])

img1d = np.fromstring(responses[0].image_data_uint8, dtype=np.uint8)
img_rgb = img1d.reshape(responses[0].height, responses[0].width, 3)
print(np.unique(img_rgb.reshape(-1, img_rgb.shape[-1]), axis=0))

With PR, we get the following BGR values:

[[ 17 194 164]
 [103 216  44]
 [121   9 150]
 [205  72  54]
 [225  35  73]]

With 1.3.1 binaries, we get:

[[ 73 226 210]
 [170 237 115]
 [183  53 202]
 [232 145 127]
 [241 104 146]]

break;

default:
updateCaptureComponentSetting(captures_[image_type], render_targets_[image_type], true, image_type_to_pixel_format_map_[image_type], capture_setting, ned_transform, false);
break;
}

setNoiseMaterial(image_type, captures_[image_type], captures_[image_type]->PostProcessSettings, noise_setting);
}
Expand All @@ -304,15 +313,16 @@ void APIPCamera::setupCameraFromSettings(const APIPCamera::CameraSetting& camera
}

void APIPCamera::updateCaptureComponentSetting(USceneCaptureComponent2D* capture, UTextureRenderTarget2D* render_target,
bool auto_format, const EPixelFormat& pixel_format, const CaptureSetting& setting, const NedTransform& ned_transform)
bool auto_format, const EPixelFormat& pixel_format, const CaptureSetting& setting, const NedTransform& ned_transform,
bool force_linear_gamma)
{
if (auto_format)
{
render_target->InitAutoFormat(setting.width, setting.height); //256 X 144, X 480
}
else
{
render_target->InitCustomFormat(setting.width, setting.height, pixel_format, false);
render_target->InitCustomFormat(setting.width, setting.height, pixel_format, force_linear_gamma);
}

if (!std::isnan(setting.target_gamma))
Expand Down
5 changes: 3 additions & 2 deletions Unreal/Plugins/AirSim/Source/PIPCamera.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -75,8 +75,9 @@ class AIRSIM_API APIPCamera : public ACameraActor

static unsigned int imageTypeCount();
void enableCaptureComponent(const ImageType type, bool is_enabled);
static void updateCaptureComponentSetting(USceneCaptureComponent2D* capture, UTextureRenderTarget2D* render_target,
bool auto_format, const EPixelFormat& pixel_format, const CaptureSetting& setting, const NedTransform& ned_transform);
static void updateCaptureComponentSetting(USceneCaptureComponent2D* capture, UTextureRenderTarget2D* render_target,
bool auto_format, const EPixelFormat& pixel_format, const CaptureSetting& setting, const NedTransform& ned_transform,
bool force_linear_gamma);
void setNoiseMaterial(int image_type, UObject* outer, FPostProcessSettings& obj, const NoiseSetting& settings);
static void updateCameraPostProcessingSetting(FPostProcessSettings& obj, const CaptureSetting& setting);
static void updateCameraSetting(UCameraComponent* camera, const CaptureSetting& setting, const NedTransform& ned_transform);
Expand Down