Ray Gizmos is not available in Heuristic mode

[BlueFisher]

The Gizmos of RayPerceptionSensorComponent is not available in Heuristic mode.

When the NN model is None and I click the play button, ML-Agents will get into the Heuristic mode. But I find the RayPerceptionSensor will not work. Specifically the int Write(WriteAdapter adapater); in ISensor will never be called, thus the m_RaySensor.debugDisplayInfo.rayInfos will always be Null.

BTW, I sometimes need m_RaySensor to draw rays with LineRender. Could you just make the m_RaySensor in RayPerceptionSensorComponentBase be protecetd.

• ML-Agents version: v0.14.0

[chriselion]

Hi @BlueFisher,
The gizmo problem has already been reported (#3412), plus there were some other requests to draw the gizmos outside of play mode. This was resolved in #3484. If you want a "smaller" workaround, #3542 (just merged) will step the sensors in Heuristic mode, which should also be enough to get the gizmos drawing.

I'll look into making the sensor more accessible - would a public get property be enough instead of making it protected?

[chriselion]

BTW, I'm curious what you're doing for the line rendering. Are you using https://docs.unity3d.com/Manual/class-LineRenderer.html or something else?

[BlueFisher]

Hi @chriselion ,
Thanks for your reply. The Gizmos only shows in the "scene" window of Unity editor but not in the "game" window or in executables. I sometimes need to record the inference process and want to visualize all the rays. So LineRenderer might help me achieve this.

Actually, it would be great if you guys decide to integrate this kind of visualization into RayPerceptionSensorComponent. Otherwise, I can implement a simple visualization by myself if I can access to the some properties of RayPerceptionSensorComponentBase like m_RaySensor.

[chriselion]

Sorry, I forgot to update:
There's now a get property for the RayPerceptionSensor:

ml-agents/com.unity.ml-agents/Runtime/Sensors/RayPerceptionSensorComponentBase.cs

Lines 147 to 150 in 97e4fea

	public RayPerceptionSensor raySensor
	{
	get => m_RaySensor;
	}

and I made the method for getting the RayPerceptionInput public:

ml-agents/com.unity.ml-agents/Runtime/Sensors/RayPerceptionSensorComponentBase.cs

Lines 238 to 254 in 97e4fea

	public RayPerceptionInput GetRayPerceptionInput()
	{
	var rayAngles = GetRayAngles(raysPerDirection, maxRayDegrees);
	var rayPerceptionInput = new RayPerceptionInput();
	rayPerceptionInput.rayLength = rayLength;
	rayPerceptionInput.detectableTags = detectableTags;
	rayPerceptionInput.angles = rayAngles;
	rayPerceptionInput.startOffset = GetStartVerticalOffset();
	rayPerceptionInput.endOffset = GetEndVerticalOffset();
	rayPerceptionInput.castRadius = sphereCastRadius;
	rayPerceptionInput.transform = transform;
	rayPerceptionInput.castType = GetCastType();
	rayPerceptionInput.layerMask = rayLayerMask;
	return rayPerceptionInput;
	}

Hopefully that should have you covered.

I'm not sure what other packages like ours do for visualization; do you have any other good examples of "in-game" visualization?

[BlueFisher]

Hi @chriselion, Thank you so much.
As for "in-game" visualization, I am so sorry that I'm new to Unity. All I know is the simple LineRenderer. Sorry that I can't give you a good recommendation.

[chriselion]

OK, in that case I don't think we'll plan on adding that as a feature; in general, I think we should provide debug visualization via gizmos, but shouldn't draw anything in the "game'.

[github-actions]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.