Add support for floating point textures to OpenGL

[processing-bot]

Created by: AmnonOwed

I'm packing data in a texture (PImage) to send to a fragment shader. I'm running into the limitations of the 32bit RGBA format that seems to be the default for every texture in Processing. It's causing banding effects in my output. Switching to floating point textures will most likely solve these problems, because of the increased accuracy.

I have searched high and low (all the forums, the source code, the issue list), but I cannot find a way to use floating points textures in Processing. Is this possible? If not, I would like to post this issue as a feature request.

[processing-bot]

Created by: REAS

@codeanticode will know the answer

[processing-bot]

Created by: codeanticode

Hi @AmnonOwed, it is not currently possible to define a floating point texture through Processing's API, the internal format can only be normal RGB or RGBA (i.e.: 1 byte per channel).

Quickly inspecting the code, seems like it shouldn't be too difficult to add GL.GL_RGBA32F or GL.GL_RGB32F as fp formats, but this would probably require an additional hint, or some other method to signal to the renderer that the user wishes to use fp textures.

[processing-bot]

Created by: AmnonOwed

Hey guys,

Thanks for the quick response. Would be great to add this functionality. I think the two options to let the user set this would be either a) per renderer or b) per texture. With regard to these two options:

a) per renderer
This option could be something like textureFormat(int format) similar to the way textureMode(int mode), textureWrap(int wrap) and textureSampling(int sampling) currently work.

b) per texture
This option could be something like GLGraphics's textureParameters solution, but specifically for each parameter. In this case setTextureFormat(int format).

I think option a (per renderer) is most in line with the current Processsing API. So it would make sense to implement this option.

The only question I would have then: can you use the global renderer setting, to set different formats for different textures in one sketch? So for example set / keep most textures to 1 byte per channel, but set one texture to 32FLOAT. I guess this could be possible by switching the global format setting back and forth at strategic points in the sketch? Not sure how this currently works for textureWrap() etc.

[processing-bot]

Created by: benfry

@codeanticode Can this be done inside begin/endPGL? If so, we should probably close. Seems akin to adding CMYK support instead of just RGB since it touches so many parts of the API.

[processing-bot]

Created by: timseverien

Any update on this? Floating point textures are great for GPGPU.

[processing-bot]

Created by: codeanticode

Hi, as mentioned earlier, this would require adding a bunch of new API to the core. I'm happy to have a more specific discussion, but this kind of functionality seems more suitable to incorporate through a contributed library... have you looked at PixelFlow, it goes pretty far in terms of GPGPU, and I believe it uses FP textures.

[processing-bot]

Created by: timseverien

Thanks for the link! PixelFlow seems to provide a lot of built-in GPGPU shaders, but I was unable to find utilities to use a custom shader. I'll have a more thorough look when I have more time.

but this kind of functionality seems more suitable to incorporate through a contributed library

Because Processing supports shaders, I think Processing should also provide the ability to choose how data is transferred to the GPU. The latter is essential for many projects. I was stoked finding out Processing supports shaders, and it feels counter-intuitive to download a library to work around this limitation. Nonetheless, I enjoy Processing a lot and am grateful of all your efforts.

Anyway, I like @AmnonOwed per-renderer approach. How about textureType(int type)? Additional constants can be introduced for various types. GL_BYTE, GL_FLOAT, etc.

[processing-bot]

Created by: cacheflowe

The more I've been using shaders for advanced feedback & particle techniques, the more I've run into this need. I didn't quite realize why my calculations were suffering until an OpenGL expert suggested that I might not be using a 32-bit texture for storage & computation. That explained my limitation of 1./255. as the smallest number that worked inside a shader. I've browsed the source, but think that I'm woefully underskilled to try to add this feature. I'll look some more though. In the meantime, I give this enhancement a 👍

[processing-bot]

Created by: timseverien

Eventually I did manage to use PixelFlow to achieve what I couldn't with processing using the following code:

import com.jogamp.opengl.GL;
import com.thomasdiewald.pixelflow.java.DwPixelFlow;
import com.thomasdiewald.pixelflow.java.dwgl.DwGLSLProgram;
import com.thomasdiewald.pixelflow.java.dwgl.DwGLTexture;

DwPixelFlow context;
PGraphics2D canvas; // A PGraphics object to render the data to

// Two shaders: one to update data, one to visualise data to screen
DwGLSLProgram shader;
DwGLSLProgram shaderScreen;

// An object containing two frame buffer objects (textures) I can swap
DwGLTexture.TexturePingPong buffer = new DwGLTexture.TexturePingPong();

void setup() {
    context = new DwPixelFlow(this);
    canvas = (PGraphics2D) createGraphics(512, 512, P2D);

    shader = context.createShader("my-shader.glsl");
    shaderScreen = context.createShader("my-shader-screen.glsl");

    buffer.resize(context, GL.GL_RGBA32F, 512, 512, GL.GL_RGBA, GL.GL_FLOAT, GL.GL_NEAREST, GL.GL_REPEAT, 4, 1);
}

// Draw shader output to buffer.dest, where buffer.src acts as input data
void update() {
    context.begin();
    context.beginDraw(buffer.dest);
  
    shader.begin();
    shader.uniformTexture("uTexture", buffer.src);
    shader.drawFullScreenQuad();
    shader.end();
  
    context.endDraw();
    context.end();

    buffer.swap();
}

void draw() {
    update();

    // Render data to PGraphics object with a shader to visualise data
    context.begin();
    context.beginDraw(canvas);
    
    shaderScreen.begin();
    shaderScreen,uniformTexture("uTexture", buffer.src);
    shaderScreen.drawFullScreenQuad();
    shaderScreen.end();
    
    context.endDraw();
    context.end();

    // Render PGraphics object to screen
    image(canvas, 0, 0);
}

Resources:

• http://thomasdiewald.com/processing/libraries/pixelflow/reference/com/thomasdiewald/pixelflow/java/DwPixelFlow.html
• http://thomasdiewald.com/processing/libraries/pixelflow/reference/com/thomasdiewald/pixelflow/java/dwgl/DwGLTexture.html
• http://thomasdiewald.com/processing/libraries/pixelflow/reference/com/thomasdiewald/pixelflow/java/dwgl/DwGLTexture.TexturePingPong.html

[processing-bot]

Created by: codeanticode

@timseverien thanks for posting your solution, it sounds like the floating texture support in PixelFlow can be nicely integrated with the core API. Do you think this makes adding FP texture support in the code superfluous?

Perhaps we should write a tutorial about these advanced techniques... I wanted to write a follow up to the intro PShader tutorial for a long time, but never get the chance to do it :-)

[processing-bot]

Created by: cacheflowe

I did some research and just got 32-bit textures working as a proof-of-concept with very minimal changes. In PJOGL, where textures are created, I overrode the internalFormat and type properties (with GL.GL_RGBA32F & GL.GL_FLOAT) in 3 functions:

@Override
public void texImage2D(int target, int level, int internalFormat, int width, int height, int border, int format, int type, Buffer data) {
  gl.glTexImage2D(target, level, GL.GL_RGBA32F, width, height, border, format, GL.GL_FLOAT, data);
}

@Override
public void copyTexImage2D(int target, int level, int internalFormat, int x, int y, int width, int height, int border) {
  gl.glCopyTexImage2D(target, level, GL.GL_RGBA32F, x, y, width, height, border);
}

@Override
public void compressedTexImage2D(int target, int level, int internalFormat, int width, int height, int border, int imageSize, Buffer data) {
  gl.glCompressedTexImage2D(target, level, GL.GL_RGBA32F, width, height, border, imageSize, data);
}

That's it! This change allows shaders to work as they do in WebGL and other 32-bit GLSL contexts (like Shadertoy), where textures can be used as high-precision vec4 float data! This allows for proper GPU particles and feedback effects, and would probably benefit most shaders' fidelity.

I've obviously just broken the intent of the PJOGL code, but I think there's the potential of adding a 32-bit texture option without much extra work. I'll try to think of a way to make this a reality, but I feel that someone like @codeanticode would have a good opinion on how to make this possible. Perhaps a toggle in settings() to switch everything over to 32-bit if the developer wants to?

Speaking to the previous question about PixelFlow's 32-bit support, you have to switch wholesale into the PixelFlow rendering techniques, and lose the option of using much of Processing's core functionality (this is my experience - I could be wrong). I think adding the 32-bit option into Processing's core would be a huge step forward.

[processing-bot]

Created by: cacheflowe

After further testing, I found some new info:

• This slows down the current Movie object in the processing-video library. There are potential performance impacts elsewhere, but Movie was noticeable.
• It seems to break the processing-video beta v2.0.4 version, which I've been using lately
• It has the potential of slowing things down, so we probably would want something like createGraphics(w, h, P32), to specifically enable this type of texture. This would also ensure backwards compatibility.

If anyone wants to try my recompiled core.jar, you can find it here.

[processing-bot]

Created by: cacheflowe

I'm in way over my head on this, but after seeing another performance hit while using a different camera library, I'm wondering if the realtime conversion between OpenGL texture formats could be causing the slowdown. But perhaps it's something else. I'll keep investigating towards a real implementation of this feature. Maybe it's a lot more involved than I'd hoped 😬 Any advice on OpenGL texture formats and how they interplay in PGL would be very welcome.

https://www.khronos.org/opengl/wiki/Pixel_Transfer
https://stackoverflow.com/a/34497547/352456

[processing-bot]

Created by: cacheflowe

I've revisited this after some conceptual direction from @benfry and arrived at a much simpler distillation that has little-to-no impact on the core. By overriding just a couple of things in PGraphics3D, we get a 32-bit buffer. The full example is seen below, and can be initialized with the normal PApplet.createGraphics() call if we add the new renderer class to PConstants:

PConstants.java addition

static final String P32 = "processing.opengl.PGraphics32";

PGraphics32.java (new class)

package processing.opengl;

import java.nio.Buffer;
import com.jogamp.opengl.GL;
import processing.core.PApplet;
import processing.core.PGraphics;
import processing.opengl.PGL;
import processing.opengl.PGraphics3D;
import processing.opengl.PGraphicsOpenGL;
import processing.opengl.PJOGL;

public class PGraphics32 extends PGraphics3D {
	// override minimal PJOGL code necessary to return 32-bit float texture
	@Override
	protected PGL createPGL(PGraphicsOpenGL pg) {
		return new PJOGL32(pg);
	}

	public class PJOGL32 extends PJOGL {
		public PJOGL32(PGraphicsOpenGL pg) {
			super(pg);
		}

		@Override
		public void texImage2D(int target, int level, int internalFormat, int width, int height, int border, int format, int type, Buffer data) {
			gl.glTexImage2D(target, level, GL.GL_RGBA32F, width, height, border, format, type, data);
		}
	}
}

It was suggested that this could be built as a library, but this is a much more minimal addition than I expected. I'd love to hear opinions about how this might get integrated as an option for creating PGraphics instances - I'm happy to go down either core or library route, depending on informed opinions. This code above would be a really easy pull request.

Some more info on what I'm doing (and I'll try to put together some complete examples), when you're creating a 32-bit texture to use as data, and manipulating each pixel in a shader to move a particle, you want to prep the PGraphics with the following configuration to have it behave properly. Otherwise, you'll get some funky artifacts:

// Create data texture for vertex shader
PGraphics dataPG = createGraphics(w, h, PConstants.P32);
dataPG.noSmooth();
((PGraphicsOpenGL) dataPG).textureSampling(2);
dataPG.beginDraw();
dataPG.hint(PConstants.DISABLE_DEPTH_SORT);
dataPG.hint(PConstants.DISABLE_DEPTH_TEST);
dataPG.hint(PConstants.DISABLE_DEPTH_MASK);
dataPG.background(0, 0);
dataPG.noStroke();
dataPG.endDraw();

Someone who's familiar with even more advanced shader techniques might have further additions or tweaks to OpenGL formats & specific texture/data configuration, but this example has been really helpful for my use-cases.

[processing-bot]

Created by: benfry

Much nicer solution! That looks simple enough that adding a hint() for it might be appropriate. That way it can have the appropriate caveats, since it's really helpful in some circumstances, but can't be made the default due to the performance issues you mentioned.