How to add custom GLSL effects to Media3 GLeffect to use in composition?

[muhammad-najam-cmp]

I required to add custom .glsl effect in media3 composition or transformer?

I'm able to add Provide transitions but required to ad different shader programs?

[droid-girl]

You can take a look at different effect implementations in the effect module to see how it can be done. For example, HslAdjustment or AlphaScale.
You will need to implement a CustomShaderProgram that extends BaseGlShaderProgram like it is done in AlphaScaleShaderProgram
Feel free to share more details about your use case, so we can advice you specifically based on your case.

[muhammad-najam-cmp]

I required to transition in between two media items using glsl shader programs.i.e. flip, swipe, pop etc
Moreover can you confirm me is BasGlShaderProgram or any other class provide support for multiple texture support in one time,currently it has only one textureId in onDraw method??

[droid-girl]

To be able to transition between media items we are working on adding APIs that will allow developers to do this easily. You will not be able to achieve a transition with a custom effect and BaseGlShaderProgram.
@ychaparov, do you know if there is a way to make a transition work before we finalize our transition APIs?

[muhammad-najam-cmp]

I would appreciate if you provide some way out to play some basic transitions.

    "Zoom" -> MatrixTransformationFactory.createZoomInTransition()
    "Crop" -> MatrixTransformationFactory.createDizzyCropEffect()
    "Spin" -> MatrixTransformationFactory.createSpin3dEffect()

these working for some instance but required more smooth results

[ychaparov]

Using the code here https://github.com/androidx/media/compare/main...ychaparov:media:ychaparov/transitions?expand=1

I generated the video attached

transition.mp4

We don't yet have a high-level API to easily set up transitions. In order to transition between VideoA and VideoB, we need to manually do the following:

• Create two EditedMediaItemSequence objects: first one containing first VideoA and then a gap ; the second one containing first a gap and then VideoB. See the table below
• Put the two sequences in a composition and use compositionBuilder.setVideoCompositorSettings to define what the transition looks like
• The basic implementation from the PR above returns OverlaySettings that scale down VideoA based on the presentationTimeUs that reaches the video compositor

	Only VideoA	transition	only VideoB
Sequence 1	VideoA	VideoA	gap
Sequence 2	gap	VideoB	VideoB

Please try out the configuration in the basic PR and let us know if it works for you!

[muhammad-najam-cmp]

Hi Its fine to use 2D-transitions (i.e. full_fading, zoom, rotate) with above provided solution, but how could to implement different glsl shader programs keeping more then two inputs? Explain a bit how sequence.Gap is handled for more then two inputs with different presentation time of different transitions.

[ychaparov]

We do not yet support glsl programs that take input from multiple video streams.

What transition effect are you trying to achieve? Can you use alpha-blending for that (OverlaySettings.getAlphaScale)?

You could use OverlaySettings to put the textures from two input videos (e.g. size 1280x720 each) into one larger texture side by side (size 1280x1440). And then you can use Composition-level effects to apply a GlEffect / GlShaderProgram to the multi-video texture.

EditedMediaItemSequence.Builder().addGap(durationUs) adds 1x1 black images to the video stream, for the specified duration. This allows you to align multiple videos in time.

Currently, the black images must be discarded via compositionBuilder.setVideoCompositorSettings

When you're dealing with more than two (e.g. three) inputs, you can again split them into sequences and ensure they align in time as needed

	only VideoA	transition A->B	only VideoB	transition B->C	only VideoC
Sequence 1	VideoA	VideoA	gap	VideoC	VideoC
Sequence 2	gap	VideoB	VideoB	VideoB	gap

[muhammad-najam-cmp]

Strange! I found that composition player is not allowing to add Gaps between the EditedMediaItem Sequences,
then how to get preview available without exporting from transformer.
Though GLEffect are supporting different filterations effect, but while transiting video with composition player hit in vain.

Is there any way out? when will the library supports the transitions properly with composition player and transformer export???

  public void setComposition(Composition composition) {
    verifyApplicationThread();
    checkArgument(!composition.sequences.isEmpty());
    checkArgument(!composition.hasGaps());
    checkState(this.composition == null);
}

[droid-girl]

CompositionPlayer does not support Gaps yet.
Please keep in mind that CompositionPlayer is still under active development and we are gradually adding these APIs to CompositionPlayer.
The sample @ychaparov shared should work for export, the preview functionality will be available shortly in the upcoming releases.