DASH: Parse MIME type, width and height into Format for image AdaptationSet

[mekptang]

Hello guys, I want to extract the width, height, and value of the image from the following manifest.

<AdaptationSet id="3" mimeType="image/jpeg" contentType="image">
    <SegmentTemplate media="$RepresentationID$/tile_$Number$.jpg" duration="100" startNumber="1"/>
      <Representation bandwidth="12288" id="thumbnails_320x180" width="3200" height="180">
      <EssentialProperty schemeIdUri="http://dashif.org/thumbnail_tile" value="10x1"/>
    </Representation>
</AdaptationSet>

I have referred to #8577 as an example but it only works for mimeType="video/mp4".

code for image/jpeg:

if("image/jpeg".equals(mimeType)){
    Representation.MultiSegmentRepresentation multiSegmentRepresentation = (Representation.MultiSegmentRepresentation) representation;
    Log.d("format.width" , String.valueOf(multiSegmentRepresentation.format.width));
    for(int l = 1; l < multiSegmentRepresentation.getSegmentCount(periodDurationUs); l++) {
    Log.d("sprite_uri" , String.valueOf(multiSegmentRepresentation.getSegmentUrl(l).resolveUri(representation.baseUrls.get(0).url)));
    }
    return;
}

result:

2021-09-28 17:34:23.146 25820-25820/com.example.exoplayerdemo D/format.width: -1

code for video/mp4:

if("video/mp4".equals(mimeType)){
    Representation.MultiSegmentRepresentation multiSegmentRepresentation = (Representation.MultiSegmentRepresentation) representation;
    Log.d("format.width" , String.valueOf(multiSegmentRepresentation.format.width));
    Log.d("format.height" , String.valueOf(multiSegmentRepresentation.format.height));
    return;
}

result:

2021-09-28 17:52:07.574 23252-23252/com.example.exoplayerdemo D/format.width: 1024
2021-09-28 17:52:07.574 23252-23252/com.example.exoplayerdemo D/format.height: 576

Any suggestions to parse this thing in exoplayer?
here is the manifest URL: https://dash.akamaized.net/akamai/bbb_30fps/bbb_with_tiled_thumbnails.mpd

[krocard]

@ojw28 could our dash parser be reuse outside of ExoPlayer as asked?

[mekptang]

Thank you for your comments.
I got the same result by using the DashManifestParser. I am not sure if I use it right.

Here is my code using dash parser:

        new Thread(new Runnable(){
            @Override
            public void run(){

                DashManifest manifest = null;

                try {
                    String dashManifestUri = "https://dash.akamaized.net/akamai/bbb_30fps/bbb_with_tiled_thumbnails.mpd";
                    DashManifestParser dashParser = new DashManifestParser();
                    InputStream input = new URL(dashManifestUri).openStream();
                    Uri uri = Uri.parse(dashManifestUri);
                    manifest = dashParser.parse(uri, input);
                } catch (IOException e) {
                    e.printStackTrace();
                }

                for (int i = 0; i < manifest.getPeriodCount(); i++) {
                    Period period = manifest.getPeriod(i);
                    long periodDurationUs = manifest.getPeriodDurationUs(i);
                    for (int j = 0; j < period.adaptationSets.size(); j++) {
                        AdaptationSet adaptationSet = period.adaptationSets.get(j);
                        for (int k = 0; k < adaptationSet.representations.size(); k++) {
                            Representation representation = adaptationSet.representations.get(k);
                            String format = adaptationSet.representations.get(k).format.toString();
                            Log.d("format", format);
                        }
                    }
                }
                
            }
        }).start();

result:

2021-09-30 11:02:24.840 28646-28675/com.example.exoplayerdemo D/format: Format(bbb_30fps_1024x576_2500k, null, video/mp4, video/avc, avc1.64001f, 3134488, null, [1024, 576, 30.0], [-1, -1])
2021-09-30 11:02:24.842 28646-28675/com.example.exoplayerdemo D/format: Format(bbb_30fps_1280x720_4000k, null, video/mp4, video/avc, avc1.64001f, 4952892, null, [1280, 720, 30.0], [-1, -1])
2021-09-30 11:02:24.842 28646-28675/com.example.exoplayerdemo D/format: Format(bbb_30fps_1920x1080_8000k, null, video/mp4, video/avc, avc1.640028, 9914554, null, [1920, 1080, 30.0], [-1, -1])
2021-09-30 11:02:24.842 28646-28675/com.example.exoplayerdemo D/format: Format(bbb_30fps_320x180_200k, null, video/mp4, video/avc, avc1.64000d, 254320, null, [320, 180, 30.0], [-1, -1])
2021-09-30 11:02:24.842 28646-28675/com.example.exoplayerdemo D/format: Format(bbb_30fps_320x180_400k, null, video/mp4, video/avc, avc1.64000d, 507246, null, [320, 180, 30.0], [-1, -1])
2021-09-30 11:02:24.843 28646-28675/com.example.exoplayerdemo D/format: Format(bbb_30fps_480x270_600k, null, video/mp4, video/avc, avc1.640015, 759798, null, [480, 270, 30.0], [-1, -1])
2021-09-30 11:02:24.843 28646-28675/com.example.exoplayerdemo D/format: Format(bbb_30fps_640x360_1000k, null, video/mp4, video/avc, avc1.64001e, 1254758, null, [640, 360, 30.0], [-1, -1])
2021-09-30 11:02:24.843 28646-28675/com.example.exoplayerdemo D/format: Format(bbb_30fps_640x360_800k, null, video/mp4, video/avc, avc1.64001e, 1013310, null, [640, 360, 30.0], [-1, -1])
2021-09-30 11:02:24.843 28646-28675/com.example.exoplayerdemo D/format: Format(bbb_30fps_768x432_1500k, null, video/mp4, video/avc, avc1.64001e, 1883700, null, [768, 432, 30.0], [-1, -1])
2021-09-30 11:02:24.843 28646-28675/com.example.exoplayerdemo D/format: Format(bbb_30fps_3840x2160_12000k, null, video/mp4, video/avc, avc1.640033, 14931538, null, [3840, 2160, 30.0], [-1, -1])
2021-09-30 11:02:24.848 28646-28675/com.example.exoplayerdemo D/format: Format(bbb_a64k, null, audio/mp4, audio/mp4a-latm, mp4a.40.5, 67071, null, [-1, -1, -1.0], [2, 48000])
2021-09-30 11:02:24.848 28646-28675/com.example.exoplayerdemo D/format: Format(thumbnails_320x180, null, image/jpeg, null, null, 12288, null, [-1, -1, -1.0], [-1, -1])

Format.java:

 @Override
  public String toString() {
    return "Format("
        + id
        + ", "
        + label
        + ", "
        + containerMimeType
        + ", "
        + sampleMimeType
        + ", "
        + codecs
        + ", "
        + bitrate
        + ", "
        + language
        + ", ["
        + width
        + ", "
        + height
        + ", "
        + frameRate
        + "]"
        + ", ["
        + channelCount
        + ", "
        + sampleRate
        + "])";
  }

[giladna]

@mekptang

you can check our approach how to achieve this with Custom classes for Parsing the manifest

https://github.com/kaltura/playkit-android/pull/686/files

[ojw28]

Am I correct in thinking that you're already managing to extract the sprite URIs? The width, format and MIME type are not being set in the Format, but this is something that we can easily add to DashManifestParser.

We can make the necessary change on our side. Once we've made the change, you can either wait for the next release to get the functionality, or temporarily fork the dev-v2 version of DashManifestParser into your own code.

[mekptang]

Am I correct in thinking that you're already managing to extract the sprite URIs? The width, format and MIME type are not being set in the Format, but this is something that we can easily add to DashManifestParser.

We can make the necessary change on our side. Once we've made the change, you can either wait for the next release to get the functionality, or temporarily fork the dev-v2 version of DashManifestParser into your own code.

Thank you! URIs can be extracted by referring #8577. I don't think this is the best practice but enough for my use case.

[mekptang]

@mekptang

you can check our approach how to achieve this with Custom classes for Parsing the manifest

https://github.com/kaltura/playkit-android/pull/686/files

Thank you. This is useful as a reference to my application.

[ojw28]

The commit above will parse the MIME type, width and height for image adaptation sets. As noted above, you can either wait for the next release to get the functionality, or temporarily fork the dev-v2 version of DashManifestParser into your own code.