App: Remember login & Buffering performance

[mcrook250]

Issue description

Feature request

• Remember me check box for both website and android app
• android app to be able to pick profile
• better buffering, maybe look at mp4box.js and using on the fly adaptive streaming. uses http and mpeg-dash and uses a single mp4 file.

Steps to Reproduce

1. use login page
2. use android app
3. watch video served by streama and the three dots loading not going away after under buffer

Expected Behaviour

not having to enter login info all the time
and smooth streaming without so much buffering, like netflix

Actual Behaviour

always having to enter login info and always buffering

Environment Information

• Operating System: rq intel Q4 4.5 ghz with home server 2011 8 gb of ram 155 mbps connection
• Streama version: 1.6.1
• Custom streama build, describe customizations or provide link to fork (If Applicable): TODO
• Container Version (If Applicable): TODO

[dularion]

The profiles are added now, it was in the pipeline for release the entire time.

The part about the login credentials is annoying me as well, I will have to take a look at it.

Streaming performance on mobile is pretty bad, might have to do with inoic1 .upgrading to ionic4 might fix it. I will take a look at it some time.

(PS: I will rename your issue for easier identification)

[mcrook250]

This is for the website and html5 player also. buffering is easy.... just java to start downloading and feed via buffer stream to the player.... have setting in settings for length of buffer and you are golden. As for the streams to the app, that can be handled by buffering via java script. I was already poking around....

https://stackoverflow.com/questions/35239044/mpeg-dash-video-stream-with-just-single-mp4-file
https://gpac.wp.imt.fr/dashcast/
https://gpac.github.io/mp4box.js/test/index.html
https://github.com/gpac/mp4box.js

I remember seeing how to use java to do buffering, I just can't remember where.
https://stackoverflow.com/questions/39267207/how-can-i-add-buffering-to-my-html-video
https://gist.github.com/chrisallick/3648116
http://www.tuxxin.com/php-mp4-streaming/

must read
https://www.inserthtml.com/2013/03/custom-html5-video-player/

Thanks @dularion !
Awesome code so far..... hows the transcoding going? You need something faster then ffmpeg, like handbrake....?

[mcrook250]

var assetURL = 'frag_bunny.mp4';
// Need to be specific for Blink regarding codecs
// ./mp4info frag_bunny.mp4 | grep Codec
var mimeCodec = 'video/mp4; codecs="avc1.42E01E, mp4a.40.2"';

if ('MediaSource' in window && MediaSource.isTypeSupported(mimeCodec)) {
  var mediaSource = new MediaSource;
  //console.log(mediaSource.readyState); // closed
  video.src = URL.createObjectURL(mediaSource);
  mediaSource.addEventListener('sourceopen', sourceOpen);
} else {
  console.error('Unsupported MIME type or codec: ', mimeCodec);
}

function sourceOpen (_) {
  //console.log(this.readyState); // open
  var mediaSource = this;
  var sourceBuffer = mediaSource.addSourceBuffer(mimeCodec);
  fetchAB(assetURL, function (buf) {
    sourceBuffer.addEventListener('updateend', function (_) {
      mediaSource.endOfStream();
      video.play();
      //console.log(mediaSource.readyState); // ended
    });
    sourceBuffer.appendBuffer(buf);
  });
};

taken from
https://developer.mozilla.org/en-US/docs/Web/API/MediaSource/isTypeSupported

link to source code for player, this is what I think you could add and would solve most issues.
https://github.com/nickdesaulniers/netfix
is this something you could use for the project?

Thanks,
Matt

[dularion]

@mcrook250 we do handle the buffering already via Accept-Ranges header. If thats what you mean. see the code here https://github.com/streamaserver/streama/blob/master/grails-app/services/streama/FileService.groovy#L14

however, this seems broken on android, as it loads all the content at once instead of chunking it.
I will build an interceptor soon-ish where i check what the header-data is from android compared to web, in order to find out whats the matter.

If you are referring to something else, let me know.

[mcrook250]

No, this is to do with when you watch a movie and you are relying on the browser to buffer enough and sometimes you can not.
I have been doing some experimenting, so bare with me if im a little of. I'll look into the groovy server code, but you need to look at the player and the java behind it, look at media extensions for java.

workflow:
webserver -> MP4 -> html5 player -> JavaScript with media extensions -> JavaScript downloads "chunks" and feeds it to the html5 player as MP4

    
const NUM_CHUNKS = 5;

var video = document.querySelector('video');
video.src = video.webkitMediaSourceURL;

video.addEventListener('webkitsourceopen', function(e) {
  var chunkSize = Math.ceil(file.size / NUM_CHUNKS);

  // Slice the video into NUM_CHUNKS and append each to the media element.
  for (var i = 0; i < NUM_CHUNKS; ++i) {
    var startByte = chunkSize * i;

    // file is a video file.
    var chunk = file.slice(startByte, startByte + chunkSize);

    var reader = new FileReader();
    reader.onload = (function(idx) {
      return function(e) {
        video.webkitSourceAppend(new Uint8Array(e.target.result));
        logger.log('appending chunk:' + idx);
        if (idx == NUM_CHUNKS - 1) {
          video.webkitSourceEndOfStream(HTMLMediaElement.EOS_NO_ERROR);
        }
      };
    })(i);

    reader.readAsArrayBuffer(chunk);
  }
}, false);

[mcrook250]

@mcrook250 we do handle the buffering already via Accept-Ranges header. If thats what you mean. see the code here https://github.com/streamaserver/streama/blob/master/grails-app/services/streama/FileService.groovy#L14
however, this seems broken on android, as it loads all the content at once instead of chunking it.
I will build an interceptor soon-ish where i check what the header-data is from android compared to web, in order to find out whats the matter.
If you are referring to something else, let me know.

this might fix the android problem though... add this and some code to figure out the mime type of the file.

also found
https://stackoverflow.com/questions/46310388/streaming-mp4-requests-via-http-with-range-header-in-grails

which explains why videos aren't working on iOS or the applewebkit.

response.addHeader("Etag", file.sha256Hex)
response.addHeader("Content-Type", "Video/MP4")

[dularion]

Cool, Ill give it a try!

[dularion]

About the buffering: I'll have to take a closer look at it.

[mcrook250]

Here is something to get you started on the buffering, im not really sure if its what is needed or maybe you will need a java class like Mp4box.js

<!DOCTYPE html>
<html>
  <head>
    <meta charset="utf-8"/>
  </head>
  <body>
    <video controls></video>
    <script>
      var video = document.querySelector('video');

      var assetURL = 'sample.mp4';
      // Need to be specific for Blink regarding codecs
      // ./mp4info frag_bunny.mp4 | grep Codec
      var mimeCodec = 'video/mp4; codecs="avc1.640016, mp4a.40.2"';
      var totalSegments = 5;
      var segmentLength = 0;
      var segmentDuration = 0;
      var bytesFetched = 0;
      var requestedSegments = [];

      for (var i = 0; i < totalSegments; ++i) requestedSegments[i] = false;

      var mediaSource = null;
      if ('MediaSource' in window && MediaSource.isTypeSupported(mimeCodec)) {
        mediaSource = new MediaSource;
        //console.log(mediaSource.readyState); // closed
        video.src = URL.createObjectURL(mediaSource);
        mediaSource.addEventListener('sourceopen', sourceOpen);
      } else {
        console.error('Unsupported MIME type or codec: ', mimeCodec);
      }

      var sourceBuffer = null;
      function sourceOpen (_) {
        sourceBuffer = mediaSource.addSourceBuffer(mimeCodec);
        getFileLength(assetURL, function (fileLength) {
          console.log((fileLength / 1024 / 1024).toFixed(2), 'MB');
          //totalLength = fileLength;
          segmentLength = Math.round(fileLength / totalSegments);
          //console.log(totalLength, segmentLength);
          fetchRange(assetURL, 0, segmentLength, appendSegment);
          requestedSegments[0] = true;
          video.addEventListener('timeupdate', checkBuffer);
          video.addEventListener('canplay', function () {
            segmentDuration = video.duration / totalSegments;
            video.play();
          });
          video.addEventListener('seeking', seek);
        });
      };

      function getFileLength (url, cb) {
        var xhr = new XMLHttpRequest;
        xhr.open('head', url);
        xhr.onload = function () {
            cb(xhr.getResponseHeader('content-length'));
          };
        xhr.send();
      };

      function fetchRange (url, start, end, cb) {
        var xhr = new XMLHttpRequest;
        xhr.open('get', url);
        xhr.responseType = 'arraybuffer';
        xhr.setRequestHeader('Range', 'bytes=' + start + '-' + end);
        xhr.onload = function () {
          console.log('fetched bytes: ', start, end);
          bytesFetched += end - start + 1;
          cb(xhr.response);
        };
        xhr.send();
      };

      function appendSegment (chunk) {
        sourceBuffer.appendBuffer(chunk);
      };

      function checkBuffer (_) {
        var currentSegment = getCurrentSegment();
        if (currentSegment === totalSegments && haveAllSegments()) {
          console.log('last segment', mediaSource.readyState);
          mediaSource.endOfStream();
          video.removeEventListener('timeupdate', checkBuffer);
        } else if (shouldFetchNextSegment(currentSegment)) {
          requestedSegments[currentSegment] = true;
          console.log('time to fetch next chunk', video.currentTime);
          fetchRange(assetURL, bytesFetched, bytesFetched + segmentLength, appendSegment);
        }
        //console.log(video.currentTime, currentSegment, segmentDuration);
      };

      function seek (e) {
        console.log(e);
        if (mediaSource.readyState === 'open') {
          sourceBuffer.abort();
          console.log(mediaSource.readyState);
        } else {
          console.log('seek but not open?');
          console.log(mediaSource.readyState);
        }
      };

      function getCurrentSegment () {
        return ((video.currentTime / segmentDuration) | 0) + 1;
      };

      function haveAllSegments () {
        return requestedSegments.every(function (val) { return !!val; });
      };

      function shouldFetchNextSegment (currentSegment) {
        return video.currentTime > segmentDuration * currentSegment * 0.8 &&
          !requestedSegments[currentSegment];
      };
    </script>
  </body>
</html>

[mcrook250]

tested with response.addHeader("Content-Type", "Video/MP4") and it works!
Also usernames shouldn't be case sensitive?

I also noticed in the <video tag you don't have onPreload="auto" ?

Sorry, I have a great connection back home, not so great where I am and I am having really bad buffering issues.

[dularion]

@mcrook250 thanks for investigating this. I will add the content-type asap to try it out :) I will use apache tika to check for the real contentType i think, not sure yet.

username: good point, we use springsecurity standard stuff, but maybe we can configure that somehow.

the videotag preloading: didnt know that was a thing, ill check it out :)

[dularion]

your issue has now become 5 issues btw :D

[mcrook250]

haha yeah sorry, hopefully I can help a bit by doing some research for you. A simple check box to remember usernames and password should be as simple as setting a setting via the auth cookie sent to the browser or just setting a simple cookie. You can encrypt the password just for safety.

the content-type and preloading should help a lot, but I came cross this today that might help with buffering ad serving files.
https://www.nurkiewicz.com/2015/06/writing-download-server-part-i-always.html
based off grails as far as I can see, most of the code work has already been done for you via an example.

Matt

PS I think I found the code you are looking for....

import grails.compiler.GrailsTypeChecked
import grails.plugin.springsecurity.annotation.Secured
import asset.pipeline.grails.AssetResourceLocator
import grails.util.BuildSettings
import org.codehaus.groovy.grails.commons.GrailsApplication
import org.springframework.core.io.Resource

class VideoController {
    GrailsApplication grailsApplication
    AssetResourceLocator assetResourceLocator

    public index() {
        Resource mp4Resource = assetResourceLocator.findAssetForURI('/../lol.mp4')

        String range = request.getHeader('range')
        if(range) {
            String[] rangeKeyValue = range.split('=')
            String[] rangeEnds = rangeKeyValue[1].split('-')
            if(rangeEnds.length  > 1) {
                int startByte = Integer.parseInt(rangeEnds[0])
                int endByte = Integer.parseInt(rangeEnds[1])
                int contentLength = (endByte - startByte) + 1
                byte[] inputBytes = new byte[contentLength]
                def inputStream = mp4Resource.inputStream
                inputStream.skip(startByte) // input stream always starts at the first byte, so skip bytes until you get to the start of the requested range
                inputStream.read(inputBytes, 0, contentLength) // read from the first non-skipped byte
                response.reset() // Clears any data that exists in the buffer as well as the status code and headers
                response.status = 206
                response.addHeader("Content-Type", "video/mp4")
                response.addHeader( 'Accept-Ranges', 'bytes')
                response.addHeader('Content-Range', "bytes ${startByte}-${endByte}/${mp4Resource.contentLength()}")
                response.addHeader( 'Content-Length', "${contentLength}")
                response.outputStream << inputBytes
            }
        }
    }
}

taken from http://qaru.site/questions/6858371/streaming-mp4-requests-via-http-with-range-header-in-grails

to figure out the mime type

import com.google.common.net.MediaType;
import java.io.*;
import java.time.Instant;
 
public class FileSystemPointer implements FilePointer {
 
    private final MediaType mediaTypeOrNull;
 
    public FileSystemPointer(File target) {
        final String contentType = java.nio.file.Files.probeContentType(target.toPath());
        this.mediaTypeOrNull = contentType != null ?
                MediaType.parse(contentType) :
                null;
    }

and

private ResponseEntity<Resource> response(FilePointer filePointer, HttpStatus status, Resource body) {
    final ResponseEntity.BodyBuilder responseBuilder = ResponseEntity
            .status(status)
            .eTag(filePointer.getEtag())
            .contentLength(filePointer.getSize())
            .lastModified(filePointer.getLastModified().toEpochMilli());
    filePointer
            .getMediaType()
            .map(this::toMediaType)
            .ifPresent(responseBuilder::contentType);
    return responseBuilder.body(body);
}
 
private MediaType toMediaType(com.google.common.net.MediaType input) {
    return input.charset()
            .transform(c -> new MediaType(input.type(), input.subtype(), c))
            .or(new MediaType(input.type(), input.subtype()));
}
 
@Override
public Optional<MediaType> getMediaType() {
    return Optional.ofNullable(mediaTypeOrNull);
}