Project

This project is based on the paper Towards Semantic Fast-Forward and Stabilized Egocentric Videos on the First International Workshop on Egocentric Perception, Interaction and Computing at European Conference on Computer Vision Workshops (EPIC@ECCVW 2016). It implements a semantic fast-forward method for First-Person videos with a proper stabilization method.

For more information and visual results, please access the project page.

Contact

Authors

• Michel Melo da Silva - PhD student - UFMG - michelms@dcc.ufmg.com
• Washington Luis de Souza Ramos - MSc student - UFMG - washington.ramos@outlook.com
• João Pedro Klock Ferreira - Undergraduate Student - UFMG - jpklock@ufmg.br
• Mario Fernando Montenegro Campos - Advisor - UFMG - mario@dcc.ufmg.br
• Erickson Rangel do Nascimento - Advisor - UFMG - erickson@dcc.ufmg.br

Institution

Federal University of Minas Gerais (UFMG)
Computer Science Department
Belo Horizonte - Minas Gerais -Brazil

Laboratory

VeRLab: Laboratory of Computer Vison and Robotics
http://www.verlab.dcc.ufmg.br

Code

This project is a two-fold source code. The first fold¹ is composed of MATLAB code to describe the video semantically and to fast-forward it. A stabilizer proper to fast-forwarded video written in C++ using OpenCV is the second fold². You can run each fold separately.

Dependencies

• ¹MATLAB 2015a or higher
• ²OpenCV 2.4 (Tested with 2.4.9 and 2.4.13)
• ²Armadillo 6 (Tested with 6.600.5 -- Catabolic Amalgamator)
• ²Boost 1 (Tested with 1.54.0 and 1.58.0)
• ²Doxygen 1 (for documentation only - Tested with 1.8.12)

Usage

The project processing is decribed by the following flowchart:

1. Optical Flow Estimator:

   The first step processing is to estimate the Optical Flow of the Input VIdeo.

   1. First, you should download the Poleg et al. 2014 Flow Estimator code from the link.
   2. Navigate to the download folder and unzip the code.
   3. Into the Vid2OpticalFlowCSV\Example folder, run the command:

Vid2OpticalFlowCSV.exe -v < video_filename > -c < config.xml > -o < output_filename.csv >

Options	Description	Type	Example
< Video_filename >	Path and filename of the video.	String	~/Data/MyVideos/myVideo.mp4
< config.xml >	Path to the configuration XML file.	String	../default-config.xml
< output_filename.csv >	Path to save the output CSV file.	String	myVideo.csv

Save the output file using the same name of the input video with extension .csv.

2. Semantic Extractor:

   The second step is to extract the semantic information over all frames of the Input video and save it in a CSV file. On the MATLAB console, go to the project folder and run the command:

>> ExtractAndSave(< Video_filename >, < Semantic_extractor_name >)

Parameters	Description	Type	Example
< Video_filename >	Path and filename of the video.	String	~/Data/MyVideos/Video.mp4
< Semantic_extractor_name >	Semantic extractor algorithm.	String	'face' or 'pedestrian'

3. Calculate Speed-up rates

   To calculate the speed-up rates for each type of segment, on the MATLAB console, go to the project folder and run the following commands:

>> [~, < Num_non_semantic_Frames >, < Num_semantic_frames >] = GetSemanticRanges(< Semantic_Data_MAT_filename >);

Parameters	Description	Type	Example
< Semantic_Data_MAT_filename >	File created by the step 2	String	'../example_face_extracted.mat'

Output	Description	Type	Example
< Num_non_semantic_Frames >	Number of frames in the Non-Semantic segments. (Will be used below.)	Integer	Tns
< Num_semantic_Frames >	Number of frames in the Semantic segments. (Will be used below.)	Integer	Ts

>> SpeedupOptimization( < Num_non_semantic_Frames >, < Num_semantic_frames >, < Desired_speedup >, < Max_speedup >, < lambda_1 >, < lambda_2 >, < show_plot > )

Parameters	Description	Type	Example
< Num_non_semantic_Frames >	Number of frames in the Non-Semantic segments. (Obtained from the previous code)	Integer	Tns
< Num_semantic_Frames >	Number of frames in the Semantic segments. (Obtained from the previous code)	Integer	Ts
< Desired_speedup >	Desired speed-up rate to the whole video.	Integer	10
< Max_speedup >	Maximum allowed jump.	Integer	100
< Lambda_1 >	Value of Lambda 1 in the optimization function.	Integer	40
< Lambda_2 >	Value of Lambda 2 in the optimization function.	Integer	8
< show_plot >	Flag to show the search space create by the optimization function.	Boolean	false

4. Create Experiment

   To run the code, you should create an experiment entry. On a text editor, add a case to the GetVideoDetails function in the file SemanticSequenceLibrary.m:

function [videoFile, startInd, endInd, filename, fps] = GetVideoDetails(video_dir,exp_name)

 ...

 case < Experiment_name >
     filename = < video_filename >;
     startInd = < start_index_frame > ;
     endInd   = < final_index_frame >;
     fps      = < video_frames_per_second >;
					
  ... 
  

Fields	Description	Type	Example
< Experiment_name >	Name to identify the experiment.	String	MyVideo
< video_filename >	Filename to the video.	String	myVideo.mp4
< start_index_frame >	Frame index to start the processing.	Integer	1
< final_index_frame >	Frame intex to stop the processing.	Integer	16987
< video_frames_per_second >	Frames per second of the video.	Integer	30

5. Semantic Fast-Forward

   After the previous steps, you are ready to accelerate the Input Video. On MATLAB console, go to the project folder and run the command:

>> SpeedupVideo(< Video_dir >, < Experiment_name >, < Semantic_extractor_name >, < Semantic_speedup >, < Non_semantic_speedup >, < Shakiness_weights >, < Velocity_weights >, < Appearance_weights >, < Semantic_weights > )

Parameters	Description	Type	Example
< Video_dir >	Path to the folder where the video file is.	String	~/Data/MyVideos
< Experiment_name >	Name set in the SemanticSequenceLibrary.m file.	String	My_Video
< Semantic_extractor_name >	Semantic extractor algorithm used in the Semantic Extractor step.	String	'face' or 'pedestrian'
< Semantic_speedup >	Desired speed-up assign to the semantic segments of the input video.	Integer	3
< Non_semantic_speedup >	Desired speed-up assign to the semantic segments of the input video.	Integer	14
< Shakiness_weights >	Tuple of weights related to the shakiness term in the edge weight formulation.	[Integer, Integer]	[10,250]
< Velocity_weights >	Tuple of weights related to the shakiness term in the edge weight formulation.	[Integer, Integer]	[50,3000]
< Appearance_weights >	Tuple of weights related to the shakiness term in the edge weight formulation.	[Integer, Integer]	[0,100]
< Semantic_weights >	Tuple of weights related to the shakiness term in the edge weight formulation.	[Integer, Integer]	[500,8]

6. Configure Video Parameters

  After the Semantic Fast-Forward step, the accelerated video is create. Now we are going to stabilize the output video. The first stabilization step is to configure the video parameters in the file acceleratedVideoStabilizer/experiment.xml. Follow the instructions described into the file.

7. Accelerate Video Stabilizer

   Navigate to the <project_folder>/acceleratedVideoStabilizer/ folder. Follow the instructions described into the <project_folder>/acceleratedVideoStabilizer/README.md file to compile and run the code.

The output of this step is the stabilized semantic fast-forward video.

Acknowledgements

1. EgoSampling: Y. Poleg, T. Halperin, C. Arora, S. Peleg, Egosampling: Fast-forward and stereo for egocentric videos, in: IEEE Conference on Computer Vision and Pattern Recognition, 2015, pp. 4768–4776. doi:10.1109/CVPR.2015.7299109.
2. LK_Blocked_Optical_Flow: Y. Poleg, C. Arora, S. Peleg, Temporal segmentation of egocentric videos, in: IEEE Conference on Computer Vision and Pattern Recognition, 2014, pp. 2537–2544. doi:10.1109/CVPR.2014.325.
3. NPD_Face_Detector: S. Liao, A. K. Jain, S. Z. Li, A fast and accurate unconstrained face detector, IEEE Transactions on Pattern Analysis and Machine Intelligence 38 (2) (2016) 211–223. doi:10.1109/TPAMI.2015.2448075.
4. PMT_Pedestrian_Detector: P. Dollar, Piotr’s Computer Vision Matlab Toolbox (PMT), https://github.com/pdollar/toolbox.

Citation

If you are using it for academic purposes, please cite:

M. M. Silva, W. L. S. Ramos, J. P. K. Ferreira, M. F. M. Campos, E. R. Nascimento, Towards semantic fast-forward and stabilized egocentric videos, in: European Conference on Computer Vision Workshops, Springer International Publishing, Amsterdam, NL, 2016, pp. 557–571. doi:10.1007/978-3-319-46604-0_40.

Bibtex entry

@InBook{Silva2016,
Title = {Towards Semantic Fast-Forward and Stabilized Egocentric Videos},
Author = {Silva, Michel Melo and Ramos, Washington Luis Souza and Ferreira, Joao Pedro Klock and Campos, Mario Fernando Montenegro and Nascimento, Erickson Rangel},
Editor = {Hua, Gang and J{'e}gou, Herv{'e}},
Pages = {557--571},
Publisher = {Springer International Publishing},
Year = {2016},
Address = {Cham},
Booktitle = {Computer Vision -- ECCV 2016 Workshops: Amsterdam, The Netherlands, October 8-10 and 15-16, 2016, Proceedings, Part I},
Doi = {10.1007/978-3-319-46604-0_40},
ISBN = {978-3-319-46604-0},
Url = { http://dx.doi.org/10.1007/978-3-319-46604-0_40 }
}

Enjoy it.