Human Activity Recognition

A Comparative Study between Different Pre-processing Approaches and Classifiers

📖 Table of Contents

Table of Contents

1. ➤ About The Project
2. ➤ Prerequisites
3. ➤ Folder Structure
4. ➤ Dataset
5. ➤ Roadmap
6. ➤ Preprocessing
   • Pre-processed data
   • Statistical feature
   • Topological feature
7. ➤ Results and Discussion
8. ➤ References
9. ➤ Contributors

📝 About The Project

This project aims to classify human activities using data obtained from accelerometer and gyroscope sensors from phone and watch. The raw data will be preprocessed using two different approaches such as topological data analysis and statistical features extraction from segmented time series. The goal is to compare and evaluate the performance of different classifiers (Decision Tree, k Nearest Neighbors, Random Forest, SVM and CNN) which are trained on the two sets of preprocessed data.

🍴 Prerequisites

The following open source packages are used in this project:

• Numpy
• Pandas
• Matplotlib
• Scikit-Learn
• Scikit-tda
• Giotto-tda
• TensorFlow
• Keras

🌵 Folder Structure

code
.
│
├── data
│   ├── raw_data
│   │   ├── phone
│   │   │   ├── accel
│   │   │   └── gyro
│   │   ├── watch
│   │       ├── accel
│   │       └── gyro
│   │
│   ├── transformed_data
│   │   ├── phone
│   │   │   ├── accel
│   │   │   └── gyro
│   │   ├── watch
│   │       ├── accel
│   │       └── gyro
│   │
│   ├── feature_label_tables
│   │    ├── feature_phone_accel
│   │    ├── feature_phone_gyro
│   │    ├── feature_watch_accel
│   │    ├── feature_watch_gyro
│   │
│   ├── wisdm-dataset
│        ├── raw
│        │   ├── phone
│        │   ├── accel
│        │   └── gyro
│        ├── watch
│            ├── accel
│            └── gyro
│
├── CNN_Impersonal_TransformedData.ipynb
├── CNN_Personal_TransformedData.ipynb  
├── CNN_Impersonal_RawData.ipynb    
├── CNN_Personal_RawData.ipynb 
├── Classifier_SVM_Personal.ipynb
├── Classifier_SVM_Impersonal.ipynb
├── statistical_analysis_time_domain.py
├── Topological data analysis.ipynb  

💾 Dataset

The WISDM (Wireless Sensor Data Mining) dataset includes raw time-series data collected from accelerometer and gyroscope sensors of a smartphone and smartwatch with their corresponding labels for each activity. The sensor data was collected at a rate of 20 Hz (i.e., every 50ms). Weiss et.al., collected this dataset from 51 subjects who performed 18 different activities listed in Table 2, each for 3 minutes, while having the smartphone in their right pant pocket and wearing the smartwatch in their dominant hand. Each line of the time-series sensor file is considered as input.

The WISDM dataset is publicly available. Please refer to the Link

The following table shows the 18 activities represented in data set.

🎯 Roadmap

Weiss et. al. has trained three models namely Decision Tree, k-Nearest Neighbors, and Random Forest for human activity classification by preprocessing the raw time series data using statistical feature extraction from segmented time series. The goals of this project include the following:

1. Train the same models - Decision Tree, k Nearest Neighbors, and Random Forest using the preprocessed data obtained from topological data analysis and compare the performance against the results obtained by Weiss et. al.

2. Train SVM and CNN using the preprocessed data generated by Weiss et. al. and evaluate the performance against their Decision Tree, k Nearest Neighbors, and Random Forest models.

🔨 Preprocessing

The WISDM (Wireless Sensor Data Mining) dataset includes raw time-series data collected from accelerometer and gyroscope sensors of a smartphone and smartwatch with their corresponding labels for each activity. The sensor data was collected at a rate of 20 Hz (i.e., every 50ms). Weiss et.al., collected this dataset from 51 subjects who performed 18 different activities listed in the previous table, each for 3 minutes, while having the smartphone in their right pant pocket and wearing the smartwatch in their dominant hand.
In this project we tried three different feature sets, extracted from the raw data, which are as follows:

1. Pre-processed data generated by Weiss et. al.
2. Statistical feature extraction
3. Topological feature extraction

All these three approaches used windowing technique to segment the raw time series and extract features from each segment.

💠 Pre-processed data

Weiss et.al used windowing technique with window size of 10 seconds to extract statistical features. They extracted 93 features out of which 43 were used to train their models. We also used the same 43 features to train our SVM and CNN. The 43 features are 1. average sensor value 2. standard deviation 3. absolute difference 4. average resultant acceleration 5. Binned distribution (10 equal sized bins per axis) and 5. time between peaks, for each axis.

🔶 Statistical feature

For this approach, we segmented the dataset using 10 second window size (200 datapoints) with no overlapping. We decided to keep the window size same as whatWeiss et.al. applied in their study, for the sake of comparison. After segmentation, for each segment we calculated eight statistical features, namely, ‘min’, ‘max’, ‘mean’, ‘standard deviation’, ‘median’, ‘variance’, ‘zero crossing’ and ‘mean crossing’, for each axes. The zero and mean crossing features are calculated by counting the rate of when a signal passes line y=0 (if we let y-axis to be the specific measurement and x-axis to represent time) and the frequency at which the signal passes the line y = mean(signal), respectively. However, these two features did not show a significant difference between different activities, so we decided to ignore them.

🔷 Topological feature

Topological data analysis provides various techniques toexplore the topological properties and shape of the data. Since time series sensor data obtained from performing an activity may have topological properties, we tried extracting features using the topology of the data and perform the classification task on those features. For a given time segment we explore the topology of each segment using persistence diagram generated via persistence homology. Persistent homology can be created through filtrations such as Vietoris- Rips, SparseRips, Cubical Persistence etc., on the data and capture the growth, birth, and death of different topological features across dimensions (e.g., components, tunnels, voids) [2]. One of the main challenges in computing the persistent homology is finding the appropriate filtration for the time segments. In total 18 topological features where extracted for each time segment.

🔍 Results and Discussion

The overall accuracy score of personal and impersonal models are shown in the following tables. Some of the results we observed are similar to the results obtained by Weiss et.al and they are discussed below:

• Since accelerometers senses acceleration based on vibration which can be more prominent during an activity and gyroscope only senses rotational changes, accelerometers outperformed gyroscope in all our models.
  
• As the style of performing an activity differs from each person, it is difficult to aggregate those features among all subjects. So our personal models vastly outperformed our impersonal models.
• It is also observed that non hand-oriented activities are classified better with sensors from smartphone and handoriented activities are classified better with sensors from smartwatch. Refer appendix for activity wise recall scores. Some key take-aways based on our results are listed below:
• CNN trained on raw sensor data performed better for personal models, however it performed poorly on impersonal models.

📚 References

• Matthew B. Kennel, Reggie Brown, and Henry D. I. Abarbanel. Determining embedding dimension for phase-space reconstruction using a geometrical construction. Phys. Rev. A, 45:3403–3411, Mar 1992.

• L. M. Seversky, S. Davis, and M. Berger. On time-series topological data analysis: New data and opportunities. In 2016 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pages 1014–1022, 2016.

• Floris Takens. Detecting strange attractors in turbulence. In David Rand and Lai-Sang Young, editors, Dynamical Systems and Turbulence, Warwick 1980, pages 366–381, Berlin, Heidelberg, 1981. Springer Berlin Heidelberg.

• Guillaume Tauzin, Umberto Lupo, Lewis Tunstall, Julian Burella P´erez, Matteo Caorsi, Anibal Medina-Mardones, Alberto Dassatti, and Kathryn Hess. giotto-tda: A topological data analysis toolkit for machine learning and data exploration, 2020.

• G. M. Weiss and A. E. O’Neill. Smartphone and smartwatchbased activity recognition. Jul 2019.

• G. M. Weiss, K. Yoneda, and T. Hayajneh. Smartphone and smartwatch-based biometrics using activities of daily living. IEEE Access, 7:133190–133202, 2019.

• Jian-Bo Yang, Nguyen Nhut, Phyo San, Xiaoli li, and Priyadarsini Shonali. Deep convolutional neural networks on multichannel time series for human activity recognition. IJCAI, 07 2015.

📜 Contributors

🎓 All participants in this project are graduate students in the Department of Computer Science and Software Engineering @ Concordia University

👩 Divya Bhagavathiappan Shiva
      Email: divya.bhagavathiappanshiva@mail.concordia.ca
      GitHub: @divyabhagavathiappan

👩 Reethu Navale
      Email: reethu.navale@mail.concordia.ca
      GitHub: @reethunavale

👩 Mahsa Sadat Afzali Arani
      Email: m_afzali93@yahoo.com
      GitHub: @MahsaAfzali

👦 Mohammad Amin Shamshiri
      Email: mohammadamin.shamshiri@mail.concordia.ca
      GitHub: @ma-shamshiri

✤ This was the final project for the course COMP 6321 - Machine Learning (Fall 2020), at Concordia University