Text document clustering using Spectral Clustering algorithm with Particle Swarm Optimization (inclusion of PCA)

Team Members

• RISHI SHARMA - 171CO135
• VEDANT MEHRA - 171CO250

Libraries / Set up environment

• Jupyter Notebook
• Anaconda
• NLTK
• Numpy
• Pandas
• Matplotlib
• Gensim

What things you need to install the software and how to install them

Following are the dependencies of python which you have to install in your system

 - python 3.6.8 
 - python pip 

After installing python and pip you have to just write

  $ pip install 'below library names'

To install below dependencies

 - numpy
 - nltk
 - matplotlib
 - pandas
 - gensim
 - juyterlab 

How to run the code

• Clone this repository and open the notebook in jupyter notebook.
  
• Now one can run each and every cell of the notebook. Furthur details of what each section of the code contains is given below in furthur steps.
  
• The first section of the code contains preliminary work which is needed. We imported the libraries that are required, then downloading stopwords and the function for creating the tf-idf vector.
  
• The next section will use the imported Reuters dataset and divide it into training and testing data, form the tf-idf vector from the training data.
  
• Now the section of Visualization has importing gensim library, tokenising the single document text, converting the tokenised vector to pandas dataframe and then visualising the word embeddings.
  
• Then we move to the Particle Swarm Optimization section where we have a function for PSO algorithm.
  
• The next section is for Spectral Clustering which will import necessary libraries, fit the data and calculate the Adjusted Random Index (ARI).
  
• The next section is our own ideas which involves the idea of using Principle Component Analysis(PCA) on Affinity matrix with Euclidean Distance. Here we applied
• PCA on Affinity matrix with Euclidean Distance and then calculated the ARI for the model.
  
• The next section has our other idea which is to use Principle Component Analysis(PCA) on Affinity matrix with Gaussian Kernel. Here we applied PCA on Affinity matrix with Gaussian Kernel and then calculated the ARI for the model.
  
• The last section is the Comparison of Adjusted Rand Index for various models.