Similarity

Overview

A Ruby library for calculating the similarity between pieces of text using a Term Frequency-Inverse Document Frequency method.

A bag of words model is used. Terms in the source documents are downcased and punctuation is removed, but stemming is not currently implemented.

This library was written to facilitate the creation of diagrams talked about by Jonathan Stray in his full-text visualization of the Iraq War Logs post. An example of how to generate a Gephi compatible file including labelling of nodes with key words is included in the examples directory.

The library depends on the GNU Scientific Library, and the gsl ruby gem but does not use sparse matrix representations to speed up the calculations, since there is no support for them in the GSL. I am currently looking into fixing this, and would appreciate any help!

Dependencies

Similarity depends on the GNU Scientific Library, and the gsl ruby gem. On OSX with https://github.com/mxcl/homebrew the GSL can be installed with

brew install gsl

The gsl gem should then install normally. For other platforms, please add the information to the wiki and I’ll add them to this readme.

Usage

First we load some documents into the corpus

require 'similarity'

:

corpus = Corpus.new

:

doc1 = Document.new(:content => "A document with a lot of additional words some of which are about chunky bacon")
doc2 = Document.new(:content => "Another longer document with many words and again about chunky bacon")
doc3 = Document.new(:content => "Some text that has nothing to do with pork products")

:

[doc1, doc2, doc3].each { |doc| corpus << doc }

Then to compare documents we can use the similar_documents method

corpus.similar_documents(doc1).each do |doc, similarity|
 puts "Similarity between doc #{doc1.id} and doc #{doc.id} is #{similarity}"
end

:

#=>
 Similarity between doc 70137042580340 and doc 70137042580340 is 0.9999999999999997
 Similarity between doc 70137042580340 and doc 70137042580240 is 0.06068602112714361
 Similarity between doc 70137042580340 and doc 70137042580160 is 0.04882114791611661

The cross-similarity matrix (useful for creating graphs) is also available

similarity_matrix = corpus.similarity_matrix

For more examples, see the examples directory.

Todo

• Performance improvements
  • Switch to storing document vector spaces in sparse form, using linalg or csparse?
• (Optional) stemming of source terms

Contributing

• Fork the project
• Send a pull request
• Don’t touch the .gemspec, I’ll do that when I release a new version

Author

Chris Lowis - BBC R&D