TF-IDF-implementation-using-map-reduce-Hadoop-python-

Terminologies:

1. TF-IDF is the product of two statistics : Term Frequency (TF) and Inverse Document Frequency (IDF).
2. TF is the number of times a term (word) occurs in a document.
3. IDF is a numerical statistic that is intended to reflect how important a word is to a document.
4. Stop Words are the words which donot contain important significance to the search queries.
5. MapReduce is a programming model and an associated implementation for processing and generating large data sets with a parallel, distributed algorithm on a cluster.
6. Apache Hadoop is an open-source software framework used for distributed storage and processing of big data sets using the MapReduce programming model.

Formulas:

TFIDF = n/N * log(D/m) n is the number of times a word is in a document N is the sum of all n's of a document

References:

1. https://janav.wordpress.com/2013/10/27/tf-idf-and-cosine-similarity/
2. https://en.wikipedia.org/wiki/Tf%E2%80%93idf#Term_frequency
3. https://en.wikipedia.org/wiki/Apache_Hadoop