Add K-means clustering feature

[geekpete]

Add k-means clustering to allow detection of clusters in data sets.
http://en.wikipedia.org/wiki/K-means_clustering

Would be useful for geo points but also other use cases too.

Thanks to https://github.com/koobs for suggesting this one in Sydney Elastic Training.

[savioteles]

It would be great! I really need this feature. Is there any estimate of when you will start coding?

[geekpete]

Not that I'm seeking to drop her in it, but Britta https://twitter.com/a2tirb would definitely have the madskillz to build this feature but not sure of her priorities/bandwidth/interest to attack this feature with sticks.

I'm also not sure how new features are selected or voted up for prioritisation by elasticsearch overlords either.

[brwe]

If I recall correctly, @geekpete proposed to have that in the context of aggregations, that is, build cluster and then use these as buckets inside the aggregations framework. Indeed, this would be an extremely useful feature.

While it would be very much fun to implement unfortunately I do not think we will implement it in the near future. Anyone coming up with a pull request for this is of course more than welcome :-)

For now I can only point you to the carrot2 plugin which does an excellent job in clustering search results.

[koobs]

I'll add the comment that 'k' clusters ought to user-suppliable as an argument to the aggregation for maximum value, with possible k-values being:

• N
• Random
• Basic Rule-Of-Thumb
• Document (m) by Term(n) matrix
• Some other cool method :)

For context, I brought this up @ the ElasticSearch Training in response to a brief conversation about search vs 'insight' in relation to data, the former where you know what you're looking for, the latter where you dont, or might not. The specific example was geospatial result sets with arbitrary demography data fields. It was a great session @brwe!

[mishakogan]

I also would like to cast my vote for some kind of automated clustering feature. Carrot2 is great but as far as understand can only work on small amount of data. Would be great to have something that clusters ALL the data all the time. Maybe custom clustering analyzer?

[clintongormley]

@brwe would #8110 help here?

[brwe]

@clintongormley not really. Bucket reducers from #8110 would run on the final aggregation but clustering needs the documents.

[jpountz]

@brwe I think implementing clustering as a reducer could help reduce the cost very significantly? K-means is costly so running such an algorithm on a dataset containing lots of documents could be very slow. On the other hand, if we take geo-clustering as an example, we could make it very fast (though a bit lossy) by working on top of the output of the geo-hash grid aggregation as a bucket reducer?

[brwe]

True, I should distinguish use cases. For up to 2d it might help indeed. For text clustering I do not see it.

[yehosef]

just found this - would be great. +1

[dsingley]

+1

search for this... this would be a very great feature. Also other Mining-algorithms.