Clustering metrics

[SkafteNicki]

🚀 Feature

Lets add clustering metrics to TM:

• Dunn index -> New Metric: Dunn Index #2049
• Silhouette @stancld
• Rand score -> New metric: Rand Score  #2025
• Adjusted_mutual_info_score -> New metric: Adjusted mutual info score #2058
• Adjusted rand score -> New metric: Adjusted Rand Score #2032
• Calinski Harabasz score -> New metric: Calinski Harabasz Score #2036
• Completeness score -> New metrics: Homogeneity, Completness, V-Measure #2053
• Fowlkes mallows score -> New metric: Fowlkes-Mallows Index #2066
• Homogeneity completeness measure -> New metrics: Homogeneity, Completness, V-Measure #2053
• Mutual info score -> Mutual Information Score #2008
• Normalized mutual info score -> New metric: Normalized Mutual Information Score #2029
• Davies–Bouldin index -> New metric: Davies bouldin score #2071
• V measure score -> New metrics: Homogeneity, Completness, V-Measure #2053

Motivation

In Supervised Learning, the labels are known and evaluation can be done by calculating the degree of correctness by comparing the predicted values against the labels. However, in Unsupervised Learning, the labels are not known, which makes it hard to evaluate the degree of correctness as there is no ground truth.

That being said, it is still consistent that a good clustering algorithm has clusters that have small within-cluster variance (data points in a cluster are similar to each other) and large between-cluster variance (clusters are dissimilar to other clusters).

ref: https://towardsdatascience.com/7-evaluation-metrics-for-clustering-algorithms-bdc537ff54d2

CTA

pls also check our contribution guide: https://torchmetrics.readthedocs.io/en/stable/generated/CONTRIBUTING.html

[github-actions]

Hi! thanks for your contribution!, great first issue!

[matsumotosan]

Hi @SkafteNicki, I'd like to give Mutual Info Score a shot!

Should the implementation be done in a new directory like torchmetrics/clustering and torchmetrics/functional/clustering?

[SkafteNicki]

@matsumotosan cool thanks for wanting to contribute :]
Yes, please add new directories as I consider this a new domain of metrics.

[SkafteNicki]

Also if it help, I was fooling around with some code in the weekend, trying to calculate the contingency matrix and pair matrix which seems to be two common objects used for calculating clustering metrics:

p = torch.tensor([0, 0, 1, 2])
t = torch.tensor([0, 0, 1, 1])

p_u = len(torch.unique(p))
t_u = len(torch.unique(t))

lin = p + p_u * t
contingency_matrix = torch.bincount(t + t_u * p, minlength=p_u * t_u).reshape(p_u, t_u)

n_c = contingency_matrix.sum(dim=1)
n_k = contingency_matrix.sum(dim=0)
sum_squared = (contingency_matrix ** 2).sum()

pair_matrix = torch.zeros(2,2)
pair_matrix[1,1] = sum_squared - len(p)
pair_matrix[0,1] = (contingency_matrix * n_k).sum() - sum_squared
pair_matrix[1,0] = (contingency_matrix.T * n_c).sum() - sum_squared
pair_matrix[0,0] = len(p) ** 2 - pair_matrix[0,1] - pair_matrix[1,0] - sum_squared

probably needs to be improved but maybe useful.

[matsumotosan]

I tried calculating the contingency matrix based on sklearn's implementation. I'll compare it to the method you suggested since it does seem to be a key computation.

[matsumotosan]

I'd like to work on normalized mutual info and Dunn index next!

[SkafteNicki]

I'd like to work on normalized mutual info and Dunn index next!

Assigned :]