Complete pairwise ranking #396
Conversation
| data = [] | ||
| for i, j in [ | ||
| (i, j) | ||
| for i in range(len(self.items)) |
There was a problem hiding this comment.
on a separate PR, can this be vectorized?
looks like it might be slow
How long does it take for N=200?
There was a problem hiding this comment.
0.4 seconds for 200 items, but this comparator takes no time..
|
@YoelShoshan I switched to choix dense method, with expects a matrix of comparison, and now expect to get the comparisons from outside. Let me know what do you think. |
| class PairwiseRanking: | ||
| """ | ||
| Rank a set of items [a,b,c,...n] based on pairwise comparisons. | ||
| The pairwise comparisons should be given as a matrix, and may be complete (n^2) or parital. |
There was a problem hiding this comment.
looks great!
We need to figure out what people should do if their model return True to both item(i)<item(j) and item(j)<item(i)
also for when their model returned False for both
We can start simple and just ask them to consider model(target, i, j) to mean "i won against j" and they should do:
matrix[i,j]+= 1
and we can start simply with that.
(We can always rectroactively detect the cases mentioned above and modify them with preprocessing in this class)
There was a problem hiding this comment.
Yes I think the most straight forward is setting matrix[i,j] = 1 if i<j, and leave it zero if not.
This way we can also have matrix[i,j] =1 and matrix[j,i] = 1 if the model predicts both, or they can both be zero if it doesn't predict any over them.
The ranking algorithm will product the most probably overall ranking considering these inconsistencies.
add an option to create a ranking based on complete pairwise ranking of a list of items.
For a list of items and a given pairwise comparator:
Currently only one such algorithm is implemented - the Bradely-Terry model, via the
choixpackage implementation.Note this adds the
choixpackage to the requirements, its license is MIT.see: https://en.wikipedia.org/wiki/Bradley%E2%80%93Terry_model
and: https://choix.lum.li/en/latest/index.html
The implementation is simple and follows this notebook: https://github.com/lucasmaystre/choix/blob/master/notebooks/intro-pairwise.ipynb
This approach uses N^2 comparisons, so it fit best to small input.
The test on synthetic data gets very high spearman correlation, even with some noise (N-200, noise.= 0.5, spearman = 0.98)