Draft rankings for unranked cards

[Drecon84]

If the card in this set doesn't have a rating, checks other sets if that same card does have a rating.

Very relevant for drafting with a random selection of all cards in the game, for example.

[Eradev]

Wouldn't that be misleading for cards that are good in a certain draft situation and not in another?

[Drecon84]

True, but it's as an alternative to them having 0 points.

Another possibility would be to go through all sets where the card is rated and average over those, but that doesn't really do all that much if the rating is 70 in one set and 10 in another one.

To be fair, this is a fix for a very niche issue anyway. For most draftable sets cards just get rated. So it's more for sets that we don't have data for or weird formats like drafting from all cards on Forge (which I like to do).

All of that said: if there's an alternative, I'm very open to it. I'm sure I haven't thought of all options.

[tool4ever]

might need a visual hint or something
otherwise we'd just get bug reports of people complaining Forge uses wrong data

[Drecon84]

I'm not sure how many people would actually ever interact with this feature, but that's possible yeah.

I don't know if I'm able to actually do that though. It might require a complete rewrite of multiple systems for something that we're not even sure that people will actually see.

I mean... I'm not against it... but I'm not sure if I'm up to that challenge.

[Drecon84]

So... I've been thinking about this a bit more and I want to make sure that we all really understand in what situations this code might actually come up. I personally think it won't be a problem, but I think it's important to actually think about this together. I have thought of 3 scenarios where this could be relevant. The third is the most relevant, so stay with me here.

1: Someone finds a way to draft an otherwise undraftable set (like a commander product or something) and they see that the cards are not all rated 0. They look in the files and see that there is no ratings for these cards. I doubt someone would ever file a bug report for a situation where they are doing something strange and the game overperforms for them.

2: Someone chooses to draft with all cards in the game (it's actually the top option when you look at the available drafts, so that could definitely happen. It's also my own favourite way to play). This format is filled with cards that will normally never be drafted but that can get a rating this way. Usually about half the cards in a pack have a rating of 0.
Now... I don't even know how someone would figure out that a score isn't obtained the regular way. But if they were, they would just figure out that there aren't any draft rankings for the card it's about. I can't personally imagine someone objecting to the card getting a rating when it normally would just get a 0.

3: I think the most realistic scenario is when there would be a card in regular set (for example: Innistrad), that just hadn't been drafted enough to get a ranking. That happens sometimes. In that case there might be another set (like: Innistrad Remastered) that would have the rating.
So what if a player finds the ranking strange? I have had moments where I saw a ranking that I found difficult to believe, where I actually looked it up in the files. It invariably just is the right ranking.
Now, let's assume that someone sees this card and sees that it's not actually in the file. This is someone who actually know where these files are (which is really, really rare) and who goes through all of the trouble to actually comb through them.
Do we really think that person would figure that something is wrong here? Or would they just be glad that when there is information missing, the game does actually find a way to give its best guess?

I mean... If we're really worried about people diving deep into the files in order to figure out why a number that could have been correct is what it is and that they would go so far to file a bug report for the fact that the game actually gives better options than it should... I guess we could provide for that?

My point is that I think you are making this way too big. The situations where someone would ever notice this are very rare and the idea that people would objeect to it seems very unlikely to me.

That said... maybe I missed something important here. These are just the use cases I can think of and there might be other cases. But I do want to stress that I don't think I've seen many people object to cards randomly having a rating of 0, which is the alternative.

Finally: I'm not trying to be difficult or obstinate. I just want to make sure that we're all working with the same information when thinking about this.

Also sorry for the big wall of text. I've been thinking about this all day, so I had a lot to say ;)

[Eradev]

For me, draft ranking is mostly there to guide users who don't usually draft. The ranking shouldn't matter much to experienced drafters as they know what works or don't, and generally know how to build a deck.

Even if just a number doesn't tell the whole story why the card is good, it at least give them a hint of what's considered playable/sought after in the context. Giving them an arbitrary number that doesn't mean anything in the draft they are playing in is just confusing and misleading them. (ie. a high cost removal could be sought after in a draft environment without much of it, but would be very bad in a very fast format.)

Maybe we should look at not displaying that information at all when the info is not present.

[Drecon84]

That's fair. Although I think you're grossly overestimating the amount of cases where the rating would differ from the expected rating.

I would be really sad if I had to roll this back. I tried to ask around on the Discord about this multiple times earlier and the only reply I got was that it would be fine to have. Now I spent a lot of time finding all of it and now that I finally got it to work I hear that it's a mistake to have spend all of my time on it.

I mean... I will accept it if you all feel like it would not be good... but it's not a great feeling having tried multiple times to try to figure out if this would be something good to have or not and after the fact having everyone tell me that I shouldn't have done it. I could have done a lot of other stuff in that time.

[Drecon84]

To be very clear here. I don't think anyone did anything wrong here. But I am very demotivated currently to keep on spending energy on Forge dev. This was my pet project that I was very motivated to do and that I tried to get feedback on multiple times. To have it get shot down suddenly after the fact is just not great.

I completely understand it. We are all just looking out to make sure the game is the best it can be. Do you want me to retract this commit? I can do that.

[tool4ever]

uhm relax please: nothing is "shot down" at this point :)
it's not like we haven't merged your PRs in the past eventually, but sometimes things take their time (e.g. Hanmac and me both had and have stuff open for 2+ years to refine them, though that's arguably quite rare)
it's perfectly normal in software development to get feedback based on the implementation and your approach happens to be on a very low level in the engine:
Are you aware this means you're also influencing the AI pick + deck building decisions this way? ⚠️

Especially in your full card pool use case this means mixing rankings from like 10+ years of powercreep together and unfortunately they would be weighted way more than other hardcoded AI constraints.
Personally I have doubts this doesn't do more harm than good for the final deck but it's obviously tricky to evaluate.
Or phrased differently, this seems more like an experimental feature than standard behaviour to expect.
That said, I don't think a "complete rewrite" would be required to make this more flexible 🤷‍♂️

Anyway, I'm not drafting that much so maybe @tehdiplomat has some better insights 🤔

[Jetz72]

I think the fact that this applies to the AI's evaluation means there's all the more reason to make a change like this. I've had the AI pass me packs with incredibly powerful cards because it didn't know how to evaluate them and defaulted to 0. It'd be better if it could default to some number that at least loosely approximates the card's value.

One thing to consider is where the estimated ranking is coming from. Looks like this implementation uses the first printing it can find with a known ranking. Hard to say if something more specific would be an improvement. The average of all others? The most recent? The highest or lowest among them? Tricky, but it wouldn't be hard to tweak.

More complicated would be to bundle the info "not a perfectly accurate ranking" in with the ranking itself in a way that can be represented in the UI. Could then wrap the number in parentheses or prefix it with a tilde to indicate it may be off by some amount. That'd involve tweaking the whole pipeline from here to the UI, though.

[Drecon84]

@tool4ever sorry about the emotion. I honestly don't mind having to work on it a lot more and waiting for discussion and tweaks. I'm not in a hurry. I think I might have misunderstood the tone of the conversation. I'll have more faith in the process from this point on.

I was aware of the AI thing and I honestly think this makes the AI better rather than worse. But I do agree that if we can find a way to make the score better than this we definitely should. I'm very open to all suggestions.

And as Jetz said: this would require passing extra variables and keeping a lot of extra information. I also think that adding UI information for this might confusion for people (are we ready for a lot of "what are those brackets" questions?).
It's something we could do, but there's a real cost to that as well. Doesn't mean that it's a bad thing, and I think I could make it work I'm pretty sure. But we should all be on board with it before I start on the work probably :)

[tehdiplomat]

Yea I worry about the changes made here won't really have the effect we're looking for. I think what we probably actually want is something similar to the cubecobra rankings file (which currently has 4000 cards in it), which is the closest thing we have to a global card ranking. By taking rankings into mind from other sets, reprints will be much more greatly drafted by AI just because they will have more opportunities to be rated at all, which will lead the AI to draft those rather than the rarity based drafting it currently does when it doesn't know.

If the goal here is to just not show 0s for unranked cards, I have no issues with that.

[Drecon84]

Well... the original goal was to not show zeroes for cards but I do think this can be bigger and better than that. It should definitely improve things if it's done right.

I did some tests (just to figure out what we're actually trying to do here) and it turns out that for a lot of cards, the available rankings are pretty similar. But probably not similar enough.
An interesting one I came across is Disenchant. It has been reprinted a lot of times and the first ranking set you come across is very far from representative (0.36 where most scores are around 0.9 (this is before the scores get inverted, so lower numbers are better here)).
So it's clear that this dirty method of just finding the first one is not good enough.

I'm thinking that the ideal method might be to try to find the biggest amount of scores that are not too far apart and taking the average of those (so for the Disenchant we would disregard the 0.36 and base the score around the 14 scores that are around 0.9). I'll have a think about the best way to try that implementation, but it seems very doable.

@tehdiplomat: I think the idea of a Master list is a really good one, but it seems difficult to set that up. We can't generate the data ourselves (because the client doesn't send data back to the server as far as I know, so we can't get data from our users) and I'm not sure if we can find a suitable list somewhere. And even if we use a list we find online (like the one from CubeCobra), we would also have to keep it up-to-date I guess?
It feels like the best option if there's not practical problems. But I have no idea how we would set it up. (but... if we could do it, it would almost certainly be our best option).

[Drecon84]

Thinking about it, maybe just using the median value would be ideal. That will work in all normal cases. Only if we have data where there are multiple groupings of data is when it might go really wrong. I haven't really seen evidence of that happening yet.

[Eradev]

@tehdiplomat: I think the idea of a Master list is a really good one, but it seems difficult to set that up. We can't generate the data ourselves (because the client doesn't send data back to the server as far as I know, so we can't get data from our users) and I'm not sure if we can find a suitable list somewhere. And even if we use a list we find online (like the one from CubeCobra), we would also have to keep it up-to-date I guess? It feels like the best option if there's not practical problems. But I have no idea how we would set it up. (but... if we could do it, it would almost certainly be our best option).

We already have the CubeCobra list inside Forge, which is the default one for custom rankings.

[Drecon84]

Ah. I had no idea. There's so much in this project that I have no idea about yet :)

In that case I think the best course of action is to do the following:
1: Get the rating for the current set
2: If that one doesn't exist, get the rating from the master list
3: If that one doesn't exist, find the median from the values in other sets
4: Otherwise default to 0.

How about that?

[Drecon84]

I see that the cubecobra rankings file is in another folder. It's accessible but doing that is a little messy.

We could have a master list for draft rankings in the rankings folder if we wanted, that would make it very easy to access.

Also: if we do the master list, would it be worth it to try and get the full list from CubeCobra instead of just the top 4000? That would make it very useful.

Just some thoughts to try and make the process as streamlined as possible.

In the end, the scale won't ever be perfect, but this could very well be the best option we have available to us.