Rename collections

[ssardina]

@haz , maybe better here than email to keep it all contained and clean.

I suggest the collections are given more informative names. For example, new-fond-papers should become fondsat, as those were clearly designed in that paper.

Now original-fond-papers is more complex, I was reading your PRP paper and some come from FOND-IPC08 track:

And then some come from IPC06 and Little&Thiebeaux paper:

If anything at all I would call them all prp, but may be worth splitting them into exactly where they come from. And that doesn't mean we cannot have prp as well as many subcollections together.

[haz]

Hrmz...it's an interesting thought, but I don't think labeling a collection based on what a paper used (e.g., the "prp set") would offer much. The newly introduced domains -- e.g., "fondsat set" -- certainly does make sense. The distinction you see is actually a mirror of the paladinus paper:

One collection worth considering in detail is the defactor "this is what you should use for comparison" one -- kind of like the "all ipc strips" one for the classical benchmarks. This shouldn't have too many duplicates, cover all the benchmarks, and be pretty non-controversial as a choice for "we tested our FOND planner against other FOND planners on a suite of benchmarks".

[ssardina]

but you see that Paladinus refers to the "benchmarks in Muise 2012"

It seems the ones you used came from IPC-FOND and probabilistic tracks. I would provide clear source IDs to each subset, and why not also the sets used by important papers. Somehow the ones that PRP used are "the original ones", but themselves come from IPC-FOND right?

So we can have

• ipc08-fond
• ipc-prob
• prp
• fondsat

and so on...

[haz]

Aye, I s'pose...since it's not a partition, we can define as many collections as we'd like. Just feels odd to lay claim to a set of benchmarks when we didn't introduce a single one in that paper :P. So would each paper-related collection contain all the domains used in the evals? Or just those "added" in some sense?

[ssardina]

mmm good question... Say, when you say fondsat, I only want the new ones introduced there, not also the PRP ones (that also fondsat used).

unless we start saying: fondsat (all) and fondsat-new (the 4 new)

[haz]

Aye, I'm with you. The reason I delineate FOND-SAT and (down-play) PRP is because the domains for FOND-SAT were uniquely authored -- it was a contribution of the work. I guess you could argue that I dug the IPC2008 domains from whatever archane compression format was used, and went through the (20min) effort to strip the probabilities of the IPPC domains to make them FOND, but it feels a bit icky to stack that contribution up against a suite of newly crafted domains.

[ssardina]

I think that's all worth mentioning:

• domain X comes from the probabilistic track of IPC-N by stripping down the probabilities, etc. If anything it is important they are the "same" domain with a tweak.
• domain X was introduce in paper A as new domain (and if we can give a brief idea, like, "probabilistic interesting" or "risky non-determinism", etc.., better)

I started doing this in PR #3