Unknown error in WOSAC submission?

[pengzhenghao]

Hi I just reused my WOSAC submission generation pipeline half year ago and made a submission. But it seems an unknown error happens:

(original post has a wrong link. Here is the real one:)

https://waymo.com/open/challenges/sim-agents/results/92d57ec4-798b/1727496916523000/

Is there anyway I can know what's the error? Thanks!

[pengzhenghao]

I believe this might be the same bug we previously encountered: #807 (comment)

Close this for now.

[pengzhenghao]

Sorry after reusing a previously working pipeline I still get unknown error. Can someone help me take a look? Thanks!

https://waymo.com/open/challenges/sim-agents/results/92d57ec4-798b/1727718185742000/

[pengzhenghao]

Hello?

[jlambert-waymo]

Hi @pengzhenghao, it looks like you had a previously successful submission on April 10/11: 1712813129292000.tar.gz.

Please see the April 11, 2024 update: https://github.com/waymo-research/waymo-open-dataset?tab=readme-ov-file#april-2024-update

This commit contains the changes:
22e51ef

New required metadata fields are added to this proto (describing your submission):
src/waymo_open_dataset/protos/sim_agents_submission.proto

Have you added those?

[pengzhenghao]

I have added the required metadata fields and was able to make a submission with those new fields in.

I just made another attempt and there is no error printed out:

https://waymo.com/open/challenges/sim-agents/results/92d57ec4-798b/1729809749033000/

Could you help me take a look? Thank you!

[zhejz]

I'm facing exactly the same problem. Here is my submission to validation split:
https://waymo.com/open/challenges/sim-agents/results/5ea7a3eb-7337/1730040554700000/

However, the leaderboard works fine when I use the same code but pack the ground-truth future into the submission file. So it's related to the trajectories, not the metadata.

[zhejz]

Seems to be the same as #864 ? @jlambert-waymo @nicomon24

[zhejz]

I uploaded exactly the same submission file again, now this time it works. Somehow the evaluation server fails randomly from time to time.

[pengzhenghao]

@jlambert-waymo Could you please take a look? It's even better if you can update the server side error messsage. Thanks!