New dataset and metadata integration

[daphne-cornelisse]

Description

This PR integrates a new version of the dataset with GPUDrive.

Todo's

• @aaravpandya Verify with Madrona viewer
• @aaravpandya Review PR
• @daphnecor Test run on 10 scenarios + videos to verify correctness
• @nadarenator Fix metadata design
• @daphnecor Update tutotrials

[nadarenator]

Looks good to me

[aaravpandya]

This PR has breaking changes with the datasets. Please change the dataset before merging in this PR.

Also, can you change the file in tests/pytest_data with the new format? Tests wont work with the old file format.

[daphne-cornelisse]

This PR has breaking changes with the datasets. Please change the dataset before merging in this PR.

Also, can you change the file in tests/pytest_data with the new format? Tests wont work with the old file format.

Yes, we will do that this week. Since this PR cannot be merged until we fix the data, I have taken your code and applied it to our integrations/vbd branch temporarily.

[daphne-cornelisse]

@nadarenator Let's see if we can update the dataset links in the readme today so we can merge this PR

[nadarenator]

I think we should hold off on regenerating the dataset until we finish vbd alignment, unless you're confident that the current processing script extracts all required info

[eugenevinitsky]

Is regenerating that expensive? We can just regenerate a small number of files for testing

[nadarenator]

Yes that's what I meant, using a small set for our purposes. But before merging this pr we'd need updated links, which would involve regenerating the whole dataset. And if we find out during vbd integration that we need yet another piece of info from the raw tfrecords we'd have to regenerate the full dataset again.

[nadarenator]

Only thing remaining: Regenerate dataset and upload to hf, add link to readme.

[daphne-cornelisse]

Only thing remaining: Regenerate dataset and upload to hf, add link to readme.

Can you send me a link to the new data once it's up? I'd like to do a test PPO run

[nadarenator]

Only thing remaining: Regenerate dataset and upload to hf, add link to readme.

Can you send me a link to the new data once it's up? I'd like to do a test PPO run

It should be available now on my public directory on greene (the same one I shared before).

[nadarenator]

Would be great if we could redo some of the tutorial notebooks, most of the indexing is outdated, and there's also datatypes now that handle the indexing internally

[aaravpandya]

Okay so 3 issues to fix:

1. Revert headless paths
2. Remove metadata.id unless this is absolutely required.
3. Use a hashmap in json serialization to make the code cleaner

[aaravpandya]

I am approving this because I am unable to fix the tests. However, I have done a smell test, visualized the scenarios and reverified the code. I am confident that everything should be working fine. But considering that this feature is blocking work on vbd integration, I think its fine to merge it in.

I will be working on getting the tests to work in a separate PR.

Also, @daphnecor since many people are using GPUDrive, I would recommend bumping the version number of our current release, and putting a "warning/info" card on the readme to notify people that this new version has breaking changes with the previous dataset and that they should download a new version.

[daphne-cornelisse]

I am approving this because I am unable to fix the tests. However, I have done a smell test, visualized the scenarios and reverified the code. I am confident that everything should be working fine. But considering that this feature is blocking work on vbd integration, I think its fine to merge it in.

I will be working on getting the tests to work in a separate PR.

Also, @daphnecor since many people are using GPUDrive, I would recommend bumping the version number of our current release, and putting a "warning/info" card on the readme to notify people that this new version has breaking changes with the previous dataset and that they should download a new version.

Great suggestion. I will add the release and description