Replace nlohmann json with simdjson #406
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
daphne-cornelisse
changed the title
daphne-cornelisse
requested changes
Apr 20, 2025
Contributor
daphne-cornelisse
left a comment
daphne-cornelisse
left a comment
There was a problem hiding this comment.
There is a conflict in json_serialization.hpp, could you please take a look, Aarav? Thank you
| @@ -1,36 +1,36 @@ | |||
| # Assuming you're in a sub-directory "tests" in your project directory | |||
| # # Assuming you're in a sub-directory "tests" in your project directory | |||
Contributor
There was a problem hiding this comment.
Why is all this commented out?
Collaborator
Author
There was a problem hiding this comment.
Yeah this is TODO. The tests were not working as I recall. I was just wondering how much effort should I put to convert the json code in there too.
Collaborator
Author
There was a problem hiding this comment.
We are not using the tests currently anyways I think ? I would say lets merge it and then circle back later.
* Refactor: Unify guidance mechanisms (#421) * Update settings * Add warning to catch unnormalized features * Clean done logic. * Restructure expert suggestions under _guidance_ * Update settings * Add warning to catch unnormalized features * Clean done logic. * Restructure expert suggestions under _guidance_ * Update WOSAC eval readme * Provide agents with full speed guidance trajs * Add support for headings * Add vel_xy * Improve code efficiency * Small shape bug fix * Improve wosac eval setting * Update best 1-scene policy * Refactor gym env with unified guidance mode. * Update visualizer with unified plotting of reference traj * Minor * Transform reference headings to local coordinate frame. * Give value network a bit more capacity * Give value network a bit more capacity * Add dataframe code * Minor * Add optional video logging to wosac script * Update init mode * Visualizer bug fix * Minor * wosac eval updates and new cpts * Add WOSAC initialization modes (#426) * detect parked script * default arg change * parked vehicle mask * train and eval init modes * Minor fixes --------- Co-authored-by: kevin <kevinwinston184@gmail.com> * Better agent POV visualization and add LiDAr (#424) * Improve agent pov plotting utils * Delete old unused function * Fix * Add lidar obs option to plot agent observation * wip * Revert defaults * Hot model fix * Support for online guidance with `vbd_online` (#431) * Delete obsolete vbd functions * Format * Remove all instances of use_vbd and alike -> unified guidance mode * Create dataclass struct for VBD online predictionsions * Delete world_time_steps as it is no longer used * Leaving a todo * Make sure to demean the VBD predicted trajectories * Data analysis and minor changes * Guidance data analysis notebook * Script to process guidance data * Fixes * Fixes * Update nb * Fixes and make sure to always wrap the yaws. * Fixes and make sure to always wrap the yaws. * Example data to work with * Integrate optional smoothening pipeline for guidance data. (#434) * Temporary fix: Access average z position (elevation) from log replays (#427) * Intended usage for z-axis * Access average z pos (elevation) from logs * Fix intended usage of avg_z * fixes * z pos fixes * Fix bug by converting all vals in tensor to floats * remove print statement * Cleanup --------- Co-authored-by: kevin <kevinwinston184@gmail.com> * Res/scaling (#436) * Small improvements * Minor training improvements * Rebase over dev * Bookkeeping * Bookkeeping * Small fixes for gpu * wip: config * Make adding the action optional * wip * Analysis notebooks * WIP: new reward mode * Frequency * Make sure to normalize rewards and other changes * Minor * Update model * Add bonus at end of episode mechanism * Fixes * Improve renderer * Rmv one-hot encoding * Fix render * Adding back the whole shazam * Naming stuff * Add data folder as cmdline arg * Add data folder as cmdline arg * Minor bug fixes * Cleanup and better error message * Align reward components scale * model looks kinda okay? (#439) * model looks kinda okay? * think we added the steer angle correctly to the env * Small bug fix --------- Co-authored-by: Daphne Cornelisse <cor.daphne@gmail.com> * Ev/memory decrease (#442) * add support for model switching * remove memory pinning * revert changed values * update log for each scenario (#433) * Bug fixes to get rid of large position values with `vbd_online` (#435) * Bug fix: Ensure there is always a warmup period of at least 10 steps when using vbd_online * online guidance fixes * minor fix * changes to get 91 steps * Keep episodeLen at 90, and don't use it toread in the logs * Add back warnings / idiot proofing * tiny fix * fix online guidance * Set max agents to 32 --------- Co-authored-by: kevin <kevinwinston184@gmail.com> * RL training improvements (#440) * Small improvements * Minor training improvements * Rebase over dev * Bookkeeping * Bookkeeping * Small fixes for gpu * wip: config * Make adding the action optional * wip * Analysis notebooks * WIP: new reward mode * Frequency * Make sure to normalize rewards and other changes * Minor * Update model * Add bonus at end of episode mechanism * Fixes * Improve renderer * Rmv one-hot encoding * Fix render * Adding back the whole shazam * Naming stuff * Add data folder as cmdline arg * Add data folder as cmdline arg * Minor bug fixes * Cleanup and better error message * Align reward components scale * Push models * Viz improvements * Data analysis * Add guidance dropout mask * Add dropout as option for training * Add dropout as option for training * Bug fix: only count valid points * Increase network size to 200K params * Bug fix * Larger net * Set controlled agent default to 32 * Minor improvements * WIP * Make agents their original size * Minor * Remove collision state from ego * Plotting stuff * wip * More plotting stuff * Minor * Fixes * Reward improvements * Reward improvements * Give agent a bit more road graph information * Visualizer bug fix: clean up axis before generating new plot * Formatting * Change dropout mechanism such that dguidance_dropout_prob represents the maximum dropout probability [cover wide range]. * Add mechanism to discourage agents from turning around for bonus * Fix sbatch generation script * Add speed penalty if end of traj is reached * wip * Some eval fixes * Eval wip * Fix action space bug * Fix defaults * VBD amortize shape bug fix * rmv speed penalty * Settings * Fig * huh? * Replace end of route bonus with small jerk penalty * Many improvements * Exclude data * Delete checkpoints * Minor * make vbd 91 steps (#446) * Fix dynamics model and some setting updates (#447) * Fix: ensure vbd_online can be used on cuda and cpu. * Small updates * WOSAC defaults * Dynamics model fix * Minor * wosac dataset (#448) * dataset script * wosac dataset script * parallelized wosac script * pkl support * vbd cuda --------- * Intermediate update (#449) * Fix: ensure vbd_online can be used on cuda and cpu. * Small updates * WOSAC defaults * Dynamics model fix * Minor * Update sbatch * Code to make latex wosac tables * pkl file * Add smoothness * Add new dropout mode * WOSAC eval improvements * Add figures * Increment progress based on visible route points only. * Files * Some small updates * More updates (#450) * Fix: ensure vbd_online can be used on cuda and cpu. * Small updates * WOSAC defaults * Dynamics model fix * Minor * Update sbatch * Code to make latex wosac tables * pkl file * Add smoothness * Add new dropout mode * WOSAC eval improvements * Add figures * Increment progress based on visible route points only. * Files * Some small updates * Minor * WOSAC eval * Small fix * fix the wosac eval * fix amortization script (#452) * Added configs, params and code for the view cone setting (#454) Co-authored-by: Pragnay Mandavilli <pm3881@ga014.hpc.nyu.edu> * Type-aware action space (#455) * Testing agent type in obs * Implement type-aware action space * Remove padding agent * Cleanup * Comment out cone view line for now * Update angle ranges * Fix out of range error for VBD trajectory (#456) * fix empty geometry handling and out of bounds indices * add safer access and overflow prevention * allow flexible no of elements * simple occlusion check working * add occlusion check with nested loop instead of bvh * ray trace against 8 corner points instead of only center * change vector to array for gpu compatibility * resolved rebase conflicts with dev * add action for head tilt animation * change line style * update config files to new parameters * set full view and no occlusion as default * add addtional sampling points for occlusion check * rename parameter and add documentation * pass config parameters into config object * Integrate traffic light states (#425) * Add tl_states to extraction script. * WIP * Add back metadata * Add tl_states struct and access through simulator. * Add time index * Cleanup * Cleanup * SMall fix * Add tl state data struct * Add minimum test script * Add minimum test script * [Work In progress] Fixed initialization and made TL a singleton tensor * Fix json init and export * mean centering * Test and omit unnesecessary code in tl obs * Changed data access format in python code * Added a more interpretable positional element arrays * Unpack tl_states correctly * Add tl state plotting function in visualizer * mini bug * Fix traffic lights by exporting everything as float32 * Improve colors * Remove test file --------- Co-authored-by: Aarav Pandya <ap7641@nyu.edu> Co-authored-by: Eugene Vinitsky <eugenevinitsky@users.noreply.github.com> Co-authored-by: Pragnay Mandavilli <pm3881@ga034.hpc.nyu.edu> Co-authored-by: Pragnay Mandavilli <pm3881@ga014.hpc.nyu.edu> * [FIX] adapt TL+other missing changes to new json parsing --------- Co-authored-by: Daphne Cornelisse <33460159+daphne-cornelisse@users.noreply.github.com> Co-authored-by: kevin <kevinwinston184@gmail.com> Co-authored-by: Daphne Cornelisse <cor.daphne@gmail.com> Co-authored-by: Eugene Vinitsky <eugenevinitsky@users.noreply.github.com> Co-authored-by: Zixu Zhang <zixu@umich.edu> Co-authored-by: Zixu Zhang <zixuz@princeton.edu> Co-authored-by: Pragnay Mandavilli <108453901+mpragnay@users.noreply.github.com> Co-authored-by: Pragnay Mandavilli <pm3881@ga014.hpc.nyu.edu> Co-authored-by: Robin <49201699+rjs02@users.noreply.github.com> Co-authored-by: Robin Sieber <rjs02@hotmail.com> Co-authored-by: Aarav Pandya <ap7641@nyu.edu> Co-authored-by: Pragnay Mandavilli <pm3881@ga034.hpc.nyu.edu> Co-authored-by: Ellington Kirby <ellingtonkirby@gmail.com>
EllingtonKirby
requested changes
Jun 26, 2025
Collaborator
EllingtonKirby
left a comment
EllingtonKirby
left a comment
There was a problem hiding this comment.
need to change src/json_serialization.hpp line 711
| if (tl_states_element.is_object() && tl_states_element.get_object().get(tl_states) == simdjson::SUCCESS) { | ||
| if (std::begin(tl_states) != std::end(tl_states)) { | ||
| // non-empty object | ||
| map.numTrafficLights = std::max(tl_states.size(), static_cast<size_t>(consts::kMaxTrafficLightCount)); |
Collaborator
There was a problem hiding this comment.
this is a bad mistake on my part, we need to change this to std::min otherwise we can load way too many traffic lights
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Work in progress. Will add more description once I test.
Some test results