Skip to content

Conversation

smadhuk
Copy link
Collaborator

@smadhuk smadhuk commented Jul 2, 2025

With the removal of the IntermediateWeightedGraph, we can move all of the code from the io.* files into user_graph.* files. This also helps prepare the correlated-matching dev branch for upcoming changes where a new iter_dem_include_correlations method will depend on the UserEdge inside user_graph.h and prevent circular dependencies if the new method lived beside the currently existing iter method inside io.h.

This PR is a no-op on performance.

Before:

[.................*<<|....................] 530 us (vs 290 us) (2.8 Mdets/s) (9.6 Mlayers/s) (1.9 Mshots/s) Decode_surface_r5_d5_p1000
[................*<<<|....................]  23 ms (vs  10 ms) (920 kdets/s) ( 59 klayers/s) (5.3 kshots/s) Decode_surface_r11_d11_p100
[................*<<<|....................] 3.4 ms (vs 1.5 ms) (2.9 Mdets/s) (1.6 Mlayers/s) (140 kshots/s) Decode_surface_r11_d11_p1000
[..................*<|....................] 140 us (vs  83 us) (3.6 Mdets/s) ( 19 Mlayers/s) (1.8 Mshots/s) Decode_surface_r11_d11_p10000
[..................*<|....................]  56 us (vs  33 us) (3.3 Mdets/s) (190 Mlayers/s) ( 18 Mshots/s) Decode_surface_r11_d11_p100000
[................*<<<|....................]  16 ms (vs 7.5 ms) (600 kdets/s) (9.9 klayers/s) (470 shots/s) Decode_surface_r21_d21_p100
[...............*<<<<|....................]  22 ms (vs 7.8 ms) (450 kdets/s) (7.4 klayers/s) (350 shots/s) Decode_surface_r21_d21_p100_with_dijkstra
[................*<<<|....................]  23 ms (vs 8.2 ms) (440 kdets/s) (7.3 klayers/s) (340 shots/s) Decode_surface_r21_d21_p100_to_edges
[.................*<<|....................]  13 ms (vs 6.3 ms) (2.8 Mdets/s) (410 klayers/s) ( 19 kshots/s) Decode_surface_r21_d21_p1000
[...............*<<<<|....................]  26 ms (vs 7.7 ms) (1.3 Mdets/s) (200 klayers/s) (9.6 kshots/s) Decode_surface_r21_d21_p1000_with_dijkstra
[...............*<<<<|....................]  28 ms (vs 8.4 ms) (1.3 Mdets/s) (190 klayers/s) (9.0 kshots/s) Decode_surface_r21_d21_p1000_to_edges
[................*<<<|....................] 2.3 ms (vs 980 us) (3.2 Mdets/s) (4.5 Mlayers/s) (210 kshots/s) Decode_surface_r21_d21_p10000
[..............*<<<<<|....................] 4.8 ms (vs 1.3 ms) (1.6 Mdets/s) (2.2 Mlayers/s) (100 kshots/s) Decode_surface_r21_d21_p10000_with_dijkstra
[..............*<<<<<|....................] 6.0 ms (vs 1.4 ms) (1.2 Mdets/s) (1.7 Mlayers/s) ( 84 kshots/s) Decode_surface_r21_d21_p10000_to_edges
[.................*<<|....................] 180 us (vs  94 us) (3.9 Mdets/s) ( 58 Mlayers/s) (2.7 Mshots/s) Decode_surface_r21_d21_p100000
[...............*<<<<|....................] 400 us (vs 130 us) (1.8 Mdets/s) ( 26 Mlayers/s) (1.2 Mshots/s) Decode_surface_r21_d21_p100000_with_dijkstra
[...............*<<<<|....................] 420 us (vs 130 us) (1.7 Mdets/s) ( 25 Mlayers/s) (1.2 Mshots/s) Decode_surface_r21_d21_p100000_to_edges
[..................*<|....................]  54 ms (vs  35 ms) (180 loads/s) Load_dem_r11_d11_p100
[..................*<|....................] 440 ms (vs 280 ms) ( 22 loads/s) Load_dem_r21_d21_p100
[..................*<|....................]  54 ms (vs  35 ms) (180 loads/s) Load_dem_r11_d11_p100_correlations
[..................*<|....................] 430 ms (vs 280 ms) ( 23 loads/s) Load_dem_r21_d21_p100_correlations
[................*<<<|....................] 9.6 ms (vs 3.5 ms) (1.0 Gcalls/s) Varying32_get_distance_at_time
[..............*<<<<<|....................]  25 ms (vs 5.9 ms) (390 Mcalls/s) Varying64_get_distance_at_time
[....................*....................] 5.0 us (vs 4.9 us) (190 MEnqueueDequeues/s) bucket_queue_sort
[..................*<|....................] 140 us (vs  99 us) ( 67 MEnqueueDequeues/s) bucket_queue_stream

After:

[.................*<<|....................] 530 us (vs 290 us) (2.8 Mdets/s) (9.5 Mlayers/s) (1.9 Mshots/s) Decode_surface_r5_d5_p1000
[................*<<<|....................]  23 ms (vs  10 ms) (930 kdets/s) ( 59 klayers/s) (5.4 kshots/s) Decode_surface_r11_d11_p100
[................*<<<|....................] 3.4 ms (vs 1.5 ms) (2.9 Mdets/s) (1.6 Mlayers/s) (140 kshots/s) Decode_surface_r11_d11_p1000
[..................*<|....................] 140 us (vs  83 us) (3.6 Mdets/s) ( 19 Mlayers/s) (1.7 Mshots/s) Decode_surface_r11_d11_p10000
[..................*<|....................]  56 us (vs  33 us) (3.3 Mdets/s) (190 Mlayers/s) ( 18 Mshots/s) Decode_surface_r11_d11_p100000
[.................*<<|....................]  16 ms (vs 7.5 ms) (620 kdets/s) ( 10 klayers/s) (480 shots/s) Decode_surface_r21_d21_p100
[...............*<<<<|....................]  22 ms (vs 7.8 ms) (460 kdets/s) (7.5 klayers/s) (350 shots/s) Decode_surface_r21_d21_p100_with_dijkstra
[...............*<<<<|....................]  24 ms (vs 8.2 ms) (420 kdets/s) (6.9 klayers/s) (320 shots/s) Decode_surface_r21_d21_p100_to_edges
[.................*<<|....................]  13 ms (vs 6.3 ms) (2.8 Mdets/s) (410 klayers/s) ( 19 kshots/s) Decode_surface_r21_d21_p1000
[...............*<<<<|....................]  26 ms (vs 7.7 ms) (1.3 Mdets/s) (200 klayers/s) (9.5 kshots/s) Decode_surface_r21_d21_p1000_with_dijkstra
[...............*<<<<|....................]  27 ms (vs 8.4 ms) (1.3 Mdets/s) (190 klayers/s) (9.2 kshots/s) Decode_surface_r21_d21_p1000_to_edges
[................*<<<|....................] 2.3 ms (vs 980 us) (3.3 Mdets/s) (4.5 Mlayers/s) (210 kshots/s) Decode_surface_r21_d21_p10000
[..............*<<<<<|....................] 4.6 ms (vs 1.3 ms) (1.6 Mdets/s) (2.3 Mlayers/s) (110 kshots/s) Decode_surface_r21_d21_p10000_with_dijkstra
[...............*<<<<|....................] 4.9 ms (vs 1.4 ms) (1.5 Mdets/s) (2.1 Mlayers/s) (100 kshots/s) Decode_surface_r21_d21_p10000_to_edges
[.................*<<|....................] 180 us (vs  94 us) (4.0 Mdets/s) ( 59 Mlayers/s) (2.8 Mshots/s) Decode_surface_r21_d21_p100000
[...............*<<<<|....................] 400 us (vs 130 us) (1.8 Mdets/s) ( 26 Mlayers/s) (1.2 Mshots/s) Decode_surface_r21_d21_p100000_with_dijkstra
[...............*<<<<|....................] 420 us (vs 130 us) (1.7 Mdets/s) ( 25 Mlayers/s) (1.2 Mshots/s) Decode_surface_r21_d21_p100000_to_edges
[................*<<<|....................] 9.5 ms (vs 3.5 ms) (1.0 Gcalls/s) Varying32_get_distance_at_time
[..............*<<<<<|....................]  22 ms (vs 5.9 ms) (430 Mcalls/s) Varying64_get_distance_at_time
[....................*....................] 5.1 us (vs 4.9 us) (190 MEnqueueDequeues/s) bucket_queue_sort
[..................*<|....................] 140 us (vs  99 us) ( 67 MEnqueueDequeues/s) bucket_queue_stream

@smadhuk smadhuk requested a review from oscarhiggott July 2, 2025 20:33
Copy link
Owner

@oscarhiggott oscarhiggott left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice!

@oscarhiggott oscarhiggott merged commit 113303d into correlated-matching Jul 2, 2025
36 checks passed
@oscarhiggott oscarhiggott deleted the u/smadhuk/refactor-io branch July 2, 2025 22:01
oscarhiggott added a commit that referenced this pull request Aug 10, 2025
* Use UserGraph for MWPM creation instead of IntermediateWeightedGraph

* Format user_graph.h

* Flag-protect (unimplemented) correlated matching

* Don't throw if correlations are enabled; Add perf tests

* PR comment; Ensure backwards compatibility with C++ API

* Remove pymatching binary

* Fix CI

* Fix CI

* Fix pybind source files

* Refactor io.* into user_graph.* (#125)

Co-authored-by: Sid Madhuk <smadhuk@google.com>

* Add method to iterate through a DEM which decomposes Hyper Edges

* Small cleanup

* pr comments: remove comments and rename variable

* throw on decomposed errors with hyperedge components

* extend: throw on probabilities greater than half

* Track joint probabilities of correlated errors

* clenaup: remove extra newline

* pr feedback: keep edges sorted in the user_graph and joint_probabilities

* Calculate implied edge weights from joint probabilities

* remove unnecessary test

* small cleanup

* Populate implied edge weights in edges in the User Graph

* Populate unconverted implied weights during matching graph creation

* Convert implied weights for decoding

* Implement reweight logic for the search and matching graphs

* Pipe reweighting logic all the way to the Pymatching binary CLI

* Pipe reweighting logic all the way to the Pymatching binary CLI

* Clean up search graph logic so

* Fix tests

* Add correlated matching branches to methods missed in earlier PRs

* Fix pybind

* Update pybindings to handle correlations

* 2 fixes: undo weights at end of shots; reweight search flooder

* Fix edge reweighting logic

* Fix test failures

* Update src/pymatching/sparse_blossom/driver/user_graph.cc

Co-authored-by: oscarhiggott <29460323+oscarhiggott@users.noreply.github.com>

* Update src/pymatching/sparse_blossom/driver/user_graph.cc

Co-authored-by: oscarhiggott <29460323+oscarhiggott@users.noreply.github.com>

* Update src/pymatching/sparse_blossom/flooder/graph.cc

Co-authored-by: oscarhiggott <29460323+oscarhiggott@users.noreply.github.com>

* Update src/pymatching/sparse_blossom/search/search_graph.cc

Co-authored-by: oscarhiggott <29460323+oscarhiggott@users.noreply.github.com>

* Update src/pymatching/sparse_blossom/search/search_graph.h

Co-authored-by: oscarhiggott <29460323+oscarhiggott@users.noreply.github.com>

* Update src/pymatching/sparse_blossom/driver/user_graph.cc

Co-authored-by: oscarhiggott <29460323+oscarhiggott@users.noreply.github.com>

* fix tests

* Only discretize weights when converting to MatchingGraphs

* Update src/pymatching/matching.py

Co-authored-by: oscarhiggott <29460323+oscarhiggott@users.noreply.github.com>

* Update src/pymatching/matching.py

Co-authored-by: oscarhiggott <29460323+oscarhiggott@users.noreply.github.com>

* Update src/pymatching/matching.py

Co-authored-by: oscarhiggott <29460323+oscarhiggott@users.noreply.github.com>

* Update src/pymatching/matching.py

Co-authored-by: oscarhiggott <29460323+oscarhiggott@users.noreply.github.com>

* Fix tests

* fix formatting

* Remove unused method

* Update src/pymatching/sparse_blossom/driver/user_graph.h

Co-authored-by: oscarhiggott <29460323+oscarhiggott@users.noreply.github.com>

* Update src/pymatching/sparse_blossom/driver/user_graph.h

Co-authored-by: oscarhiggott <29460323+oscarhiggott@users.noreply.github.com>

* Update src/pymatching/sparse_blossom/driver/user_graph.h

Co-authored-by: oscarhiggott <29460323+oscarhiggott@users.noreply.github.com>

* Make previously free edges to implied weights unconverted variable a class member

* Add default new parameters to prevent breaking API changes

* remove unused method

* const ref for std::vector<ImpliedWeightUnconverted>

* check implied weight validity when computing normalizing constant

* return bool in get_edge_or_boundary_edge_weight and add test

* pr comment: use SIZE_MAX instead of -1 for consistency

* Add tests for correlations with pybind flow

* Remove unused blob of code

* Decode to edges array also allows correlated matching (#157)

* Decode to edges array also allows correlated matching

* format

* Update src/pymatching/matching.py

---------

Co-authored-by: Sid Madhuk <smadhuk@google.com>
Co-authored-by: oscarhiggott <29460323+oscarhiggott@users.noreply.github.com>

* Check for invalid pymatching.Matching configuration when decoding with enable_correlations=True (#160)

* Catch attempted decoding using enable_correlations=True when pymatching.Matching is not configured for correlations

* add to docstrings on correlations expose enable_correlation flags to more loading methods

* fix docstring

* turn on decompose_errors in perf

* decompose_errors in dem loading tests

* test for exception with undecomposed hyperedge

* flake8

* add enable_correlations to more docstrings

* flake8

* update undecomposed hyperedges error

* Ensure that pymatching CLI includes a search flooder for predict (#163)

Co-authored-by: Sid Madhuk <smadhuk@google.com>

* Correlated matching docs (#164)

* correlated matching docs

* readme

---------

Co-authored-by: Sid Madhuk <smadhuk@google.com>
Co-authored-by: oscarhiggott <29460323+oscarhiggott@users.noreply.github.com>
Co-authored-by: Oscar Higgott <oscarhiggott@google.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants