Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

waypoints, segments, clinched #213

Closed
1 of 8 tasks
yakra opened this issue Mar 19, 2022 · 2 comments · Fixed by TravelMapping/DataProcessing#619
Closed
1 of 8 tasks

waypoints, segments, clinched #213

yakra opened this issue Mar 19, 2022 · 2 comments · Fixed by TravelMapping/DataProcessing#619

Comments

@yakra
Copy link
Owner

yakra commented Mar 19, 2022

Ideas for speeding up the highest-impact DB tables

C++:

[157.8] ...waypoints	4.0
[161.8] ...segments	6.8
[168.6] ...clinched	1.7
[170.3] ...

Python:

[444.9] ...waypoints	6.8
[451.7] ...segments	3.4
[465.1] ...clinched	5.7
[470.8] ...
  • C++
    • waypoints: The the same approach as for the routes & connectedRoutes tables, and nix std::string Waypoint::csv_line in favor of inserting the info directly into the ofstream.
    • segments has 2 components to improve:
      • Similarly replacing std::string HighwaySegment::csv_line
      • clinched_list setup: just construct std::pair<TravelerList*,unsigned int>s
    • clinched has nothing much to do other than adapt to the new clinched_list set up while processing segments. Will probably run a wee bit slower than just inserting already-constructed strings as we do now, but that should be outweighed by the improvement to segments.
  • Python:
    • Odd that in C++, segments took 1.7x as long as waypoints but here it takes half as long. Whatever, I'll blame the mysteries that go on behind the curtain, like string interning or whatever.
    • waypoints
      • Including the parens & newline in Waypoint.csv_line may help, avoiding constructing one string and then another string based off of it.
      • Would eliminating Waypoint.csv_line altogether help? AIUI, a string object is still constructed for the file object to write. As a temporary value, is it interned?
    • segments
      • Follow the lead of waypoints with parens, newlines, and the csv_line function.
      • Not much to do with clinched_list, as that has to be stringified before writing anyway. Except:
    • clinched can lose the parens & newlines, with those constructed directly in clinched_list earlier. Mirror image of the C++ diffs -- slower in segments, made up for by being faster here.
@yakra
Copy link
Owner Author

yakra commented Mar 21, 2022

related: TravelMapping#324

@yakra yakra mentioned this issue Mar 21, 2022
@yakra
Copy link
Owner Author

yakra commented Mar 31, 2022

  • waypoints DB table needs Python-style floats for latitude & longitude.
  • more broadly, search & destroy csv_line functions project-wide
    (Only HighwaySegment::csv_line left)
  • Waypoint::out_of_bounds needs pyfloats
  • DUPLICATE_COORDS too

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant