Improve performance of the Bazaar VCS backend #5445

jelmer · 2018-05-26T18:46:25Z

Improve performance of the Bazaar VCS backend:

Use lightweight checkouts rather than a full branch clone
Export directly from the remote branch

This significantly improves performance for Bazaar branches. E.g. for installing bzr itself ("bzr+lp:bzr"):

Performance on my system for 'bzr co --lightweight lp:bzr':

 0.60s user 0.11s system 5% cpu 12.234 total

Performance on my system for 'bzr branch lp:bzr' (current behaviour):

 65.41s user 1.48s system 39% cpu 2:47.91 total

Fixes #5443
Fixes #5444

RonnyPfannschmidt · 2018-05-26T19:21:41Z

the failing build was a travis issue, i restarted the offending element

pradyunsg · 2018-05-30T17:40:15Z

Looks like the same as git's --depth parameter and related discussion.

jelmer · 2018-05-30T20:45:43Z

A distinction between git's --depth parameter and bzr's lightweight checkouts is that history in bzr's case is not truncated; any access of history (which should be rare) will result in bzr contacting the remote repository to retrieve the missing history (lazily).

RonnyPfannschmidt · 2018-05-31T13:37:39Z

@pradyunsg basically gits '--depth' is a half-assed feature that breaks half the world, while bzr pretty much did light checkouts correctly

pradyunsg · 2018-06-22T22:04:39Z

@jelmer Could you link to the issues that this PR should close (using the GitHub auto close 'magic')?

(on mobile)

jelmer · 2018-06-23T16:06:40Z

Done.

pradyunsg · 2018-06-23T16:37:01Z

FYI - linter check is failing.

jelmer · 2018-06-23T17:05:49Z

Sorry, fixed now.

BrownTruck · 2018-07-09T03:45:02Z

Hello!

I am an automated bot and I have noticed that this pull request is not currently able to be merged. If you are able to either merge the master branch into this pull request or rebase this pull request against master then it will eligible for code review and hopefully merging!

src/pip/_internal/vcs/bazaar.py

jelmer · 2018-07-31T00:52:11Z

Done.

BrownTruck · 2018-08-17T12:15:05Z

Hello!

I am an automated bot and I have noticed that this pull request is not currently able to be merged. If you are able to either merge the master branch into this pull request or rebase this pull request against master then it will eligible for code review and hopefully merging!

src/pip/_internal/vcs/bazaar.py

jelmer · 2018-08-18T13:42:52Z

Thanks, cjerdonek. Is there anything I can do to help get this merged? upstream keeps changing under me.

BrownTruck · 2018-09-27T22:15:02Z

Hello!

I am an automated bot and I have noticed that this pull request is not currently able to be merged. If you are able to either merge the master branch into this pull request or rebase this pull request against master then it will be eligible for code review and hopefully merging!

* Use lightweight checkouts rather than a full branch clone * Export directly from the remote branch Fixes pypa#5444, pypa#5443

jelmer · 2018-09-27T22:46:15Z

ping, any news on this?

pradyunsg · 2018-11-13T07:56:58Z

Thanks @jelmer for being so patient!

@cjerdonek could you please take a look at this?

jelmer · 2019-01-01T17:21:33Z

ping @cjerdonek

cjerdonek · 2019-01-06T22:48:45Z

Here are some high-level comments on this PR. My apologies for not taking a look sooner, but I've never used Bazaar, so I felt like I might not be qualified to review the PR. My comments below are merely high-level.

This PR changes three different operations for the Bazaar VCS: export(), fetch_new(), and update(). I think they should be proposed and considered separately since the rationale and decision may be different for each, and they may each require different tests, etc.
For each change there should be at least one test to make sure the code still works with the change. It's possible there aren't yet any functional tests using Bazaar.
The news files have extension .bugfix, but these are performance enhancements and so should be marked .feature.
One of the proposed changes here is to do lightweight checkouts, but for consistency reasons it doesn't seem like we should be doing this for Bazaar if we're not going to do it for Git. And for Git, this idea (shallow clones) was already rejected in the past here: git checkout could/should be done with --depth=1 #2432

jelmer · 2019-01-06T23:42:20Z

As discussed earlier in this pull request, lightweight checkouts in bzr are different from shallow checkouts in Git and the considerations for dropping git shallow clones don't apply here:

A distinction between git's --depth parameter and bzr's lightweight checkouts is that history in bzr's case is not truncated; any access of history (which should be rare) will result in bzr contacting the remote repository to retrieve the missing history (lazily).

I'll see if I can split up this pull request into two separate ones.

jelmer · 2019-01-15T22:46:06Z

I've split the export part out into a separate proposal: #6139

BrownTruck · 2019-03-01T17:00:54Z

Hello!

I am an automated bot and I have noticed that this pull request is not currently able to be merged. If you are able to either merge the master branch into this pull request or rebase this pull request against master then it will be eligible for code review and hopefully merging!

pradyunsg · 2019-05-23T04:31:19Z

Closing this since it's bitrotten. Please file a new PR if there's still interest in this.

pradyunsg added type: enhancement Improvements to functionality C: vcs pip's interaction with version control systems like git, svn and bzr state: needs discussion This needs some more discussion labels May 30, 2018

jelmer force-pushed the bazaar-perf branch from cc901e5 to 46960e6 Compare June 23, 2018 16:06

jelmer force-pushed the bazaar-perf branch from 46960e6 to 2c928a0 Compare June 23, 2018 16:43

BrownTruck added the needs rebase or merge PR has conflicts with current master label Jul 9, 2018

jelmer force-pushed the bazaar-perf branch from 2c928a0 to 001e405 Compare July 9, 2018 11:21

pypa-bot removed the needs rebase or merge PR has conflicts with current master label Jul 9, 2018

cjerdonek reviewed Jul 9, 2018

View reviewed changes

src/pip/_internal/vcs/bazaar.py Outdated Show resolved Hide resolved

jelmer force-pushed the bazaar-perf branch from 001e405 to 83d4895 Compare July 9, 2018 19:33

cjerdonek reviewed Jul 9, 2018

View reviewed changes

src/pip/_internal/vcs/bazaar.py Outdated Show resolved Hide resolved

jelmer force-pushed the bazaar-perf branch from 83d4895 to c300c29 Compare July 9, 2018 21:13

cjerdonek reviewed Jul 10, 2018

View reviewed changes

src/pip/_internal/vcs/bazaar.py Outdated Show resolved Hide resolved

jelmer force-pushed the bazaar-perf branch from c300c29 to f8cc8b3 Compare July 10, 2018 22:44

BrownTruck added the needs rebase or merge PR has conflicts with current master label Aug 17, 2018

jelmer force-pushed the bazaar-perf branch from f8cc8b3 to c73c8d8 Compare August 17, 2018 15:43

pypa-bot removed the needs rebase or merge PR has conflicts with current master label Aug 17, 2018

cjerdonek reviewed Aug 18, 2018

View reviewed changes

src/pip/_internal/vcs/bazaar.py Outdated Show resolved Hide resolved

pradyunsg removed the state: needs discussion This needs some more discussion label Aug 18, 2018

pradyunsg approved these changes Aug 18, 2018

View reviewed changes

BrownTruck added the needs rebase or merge PR has conflicts with current master label Sep 27, 2018

jelmer and others added 2 commits September 27, 2018 23:45

Some performance improvements for Bazaar:

026ab4d

* Use lightweight checkouts rather than a full branch clone * Export directly from the remote branch Fixes pypa#5444, pypa#5443

Add url parameter to bazaar's update method

e63799e

jelmer force-pushed the bazaar-perf branch from cef8654 to e63799e Compare September 27, 2018 22:45

pypa-bot removed the needs rebase or merge PR has conflicts with current master label Sep 27, 2018

Remove unused import.

6468e8b

jelmer mentioned this pull request Jan 15, 2019

Bazaar: Export directly from the remote branch. #6139

Merged

BrownTruck added the needs rebase or merge PR has conflicts with current master label Mar 1, 2019

pradyunsg closed this May 23, 2019

lock bot added the auto-locked Outdated issues that have been locked by automation label Jun 22, 2019

lock bot locked as resolved and limited conversation to collaborators Jun 22, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Improve performance of the Bazaar VCS backend #5445

Improve performance of the Bazaar VCS backend #5445

jelmer commented May 26, 2018 •

edited by pradyunsg

Loading

RonnyPfannschmidt commented May 26, 2018

pradyunsg commented May 30, 2018

jelmer commented May 30, 2018 •

edited

Loading

RonnyPfannschmidt commented May 31, 2018

pradyunsg commented Jun 22, 2018

jelmer commented Jun 23, 2018

pradyunsg commented Jun 23, 2018

jelmer commented Jun 23, 2018

BrownTruck commented Jul 9, 2018

jelmer commented Jul 31, 2018

BrownTruck commented Aug 17, 2018

jelmer commented Aug 18, 2018

BrownTruck commented Sep 27, 2018

jelmer commented Sep 27, 2018

pradyunsg commented Nov 13, 2018

jelmer commented Jan 1, 2019

cjerdonek commented Jan 6, 2019

jelmer commented Jan 6, 2019

jelmer commented Jan 15, 2019

BrownTruck commented Mar 1, 2019

pradyunsg commented May 23, 2019

Improve performance of the Bazaar VCS backend #5445

Improve performance of the Bazaar VCS backend #5445

Conversation

jelmer commented May 26, 2018 • edited by pradyunsg Loading

RonnyPfannschmidt commented May 26, 2018

pradyunsg commented May 30, 2018

jelmer commented May 30, 2018 • edited Loading

RonnyPfannschmidt commented May 31, 2018

pradyunsg commented Jun 22, 2018

jelmer commented Jun 23, 2018

pradyunsg commented Jun 23, 2018

jelmer commented Jun 23, 2018

BrownTruck commented Jul 9, 2018

jelmer commented Jul 31, 2018

BrownTruck commented Aug 17, 2018

jelmer commented Aug 18, 2018

BrownTruck commented Sep 27, 2018

jelmer commented Sep 27, 2018

pradyunsg commented Nov 13, 2018

jelmer commented Jan 1, 2019

cjerdonek commented Jan 6, 2019

jelmer commented Jan 6, 2019

jelmer commented Jan 15, 2019

BrownTruck commented Mar 1, 2019

pradyunsg commented May 23, 2019

jelmer commented May 26, 2018 •

edited by pradyunsg

Loading

jelmer commented May 30, 2018 •

edited

Loading