Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: prevent progress bar from panicking using workaround #6940

Merged
merged 4 commits into from
Jun 14, 2023

Conversation

conradoplg
Copy link
Collaborator

@conradoplg conradoplg commented Jun 13, 2023

Motivation

Running Zebra with progress bar makes it panic after a while.

This seems to be caused by a bug in howudoin/indicatif where if a progress bar gets stuck, the ETA keeps growing and growing until it overflows. I'll eventually try to get a simple reproducer and report it, but in the meantime we can do a workaround.

Solution

This removes the limit on the progress bars that are not guaranteed to complete, which removes the ETA and avoids the panic. The other progress bars (e.g. Blocks) could still panic if the node gets offline but this seems enough for now.

We could copy&paste and modify the TermLine implementation from howudoin to not display the ETA, but seems more work and would duplicate code.

They look like this with this PR:

 | Known Peers: 70                                                                                                    
 | Failed Peers: 3                                                                                                    
 | Inbound Connections: 0                                                                                             
 | Outbound Connections: 39                                                                                           
 / Queued Checkpoint Blocks (6y)
 [=============================>]  99% (2092738/2112016)                                                              
 / Verified Checkpoints (39w)
 [=============================>]  99% (10795/10875)                                                                  
 / Blocks (2h)
 [=============================>]  99% (2091696/2120401) Nu5                                                          
 | Never Attempted Peers: 27                                                             

Closes #6926

Review

This is not urgent since progress bars are optional. But I think it would be nice to get this in the stable release so that we can show the progress bars working during demos.

Anyone can review.

Reviewer Checklist

  • Will the PR name make sense to users?
    • Does it need extra CHANGELOG info? (new features, breaking changes, large changes)
  • Are the PR labels correct?
  • Does the code do what the ticket and PR says?
    • Does it change concurrent code, unsafe code, or consensus rules?
  • How do you know it works? Does it have tests?

Follow Up Work

Need to report the bug in indicatif (and maybe propose a PR to fix), will do that later

@conradoplg conradoplg requested a review from a team as a code owner June 13, 2023 15:07
@conradoplg conradoplg requested review from oxarbitrage and removed request for a team June 13, 2023 15:07
@github-actions github-actions bot added the C-bug Category: This is a bug label Jun 13, 2023
@codecov
Copy link

codecov bot commented Jun 13, 2023

Codecov Report

Merging #6940 (e9ec8ce) into main (c058f77) will increase coverage by 0.06%.
The diff coverage is 100.00%.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #6940      +/-   ##
==========================================
+ Coverage   77.59%   77.66%   +0.06%     
==========================================
  Files         310      310              
  Lines       41475    41463      -12     
==========================================
+ Hits        32183    32201      +18     
+ Misses       9292     9262      -30     

@conradoplg conradoplg requested a review from a team as a code owner June 13, 2023 17:37
Copy link
Contributor

@teor2345 teor2345 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this fix!

I think we should restart the Zebra instance before the demo as a workaround.

The long-term fix would be disabling the ETA on those progress bars (an API change), and fixing the overflow bug in those libraries. But that's not a priority right now.

Would you mind moving these limits into comments? It's important that we document them anyway, and we might as well refer to the existing constants.

zebra-consensus/src/checkpoint.rs Outdated Show resolved Hide resolved
@teor2345 teor2345 added P-Low ❄️ I-panic Zebra panics with an internal error message do-not-merge Tells Mergify not to merge this PR labels Jun 13, 2023
@conradoplg
Copy link
Collaborator Author

Thanks for this fix!

I think we should restart the Zebra instance before the demo as a workaround.

It takes 4m30s to crash on my machine, so that may lead to a very stressful demo 😅
This seems a very low risk change, but it's up to you. We can always cherry pick this when doing any demos.

Would you mind moving these limits into comments? It's important that we document them anyway, and we might as well refer to the existing constants.

Done in 74365c4

@teor2345
Copy link
Contributor

Thanks for this fix!
I think we should restart the Zebra instance before the demo as a workaround.

It takes 4m30s to crash on my machine, so that may lead to a very stressful demo 😅 This seems a very low risk change, but it's up to you. We can always cherry pick this when doing any demos.

Oh wow, that's not what was happening when I wrote the feature. This seems like a new bug in the libraries we're using.

Co-authored-by: teor <teor@riseup.net>
Copy link
Contributor

@teor2345 teor2345 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the fix!

@dconnolly dconnolly removed the do-not-merge Tells Mergify not to merge this PR label Jun 14, 2023
mergify bot added a commit that referenced this pull request Jun 14, 2023
@mergify mergify bot merged commit fd78556 into main Jun 14, 2023
@mergify mergify bot deleted the fix-progress-bar-panic branch June 14, 2023 19:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
C-bug Category: This is a bug I-panic Zebra panics with an internal error message
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Panic running 1.0.0-rc.9+23.gc058f77 with progress-bar feature
3 participants