Is it possible to get (or compile) .pdb file for sharp.node for specfic sharp version? #3569
Comments
|
Rather than the sharp binary, I suspect this will be coming from one of libvips dependencies. This means you'll need to build your own debug binaries for the specific version of libvips you're using with https://github.com/libvips/build-win64-mxe and its I'm 80% certain this will relate to font discovery/rendering. The next v0.32.0 release of sharp will include prebuilt binaries that include the latest cairo, pango and fontconfig, all of which have seen a lot of font-related fixes and improvements in the last year, so waiting might be easier. |
|
Thank you for your reply! My next question was about libvips, since I have crashes with this one too, and now I will try building it myself and saving PDB file for future. However, my original question was indeed about sharp, as it does crash. Any ideas how to debug this? |
|
Please can you provide a full stack trace of a crash that you believe is caused directly by sharp (as opposed to a crash from one of libvips dependencies where a call to sharp happens to appear in the call stack). |
|
Sure. This archive includes sample .dmp file as well as stack trace decoded by cdb.exe |
|
Thanks, the salient part appears to be: This looks like it might be an attempt to shutdown libuv whilst threads from its worker pool are still processing data. If you've not seen it, Electron provides an API to handle graceful shutdowns - https://www.electronjs.org/docs/latest/api/app#event-before-quit The stack dump mentions Electron v14.2.6, which is EOL and uses an out-dated Node.js, so I'd recommend upgrading that bit first. |
|
Sorry, I am not sure I understand. Yes, there is 'before quit' event in electron and I use it. Is there something I should also do with sharp library too in that call in order to avoid such crashes? Updating Electron is huge effort as it contains lots of breaking changes. And it works just fine. In any case, it does not look like it is who is causing crashes, so I don't think this is the priority right now. |
|
By the way, I provided you with just one crash dump, but the thing is, this is what happens regularly for different users. And the stack is always the same, it always has this sharp_win32_x64+0x294cc line. |
|
Did you see https://sharp.pixelplumbing.com/api-utility#counters |
|
Upgrading Electron to a version with Node.js 16 would bring in nodejs/node#35021 that might fix this. |
|
Building a PDB locally for sharp v0.29.3 using this patch: --- a/node_modules/sharp/binding.gyp
+++ b/node_modules/sharp/binding.gyp
@@ -205,7 +205,8 @@
'VCCLCompilerTool': {
'ExceptionHandling': 1,
'Optimization': 1,
- 'WholeProgramOptimization': 'true'
+ 'WholeProgramOptimization': 'true',
+ 'DebugInformationFormat': 3 # Generate a PDB
},
'VCLibrarianTool': {
'AdditionalOptions': [Produces this stack trace in WinDbg (the So, it looks like you ran into #2999, which was fixed in sharp v0.30.0. |
Actually, the stack trace of that issue was: DetailsSo, it's probably not related to that. |
|
@lovell thank you for clarification. Actually, I have lots of other native stuff going on in background of my application, but I don't need to do any magic in 'before-quit' to wait for it to finish, as electron (the node core) just waits for all running tasks to finish before killing the process. I have pretty heavy computations for face recognition, and node waits for it to finish without issue. Therefore my quess is that in sharp case it will behave the same way, unless there is bug in sharp. My point is, rather than building strange workarounds it would make sense to figure out what exactly is wrong and maybe fix it. As I can see from discussion (thank you guys!), there is no clarity on what exaclty this bug is. Therefore my plan probably should remain as it was:
Please let me know what you think. In the mean time I will try to figure out how to compile sharp.node when it is installed as npm library. |
|
Were you able to add a call to |
|
I don't believe this will give us anything regarding this issue for multiple reasons:
|
|
Commit 4ec883e addresses what could be the source of a possible race condition at shutdown, especially when using Node.js 14, as it sometimes becomes no longer possible to call back into JS from C++. This will be in v0.32.0. Node.js 16 improves the situation too, via commits nodejs/node@e326c41 and nodejs/node@602fb3b |
|
Quick update from my side. I managed to modify my build system so now I can actually have matching Long story short, now I have full stack of the crash. However, it does not really shares more light (at least to me) on the reasons for this, as this all is happening during closing stage. But we already knew that. In any case I am updating my production to 0.31.3 now and we will see if that will fix the that particular issue. So far in my tests everything seems to work fine and I don't see any incompatibilities. We will see about crashed after I will release it. However, though I can now reproduce crash stack with symbols for sharp.node, apparently this is not the biggest issue. There is much more crashes inside libvips-42. And that one is much trickier to build, let alone build with PDB. I spent few hours figuring out how to build library and managed to do that using build-win-64, but I can't really go any further and build PDB, as I have zero knowledge about all the building tools you guys are using. I hope some of errors will go away with updating version, but it is hard to believe that all of them will go, as there is wide variety of them. I would like to ask your help with instructions on how to build libvips with PDB. I assume it will be beneficial for everyone if I would be able to provide you with crash dumps with symbols from actual production use. However, ideal solution be, if you add PDB as target to your build system, so every time you publish .zip with prebuilt binaries to github, you would also put .zip with matching .PDB's. But that is more chirtmas wish, I would be grateful for any help, such as hint on how to build PDB's myself. |
|
Status update. I have updated sharp to 0.31.3 (latest at that moment) and now enough time has passed sice my release to analyze the results. Overall, the amount of crashes seems to decrease couple times (though this is not an acurate calculation, rather than an guesstimate.) Nevertheless, the issue that I described in the beginning of this thread is still there, thougn it is not that frequent now. I can provide dump files if you'd like. Any suggestions? Also, any ideas on how to make PDB's for libvips? |
|
@TomaterID Are you still seeing this problem with the latest sharp v0.32.1? |
|
I still see it on 0.31.3 which was latest at that moment. Please give me some to update to 0.32.1 |
|
Status update. I did some sorting of crash reports after updating to 0.32.1 I'm afraid, the problem is still there:
I can send dmp files if you want, there are many of them. I am actually getting owerwhelmed with sharp crashes (most of them in libvips though). As popularity of my application is growing I am getting less confident I can continue with sharp being so unreliable. Currenly it is responsible for 95%+ of all crashes on customers side of my application. I would appreciate any thoughts and suggestions. Not only about this paricular crash, but also about all other crashes that come from libvips. I can provide you with lots of dmp files for debugging, but having no access to libvips pdb's or even not having opportunity to build ones myself I am not sure those dmp files will be of any use to anyone. Please help. |
|
Thanks for the updates. Which version of Electron are you using now? Does it include a version of Node.js with commits nodejs/node@e326c41 and nodejs/node@602fb3b ? |
|
I am currently using electron 14.2.6, not sure how to check if certain commit is there. |
|
https://github.com/electron/electron/blob/v14.2.6/DEPS#L20 says Node.js v14.17.0, which would suggest not, unless it has been patched. A possible side effect of not having commit nodejs/node@e326c41 would look like the crash highlighted in #3569 (comment) I highly recommend you upgrade to a version of Electron that uses Node.js 16+. |
|
@lovell ok, thanks. I will update to electron 22 (last one to support winows 7/8/8.1 which is still significiant poriton of users). That will include lots of testing, but I had to do that anyway, sooner or later. I hope that will help us to get rid of this particular issue at least, it accounts to roughly 25% of crashes, so that is a good start. I will let you know about results after a while. |
|
Electron v.22.3.8, sharp v.0.32.1 the error is still there. If you want, I can provide .dmp files.
|
I'll look into producing PDBs by default for release builds, to facilitate post-mortem debugging with Windows Debugger. When building libvips' Windows binaries with the Looking at the stack trace above, I'm not sure if this is a problem with libvips (or any of its dependencies), as I'm not seeing any Without a reproduction it'll be hard to debug further. Does this crash only happen during shutdown? If so, it may be worth recompiling sharp with the
|
Thank you very much, @kleisauke ! That would help a lot, since unfortunately, sharp is not yet stable enough and crashes a lot, and most of happens inside libvips. Unfortunately that happens on end-user side and all I can get is crash dumps. Having PDB's would simplify troubleshooting a lot.
You are totally right, this one happens inside sharp itself, that is why I started from reporitng it first. This is very frequent one, but this is just one of many, and most of others are inside libvips. I will be posting separate issues as I process those dumps and triage them (unfortunately, this very laborous process). One of them I have already posted: #3677 Please stay tuned for more.
Crash dumps is all we have, unfortunately. I was not able to reproduce any of it myself and have zero idea about circumstances. I hoped that reading stack trace will give you more info about situation when it happens and hint about what would be the way to reproduce. @lovell had theory that this one of the already fixed bugs, but unfortiately I cannot confirm that - after updating sharp and electron it is still there. However, if this one is only happening during shutdow, then it may appear not to be a real problem for users, unlike other crashes that happen during photo archive processing and are pretty annoying, as you have to restart program many times until your archive gets processed. Anyway, I am open for your suggestions and ready to provide any support, including providing you with crash dumps. I would even love to debug myselft and push PR, but unfortunately without PDB's there is nothing I can do. |
|
Commit libvips/build-win64-mxe@676260e adds support for this. The statically-linked 64-bit libvips Windows binaries and corresponding PDB files built from that commit can be found here: Hope this helps. |
|
@kleisauke thank you so much! What do I need to do to make sure my production code has corresponding PDB files? Do I wait for next release or just replace binaries in latest version? |
|
Have just submit another crash: #3679 If we can fix these three (#3569, #3677 and #3679) that would mean my app would crash roughly four times less often, which is a big deal. I am looking forward for more info on how to apply PDB's to production build so I can get actionable crash dumps. As you know you can only use PDB that was build togeter with your binary that crashed, so I need to release those ones first to the public, then we will be able to collect dumps that match those PDB's |
As a best effort, I rebased the above commit on top of the released v8.14.2 binaries. You can find these PDB files here: However, I'm not sure if the Visual Studio debugger is able to associate these PDBs to the previous released binaries, since the COFF debug directory is not available on those DLLs. AFAIK, WinDbg would load these mismatched PDBs without any issues using the |
|
You can be sure that VC will not load those PDB's even if you build them from same sources. For some reason it builds different ones every time, and I guess that VC checks the checksums. Due to that reason I now build sharp myself for production in order to get matching PDB's. I never heard about |
This approach seems to work fine with WinDbg, thank you! I will soon provide more info about crashes. |
|
Just in case, fresh stack trace for this particular crash, using symopt trick I can have line numers now: |
|
Thanks for the updates. Given you're compiling your own sharp, and if you've not already tried, please can you add Lines 72 to 74 in 3340120 |
|
OK, will do. We'll have to wait for results. |
|
I've added |
|
Closing as this was superceded by #3677 |
Question about an existing feature
What are you trying to achieve?
I am currently using sharp in my photo management application Tonfotos. Library does great job, thank you guys!
However, it crushes from time to time at customer side. I am collecting crash dumps, but unfortunately I cannot get much of useful information from it as I don't have .pdb file corresponding to the binary that is being shipped with npm module. Best stack trace I can get looks like that:
(... skipped windows internals...)
KERNELBASE!RaiseException+0x69
sharp_win32_x64+0x294cc
sharp_win32_x64+0xbac6
sharp_win32_x64+0x421e2
sharp_win32_x64+0x2d680
sharp_win32_x64+0x2c5dd
(... more skipped ....)
Obviously somehting is happened inside sharp, but we will never know what exactly and where until we have corresponding .pdb file. I am using sharp 0.29.3, and I would love to update, if I would be able to check if that bug was indeed fixed in later versions. But this amount of information is far from enought to find similar issues here. Having name of the failing function will be a great help. I would be able to see the code changes history and make educated decision about version to chose.
Last resort would be to upgrade to latest and pray, but that would be a lottery. This will not guarantee that this issue will go, but would give non-zero probability of adding new issues, too. (No offense, it is just the way life is.)
When you searched for similar issues, what did you find that might be related?
Unfortunately, I was not able to find relevant discussions. Everything with 'pdb' keyword looks to be installation issue.
Please provide a minimal, standalone code sample, without other dependencies, that demonstrates this question
Hmm... not sure this is relevant to my question.
Please provide sample image(s) that help explain this question
I have no idea what causes crash, is it connected to any images, etc. It is happening at customer side and all I get is crash dump from crashpad.
So I would be greateful for any help:
Thank you very much!
The text was updated successfully, but these errors were encountered: