Integrate ebook-converter (calibre binaries) #28
Comments
|
integrated the changes via commit 3675716 |
|
NICE!!! Redid my images/containers with the latest build. Kindlegen works, however having an issue with ebook-convert. Only 1% completed in the conversion after 10 min! The only thing I did in the configuration was select the ebook-convert and put in the path to the tool. I did not put any additional settings. Was I suppose to? Thanks for this build! |
|
Hi @bodybybuddha, thanks for testing. Didn't tried to convert an ebook via ebook-convert yet. I've to test a little bit and check if it's necessary to set specific options. I'll connect into the container too to execute it directly on command line to see if an error happens. |
|
Same here, with your docker and calibres' one as well. Calibre web stucks at 1% ad eternum, while the command line aslo stucks... /books/Joe Abercrombie/Tierras rojas (14677) # ls
'Tierras rojas - Joe Abercrombie.epub' cover.jpg metadata.opf
/books/Joe Abercrombie/Tierras rojas (14677) # ebook-convert Tierras\ rojas\ -\
Joe\ Abercrombie.epub Tierras\ rojas\ -\ Joe\ Abercrombie.mobi
1% Eingabe wird in HTML konvertiert …
InputFormatPlugin: EPUB Input running
on /books/Joe Abercrombie/Tierras rojas (14677)/Tierras rojas - Joe Abercrombie.epub
Found HTML cover OEBPS/Text/cubierta.xhtml
Parsing all content...
34% Umwandlungen am eBook werden ausgeführt …
Merging user specified metadata...
Detecting structure...
Flattening CSS and remapping font sizes...
Source base font size is 12.00000pt
Removing fake margins...
Removing level div_1 left margin of: 1.5em
Removing level div_1 right margin of: 1.5em
Cleaning up manifest...
Trimming unused files from manifest...
Trimming u'OEBPS/Text/cubierta.xhtml' from manifest
Creating MOBI Output...
67% MOBI Output-Erweiterung wird ausgeführtI've tried both copying the epub to /tmp (to discard docker volume issues) and renaming to test.epub (to avoid spaces and so on the name). No luck. The same epub on a normal linux desktop converts just fine, this is the output: $ ebook-convert test.epub test.mobi
1% Convirtiendo entrada a HTML...
InputFormatPlugin: EPUB Input running
on /tmp/test.epub
Found HTML cover OEBPS/Text/cubierta.xhtml
Parsing all content...
34% Aplicando transformaciones al libro electrónico...
Merging user specified metadata...
Detecting structure...
Flattening CSS and remapping font sizes...
Source base font size is 12.00000pt
Removing fake margins...
Removing level div_1 left margin of: 1.5em
Removing level div_1 right margin of: 1.5em
Cleaning up manifest...
Trimming unused files from manifest...
Trimming u'OEBPS/Text/cubierta.xhtml' from manifest
Creating MOBI Output...
67% Ejecutando el complemento MOBI Output
Serializing resources...
Creating MOBI 6 output
Generating in-line TOC...
Applying case-transforming CSS...
Rasterizing SVG images...
Converting XHTML to Mobipocket markup...
Serializing markup content...
Compressing markup content...
Generating MOBI index for a book
MOBI output written to /tmp/test.mobi
|
|
Today I've digged into the issue and found the following: So To get more detailed information I've changed then the level of LD_DEBUG to all and checked the output: Therefore the issue happens then at the GLIBC library from https://github.com/sgerrand/alpine-pkg-glibc. I tried the following things today:
At the Calibre site the following info appears: So at the moment I've no idea how to fix the issue, but I'll stay tuned to find a solution. |
|
Wow @Technosoft2000! That's a lot of testing. Thanks for doing that. Here's a shot in the dark: I've been running aptalca/docker-rdp-calibre docker image (Ubuntu base) for Calibre and I took a peek at its configuration. The only instance for that particular library, libQt5Core.so.5, is under /opt/calibre/lib. As in the above log file, the /opt/calibre/lib directory was the first directory checked for the file - and then it went onto using the version in /user/lib. I'll assume that Calibre isn't using the library files supplied with its installation. Also, at one time, I was running Calibre in 'headless' mode on my Ubuntu server via an X-server. So I checked out it's configuration and notice the same thing - the libQt5Core.so.5 file is only found in the /opt/calibre/lib directory and not the /user/lib. Don't know if the above was helpful or not. If I can help test, let me know what scenarios you'd like for me to test and I'll see if I can give it a whirl. Thanks! |
|
I want to post here my last try with this. I realized is on my desktop (arch linux) my ebook-convert is a python script While on the docker is So as it's dynamically linked I saw It fails now with: I guess ebook-convert can be compiled to be a statically linked binary (but I think we can enter in a dark and deep hole, ignoring the warning developers advice to avoid). Perhaps we could create a wrapper (just like arch linux's calibre does). The content of ebook-convert python is: What do you think? |
|
Just a quick update. I've just renamed the ebook-convert ELF binary and created a python wrapper instead with this content It will fail. On my desktop, in |
|
I think I may have found the solution... I simply changed the PATH for calibre binaries from: /opt/calibre/bin to: /opt/calibre I ran the ebook-convert manually (with verbose on) on an ePub book to Mobi and it worked! However, the tool didn't update the calibre database with the new data. Not sure if I needed to include a parameter or if another command was used to add the book to the database. But the conversion worked. Not exactly where to change the PATH for the image to do a full test. (EDIT: line 206 of the dockerfile seems to be the place.) Can one of you see if this simple change resolve the conversion issue? Thanks! |
|
Those are good news indeed!! You cannot simply put an ebook on a directory and hope it shows in calibre :| I guess when we convert the book through the GUI, it usually updates de calibredb, so perhaps just recreating the docker image with the right PATH just works, did you tried it? |
|
I was able to create a new image with a modified dockerfile. The manual ebook-convert work correctly. However, I ran two tests with the GUI. One, with the converter tool command line setting set to /opt/calibre and in the second test, the command line setting was set to /opt/calibre/bin. Both got stuck at the 1% mark for over 5 minutes in both tests. -- I did do a manual convert of the same file and it took under 30 seconds to complete. Here's the log of the working manual conversion: Now that manual conversion is working, I think the next steps is to take a look at how the feature is implemented in calibre-web. Just to rule out any 'basic' issues - before we go back into looking at libraries. I did not do a pull-request of the change I made to @Technosoft2000's docker image. I was hoping to solve this issue before making the request. |
|
Okay - I've learned a lot. Here's my update. First, I found out the calibre-web has changed in the last few days, so I had to do an update. Now instead of being at 1% and never ending, it will fail the conversion immediately. This is with the modified PATH solution above. This is better than before as now the system doesn't have a subprocess going forever. (Side note - it doesn't really go forever. If you look at the processes you'll see a new process started for the conversion. It'll have the cmd string as verbiage. After some time out the verbiage will change on the process to "[EBOOK-CONVERT]". So it doesn't look like the process is doing anything other just being there. I digress.) Then, I wanted to make sure the calibre-web was sending the right command with parameters to convert the book properly. The conversion code is the worker.py. Line 236 handles the gathering of the appropriate command: In my version I changed the file_path to be unicode: The real magic is line 246, where the subprocess is executed: To see what was in the command object, I put in a couple of debug lines just before line 246: Then I used the website to convert a book and this is what the log came back with: The good news - the command object is done correctly. The first item in the sequence the executable, then a list of args you want the subprocess to execute: So in the above, execute ebook-convert from the right directory, then the ePub you want converted, and where you want the mobi file to be. So far so good. The interesting bit is the error message that is given when the job fails: Note that '/calibre-web/app/' was added to the beginning of the ePub file location. The ebook-convert program is getting this string from somewhere - not sure if it means its the current directory (b/c if you do open a bash session in the docker image, you are in the /calibre-web/app directory) or if it's pulling it from something else. So I changed the subprocess.Popen command to this: I changed the current working directory to /. This is the result of that test: Ahh.. so what if I change the current working directory to cwd='' - different result: We are executing the ebook-converter, but we aren't passing in the right parameters? Oh yes - I also opened up a bash session to the container and from the default directory (/calibre-web/app) I executed what should be the correct command line for the conversion for the book I used in the tests above and it converted perfectly and in the right directory. This rules out any string issues with the path, etc. So now I'm stuck. Any suggestions where to go from here? Lastly, if one were to look into the code in the Worker.py file after the points mentioned above, one will see that on a successful conversion, the code will then add the new version of the book to the calibre database. I'll start looking into the ebook-convert tool a little bit more to see why it would have the default directory added to the ePub portion of the string and more into the pOpen subprocess interface (although I suspect pOpen isn't the issue.) |
|
Another quick update - I notice that any conversion to mobi may not have the requisite [PDOC] tag if you want to use WhispherSync, so I added a couple of extra parameters via the GUI screen. The new parameters did make it to the command object - but in the end I got the same failure. |
|
Last update on this one for tonight :-) I decided to write a small python script that used the same key lines from calibre-web to test it with the command object (array... whatever it is..): After many different variations I FINALLY got it to work. The only way the above code work was to remove all the double quotes from the book paths: So as it turns out, once we have the PATH corrected in @Technosoft2000's image - we will need to change calibre-web's worker.py to strip out the unneeded double quotes. However, after reading some of @OzzieIsaacs commits, there may have been some Window OS considerations. Thus the code may need to test for OS first before determining to strip out the double quotes. I was wondering if the jim3ma/docker-calibre-web has the same issue? Thoughts? |
|
Hi all, many thanks for additional testing and the new ideas and tests to get it work, awesome. I'll check this after my work and try to include the fix into the docker image. Thanks for the support :) |
|
On windows I had the same issue you are also stuck with: for an unknown reason something was added to the path, and therefore it didn't work. Another issue on window was, that without the double quotes it could not handle path with spaces in it. I'll doublecheck if it is working on "normal linux" without the qoutes, I can remove tham, and only add them for the windows version (which has a special processing anyway) |
That would be great, because then I don't have to patch the file inside the docker image. |
|
Thanks for the quick replies everyone! @Technosoft2000 - Just verifying as I'm already setup to do the tests: I ran a test of the working test script against your original image and it worked. So if @OzzieIsaacs can verify that removing the double quotes is the solution and finds time to do so - I think we are good to go! Thank you both for all your efforts! @adocampo - thanks for your contributions to this as well! |
…worked correct, therefore changed the internal Calibre path from `/opt/calibre/bin` to `/opt/calibre`; thanks to @bodybybuddha and @adocampo for testing and finding a solution for this issue; see for details at issue #28
|
I changed calibre-web to excecute the command without qoutes on linux. But to make it work I had to add an . |
|
Thanks @OzzieIsaacs! I'll test a little later today. |
|
Perfect! It worked! I updated my 'production' container to the latest calibre-web release and was able to send an ePub to my Kindle after a mobi conversion. Woot!!
Thank you @OzzieIsaacs! |
|
Just tried. Works flawlessly! |
|
Do I need to so something other than restart the docker to pick up the latest version of calibre-web? For me, it still shows not installed for Calibre converter. |
|
@astrodad - Stop your container, then do a docker pull of the image, start container. That will get you the latest version of calibre-web that was configured in the image. To get updates of calibre-web that occur after the image refresh - go into the admin page of calibre-web and you'll see a Check for Updates button. Click it and it'll change to Update if a new version exists. The update button will restart calibre-web for you. FYI - to use the calibre ebook-converter tool, go to the admin page of calibre-web and click on Basic configuration. Then fill in the appropriate fields: Hope that helps! |
|
I issued a docker pull, restarted and then also checked for updates. It found one and I applied. When re-booted it still shows Not Installed for the calibre converter tool. Log shows this: [2018-08-25 23:07:13,825] {/calibre-web/app/cps/web.py:170} INFO - Starting Calibre Web... |
|
That's interesting. When you did the pull, did it pull any new layers? I removed my container and the images for calibre-web and pull a new copy image: I also did another pull to get the following image info: When I did a check for an update on calibre-web, there was no further updates. Here's the date stamp of the latest commit: So the version in the image should be the latest as of right now. The about page did list the ebook-convert program (I didn't change my configuration files in the steps above.) I was also able to successfully send an ePub book to my Kindle, with a conversion. The only thing I can think of is maybe you have two images of the docker file? Can you do a "docker image ls" and double-check? I had this happen to me once - I had two images with the same repo name, but different tags; one with and the other with 'latest'. I got rid of the one and all was well. |
|
UPDATE: removing the image and the container fixed the problem. I never got the document but I didn’t get any errors. That’s a separate issue, but it looks like the conversion was good. Thanks for the help. Thanks for your help on this. I checked and didn't see any other images: $ docker image ls | grep calibre $ docker image pull technosoft2000/calibre-web Administration I am going to drop the image, drop the container, and try again from scratch. |
|
Hi @Technosoft2000, did you see the @astrodad UPDATE? He got it work by deleting the container and re-fetching it again. I'm not very experimented with docker, but I always create the containers with persistent volumes and when there is an update, I remove the container, then the image, and then grab the latest version. I haven't had any problem this way. |
|
@adocampo thanks for the hint, I've overlooked the UPDATE :-|
I do it in the same way :) |
|
Thanks! I’ll use this routine from now on. |
Merge the changes from https://github.com/jim3ma/docker-calibre-web
so that the ebook-converter from the original calibre application is included and can be used by calibre-web.
see also at janeczku/calibre-web#411
The text was updated successfully, but these errors were encountered: