utf-8' codec can't decode byte 0xff in position 0

[scribas]

Hi

I am suddenly unable to open my project. This bug sounds like a duplicate of issue #470. However, I downloaded version 0.9.0 with the hope of importing your solution for issue #470, but still getting the same error. I am not storing as a single file and tracking in git. I unfortunately have no idea which exact file is causing the problem. Running Linux Mint 19.1 on Linux Kernel 5.0.0-15.

Debug: Web rendering engine used: QWebEngineView
Running manuskript version 0.9.0.
Note: No translator found or loaded from system locale for locale en_ZA.
Detected file format version: 1. Zip: False.
Traceback (most recent call last):
  File "/usr/share/manuskript/manuskript/ui/welcome.py", line 146, in loadRecentFile
    self.mw.loadProject(act.data())
  File "/usr/share/manuskript/manuskript/mainWindow.py", line 565, in loadProject
    self.loadDatas(project)
  File "/usr/share/manuskript/manuskript/mainWindow.py", line 792, in loadDatas
    errors = loadSave.loadProject(project)
  File "/usr/share/manuskript/manuskript/loadSave.py", line 66, in loadProject
    v1.loadProject(project, zip=isZip)
  File "/usr/share/manuskript/manuskript/load_save/version_1.py", line 663, in loadProject
    files[os.path.join(p, f)] = fo.read()
  File "/home/scribas/anaconda3/lib/python3.7/codecs.py", line 322, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 0: invalid start byte
Fatal Python error: Aborted

Current thread 0x00007f809b5cd740 (most recent call first):
  File "/usr/share/manuskript/manuskript/main.py", line 145 in launch
  File "/usr/share/manuskript/manuskript/main.py", line 158 in run
  File "/usr/bin/manuskript", line 30 in <module>
Aborted (core dumped)

[scribas]

Oh, I seem to have found the problem. I recently introduced some images of my characters. I deleted these and now the project is opening. I am not 100% sure anymore, where and whether I referenced the images in my project, but at least the project now opens.

Is it possible that Manuskript tries to open the foreign binary image files within the project folder structure as text files? It almost seems as if it is hitting binary characters, which make it bomb out?

[gedakc]

How did you add the character images?

If you Run Manuskript from Source Code you can learn which file is being read when the error occurs by by adding a print() statement before the file is read.

For example:

diff --git a/manuskript/load_save/version_1.py b/manuskript/load_save/version_1.py                                                                                                                                
index 048ae26..3af5d64 100644                                                                                                                                                                                     
--- a/manuskript/load_save/version_1.py                                                                                                                                                                           
+++ b/manuskript/load_save/version_1.py                                                                                                                                                                           
@@ -660,6 +660,7 @@ def loadProject(project, zip=None):                                                                                                                                                           
                         files[os.path.join(p, f)] = fo.read()                                                                                                                                                    
                 else:                                                                                                                                                                                            
                     with open(os.path.join(dirpath, f), "r", encoding="utf8") as fo:                                                                                                                             
+                        print ("DEBUG: os.path.join(p, f): ", os.path.join(p, f))                                                                                                                                
                         files[os.path.join(p, f)] = fo.read()                                                                                                                                                    
                                                                                                                                                                                                                  
         # Saves to cache (only if we loaded from disk and not zip)

Alternatively if you can provide a sample project file that produces the error then we can take a look at it when we have time.

[scribas]

Apologies for the delay.

I tested my theory on both Windows and Linux. To reconstruct the problem, simply add any image into the project folder and open up Manuskript. It then crashes.

So, for example, if I create a Shakespeare,msk project and save it with the option not to save it into a single folder, there will be a Shakespeare folder within the same folder where Shakespeare,msk resides. I wanted to add a couple of images into this Shakespeare folder and reference them with markdown notation inside Manuskript. Please note that no markdown code is required for Manuskript to crash. Just simply the presence of the image binary in the project folder already causes the problem.

[gedakc]

... add any image into the project folder and open up Manuskript. It then crashes.

Ah, that explains it. The project file format for Manuskript was not designed to contain files such as images. If Save to one single file was enabled, then this would be equivalent to storing the images within the zip file. This is not currently supported, but has been requested. See Adding images to docs.

What Works Currently:

To reference images with current versions of Manuskript be sure to store the images outside of the project file(s) and directories.

One option is to store the images in a directory at the same level as the project.msk file. For example:

• Create an images folder at the same level as the project.msk file
• To reference an image use: ![My Image One Title](images/myimage1.jpg)

.
|
+-- images --+-- myimage1.jpg
|            |
|            +-- myimage2.png
|            |
|            +-- ...
|
+-- myproject --+-- infos.txt
|               |
|               +-- labels.txt
|               |
|               +-- ...
|               |
|               +-- outline --+-- Chapter 1 --+-- Scene 1
|               |             |               |
|               |             |               +-- Scene 2
|               |             |               |
|               |             |               +-- ...
|               |             |
|               |             +-- Chapter 2 --+-- ...
|               |             |
|               |             +-- ...
|               |
|               +-- ...
|               |
|               +-- world.opml
|
+-- myproject.msk

This method also works when the project has Save to one single file enabled because the images are not stored within the manuskript project file.

.
|
+-- images --+-- myimage1.jpg
|            |
|            +-- myimage2.png
|            |
|            +-- ...
|
+-- myproject.msk

Then if this was posted on the web, you might compile/export the project as HTML and then place the project.html and images folder at the same location on the web server.

Is the above listed solution sufficient for your needs,
or should this issue be considered a request (duplicate) for issue #22?

[scribas]

No, this is a pretty cool solution! I saw that you added some notes to the wiki pages on this. I will close.

I must say, that version 0.9 is working very well for me. It is really now becoming a well-polished piece of software.

[scribas]

I tried to follow the instructions on the wiki page, for referencing the images inside the images folder: ![My Image One Title](images/myimage1.jpg). Firstly, it complained that it does not understand the protocol. I also tried various combinations in linux. I finally had to put the full path in, with a file:// upfront: ![My Image Title](file:///home/scribas/git_repos/manuskript/images/myimage1.jpg). I suspect that the reason for this is, that the present working directory is in the folder where the manuskript binary is kept?

Aside from this, if I put the correct reference into one of the pages within the Editor environment and hover over the link with my mouse, the image pops up. However, if I try and do this in the Characters environment, the image only flickers up for a second and then disappears again. As I move my mouse around over the link, the image will keep on flickering on and off very rapidly. It would be great if this could be improved?

[gedakc]

Your theory is likely correct in that a relative URL of the form images/myimage1.jpg is being referenced relative to the location of the manuskript executable, and not the location of the project.msk file. This should be changed.

Since the problem you discovered is different from the crash raised in this issue, would you please create a new issue. Then this issue can be closed.

[scribas]

As requested, I have opened a new issue with my last comment and will close this one.