Nikon D40x: Failed to read some raw files

[pengowray]

Describe the bug

Darktable 4.4.1 (Windows 10) could not load this image "failed to read white balance information" and "exits" darkroom to return to lighttable. Other software such as Rawtherapee has no problems. Darktable could load some other images in the series of photos but not all, so I don't believe the file is corrupted.

I've uploaded the image to raw.pixls.us as DSC_0839.NEF — 9.98 MB (10,469,923 bytes) [Uploaded five failing raw files to Dropbox now]

I can upload more images if needed, but I no longer have access to the camera to make new images. The Nikon D40x is an old camera, but note that it's quite a different model to the D40.

Steps to reproduce

Double click DSC_0839.NEF in lighttable mode.

Expected behavior

Open the image without an error

Logfile | Screenshot | Screencast

Commit

No response

Where did you install darktable from?

darktable.org

darktable version

4.4.1

What OS are you using?

Windows

What is the version of your OS?

Windows 10 Pro

Describe your system?

No response

Are you using OpenCL GPU in darktable?

No

If yes, what is the GPU card and driver?

No response

Please provide additional context if applicable. You can attach files too, but might need to rename to .txt or .zip

No response

[kmilos]

The D40X should be supported and the already existing sample https://raw.pixls.us/getfile.php/804/nice/Nikon%20-%20D40X%20-%2012bit%20compressed%20(Lossy%20(type%201))%20(3:2).NEF works.

Did you open this file perhaps in some Nikon software before? It's been known to corrupt it... How did you transfer it?

Can you please share the sample here as well (via some cloud service)?

[victoryforce]

I've uploaded the image to raw.pixls.us as DSC_0839.NEF — 9.98 MB (10,469,923 bytes)

raw.pixls.us is not a place where all darktable developers have access to files once uploaded. Also, your file is unlikely to be accepted and available through the site because it obviously has issues. It is better to share this file with some cloud service (google drive, dropbox, etc., whatever is more convenient for you).

[pengowray]

Sorry, assumed raw.pixls.us was more open.

Dropbox link:
https://www.dropbox.com/scl/fi/fcr4iyo1ia2t7o4vgrtfg/DSC_0839.NEF?rlkey=a402867pxqksb67ptvj280wvl&dl=0

[kmilos]

raw.pixls.us just serves a different purpose, let's analyze here first...

[pengowray]

This file is from 2009, so my exact process is hazy, and hopefully it hasn't been ravaged by random software over the years. I've never thought to use Nikon's own software and I don't believe I've ever used any Nikon software. I would have either copied the file directly from the memory card, or used some version of Lightroom (circa 2009) to copy it up. (Hopefully Lightroom doesn't modify raw files)

There are a bunch of photos in this series, and some get an error message with Darktable and some don't. I also tried in viewing this one in Rawtherapee and Windows 10's Photo viewer without problem.

[kmilos]

It doesn't seem to have been re-written by other Nikon software indeed...

Getting this error from rawspeed:

ERROR: [rawspeed] rawspeed::RawImage rawspeed::RawDecoder::decodeRaw(), line 304: const uint8_t* raw
speed::BitStreamForwardSequentialReplenisher<Tag>::getInput() [with Tag = rawspeed::MSBBitPumpTag; u
int8_t = unsigned char], line 158: Buffer overflow read in BitStream

@LebedevRI Any ideas?

@pengowray As this has nothing to do w/ white balance (just an "after effect" message), please update the title.

[LebedevRI]

Bit rot?

[LebedevRI]

Looks like we fail at column 3724 (out of 3904) of last row.
This is most certainly a bit rot.

[LebedevRI]

Looks like we fail at column 3724 (out of 3904) of last row. This is most certainly a bit rot.

Looking at that file, the raw data buffer specified by STRIPOFFSETS+STRIPBYTECOUNTS EXIF tags
does span to the end of the file, so those tags themselves weren't corrupted.
Either the camera, for whatever reason, did not write the data for those last pixels into the buffer,
(and thus the EXIF tags are faithful), OR there has been a lot of bitrot on that file,
in a way that would cause the decoding to consume more bits than would be necessary,
thus running out of bits before finishing decoding. I'm not sure what the third explanation is.

The first possibility is the most likely, IMO.

[pengowray]

Thanks all for looking into it.

I'm not always great with backups, but I went to check it against a recent backup and discovered for this photo (and others in the series) I have at least 3 different copies of the file, and they were made around the time as when the photo was taken. All their SHA256 hashes are identical.

So if the file's been corrupted it was likely either by the camera, by its memory card, or in the initial copying process to PC, and unlikely bit rot over time.

Here's four more files that are also wont open in darktable, but appear fine in other software, in case anyone still wants to look for consistency to the error: All 5 files in a folder

[ralfbrown]

Note that nearly all image viewers which can handle raw formats show the embedded JPEG preview rather than rendering the raw data. And in the raw formats I'm familiar with, the raw data comes after the preview, so a truncated file would still appear fine in such viewers....
Can't answer why Rawtherapee can display the files that cause trouble in dt.

[github-actions]

This issue has been marked as stale due to inactivity for the last 60 days. It will be automatically closed in 300 days if no update occurs. Please check if the master branch has fixed it and report again or close the issue.