File Corruption

[anywhichway]

• Version:
  { version: '0.47.0', repo: 7, commit: '' }

• Platform:
  Linux newscart 4.19.0-9-amd64 Add to cli: ipfs pin [-r] <ipfs-path> #1 SMP Debian 4.19.118-2+deb10u1 (2020-06-07) x86_64 GNU/Linux

• Subsystem:

files

Severity:

One of following: HIGH

Description:

• What you did: wrote this to a file:

{"value":{"#":"Index@3Uq21ZYksMWiAeefCvpJ9dkPrbmJRt4s","User@P1949ohXiPvAzxfz36jTl6Ztjzj5kixv":{"id":"User@P1949ohXiPvAzxfz36jTl6Ztjzj5kixv","keys":{"User@/email/":true,"User@/name/":true,"User@/page/":true,"User@/#/":true}},"User@ZbZ9wIevnsMZwNnRwK8iz3zFhBLyEwsx":{"id":"User@ZbZ9wIevnsMZwNnRwK8iz3zFhBLyEwsx","keys":{"User@/email/":true,"User@/name/":true,"User@/#/":true}},"^":{"id":"Index@3Uq21ZYksMWiAeefCvpJ9dkPrbmJRt4s","version":1,"cname":"Object","ctime":1593560714371,"atime":1593560714371,"mtime":1593560714371,"birthtime":1593560684044}}}

• What happened

Read it just a few seconds later and got this, note the repeat of characters at the end, which is corrupt JSON:

{"value":{"#":"Index@3Uq21ZYksMWiAeefCvpJ9dkPrbmJRt4s","User@P1949ohXiPvAzxfz36jTl6Ztjzj5kixv":{"id":"User@P1949ohXiPvAzxfz36jTl6Ztjzj5kixv","keys":{"User@/email/":true,"User@/name/":true,"User@/page/":true,"User@/#/":true}},"User@ZbZ9wIevnsMZwNnRwK8iz3zFhBLyEwsx":{"id":"User@ZbZ9wIevnsMZwNnRwK8iz3zFhBLyEwsx","keys":{"User@/email/":true,"User@/name/":true,"User@/#/":true}},"^":{"id":"Index@3Uq21ZYksMWiAeefCvpJ9dkPrbmJRt4s","version":1,"cname":"Object","ctime":1593560714371,"atime":1593560714371,"mtime":1593560714371,"birthtime":1593560684044}}}e":1593560684044}}}

• What you expected to happen

Expected to read back the same string written. Note, we are writing strings not buffers, they are generated with JSON.stringify.

If we run the exact same code with Node fs calls instead of IPFS calls, things work ok.

Steps to reproduce the error:

This does not demonstrate the problem, but it is the wrapper that makes IPFS look like FS for our purposes:

async readFile(path) {
		path.startsWith("/") || (path = "/"+path);
		const chunks = [];
		try {
			for await(const chunk of this._ipfs.files.read(path)) {
			  chunks.push(chunk)
			}
			return Buffer.concat(chunks).toString();
		} catch(e) {
			if(e.code!=="ERR_NOT_FOUND") {
				throw e;
			}
		}
	}

async writeFile(path,string) {
		path.startsWith("/") || (path = "/"+path);
		try {
			await this._ipfs.files.write(path,string,{create:true,parents:true}});
		} catch(e) {
			throw e;
		}
	}

[anywhichway]

We tried deleting the file every time this happened (note it always happens on the same file), which did not help. However, if we set the truncate flag to true when we do the initial write the problem disappears. This would take the issue from HIGH to MEDIUM since there is a simple workaround. However, I would not want to get surprised by this in production code. So, either a core fix or some documentation guidance would be useful to others.

[achingbrain]

As you've found, you need to pass the truncate argument if you're writing data that is smaller than the file at the path provided.

Please see the docs for ipfs.files.write