ipfs dag get behavior for nodes that were not added explicitly using dag put

[AndreaCensi]

Version information:

go-ipfs version: 0.4.20-
Repo version: 7
System version: amd64/darwin
Golang version: go1.12.4

Also reproduced on version 0.4.19 and on linux machines.

Type:

bug

Description:

I encountered a behavior that I didn't expect with the combination of ipfs block put and ipfs dag get.

In short: I create a DAG node on IPFS node N1. A second IPFS node N2 can see and retrieve
the block, but it cannot retrieve the DAG node. From the spec I expect that N2 should be able to retrieve the DAG node even though it has not been explicitly stored.

How to reproduce

Put a DAG node on IPFS node N1

Consider this simple file:

N1 $ cat test-data/one.json
{"a":"test"}

We add it using ipfs dag put:

N1 $ ipfs dag put test-data/one.json
zdpuAxm8CKxE6NDM52feyqpfzzY8mX6pkbS3YujhvbU6Yf9xt

We can then get it back with ipfs dag get:

N1 $ ipfs dag get zdpuAxm8CKxE6NDM52feyqpfzzY8mX6pkbS3YujhvbU6Yf9xt
{"a":"test"}

The CID zdpuAxm8CKxE6NDM52feyqpfzzY8mX6pkbS3YujhvbU6Yf9xt corresponds to the multihash
QmagLRWbVtjZ4tpD3DWJYCWVNGGUUgzMd6tXPpjwgcow44. We can see this by using:

N1 $ ipfs block get zdpuAxm8CKxE6NDM52feyqpfzzY8mX6pkbS3YujhvbU6Yf9xt | ipfs block put
QmagLRWbVtjZ4tpD3DWJYCWVNGGUUgzMd6tXPpjwgcow44    

Try to retrieve the data from a different IPFS node N2

Suppose another node N2 has the same block.

If the block above is not available anymore on the network, you can do:

N2 $ echo oWFhZHRlc3Q= | base64 -d | ipfs block put
QmagLRWbVtjZ4tpD3DWJYCWVNGGUUgzMd6tXPpjwgcow44

Now, on N2, I write:

N2 $ ipfs dag get zdpuAxm8CKxE6NDM52feyqpfzzY8mX6pkbS3YujhvbU6Yf9xt
(hangs)

What I would expect:

1. N2 recognizes that the CID zdpu...9xt corresponds to block Qmag...cow44.
2. The block Qmag...cow44 is readily available.
3. The DAG node is parsed and returned.

What happens: The node hangs - not sure from the logs what's going on, but it is doing something related to getting zdpuAxm8CKxE6NDM52feyqpfzzY8mX6pkbS3YujhvbU6Yf9xt from the network, while the block is already available.

Broader context

This behavior prevents having a valid workaround for the issues discussed in:

• "ipfs dag get" does not have CBOR output; hence data roundtrip is not guaranteed as JSON mangles data stored by application #6239
• Cbor http API #4313
• Question regarding the dag-cbor encoding of objects (trying to reproduce in Python) ipld/ipld#65

ipfs dag put has known problems in mangling the data because of JSON transport (ints get transferred to float, etc.) so I need to use ipfs block put to store my DAG nodes to make sure that the data is not mangled. However when I do so, IPFS doesn't seem to be able to retrieve the DAG nodes via the CID, even though the blocks are properly stored. The example above is the simplest minimum case that reproduces the issue.

[magik6k]

The issue is basically #5231 (#5285 partially addresses it, but only for cid0v-cidv1 with batching codecs) - tl;dr we address blocks by CIDs, not multihashes

As you can see in https://cid.ipfs.io/#zdpuAxm8CKxE6NDM52feyqpfzzY8mX6pkbS3YujhvbU6Yf9xt, the CID you get from dag put has a different codec from what you get after block put.

To make the CID retrivable after block put, you need to do:

$ echo oWFhZHRlc3Q= | base64 -d | ipfs block put -f cbor
zdpuAxm8CKxE6NDM52feyqpfzzY8mX6pkbS3YujhvbU6Yf9xt

[AndreaCensi]

Thanks! Everything works as expected when the cbor format is specified.