[4/N][zero serialization] Add fallback to the encoder client when it cannot tell chunk encoding format

[jianoaix]

Why are these changes needed?

We need the batcher to fallback to Gob chunk encoding, when chunks returned from Encoder do not have encoding format specified.

The reason is this case:

• Encoder: running at a release before this PR [3/N][zero serialization] Make Encoder produce and pass chunks in desired format #735
• Batcher: running at a release after the PR

So the Encoder will produce Gob chunks, and when Batcher tries to find out the format, it'll get ChunkEncodingFormat_UNKNOWN. So this shouldn't be treated as an error; instead, it should be treated as Gob.

Compatibility/correctness reasoning

Let's denote the versions of Encoder as:

• E0: any commit before PR [3/N][zero serialization] Make Encoder produce and pass chunks in desired format #735
• E1: any commit at or after this PR, but with ENCODER_ENABLE_GNARK_CHUNK_ENCODING as false
• E2: any commit at or after this PR, but with ENCODER_ENABLE_GNARK_CHUNK_ENCODING as true

And versions of Bather as:

• B0: any commit before PR [3/N][zero serialization] Make Encoder produce and pass chunks in desired format #735
• B1: any commit at or after this PR, but with BATCHER_ENABLE_GNARK_CHUNK_ENCODING as false
• B2: any commit at or after this PR, but with BATCHER_ENABLE_GNARK_CHUNK_ENCODING as true

The reasoning and testing of the compatibility of 9 combinations:

• E0-B0: current state
• E0-B1: the Encoder returns Gob, and Batcher sees ChunkEncodingFormat_UNKNOWN, but correctly falls back to Gob
• E0-B2: same as E0-B1, but additionally the Batcher will convert Gob to Gnark on the fly before sending to Node, at

  eigenda/disperser/batcher/grpc/dispatcher.go

  Line 368 in cacdc21

  chunksData, err = chunksData.ToGnarkFormat()

• E1-B0: the Encoder returns Gob and with the chunk_encoding_format set as GOB in the response; the Batcher doesn't care about chunk_encoding_format and just get the chunks and treat them as Gob, which is correct
• E1-B1: similar to E1-B0, it's just the Batcher will pass the chunk_encoding_format to Dispatcher, which will send chunk to Node as it is
• E1-B2: similar to E1-B1, it's just the Dispatcher will convert Gob to Gnark on the fly before sending to Node, at

  eigenda/disperser/batcher/grpc/dispatcher.go

  Line 368 in cacdc21

  chunksData, err = chunksData.ToGnarkFormat()

• E2-B0: N/A -- the Batcher will get new release(s) before the Encoder enable the ENCODER_ENABLE_GNARK_CHUNK_ENCODING as true
• E2-B1: the Encoder will return chunks in Gnark, with chunk_encoding_format set to GNARK; and then Batcher will understand it's GNARK, but it'll convert them from GNARK to GOB on the fly before sending to Node, at

  eigenda/disperser/batcher/grpc/dispatcher.go

  Line 393 in cacdc21

  chunksData, err = chunksData.ToGobFormat()

• E2-B2: similar to E2-B1, but now the Batcher will just send the chunks as it is to Node

Testing

All of the above combinations tested in preprod.

Checks

• I've made sure the lint is passing in this PR.
• I've made sure the tests are passing. Note that there might be a few flaky tests, in that case, please comment that they are not relevant.
• Testing Strategy
  • Unit tests
  • Integration tests
  • This PR is not tested :(