feat: backfill provider scaffolder #19
Conversation
SEQ-204 Scaffolder: OP-translator <-> Metafiller
We need to scaffold the connection between the OP-translator and the Metafiller as part of the backfilling process . At this stage, we are uncertain about the exact data the Metafiller will return and the mechanism by which the OP-translator will communicate with the Metafiller. This ticket focuses on building the scaffolding on the OP-translator side to prepare for this connection. Once more information is available regarding the interaction details and data structure, we will update and refine the implementation accordingly. AC:
|
ericvelazquez
changed the title
There was a problem hiding this comment.
Thinking about moving this to its own folder /pkg/backfill. Any thoughts?
There was a problem hiding this comment.
support that since I don't believe the backfiller is specific to OP, right?
There was a problem hiding this comment.
I would leave it in the translator package, since it's not used anywhere else, but either is fine
There was a problem hiding this comment.
Moved it to its own package since it isn't specific to the translator
| } | ||
|
|
||
| var data *BackFillData | ||
| err = json.Unmarshal(body, &data) |
There was a problem hiding this comment.
Just a thought, I know we haven't designed the backfiller-indexer service yet.
Could we just write/read raw binary data in the response body? That would save us some overhead on producing / unpacking JSON and hex strings.
There was a problem hiding this comment.
We could but we need the EpochHash too.
There was a problem hiding this comment.
We could set it in the http headers
There was a problem hiding this comment.
Will need to think more about this. Don't love the header solution. But will work on a solution when working on this ticket: https://linear.app/syndicate/issue/SEQ-209/write-logic-in-op-translator-for-switching-between-data-retrieval
| func InitBackFillerProvider(cfg *config.Config) *BackFillProvider { | ||
| return &BackFillProvider{ | ||
| MetafillerURL: cfg.MetafillerURL, | ||
| Client: &http.Client{}, |
There was a problem hiding this comment.
could you add a TODO comment referencing SEQ-141
| } | ||
|
|
||
| type BackFillData struct { | ||
| Data string `json:"data"` // Hex format |
There was a problem hiding this comment.
nit: you could enforce that it's a hex string with a custom type, something like
type BackFillData struct {
Data HexString `json:"data"`
EpochHash common.Hash `json:"epochHash"`
}
...
func isHex(s string) bool {
// Remove 0x prefix if present
s = strings.TrimPrefix(s, "0x")
_, err := hex.DecodeString(s)
return err == nil
}
and use alongside the existing std hex package
There was a problem hiding this comment.
alt approach might be to make the data field private and add getter and setter, where you do isHex() validation in the setter. Though that might be more annoying than valuable
There was a problem hiding this comment.
will leave it like this for now for simplicity reasons but will spin up a ticket to look at best ways to handle hex values. That would be useful in the whole app, not just this file.
Ticket
What does this PR do?
Does this PR introduce any breaking changes (API/schema)?
Do any environment variables need to be updated or added before deployment?
How can this PR be tested?
Unit & integration tests