Tendermint integration (graphprotocol#3212)

* initial changes

* codec changes

* Resolver + runtime fixes

* add chain init

* latest types corrections

* fix bad rebase solution

* initial work on data structs and hashes

* reference fixes

* One batch post-merge cleanup

* few next corrections

* working version

* change hash to struct

* remove unwraps

* move rendermint to chain folder

* near augmentations

* Netx batch of changes

* Extend some types

* almost done with mapping tendermint acss

* small tx test

* tx fixes

* proto + structures next update

* working demo

* FIx network mapping in tendermint

* fix git breakage

* change graph types

* Align with latest - working version

* Some further cleanup

* fmt

+ adjusting some comment indentation

* Some refactoring

* Latest structure bindings

* Implement ToAscObj for more codec types

* Fix EventList::parent_ptr

* remove unwrap
* return None if no last_block_id available

* Some refactoring

* Fix abi Asc conversions

* based on protobuf definitions return errors when missing values must be present
* and null when fields are optional

* Remove some unused imports

* Remove reimplementation of default trait fn

* Refactoring some imports

* Small fmt

* Use proper block hash

* Refactoring some imports

* Refactor anyhow::Error usage

* Implement DataSource::as_stored_dynamic_data_source

* Add paddings to fix structure alignment

Source: https://gist.github.com/sosedoff/26beea5429ce86530d4b055ed09859d5

* Remove unused imports

* Add the event trigger

* Gather events from transactions

* Refactor the code

* Bring back the `data` and `evidence` fields

* Refactor the code

* Ensure there's only one handler for each event type

* Fix the data conversion bug

* Use the `u8` type instead of `bool` for padding

* Error on DataSource try_from template

* Optimize DataSource event handlers validation

* Remove itertools

* Refactor some errors

* Some formatting

* Remove unnecessary async_trait

* Refactor some imports

* Add TendermintTrigger::block_hash

* Remove some comments

* Fix some clippy warnings

* Small fmt

* Fix compilation errors and warnings after merge

* Update Tendermint kind

* Update Tendermint definitions

* Tendermint fixes

* remove IngestorAdapter
* use new DeterministicHostError enum
* use new firehose types

* Update tendermint definitions and mapping

* Update tendermint definitions and mapping

* Remove comments

* Format files

* Update tendermint chain

* Pass a block header instead of a full block

* Fix compilation errors

* Small refactor

* Fix EventData block_header

* Remove unused variables

* Remove unused tendermint hash module

* Remove outdated chain data source filtering

* Remove unused tendermint types module

* Fix the block hash mismatch bug

* Remove the unused Hash type

* Remove unused tendermint adapter

* Some formatting

* fmt

* Add event_type to MappingTriggerWithoutBlock

* Regenerate asc tendermint

* Add paddings

* Format document

* Fix enum mapping

* Files refactor

* Use the stable protobuf definitions

* Derive PartialOrd for NodeCapabilities

* Remove unnecessary clone

* Change NEAR FIXMEs to Tendermint

* Use cheap_clone for Tendermint chain chain_store

* Add optimization FIXME to full block cloning

* Return Event Iterators instead of Vec

* Remove subtle-encoding dep

* Update tendermint crate version

* Change Figment to Firehose in build.rs

* Derive PartialOrd for TendermintTrigger

* Fix comment in main.rs

* Revert TendermintTrigger PartialOrd derive

* TendermintTrigger variant fields are generated by prost and don't implement PartialOrd
* need to rely on the Ord impl

* Remove TendermintBlockFilter

* Add custom tendermint Source

* add dynamic data sources error

* Add template error

* Remove Mapping kind field

* Add EventOrigin to TendermintTrigger::Event

* to filter handler events by origin as well

* Validate event handlers with origins

* Fix tendermint data source validation unit test

* Fix tests

* Fmt

* Implement final_block_ptr_for Tendermint blocks

* Simplify some anyhow errors

* Add tendermint TriggerFilter::to_firehose_filter

* Update tendermint Blockchain::new_firehose_block_stream

Co-authored-by: Łukasz Miłkowski <lukanus@uaznia.net>
Co-authored-by: Robert Ignat <robert.ignat91@gmail.com>
Co-authored-by: Patrycja Zawadka <patrycja@figment.io>
Co-authored-by: patiee <76398520+patiee@users.noreply.github.com>
Co-authored-by: roignpar <47150492+roignpar@users.noreply.github.com>

chain/near/src/codec.rs

@@ -26,15 +26,15 @@ impl LowerHex for &CryptoHash {
 impl BlockHeader {
     pub fn parent_ptr(&self) -> Option<BlockPtr> {
         match (self.prev_hash.as_ref(), self.prev_height) {
-            (Some(hash), number) => Some(BlockPtr::from((hash.into(), number))),
+            (Some(hash), number) => Some(BlockPtr::from((H256::from(hash), number))),
             _ => None,
         }
     }
 }
 
 impl<'a> From<&'a BlockHeader> for BlockPtr {
     fn from(b: &'a BlockHeader) -> BlockPtr {
-        BlockPtr::from((b.hash.as_ref().unwrap().into(), b.height))
+        BlockPtr::from((H256::from(b.hash.as_ref().unwrap()), b.height))
     }
 }
 

chain/tendermint/Cargo.toml

@@ -0,0 +1,24 @@
+[package]
+name = "graph-chain-tendermint"
+version = "0.25.1"
+edition = "2018"
+
+[build-dependencies]
+tonic-build = "0.5.1"
+
+[dependencies]
+graph = { path = "../../graph" }
+mockall = "0.9.1"
+prost = "0.8.0"
+prost-types = "0.8.0"
+serde = "1.0"
+anyhow = "1.0"
+semver = "1.0.3"
+
+graph-runtime-wasm = { path = "../../runtime/wasm" }
+graph-runtime-derive = { path = "../../runtime/derive" }
+
+[dev-dependencies]
+graph-core = { path = "../../core" }
+pretty_assertions = "0.7.2"
+test-store = { path = "../../store/test-store" }

chain/tendermint/build.rs

@@ -0,0 +1,8 @@
+fn main() {
+    println!("cargo:rerun-if-changed=proto");
+    tonic_build::configure()
+        .out_dir("src/protobuf")
+        .format(true)
+        .compile(&["proto/codec.proto"], &["proto"])
+        .expect("Failed to compile Firehose Tendermint proto(s)");
+}

chain/tendermint/proto/codec.proto

@@ -0,0 +1,280 @@
+syntax = "proto3";
+
+package fig.tendermint.codec.v1;
+
+option go_package = "github.com/figment-networks/tendermint-protobuf-def/codec";
+
+import "gogo.proto";
+import "google/protobuf/descriptor.proto";
+
+// used in Vote
+enum SignedMsgType {
+  option (gogoproto.goproto_enum_stringer)  = true;
+  option (gogoproto.goproto_enum_prefix)    = false;
+
+  SIGNED_MSG_TYPE_UNKNOWN   = 0 [(gogoproto.enumvalue_customname) = "UnknownType"];
+  // Votes
+  SIGNED_MSG_TYPE_PREVOTE   = 1 [(gogoproto.enumvalue_customname) = "PrevoteType"];
+  SIGNED_MSG_TYPE_PRECOMMIT = 2 [(gogoproto.enumvalue_customname) = "PrecommitType"];
+  // Proposals
+  SIGNED_MSG_TYPE_PROPOSAL  = 32 [(gogoproto.enumvalue_customname) = "ProposalType"];
+}
+
+// BlockIdFlag indicates which BlockID the signature is for
+enum BlockIDFlag {
+  option (gogoproto.goproto_enum_stringer)  = true;
+  option (gogoproto.goproto_enum_prefix)    = false;
+
+  BLOCK_ID_FLAG_UNKNOWN = 0 [(gogoproto.enumvalue_customname) = "BlockIDFlagUnknown"];
+  BLOCK_ID_FLAG_ABSENT  = 1 [(gogoproto.enumvalue_customname) = "BlockIDFlagAbsent"];
+  BLOCK_ID_FLAG_COMMIT  = 2 [(gogoproto.enumvalue_customname) = "BlockIDFlagCommit"];
+  BLOCK_ID_FLAG_NIL     = 3 [(gogoproto.enumvalue_customname) = "BlockIDFlagNil"];
+}
+
+message EventList {
+  EventBlock                new_block             = 1;
+  repeated EventTx          transaction           = 2;
+  EventValidatorSetUpdates  validator_set_updates = 3;
+}
+
+message EventData {
+  Event       event = 1;
+  EventBlock  block = 2;
+}
+
+message Block {
+  Header        header      = 1 [(gogoproto.nullable) = false];
+  Data          data        = 2 [(gogoproto.nullable) = false];
+  EvidenceList  evidence    = 3 [(gogoproto.nullable) = false];
+  Commit        last_commit = 4;
+}
+
+message BlockID {
+  bytes         hash            = 1;
+  PartSetHeader part_set_header = 2 [(gogoproto.nullable) = false];
+}
+
+message BlockParams {
+  int64 max_bytes = 1; // Note: must be greater than 0
+  int64 max_gas   = 2; // Note: must be greater or equal to -1
+}
+
+message Commit {
+  int64               height      = 1; // must be > 0
+  int32               round       = 2; // must be > 0
+  BlockID             block_id    = 3 [(gogoproto.nullable) = false, (gogoproto.customname) = "BlockID"];
+  repeated CommitSig  signatures  = 4 [(gogoproto.nullable) = false];
+}
+
+message CommitSig {
+  BlockIDFlag block_id_flag     = 1; // must match those in the enum above
+  bytes       validator_address = 2; // hash 256 20 bytes - this is the first 20 characters of a 32-byte key - SHA256(pubkey)[:20]
+  Timestamp   timestamp         = 3 [(gogoproto.nullable) = false, (gogoproto.stdtime) = true];
+  bytes       signature         = 4; // length should be > 0 and < 64
+}
+
+message Consensus {
+  option (gogoproto.equal) = true;
+
+  uint64 block = 1;
+  uint64 app   = 2;
+}
+
+message ConsensusParams {
+  BlockParams     block     = 1 [(gogoproto.nullable) = false];
+  EvidenceParams  evidence  = 2 [(gogoproto.nullable) = false];
+  ValidatorParams validator = 3 [(gogoproto.nullable) = false];
+  VersionParams   version   = 4 [(gogoproto.nullable) = false];
+}
+
+message Data {
+  repeated bytes txs = 1;
+}
+
+message Duration {
+  int64 seconds = 1;
+  int32 nanos   = 2;
+}
+
+message DuplicateVoteEvidence {
+  EventVote   vote_a              = 1;
+  EventVote   vote_b              = 2;
+  int64       total_voting_power  = 3;
+  int64       validator_power     = 4;
+  Timestamp   timestamp           = 5 [(gogoproto.nullable) = false, (gogoproto.stdtime) = true];
+}
+
+message Event {
+  string                  event_type = 1 [json_name = "type"];
+  repeated EventAttribute attributes = 2 [
+    (gogoproto.nullable) = false,
+    (gogoproto.jsontag)  = "attributes,omitempty"
+  ];
+}
+
+message EventAttribute {
+  string  key   = 1;
+  string  value = 2;
+  bool    index = 3; // nondeterministic
+}
+
+message EventBlock {
+  Block               block               = 1;
+  BlockID             block_id            = 2; // not present in v0.34.13
+  ResponseBeginBlock  result_begin_block  = 3;
+  ResponseEndBlock    result_end_block    = 4;
+}
+
+message EventTx {
+  TxResult tx_result = 1;
+}
+
+message EventValidatorSetUpdates {
+  repeated Validator validator_updates = 1;
+}
+
+message EventVote {
+  SignedMsgType event_vote_type   = 1 [json_name = "type"]; // should be present in the enum at the top of this file
+  uint64        height            = 2; // must be > 0
+  int32         round             = 3; // must be > 0
+  BlockID       block_id          = 4 [(gogoproto.nullable) = false, (gogoproto.customname) = "BlockID"];
+  Timestamp     timestamp         = 5 [(gogoproto.nullable) = false, (gogoproto.stdtime) = true];
+  bytes         validator_address = 6; // hash with a length of 20
+  int32         validator_index   = 7;
+  bytes         signature         = 8; // hash length should be >0 and <64
+}
+
+message Evidence {
+  oneof sum {
+    DuplicateVoteEvidence     duplicate_vote_evidence       = 1;
+    LightClientAttackEvidence light_client_attack_evidence  = 2;
+  }
+}
+
+message EvidenceList {
+  repeated Evidence evidence = 1 [(gogoproto.nullable) = false];
+}
+
+message EvidenceParams {
+  int64     max_age_num_blocks  = 1;
+  Duration  max_age_duration    = 2
+      [(gogoproto.nullable)     = false, (gogoproto.stdduration) = true];
+  int64     max_bytes           = 3; // Default is 1048576 or 1MB
+}
+
+message Header {
+  Consensus version               = 1 [(gogoproto.nullable) = false];
+  string    chain_id              = 2 [(gogoproto.customname) = "ChainID"];
+  uint64    height                = 3;
+  Timestamp time                  = 4 [(gogoproto.nullable) = false, (gogoproto.stdtime) = true];
+  BlockID   last_block_id         = 5 [(gogoproto.nullable) = false];
+  bytes     last_commit_hash      = 6; // hash 256 32 bytes
+  bytes     data_hash             = 7; // hash 256 32 bytes
+  bytes     validators_hash       = 8; // hash 256 32 bytes
+  bytes     next_validators_hash  = 9; // hash 256 32 bytes
+  bytes     consensus_hash        = 10; // hash 256 32 bytes
+  bytes     app_hash              = 11; // determined by application, not a fixed length
+  bytes     last_results_hash     = 12; // hash 256 32 bytes - first block will be results of an empty hash
+  bytes     evidence_hash         = 13; // hash sha256 32 bytes
+  bytes     proposer_address      = 14; // hash 256 20 bytes - this is the first 20 characters of a 32-byte key - SHA256(pubkey)[:20]
+}
+
+message LightBlock {
+  SignedHeader signed_header = 1; // cannot be nil
+  ValidatorSet validator_set = 2; // cannot be nil
+}
+
+message LightClientAttackEvidence {
+  LightBlock          conflicting_block     = 1;
+  int64               common_height         = 2;
+  repeated Validator  byzantine_validators  = 3;
+  int64               total_voting_power    = 4;
+  Timestamp           timestamp             = 5 [(gogoproto.nullable) = false, (gogoproto.stdtime) = true];
+}
+
+message PublicKey {
+  option (gogoproto.compare) = true;
+  option (gogoproto.equal) = true;
+
+  oneof sum {
+    bytes ed25519   = 1;
+    bytes secp256k1 = 2;
+  }
+}
+
+message PartSetHeader {
+  uint32  total = 1;
+  bytes   hash  = 2; // hash 256 32 bytes
+}
+
+message ResponseBeginBlock {
+  repeated Event events = 1
+      [(gogoproto.nullable) = false, (gogoproto.jsontag) = "events,omitempty"];
+}
+
+message ResponseEndBlock {
+  repeated ValidatorUpdate  validator_updates       = 1;
+  ConsensusParams           consensus_param_updates = 2;
+  repeated Event            events                  = 3;
+}
+
+message ResponseDeliverTx {
+  uint32          code        = 1;
+  bytes           data        = 2;
+  string          log         = 3;
+  string          info        = 4;
+  int64           gas_wanted  = 5;
+  int64           gas_used    = 6;
+  repeated Event  events      = 7 [(gogoproto.nullable) = false, (gogoproto.jsontag) = "events,omitempty"];
+  string          codespace   = 8;
+}
+
+message SignedHeader {
+  Header header = 1;
+  Commit commit = 2;
+}
+
+message Timestamp {
+  int64 seconds = 1;
+  int32 nanos = 2;
+}
+
+message TxResult {
+  uint64            height  = 1;
+  uint32            index   = 2;
+  bytes             tx      = 3;
+  ResponseDeliverTx result  = 4;
+}
+
+message Validator {
+  bytes     address           = 1;
+  PublicKey pub_key           = 2 [(gogoproto.nullable) = false]; // must be > 0
+  int64     voting_power      = 3;
+  int64     proposer_priority = 4;
+}
+
+message ValidatorParams {
+  option (gogoproto.populate) = true;
+  option (gogoproto.equal)    = true;
+
+  repeated string pub_key_types = 1;
+}
+
+message ValidatorSet {
+  repeated Validator  validators         = 1; // cannot be empty or nil
+  Validator           proposer           = 2; // cannot be nil
+  int64               total_voting_power = 3;
+}
+
+message ValidatorUpdate {
+  bytes     address = 1;
+  PublicKey pub_key = 2 [(gogoproto.nullable) = false];
+  int64     power   = 3;
+}
+
+message VersionParams {
+  option (gogoproto.populate) = true;
+  option (gogoproto.equal)    = true;
+
+  uint64 app_version = 1;
+}