Skip to content

Commit

Permalink
Add chain reader and codec EVM implementation (#11842)
Browse files Browse the repository at this point in the history
* Implement skeleton interfaces, structs, & methods for ChainReader EVM POC

- Read ChainReader config in from RelayConfig
- Add some initialization and validation relay skeletons

- Use medianProviderWrapper instead of passing medianContract separately

This avoids us having to modify the signature of NewMedianFactory, which
would require further modifications to all non-evm repos and chainlink-relay

- Add chain_reader_test.go with some basic relay tests

Co-authored-by: Jordan Krage <jmank88@gmail.com>

- Add chain reader config validation
- Add chain reader config validation tests
- Add config for chain reader median contract to cr validation testcases
- Add unimplemented Encode(), Decode(), GetMaxEncodingSize(), GetMaxDecodingSize()
- Add ChainReader() method to mock provider for plugin test
- Rename relaymercury.ChainReader to MercuryChainReader, resolve name collisions
- Add tests for errors during ChainReader construction
- Propagate InvalidConfig & any other errors back to client

We should ignore Unsupported until node ops have been given ample time to migrate to the new job spec
(including a section for ChainReader config) so that we can remove the old product-specific
MedianContract component from MedianProvider. All other errors we can immediately start passing back
to the client, letting the core node decide how to handle them (eg. displaying an "invalid job spec"
message to the UI if the RelayConfig was invalid or the ContractID missing)

* Fix lint

* Fix go imports lint

* Fix go.mod require got split into two

Co-authored-by: Jordan Krage <jmank88@gmail.com>

* Run make gomodtidy

* Add tickets to chainreader TODOs

* Log when we're falling back to internal MedianContract

We had this before, at some point it accidentally got dropped

* Pass contractID, relayConfig instead of relayOpts

Also: move test for invalid contract ID to evm_test.go, & add mismatched ChainID
test while we're here since it's pretty similar

* Revert get solana sha workflow to ref develop

Co-authored-by: Jordan Krage <jmank88@gmail.com>

* Update err handling for when chain reader config is missing

* Update solana repo hash

* Improve chain reader config validation tests

* Run go mod tidy

* Update solana repo go mod reference and tidy

* fix logs dir for Solana tests

* fix solana logs path once more

* print absolue log folder location

* get abs folder in a correct way

* do it some other way

* update solana logs dir in github workflow

* Add chain Reader enabled log to NewMedianServices

* Update chainlink-solana ref and tidy

* Add logs and handle disabled chain reader when median loop is enabled

* Update go mod solana repo refs

* Improve error messages in evm chain reader and add test cases

* Update minor comment

* Run tidy

* Remove unneeded chain reader codec methods

* Remove chain reader return values as they are out of scope for this PR

* Add types for the evm

* Create a codec entry with from args, allowing us to do type checking and to have structs to use with abi.Arguments's Pack function

* Craetes a function to get the max size for abi.Argumetns, given any outermost slices have N elements

* Add codec and make chain reader use it.  Run the interface tests for both chain reader and codec.

* Add modifiers to evm chain reader and codec

* Add modifiers to evm chain reader and codec

* small fixes after merging

* Small cleanups and comments

* fix after merge

* add events

* Don't pass ctx around in tests

* Support multiple contract names and mapping them to multiple contracts.

* Upadte chain reader interface

* update go mod

* Fix address type and start the chain reader

* Update go mod references

* add epoch converting

* Simplify chain reader's bindings.

* Add smoke test and feature test for chain reader.

* lint again

* Update solana to point to newer other ones in its own integration tests...

* linter

* .

* newer solana again

* newer solana again...

* Fix bug with unnamed return params and invalid params containing the same name multiple times

* Add generated files to sonar exclude

* Update sonar again

* Methods per sonar suggestions

* Reanme function and type per PR feedback

* Generate abi, bin & wrapper for chainreader test with gethwrappers

* Update generated-wrapper-dependency-versions-do-not-edit.txt

Also: remove extra unused files

* prettier -w ChainReaderTestContract.sol

* Update to use the ErrSliceWrongLen

* Move test contract from contracts/tests to contrats/shared/test

* Fix SQ coverage exclusions

* Use modifier for time

* Update go.mod

* .github/workflows: dedupe log artifact name with tag_suffix

* [TT-792] Add Resources for New OCR2 Tests (#11702)

* Add Resources for New OCR2 Tests

* More power to the engines

* Log everything

* Chill on the power now

* Fix missing container env

* Update common

* Fix broken test

* Add encoder defs for events

* Revert "Add encoder defs for events"

This reverts commit dd99acc.

* revert the change to codec entry for event and simplify how it's done

* Add Marshal method for custom codec types.

* Updated common, still seeing not found, not sure why...

* Update feeds

* Update feeds and common to remove accidental debugging log line

* Update feeds to remove accidental log line and to add more logging to warning messages

* Fixup go mod

* Update feeds

* Remove json tag that had no business being where it was in codec entry

* Fix small lint error, likely detected from better linter in merge from dev

* Add fuzz test for codec

* Support indexed events

* PR feedback, fix typo, clear up comment, revert making a function public

* Refactored function for getting event input.

* --fix and add comments

* use maps instaed fo ToCamelCase to allow consistnacy, also safer if customers have foo and foo_ both as varibles.

* Scope native and checked type conversion to a single struct and add tests to verify that NewAt is safe and fuzz tests for sanity around it too

* Chain Reader config improvements (#11679)

* core/services/job: remove JSONConfig.Bytes hack

* Shorten chain reader cfg abi in median test, change readType string

* minimize abi; pretty formatting; use text marshaller

* integrations-tests/client: use TOML for relayconfig

* minimize chain reader median contract abi in features ocr2 test

* pretty ABI JSON

* add ModifiersConfig wrapper to include type field

* fix TestOCR2TaskJobSpec_String

---------

Co-authored-by: ilija <pavlovicilija42@gmail.com>

* Template must in types gen and small json rename for mods in chain reader def fields

* Update test contract for lint, fix toml test files after renaming the mod in prior commit

* Update common for a rename and fix tests that I broke a couple commits ago

* Move test contract

* Commit with fixed contract for codec test and add small check to if statement for arg len

* small fix to encoder, and fix config in feature test

* Remove debug line since smoke test is passing now and I need to be ready to merge

* Fix bad merge conflict

* small PR fixes

* Couple more error fixes

* update common

* update feeds to point to main

* Fix typo

* Update Solana repo refs

* Pretty chain reader test solitidty file

* Update solana

* Update solana in integation tests

* Needed to update more than just integation tests for solana.

* Update Solana ref to fix relayConfig

* Update Solana ref again

* Update solana and starknet to point to the develop branches.

* update relayer for starknet

---------

Co-authored-by: Domino Valdano <2644901+reductionista@users.noreply.github.com>
Co-authored-by: ilija <pavlovicilija42@gmail.com>
Co-authored-by: ilija42 <57732589+ilija42@users.noreply.github.com>
Co-authored-by: Jordan Krage <jmank88@gmail.com>
Co-authored-by: Bartek Tofel <bartek.tofel@smartcontract.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Adam Hamrick <adam.hamrick@smartcontract.com>
  • Loading branch information
8 people authored Jan 22, 2024
1 parent dbcba88 commit 091a9a3
Show file tree
Hide file tree
Showing 62 changed files with 7,955 additions and 716 deletions.
2 changes: 1 addition & 1 deletion .github/workflows/ci-core.yml
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@ jobs:
strategy:
fail-fast: false
matrix:
cmd: ["go_core_tests", "go_core_race_tests"]
cmd: ["go_core_tests", "go_core_race_tests", "go_core_fuzz"]
name: Core Tests (${{ matrix.cmd }})
runs-on: ubuntu20.04-64cores-256GB
env:
Expand Down
6 changes: 3 additions & 3 deletions .github/workflows/integration-tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -340,12 +340,12 @@ jobs:
file: ocr
pyroscope_env: ci-smoke-ocr-evm-simulated
- name: ocr2
nodes: 4
nodes: 6
os: ubuntu-latest
file: ocr2
pyroscope_env: ci-smoke-ocr2-evm-simulated
- name: ocr2
nodes: 4
nodes: 6
os: ubuntu-latest
pyroscope_env: ci-smoke-ocr2-plugins-evm-simulated
tag_suffix: "-plugins"
Expand Down Expand Up @@ -472,7 +472,7 @@ jobs:
cl_repo: ${{ env.CHAINLINK_IMAGE }}
cl_image_tag: ${{ github.sha }}${{ matrix.product.tag_suffix }}
aws_registries: ${{ secrets.QA_AWS_ACCOUNT_NUMBER }}
artifacts_name: ${{ matrix.product.name }}-test-logs
artifacts_name: ${{ matrix.product.name }}${{ matrix.product.tag_suffix }}-test-logs
artifacts_location: ./integration-tests/smoke/logs/
publish_check_name: ${{ matrix.product.name }}
token: ${{ secrets.GITHUB_TOKEN }}
Expand Down
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -85,3 +85,6 @@ tools/flakeytests/coverage.txt

.test_summary/
.run.id

# Fuzz tests can create these files
**/testdata/fuzz/*
3 changes: 2 additions & 1 deletion common/txmgr/confirmer.go
Original file line number Diff line number Diff line change
Expand Up @@ -14,10 +14,11 @@ import (
"github.com/prometheus/client_golang/prometheus/promauto"
"go.uber.org/multierr"

commonhex "github.com/smartcontractkit/chainlink-common/pkg/utils/hex"

"github.com/smartcontractkit/chainlink-common/pkg/chains/label"
"github.com/smartcontractkit/chainlink-common/pkg/logger"
"github.com/smartcontractkit/chainlink-common/pkg/services"
commonhex "github.com/smartcontractkit/chainlink-common/pkg/utils/hex"
"github.com/smartcontractkit/chainlink-common/pkg/utils/mailbox"

"github.com/smartcontractkit/chainlink/v2/common/client"
Expand Down
1 change: 1 addition & 0 deletions contracts/scripts/native_solc_compile_all_shared
Original file line number Diff line number Diff line change
Expand Up @@ -32,3 +32,4 @@ compileContract shared/token/ERC677/BurnMintERC677.sol
compileContract shared/token/ERC677/LinkToken.sol
compileContract shared/mocks/WERC20Mock.sol
compileContract vendor/openzeppelin-solidity/v4.8.3/contracts/token/ERC20/ERC20.sol
compileContract shared/test/helpers/ChainReaderTestContract.sol
117 changes: 117 additions & 0 deletions contracts/src/v0.8/shared/test/helpers/ChainReaderTestContract.sol
Original file line number Diff line number Diff line change
@@ -0,0 +1,117 @@
// SPDX-License-Identifier: MIT
pragma solidity ^0.8;

struct TestStruct {
int32 Field;
string DifferentField;
uint8 OracleId;
uint8[32] OracleIds;
address Account;
address[] Accounts;
int192 BigField;
MidLevelTestStruct NestedStruct;
}

struct MidLevelTestStruct {
bytes2 FixedBytes;
InnerTestStruct Inner;
}

struct InnerTestStruct {
int64 IntVal;
string S;
}

contract LatestValueHolder {
event Triggered(
int32 indexed field,
string differentField,
uint8 oracleId,
uint8[32] oracleIds,
address Account,
address[] Accounts,
int192 bigField,
MidLevelTestStruct nestedStruct
);

event TriggeredEventWithDynamicTopic(string indexed fieldHash, string field);

// First topic is event hash
event TriggeredWithFourTopics(int32 indexed field1, int32 indexed field2, int32 indexed field3);

TestStruct[] private s_seen;
uint64[] private s_arr;

constructor() {
// See chain_reader_interface_tests.go in chainlink-relay
s_arr.push(3);
s_arr.push(4);
}

function addTestStruct(
int32 field,
string calldata differentField,
uint8 oracleId,
uint8[32] calldata oracleIds,
address account,
address[] calldata accounts,
int192 bigField,
MidLevelTestStruct calldata nestedStruct
) public {
s_seen.push(TestStruct(field, differentField, oracleId, oracleIds, account, accounts, bigField, nestedStruct));
}

function returnSeen(
int32 field,
string calldata differentField,
uint8 oracleId,
uint8[32] calldata oracleIds,
address account,
address[] calldata accounts,
int192 bigField,
MidLevelTestStruct calldata nestedStruct
) public pure returns (TestStruct memory) {
return TestStruct(field, differentField, oracleId, oracleIds, account, accounts, bigField, nestedStruct);
}

function getElementAtIndex(uint256 i) public view returns (TestStruct memory) {
// See chain_reader_interface_tests.go in chainlink-relay
return s_seen[i - 1];
}

function getPrimitiveValue() public pure returns (uint64) {
// See chain_reader_interface_tests.go in chainlink-relay
return 3;
}

function getDifferentPrimitiveValue() public pure returns (uint64) {
// See chain_reader_interface_tests.go in chainlink-relay
return 1990;
}

function getSliceValue() public view returns (uint64[] memory) {
return s_arr;
}

function triggerEvent(
int32 field,
string calldata differentField,
uint8 oracleId,
uint8[32] calldata oracleIds,
address account,
address[] calldata accounts,
int192 bigField,
MidLevelTestStruct calldata nestedStruct
) public {
emit Triggered(field, differentField, oracleId, oracleIds, account, accounts, bigField, nestedStruct);
}

function triggerEventWithDynamicTopic(string calldata field) public {
emit TriggeredEventWithDynamicTopic(field, field);
}

// first topic is the event signature
function triggerWithFourTopics(int32 field1, int32 field2, int32 field3) public {
emit TriggeredWithFourTopics(field1, field2, field3);
}
}

Large diffs are not rendered by default.

Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ batch_blockhash_store: ../../contracts/solc/v0.8.6/BatchBlockhashStore/BatchBloc
batch_vrf_coordinator_v2: ../../contracts/solc/v0.8.6/BatchVRFCoordinatorV2/BatchVRFCoordinatorV2.abi ../../contracts/solc/v0.8.6/BatchVRFCoordinatorV2/BatchVRFCoordinatorV2.bin d0a54963260d8c1f1bbd984b758285e6027cfb5a7e42701bcb562ab123219332
batch_vrf_coordinator_v2plus: ../../contracts/solc/v0.8.6/BatchVRFCoordinatorV2Plus/BatchVRFCoordinatorV2Plus.abi ../../contracts/solc/v0.8.6/BatchVRFCoordinatorV2Plus/BatchVRFCoordinatorV2Plus.bin 7bb76ae241cf1b37b41920830b836cb99f1ad33efd7435ca2398ff6cd2fe5d48
blockhash_store: ../../contracts/solc/v0.8.6/BlockhashStore/BlockhashStore.abi ../../contracts/solc/v0.8.6/BlockhashStore/BlockhashStore.bin 12b0662f1636a341c8863bdec7a20f2ddd97c3a4fd1a7ae353fe316609face4e
chain_reader_example: ../../contracts/solc/v0.8.19/ChainReaderTestContract/LatestValueHolder.abi ../../contracts/solc/v0.8.19/ChainReaderTestContract/LatestValueHolder.bin de88c7e68de36b96aa2bec844bdc96fcd7c9017b38e25062b3b9f9cec42c814f
chain_specific_util_helper: ../../contracts/solc/v0.8.6/ChainSpecificUtilHelper/ChainSpecificUtilHelper.abi ../../contracts/solc/v0.8.6/ChainSpecificUtilHelper/ChainSpecificUtilHelper.bin 5f10664e31abc768f4a37901cae7a3bef90146180f97303e5a1bde5a08d84595
consumer_wrapper: ../../contracts/solc/v0.7/Consumer/Consumer.abi ../../contracts/solc/v0.7/Consumer/Consumer.bin 894d1cbd920dccbd36d92918c1037c6ded34f66f417ccb18ec3f33c64ef83ec5
cron_upkeep_factory_wrapper: ../../contracts/solc/v0.8.6/CronUpkeepFactory/CronUpkeepFactory.abi - dacb0f8cdf54ae9d2781c5e720fc314b32ed5e58eddccff512c75d6067292cd7
Expand Down
4 changes: 3 additions & 1 deletion core/gethwrappers/go_generate.go
Original file line number Diff line number Diff line change
Expand Up @@ -139,7 +139,9 @@ package gethwrappers
//go:generate go run ./generation/generate/wrap.go ../../contracts/solc/v0.8.6/MockAggregatorProxy/MockAggregatorProxy.abi ../../contracts/solc/v0.8.6/MockAggregatorProxy/MockAggregatorProxy.bin MockAggregatorProxy mock_aggregator_proxy

// Log tester
//go:generate go run ./generation/generate/wrap.go ../../contracts/solc/v0.8.19/LogEmitter/LogEmitter.abi ../../contracts/solc/v0.8.19/LogEmitter/LogEmitter.bin LogEmitter log_emitter

// ChainReader test contract
//go:generate go run ./generation/generate/wrap.go ../../contracts/solc/v0.8.19/ChainReaderTestContract/LatestValueHolder.abi ../../contracts/solc/v0.8.19/ChainReaderTestContract/LatestValueHolder.bin LatestValueHolder chain_reader_example

// Chainlink Functions
//go:generate go generate ./functions
Expand Down
Loading

0 comments on commit 091a9a3

Please sign in to comment.