fix(neard): fixed failed CI tests due to features mismatches

[miraclx]

We're sort of in a weird spot as it is, we have features within nearcore that are required but can't be enabled if they depend on crates listed under dev-dependencies (bug in cargo: rust-lang/cargo#6915). Temporarily listing them under dependencies, fixed the CI testing failures we had while working on #4292 (same error @ailisp highlighted: #4333 (comment)), but as pointed out in #4331 (comment), this method should be out of the question. If we remove it, however, we can't work with the tests that depend on those features.

This PR moves the previously top-level tests into a new crate to have better dependency and feature handling as @matklad suggested

• move tests from /test folder somewhere -- either to some of the existing packages (neard perhaps?) or into a new integration-tests package

This would ensure we have dependencies like testlib, runtime-params-estimator and restored-receipts-verifier that are only needed for testing and depend on nearcore without cyclic dependency issues while having all features relevant to testing in order.

The features within neard have been reduced to proxies to nearcore features, which partially resolves #4325 (though the intent for that has morphed a bit). This also makes for a cleaner dependency graph from neard perspective, removing extraneous dependencies that were previously required.

I also noticed removed the old_tests feature that should've been removed along with the rest of it in #928.

Updated some docs and code comments referencing the old neard path too.

[miraclx]

All required CI checks pass so far ✔️ except the untested runtime-params-estimator which hasn't passed in a while anyways, I see the issues with this are as follows:

• First, we add the rustfmt component for a missing toolchain

  nearcore/runtime/runtime-params-estimator/emu-cost/Dockerfile

  Line 12 in 4f300bb

  RUN rustup component add rustfmt --toolchain 1.51.0-x86_64-unknown-linux-gnu

  I'm not sure what toolchain is installed within the docker image but is this line even necessary?

• Next, we need to properly escape this

  nearcore/runtime/runtime-params-estimator/emu-cost/ci.sh

  Line 17 in 4f300bb

  $(cd test-contract && ./build.sh)

  else bash does the cmd substitution outside of the nested bash invocation.

[chefsale]

The only thing I believe we can remove performance stats feature flag from all the binaries except neard in the makefile like I proposed in #4338 @bowenwang1996 @frol

[ailisp]

we probably should check these build in .buildkite/pipeline.yml:

cargo check -p state-viewer --features nearcore/performance_stats,nearcore/memory_stats
cargo check -p store-validator --features nearcore/performance_stats,nearcore/memory_stats

because existing check doesn't check these feature combinations

[miraclx]

Aren't these covered by this?

nearcore/.buildkite/pipeline.yml

Line 30 in 928f15d

RUSTFLAGS='-D warnings' cargo check --workspace --all-targets --all-features

And @chefsale, what do you think? Regarding your remark in #4338 (comment), if those lines aren't necessary, checks for them might not as well.

[ailisp]

No it doesn't because there's no workspace features, see this two commits' buildkite:
1a1256c
08e7e31

So the first one only checks

       RUSTFLAGS='-D warnings' cargo check --workspace --all-targets --all-features 

It passed. The second commit add a cargo check -p neard --features sandbox feature and it failed. This means sandbox feature of neard was not checked by cargo check --workspace --all-targets --all-features

[ailisp]

@miraclx I think you're correct, all-features do include all features, mine case was failing because it's missing some features instead of --all-features didn't work. So the ci doesn't check all feature combinations. cargo check --workspace --all-targets --all-features ensures when all features are included, all packages can compile. But real build includes package build with no feature, or a subset of features, and they can fail if feature dependency wasn't correctly specified.

So for example this one from Makefile

cargo build -p state-viewer --features nearcore/performance_stats,nearcore/memory_stats

since we don't do make on ci, it makes sense to cargo check these compile, not introduced by your PR, but a thing we probably should do to gurantee make works on master commits.

[ailisp]

Same check in buildkite here

[bowenwang1996]

why do we need all these features?

[miraclx]

Before now, lines like these enabled features that prior to the decoupling, existed in the workspace.

nearcore/Makefile

Line 37 in 928f15d

cargo $(RUST_OPTIONS) build -p near-vm-runner-standalone --release --features nightly_protocol,nightly_protocol_features,performance_stats,memory_stats

Now those features are defined in nearcore. This crate, however, doesn't directly depend on nearcore. So we have to adapt the needed features to directly match the available dependencies.

[matklad]

Added auto-merge -- feels like the faster we do this, the better )