Skip to content

Refactor code generation verification #2108

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
tsachiherman merged 34 commits into from
May 21, 2021
Merged

Refactor code generation verification #2108

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
34 commits
Select commit Hold shift + click to select a range
cdbdb47
try lazy swagger
tsachiherman Apr 26, 2021
c464d56
update
tsachiherman Apr 27, 2021
4cb51fd
constant width hex dump formatting
tsachiherman Apr 27, 2021
ba4487c
remove dependency
tsachiherman Apr 27, 2021
3a5e182
update kmd json
tsachiherman Apr 27, 2021
ef32b2f
update
tsachiherman Apr 27, 2021
7a28526
update
tsachiherman Apr 27, 2021
3a7d29e
Merge branch 'master' into tsachi/lazyswagger
tsachiherman May 18, 2021
0f208f2
remove dependencies when these aren't needed.
tsachiherman May 18, 2021
5428faf
undo unwanted changes.
tsachiherman May 18, 2021
ea808a3
update makefile
tsachiherman May 18, 2021
50966e3
update
tsachiherman May 18, 2021
2c4c863
move build tools to its own directory.
tsachiherman May 19, 2021
ca1c7ef
update go.sum
tsachiherman May 19, 2021
ed770bf
try
tsachiherman May 19, 2021
f8f6014
fix go.sum
tsachiherman May 19, 2021
6cf5409
fix go.mod/sum
tsachiherman May 19, 2021
3e3a9bf
fix go.sum for windows.
tsachiherman May 19, 2021
4c00cb8
swagger fix.
tsachiherman May 19, 2021
ebbacc8
create separate code verification path
tsachiherman May 19, 2021
4627280
add sh
tsachiherman May 19, 2021
0b2a51c
update
tsachiherman May 19, 2021
73a6060
bugfix
tsachiherman May 19, 2021
aca46e0
cleanups
tsachiherman May 19, 2021
03b369e
move fixcheck to pregen
tsachiherman May 19, 2021
37043a5
step
tsachiherman May 19, 2021
8f4d4e6
Move gen step to code verification.
tsachiherman May 20, 2021
6d733f1
cleanup
tsachiherman May 20, 2021
df81c08
Add cleanup of generated directories.
tsachiherman May 20, 2021
f1561da
remove obsolete genesis.json file copy.
tsachiherman May 20, 2021
8b20c95
mark test as flacky
tsachiherman May 20, 2021
d792074
Fix travis output timeout on mac.
tsachiherman May 20, 2021
9c6c511
Avoid using generated genesis.json files.
tsachiherman May 20, 2021
79a17e6
update per reviewer feedback.
tsachiherman May 21, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 0 additions & 6 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,12 +8,6 @@ coverage.html
*.cdv
*.cdv.archive

# swagger
swagger.json
swagger.json.validated
kmdSwaggerWrappers.go
bundledSpecInject.go

# Exclude GoLand files
.idea/

Expand Down
5 changes: 5 additions & 0 deletions .travis.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,11 @@ jobs:
- travis_retry scripts/travis/build_test.sh

- stage: build_pr
os: linux
name: Ubuntu AMD64 CodeGen Verification
script:
- scripts/travis/codegen_verification.sh
- # same stage, parallel job
os: linux
name: Ubuntu AMD64 Build
script:
Expand Down
39 changes: 19 additions & 20 deletions Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -81,9 +81,6 @@ fmt:
fix: build
$(GOPATH1)/bin/algofix */

fixcheck: build
$(GOPATH1)/bin/algofix -error */

lint: deps
$(GOPATH1)/bin/golint ./...

Expand Down Expand Up @@ -148,7 +145,7 @@ $(ALGOD_API_SWAGGER_SPEC): $(ALGOD_API_FILES) crypto/libs/$(OS_TYPE)/$(ARCH)/lib
PATH=$(GOPATH1)/bin:$$PATH \
go generate ./...

$(ALGOD_API_SWAGGER_INJECT): $(ALGOD_API_SWAGGER_SPEC) $(ALGOD_API_SWAGGER_SPEC).validated
$(ALGOD_API_SWAGGER_INJECT): deps $(ALGOD_API_SWAGGER_SPEC) $(ALGOD_API_SWAGGER_SPEC).validated
./daemon/algod/api/server/lib/bundle_swagger_json.sh

# Regenerate kmd swagger spec files
Expand All @@ -175,35 +172,45 @@ $(KMD_API_SWAGGER_SPEC): $(KMD_API_FILES) crypto/libs/$(OS_TYPE)/$(ARCH)/lib/lib
touch $@; \
fi

$(KMD_API_SWAGGER_INJECT): $(KMD_API_SWAGGER_SPEC) $(KMD_API_SWAGGER_SPEC).validated
$(KMD_API_SWAGGER_INJECT): deps $(KMD_API_SWAGGER_SPEC) $(KMD_API_SWAGGER_SPEC).validated
./daemon/kmd/lib/kmdapi/bundle_swagger_json.sh

# generated files we should make sure we clean
GENERATED_FILES := \
$(ALGOD_API_SWAGGER_INJECT) \
$(KMD_API_SWAGGER_INJECT) \
$(ALGOD_API_SWAGGER_SPEC) $(ALGOD_API_SWAGGER_SPEC).validated \
$(KMD_API_SWAGGER_SPEC) $(KMD_API_SWAGGER_SPEC).validated

rebuild_swagger: deps
rm -f $(GENERATED_FILES)
# we need to invoke the make here since we want to ensure that the deletion and re-creating are sequential
make $(KMD_API_SWAGGER_INJECT) $(ALGOD_API_SWAGGER_INJECT)

# develop

build: buildsrc gen
build: buildsrc

# We're making an empty file in the go-cache dir to
# get around a bug in go build where it will fail
# to cache binaries from time to time on empty NFS
# dirs
buildsrc: check-go-version crypto/libs/$(OS_TYPE)/$(ARCH)/lib/libsodium.a node_exporter NONGO_BIN deps $(ALGOD_API_SWAGGER_INJECT) $(KMD_API_SWAGGER_INJECT)
buildsrc: check-go-version crypto/libs/$(OS_TYPE)/$(ARCH)/lib/libsodium.a node_exporter NONGO_BIN
mkdir -p tmp/go-cache && \
touch tmp/go-cache/file.txt && \
GOCACHE=$(SRCPATH)/tmp/go-cache go install $(GOTRIMPATH) $(GOTAGS) $(GOBUILDMODE) -ldflags="$(GOLDFLAGS)" ./...

check-go-version:
./scripts/check_golang_version.sh build

SOURCES_RACE := github.com/algorand/go-algorand/cmd/kmd

## Build binaries with the race detector enabled in them.
## This allows us to run e2e tests with race detection.
## We overwrite bin-race/kmd with a non -race version due to
## the incredible performance impact of -race on Scrypt.
build-race: build
@mkdir -p $(GOPATH1)/bin-race
GOBIN=$(GOPATH1)/bin-race go install $(GOTRIMPATH) $(GOTAGS) -race -ldflags="$(GOLDFLAGS)" ./...
GOBIN=$(GOPATH1)/bin-race go install $(GOTRIMPATH) $(GOTAGS) -ldflags="$(GOLDFLAGS)" $(SOURCES_RACE)
cp $(GOPATH1)/bin/kmd $(GOPATH1)/bin-race

NONGO_BIN_FILES=$(GOPATH1)/bin/find-nodes.sh $(GOPATH1)/bin/update.sh $(GOPATH1)/bin/COPYING $(GOPATH1)/bin/ddconfig.sh

Expand Down Expand Up @@ -238,28 +245,20 @@ integration: build-race

testall: fulltest integration

# generated files we should make sure we clean
GENERATED_FILES := daemon/algod/api/bundledSpecInject.go \
daemon/algod/api/lib/bundledSpecInject.go \
daemon/kmd/lib/kmdapi/bundledSpecInject.go \
$(ALGOD_API_SWAGGER_SPEC) $(ALGOD_API_SWAGGER_SPEC).validated \
$(KMD_API_SWAGGER_SPEC) $(KMD_API_SWAGGER_SPEC).validated

clean:
go clean -i ./...
rm -f $(GOPATH1)/bin/node_exporter
rm -f $(GENERATED_FILES)
Copy link
Contributor

@algonautshant algonautshant May 20, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will it be useful to add a cleanall option which includes generated_files?

Copy link
Contributor Author

@tsachiherman tsachiherman May 20, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think so. That would cause more issues, since cleaning them without regenerating them would cause build failures ( keep in mind that I've removed the dependencies on these, since they're alway there ).
That's the reason I added rebuild_swagger target.

cd crypto/libsodium-fork && \
test ! -e Makefile || make clean
rm -rf crypto/lib
rm -rf crypto/libs
rm -rf crypto/copies
rm -rf ./gen/devnet ./gen/mainnetnet ./gen/testnet

# clean without crypto
cleango:
go clean -i ./...
rm -f $(GOPATH1)/bin/node_exporter
rm -f $(GENERATED_FILES)

# assign the phony target node_exporter the dependency of the actual executable.
node_exporter: $(GOPATH1)/bin/node_exporter
Expand Down Expand Up @@ -312,7 +311,7 @@ dump: $(addprefix gen/,$(addsuffix /genesis.dump, $(NETWORKS)))
install: build
scripts/dev_install.sh -p $(GOPATH1)/bin

.PHONY: default fmt vet lint check_shell sanity cover prof deps build test fulltest shorttest clean cleango deploy node_exporter install %gen gen NONGO_BIN check-go-version
.PHONY: default fmt vet lint check_shell sanity cover prof deps build test fulltest shorttest clean cleango deploy node_exporter install %gen gen NONGO_BIN check-go-version rebuild_swagger

###### TARGETS FOR CICD PROCESS ######
include ./scripts/release/mule/Makefile.mule
Expand Down
2 changes: 1 addition & 1 deletion auction/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ Each step will first be explained in prose, followed by the console commands. Wh
## Set up `algod`
Create a data directory (here, called `xx`). Copy the genesis block, `genesis.json`, to the data directory. Start an algorand node using that data directory.
- `mkdir xx`
- `cp gen/devnet/genesis.json xx`
- `cp installer/genesis/devnet/genesis.json xx`
- `algod -d xx`

## Set up `auctionbank`
Expand Down
2 changes: 2 additions & 0 deletions catchup/service_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -421,6 +421,8 @@ func TestServiceFetchBlocksMalformed(t *testing.T) {
}

func TestOnSwitchToUnSupportedProtocol(t *testing.T) {
t.Skip("This test is flacky and need to be fixed.")
Copy link
Contributor

@algonautshant algonautshant May 20, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we have a ticket to follow up on this?

Copy link
Contributor Author

@tsachiherman tsachiherman May 20, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will create a follow up ticket for this one, once this PR is approved and merged in.

Copy link
Contributor

@winder winder May 20, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are a bunch of these, I think there's an umbrella ticket to fix:

=== Skipped
=== SKIP: agreement TestCryptoVerifierBuffers (0.00s)
    cryptoVerifier_test.go:139: Test is flaky

=== SKIP: agreement/gossip TestNetworkImplFullStackQuick (0.00s)
    networkFull_test.go:136: 

=== SKIP: config TestConfigMigrate (0.00s)
    config_test.go:247: 

=== SKIP: daemon/algod TestFirstListenerSetupGetsPort8080WhenPassedPortZero (0.00s)
    server_test.go:43: Cannot run this test since port 8080 is already in use.

=== SKIP: gen TestLoadMultiRootKeyConcurrent (0.00s)
    generate_test.go:37: 

=== SKIP: gen TestLoadSingleRootKeyConcurrent (0.00s)
    generate_test.go:78: 

=== SKIP: ledger TestArchival (0.00s)
    archival_test.go:127: Skipping the TestArchival as it tend to randomally fail on travis linux-amd64

=== SKIP: ledger TestLedgerMemoryLeak (0.00s)
    ledger_test.go:1587: 

=== SKIP: ledger TestRandomizedEncodingcatchpointFileBalancesChunk (0.00s)
    codec_tester.go:329: 

=== SKIP: ledger TestRandomizedEncodingencodedBalanceRecord (0.00s)
    codec_tester.go:329: 

=== SKIP: logging TestAsyncTelemetryHook_Close (0.00s)
    telemetryhook_test.go:174: We no longer ensure 100% delivery. To not block, we drop messages when they come in faster than the network sends them.

=== SKIP: logging TestAsyncTelemetryHook_QueueDepth (0.00s)
    telemetryhook_test.go:199: flakey test can fail on slow test systems

=== SKIP: node TestSyncingFullNode (0.00s)
    node_test.go:180: This is failing randomly again - PLEASE FIX!

=== SKIP: node TestInitialSync (0.00s)
    node_test.go:237: flaky TestInitialSync 

=== SKIP: node TestSimpleUpgrade (0.00s)
    node_test.go:270: Randomly failing: node_test.go:~330 : no block notification for account. Re-enable after agreement bug-fix pass

=== SKIP: rpcs TestStartAndStop (0.00s)
    txSyncer_test.go:263: TODO: replace this test in new client paradigm

=== SKIP: tools/network TestRealNamesWithResolver (0.00s)
    resolveController_test.go:76: 

=== SKIP: tools/network/dnssec TestDeadNS (0.00s)
    dnssec_test.go:315: 

=== SKIP: tools/network/dnssec TestRealRequests (0.00s)
    dnssec_test.go:332:

Copy link
Contributor Author

@tsachiherman tsachiherman May 21, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, I'm aware of that.. The important thing is that we need to keep tracking these explicitly so that we won't forget these.


// Test the interruption in the initial loop
// This cannot happen in practice, but is used to test the code.
{
Expand Down
4 changes: 2 additions & 2 deletions daemon/algod/api/server/lib/bundle_swagger_json.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,12 +3,12 @@
THISDIR=$(dirname $0)

cat <<EOM | gofmt > $THISDIR/bundledSpecInject.go
// Code generated during build process, along with swagger.json. DO NOT EDIT.
// Code generated by bundle_swagger_json.sh, along with swagger.json. DO NOT EDIT.
package lib

func init() {
SwaggerSpecJSON = string([]byte{
$(cat $THISDIR/../../swagger.json | hexdump -v -e '1/1 "0x%02X, "' | fmt)
$(cat $THISDIR/../../swagger.json | hexdump -v -e '1/1 "0x%02X, "' | fmt -w 100)
})
}

Expand Down
Loading