Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

VM testnet tx-nodes & blockscout #687

Merged
merged 53 commits into from
Aug 22, 2019
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
53 commits
Select commit Hold shift + click to select a range
661e63f
VM testnet works, celotool initial and destroy works
tkporter Aug 6, 2019
079e3e0
Make names of resources specific to a celo_env
tkporter Aug 6, 2019
e0d23dc
Remove hardcoded vars for bootnode
tkporter Aug 6, 2019
607b991
Add small READMEs to terraform modules
tkporter Aug 6, 2019
e1935c6
Fix failing ci-lint
tkporter Aug 8, 2019
9b60aff
Specify kube-hosted ethstats in vm testnet
tkporter Aug 12, 2019
6bb0fc9
Make names of resources specific to a celo_env
tkporter Aug 6, 2019
523bb94
Remove hardcoded vars for bootnode
tkporter Aug 6, 2019
996b96e
Generate and upload vm testnet node secrets to google cloud
tkporter Aug 12, 2019
baeada5
Refactor to separate terraform logic from vm testnet logic
tkporter Aug 12, 2019
de2065f
Move toward using secrets via google storage
tkporter Aug 12, 2019
0446c52
Move terraform validators to use secrets from gcloud storage
tkporter Aug 13, 2019
387aac9
Make bootnode use cloud storage secrets
tkporter Aug 13, 2019
225fb7e
Go back to a load test bootnode key
tkporter Aug 13, 2019
17d72b1
Work on credentials
tkporter Aug 13, 2019
54fb33f
Add a service account for VMs whose permissions are limited
tkporter Aug 14, 2019
89ec2ae
Use correct ethstats secret
tkporter Aug 14, 2019
2525185
Merge master
tkporter Aug 14, 2019
6fda0e8
Fix unresolved merge conflict
tkporter Aug 14, 2019
99cacf4
Use correct ethstats secret
tkporter Aug 14, 2019
ac6cef2
Clean up
tkporter Aug 14, 2019
6af8e9e
Merge branch 'master' into trevor/vm-testnet-secrets
tkporter Aug 14, 2019
99f477d
Merge branch 'master' into trevor/vm-testnet-secrets
tkporter Aug 14, 2019
fb551b1
Merge branch 'master' into trevor/vm-testnet-secrets
tkporter Aug 14, 2019
b0a3aa1
Remove note in comment
tkporter Aug 15, 2019
e8082ec
Merge branch 'trevor/vm-testnet-secrets' of github.com:celo-org/celo-…
tkporter Aug 15, 2019
43a0b87
Merge branch 'master' into trevor/vm-testnet-secrets
tkporter Aug 15, 2019
ad59ded
Merge branch 'master' of github.com:celo-org/celo-monorepo into trevo…
tkporter Aug 16, 2019
ca5aa20
Add tx nodes
tkporter Aug 19, 2019
c9f8e91
Working internal tx-node load balancer
tkporter Aug 20, 2019
4b3dc39
Run terraform fmt
tkporter Aug 20, 2019
053d827
Set up blockscout
tkporter Aug 20, 2019
a6079ff
Blockscout working with VPC network destruction safety
tkporter Aug 21, 2019
009e5a3
Merge branch 'master' of github.com:celo-org/celo-monorepo into trevo…
tkporter Aug 21, 2019
417d91d
Minor fixes
tkporter Aug 21, 2019
b42201d
Terraform fmt
tkporter Aug 21, 2019
f05ff9a
Add env-utils.ts
tkporter Aug 21, 2019
c9c6659
More refactor
tkporter Aug 21, 2019
b06148f
Some more refactor
tkporter Aug 21, 2019
5bd70c0
Get rid of error
tkporter Aug 21, 2019
8121915
Get blockscout celotool working
tkporter Aug 21, 2019
b7bae58
Fix tsc build error
tkporter Aug 21, 2019
ba391b3
Clean up
tkporter Aug 21, 2019
595b034
Only use non default network for vm testnets
tkporter Aug 21, 2019
05dc5df
Merge branch 'master' of github.com:celo-org/celo-monorepo into trevo…
tkporter Aug 21, 2019
c182d38
Fix lint errors
tkporter Aug 21, 2019
d0d1032
Merge branch 'master' into trevor/vm-testnet-tx-nodes
tkporter Aug 22, 2019
7e9fbfa
Merge branch 'master' into trevor/vm-testnet-tx-nodes
tkporter Aug 22, 2019
161f69f
Hopefully fix import path error
tkporter Aug 22, 2019
69ad8e3
Merge branch 'master' of github.com:celo-org/celo-monorepo into trevo…
tkporter Aug 22, 2019
4a3d92a
Merge branch 'trevor/vm-testnet-tx-nodes' of github.com:celo-org/celo…
tkporter Aug 22, 2019
5a5d602
Fix create before destroy issue
tkporter Aug 22, 2019
348ba89
Merge branch 'master' into trevor/vm-testnet-tx-nodes
tkporter Aug 22, 2019
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Clean up
  • Loading branch information
tkporter committed Aug 21, 2019
commit ba391b3f663422d310f235ab0426959eb4c20141
6 changes: 4 additions & 2 deletions packages/celotool/src/cmds/deploy/upgrade/blockscout.ts
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ import sleep from 'sleep-promise'
import { installHelmChart, removeHelmRelease, upgradeHelmChart } from 'src/lib/blockscout'
import { switchToClusterFromEnv } from 'src/lib/cluster'
import { resetCloudSQLInstance, retrieveCloudSQLConnectionInfo } from 'src/lib/helm_deploy'
import { execCmd } from 'src/lib/utils'
import { execCmdWithExitOnFailure } from 'src/lib/utils'
import * as yargs from 'yargs'

export const command = 'blockscout'
Expand Down Expand Up @@ -53,7 +53,9 @@ export const handler = async (argv: BlockscoutUpgradeArgv) => {
} else {
console.info(`Delete blockscout-migration`)
try {
await execCmd(`kubectl delete job ${argv.celoEnv}-blockscout-migration -n ${argv.celoEnv}`)
await execCmdWithExitOnFailure(
`kubectl delete job ${argv.celoEnv}-blockscout-migration -n ${argv.celoEnv}`
)
} catch (error) {
console.error(error)
}
Expand Down
39 changes: 12 additions & 27 deletions packages/celotool/src/lib/terraform.ts
Original file line number Diff line number Diff line change
Expand Up @@ -15,8 +15,6 @@ export async function initTerraformModule(
vars: TerraformVars,
backendConfigVars: TerraformVars
) {
console.info('Resetting local state to force pull remote backend...')
await resetLocalTerraformState()
const modulePath = getModulePath(moduleName)
return execTerraformCmd(
'init',
Expand Down Expand Up @@ -57,6 +55,11 @@ export function destroyTerraformModule(moduleName: string, vars: TerraformVars)
return execTerraformCmd('destroy', getModulePath(moduleName), getVarOptions(vars), '-force')
}

// pulls remote state
function refreshTerraformModule(moduleName: string, vars: TerraformVars) {
return execTerraformCmd('refresh', getModulePath(moduleName), getVarOptions(vars))
}

// This is a workaround to `terraform output` requiring users to run it from
// inside the Terraform module directory. See:
// https://github.com/hashicorp/terraform/issues/15581
Expand All @@ -66,29 +69,11 @@ export async function getTerraformModuleOutputs(
vars: TerraformVars,
backendConfigVars: TerraformVars
) {
// console.info('Resetting local state to force pull remote backend...')
// await resetLocalTerraformState()

const initCmd = buildTerraformCmd(
'init',
getModulePath(moduleName),
getVarOptions(vars),
getVarOptions(backendConfigVars, 'backend-config')
)

const refreshCmd = buildTerraformCmd('refresh', getModulePath(moduleName), getVarOptions(vars))
const [stdout] = await execCmd(`
cd ${getModulePath(moduleName)} && \
${initCmd} > /dev/null 2>&1 && \
${refreshCmd} > /dev/null 2>&1 && \
terraform output -json
`)
console.log(stdout)
return JSON.parse(stdout)
}

function resetLocalTerraformState() {
// return execCmd('rm -f ./.terraform/terraform.tfstate')
await initTerraformModule(moduleName, vars, backendConfigVars)
await refreshTerraformModule(moduleName, vars)
const modulePath = getModulePath(moduleName)
const [output] = await execCmd(`cd ${modulePath} && terraform output -json`)
return JSON.parse(output)
}

function getModulePath(moduleName: string) {
Expand Down Expand Up @@ -122,7 +107,7 @@ function execTerraformCmd(
return execCmd(`cd ${modulePath} && ${terraformCmd}`, {}, false, true)
}

function buildTerraformCmd(command: string, actionPath: string, ...options: string[]) {
function buildTerraformCmd(command: string, cmdPath: string, ...options: string[]) {
const optionsStr = options ? options.join(' ') : ''
return `terraform ${command} -input=false ${optionsStr} ${actionPath}`
return `terraform ${command} -input=false ${optionsStr} ${cmdPath}`
}
14 changes: 7 additions & 7 deletions packages/celotool/src/lib/vm-testnet-utils.ts
Original file line number Diff line number Diff line change
Expand Up @@ -167,6 +167,13 @@ function getTestnetVars(celoEnv: string) {
}
}

function getTestnetNetworkVars(celoEnv: string) {
return {
...getEnvVarValues(testnetNetworkEnvVars),
network_name: networkName(celoEnv),
}
}

function getEnvVarValues(terraformEnvVars: TerraformVars) {
const vars: { [key: string]: string } = {}
for (const key of Object.keys(terraformEnvVars)) {
Expand All @@ -175,13 +182,6 @@ function getEnvVarValues(terraformEnvVars: TerraformVars) {
return vars
}

function getTestnetNetworkVars(celoEnv: string) {
return {
...getEnvVarValues(testnetNetworkEnvVars),
network_name: networkName(celoEnv),
}
}

export async function generateAndUploadSecrets(celoEnv: string) {
// Bootnode
const bootnodeSecrets = generateBootnodeSecretEnvVars()
Expand Down