Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Timeout Issues stemming from Puma persistent_timeout default setting #26

Closed
3 of 4 tasks
ghost opened this issue Jun 24, 2020 · 3 comments
Closed
3 of 4 tasks

Timeout Issues stemming from Puma persistent_timeout default setting #26

ghost opened this issue Jun 24, 2020 · 3 comments

Comments

@ghost
Copy link

ghost commented Jun 24, 2020

Pre issue-raising checklist

I have already (please mark the applicable with an x):

  • Confirmed this is the right place to raise the issue - only issues related to the Dockerization of the Pact Broker should be raised here. Issues related to the Pact Broker application itself should be raised in the Pact Broker project.
  • Upgraded to the latest Pact Broker Docker image OR
  • Checked the CHANGELOG to see if the issue I am about to raise has been fixed
  • Read the Troubleshooting page

Software versions

  • pact-broker gem version: eg 2.57.0
  • pact-broker docker version: 2.57.0.0
  • OS: N/A
  • pact broker client details: pact-js v9.11.0

Expected behaviour

Running Pact verification tests should successfully publish results to the Pact Broker.

Actual behaviour

We currently run this Docker image on AWS behind an ALB, which has an idle timeout. This Docker image makes use of Puma, which has a default keep alive of 20 seconds. We have seen issues where if verification tests take longer than 20 seconds, we see errors coming from the load balancer that stem from the fact that the connection between the load balancer and the Pact Broker has been closed, leading to the ALB to return a 504 status code. This leads to errors in our CI server since the Pact binary is unable to publish verification results to the Pact Broker.

Steps to reproduce

Unfortunately the source code is for a closed-source application. Issue seems to stem from running verification tests lasting longer than 20 seconds. Should be able to reproduce with a simple delay of longer than 20 seconds.

Relevent log files

Logs from verification test run:

/usr/src/app/node_modules/@pact-foundation/pact-node/standalone/linux-x64-1.84.0/pact/lib/vendor/ruby/2.2.0/gems/json-2.3.0/lib/json/common.rb:156:in `parse': 757: unexpected token at '<html>
 (JSON::ParserError)
<head><title>504 Gateway Time-out</title></head>
<body bgcolor="white">
<center><h1>504 Gateway Time-out</h1></center>
</body>
</html>
'

	from /usr/src/app/node_modules/@pact-foundation/pact-node/standalone/linux-x64-1.84.0/pact/lib/vendor/ruby/2.2.0/gems/json-2.3.0/lib/json/common.rb:156:in `parse'
	from /usr/src/app/node_modules/@pact-foundation/pact-node/standalone/linux-x64-1.84.0/pact/lib/vendor/ruby/2.2.0/gems/pact-1.49.3/lib/pact/hal/http_client.rb:70:in `body'
	from /usr/src/app/node_modules/@pact-foundation/pact-node/standalone/linux-x64-1.84.0/pact/lib/vendor/ruby/2.2.0/gems/pact-1.49.3/lib/pact/provider/verification_results/publish.rb:103:in `publish_verification_results'
	from /usr/src/app/node_modules/@pact-foundation/pact-node/standalone/linux-x64-1.84.0/pact/lib/vendor/ruby/2.2.0/gems/pact-1.49.3/lib/pact/provider/verification_results/publish.rb:35:in `call'
	from /usr/src/app/node_modules/@pact-foundation/pact-node/standalone/linux-x64-1.84.0/pact/lib/vendor/ruby/2.2.0/gems/pact-1.49.3/lib/pact/provider/verification_results/publish.rb:21:in `call'
	from /usr/src/app/node_modules/@pact-foundation/pact-node/standalone/linux-x64-1.84.0/pact/lib/vendor/ruby/2.2.0/gems/pact-1.49.3/lib/pact/provider/verification_results/publish_all.rb:22:in `block in call'
	from /usr/src/app/node_modules/@pact-foundation/pact-node/standalone/linux-x64-1.84.0/pact/lib/vendor/ruby/2.2.0/gems/pact-1.49.3/lib/pact/provider/verification_results/publish_all.rb:19:in `collect'
	from /usr/src/app/node_modules/@pact-foundation/pact-node/standalone/linux-x64-1.84.0/pact/lib/vendor/ruby/2.2.0/gems/pact-1.49.3/lib/pact/provider/verification_results/publish_all.rb:19:in `call'
	from /usr/src/app/node_modules/@pact-foundation/pact-node/standalone/linux-x64-1.84.0/pact/lib/vendor/ruby/2.2.0/gems/pact-1.49.3/lib/pact/provider/verification_results/publish_all.rb:10:in `call'
	from /usr/src/app/node_modules/@pact-foundation/pact-node/standalone/linux-x64-1.84.0/pact/lib/vendor/ruby/2.2.0/gems/pact-1.49.3/lib/pact/provider/rspec/pact_broker_formatter.rb:28:in `close'
	from /usr/src/app/node_modules/@pact-foundation/pact-node/standalone/linux-x64-1.84.0/pact/lib/vendor/ruby/2.2.0/gems/rspec-core-3.9.1/lib/rspec/core/reporter.rb:209:in `block in notify'
	from /usr/src/app/node_modules/@pact-foundation/pact-node/standalone/linux-x64-1.84.0/pact/lib/vendor/ruby/2.2.0/gems/rspec-core-3.9.1/lib/rspec/core/reporter.rb:208:in `each'
	from /usr/src/app/node_modules/@pact-foundation/pact-node/standalone/linux-x64-1.84.0/pact/lib/vendor/ruby/2.2.0/gems/rspec-core-3.9.1/lib/rspec/core/reporter.rb:208:in `notify'
	from /usr/src/app/node_modules/@pact-foundation/pact-node/standalone/linux-x64-1.84.0/pact/lib/vendor/ruby/2.2.0/gems/rspec-core-3.9.1/lib/rspec/core/reporter.rb:243:in `close'
	from /usr/src/app/node_modules/@pact-foundation/pact-node/standalone/linux-x64-1.84.0/pact/lib/vendor/ruby/2.2.0/gems/rspec-core-3.9.1/lib/rspec/core/reporter.rb:196:in `close_after'
	from /usr/src/app/node_modules/@pact-foundation/pact-node/standalone/linux-x64-1.84.0/pact/lib/vendor/ruby/2.2.0/gems/rspec-core-3.9.1/lib/rspec/core/reporter.rb:174:in `finish'
	from /usr/src/app/node_modules/@pact-foundation/pact-node/standalone/linux-x64-1.84.0/pact/lib/vendor/ruby/2.2.0/gems/rspec-core-3.9.1/lib/rspec/core/reporter.rb:76:in `report'
	from /usr/src/app/node_modules/@pact-foundation/pact-node/standalone/linux-x64-1.84.0/pact/lib/vendor/ruby/2.2.0/gems/rspec-core-3.9.1/lib/rspec/core/runner.rb:115:in `run_specs'
	from /usr/src/app/node_modules/@pact-foundation/pact-node/standalone/linux-x64-1.84.0/pact/lib/vendor/ruby/2.2.0/gems/rspec-core-3.9.1/lib/rspec/core/runner.rb:89:in `run'
	from /usr/src/app/node_modules/@pact-foundation/pact-node/standalone/linux-x64-1.84.0/pact/lib/vendor/ruby/2.2.0/gems/rspec-core-3.9.1/lib/rspec/core/runner.rb:71:in `run'
	from /usr/src/app/node_modules/@pact-foundation/pact-node/standalone/linux-x64-1.84.0/pact/lib/vendor/ruby/2.2.0/gems/pact-1.49.3/lib/pact/provider/pact_spec_runner.rb:88:in `run_specs'
	from /usr/src/app/node_modules/@pact-foundation/pact-node/standalone/linux-x64-1.84.0/pact/lib/vendor/ruby/2.2.0/gems/pact-1.49.3/lib/pact/provider/pact_spec_runner.rb:34:in `run'
	from /usr/src/app/node_modules/@pact-foundation/pact-node/standalone/linux-x64-1.84.0/pact/lib/vendor/ruby/2.2.0/gems/pact-1.49.3/lib/pact/cli/run_pact_verification.rb:69:in `run_with_pact_uri_object'
	from /usr/src/app/node_modules/@pact-foundation/pact-node/standalone/linux-x64-1.84.0/pact/lib/vendor/ruby/2.2.0/gems/pact-1.49.3/lib/pact/cli/run_pact_verification.rb:50:in `run_specs'
	from /usr/src/app/node_modules/@pact-foundation/pact-node/standalone/linux-x64-1.84.0/pact/lib/vendor/ruby/2.2.0/gems/pact-1.49.3/lib/pact/cli/run_pact_verification.rb:21:in `call'
	from /usr/src/app/node_modules/@pact-foundation/pact-node/standalone/linux-x64-1.84.0/pact/lib/vendor/ruby/2.2.0/gems/pact-1.49.3/lib/pact/cli/run_pact_verification.rb:13:in `call'
	from /usr/src/app/node_modules/@pact-foundation/pact-node/standalone/linux-x64-1.84.0/pact/lib/vendor/ruby/2.2.0/gems/pact-provider-verifier-1.30.1/lib/pact/provider_verifier/app.rb:169:in `verify_pact'
	from /usr/src/app/node_modules/@pact-foundation/pact-node/standalone/linux-x64-1.84.0/pact/lib/vendor/ruby/2.2.0/gems/pact-provider-verifier-1.30.1/lib/pact/provider_verifier/app.rb:43:in `block in call'
	from /usr/src/app/node_modules/@pact-foundation/pact-node/standalone/linux-x64-1.84.0/pact/lib/vendor/ruby/2.2.0/gems/pact-provider-verifier-1.30.1/lib/pact/provider_verifier/app.rb:42:in `collect'
	from /usr/src/app/node_modules/@pact-foundation/pact-node/standalone/linux-x64-1.84.0/pact/lib/vendor/ruby/2.2.0/gems/pact-provider-verifier-1.30.1/lib/pact/provider_verifier/app.rb:42:in `call'
	from /usr/src/app/node_modules/@pact-foundation/pact-node/standalone/linux-x64-1.84.0/pact/lib/vendor/ruby/2.2.0/gems/pact-provider-verifier-1.30.1/lib/pact/provider_verifier/app.rb:34:in `call'
	from /usr/src/app/node_modules/@pact-foundation/pact-node/standalone/linux-x64-1.84.0/pact/lib/vendor/ruby/2.2.0/gems/pact-provider-verifier-1.30.1/lib/pact/provider_verifier/cli/verify.rb:56:in `verify'
	from /usr/src/app/node_modules/@pact-foundation/pact-node/standalone/linux-x64-1.84.0/pact/lib/vendor/ruby/2.2.0/gems/thor-0.20.3/lib/thor/command.rb:27:in `run'
	from /usr/src/app/node_modules/@pact-foundation/pact-node/standalone/linux-x64-1.84.0/pact/lib/vendor/ruby/2.2.0/gems/thor-0.20.3/lib/thor/invocation.rb:126:in `invoke_command'
	from /usr/src/app/node_modules/@pact-foundation/pact-node/standalone/linux-x64-1.84.0/pact/lib/vendor/ruby/2.2.0/gems/thor-0.20.3/lib/thor.rb:387:in `dispatch'
	from /usr/src/app/node_modules/@pact-foundation/pact-node/standalone/linux-x64-1.84.0/pact/lib/vendor/ruby/2.2.0/gems/thor-0.20.3/lib/thor/base.rb:466:in `start'
	from /usr/src/app/node_modules/@pact-foundation/pact-node/standalone/linux-x64-1.84.0/pact/lib/vendor/ruby/2.2.0/gems/pact-provider-verifier-1.30.1/lib/pact/provider_verifier/cli/custom_thor.rb:17:in `start'
	from /usr/src/app/node_modules/@pact-foundation/pact-node/standalone/linux-x64-1.84.0/pact/lib/app/pact-provider-verifier.rb:33:in `<main>'
]

      at internal/process/task_queues.js:97:5

Would it be possible to expose the relevant Puma configuration for PERSISTENT_TIMEOUT via some configuration within this Dockerfile?
@bethesque
Copy link
Member

I haven't tested this myself, but I've made this change to the edge branch: 1e51b9a

Grab pactfoundation/pact-broker:edge and see if that fixes your issue. If it does, I'll put it on master.

@ghost
Copy link
Author

ghost commented Jun 25, 2020

@bethesque - That seems to have done the trick! Thank you so much for the quick turnaround time. It is most appreciated.

@bethesque
Copy link
Member

This has been released in tag 2.58.0.0

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@bethesque