Problems when running one-off dynos

[cabello]

The agent does not start properly and all tracing calls fail and report unable to connect to localhost port 7777.

Is there a way to tell a one-off dyno apart from other regular services?

This is the output that I get when running heroku run console -a my-app:

2017-03-08 23:49:24,869 | ERROR | dd.dogstatsd | utils.checkfiles(checkfiles.py:21) | Couldn't find the check configuration folder, this shouldn't happen.
2017-03-08 23:49:24,870 | ERROR | dd.dogstatsd | utils.dockerutil(dockerutil.py:116) | No instance was found in the docker check configuration. Docker related collection will not work.
2017-03-08 23:49:24,876 | INFO | dd.dogstatsd | utils.pidfile(pidfile.py:35) | Pid file is: /app/.apt/opt/datadog-agent/run/dogstatsd.pid
2017-03-08 23:49:24,877 | INFO | dd.dogstatsd | daemon(daemon.py:157) | Starting

[miketheman]

This was attempted first in #22 and nothing ever finalized from it.
There's the DISABLE_DATADOG_AGENT env var, which prevents startup, but I don't know how exactly one might set that for the one-off.

How does the Agent startup interfere with the one-off dyno?

[cabello]

As a short gap measure I did the following.

On integration code, Sinatra for example:

settings.datadog_tracer.configure enabled: ENV['DATADOG_APM_ENABLED'].present?

On Procfile:

web: env DATADOG_APM_ENABLED=true bundle exec ...

This will require me to add this to every other entry in the Procfile that I want to support, but for now it's okay.

---

How does the Agent startup interfere with the one-off dyno?

We have tracing on ActiveRecord queries, so when you do something like User.find(...) you start getting output unable to connect to localhost : 7777 and it loops indefinitely and it gets really hard to type on the console, while this message is being print multiple times a second.

[miketheman]

@cabello Thanks for the details.

We're looking at how to best include the tracing agent into this buildpack (see #28) - so if you're using this one, the trace agent doesn't start yet, and the tracer client code is the one spamming the log messages, regardless of web or one-off dyno.

This seems to be an issue with the trace library behaviors when it cannot connect to the trace agent endpoint.
See DataDog/dd-trace-rb#53 for how this has been reported and then solved for Rails/ActiveRecord.

I hope to incorporate the trace agent to the mainline buildback soon.