Skip to content

ActiveSupport Notifications

Jump to bottom
adham90 edited this page Feb 20, 2026 · 1 revision

ActiveSupport Notifications

RubyLLM::Agents emits ActiveSupport::Notifications events throughout the middleware pipeline, giving you real-time observability into every execution, cache interaction, budget check, and reliability event.

Notifications fire independently of database tracking — even if track_executions is disabled, subscribers still receive events.

Event Taxonomy

All events use the ruby_llm_agents. prefix and are organized by domain:

Event Domain Description
ruby_llm_agents.execution.start Execution Agent execution begins
ruby_llm_agents.execution.complete Execution Agent execution succeeded
ruby_llm_agents.execution.error Execution Agent execution failed
ruby_llm_agents.cache.hit Cache Response served from cache
ruby_llm_agents.cache.miss Cache Cache lookup found no match
ruby_llm_agents.cache.write Cache Response written to cache
ruby_llm_agents.budget.check Budget Budget check performed before execution
ruby_llm_agents.budget.exceeded Budget Execution blocked by budget limit
ruby_llm_agents.budget.record Budget Spend recorded after execution
ruby_llm_agents.reliability.fallback_used Reliability Fallback model succeeded after primary failed
ruby_llm_agents.reliability.all_models_exhausted Reliability All models (primary + fallbacks) failed

Execution Events

execution.start

Fired when an agent execution begins, before the LLM call.

Payload:

Field Type Description
:agent_type String Agent class name
:model String Configured model
:tenant_id String/nil Tenant identifier
:execution_id Integer/nil Database execution ID 
ActiveSupport::Notifications.subscribe("ruby_llm_agents.execution.start") do |*, payload|
  Rails.logger.info "[LLM] Starting #{payload[:agent_type]} with #{payload[:model]}"
end

execution.complete

Fired when an agent execution succeeds.

Payload:

Field Type Description
:agent_type String Agent class name
:agent_type_symbol Symbol Agent type (:chat, :embed, etc.)
:execution_id Integer/nil Database execution ID
:model String Configured model
:model_used String Model that actually ran
:tenant_id String/nil Tenant identifier
:status String "success"
:duration_ms Integer Execution time in milliseconds
:input_tokens Integer Prompt tokens
:output_tokens Integer Response tokens
:total_tokens Integer Sum of input + output
:input_cost Float Cost of input tokens
:output_cost Float Cost of output tokens
:total_cost Float Total execution cost
:cached Boolean Whether response came from cache
:attempts_made Integer Number of attempts (retries + fallbacks)
:finish_reason String "stop", "length", "tool_calls", etc.
:time_to_first_token_ms Integer/nil TTFT (streaming only)
:error_class nil Always nil on success
:error_message nil Always nil on success 
ActiveSupport::Notifications.subscribe("ruby_llm_agents.execution.complete") do |*, payload|
  StatsD.timing("llm.duration", payload[:duration_ms])
  StatsD.increment("llm.executions", tags: [
    "agent:#{payload[:agent_type]}",
    "model:#{payload[:model_used]}"
  ])
  StatsD.gauge("llm.cost", payload[:total_cost])
  StatsD.histogram("llm.tokens", payload[:total_tokens])
end

execution.error

Fired when an agent execution fails (after all retries and fallbacks are exhausted).

Payload: Same fields as execution.complete, but with:

Field Type Description
:status String "error" or "timeout"
:error_class String Exception class name
:error_message String Exception message 
ActiveSupport::Notifications.subscribe("ruby_llm_agents.execution.error") do |*, payload|
  StatsD.increment("llm.errors", tags: [
    "agent:#{payload[:agent_type]}",
    "error:#{payload[:error_class]}"
  ])

  if payload[:error_class] == "Timeout::Error"
    Slack::Notifier.new(ENV['SLACK_WEBHOOK']).ping(
      "Timeout in #{payload[:agent_type]} after #{payload[:duration_ms]}ms"
    )
  end
end

Cache Events

cache.hit

Fired when a cached response is returned instead of calling the LLM.

Payload:

Field Type Description
:agent_type String Agent class name
:cache_key String The cache key that matched

cache.miss

Fired when a cache lookup finds no match and the LLM will be called.

Payload: Same as cache.hit.

cache.write

Fired after a successful LLM response is written to cache.

Payload: Same as cache.hit.

# Track cache hit rate
hits = 0
total = 0

ActiveSupport::Notifications.subscribe(/ruby_llm_agents\.cache\.(hit|miss)/) do |name, *, payload|
  total += 1
  hits += 1 if name.end_with?(".hit")
  StatsD.gauge("llm.cache.hit_rate", hits.to_f / total) if total > 0
end

Budget Events

budget.check

Fired before execution when budget limits are checked.

Payload:

Field Type Description
:agent_type String Agent class name
:tenant_id String/nil Tenant identifier

budget.exceeded

Fired when an execution is blocked because budget limits have been reached.

Payload: Same as budget.check.

ActiveSupport::Notifications.subscribe("ruby_llm_agents.budget.exceeded") do |*, payload|
  PagerDuty.trigger(
    summary: "Budget exceeded for #{payload[:agent_type]}",
    details: { tenant_id: payload[:tenant_id] }
  )
end

budget.record

Fired after spend is recorded following a successful execution.

Payload:

Field Type Description
:agent_type String Agent class name
:tenant_id String/nil Tenant identifier
:total_cost Float Cost recorded
:total_tokens Integer Tokens used 
ActiveSupport::Notifications.subscribe("ruby_llm_agents.budget.record") do |*, payload|
  StatsD.increment("llm.spend", payload[:total_cost], tags: [
    "agent:#{payload[:agent_type]}",
    "tenant:#{payload[:tenant_id]}"
  ])
end

Reliability Events

reliability.fallback_used

Fired when the primary model fails and a fallback model succeeds.

Payload:

Field Type Description
:agent_type String Agent class name
:primary_model String The model that failed
:used_model String The fallback model that succeeded
:attempts_made Integer Total attempts across all models 
ActiveSupport::Notifications.subscribe("ruby_llm_agents.reliability.fallback_used") do |*, payload|
  StatsD.increment("llm.fallback", tags: [
    "agent:#{payload[:agent_type]}",
    "primary:#{payload[:primary_model]}",
    "used:#{payload[:used_model]}"
  ])
end

reliability.all_models_exhausted

Fired when all models (primary + fallbacks) fail, just before raising AllModelsExhaustedError.

Payload:

Field Type Description
:agent_type String Agent class name
:models_tried Array List of all models attempted 
ActiveSupport::Notifications.subscribe("ruby_llm_agents.reliability.all_models_exhausted") do |*, payload|
  Slack::Notifier.new(ENV['SLACK_WEBHOOK']).ping(
    ":rotating_light: All models exhausted for #{payload[:agent_type]}: #{payload[:models_tried].join(', ')}"
  )
end

Subscribing to Events

Single Event

ActiveSupport::Notifications.subscribe("ruby_llm_agents.execution.complete") do |*, payload|
  # Handle event
end

Pattern Matching

Subscribe to all events in a domain:

# All execution events (start, complete, error)
ActiveSupport::Notifications.subscribe(/ruby_llm_agents\.execution\./) do |name, *, payload|
  Rails.logger.info "[LLM] #{name}: #{payload[:agent_type]}"
end

# All reliability events
ActiveSupport::Notifications.subscribe(/ruby_llm_agents\.reliability\./) do |name, *, payload|
  Rails.logger.warn "[LLM] #{name}: #{payload[:agent_type]}"
end

# All events
ActiveSupport::Notifications.subscribe(/ruby_llm_agents\./) do |name, *, payload|
  Rails.logger.debug "[LLM] #{name}: #{payload.inspect}"
end

Unsubscribing

subscriber = ActiveSupport::Notifications.subscribe("ruby_llm_agents.execution.complete") do |*, payload|
  # Handle event
end

# Later
ActiveSupport::Notifications.unsubscribe(subscriber)

Production Integration Examples

StatsD / Datadog

# config/initializers/ruby_llm_agents.rb
ActiveSupport::Notifications.subscribe("ruby_llm_agents.execution.complete") do |*, payload|
  tags = ["agent:#{payload[:agent_type]}", "model:#{payload[:model_used]}"]
  StatsD.timing("llm.duration_ms", payload[:duration_ms], tags: tags)
  StatsD.increment("llm.executions.success", tags: tags)
  StatsD.histogram("llm.cost", payload[:total_cost], tags: tags)
  StatsD.histogram("llm.tokens", payload[:total_tokens], tags: tags)
end

ActiveSupport::Notifications.subscribe("ruby_llm_agents.execution.error") do |*, payload|
  tags = ["agent:#{payload[:agent_type]}", "error:#{payload[:error_class]}"]
  StatsD.increment("llm.executions.error", tags: tags)
end

ActiveSupport::Notifications.subscribe("ruby_llm_agents.reliability.fallback_used") do |*, payload|
  StatsD.increment("llm.fallback", tags: [
    "agent:#{payload[:agent_type]}",
    "from:#{payload[:primary_model]}",
    "to:#{payload[:used_model]}"
  ])
end

Slack Alerts

# config/initializers/ruby_llm_agents.rb
slack = Slack::Notifier.new(ENV['SLACK_WEBHOOK'])

ActiveSupport::Notifications.subscribe("ruby_llm_agents.budget.exceeded") do |*, payload|
  slack.ping(":money_with_wings: Budget exceeded for #{payload[:agent_type]} (tenant: #{payload[:tenant_id]})")
end

ActiveSupport::Notifications.subscribe("ruby_llm_agents.reliability.all_models_exhausted") do |*, payload|
  slack.ping(":rotating_light: All models exhausted for #{payload[:agent_type]}: #{payload[:models_tried].join(', ')}")
end

Custom Logging

# config/initializers/ruby_llm_agents.rb
ActiveSupport::Notifications.subscribe(/ruby_llm_agents\./) do |name, started, finished, id, payload|
  duration = ((finished - started) * 1000).round(2)
  Rails.logger.tagged("LLM") do
    Rails.logger.info "#{name} (#{duration}ms) #{payload.except(:error_message).inspect}"
  end
end

Safety

All notification calls are wrapped in a rescue block to ensure that subscriber errors never break agent execution. If a subscriber raises an exception, the notification is silently dropped and execution continues normally.

# This is safe — a buggy subscriber won't crash your agent
ActiveSupport::Notifications.subscribe("ruby_llm_agents.execution.complete") do |*, payload|
  raise "oops" # Will not affect agent execution
end

Related Pages

Clone this wiki locally