I noticed you've actually highlighted this in the current tracer code:

https://github.com/DataDog/dd-trace-rb/blob/master/lib/ddtrace/contrib/rails/action_controller.rb#L66-L68

Not every exception raised during controller processing maps to a 500 response. By default, current rails inserts the ActionDispatch::ShowExceptions middleware which calculates the appropriate status code for an exception using ActionDispatch::ExceptionWrapper. You could also call the exception wrapper to calculate the probable status code for any exception so that it is more accurate.

We recently starting using ShowExceptions more for serving proper dynamic 404 pages and rely on ExceptionWrapper to translate common exceptions into client errors (4XX response) which we don't care about, but these responses now appear incorrectly in Datadog APM as Error responses with 500 response codes.

P.S. Really enjoying watching the product grow!