Quotes are stripped from pipeline configuration if it includes an environment variable reference in LogStash v8.15.1

[mark-VIII]

Logstash information:

LogStash version is 8.15.1.
Expanded from tar archive.
Started via command line.

Plugins installed: (bin/logstash-plugin list --verbose)

Plugins included by default in tar archive distribution.

JVM (e.g. java -version):

Using bundled JDK.

OS version (uname -a if on a Unix-like system):

5.15.0-119-generic #129-Ubuntu SMP Fri Aug 2 19:25:20 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

Description of the problem including expected versus actual behavior:

Expected behaviour is that LogStash will start a pipeline without error when the configuration includes an environment variable reference.

Actual behaviour is that the configuration is invalidated when it includes an environment variable reference such that LogStash fails to run the pipeline. If all env var references are removed, the pipeline starts successfully.

Steps to reproduce:

1. Create a file (test.conf) containing the following configuration:

input { stdin { } }

filter {
  ruby {
    init => '
      require "uri"
      require "net/http"
      require "json"
      $version = ${VERSION}
      uri = URI("http://localhost:9600")
      http = Net::HTTP.new(uri.host, uri.port)
      req = Net::HTTP::Get.new(uri.request_uri)
      res = http.request(req)
      if res.is_a?(Net::HTTPSuccess) then
        json = JSON.parse(res.body)
        $version = json["version"]
      end
    '
    code => "event.set('ls_version', $version)"
  }
}

output { stdout { } }

2. Start LogStash using command: bin/logstash -f test.conf
3. Observe the following error:

[2024-09-10T02:01:18,645][INFO ][logstash.javapipeline    ] Pipeline `main` is configured with `pipeline.ecs_compatibility: v8` setting. All plugins in this pipeline will default to `ecs_compatibility => v8` unless explicitly configured otherwise.
[2024-09-10T02:01:18,667][ERROR][logstash.filters.ruby    ][main] error in register {:message=>"(ruby filter init):6: unknown regexp options - lcalht\n      uri = URI(http://localhost:9600)\n                     ^", :exception=>SyntaxError, :backtrace=>["org/jruby/RubyKernel.java:1274:in `eval'", "/root/logstash-8.15.1/vendor/bundle/jruby/3.1.0/gems/logstash-filter-ruby-3.1.8/lib/logstash/filters/ruby.rb:62:in `register'", "org/logstash/config/ir/compiler/AbstractFilterDelegatorExt.java:75:in `register'", "/root/logstash-8.15.1/logstash-core/lib/logstash/java_pipeline.rb:237:in `block in register_plugins'", "org/jruby/RubyArray.java:1981:in `each'", "/root/logstash-8.15.1/logstash-core/lib/logstash/java_pipeline.rb:236:in `register_plugins'", "/root/logstash-8.15.1/logstash-core/lib/logstash/java_pipeline.rb:617:in `maybe_setup_out_plugins'", "/root/logstash-8.15.1/logstash-core/lib/logstash/java_pipeline.rb:249:in `start_workers'", "/root/logstash-8.15.1/logstash-core/lib/logstash/java_pipeline.rb:194:in `run'", "/root/logstash-8.15.1/logstash-core/lib/logstash/java_pipeline.rb:146:in `block in start'"]}
[2024-09-10T02:01:18,670][ERROR][logstash.javapipeline    ][main] Pipeline error {:pipeline_id=>"main", :exception=>#<RuntimeError: unexpected error: (ruby filter init):6: unknown regexp options - lcalht
      uri = URI(http://localhost:9600)
                     ^>, :backtrace=>["/root/logstash-8.15.1/vendor/bundle/jruby/3.1.0/gems/logstash-filter-ruby-3.1.8/lib/logstash/filters/ruby.rb:71:in `register'", "org/logstash/config/ir/compiler/AbstractFilterDelegatorExt.java:75:in `register'", "/root/logstash-8.15.1/logstash-core/lib/logstash/java_pipeline.rb:237:in `block in register_plugins'", "org/jruby/RubyArray.java:1981:in `each'", "/root/logstash-8.15.1/logstash-core/lib/logstash/java_pipeline.rb:236:in `register_plugins'", "/root/logstash-8.15.1/logstash-core/lib/logstash/java_pipeline.rb:617:in `maybe_setup_out_plugins'", "/root/logstash-8.15.1/logstash-core/lib/logstash/java_pipeline.rb:249:in `start_workers'", "/root/logstash-8.15.1/logstash-core/lib/logstash/java_pipeline.rb:194:in `run'", "/root/logstash-8.15.1/logstash-core/lib/logstash/java_pipeline.rb:146:in `block in start'"], "pipeline.sources"=>["/root/logstash-8.15.1/pipeline/example/test.conf"], :thread=>"#<Thread:0x61b5b709 /root/logstash-8.15.1/logstash-core/lib/logstash/java_pipeline.rb:134 run>"}
[2024-09-10T02:01:18,671][INFO ][logstash.javapipeline    ][main] Pipeline terminated {"pipeline.id"=>"main"}
[2024-09-10T02:01:18,677][ERROR][logstash.agent           ] Failed to execute action {:id=>:main, :action_type=>LogStash::ConvergeResult::FailedAction, :message=>"Could not execute action: PipelineAction::Create<main>, action_result: false", :backtrace=>nil}
[2024-09-10T02:01:18,696][INFO ][logstash.runner          ] Logstash shut down.

Notice that the " have been removed from around the URI string value.

If the line $version = ${VERSION} is removed from the configuration or the same configuration is used in LogStash v8.15.0, the pipeline will start and operate correctly.

We believe this problem has been introduced by this change: #16375

Could also be related to #16433 ?

[akbwm]

We are seeing the same issue when using JDBC input plugin and specifying an SQL statement which includes some quoted values and a passing in ENV variable.

statement => "SELECT id, name, IF(country_code = 'uk', 'gb', country_code) AS country_code, '${SOURCE_NAME}' AS external_source FROM ${FROM_TABLE} ORDER BY id

During execution, the query is completely stripped of all quotes (single or double) and thus results in an error:

[ERROR] 2024-09-09 13:16:45.283 [[main]<jdbc] jdbc - Java::JavaSql::SQLSyntaxErrorException: (conn=2248) Unknown column 'uk' in 'field list': SELECT count(*) AS "COUNT" FROM (SELECT id, name, IF(country_code = uk, gb, country_code) AS country_code, some_source_from_env_var AS external_source FROM some_table_20240910 ORDER BY id ) AS "T1" LIMIT 1

The same setup worked (and works) when using version 8.15.0.
The problem seems to be related to this change introduced in 8.15.1 where all quotes are stripped if an ENV var has been encountered.

https://github.com/elastic/logstash/blob/v8.15.1/logstash-core/lib/logstash/util/substitution_variables.rb#L88C5-L96C8

refined_value = placeholder_value.gsub(/[\\"\\']/, '')

[mashhurs]

Confirming that #16365 caused the regression.