Set pipeline name automatically in transform definitions #2973

mrodm · 2025-10-02T11:08:19Z

This PR adds support to set automatically the pipeline name used in transforms adding the current package version as a prefix.

In order to achieve that elasticsearch/transform/*/transform.yml is managed by elastic-package as a template file (Golang html template). And it is allowed to use a function in the template named IngestPipelineName that adds the current version of the package to the pipeline.

Example:

dest:
  index: "logs-ti_anomali_latest.threatstream-2"
  pipeline: '{{ IngestPipelineName "transform" }}'
  aliases:
    - alias: "logs-ti_anomali_latest.threatstream"
      move_on_creation: true

This rendering of the template happens at different stages to ensure always is performed:

Building package: elastic-package build
Installing package: elastic-package install
System tests: elastic-package test system

Author's checklist

Test rendering while building
Test installing package
Test running system tests
Add note in Revert stack version pinning for packages failing with 9.x #2773 for the new test package.

How to test this PR locally

elastic-package stack up -v -d
package_path="test/packages/parallel/ti_anomali_template"

# Build and check that pipeline has been replaced
elastic-package -C "${package_path}" build -v
grep -A 5 "dest:"  build/packages/ti_anomali_template/1.22.1/elasticsearch/transform/latest_ioc/transform.yml 
# Expected output:
# dest:
#   index: "logs-ti_anomali_latest.threatstream-2"
#   pipeline: '1.22.1-transform'
#   aliases:
#     - alias: "logs-ti_anomali_latest.threatstream"
#       move_on_creation: true


# Install package
elastic-package -C "${package_path}" install -v

# check in Kibana that the transform has been installed with the right pipeline
curl -k -s -u elastic:changeme 'https://localhost:9200/_transform/logs-ti_anomali_template.latest_ioc-default-0.3.0' | jq -r '.transforms[] | .dest'
# Expected output:
# {
#   "index": "logs-ti_anomali_latest.threatstream-2",
#   "aliases": [
#     {
#       "alias": "logs-ti_anomali_latest.threatstream",
#       "move_on_creation": true
#     }
#   ],
#   "pipeline": "1.22.1-transform"
# }

# Running system tests for this package should work successfully
# Transform should be detected and validated
elastic-package -C "${package_path}" test system -v

elastic-package stack down -v

Allow to set an IngestPipelineName function in these templates in order to set the right ingest template filename. This filename must contain as a prefix the package version (from the manifest).

mrodm · 2025-10-02T14:44:53Z

/test

mrodm · 2025-10-02T15:55:43Z

/test

mrodm · 2025-10-02T17:03:04Z

/test

mrodm · 2025-10-02T17:42:45Z

/test

mrodm · 2025-10-02T18:17:45Z

/test

mrodm · 2025-10-03T08:14:18Z

/test

mrodm · 2025-10-03T11:09:20Z

/test

jsoriano

LGTM, some nitpicking and wondering if we should use text/template instead of html/template.

internal/packages/packages.go

jsoriano · 2025-10-07T14:34:24Z

internal/packages/packages.go

+	_, err = os.Stat(filepath.Join(packageRootPath, "elasticsearch", "ingest_pipeline", pipelineFileName))
+	if err != nil {
+		return nil, TransformDefinition{}, fmt.Errorf("destination ingest pipeline file %s not found: %w", pipelineFileName, err)
+	}


Did you test with integrations? Maybe some package fails with this 😬 Though if it does we should fix the package.

Testing to build locally all packages in the integrations repository, there is one package that sets as dest.pipeline the ingest pipeline from the data stream:

dest: index: "aws_billing.cur-v1" pipeline: "metrics-aws_billing.cur-0.1.0-pipeline_extract_metadata" aliases: - alias: "aws_billing.cur_latest" move_on_creation: true

That fails with the current validation 🤔

That package (aws_billing) is failing this validation because of:

It's using a pipeline from a data stream

It set an old version in the destination pipeline, it should be 0.2.0.

That pipeline from cur data stream looks like it is not used:
https://github.com/elastic/integrations/blob/34df8c08914a678dc3310bd000e7908a1af42ef6/packages/aws_billing/data_stream/cur/elasticsearch/ingest_pipeline/pipeline_extract_metadata.yml

At least from the default pipeline there is no pipeline processor to trigger that other pipeline:
https://github.com/elastic/integrations/blob/34df8c08914a678dc3310bd000e7908a1af42ef6/packages/aws_billing/data_stream/cur/elasticsearch/ingest_pipeline/default.yml

Here I've added the support to detect if the ingest pipeline exists on any data stream:
4f2756e

Probably, it could be moved the pipeline in that package to be on the elasticsearch folder in the root, but in any case it could be interesting to keep the support to check if pipelines are located in data streams.

WDYT ?

I thought that for those cases using ingest pipelines for data streams, the transform definition could use the function like this:

dest: index: "aws_billing.cur-v1" pipeline: "metrics-aws_billing.cur{{ ingestPipelineName "pipeline_extract_metadata"}}" aliases: - alias: "aws_billing.cur_latest" move_on_creation: true

Currently, this would be the minimum change required for aws_billing package
elastic/integrations#15593 since the latest version of the package is 0.2.0.

Another option could be move the Ingest Pipeline to the elasticsearch folder from the root of the package, but I'm not totally sure if this could lead to some issue.

Currently, this would be the minimum change required for aws_billing package
elastic/integrations#15593 since the latest version of the package is 0.2.0.

Yep, we can apply this change by now, but using a pipeline from a data stream in a transform sounds wrong.

We can also use the placeholder you proposed in the previous comment, right?

pipeline: "metrics-aws_billing.cur{{ ingestPipelineName "pipeline_extract_metadata"}}"

Co-authored-by: Jaime Soriano Pastor <jaime.soriano@elastic.co>

mrodm · 2025-10-07T19:18:32Z

test integrations

elastic-vault-github-plugin-prod · 2025-10-07T19:25:16Z

Created or updated PR in integrations repository to test this version. Check elastic/integrations#15594

elasticmachine · 2025-10-07T19:39:46Z

💚 Build Succeeded

Buildkite Build
Commit: 4f2756e

History

💔 Build #6300 failed a44d6cc
💚 Build #6288 succeeded 351171c
💚 Build #6287 succeeded 351171c
💚 Build #6284 succeeded 351171c
💔 Build #6283 failed 351171c

cc @mrodm

mrodm added 2 commits October 2, 2025 12:26

Set transform manifest as template

07f3e15

Allow to set an IngestPipelineName function in these templates in order to set the right ingest template filename. This filename must contain as a prefix the package version (from the manifest).

Add a copy of ti_anomaly with a dummy ingest pipeline for transform

5961b7e

mrodm self-assigned this Oct 2, 2025

mrodm added 6 commits October 2, 2025 13:29

Skip check if there is no destination pipeline

951defe

Check for empty pipelines

6cf6d08

Add test

9093315

Pin stack version for ti_anomali_template

ff9aa3f

Update stack version used for ti_anomali and ti_anomali_logsdb

bf90e2b

Update index and alias names in tranform

351171c

mrodm marked this pull request as ready for review October 3, 2025 11:46

mrodm requested a review from a team October 6, 2025 08:42

jsoriano reviewed Oct 7, 2025

View reviewed changes

mrodm and others added 5 commits October 7, 2025 19:24

Apply suggestions from code review

789693b

Co-authored-by: Jaime Soriano Pastor <jaime.soriano@elastic.co>

Set to lowercase first letter in template function

d8f0447

Use text/template

ff0daa5

Re-phrase error message

a44d6cc

Validate if ingest pipeline exists on data stream

4f2756e

elastic-vault-github-plugin-prod bot mentioned this pull request Oct 7, 2025

Test elastic-package#2973 - DO NOT MERGE elastic/integrations#15594

Draft

mrodm mentioned this pull request Oct 8, 2025

Revert stack version pinning for packages failing with 9.x #2773

Open

mrodm requested a review from jsoriano October 8, 2025 15:11

jsoriano approved these changes Oct 9, 2025

View reviewed changes

Set pipeline name automatically in transform definitions #2973

Are you sure you want to change the base?

Set pipeline name automatically in transform definitions #2973

Conversation

mrodm commented Oct 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Author's checklist

How to test this PR locally

Uh oh!

mrodm commented Oct 2, 2025

Uh oh!

mrodm commented Oct 2, 2025

Uh oh!

mrodm commented Oct 2, 2025

Uh oh!

mrodm commented Oct 2, 2025

Uh oh!

mrodm commented Oct 2, 2025

Uh oh!

mrodm commented Oct 3, 2025

Uh oh!

mrodm commented Oct 3, 2025

Uh oh!

jsoriano left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jsoriano Oct 7, 2025

Choose a reason for hiding this comment

Uh oh!

mrodm Oct 7, 2025

Choose a reason for hiding this comment

Uh oh!

mrodm Oct 7, 2025

Choose a reason for hiding this comment

Uh oh!

mrodm Oct 7, 2025

Choose a reason for hiding this comment

Uh oh!

mrodm Oct 8, 2025

Choose a reason for hiding this comment

Uh oh!

mrodm Oct 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jsoriano Oct 9, 2025

Choose a reason for hiding this comment

Uh oh!

mrodm commented Oct 7, 2025

Uh oh!

elastic-vault-github-plugin-prod bot commented Oct 7, 2025

Uh oh!

elasticmachine commented Oct 7, 2025

💚 Build Succeeded

History

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

mrodm commented Oct 2, 2025 •

edited

Loading

mrodm Oct 8, 2025 •

edited

Loading