[BUG] Parsed field data missing when source field not included #3265
Description
What is the bug?
When using the PPL parse command, if the source field that was parsed is not included in the subsequent fields command, the parsed field data is missing from the final result.
This behavior appears to be by design in the code: ProjectOperator.java#L76.
How can one reproduce the bug?
POST parse_command_test/_doc
{
"@timestamp": "2025-01-04T04:00:00Z",
"line": {
"@message": "Request from AWS for client ID 123 is OVER_LIMIT"
}
}
POST _plugins/_ppl
{
"query": """
search source=parse_command_test
| parse line.@message 'Request from (?<service>.+) for client ID (?<clientId>.+) is OVER_LIMIT'
| fields @timestamp, service, clientId
"""
}
{
"schema": [
{
"name": "@timestamp",
"type": "timestamp"
},
{
"name": "service",
"type": "string"
},
{
"name": "clientId",
"type": "string"
}
],
"datarows": [
[
"2025-01-04 04:00:00"
]
],
"total": 1,
"size": 1
}
What is the expected behavior?
- If intentional: Provide clear documentation to explain why parsed field data depends on the inclusion of the original field in the fields command.
- Otherwise: Modify the behavior to ensure parsed fields remain available even if the original field is excluded.
Do you have any screenshots?
N/A
Do you have any additional context?
N/A