Add spec for shared folder and link files

[marc-gr]

What does this PR do?

Adds a spec definition for a shared folder under _dev that will allow users to put in there any files that are going to be linked from multiple data_streams and/or packages.

Adds a spec definition to allow *.link files under pipelines, agent, and fields to let users reuse repeated files from other parts of the integrations repository.

Adds a test package for both.

Checklist

• I have added test packages to test/packages that prove my change is effective.
• I have added an entry in spec/changelog.yml.

Related issues

• Closes Support for shared fields mapping and pipelines between data streams #89

[jsoriano]

I think that we should take into account the linked files in validations, though they will be taken into account when the package is built, so probably not a blocker till we have more specific validation of source vs. built packages.

Apart from that, I wonder if we should include checksums in the files tested here 🤔

[marc-gr]

Added the logic to be able to do the checks here. I am not sure about what to do with the FindRoot... functionality as it is used in both here and elastic-package, but does not really makes sense to me to have it in here other than that. For now I duplicated it but open to suggestions.

I also updated elastic/elastic-package#2483 to make use of the changes

[marc-gr]

I simplified this as suggested:

• Only logic in package-spec is now related to finding the included files on validation
• Removes any extra logic and does this transparently using a custom fs.FS
• As suggested in Simple includes implementation elastic-package#2483 now included files are relative to the link location

For this to be effective I had to set required: false to agent and fields specs. This is not ideal but I am not sure how to add an anyOf condition in there. I see a similar case in https://github.com/elastic/package-spec/blob/main/spec/integration/elasticsearch/spec.yml#L24 and as far as I can tell it does not seem addressed either, so any suggestions in that direction are appreciated.

I think once we solve that this seems enough from the spec perspective.

[marc-gr]

@mrodm I added a semantic check for our case and the pipelines one, not sure this is the way you prefer them to be defined so lmk if any changes are required

[jsoriano]

Overall looks good, thanks for adding the suggestions. Added some small questions.

[mrodm]

Should we remark that this fields folder is just for the ones defined under data_stream ? Is that right?

If so, would it be better to re-phrase that item in the comment as this? WDYT @marc-gr ?

Suggested change

	// - fields
	// - data_stream/*/fields

IIUC the fields folder in transforms is not taken into account here. If it would be needed those links in transforms, I guess it could be added support in a follow-up. Let's continue with the current links defined in this PR , is that ok @jsoriano?

[marc-gr]

@mrodm I opted to add an allowLink option into the item spec after discussing it with @jsoriano as it seems more robust and not rely on the semantic validation, I think this is a cleaner approach together with the custom fs

[jsoriano]

I opted to add an allowLink option into the item spec after discussing it with @jsoriano as it seems more robust and not rely on the semantic validation, I think this is a cleaner approach together with the custom fs.

Thanks for trying this approach, it looks clearer and I feel more confident merging this. I think there can still be some corner cases, but they will be more limited.

I think this could be merged, but added a couple of questions.
Before merging I would like to double-check what parts of linkedfiles we want/need to publicly expose.

[jsoriano]

👍

[jsoriano]

This should be moved to a new version block for 3.3.6-next.

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 4f01f75

History

• 💚 Build #1117 succeeded 4c330a8
• 💚 Build #1113 succeeded 829e846
• 💔 Build #1112 failed 835ce7e
• 💚 Build #1110 succeeded 7422eec
• 💚 Build #1109 succeeded 9ab873d
• 💚 Build #1105 succeeded e03cb61

[jsoriano]

Thanks! 🎉

[mrodm]

Thanks!