Discuss: project structure from packages perspective

[mtojek]

There is no need anymore (once already split with package-storage) to apply additional versioning in directories, e.g.:

change dev/packages/beats/aws/0.0.4 to dev/packages/beats/aws

EDIT

I've renamed the issue as it's no longer a discussion about the version tag.

[mtojek]

@ruflin

what do you think about adjusting the overall project structure similarly to https://github.com/elastic/package-storage (packages in root)?

[exekias]

Actually I wonder, why not moving all packages to the root folder? Ideally we could have all tooling external to this repo, so we only store packages here

[mtojek]

@exekias @ruflin Do you suggest to create a new repo elastic/integrations-tools for all tools you can use to work with integrations (see: #14)? It's a good moment to start with this approach instead of some scripts hidden in dev directory (which we'd like to move later).

Currently, there are two places where you can keep packages: elastic/integrations (packages in development) and elastic/package-storage (released, at least "alpha").

[ruflin]

The reason I like to have a directory like packages is because there will always be other files like README.md and probably some directories for license or anything else we don't know about yet. This makes it future proof. For example the beats-import scripts are for me not generic tooling and belong in here.

I agree that some tooling might move out of this repo in the future but I would keep it in this repo for now. Like this tooling and packages can be adjusted just in 1 PR and now have to jump between two repos. I think it speeds up development. Lets get it stable first.

This repository is for package development. The package-storage is for releasing packages.

[mtojek]

@ruflin @exekias

I want to discuss another issue. We have some integrations published as "alpha". I assume we should keep them both in elastic/integrations and elastic/package-storage.

Does it mean that we should keep the current structure:

packages/
     alpha/
        mysql
            (integration source considered as alpha, no "semver" in folder name)
     beats/
        mysql
            (newest regenerated integration source, no "semver" in folder name)

Some arguments behind this model:

• we need to let contributors re-import the integration and manually apply changes to alpha packages
• there is no danger that somebody accidentally wipes the "alpha" integration with a generated one
• packages/beats stays as a playground for developers working on the migration script.

EDIT:
In this PR I wiped-out all alpha packages in favor of ones in package-storage: #21 . Not sure if this is the way to go.

[ruflin]

For me the question is, how will the development look like for the next 2-3 versions of the package. Will the dev take alpha/mysql, improve it and push a new version to the storage or will it be on top of beats/mysql and then a new version is pushed.

Brainstorming now: Lets assume under packages/mysql we have the package we publish to the storage. Under dev/alpha/ and dev/beats are the two mentioned above. Now we need a way to combine the two dev packages to the actual package. Lets assume we have a perfect conversion of ingest pipelines with the script, so we take all these from the beats/mysql directory. We have no script for the readme, so we take it from the alpha package etc. This could lead us to a situation, where we copy over the beats/mysql to packages/mysql and then select from alpha/mysql all the files that must be overwritten. All the files that we already can generate should also be removed from alpha to make it clear, this is not an actual package but only manual curated bits. This is where we will keep the manual curated bits as long as we need. In the end, we just sync files from the two directories.

The syncing should be on demand per package to make sure it is reviewed before it is updated. We keep to pulling in all the changes from Beats automatically with the script into dev/beats but do the merge for the dev/mysql package by manually running the script and then opening a PR. The reason is to make sure, we keep reviewing the packages that they are valid.

[mtojek]

I will present a different idea based on your sample (mysql). In this proposal the more important is the final effect (generated packages) rather than intermediate development stages of packages.

It might be hard for developers to jump between some resources in dev/alpha/mysql and dev/beats/mysql, so I propose the following scenario:

1. Developer A publishes dev/alpha/mysql and apply nest possible changes to this one.
2. Developer B works on adjusting the migration script to achieve the same final result as dev/alpha/mysql - to achieve a package generated in 100%.
3. Once dev/beats/mysql looks exactly like dev/alpha/mysql, the package dev/alpha/mysql can be removed and the dev/beats/mysql will be published as alpha.

It's not crucial for me where we keep templates files and similars. It would be cool if we can keep them in Beats.

[ruflin]

@mtojek In your scenario, from which of the two directories is the package built that is pushed to the package-storage? Alpha until they are 100% identical?

One thing I realised we might have a disconnect thanks to elastic/package-registry#460 is that from dev/alpha/mysql to build/mysql/0.3.0 there is a build step in between these two. Not having the versions in this repo is to simplify development, but to run it with the registry, it must have the version in the path.

[mtojek]

@mtojek In your scenario, from which of the two directories is the package built that is pushed to the package-storage? Alpha until they are 100% identical?

Yes

Not having the versions in this repo is to simplify development, but to run it with the registry, it must have the version in the path.

Currently, the PR elastic/package-registry#460 changes the dev/generator to not require version in the path dev/alpha/mysql but read it from manifest.yml instead. The created public folder stays as is:

[mtojek]

As we discussed offline, I'm starting with some changes in the repository:

1. Packages will be present in the root directory.
2. Migrated packages will be stored together with manually adjusted.
3. No version in package path.

[mtojek]

Resolving