Check and generate multiple readme files in one package #282

kaiyan-sheng · 2021-03-09T14:24:14Z

With the new package-spec update with input groups, we start to allow multiple readme files .md under _dev/build/docs and docs. This PR is to add the check and generate multiple readme files when running elastic-package build and elastic-package lint.

`elastic-package lint` when `billing.md` and `cloudtrail.md` are missing

elastic-package has been installed.
Lint the package
billing.md is outdated. Rebuild the package with 'elastic-package build'
cloudtrail.md is outdated. Rebuild the package with 'elastic-package build'
Error: checking readme files are up-to-date failed: check readme files are up-to-date failed

`elastic-package build` output

kaiyansheng@KaiyanMacBookPro:~/go/src/github.com/elastic/elastic-package/test/packages/aws (additional_readmes)$ elastic-package build
Build the package
README.md file rendered: /Users/kaiyansheng/go/src/github.com/elastic/elastic-package/test/packages/aws/docs/README.md
billing.md file rendered: /Users/kaiyansheng/go/src/github.com/elastic/elastic-package/test/packages/aws/docs/billing.md
cloudtrail.md file rendered: /Users/kaiyansheng/go/src/github.com/elastic/elastic-package/test/packages/aws/docs/cloudtrail.md
cloudwatch.md file rendered: /Users/kaiyansheng/go/src/github.com/elastic/elastic-package/test/packages/aws/docs/cloudwatch.md
dynamodb.md file rendered: /Users/kaiyansheng/go/src/github.com/elastic/elastic-package/test/packages/aws/docs/dynamodb.md
ebs.md file rendered: /Users/kaiyansheng/go/src/github.com/elastic/elastic-package/test/packages/aws/docs/ebs.md
ec2.md file rendered: /Users/kaiyansheng/go/src/github.com/elastic/elastic-package/test/packages/aws/docs/ec2.md
elb.md file rendered: /Users/kaiyansheng/go/src/github.com/elastic/elastic-package/test/packages/aws/docs/elb.md
lambda.md file rendered: /Users/kaiyansheng/go/src/github.com/elastic/elastic-package/test/packages/aws/docs/lambda.md
natgateway.md file rendered: /Users/kaiyansheng/go/src/github.com/elastic/elastic-package/test/packages/aws/docs/natgateway.md
rds.md file rendered: /Users/kaiyansheng/go/src/github.com/elastic/elastic-package/test/packages/aws/docs/rds.md
s3.md file rendered: /Users/kaiyansheng/go/src/github.com/elastic/elastic-package/test/packages/aws/docs/s3.md
sns.md file rendered: /Users/kaiyansheng/go/src/github.com/elastic/elastic-package/test/packages/aws/docs/sns.md
sqs.md file rendered: /Users/kaiyansheng/go/src/github.com/elastic/elastic-package/test/packages/aws/docs/sqs.md
transitgateway.md file rendered: /Users/kaiyansheng/go/src/github.com/elastic/elastic-package/test/packages/aws/docs/transitgateway.md
usage.md file rendered: /Users/kaiyansheng/go/src/github.com/elastic/elastic-package/test/packages/aws/docs/usage.md
vpcflow.md file rendered: /Users/kaiyansheng/go/src/github.com/elastic/elastic-package/test/packages/aws/docs/vpcflow.md
vpn.md file rendered: /Users/kaiyansheng/go/src/github.com/elastic/elastic-package/test/packages/aws/docs/vpn.md
Package built: /Users/kaiyansheng/go/src/github.com/elastic/elastic-package/build/integrations/aws/999.999.999
Done

`elastic-package lint` when there is no file missing

kaiyansheng@KaiyanMacBookPro:~/go/src/github.com/elastic/elastic-package/test/packages/aws (additional_readmes)$ elastic-package lint
Lint the package
Done

elasticmachine · 2021-03-09T14:24:35Z

Pinging @elastic/integrations (Team:Integrations)

elasticmachine · 2021-03-09T14:28:22Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Build Cause: Pull request #282 updated
Start Time: 2021-03-23T03:48:33.517+0000
Duration: 21 min 32 sec
Commit: 17506c5

Test stats 🧪

Test	Results
Failed	0
Passed	316
Skipped	1
Total	317

Trends 🧪

mtojek

Probably the command output should be adjusted:

Build the package
README.md file rendered: /Users/marcin.tojek/go/src/github.com/elastic/elastic-package/test/packages/aws/docs/README.md
Package built: /Users/marcin.tojek/go/src/github.com/elastic/elastic-package/build/integrations/aws/999.999.999
billing.md file rendered: /Users/marcin.tojek/go/src/github.com/elastic/elastic-package/test/packages/aws/docs/billing.md
Package built: /Users/marcin.tojek/go/src/github.com/elastic/elastic-package/build/integrations/aws/999.999.999
cloudtrail.md file rendered: /Users/marcin.tojek/go/src/github.com/elastic/elastic-package/test/packages/aws/docs/cloudtrail.md
Package built: /Users/marcin.tojek/go/src/github.com/elastic/elastic-package/build/integrations/aws/999.999.999
cloudwatch.md file rendered: /Users/marcin.tojek/go/src/github.com/elastic/elastic-package/test/packages/aws/docs/cloudwatch.md
Package built: /Users/marcin.tojek/go/src/github.com/elastic/elastic-package/build/integrations/aws/999.999.999
dynamodb.md file rendered: /Users/marcin.tojek/go/src/github.com/elastic/elastic-package/test/packages/aws/docs/dynamodb.md
Package built: /Users/marcin.tojek/go/src/github.com/elastic/elastic-package/build/integrations/aws/999.999.999
ebs.md file rendered: /Users/marcin.tojek/go/src/github.com/elastic/elastic-package/test/packages/aws/docs/ebs.md
Package built: /Users/marcin.tojek/go/src/github.com/elastic/elastic-package/build/integrations/aws/999.999.999
ec2.md file rendered: /Users/marcin.tojek/go/src/github.com/elastic/elastic-package/test/packages/aws/docs/ec2.md
Package built: /Users/marcin.tojek/go/src/github.com/elastic/elastic-package/build/integrations/aws/999.999.999
elb.md file rendered: /Users/marcin.tojek/go/src/github.com/elastic/elastic-package/test/packages/aws/docs/elb.md
Package built: /Users/marcin.tojek/go/src/github.com/elastic/elastic-package/build/integrations/aws/999.999.999
lambda.md file rendered: /Users/marcin.tojek/go/src/github.com/elastic/elastic-package/test/packages/aws/docs/lambda.md
Package built: /Users/marcin.tojek/go/src/github.com/elastic/elastic-package/build/integrations/aws/999.999.999
natgateway.md file rendered: /Users/marcin.tojek/go/src/github.com/elastic/elastic-package/test/packages/aws/docs/natgateway.md
Package built: /Users/marcin.tojek/go/src/github.com/elastic/elastic-package/build/integrations/aws/999.999.999
rds.md file rendered: /Users/marcin.tojek/go/src/github.com/elastic/elastic-package/test/packages/aws/docs/rds.md
Package built: /Users/marcin.tojek/go/src/github.com/elastic/elastic-package/build/integrations/aws/999.999.999
s3.md file rendered: /Users/marcin.tojek/go/src/github.com/elastic/elastic-package/test/packages/aws/docs/s3.md
Package built: /Users/marcin.tojek/go/src/github.com/elastic/elastic-package/build/integrations/aws/999.999.999
sns.md file rendered: /Users/marcin.tojek/go/src/github.com/elastic/elastic-package/test/packages/aws/docs/sns.md
Package built: /Users/marcin.tojek/go/src/github.com/elastic/elastic-package/build/integrations/aws/999.999.999
sqs.md file rendered: /Users/marcin.tojek/go/src/github.com/elastic/elastic-package/test/packages/aws/docs/sqs.md
Package built: /Users/marcin.tojek/go/src/github.com/elastic/elastic-package/build/integrations/aws/999.999.999
transitgateway.md file rendered: /Users/marcin.tojek/go/src/github.com/elastic/elastic-package/test/packages/aws/docs/transitgateway.md
Package built: /Users/marcin.tojek/go/src/github.com/elastic/elastic-package/build/integrations/aws/999.999.999
usage.md file rendered: /Users/marcin.tojek/go/src/github.com/elastic/elastic-package/test/packages/aws/docs/usage.md
Package built: /Users/marcin.tojek/go/src/github.com/elastic/elastic-package/build/integrations/aws/999.999.999
vpcflow.md file rendered: /Users/marcin.tojek/go/src/github.com/elastic/elastic-package/test/packages/aws/docs/vpcflow.md
Package built: /Users/marcin.tojek/go/src/github.com/elastic/elastic-package/build/integrations/aws/999.999.999
vpn.md file rendered: /Users/marcin.tojek/go/src/github.com/elastic/elastic-package/test/packages/aws/docs/vpn.md
Package built: /Users/marcin.tojek/go/src/github.com/elastic/elastic-package/build/integrations/aws/999.999.999

We don't have to notify the developer multiple times that the package has been built :)

cmd/build.go

cmd/lint.go

internal/docs/readme.go

test/packages/aws/_dev/build/docs/billing.md

cmd/build.go

…tional_readmes

…package into additional_readmes

cmd/build.go

cmd/lint.go

internal/docs/readme.go

kaiyan-sheng · 2021-03-14T01:20:25Z

@mtojek @ycombinator The current CI failure is caused by:

kaiyansheng@KaiyanMacBookPro:~/go/src/github.com/elastic/elastic-package/test/packages/kubernetes (additional_readmes)$ elastic-package lint
Lint the package
Error: can't check if readme files are up-to-date: failed to return a list of directory entries from /Users/kaiyansheng/go/src/github.com/elastic/elastic-package/test/packages/kubernetes: open /Users/kaiyansheng/go/src/github.com/elastic/elastic-package/test/packages/kubernetes/_dev/build/docs: no such file or directory

I see in several test packages, we have docs/README.md but the package itself doesn't have _dev/build/docs/README.md. I think _dev/build/docs/README.md is required to create the actual docs/README.md right?

mtojek · 2021-03-15T09:03:25Z

I see in several test packages, we have docs/README.md but the package itself doesn't have _dev/build/docs/README.md. I think _dev/build/docs/README.md is required to create the actual docs/README.md right?

Actually it is the other way round. The doc template can be used to generate the README file, but it's not a hard requirement. Please take a look at the log integration: https://github.com/elastic/integrations/tree/master/packages/log (it's exactly this use case).

kaiyan-sheng · 2021-03-15T14:44:53Z

Actually it is the other way round. The doc template can be used to generate the README file, but it's not a hard requirement. Please take a look at the log integration: https://github.com/elastic/integrations/tree/master/packages/log (it's exactly this use case).

Thanks! OK it's fixed now I believe. This PR is ready for another review :) Thank you @mtojek and @ycombinator !

internal/docs/readme.go

mtojek · 2021-03-15T15:09:21Z

cmd/build.go

-		cmd.Printf("%s file rendered: %s\n", docs.ReadmeFile, target)
+
+	for _, target := range targets {
+		if target != "" {


What is the case that the target is empty?

It will be empty when README file is static and can't be generated from the template file. This came from generateReadme(fileName, packageRoot) function.

Ah I see! It might be cleaner to skip this file on the lower layer (in the docs.UpdateReadmes()), so "" (empty string) won't leak here.

cmd/lint.go

internal/docs/readme.go

…package into additional_readmes

kaiyan-sheng · 2021-03-16T04:33:45Z

cmd/lint.go

+	outdatedStr := ""
+	errStr := ""
+	for _, f := range readmeFiles {
+		if !f.UpToDate {


I'm trying to capture all outdated file names here to display in the error message. It's not pretty again. Is there a better way to do so?

In the other comment I suggested to return an error if something bad happens.

You can check if docs.AreReadmesUpToDate() returned an error and any readmeFiles. If so, you can print the error and conditionally iterate over readmeFiles. No need to concatenate single error message. You can return a generic error and use cmd.Printf to iterate over problematic files. We used to mix cmd.Printf and errrors.Wrap for such cases, you can take a look at cmd/build.go and internal/builder/packages.go.

mtojek

I think we're close to the final form. Thanks for all adjustments!

internal/docs/readme.go

mtojek · 2021-03-16T09:10:39Z

cmd/lint.go

+	outdatedStr := ""
+	errStr := ""
+	for _, f := range readmeFiles {
+		if !f.UpToDate {


In the other comment I suggested to return an error if something bad happens.

You can check if docs.AreReadmesUpToDate() returned an error and any readmeFiles. If so, you can print the error and conditionally iterate over readmeFiles. No need to concatenate single error message. You can return a generic error and use cmd.Printf to iterate over problematic files. We used to mix cmd.Printf and errrors.Wrap for such cases, you can take a look at cmd/build.go and internal/builder/packages.go.

internal/docs/readme.go

mtojek · 2021-03-16T09:16:29Z

cmd/build.go

-		cmd.Printf("%s file rendered: %s\n", docs.ReadmeFile, target)
+
+	for _, target := range targets {
+		if target != "" {


Ah I see! It might be cleaner to skip this file on the lower layer (in the docs.UpdateReadmes()), so "" (empty string) won't leak here.

ycombinator

Pending the ongoing changes from @mtojek's review, this PR LGTM.

…package into additional_readmes

kaiyan-sheng · 2021-03-18T08:45:39Z

@mtojek cmd.Printf is exactly what I was looking for! Thank you!!

mtojek

Just a couple of nit-picks and you're good to go :)

Regarding the issue reported by CI:

test case failed: one or more problems with fields found in documents: [0] unexpected pipeline error: field [json] not present as part of path [json]

... I fixed it in the integrations repo: elastic/integrations#791 . Most likely you need to sync it here.

internal/docs/readme.go

mtojek

Well done! Ship it :) (just wait for the green status)

mtojek

@kaiyan-sheng could you please describe/justify the fix you applied to the PR to make it green? I'm curious what exactly was the root cause :)

mtojek · 2021-03-22T14:40:28Z

internal/testrunner/runners/system/runner.go

@@ -487,6 +487,7 @@ func createPackageDatastream(
 	// Add package-level vars
 	pkgVars := kibana.Vars{}
 	input := pkg.PolicyTemplates[0].FindInputByType(streamInput)
+	input.Vars = append(input.Vars, pkg.Vars...)


@ycombinator could you please verify this fix? I'm not quite sure about implications.

My understanding of this change is that all the package-level vars, defined in the package's manifest.yml file, are being copied into to each input's vars list.

From a system test definition perspective, this means that the top-level vars in the system test definition file could be used to override not just input vars but now also the package vars. So I think this change makes sense.

The only thing I would change is move this line inside the if input != nil check below.

Done! Thanks @ycombinator for the review!!

mtojek · 2021-03-22T14:40:52Z

internal/testrunner/runners/system/runner.go

@@ -487,6 +487,7 @@ func createPackageDatastream(
 	// Add package-level vars
 	pkgVars := kibana.Vars{}
 	input := pkg.PolicyTemplates[0].FindInputByType(streamInput)


This is something we'll have to look to support multiple policy templates.

kaiyan-sheng · 2021-03-22T15:10:05Z

@kaiyan-sheng could you please describe/justify the fix you applied to the PR to make it green? I'm curious what exactly was the root cause :)

Thanks for the review! This new AWS package manifest.yml file added all AWS credential variables under vars at the package level, which is the same level as policy templates. The current code doesn't read these package level variables so they are not passed into the agent policy. This last commit is to collect the package level variables and combine them with the input level variables under the policy template so these variables can be added into the agent policy:

inputs:
  - id: f45e9a6b-0fcf-4022-aa24-a3b43e1fdab1
    name: aws-ec2_metrics
    revision: 1
    type: aws/metrics
    use_output: default
    meta:
      package:
        name: aws
        version: 999.999.999
    data_stream:
      namespace: ep
    streams:
      - id: aws/metrics-aws.ec2_metrics-f45e9a6b-0fcf-4022-aa24-a3b43e1fdab1
        data_stream:
          dataset: aws.ec2_metrics
          type: metrics
        metricsets:
          - ec2
        period: 60s
        access_key_id: abcd
        secret_access_key: abcd
        session_token: >-
          abcd
        tags_filter:
          - key: Name
            value: elastic-package-test-78669

Check and generate multiple README.md files in one package

12a17ea

kaiyan-sheng self-assigned this Mar 9, 2021

kaiyan-sheng added the Team:Integrations Label for the Integrations team label Mar 9, 2021

kaiyan-sheng requested a review from mtojek March 9, 2021 14:24

mtojek requested a review from ycombinator March 9, 2021 14:42

mtojek reviewed Mar 9, 2021

View reviewed changes

cmd/build.go Outdated Show resolved Hide resolved

cmd/lint.go Outdated Show resolved Hide resolved

internal/docs/readme.go Outdated Show resolved Hide resolved

Merge branch 'master' into additional_readmes

a3886c0

ycombinator reviewed Mar 9, 2021

View reviewed changes

test/packages/aws/_dev/build/docs/billing.md Show resolved Hide resolved

ycombinator reviewed Mar 9, 2021

View reviewed changes

cmd/build.go Outdated Show resolved Hide resolved

kaiyan-sheng added 3 commits March 10, 2021 14:34

move ReadDir into IsReadmeUpToDate and UpdateReadme

ae666b8

Merge branch 'master' of github.com:elastic/elastic-package into addi…

0981abe

…tional_readmes

Merge branch 'additional_readmes' of github.com:kaiyan-sheng/elastic-…

d57e8ba

…package into additional_readmes

mtojek reviewed Mar 10, 2021

View reviewed changes

cmd/build.go Outdated Show resolved Hide resolved

cmd/lint.go Outdated Show resolved Hide resolved

internal/docs/readme.go Outdated Show resolved Hide resolved

internal/docs/readme.go Show resolved Hide resolved

kaiyan-sheng mentioned this pull request Mar 10, 2021

Adjust aws package to use input groups elastic/integrations#767

Merged

3 tasks

ycombinator reviewed Mar 10, 2021

View reviewed changes

internal/docs/readme.go Outdated Show resolved Hide resolved

ycombinator reviewed Mar 10, 2021

View reviewed changes

internal/docs/readme.go Outdated Show resolved Hide resolved

ycombinator reviewed Mar 10, 2021

View reviewed changes

internal/docs/readme.go Outdated Show resolved Hide resolved

ycombinator and others added 5 commits March 10, 2021 07:52

Merge branch 'master' into additional_readmes

7ef4eb5

Merge branch 'master' into additional_readmes

3e762da

Create AreReadmesUpToDate and UpdateReadmes functions

303bb12

print elastic-package build output in alphabetical order

17afb55

list all file names that need to be updated

4f603f8

make _dev/build/docs not required

5898df2

Merge branch 'master' into additional_readmes

6cf1ce4

ycombinator reviewed Mar 15, 2021

View reviewed changes

internal/docs/readme.go Outdated Show resolved Hide resolved

ycombinator reviewed Mar 15, 2021

View reviewed changes

internal/docs/readme.go Outdated Show resolved Hide resolved

ycombinator reviewed Mar 15, 2021

View reviewed changes

internal/docs/readme.go Outdated Show resolved Hide resolved

mtojek reviewed Mar 15, 2021

View reviewed changes

mtojek requested a review from ycombinator March 15, 2021 15:27

kaiyan-sheng added 3 commits March 16, 2021 11:38

check os.IsNotExist and use fileName instead

e5983ce

Merge branch 'additional_readmes' of github.com:kaiyan-sheng/elastic-…

380019c

…package into additional_readmes

fix error message

41c2ff2

kaiyan-sheng commented Mar 16, 2021

View reviewed changes

Merge branch 'master' into additional_readmes

9d9039c

mtojek reviewed Mar 16, 2021

View reviewed changes

ycombinator approved these changes Mar 16, 2021

View reviewed changes

kaiyan-sheng added 2 commits March 18, 2021 16:35

use cmd.Printf for error messages

9a85bb5

Merge branch 'additional_readmes' of github.com:kaiyan-sheng/elastic-…

da89e55

…package into additional_readmes

mtojek reviewed Mar 18, 2021

View reviewed changes

internal/docs/readme.go Outdated Show resolved Hide resolved

internal/docs/readme.go Outdated Show resolved Hide resolved

internal/docs/readme.go Outdated Show resolved Hide resolved

fix error messages

cb3e976

mtojek approved these changes Mar 18, 2021

View reviewed changes

kaiyan-sheng added 3 commits March 18, 2021 18:37

add ignore_missing

9ff5e57

rerun elastic-package test pipeline for aws

1673749

test with adding package level vars

d24311c

mtojek requested review from mtojek and ycombinator March 22, 2021 14:39

mtojek reviewed Mar 22, 2021

View reviewed changes

check input is nil before merging vars

17506c5

kaiyan-sheng merged commit 96ac7e9 into elastic:master Mar 23, 2021

kaiyan-sheng deleted the additional_readmes branch March 23, 2021 05:10

mtojek mentioned this pull request Mar 23, 2021

Meta: enable input groups elastic/package-spec#144

Closed

8 tasks

kaiyan-sheng mentioned this pull request Mar 23, 2021

Update dependency on elastic-package elastic/integrations#798

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Check and generate multiple readme files in one package #282

Check and generate multiple readme files in one package #282

kaiyan-sheng commented Mar 9, 2021 •

edited

Loading

elasticmachine commented Mar 9, 2021

elasticmachine commented Mar 9, 2021 •

edited

Loading

Build stats

Test stats 🧪

Trends 🧪

mtojek left a comment

kaiyan-sheng commented Mar 14, 2021

mtojek commented Mar 15, 2021

kaiyan-sheng commented Mar 15, 2021

mtojek Mar 15, 2021

kaiyan-sheng Mar 16, 2021

mtojek Mar 16, 2021

kaiyan-sheng Mar 16, 2021

mtojek Mar 16, 2021

mtojek left a comment

mtojek Mar 16, 2021

mtojek Mar 16, 2021

ycombinator left a comment

kaiyan-sheng commented Mar 18, 2021

mtojek left a comment

mtojek left a comment

mtojek left a comment

mtojek Mar 22, 2021

ycombinator Mar 22, 2021 •

edited

Loading

kaiyan-sheng Mar 23, 2021

mtojek Mar 22, 2021

kaiyan-sheng commented Mar 22, 2021

Check and generate multiple readme files in one package #282

Check and generate multiple readme files in one package #282

Conversation

kaiyan-sheng commented Mar 9, 2021 • edited Loading

elastic-package lint when billing.md and cloudtrail.md are missing

elastic-package build output

elastic-package lint when there is no file missing

elasticmachine commented Mar 9, 2021

elasticmachine commented Mar 9, 2021 • edited Loading

💚 Build Succeeded

Build stats

Test stats 🧪

Trends 🧪

mtojek left a comment

Choose a reason for hiding this comment

kaiyan-sheng commented Mar 14, 2021

mtojek commented Mar 15, 2021

kaiyan-sheng commented Mar 15, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mtojek left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ycombinator left a comment

Choose a reason for hiding this comment

kaiyan-sheng commented Mar 18, 2021

mtojek left a comment

Choose a reason for hiding this comment

mtojek left a comment

Choose a reason for hiding this comment

mtojek left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ycombinator Mar 22, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kaiyan-sheng commented Mar 22, 2021

kaiyan-sheng commented Mar 9, 2021 •

edited

Loading

`elastic-package lint` when `billing.md` and `cloudtrail.md` are missing

`elastic-package build` output

`elastic-package lint` when there is no file missing

elasticmachine commented Mar 9, 2021 •

edited

Loading

ycombinator Mar 22, 2021 •

edited

Loading