Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Adding a metadata file to KubeHound dumps #247

Merged
merged 16 commits into from
Sep 10, 2024
Merged

Adding a metadata file to KubeHound dumps #247

merged 16 commits into from
Sep 10, 2024

Conversation

edznux-dd
Copy link
Contributor

This is barely a PoC.

TLDR:
To simplify user experience, especially for the local dumper/ingestor, we should write a metadata.json file that contains the metadata of a specific dump.

Why:
Stuffing data in the filepath as some limitation:

  • There's a limit of number of element in the data path (limited by the number of characters for each filename)
  • Force the user to never rename the archive, which can be an issue when storing historical dumps (currently, the latest dump is computed by the file last modified time)
  • Parsing the data can become more cumbersome than simply parsing a json file containing each value in the correct format.

How:
Add a metadata.json file that contains all the metadata around a specific dump.
It will then be reused by the ingestor to populate the RunID, the ClusterName and any future info we want.

@edznux-dd edznux-dd mentioned this pull request Aug 19, 2024
Closed
edznux-dd
edznux-dd commented Sep 6, 2024
View reviewed changes
pkg/collector/file.go Outdated Show resolved Hide resolved
edznux-dd
edznux-dd commented Sep 6, 2024
View reviewed changes
pkg/kubehound/core/core_ingest_local.go Outdated Show resolved Hide resolved
pkg/dump/writer/file_writer.go Outdated Show resolved Hide resolved
edznux-dd
edznux-dd commented Sep 6, 2024
View reviewed changes
pkg/collector/k8s_api.go Outdated Show resolved Hide resolved
edznux-dd
edznux-dd commented Sep 6, 2024
View reviewed changes
pkg/dump/result.go Outdated Show resolved Hide resolved
pkg/dump/pipeline/metadata.go
}

func (d *MetadataIngestor) DumpMetadata(ctx context.Context, metadata collector.Metadata) error {
data := make(map[string]*collector.Metadata)
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it expected that this is a pointer to the collector metadata ? 😅

The buffer above is make(map[string]collector.Metadata),

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no it is not 😅

pkg/kubehound/core/core_ingest_local.go Outdated Show resolved Hide resolved
@jt-dd jt-dd marked this pull request as ready for review September 10, 2024 09:56
@jt-dd jt-dd requested a review from a team as a code owner September 10, 2024 09:56
@jt-dd jt-dd changed the title [WIP] Moving to metadata file instead of meaningful filepath Adding a metadata file to KubeHound dumps Sep 10, 2024
@jt-dd jt-dd merged commit fdafcbe into main Sep 10, 2024
8 checks passed
@jt-dd jt-dd deleted the edouard/kh-metadata-file branch September 10, 2024 12:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jt-dd jt-dd jt-dd approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@edznux-dd @jt-dd