Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[TUF autoupdater] Check out latest #1185

Merged
merged 64 commits into from
May 19, 2023

Conversation

RebeccaMahany
Copy link
Contributor

@RebeccaMahany RebeccaMahany commented May 8, 2023

Relates to #954.

This is the next step toward using our new TUF infrastructure: a new implementation of findNew that looks at our new update library.

Includes some drive-by changes:

  • I've increased the download timeout from 5 to 8 minutes. I've seen just a couple download timeouts in the kolide_tuf_autoupdater_errors table that I'm hoping this will resolve.
  • Adds a GetChannelVersionFromTufServer to be used by launcher doctor and by packaging changes for Make launcher available per-architecture #1149
How we pick which version of launcher/osqueryd to run

This flowchart is also committed to the docs in this PR.

flowchart TB
    A[Library lookup] --> B{Do we have a local TUF repo?}
    B ---->|No| C[Get most recent version from update library]
    C --> D[Return path to most recent version of executable]
    D --> H[End]
    B -->|Yes| E{release.json target metadata exists?}
    E -->|No| C
    E --> |Yes| F{Target indicated by release.json\nis downloaded to update library?}
    F --> |Yes| G[Return path to selected executable in update library]
    F --> |No| C
    G --> H
Loading
Future work + linked PRs

Subsequent PRs will tackle the following (order is not set in stone):

  1. Point to release file <binary>/<os>/<arch>/<channel>/release.json, once available (in progress but on hold here: [TUF autoupdater] Include arch in release file path and download file path #1195)
  2. Trigger a reload/restart after a successful update; launcher init performs version selection using the new update library (potentially falling back to findNew); launcher performs library tidying after version selection
  3. Other improvements/refactors (ship a more up-to-date TUF repo, consolidate TUF-heavy work into a separate client for ease of readability)
  4. An "update now" functionality tied to control server
  5. Eventually removing the old notary autoupdater

Previous work:

  1. Run new TUF autoupdater side-by-side with notary autoupdater #1081
  2. Expose data and metrics about new TUF autoupdater #1103
  3. Point to production TUF infra #1108
  4. Perform retry on TUF update #1110
  5. Add library manager to handle TUF downloads #1111
  6. Small autoupdate improvements #1119
  7. Pass knapsack in to tuf autoupdater to simplify configuration #1168
  8. [TUF autoupdater] Remove osquery client dependency #1178
  9. Move TUF test setup to new package #1188

@RebeccaMahany RebeccaMahany marked this pull request as ready for review May 12, 2023 18:48
seejdev
seejdev previously approved these changes May 15, 2023
@directionless
Copy link
Contributor

I was slacking about this. It feels confusing and complex.

@RebeccaMahany
Copy link
Contributor Author

I was slacking about this. It feels confusing and complex.

@directionless I think this is simpler and more lightweight now -- let me know what you think.

Copy link
Contributor

@James-Pickett James-Pickett left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I lack the understanding to review the code on a macro level. I'll try to look again later. Left some "micro" comments =)

pkg/autoupdate/tuf/library_lookup.go Outdated Show resolved Hide resolved
pkg/autoupdate/tuf/library_lookup.go Outdated Show resolved Hide resolved
pkg/autoupdate/tuf/library_lookup.go Outdated Show resolved Hide resolved
Copy link
Contributor

@directionless directionless left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is mostly okay. (See questions)

this feels much more readable than the prior. I suspect some of this should land in the library manager, and we should call it. Having this iterate files on disk feels wrong. (What happens if we move the library into ram?) But we can iterate there if we want to.

pkg/autoupdate/tuf/read_only_tuf_client.go Show resolved Hide resolved
pkg/autoupdate/tuf/read_only_tuf_client.go Show resolved Hide resolved
pkg/autoupdate/tuf/channel_version.go Outdated Show resolved Hide resolved
pkg/autoupdate/tuf/library_lookup.go Outdated Show resolved Hide resolved
pkg/autoupdate/tuf/library_lookup.go Outdated Show resolved Hide resolved
Comment on lines 39 to 47
targets, err := metadataClient.Targets()
if err != nil {
return "", "", fmt.Errorf("could not get target: %w", err)
}

targetName, _, err := findRelease(binary, targets, channel)
if err != nil {
return "", "", fmt.Errorf("could not find release: %w", err)
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is okay, but it feels weird. These are functions that are hard to follow, and work a lot with github.com/theupdateframework/go-tuf/data. It feels like we should bake them into something like metadataClient. Not sure it's worth it though

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think there are a couple places in autoupdate.go that feel similar, where it's a lot of work with go-tuf/data. Maybe could be wrapped in a tufReleaseClient (or something) that performs the repeated tasks (update from remote TUF repo to download updated targets, find release target among those targets, find what target it's pointing at, return that target). I could do that here, or add it to the TODOs for a subsequent PR if we want to think about it more.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Your call. I think this is okay, but it's still murky feeling.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll leave as-is for now, I guess, and think on it

pkg/autoupdate/tuf/library_lookup.go Outdated Show resolved Hide resolved
Copy link
Contributor

@directionless directionless left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I know this had a lot of iteration. It seems like it's getting pretty clean

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants