Run new TUF autoupdater side-by-side with notary autoupdater #1081

RebeccaMahany · 2023-03-07T23:33:40Z

Relates to #954.

This is part 1 of moving to use our new TUF infrastructure; here, we run our new TUF autoupdater side-by-side with our older notary autoupdater implementation. The new autoupdater only maintains a TUF repo but does not attempt to perform updates yet.

Subsequent PRs will tackle the following (order is not set in stone)

Improving TUF autoupdater metrics and exposing them in a launcher table
Point to production TUF instead of development
Download, verify, and untar downloads from new TUF, moving them to a new update directory located in the root directory
findnew looks for updates in new location, falling back to old location
Other improvements/refactors
An "update now" functionality tied to control server
Eventually removing the old notary autoupdater

…om new one yet

…sable

… rolling error count

…oupdater

cmd/launcher/internal/updater/updater.go

pkg/autoupdate/tuf_autoupdate.go

pkg/make/builder.go

cmd/launcher/internal/updater/updater.go

pkg/autoupdate/tuf_autoupdate.go

directionless · 2023-03-10T00:25:55Z

pkg/autoupdate/tuf_autoupdate.go

+	strippedBinary := strings.TrimSuffix(ta.binary, ".exe")
+
+	// The target is in the form `launcher/linux/launcher-0.13.6.tar.gz` -- trim the prefix and the file extension to return the version
+	prefixToTrim := fmt.Sprintf("%s/%s/%s-", strippedBinary, ta.operatingSystem, strippedBinary)


Urk. I've lost site of how we track archtecture

I don't believe it's present in the TUF/dl.kolide.co stuff

I might have dropped it. (oops)

We want to start supporting windows and linux arm64. And I think we'll eventually want to stop shipping universal binaries.

I'm not sure if we should roll it into the platform specification, or think harder

I also think this is something we'll need. There have been a few times that folks have asked for arm64 on linux / windows. I think number will continue to grow. @RebeccaMahany , do you think this would be difficult to bolt on later?

IIRC we'd decided to punt architecture to later when I was working on the endpoint-releaser work.

Here is a fairly simple way I could imagine us implementing this in the future:

endpoint-releaser begins to publish both to binary/os/file.tar.gz and binary/os/arch/file.tar.gz

endpoint-releaser begins to maintain a binary/os/arch/channel/release.json file to point to architecture-specific releases, in addition to the binary/os/channel/release.json it already maintains

launcher's new autoupdater looks for its release in binary/os/arch/channel/release.json, falling back to preexisting binary/os/channel/release.json if it can't find it for whatever reason

So, I think it wouldn't be too bad?

pkg/autoupdate/tuf/autoupdate.go

…pdate test for same; add more docs

directionless

I think there's going to be room for simplification, but we can use this for now!

pkg/autoupdate/tuf/autoupdate.go

directionless · 2023-03-14T20:10:15Z

pkg/autoupdate/tuf/autoupdate.go

+	}
+}
+
+func NewTufAutoupdater(metadataUrl, binary, rootDirectory string, metadataHttpClient *http.Client, opts ...TufAutoupdaterOption) (*TufAutoupdater, error) {


I suspect we can remove option. But let's go with it and iterate

RebeccaMahany added 5 commits March 7, 2023 18:33

First pass at running new tuf client side-by-side with notary impl

c453221

Merge branch 'main' into becca/tuf

ca9742e

Generate root.json

e43ca07

Test for switching between updater and fallback updater

53141fb

Make TUF server URL a command-line flag

1e3165e

RebeccaMahany changed the title ~~Draft: run new TUF client side-by-side with notary client~~ Draft: Run new TUF autoupdater side-by-side with notary autoupdater Mar 8, 2023

RebeccaMahany added 12 commits March 8, 2023 11:15

Rename TUF autoupdater

95c8ac6

Add options to configure TUF autoupdater

f42e577

const for error count threshold

51777c5

Run new and old autoupdaters alongside each other; don't do update fr…

e021a99

…om new one yet

go mod tidy

9e6ceeb

Simplify error counting for now

82c286d

Log release version

95e47ca

Small fixes, begin unit tests

71e2549

Add TUF tests for updating local metadata; make threadsafe buffer reu…

662cfb8

…sable

Remove unused const; add test for timestamp checking when calculating…

beb997d

… rolling error count

Ensure failure to initialize TUF autoupdater does not stop legacy aut…

3dacb90

…oupdater

Merge branch 'main' into becca/tuf

3e2f3bc

RebeccaMahany force-pushed the becca/tuf branch from e059d23 to 3e2f3bc Compare March 9, 2023 17:03

RebeccaMahany added 5 commits March 9, 2023 12:13

Fix how we include root.json asset

f6bbb7b

Lock for error counter

2a44428

Rename TUF repo

527a878

fix up some tests

f098d4b

lint

7ce15cf

RebeccaMahany changed the title ~~Draft: Run new TUF autoupdater side-by-side with notary autoupdater~~ Run new TUF autoupdater side-by-side with notary autoupdater Mar 9, 2023

RebeccaMahany added 3 commits March 9, 2023 15:57

Merge branch 'main' into becca/tuf

fc11378

debug windows test

9847a7c

remove assertexpectations for the moment

30a2932

directionless reviewed Mar 10, 2023

View reviewed changes

Remove mirror client/url since we don't need it yet

712e95d

Remove new autoupdater from cmd/launcher/internal/updater

6b6be3f

RebeccaMahany force-pushed the becca/tuf branch from 3d9f769 to 6b6be3f Compare March 13, 2023 19:35

RebeccaMahany added 2 commits March 13, 2023 16:17

Actually fix test on windows

34eeb99

Log binary as well

37b8a05

RebeccaMahany marked this pull request as ready for review March 13, 2023 20:47

Add TUF url to ip lookups in checkpointer

3418978

RebeccaMahany force-pushed the becca/tuf branch from e08d16f to 3418978 Compare March 13, 2023 20:54

James-Pickett reviewed Mar 13, 2023

View reviewed changes

pkg/autoupdate/tuf/autoupdate.go Show resolved Hide resolved

James-Pickett reviewed Mar 14, 2023

View reviewed changes

pkg/autoupdate/tuf/autoupdate.go Outdated Show resolved Hide resolved

pkg/autoupdate/tuf/autoupdate.go Outdated Show resolved Hide resolved

pkg/autoupdate/tuf/autoupdate.go Show resolved Hide resolved

Use const for tuf directory name that we will need to update later; u…

0323c0f

…pdate test for same; add more docs

RebeccaMahany mentioned this pull request Mar 14, 2023

Expose data and metrics about new TUF autoupdater #1103

Merged

James-Pickett previously approved these changes Mar 14, 2023

View reviewed changes

directionless previously approved these changes Mar 14, 2023

View reviewed changes

Don't need to export default channel const

e3141a2

RebeccaMahany dismissed stale reviews from directionless and James-Pickett via e3141a2 March 14, 2023 20:22

directionless approved these changes Mar 14, 2023

View reviewed changes

RebeccaMahany merged commit 5126b12 into kolide:main Mar 14, 2023

RebeccaMahany deleted the becca/tuf branch March 14, 2023 20:44

This was referenced Mar 22, 2023

Point to production TUF infra #1108

Merged

Add library manager to handle TUF downloads #1111

Merged

RebeccaMahany mentioned this pull request Apr 6, 2023

Small autoupdate improvements #1119

Merged

This was referenced Apr 28, 2023

Draft: TUF find new, v2 #1165

Closed

Pass knapsack in to tuf autoupdater to simplify configuration #1168

Merged

[TUF autoupdater] Remove osquery client dependency #1178

Merged

RebeccaMahany mentioned this pull request May 8, 2023

[TUF autoupdater] Check out latest #1185

Merged

RebeccaMahany mentioned this pull request May 22, 2023

[TUF autoupdater] Use new autoupdate code for internal testing #1203

Closed

RebeccaMahany mentioned this pull request Jul 26, 2023

[TUF autoupdater] Devices on nightly channel use new autoupdate library #1268

Merged

18 tasks

This was referenced Aug 29, 2023

Small cleanup for initial TUF rollout #1305

Merged

Create a temp staging dir per-download #1306

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Run new TUF autoupdater side-by-side with notary autoupdater #1081

Run new TUF autoupdater side-by-side with notary autoupdater #1081

RebeccaMahany commented Mar 7, 2023 •

edited

Loading

directionless Mar 10, 2023

RebeccaMahany Mar 10, 2023

directionless Mar 10, 2023

James-Pickett Mar 14, 2023

RebeccaMahany Mar 14, 2023 •

edited

Loading

directionless left a comment

directionless Mar 14, 2023

Run new TUF autoupdater side-by-side with notary autoupdater #1081

Run new TUF autoupdater side-by-side with notary autoupdater #1081

Conversation

RebeccaMahany commented Mar 7, 2023 • edited Loading

directionless Mar 10, 2023

Choose a reason for hiding this comment

RebeccaMahany Mar 10, 2023

Choose a reason for hiding this comment

directionless Mar 10, 2023

Choose a reason for hiding this comment

James-Pickett Mar 14, 2023

Choose a reason for hiding this comment

RebeccaMahany Mar 14, 2023 • edited Loading

Choose a reason for hiding this comment

directionless left a comment

Choose a reason for hiding this comment

directionless Mar 14, 2023

Choose a reason for hiding this comment

RebeccaMahany commented Mar 7, 2023 •

edited

Loading

RebeccaMahany Mar 14, 2023 •

edited

Loading