deb and rpm packaging for all binaries #6456

julienduchesne · 2022-06-22T02:00:05Z

There is currently deb and rpm packages being published for logcli amd64 but they aren't signed

In this PR:

Add RPM and deb packages for logcli, loki-canary loki and promtail for arm, arm64 and amd64
Sign them with the Grafana GPG key (packages.grafana.com/gpg.key). This will be the same key for all Grafana products
Add CI steps to test the RPM and deb packages. This launches a docker image with systemd installed, then it installs both loki and promtail and does a logcli query on the Loki instance

grafanabot · 2022-06-22T02:06:43Z

./tools/diff_coverage.sh ../loki-main/test_results.txt test_results.txt ingester,distributor,querier,querier/queryrange,iter,storage,chunkenc,logql,loki

Change in test coverage per package. Green indicates 0 or positive change, red indicates that test coverage for a package fell.

+           ingester	0%
+        distributor	0%
+            querier	0%
+ querier/queryrange	0.1%
+               iter	0%
+            storage	0%
+           chunkenc	0%
+              logql	0%
+               loki	0.6%

dannykopping

Thanks for this @julienduchesne! 🎉
@KMiller-Grafana could you please have a look at the verbiage in tools/package-nfpm.jsonnet to see if there's anything you'd like to change before I merge?

chaudum

Thanks for your efforts to make OS packages happen also for Loki. However I have a big concern which I raised in a comment.

tools/package-nfpm.jsonnet

jeschkies

Thanks for getting this started 🙂

tools/package-nfpm.jsonnet

KMiller-Grafana

File contents for tools/packaging/nfpm.jsonnet look good to me.

julienduchesne · 2022-06-28T02:39:13Z

@jeschkies @chaudum. Re-requesting a review. I added systemd configs for both deb and RPM. I also added a CI test to check that everything keeps working. The CI check uses CentOS and Debian. It installs loki and promtail and then queries with logcli to see that everything works well

grafanabot · 2022-06-28T02:46:37Z

./tools/diff_coverage.sh ../loki-main/test_results.txt test_results.txt ingester,distributor,querier,querier/queryrange,iter,storage,chunkenc,logql,loki

Change in test coverage per package. Green indicates 0 or positive change, red indicates that test coverage for a package fell.

+           ingester	0%
+        distributor	0%
+            querier	0%
+ querier/queryrange	0.1%
+               iter	0%
+            storage	0%
+           chunkenc	0%
+              logql	0%
+               loki	0%

jeschkies

This is really awesome! Just so I understand. The packages are not published yet, right?

jeschkies · 2022-06-28T15:13:39Z

.drone/drone.jsonnet

+          }),
+      run('test packaging',
+          commands=[
+            'go install github.com/google/go-jsonnet/cmd/jsonnet@latest',  // Test, install in build image instead


I'm not sure I understand this step. Why do you install a Go dependency here?

Installed in the build image here: #6526

jeschkies · 2022-06-28T15:14:18Z

.drone/drone.jsonnet

+      {
+        name: 'test deb package',
+        image: 'docker',
+        commands: ['./tools/packaging/verify-deb-install.sh'],


This is awesome. I'd use some test library but am totally fine with a shell script for now.

jeschkies · 2022-06-28T15:16:46Z

tools/packaging/nfpm.sh

+for name in loki loki-canary logcli promtail; do
+    for arch in amd64 arm64 arm; do
+        config_path="dist/tmp/config-${name}-${arch}.json"
+        jsonnet -V "name=${name}" -V "arch=${arch}" "tools/packaging/nfpm.jsonnet" > "${config_path}"


...ah this is why you need to install Jsonnet 🙂

jeschkies · 2022-06-28T15:18:12Z

tools/packaging/verify-deb-install.sh

+
+cat <<EOF | docker exec --interactive "${image}" sh
+    # Install loki and check it's running
+    dpkg -i ${dir}/dist/loki_0.0.0~rc0_amd64.deb


Shouldn't the binary have a version?

oh dang, that'll fail on actual releases. Will fix

jeschkies · 2022-06-28T15:18:38Z

tools/packaging/verify-deb-install.sh

+    sleep 5
+    labels_found=\$(logcli labels)
+    echo "Found labels: \$labels_found"
+    [ "\$labels_found" != "" ] || (echo "no logs found with logcli" && exit 1)


I'd use a loop but this is a great start. We can always iterate.

julienduchesne · 2022-06-28T15:21:40Z

This is really awesome! Just so I understand. The packages are not published yet, right?

No, we're working on a global repository (apt.grafana.com and rpm.grafana.com) where we will publish all packages. It'll just be an additional Drone step. For now, the deb and rpm files will be in the releases. I'm hoping to have the repositories ready before end of July

Needed for #6456

grafanabot · 2022-06-28T16:07:05Z

./tools/diff_coverage.sh ../loki-main/test_results.txt test_results.txt ingester,distributor,querier,querier/queryrange,iter,storage,chunkenc,logql,loki

Change in test coverage per package. Green indicates 0 or positive change, red indicates that test coverage for a package fell.

-           ingester	-0.1%
-        distributor	-0.3%
+            querier	0%
+ querier/queryrange	0%
+               iter	0%
+            storage	0%
+           chunkenc	0%
+              logql	0%
+               loki	0%

rightaway · 2022-06-29T08:08:17Z

@julienduchesne The debian package will install directly on the system not through docker?

Where can I see what the .service files will look like on debian for loki and promtail packages?

chaudum · 2022-06-29T08:09:32Z

tools/packaging/nfpm.jsonnet

+        dst: '/etc/systemd/system/loki.service',
+      },
+      {
+        src: './cmd/loki/loki-local-config.yaml',


This config file configures Loki to write to /tmp/loki. Not sure if this is the ideal default location, but I think it's good enough for a first iteration.

* Add jsonnet to build image Needed for #6456 * Add pipeline step to build image

grafanabot · 2022-07-05T12:05:58Z

./tools/diff_coverage.sh ../loki-main/test_results.txt test_results.txt ingester,distributor,querier,querier/queryrange,iter,storage,chunkenc,logql,loki

Change in test coverage per package. Green indicates 0 or positive change, red indicates that test coverage for a package fell.

+           ingester	0%
+        distributor	0.3%
+            querier	0%
+ querier/queryrange	0%
+               iter	0%
+            storage	0%
+           chunkenc	0%
+              logql	0%
+               loki	0%

MAFLO321 · 2022-07-05T16:32:35Z

tools/packaging/promtail.service

+Type=simple
+User=promtail
+ExecStart=/usr/bin/promtail -config.file /etc/promtail/config.yml
+


Wouldn't it be a neat idea if promtail would also be Restart = on-failure, just like in loki.service?

There is currently deb and rpm packages being published for logcli amd64 but they aren't signed In this PR: - Add RPM and deb packages for `logcli`, `loki-canary` `loki` and `promtail` for arm, arm64 and amd64 - Sign them with the Grafana GPG key (packages.grafana.com/gpg.key). This will be the same key for all Grafana products - Add CI steps to test the RPM and deb packages. This launches a docker image with systemd installed, then it installs both loki and promtail and does a `logcli` query on the Loki instance

grafanabot · 2022-08-01T13:06:29Z

./tools/diff_coverage.sh ../loki-main/test_results.txt test_results.txt ingester,distributor,querier,querier/queryrange,iter,storage,chunkenc,logql,loki

Change in test coverage per package. Green indicates 0 or positive change, red indicates that test coverage for a package fell.

+           ingester	0%
+        distributor	0%
+            querier	0%
+ querier/queryrange	0%
+               iter	0%
+            storage	0%
+           chunkenc	0%
+              logql	0%
+               loki	0%

* deb and rpm packaging for all binaries There is currently deb and rpm packages being published for logcli amd64 but they aren't signed In this PR: - Add RPM and deb packages for `logcli`, `loki-canary` `loki` and `promtail` for arm, arm64 and amd64 - Sign them with the Grafana GPG key (packages.grafana.com/gpg.key). This will be the same key for all Grafana products - Add CI steps to test the RPM and deb packages. This launches a docker image with systemd installed, then it installs both loki and promtail and does a `logcli` query on the Loki instance * Remove specific version in test scripts * Bump build image to 0.22.0 * Add restart policy for promtail's service

**What this PR does / why we need it**: With #6456 we started providing native packages of Loki, Promtail and logcli. They've been added to Grafana's repository in the meantime. In order to spread the knowledge of the packages and the repo it should be documented. **Which issue(s) this PR fixes**: Fixes #52 **Checklist** - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [x] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` --------- Co-authored-by: René Scheibe <rene.scheibe@gmail.com> Co-authored-by: J Stickler <julie.stickler@grafana.com>

julienduchesne requested a review from a team as a code owner June 22, 2022 02:00

pull-request-size bot added the size/M label Jun 22, 2022

dannykopping approved these changes Jun 22, 2022

View reviewed changes

dannykopping requested a review from KMiller-Grafana June 22, 2022 08:02

chaudum requested changes Jun 22, 2022

View reviewed changes

tools/package-nfpm.jsonnet Outdated Show resolved Hide resolved

jeschkies reviewed Jun 22, 2022

View reviewed changes

tools/package-nfpm.jsonnet Outdated Show resolved Hide resolved

julienduchesne force-pushed the julienduchesne/packaging-deb-rpm-all-products branch from bdea709 to 9868c51 Compare June 27, 2022 16:57

pull-request-size bot added size/L and removed size/M labels Jun 27, 2022

julienduchesne force-pushed the julienduchesne/packaging-deb-rpm-all-products branch 3 times, most recently from 2f4fb48 to 505e063 Compare June 27, 2022 17:15

julienduchesne marked this pull request as draft June 27, 2022 17:21

KMiller-Grafana approved these changes Jun 27, 2022

View reviewed changes

julienduchesne force-pushed the julienduchesne/packaging-deb-rpm-all-products branch from 60d654a to 49b73e3 Compare June 28, 2022 02:32

pull-request-size bot added size/XL and removed size/L labels Jun 28, 2022

julienduchesne requested review from chaudum, jeschkies and a team June 28, 2022 02:39

julienduchesne marked this pull request as ready for review June 28, 2022 02:39

julienduchesne force-pushed the julienduchesne/packaging-deb-rpm-all-products branch from 49b73e3 to bfa9c6a Compare June 28, 2022 02:41

jeschkies mentioned this pull request Jun 28, 2022

Promtail Packages #52

Closed

jeschkies approved these changes Jun 28, 2022

View reviewed changes

julienduchesne added a commit that referenced this pull request Jun 28, 2022

Add jsonnet to build image

e1a9c0c

Needed for #6456

julienduchesne mentioned this pull request Jun 28, 2022

Add jsonnet to build image #6526

Merged

chaudum approved these changes Jun 29, 2022

View reviewed changes

julienduchesne added a commit that referenced this pull request Jun 30, 2022

Add jsonnet to build image (#6526)

d61b0fe

* Add jsonnet to build image Needed for #6456 * Add pipeline step to build image

julienduchesne force-pushed the julienduchesne/packaging-deb-rpm-all-products branch 2 times, most recently from 0b98a00 to faf6c92 Compare July 5, 2022 12:01

MAFLO321 reviewed Jul 5, 2022

View reviewed changes

julienduchesne added 4 commits August 1, 2022 08:58

Remove specific version in test scripts

9d4595a

Bump build image to 0.22.0

7f3fe14

Add restart policy for promtail's service

dcc0d74

julienduchesne force-pushed the julienduchesne/packaging-deb-rpm-all-products branch from faf6c92 to dcc0d74 Compare August 1, 2022 13:00

julienduchesne merged commit 7b70d3d into main Aug 1, 2022

julienduchesne deleted the julienduchesne/packaging-deb-rpm-all-products branch August 1, 2022 13:22

MAFLO321 mentioned this pull request Nov 15, 2022

APT package repo does not contain loki and promtail packages #7705

Open

DeadNews mentioned this pull request Nov 22, 2022

Chenge default data location to /var/lib/loki #7748

Open

jeschkies mentioned this pull request Mar 20, 2023

Document how to install Loki and Promtail using APT/DNF. #8841

Merged

5 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

deb and rpm packaging for all binaries #6456

deb and rpm packaging for all binaries #6456

julienduchesne commented Jun 22, 2022 •

edited

Loading

grafanabot commented Jun 22, 2022

dannykopping left a comment

chaudum left a comment

jeschkies left a comment

KMiller-Grafana left a comment

julienduchesne commented Jun 28, 2022

grafanabot commented Jun 28, 2022

jeschkies left a comment

jeschkies Jun 28, 2022

julienduchesne Jun 28, 2022

jeschkies Jun 28, 2022

jeschkies Jun 28, 2022

jeschkies Jun 28, 2022

julienduchesne Jun 28, 2022

jeschkies Jun 28, 2022

julienduchesne commented Jun 28, 2022 •

edited

Loading

grafanabot commented Jun 28, 2022

rightaway commented Jun 29, 2022

chaudum Jun 29, 2022

grafanabot commented Jul 5, 2022

MAFLO321 Jul 5, 2022

grafanabot commented Aug 1, 2022

deb and rpm packaging for all binaries #6456

deb and rpm packaging for all binaries #6456

Conversation

julienduchesne commented Jun 22, 2022 • edited Loading

grafanabot commented Jun 22, 2022

dannykopping left a comment

Choose a reason for hiding this comment

chaudum left a comment

Choose a reason for hiding this comment

jeschkies left a comment

Choose a reason for hiding this comment

KMiller-Grafana left a comment

Choose a reason for hiding this comment

julienduchesne commented Jun 28, 2022

grafanabot commented Jun 28, 2022

jeschkies left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

julienduchesne commented Jun 28, 2022 • edited Loading

grafanabot commented Jun 28, 2022

rightaway commented Jun 29, 2022

Choose a reason for hiding this comment

grafanabot commented Jul 5, 2022

Choose a reason for hiding this comment

grafanabot commented Aug 1, 2022

julienduchesne commented Jun 22, 2022 •

edited

Loading

julienduchesne commented Jun 28, 2022 •

edited

Loading