Update from 0.13.7 to 0.14.0 crashes container

[WAdama]

Hi all,

after updating the syncstorage-rs container from 0.13.7 to 0.14.0 the container crashes.

The log is filled with this messages you see in the attached log.
FirefoxSync_RS_logs.txt

┆Issue is synchronized with this Jira Task

[jrconlin]

Looking at the log, I see that the problem is that there's an invalid database URL being specified. specifically:
SLaka5upLEa2RfW4XNMkUrRihKQNJKQFqAzR5McFSRpWcnrD7hcYRtxh37T9+@MariaDB:3306/syncstorage_rs
What's also curious is the fact that you've specified the db in settings as mysql:, but the code is trying to use spanner.

Are you compiling with --no-default-features --features=mysql?

[WAdama]

Hi,

I'm just using the docker image you provide. This is my compose file and the env file:

--------------------- Start
version: "3.8"

services:
firefox-sync:
image: mozilla/syncstorage-rs:0.14.0
container_name: ${CONTAINER_NAME}
environment:
SYNC_HOST: 0.0.0.0
SYNC_HUMAN_LOGS: 1
SYNC_MASTER_SECRET: ${SYNC_MASTER_SECRET}
SYNC_SYNCSTORAGE__DATABASE_URL: mysql://${MYSQL_USER}:${MYSQL_PASS}@${DATABASE_SERVER}:${DATABASE_PORT}/${DATABASE_SYNC}
SYNC_TOKENSERVER__ENABLED: "true"
SYNC_TOKENSERVER__RUN_MIGRATIONS: "true"
SYNC_TOKENSERVER__NODE_TYPE: mysql
SYNC_TOKENSERVER__DATABASE_URL: mysql://${MYSQL_USER}:${MYSQL_PASS}@${DATABASE_SERVER}:${DATABASE_PORT}/${DATABASE_TOKEN}
SYNC_TOKENSERVER__FXA_EMAIL_DOMAIN: api.accounts.firefox.com
SYNC_TOKENSERVER__FXA_OAUTH_SERVER_URL: https://oauth.accounts.firefox.com/v1
SYNC_TOKENSERVER__FXA_METRICS_HASH_SECRET: ${METRICS_HASH_SECRET}
SYNC_TOKENSERVER__ADDITIONAL_BLOCKING_THREADS_FOR_FXA_REQUESTS: 2
RUST_LOG: warn
volumes:
- /etc/localtime:/etc/localtime:ro

ports:
- ${EXTERNAL_PORT}:8000
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:8000/heartbeat"]
interval: 30s
timeout: 10s
retries: 5
restart: unless-stopped
networks:
- mariadb

networks:
mariadb:
external: true
------------------------- End

------------------------- Start
COMPOSE_PROJECT_NAME=ffsync_rs
CONTAINER_NAME=FirefoxSync_RS
MYSQL_USER=sync_rs
MYSQL_PASS=
SYNC_MASTER_SECRET=
METRICS_HASH_SECRET=
DATABASE_SERVER=MariaDB
DATABASE_PORT=3306
DATABASE_SYNC=syncstorage_rs
DATABASE_TOKEN=tokenserver_rs
EXTERNAL_PORT=49997
------------------------ End

[WAdama]

I've tried it again, after it's officially released. The error is the same.

What I've not registered before: The image for 0.14.0 is quite larger than for 0.13.7, 2.2 GB to 1.4 GB. Is that normal?

[jonasled]

I yesterday did also the update and I have the same problem, after updating the Container won't start anymore. I'm currently running the syncstorage on a Kubernetes cluster with this deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: syncstorage
  namespace: mozilla
  labels:
    app: syncstorage
spec:
  replicas: 1
  selector:
    matchLabels:
      app: syncstorage
  template:
    metadata:
      labels:
        app: syncstorage
    spec:
      containers:
        - name: syncstorage
          image: mozilla/syncstorage-rs:0.13.7
          ports:
            - containerPort: 8000
          env:
            - name: SYNC_HOST
              value: 0.0.0.0
            - name: SYNC_MASTER_SECRET
              valueFrom:
                secretKeyRef:
                  name: syncstorage
                  key: masterSecret
            - name: SYNC_SYNCSTORAGE__DATABASE_URL
              valueFrom:
                secretKeyRef:
                  name: syncstorage
                  key: syncstorageDatabaseUrl
            - name: SYNC_RUN_MIGRATIONS
              value: "true"

            - name: SYNC_TOKENSERVER__ENABLED
              value: "true"
            - name: SYNC_TOKENSERVER__RUN_MIGRATIONS
              value: "true"
            - name: SYNC_TOKENSERVER__NODE_TYPE
              value: "mysql"
            - name: SYNC_TOKENSERVER__DATABASE_URL
              valueFrom:
                secretKeyRef:
                  name: syncstorage
                  key: tokenserverDatabaseUrl
            - name: SYNC_TOKENSERVER__FXA_EMAIL_DOMAIN
              value: "api.accounts.firefox.com"
            - name: SYNC_TOKENSERVER__FXA_OAUTH_SERVER_URL
              value: "https://oauth.accounts.firefox.com/v1"
            - name: SYNC_TOKENSERVER__FXA_METRICS_HASH_SECRET
              valueFrom:
                secretKeyRef:
                  name: syncstorage
                  key: fxaMetricsHashSecret
            - name: SYNC_TOKENSERVER__ADDITIONAL_BLOCKING_THREADS_FOR_FXA_REQUESTS
              value: "2"
      tolerations:
        - key: "architecture"
          operator: "Equal"
          value: "x86"
          effect: "NoExecute"
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: kubernetes.io/arch
                operator: In
                values:
                - amd64

[WAdama]

And what has Vaultwarden to do with Firefox Sync? You're talking about two different products, which are doing different things...

[WAdama]

But that's not the main goal of Firefox Sync. Otherwise Vaultwarden / Bitwarden is "only" a password safe.

And this helps not in this case.

[maboxx]

Hi all,
I have noticed the same thing. Update from 0.13.6 or even an update from 0.13.7 to 0.14.0 does not work. The container does not start anymore.
Is there any news in the topic? I think the problem will still hit many users when they want to update.

[Unfaehig]

So the issue has to do with the syncstorage-mysql part.

In short the migrations (table creations/updates for the syncstorage db) never happen in an normal start up of this image.

Which is weird because the function to migrate a new dbs exists. see here At least used by some tests here
I'm wondering why there is no SYNC_SYNCSTORAGE__RUN_MIGRATIONS env while there is a SYNC_TOKENSERVER__RUN_MIGRATIONS for the tokenserver, which actually work fine. The image also start when we disable the syncstorage completely.
So please add the migration code on startup and may be throw better issues instead of just 500, when entries in db do not exist.

Let me know if I should raise a separate ticket, for the change thx!

[Unfaehig]

OK i need to do a step back. The braking change is this commit at Line 51. Like mentioned in in one of the issues, the Docker image expecting spanner instead of mysql today. In other words this was the day where the built docker images changed from mysql as default building feature to spanner.
Results in two different compiled versions. There are two possible solutions i see:

1. Creating a building pipeline, which builds on each release two images. Take Nextcloud as reference they use docker tags they regular updating like fpm, apache, fpm-alpine...
2. compiling a version including both features in one executable. Needs more work/changes and testing i guess

So i test building the current repo with docker build --build-arg="DATABASE_BACKEND=mysql" . this works like a charm and also the auto migration and population of the database works like expected. For me the solution 1 sounds the cleanest and would make self hosting much easier!

[jrconlin]

Again, I REALLY want to encourage folk to not use our Dockerfiles in any form of a production environment (That means anything other than "I want to test syncstorage-rs".)

What you probably want to do is make a copy and modify it for folk that ARE running "stand-alone". That means compiling with --no-default-features ---features=mysql, and pointing toward a config file that does things like NOT set SYNC_MASTER_SECRET=secret0 or whatever. (Your secret should be a long base64 compatible string (I'd recommend at least 32 octets/characters) that is (ideally) derived from some cryptographically secure source.
Even something dumb like

import os
import base64

print(base64.standard_b64encode(os.urandom(32)).decode('utf-8'))

I'd also recommend putting stuff into a /standalone directory. That way, there's less chance of someone accidentally changing or breaking things. (I could get moved off this project or something.)

[porelli]

Probably not the most elegant solution but I prepared a GitHub action to re-build the image as suggested above by @Unfaehig and @jrconlin. The image is rebuilt weekly on the latest tag available on this repo.

The image is published at: ghcr.io/porelli/firefox-sync:syncstorage-rs-mysql-latest
The repo with the instructions and deails are published at: https://github.com/porelli/firefox-sync