Upgrade path and tags database

[nickbabcock]

Coming from 1.0.2-2 and upgrading to 1.1.2-1, I see several errors related to tags:

curl -s "http://graphite/tags?pretty=1"

{
  "error": "no such table: tags_tag"
}

curl -X POST "http://graphite/tags/tagSeries" \
   --data-urlencode 'path=disk.used;rack=a1;datacenter=dc1;server=web01'
{"error": "no such table: tags_series"}

These are the tables I see in graphite.db. Since I did no configuring on my end I'm assuming this includes the local tags db.

account_mygraph             dashboard_dashboard_owners
account_profile             dashboard_template
account_variable            dashboard_template_owners
account_view                django_admin_log
account_window              django_content_type
auth_group                  django_migrations
auth_group_permissions      django_session
auth_permission             events_event
auth_user                   tagging_tag
auth_user_groups            tagging_taggeditem
auth_user_user_permissions  url_shortener_link
dashboard_dashboard

Is there a migration guide / am I missing something?

[deniszh]

@nickbabcock
I think you need to use separate database (MySQL, PostgreSQL) for any serious use of tags, I doubt that SQLite will able to handle an even minimal load.
But yes, database schema migration is needed, and there's no migration guide, despite official for Graphite 1.1.x - https://graphite.readthedocs.io/en/latest/releases/1_1_1.html#upgrading
You need to run something like

cp /opt/graphite/storage/graphite.db \
      /opt/graphite/storage/graphite.db.backup-`date +%Y%m%d_%H%M%S`
PYTHONPATH=/opt/graphite/webapp django-admin.py migrate \
      --noinput --settings=graphite.settings --run-syncdb

in the container for that.

[nickbabcock]

Interestingly that command returns:

Operations to perform:
  Synchronize unmigrated apps: browser, composer, functions, metrics, render, staticfiles, whitelist
  Apply all migrations: account, admin, auth, contenttypes, dashboard, events, sessions, tagging, tags, url_shortener
Synchronizing apps without migrations:
  Creating tables...
    Running deferred SQL...
Running migrations:
  Applying account.0001_initial...Traceback (most recent call last):
  File "/usr/local/bin/django-admin.py", line 5, in <module>
    management.execute_from_command_line()
// [Snip]
  File "/usr/local/lib/python2.7/dist-packages/django/db/backends/sqlite3/base.py", line 326, in execute
    return Database.Cursor.execute(self, query)
django.db.utils.OperationalError: table "account_mygraph" already exists

Since I only use graphite-web as a mean as interfacing with carbon with functions for grafana, a quick scan of graphite.db didn't reveal anything I'd care too much about if I dropped sqlite and made a fresh start with postgres.

[deniszh]

That's a good move, indeed - postgresql should run much better.

[nickbabcock]

I did have a couple questions. I noticed that in order to get the initial migrations to run for postgres (and I'm assuming any non-sqlite db) I had to force /usr/local/bin/manage.sh to run on every startup, which I believe should be ok as the migrations appear idempotent. Plus on future graphite db updates, I should be able to automatically migrate to them without exec-ing into the container. For whatever reason, 01_conf_init.sh didn't kick off the migration process.

FROM graphiteapp/graphite-statsd:1.1.3

COPY carbon.conf storage-schemas.conf /opt/graphite/conf/

# Copy over settings for SECRET_KEY and DATABASES
COPY local_settings.py /opt/graphite/webapp/graphite/

# Graphite does not come bundled with a python postgres driver so we manually
# pip install psycopg2. We pin to 2.7.4 for reproducibility.
RUN pip install psycopg2-binary==2.7.4

CMD /usr/local/bin/manage.sh && /sbin/my_init

And my compose file if necessary:

version: '3'
services:
  graphite:
    container_name: 'graphite'
    build: .
    restart: 'unless-stopped'
    ports:
      - '8181:80'
      - '2003:2003'
    links:
      - db
    # Snipped volumes
  db:
    container_name: 'graphite-db'
    image: 'postgres:10.3-alpine'
    restart: 'unless-stopped'
    # Snipped volumes
    environment:
      - POSTGRES_PASSWORD=graphitedbpassword

Do you see anything amiss?

[deniszh]

LGTM

[nickbabcock]

Excellent, works like a charm