Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Initial db migration failure causes trotto app to return 500 internal server error #63

Open
kentoku24 opened this issue Jan 20, 2021 · 5 comments

Comments

@kentoku24
Copy link

The symptom and steps to reproduce:

  1. Launch trotto instance.
  2. Launch postgres instance.
  3. Trotto instance keeps returning 500 internal server error.

Root cause:
When trotto app and postgres instance are written in one docker-compose file,
both are launched almost simultaneously.

Solution:
a: add health check to trotto instance, so it will eventually restarted, and DB migration after restart should success.
b: add retry capability to the initial database migration process.
c: inform users to make sure to launch database prior to trotto instance.

@itsojon
Copy link
Collaborator

itsojon commented Jan 21, 2021

Thank you for submitting this, @kentoku24! I think a lazy retry could work well. Did you try any of these solutions in your own branch?

@mikedodge04
Copy link

Im seeing the same issue. @kentoku24 , were you able to get it working?

@mikedodge04
Copy link

After trying one of the suggested fix im still getting the same error. Im not sure whats going on. Below is my repro case.

this is my app.yml:
sessions_secret: QinOf8Z5W/oiTS5uJ/=
postgres:
url: "postgresql://example:example@db/golinks"
admins:

Docker-compose:
version: "3.5"
services:
go-links:
image: trotto:1.0
depends_on:
db:
condition: service_healthy
ports:
- "9095:8000"
networks:
- traefik
- internal
volumes:
- /etc/localtime:/etc/localtime:ro
- ${DATA_DIR}/go-links:/data
restart: always

db:
image: postgres
env_file:
- .env
environment:
POSTGRES_DB: golinks
POSTGRES_USER: example
POSTGRES_PASSWORD: example
networks:
- internal
healthcheck:
test: [ "CMD", "pg_isready", "-q", "-d", "golinks", "-U", "example" ]
timeout: 45s
interval: 10s
retries: 10
volumes:
- /etc/localtime:/etc/localtime:ro
- ${DATA_DIR}/go-links/db:/var/lib/postgresql/data/
restart: always

networks:
traefik:
external: true
internal:
internal: true

Logs During Start UP
go-links_db :
today at 4:08:12 PM
today at 4:08:12 PM PostgreSQL Database directory appears to contain a database; Skipping initialization
today at 4:08:12 PM
today at 4:08:12 PM 2021-05-14 16:08:12.914 PDT [1] LOG: starting PostgreSQL 13.2 (Debian 13.2-1.pgdg100+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 8.3.0-6) 8.3.0, 64-bit
today at 4:08:12 PM 2021-05-14 16:08:12.914 PDT [1] LOG: listening on IPv4 address "0.0.0.0", port 5432
today at 4:08:12 PM 2021-05-14 16:08:12.914 PDT [1] LOG: listening on IPv6 address "::", port 5432
today at 4:08:12 PM 2021-05-14 16:08:12.917 PDT [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
today at 4:08:12 PM 2021-05-14 16:08:12.924 PDT [27] LOG: database system was shut down at 2021-05-14 16:08:03 PDT
today at 4:08:12 PM 2021-05-14 16:08:12.929 PDT [1] LOG: database system is ready to accept connections

go-links_web:
today at 4:08:23 PM ../scripts/upgrade_db.sh: 1: cd: can't cd to server/src
today at 4:08:24 PM INFO [alembic.runtime.migration] Context impl PostgresqlImpl.
today at 4:08:24 PM INFO [alembic.runtime.migration] Will assume transactional DDL.
today at 4:08:24 PM [2021-05-14 16:08:24 -0700] [15] [INFO] Starting gunicorn 20.0.4
today at 4:08:24 PM [2021-05-14 16:08:24 -0700] [15] [INFO] Listening at: http://0.0.0.0:8000 (15)
today at 4:08:24 PM [2021-05-14 16:08:24 -0700] [15] [INFO] Using worker: sync
today at 4:08:24 PM [2021-05-14 16:08:24 -0700] [18] [INFO] Booting worker with pid: 18
today at 4:08:24 PM [2021-05-14 16:08:24 -0700] [19] [INFO] Booting worker with pid: 19
today at 4:08:24 PM [2021-05-14 16:08:24 -0700] [20] [INFO] Booting worker with pid: 20
today at 4:08:24 PM [2021-05-14 16:08:24 -0700] [21] [INFO] Booting worker with pid: 21

When i go to the site:
500 Internal Server Error
The server encountered an internal error and was unable to complete your request. Either the server is overloaded or there is an error in the application.

Web logs; go-links_web:
today at 4:08:46 PM [2021-05-14 16:08:46,876] ERROR in app: Exception on / [GET]
today at 4:08:46 PM Traceback (most recent call last):
today at 4:08:46 PM File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 2447, in wsgi_app
today at 4:08:46 PM response = self.full_dispatch_request()
today at 4:08:46 PM File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1952, in full_dispatch_request
today at 4:08:46 PM rv = self.handle_user_exception(e)
today at 4:08:46 PM File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1821, in handle_user_exception
today at 4:08:46 PM reraise(exc_type, exc_value, tb)
today at 4:08:46 PM File "/usr/local/lib/python3.7/site-packages/flask/_compat.py", line 39, in reraise
today at 4:08:46 PM raise value
today at 4:08:46 PM File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1950, in full_dispatch_request
today at 4:08:46 PM rv = self.dispatch_request()
today at 4:08:46 PM File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1936, in dispatch_request
today at 4:08:46 PM return self.view_functionsrule.endpoint
today at 4:08:46 PM File "/usr/src/app/server/src/main.py", line 161, in home
today at 4:08:46 PM 'default_namespace': config.get_default_namespace(current_user.organization),
today at 4:08:46 PM File "/usr/src/app/server/src/shared_helpers/config.py", line 100, in get_default_namespace
today at 4:08:46 PM if session.get('org_default_ns_exp') and session.get('org_default_ns_exp') > datetime.datetime.utcnow():
today at 4:08:46 PM TypeError: can't compare offset-naive and offset-aware datetimes
`

@swapnachagam
Copy link

swapnachagam commented Sep 29, 2022

Is there an update on the above issue?
I tried setting up the trotto in my own ec2 instance and i am getting 500 error too

@maxtechera
Copy link
Contributor

Hi @swapnachagam! We will work on this next week and provide examples around configuring this with docker-compose.
Keep you posted!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants