OpenAQW Health

This is how error tracking and uptime checks for OpenAQW's sites are set up and run. It is GlitchTip, an open-source error tracker that speaks Sentry's protocol, running from GlitchTip's official container image on Railway. It follows OpenNeo's openneo-health; only the hosting differs.

What runs

• GlitchTip: the image glitchtip/glitchtip:6.2.6 in all-in-one mode (SERVER_ROLE=all_in_one), so the web app and its background worker share one container, limited to 1 vCPU and 1 GB of memory. With VALKEY_URL empty, its cache and task queue live in Postgres and there is no Redis.
• GlitchTip DB: Postgres 18 (Railway's postgres-ssl:18 image) with a 5 GB volume at /var/lib/postgresql/data. It starts with max_locks_per_transaction=512, as GlitchTip's install guide asks, because its event tables are split into many partitions.
• It is at health.openaqw.com. Railway serves HTTPS and forwards to GlitchTip on port 8000. Sites in the same Railway project send errors over the private network, to glitchtip.railway.internal:8000.
• Sign-ups and new organizations are closed. Events are kept for 90 days.

What it watches

• AQ Closet: errors from its server and from visitors' pages, which the site passes on itself without visitors' addresses (server/errors.js in aq-closet), and an uptime check on https://aqcloset.com/health every minute.

Settings

glitchtip.env and postgres.env list the variables on the two Railway services. Secrets are placeholders here and live only in Railway. When a setting changes, change it in Railway and here together.

First start

setup.py creates the admin account (from LOGIN_EMAIL and LOGIN_PASSWORD), the OpenAQW organization, the AQ Closet project and team, and the uptime check, then prints the project's DSN. It is safe to run again. It ran as the GlitchTip service's pre-deploy command, with the script's text in the GT_SETUP variable:

python manage.py shell -c "$GT_SETUP"

Change the admin password after the first sign-in.

Upgrading

1. Read GlitchTip's changelog for every version between ours and the new one. Migrations run when the container starts.
2. Back up the database volume in Railway.
3. Change the image tag on the Railway service, and in this README.
4. Deploy, then check that /_health/ answers and a test event arrives.

Moving it

Nothing here depends on Railway except the ${{...}} references and the .railway.internal host names. The same two images with the same variables run under Docker Compose on any machine; point the sites' SENTRY_DSN at the new address.

License

MIT, see LICENSE.