Operations

How to back up and restore a server, watch it, find out why it fails, and recover from an incident. Upgrading and rolling back are in Install, upgrade, and roll back.

What to back up

What	Where	How
Messages, flows, users, alerts, snippets, config map, scripts, settings, lookups, the audit log, events, statistics	The store's PostgreSQL database (store.dialect: postgres)	pg_dump (below)
The server configuration	The file you pass to --config	Copy the file
TLS certificates, keys, and CAs	tls.certFile, tls.keyFile, and every certFile, keyFile, and caFile a flow's source or destination names	Copy the files (keep the keys private)
Secrets	secrets.dir (default /run/secrets), ~/.pgpass of the server's account, environment variables	However you manage secrets
Files a flow reads or writes	A file source's dir and moveTo, a file destination's dir	Copy the directories

ops.yaml in the examples is a connection file with the address and an operator's credentials.

With store.dialect: memory (the default) or disabled, nothing survives a restart: there is nothing to back up but the files. Use postgres for any server whose data you need to keep.

Weavster has no backup command of its own: back up the database with PostgreSQL's tools.

Back up the database

pg_dump -Fc -f weavster-2026-09-29.dump 'postgres://weavster@db.internal:5432/weavster?sslmode=verify-full'

pg_dump takes a consistent snapshot while the server runs; you do not need to stop it. Take one before every upgrade (it is what makes a rollback possible) and on a schedule that matches how many messages you can afford to lose.

Restore the database

1. If you can still reach the server, save the messages received since the backup, since they are not in it:
2. Stop every flow (POST /api/v1/flows/stop-all, see Flow lifecycle), so that no message arrives after the export.
3. Export the messages. The client's exportmessages "path" * writes the newest 10,000; page through more with GET /api/v1/messages/export (see Export and import messages), and check that the archives hold every message since the backup.
4. Stop the server.
5. Restore the dump into a new, empty database next to the current one (add the connection options you use with pg_dump). Do not restore over the current database: that would keep tables a later release added and the backup lacks, and the next upgrade would fail on them. The current database also stays your way back if the restore fails.

bash createdb -O weavster weavster_restore pg_restore --no-owner --role=weavster -d weavster_restore weavster-2026-09-29.dump

1. Point store.dsn (or the store.dsnEnv secret) at weavster_restore, and start the same release that took the backup, or a newer one (it upgrades the schema at start).
2. Import the messages you exported, replacing the older copies the backup holds: POST /api/v1/messages/import?overwrite=true. Until then, a message that was queued in the backup may be delivered again (delivery is at least once).
3. Once the restored server works, drop the old database if you no longer need it, with the same connection options (host, role, TLS) as the commands above, for example dropdb -h db.internal -U weavster weavster.

Sign-in tokens are not kept across restarts: users log in again after a restore.

Keep a copy of the configuration too

A configuration export is a portable file of flows, alerts, snippets, scripts, and settings that you can import into another server or keep in version control. It does not hold users, passwords, lookups, or messages, so it does not replace the database backup.

$ weavster -c ops.yaml
weavster> exportcfg "weavster-config.json"

See Configuration export and import, or manage the configuration from files with Config-as-code documents.

Monitor

Is it up?

Weavster has no separate health endpoint. For a liveness probe, request the API description; it needs no login:

curl -fsS -o /dev/null http://127.0.0.1:8080/api/openapi.yaml && echo up

With listen.contextPath: /weavster, the path is /weavster/api/openapi.yaml. The server opens its ports only after it has connected to the store and upgraded the schema, so an answer means it has started. The probe does not use the store, so it keeps answering if the database goes away later. To check the store too, request something the server reads from it, such as GET /api/v1/flows with an account that has flows:view: it returns an error while the store cannot be reached.

For more detail, GET /api/v1/system (any signed-in user) returns the version and uptime:

curl -s -u 'ops:PASSWORD' -H 'X-Weavster-CSRF: 1' http://127.0.0.1:8080/api/v1/system

See System information.

Metrics

Scrape GET /metrics with an account that has flows:view; see Metrics for the Prometheus configuration and every metric. Alert on:

Signal	Metric	Meaning
Deliveries failing	weavster_connector_messages_total{outcome="errored"} rising	A destination rejects messages or cannot be reached; they are retried, then dead-lettered
Server at capacity	weavster_processing_refused_total rising, or weavster_processing_in_flight equal to weavster_processing_slots	Messages are refused as busy; raise processing.maxConcurrent or add capacity
Flows not running	weavster_flows{status="started"} lower than expected	A flow was stopped, or failed to deploy at start. A status no flow has is not published at all, so write the rule as (weavster_flows{status="started"} or vector(0)) < 3 to fire when none are started

Alerts you define in Weavster are stored but do not fire in this edition; alert from your monitoring system instead.

Logs

The server writes its log to stderr as text lines (time=… level=INFO msg=…). The log level and format cannot be changed. Collect stderr: it holds the audit log (msg=audit), the first-run admin password, and warnings such as:

Log line	Meaning
store connection failed	At start, the store cannot be reached; the server tries again, up to store.maxRetry times
processing limit reached: messages refused as busy (see processing.maxConcurrent)	Every processing slot is in use
delivery retry pass failed	A pass over queued deliveries failed (often because the store cannot be reached); it runs again on the next interval
auto-deploy failed	A flow in the store could not be deployed at start
audit entry not stored	An audit entry was logged to stderr but not stored, so GET /api/v1/audit lacks it
shutdown deadline reached; unfinished messages are stored and resume on the next start	The server stopped before every message was done

API errors carry no request ID; match an error to the log by time and by the user and resource in the audit line.

Troubleshoot

The server prints why it failed to start on stderr, as one Error: line, and exits 1 (2 for a usage error).

Error	Fix
Error: listen tcp 127.0.0.1:8080: bind: address already in use	Another process (often another Weavster) has the port: stop it, or change listen.address
Error: config: …: yaml: unmarshal errors:	The next line names a misspelled or unknown key in the configuration file, for example line 2: field dialekt not found in type serverconfig.Store; see Server configuration
Error: config: the postgres dialect needs store.dsn or store.dsnEnv (one of them)	Set exactly one of them
Error: store: postgres: giving up after N attempts: store.dsnEnv: environment variable NAME is not set, and there is no file /run/secrets/NAME	Provide the secret as a variable or a file in secrets.dir (default /run/secrets)
Error: store: postgres: giving up after N attempts: …	The database cannot be reached: check the host, port, credentials, and sslmode in the connection string
Error: store: state: the database schema is at version N (written by weavster X), newer than this release supports (M): …	A newer release upgraded the database (written by an unknown weavster release when it did not record its version): run that release, or restore a backup taken before the upgrade
Error: tls: open …: no such file or directory	A path in tls.certFile or tls.keyFile is wrong (permission denied instead of no such file or directory: the server's account cannot read it)
Error: refusing to run under a privileged OS account; use a dedicated service account or set WEAVSTER_ALLOW_ROOT=1	Run the server as an unprivileged account

When the server runs but messages do not arrive:

1. The client's status command (or GET /api/v1/flows) shows whether the flow is started. A flow must be deployed and started to process messages; see Flow lifecycle.
2. GET /api/v1/messages?flowId=FLOW&status=errored lists messages that failed, with the error of each; deadletter list FLOW lists those that ran out of delivery attempts.
3. GET /api/v1/events shows what happened, such as message.errored for a failed transform or source.http.failed for a source that could not listen; see Statistics and events.

Recover from an incident

A destination was down

Deliveries to an unreachable destination are retried with backoff (delivery.backoffBaseMs, doubling up to one minute) and, after delivery.maxAttempts attempts, the message is dead-lettered. Once the destination is back, send the dead letters again:

$ weavster -c ops.yaml
weavster> deadletter requeue all adt

A requeued message goes only to the destinations that did not receive it. See Dead-lettered messages.

To hold messages while you fix a destination, stop that destination (POST /api/v1/flows/{id}/destinations/{name}/stop); the flow keeps receiving and queues them.

The server stopped or crashed

Start it again. With a PostgreSQL store, every message the server acknowledged is stored, and the server resumes queued and unfinished messages of started flows. Delivery is at least once: a message being delivered when the server stopped may be delivered again, so receivers should detect duplicates (HTTP destinations send the same Idempotency-Key on every attempt). Statistics counted since the last sample are lost; messages are not.

A bad change was deployed

Apply the previous config-as-code document (see Config-as-code documents). It changes what it lists back, and in each section it includes, removes what that section no longer lists. So if it has a flows section, it removes flows the bad change added; if it has none, flows are not managed by it, and you undeploy and delete added flows yourself.

Or import the previous configuration export:

$ weavster -c ops.yaml
weavster> importcfg "weavster-config.json" force

An import only adds and replaces. Undeploy and delete any flow the bad change added (see Flow lifecycle), or it keeps running.

Then reprocess the messages the bad change handled: POST /api/v1/messages/{id}/reprocess runs a stored message through the flow again as a new message. See Work with one message.

The database is lost

Restore the last backup into a new database, point store.dsn (or store.dsnEnv) at it, and start the server. Messages received since that backup are gone, unless their senders can send them again.