Notifications & Alerting
Vorian Connect monitors lab operations continuously and delivers email and SMS alerts when attention is needed — offline instruments, unmapped result backlogs, preanalytical specimen conditions, and QC violations. Per-tenant settings let each lab control which alerts fire, at what severity, and who receives them.
How It Works
- A dedicated notifier worker runs alongside the cloud gateway, polling the database every 60 seconds (configurable via
NOTIFIER_POLL_SECONDS) for pending alert conditions - When a condition is detected — an instrument missing heartbeats for 15 minutes, a preanalytical event firing, or unmapped results exceeding a threshold — the worker enqueues an alert notification with a deduplication key to prevent duplicate alerts
- Alert settings are tenant-scoped: each lab configures which alert types are enabled, the minimum severity threshold, recipient lists, throttle intervals, and whether SMS delivery is active
- Email delivery uses SMTP with STARTTLS support. SMS delivery uses Twilio with E.164 phone number validation. Both channels are optional and independently configurable
- Each notification is attempted up to 3 times with idempotent per-recipient delivery tracking. After exhausting retries, the notification is suppressed and logged with the failure reason
- Throttle intervals (configurable per alert type, default 15 minutes) prevent alert storms — a repeated condition won't generate another notification until the throttle window passes
What's Included
- Offline instrument alerts: When an edge agent stops sending heartbeats for more than 15 minutes, the notifier creates an alert with the instrument name and last-heartbeat timestamp. Deduplicated per instrument so each offline event generates one notification, not one per poll cycle
- Unmapped result threshold alerts: When pending unmapped results exceed a configurable threshold and minimum age, the notifier sends a warning with the pending count and a link to the Unmapped Results review page. Configurable per tenant through
unmapped_alert_settings - Preanalytical specimen alerts: When a preanalytical event fires (hemolysis, icterus, lipemia, or other specimen condition), the notifier delivers the specimen ID, event type, severity, source instrument, and observation timestamp. Legacy per-event recipient overrides are supported alongside tenant-wide settings
- QC violation alerts: When Westgard rule violations are detected, the notifier can deliver alerts to configured recipients with the violation type and affected instrument
- Generic operational alerts: Any system event can be enqueued as a generic alert with a type, severity, subject, body, and recipient list. Used for test alerts and extensible to future alert categories
- Test email and SMS: Administrators can send test notifications through the dashboard API (
/api/alert-settings/test-emailand/api/alert-settings/test-sms) to verify delivery configuration before going live - Per-tenant alert settings: Each tenant configures independently: enable/disable per alert type, set minimum severity, define email and SMS recipient lists, and set throttle intervals. Settings are stored in
alert_settingsand managed through the dashboard - Delivery tracking: Every notification attempt is recorded in
alert_notification_deliverieswith per-recipient, per-channel tracking. Operators can see which recipients received an alert, which failed, and why
Where It Fits
The notifier is a background operational service that bridges automated monitoring and human attention. It complements the Observability dashboard — observability is for active scanning, notifications are for passive awareness. When a lab team is not watching the dashboard, the notifier ensures they still know when something needs attention.
Alert settings are managed through the Team administration surface, where site administrators configure which alert types are active for their lab, who receives them, and at what severity threshold. The notifier worker runs as a separate process alongside the gateway and reprocessor, sharing the same PostgreSQL database for tenant-scoped alert state.
Technical Details (for evaluators)
The notifier worker (package notifier) uses a polling loop with configurable interval (NOTIFIER_POLL_SECONDS, default 60s). Each poll cycle runs three processors: offline instrument enqueue, unmapped threshold checks, preanalytical delivery, and generic alert delivery. Processors use FOR UPDATE SKIP LOCKED to safely handle concurrent notifier instances without duplicate delivery.
Email delivery uses Go's net/smtp with STARTTLS support (configurable via SMTP_REQUIRE_STARTTLS). SMS delivery uses the Twilio REST API (POST /2010-04-01/Accounts/{sid}/Messages.json) with HTTP Basic authentication. SMS recipients are validated as E.164 format (+1XXXXXXXXXX) before delivery is attempted. Both channels are guarded by feature flags — if SMTP credentials or Twilio credentials are missing, the channel is silently disabled.
Alert deduplication uses a dedupe_key column (e.g., offline_instrument:{source_id}) combined with the throttle window to prevent repeat notifications. The throttle is enforced at enqueue time — the INSERT checks for an existing notification with the same dedupe key within the throttle interval and skips if one exists.
Delivery idempotency is handled through the alert_notification_deliveries table with a composite unique constraint on (notification_table, notification_id, channel, recipient). Before sending to a recipient, the worker checks whether a successful delivery already exists for that combination and skips if so.