Verae Time × Zapier

Local network failures

site/packages/overview/05-network-failures.md

Local network failures

5. Local network failures

“Local network” means the operator LAN, SSH to NS1, or a partitioned archive — not Zapier’s cloud.

Failure What happens What the customer sees
NATS node unreachable Client reconnects to another cluster URL; JetStream consumers resume Wait may return pending; hook still fires later
All NATS down Middleware cannot publish jobs.watch; fleet marks workers unhealthy 503 / pending; no NATS leak to Zapier
One WORM / tree node partitioned Bloom miss = no packet; aggregator uses whoever answered Lookup may miss attachments until the node returns; seal on chain still valid
SSH to a fleet host fails Spawn fails; fleet places the next replica on another machine Floor still met if capacity remains on local or another SSH host
Chain api.veraetime.net timeout Poller retries; then timestamp.timeout event Wait → pending or failed; async + hook still the recovery path
zappier-edge 402 QuotaExceeded with upgrade URL Zap step error; no NATS involved
Tunnel to loopback NATS dropped NATS_URL=nats://127.0.0.1:14222 dies; restart nats-tunnel.sh Workers on NS1 itself still see 127.0.0.1:4222

Design choices that make partitions survivable

  1. Archive query is broadcast, not a shared queue group — a dead node does not steal the message.
  2. Bloom miss is silence — missing nodes do not send empty errors that look like “hash unknown”.
  3. Zero replies + known puts is an outage, not a miss (simulator / fleet monitors flag this).
  4. NATS is not on the public NIC — a WAN blip does not expose 4222.
  5. Idempotent seals — retrying a Zap after a network error will not double-timestamp.

See fleet RTT (min / avg / p50 / p90) when planning extra tree nodes after a flaky path.