tr
TryGhost
GitHub
Ghost
Workspace
GitHub
CI Pipeline Executions
Current execution
Failed
NY-1559_tinybird-sync-job
Runs and Agents
Configuration
Self-Healing CI
Project Graph
Circular Dependencies
Enterprise
Resource Usage
Analysis
Conformance
Enterprise
Runs
Current run
Succeeded
Tasks
Resource Usage
Insights
Compare tasks
Analytics
Sign in
Toggle sidebar
[WIP, NO MERGE]: Added recurring job to sync automation runs to Tinybird ref https://linear.app/ghost/issue/NY-1559/ghost-backend-send-events-to-traffic-analytics-service-with-recurring This cannot be merged as-is because it includes database migrations, so it will need to be broken up further before review. - Replaces the boot.js proof of concept with a tinybird-sync service scheduled through the jobs service every five minutes, with a per-process random offset - Adds a tinybird_syncs table storing a per-table watermark (last_synced_updated_at), plus updated_at indexes on automation_runs and automation_run_steps so each run only scans recently changed rows - Sends only the columns the Tinybird materialized views read; member id, member email, and lock bookkeeping never leave MySQL - Reads rows in keyset-paginated batches and splits requests by UTF-8 byte size under Tinybird's Events API limit - Uses wait=true so Tinybird acknowledges committed rows, fails the job on quarantined rows or HTTP errors without advancing the watermark, and aborts stalled requests after a timeout - Holds back rows updated within the last minute so a still-open transaction cannot be skipped by the watermark - Syncs both tables concurrently and guards against overlapping runs - Adds a backgroundJobs.tinybirdSync config flag to disable the job - Reverts the WIP changes that read stats from Tinybird; that work belongs to NY-1560 - Covers the sync logic with unit tests against an in-memory SQLite database and a fake Tinybird Express server
nx run ghost-monorepo:lint:boundaries
⌘K
Succeeded
nx run ghost-monorepo:lint:boundaries
Click to copy
Linux
4 CPU cores
read-write
access token used
0eef79a6
30472
GitHub
Ghost
Workspace