Lens

Sync Data

Data sync pulls fresh data from your connected sources into ClickHouse. You can sync manually or set up automatic schedules to keep dashboards current.

Overview

Lens doesn't query your source systems live. Instead, sync jobs pull data on a schedule and land it in ClickHouse, where boards and reports can hit it with interactive latencies. The tradeoff is data freshness: whatever was in the source at sync time is what your dashboards show until the next sync runs.

This design is deliberate. Live queries against GA4 or Airbyte sources would be slow and would eat API quotas. By staging data in ClickHouse, Lens serves dashboards in milliseconds instead of seconds and doesn't burn through your source-system rate limits every time a client reloads a report.

Manual sync

Open any table and click the Sync Now button. The job runs in the background, pulling the latest data from the source into ClickHouse, and the page updates when it finishes. For GA4 tables, a confirmation window appears first with an estimate of the backfill window, the projected number of rows, how many 30-day API chunks it will take, and a rough duration. This lets you confirm before kicking off a large pull, and points you to a BigQuery source when the volume is high.

Manual sync is the right choice when you just landed a new data point and want to refresh a dashboard on demand, or when you're iterating on a table's schema and need to test the sync end-to-end before committing to a schedule.

Set a schedule

When creating or editing a table, pick a sync schedule: Hourly, Daily, Weekly, Monthly, or Manual. Daily is the common default for GA4 tables because GA4 data itself takes up to 48 hours to finalize. Hourly makes sense when the source is real-time and the dashboard consumers expect near-live numbers.

Scheduled syncs run automatically in the background. No action is needed from you; the Lens scheduler picks up each table when its cadence fires. If a run fails, it's logged to the sync history and (depending on configuration) surfaces as an error badge on the Tables list.

Monitor sync status

The Tables list shows the last sync time and status for each table. A green indicator means the most recent sync succeeded; red means there was an error. Click into a table to see its detailed sync history, including timestamps, row counts, and any error messages from failed runs.

Check this view when a dashboard looks stale. If the last successful sync was yesterday and you expected hourly refreshes, something is wrong upstream (usually credentials or source-side API issues) and needs attention before more scheduled runs stack up failures.

Incremental vs full refresh

Most sources support incremental sync, which pulls only the data that has changed since the last successful run. This is faster, cheaper, and doesn't blow out your source-system quotas. Lens uses incremental sync by default wherever the source supports it.

A full refresh replaces every row in the table. You want this when an upstream schema change invalidates old rows, or when you've changed the table's dimensions or metrics and need to re-pull everything to match the new shape. Full refresh takes longer and uses more quota, so reach for it deliberately rather than as a default.

BigQuery tables are full refresh only for now: each sync reads the configured range and reimports it. For wildcard table sets like events_* the date range drives the _TABLE_SUFFIX filter, and for single-table or custom-SQL modes you pick a date column that anchors each row. Scope the Backfill Days to what you actually chart so the reimport stays fast.

Large GA4 backfills

The GA4 Data API returns data row by row, in 30-day chunks. A long backfill therefore means many chunks and, on a high-traffic property, millions of rows that can take hours to pull. To keep a single table from monopolizing the sync worker, GA4 backfills are capped at 400 days. If you try to set a longer window, Lens blocks the save and points you to a BigQuery source, which reads GA4 export data in bulk and is far faster for deep history.

GA4 syncs checkpoint after each 30-day chunk. If a long sync is interrupted, the next run resumes from the last completed chunk instead of starting over, so even a large backfill makes steady forward progress. When a sync is running or queued, the table shows its status; if it ever shows an error or looks stuck, the status banner has a Reset Sync Status button. Resetting clears the state and also signals a running GA4 sync to stop cleanly at its next chunk boundary, which is how you cancel a pull you started by mistake.

Tips

GA4 incremental syncs use a rolling 7-day window because GA4 data can continue to finalize for up to 48 hours after the event. CSV tables use a folder model: upload files with Upload Data and they accumulate in the table's folder, then each sync ingests new files. JSONL tables sync from S3 and new files are detected automatically on each run.

Troubleshooting

Sync fails with a credentials error

Open Settings > Service Connections and check the status of the connection tied to the source. OAuth tokens can expire if the Google account owner revokes access or if 2FA settings change. Reconnect the service, then trigger a manual sync on the table to verify.

Sync appears stuck for hours

Large GA4 backfills can take a long time, but they shouldn't stall. GA4 syncs checkpoint after each 30-day chunk, so an interrupted run resumes from where it left off rather than restarting. If progress hasn't moved in over an hour, use the Reset Sync Status button on the status banner: it clears the state and signals a running GA4 sync to stop at its next chunk boundary, after which you can start a fresh run. If a single GA4 table consistently runs for hours, its backfill window is probably too large; move that history to a BigQuery source.

New data missing after a successful sync

GA4's rolling 7-day window means today's data may still be materializing upstream. Wait for the 48-hour finalization window, then re-check. If the data is older than that and still missing, confirm the sync's filter configuration hasn't silently excluded the traffic you're looking for.