Lens

Airbyte Source

Bring data from any platform your own Airbyte instance can reach into Lens. Airbyte writes JSONL files to a drop path Lens gives you, and Lens loads each stream as a table.

When to use Airbyte

Use it for a platform Lens has no built-in source for, when you already run Airbyte. Lens does not run Airbyte or its connectors: your Airbyte syncs the platform and writes JSONL files, and Lens reads them.

TermMeans
Airbyte connectionA named drop location in Lens, such as "Partner Ads". It gets its own drop path
Drop pathThe folder your Airbyte destination writes to
StreamOne dataset inside the connection, such as campaigns. Each stream is a folder under the drop path, and each Lens table reads one stream

Several tables can share one connection, one per stream.

Before you start

  • A working Airbyte source connector for the platform you want.
  • An Airbyte S3 destination with JSONL output. Besides the drop path, it needs the bucket name and write credentials; your TagPipes contact provides those.

Step 1: Create the table and the connection

  1. Go to Configure > Data Tables, click + New Table, and choose Airbyte (JSONL).
  2. Enter a Table Name, such as "Partner Ads Campaigns".
  3. Under Airbyte Connection, choose + New Connection and enter a Connection Name. The name becomes part of the drop path: "Partner Ads" becomes partner-ads.
  4. Click Create Table. Lens saves the table and reopens it for editing so you can copy the drop path.

For a second stream from the same Airbyte connection, create another table and pick the existing connection instead of + New Connection.

Step 2: Point Airbyte at the drop path

  1. On the reopened table, the S3 Drop Path shows under the connection; click Copy.
  2. In your Airbyte S3 destination, use that path as the bucket path, so each stream writes to its own folder directly under it.
  3. Set the output format to JSONL. Gzip-compressed .jsonl.gz files are read too.
  4. Run the Airbyte sync.

Step 3: Pick a stream and map its fields

Back on the table, Lens lists the streams it finds under Stream Selection. If none appear yet, run the Airbyte sync and click Refresh Streams.

Select a stream. Lens scans sample records and lists every field under Column Mapping with a sample value. Give each field a role:

RoleUse for
DateThe record's date. Exactly one
DimensionSomething you group or filter by: campaign, channel, country
MetricA number Lens adds up: spend, clicks. At least one
IgnoreEverything else. The default

For a metric, Transform can rescale the value as it loads: / 100, / 1,000, / 1,000,000 or x 100. Use / 1,000,000 for platforms that report money in micros.

Lens tidies Airbyte records as it loads them:

  • Airbyte's own _airbyte_* fields are dropped.
  • Nested fields are flattened with _, so metrics.spend becomes metrics_spend.
  • Lists become one comma-separated value.

If the date field holds local dates, set Source Timezone to that zone. See Dates and Time Zones.

Step 4: Choose the sync mode and sync

Sync ModeWhat each sync does
Incremental (default)Loads only files written since the last sync and adds their rows. Rows already loaded stay as they are
Full RefreshClears the table and reloads every file in the stream's folder

Choose a schedule, click Save Table, then Sync Now on the table page.

Check it worked

The table moves to active and the row count fills in. Compare one day's total for a metric with the same day in the platform the data came from.

Troubleshooting

What you seeWhat to do
"No streams found yet"Airbyte has not written anything to the drop path. Check the destination's bucket path matches the drop path exactly, run the Airbyte sync, then click Refresh Streams
"No JSONL files found in this stream yet"The stream's folder has no .jsonl or .jsonl.gz files. Check the destination's output format is JSONL
Saving asks for a Date column or a MetricEvery Airbyte table needs exactly one Date field and at least one Metric
Rows are counted twiceYour Airbyte connection rewrites full copies of the data on every run, and Incremental adds each copy. Switch the Lens table to Full Refresh, or change the Airbyte connection to write only new records
Dates are one day earlySource Timezone is UTC but the date field holds local dates. Set the zone and run a full refresh