Transfer data in Fauna to your analytics tool using Airbyte

Why Airbyte

Why Fauna

Why Fauna + Airbyte

• Full refresh append sync mode copies all of your data to the destination, without deleting existing data.
• Full refresh overwrite sync mode copies the whole stream and replaces data in the destination by overwriting it.
• Incremental append sync mode periodically transfers new, changed, or deleted data to the destination.
• Incremental deduped history sync mode copies new records from stream and appends data in the destination, while providing a de-duplicated view mirroring the state of the stream in the source

Prerequisites

Create a destination database account

Set up dbt

Install Docker

Step 1: Set up the Fauna source

Full refresh sync mode

CreateRole({
  name: "airbyte-readonly",
  privileges: [{
    resource: Collection("COLLECTION_NAME"),
    actions: { read: true }
  }],
})

CreateKey({
  name: "airbyte-readonly",
  role: Role("airbyte-readonly"),
})

{
  ref: Key("341909050919747665"),
  ts: 1662328730450000,
  role: Role("airbyte-readonly"),
  secret: "fnAEjXudojkeRWaz5lxL2wWuqHd8k690edbKNYZz",
  hashed_secret: "$2a$05$TGr5F3JzriWbRUXlKMlykerq1nnYzEUr4euwrbrLUcWgLhvWmnW6S"
}

Incremental append sync mode

CreateIndex({
  name: "INDEX_NAME",
  source: Collection("COLLECTION_NAME"),
  terms: [],
  values: [
    { "field": "ts" },
    { "field": "ref" }
  ]
})

|Index values|Description|
| --- | ----------- |
|`ts`| Last modified timestamp.|
|`ref`|Unique document identifier.|

CreateRole({
  name: "airbyte-readonly",
  privileges: [
    {
      resource: Collection("COLLECTION_NAME"),
      actions: { read: true }
    },
    {
      resource: Index("INDEX_NAME"),
      actions: { read: true }
    },
    {
      resource: Indexes(),
      actions: { read: true }
    }
  ],
})

CreateKey({
  name: "airbyte-readonly",
  role: Role("airbyte-readonly"),
})

{
  ref: Key("341909050919747665"),
  ts: 1662328730450000,
  role: Role("airbyte-readonly"),
  secret: "fnAEjXudojkeRWaz5lxL2wWuqHd8k690edbKNYZz",
  hashed_secret: "$2a$05$TGr5F3JzriWbRUXlKMlykerq1nnYzEUr4euwrbrLUcWgLhvWmnW6S"
}

1. The source must be able to Get() the index, which means it needs read access to this index.
2. The source of the index must be a reference to the collection you are trying to sync
3. The number of values must be two.
4. The number of terms must be zero.
5. The values must be equal to:

   {"field": "ts"}
   {"field": "ref"}

Step 2: Deploy and launch Airbyte

Deploy and launch locally

1. Refer to the Deploy Airbyte instructions to install and deploy Airbyte. Enter the following commands to deploy the Airbyte server:

   git clone https://github.com/airbytehq/airbyte.git
   cd airbyte
   docker-compose up

2. When the Airbyte banner displays, launch the Airbyte dashboard at http://localhost:8000.
3. To log in, enter the default credentials found in the .env file of the cloned repository:

   BASIC_AUTH_USERNAME=airbyte
   BASIC_AUTH_PASSWORD=password

4. Choose the Connections menu item to start setting up your data source.
5. In the Airbyte dashboard, click the  + New connection button
6. Skip to Step 3: Set up the Fauna source

Deploy and launch with Airbyte Cloud

1. Refer to the Getting Started with Airbyte Cloud guide for the fastest and most reliable way to run Airbyte.
2. If this is your first time, go to https://cloud.airbyte.com/signup, sign up for an Airbyte account, and click Creat your first connection.

Step 3: Set up the Fauna source

1. In the Source type dropdown, choose Fauna, which lists the configurable Fauna connector parameters. If you previously set up a source, click the Use existing source button to choose that source. A Setup Guide in the right-side panel also gives detailed setup instructions.
2. Set the following required parameters:
   Parameter	Description
   Name	Enter a descriptive name for this connection. The name is displayed in the Connections window connections list.
   Domain	Enter the domain of the collection you want to export. See Region Groups for region domains.
   Port	Enter the default port number: 443.
   Scheme	Enter the scheme used to connect to Fauna: https.
   Fauna Secret	Enter the saved Fauna secret that you use to authenticate with the database.
   Page Size	The page size lets you control the memory size, which affects connector performance.
   Deletion Mode	The deletion mode lets you specify whether to ignore document deletions or flag documents as deleted, depending on your use case. + Choose from the following options: The Ignore option ignores document deletions. The Deleted Field option adds a date column that has the date when you deleted the document. This maintains document history while letting the destination reconstruct deletion events.
3. After setting up the source, click the Set up source button.
   The "All connection tests passed!" message confirms successful connection to the Fauna source. This minimally confirms:
   • The secret is valid.
   • The collection exists.

Step 4: Set up the destination

1. If you previously set up a destination, click the Use existing destination button to select and use that destination. Otherwise, choose the Destination type.
2. Destination connector configuration parameters are unique to the destination. Populate the Set up the destination fields according to the connector requirements, including authentication information if needed. A Setup Guide is provided in the right-side panel with detailed setup instructions.
3. When you are done, click the Set up destination button and wait for the destination testing to successfully complete.

Step 5: Set up the connection

1. Enter a descriptive name for the connection in the Connection name field.
2. Choose a Replication frequency, which is the data sync period.
   You can choose the Manual option to manually sync the data.
3. In the Destination Namespace field, click the Edit button to choose a destination namespace where the data is stored. Options include:
   Option	Description
   Mirror source structure	Sets the name in the destination database to the name used for the Fauna source.
   Other	Uses another naming option, such as prefixing the database name with a string.

1. In the Non-breaking schema updates detected field, choose Ignore or Disable connection for how Airbyte handles syncs when it detects a non-breaking schema change in the source.
2. In the Activate the streams you want to sync section, click the > arrow to expand the available fields:
   Field	Description
   data	Collection data.
   ref	Unique document identifier.
   ts	Data timestamp.
   ttl	Time-to-live interval.
   The document is deleted if it is not modified in the ttl time interval. The default value is null for not used. After document deletion, it is not displayed in temporal queries and the connector does not emit a deleted_at row.
3. Select ref as the Primary key. This uniquely identifies the document in the collection.
4. In the Sync mode click the options to choose the combination of options that define the source sync behavior:
   Sync mode	Description
   Incremental - Deduped history	Sync new records from stream and append data in destination, also provides a de-duplicated view mirroring the state of the stream in the source.
   Full refresh - Overwrite	Sync the whole stream and replace data in destination by overwriting it.
   Incremental - Append	Sync new records from stream and append data in destination.
   Full refresh - Append	Sync the whole stream and append data in destination.
   Fewer than four options indicates that the index is incorrectly set up. See Step 1: Set up the Fauna source. A new incremental sync gets the full database, the same as a full sync.
5. Choose the Normalization data format:
   Data format	Description
   Raw data (JSON)	Put all the source data in a single column.
   Normalized tabular data	Put the ref, ts, ttl, and data fields in separate columns.
6. Click the + Add transformation button to add the dbt transform.
   To extract the fields in the source data column, you need to configure dbt to map source data to destination database columns. For example, the following SQL-based query extracts the name, account_balance, and credit_card/expires fields from the source data column to populate three separate columns of the destination data:

   with output as (
     select
       data:name as name
       data:account_balance as balance
       data:credit_card:expires as cc_expires
     from airbyte_schema.users
   )
   
   select *from output

7. Click the Set up connection button.

Step 6: Sync the data

• The number of bytes transferred.
• The number of records emitted and committed.
• The sync duration.

Step 7: Verify the integration

Conclusion