Documentation Experience Platform Source Connectors Guide

Ultimate

Create a Google BigQuery base connection using the Flow Service API

Last update: Thu Jun 08 2023 00:00:00 GMT+0000 (Coordinated Universal Time)

Topics:
Sources

CREATED FOR:

Developer
User
Admin
Leader

IMPORTANT

The Google BigQuery source is available in the sources catalog to users who have purchased Real-Time Customer Data Platform Ultimate.

A base connection represents the authenticated connection between a source and Adobe Experience Platform.

This tutorial walks you through the steps to create a base connection for Google BigQuery using the Flow Service API.

Getting started

This guide requires a working understanding of the following components of Experience Platform:

Sources: Experience Platform allows data to be ingested from various sources while providing you with the ability to structure, label, and enhance incoming data using Platform services.
Sandboxes: Experience Platform provides virtual sandboxes which partition a single Platform instance into separate virtual environments to help develop and evolve digital experience applications.

The following sections provide additional information that you will need to know in order to successfully connect to Google BigQuery using the Flow Service API.

Gather required credentials

In order for Flow Service to connect Google BigQuery to Platform, you must provide the following OAuth 2.0 authentication values:

Credential

Description

project

The project ID of the default Google BigQuery project to query against.

clientID

The ID value used to generate the refresh token.

clientSecret

The secret value used to generate the refresh token.

refreshToken

The refresh token obtained from Google used to authorize access to Google BigQuery.

largeResultsDataSetId

The pre-created Google BigQuery dataset ID that is required in order to enable support for large result sets.

connectionSpec.id

The connection specification returns a source’s connector properties, including authentication specifications related to creating the base and source connections. The connection specification ID for Google BigQuery is: 3c9b37f8-13a6-43d8-bad3-b863b941fedd.

For more information about these values, refer to this Google BigQuery document.

Using Platform APIs

For information on how to successfully make calls to Platform APIs, see the guide on getting started with Platform APIs.

Create a base connection

A base connection retains information between your source and Platform, including your source’s authentication credentials, the current state of the connection, and your unique base connection ID. The base connection ID allows you to explore and navigate files from within your source and identify the specific items that you want to ingest, including information regarding their data types and formats.

To create a base connection ID, make a POST request to the /connections endpoint while providing your Google BigQuery authentication credentials as part of the request parameters.

API format

POST /connections

Request

The following request creates a base connection for Google BigQuery:

curl -X POST \
    'https://platform.adobe.io/data/foundation/flowservice/connections' \
    -H 'Authorization: Bearer {ACCESS_TOKEN}' \
    -H 'x-api-key: {API_KEY}' \
    -H 'x-gw-ims-org-id: {ORG_ID}' \
    -H 'x-sandbox-name: {SANDBOX_NAME}' \
    -H 'Content-Type: application/json' \
    -d '{
        "name": "Google BigQuery connection",
        "description": "Google BigQuery connection",
        "auth": {
            "specName": "Basic Authentication",
            "type": "OAuth2.0",
            "params": {
                    "project": "{PROJECT}",
                    "clientId": "{CLIENT_ID},
                    "clientSecret": "{CLIENT_SECRET}",
                    "refreshToken": "{REFRESH_TOKEN}"
                }
        },
        "connectionSpec": {
            "id": "3c9b37f8-13a6-43d8-bad3-b863b941fedd",
            "version": "1.0"
        }
    }'

Property

Description

auth.params.project

The project ID of the default Google BigQuery project to query. against.

auth.params.clientId

The ID value used to generate the refresh token.

auth.params.clientSecret

The client value used to generate the refresh token.

auth.params.refreshToken

The refresh token obtained from Google used to authorize access to Google BigQuery.

connectionSpec.id

The Google BigQuery connection specification ID: 3c9b37f8-13a6-43d8-bad3-b863b941fedd.

Response

A successful response returns details of the newly created connection, including its unique identifier (id). This ID is required to explore your data in the next tutorial.

{
    "id": "6990abad-977d-41b9-a85d-17ea8cf1c0e4",
    "etag": "\"ca00acbf-0000-0200-0000-60149e1e0000\""
}

Next steps

By following this tutorial, you have created a Google BigQuery base connection using the Flow Service API. You can use this base connection ID in the following tutorials:

recommendation-more-help

337b99bb-92fb-42ae-b6b7-c7042161d089