Create an Amazon Kinesis source connection using the Flow Service API
This tutorial walks you through the steps to connect Amazon Kinesis (hereinafter referred to as “Kinesis”) to Experience Platform, using the Flow Service API.
Getting started
This guide requires a working understanding of the following components of Adobe Experience Platform:
- Sources: Experience Platform allows data to be ingested from various sources while providing you with the ability to structure, label, and enhance incoming data using Platform services.
- Sandboxes: Experience Platform provides virtual sandboxes which partition a single Platform instance into separate virtual environments to help develop and evolve digital experience applications.
The following sections provide additional information that you will need to know in order to successfully connect Kinesis to Platform using the Flow Service API.
Gather required credentials
In order for Flow Service to connect with your Amazon Kinesis account, you must provide values for the following connection properties:
accessKeyId
secretKey
region
connectionSpec.id
86043421-563b-46ec-8e6c-e23184711bf6
.For more information on Kinesis access keys and how to generate them, refer to this AWS guide on managing access keys for IAM users.
Using Platform APIs
For information on how to successfully make calls to Platform APIs, see the guide on getting started with Platform APIs.
Create a base connection
The first step in creating a source connection is to authenticate your Kinesis source and generate a base connection ID. A base connection ID allows you to explore and navigate files from within your source and identify specific items that you want to ingest, including information regarding their data types and formats.
To create a base connection ID, make a POST request to the /connections
endpoint while providing your Kinesis authentication credentials as part of the request parameters.
API format
POST /connections
Request
curl -X POST \
'https://platform.adobe.io/data/foundation/flowservice/connections' \
-H 'Authorization: Bearer {ACCESS_TOKEN}' \
-H 'x-api-key: {API_KEY}' \
-H 'x-gw-ims-org-id: {ORG_ID}' \
-H 'x-sandbox-name: {SANDBOX_NAME}' \
-H 'Content-Type: application/json' \
-d '{
"name": "Amazon Kinesis connection",
"description": "Connector for Amazon Kinesis",
"providerId": "521eee4d-8cbe-4906-bb48-fb6bd4450033",
"auth": {
"specName": "Aws Kinesis authentication credentials",
"params": {
"accessKeyId": "{ACCESS_KEY_ID}",
"secretKey": "{SECRET_KEY}",
"region": "{REGION}"
}
},
"connectionSpec": {
"id": "86043421-563b-46ec-8e6c-e23184711bf6",
"version": "1.0"
}
}'
auth.params.accessKeyId
auth.params.secretKey
auth.params.region
connectionSpec.id
86043421-563b-46ec-8e6c-e23184711bf6
Response
A successful response returns details of the newly created base connection, including its unique identifier (id
). This ID is required in the next step to create a source connection.
{
"id": "4cb0c374-d3bb-4557-b139-5712880adc55",
"etag": "\"6507cfd8-0000-0200-0000-5e18fc600000\""
}
Create a source connection source
A source connection creates and manages the connection to the external source from where data is ingested. A source connection consists of information like data source, data format, and the source connection ID needed to create a dataflow. A source connection instance is specific to a tenant and organization.
To create a source connection, make a POST request to the /sourceConnections
endpoint of the Flow Service API.
API format
POST /sourceConnections
Request
curl -X POST \
'https://platform.adobe.io/data/foundation/flowservice/sourceConnections' \
-H 'Authorization: Bearer {ACCESS_TOKEN}' \
-H 'x-api-key: {API_KEY}' \
-H 'x-gw-ims-org-id: {ORG_ID}' \
-H 'x-sandbox-name: {SANDBOX_NAME}' \
-H 'Content-Type: application/json' \
-d '{
"name": "AWS Kinesis source connection",
"description": "A source connection for AWS Kinesis",
"baseConnectionId": "4cb0c374-d3bb-4557-b139-5712880adc55",
"connectionSpec": {
"id": "86043421-563b-46ec-8e6c-e23184711bf6",
"version": "1.0"
},
"data": {
"format": "json"
},
"params": {
"stream": "{STREAM}",
"dataType": "raw",
"reset": "latest"
}
}'
name
description
baseConnectionId
connectionSpec.id
86043421-563b-46ec-8e6c-e23184711bf6
data.format
json
.params.stream
params.dataType
raw
and xdm
.params.reset
latest
to start reading from the most recent data, and use earliest
to start reading from the first available data in the stream.Response
A successful response returns the unique identifier (id
) of the newly created source connection. This ID is required in the next tutorial to create a dataflow.
{
"id": "e96d6135-4b50-446e-922c-6dd66672b6b2",
"etag": "\"66013508-0000-0200-0000-5f6e2ae70000\""
}
Next steps
By following this tutorial, you have created a Kinesis source connection using the Flow Service API. You can use this source connection ID in the next tutorial to create a streaming dataflow using the Flow Service API.