Aggregation policy

To ensure maximum efficiency when exporting data to your API endpoint, you can use various settings to aggregate exported profiles into larger or smaller batches, group them by identity, and other use cases. This also allows you to tailor data exports to any downstream limitations on your API endpoint (rate limiting, number of identities per API call, etc.).

Use configurable aggregation to dive deep into the settings provided by Destination SDK or use best effort aggregation to tell Destination SDK to batch the API calls as best as it can.

When building a real-time (streaming) destination with Destination SDK, you can configure how the exported profiles should be combined in the resulting exports. This behavior is determined by the aggregation policy settings.

To understand where this component fits into an integration created with Destination SDK, see the diagram in the configuration options documentation or see the guide on how to use Destination SDK to configure a streaming destination.

You can configure the aggregation policy settings via the /authoring/destinations endpoint. See the following API reference pages for detailed API call examples where you can configure the components shown in this page.

This article describes all the supported aggregation policy settings that you can use for your destination.

After reading through this document, refer to the documentation on using templating and the aggregation key examples to understand how to include the aggregation policy in your message transformation template based on your selected aggregation policy.

Supported integration types supported-integration-types

Refer to the table below for details on which types of integrations support the functionality described on this page.

Best effort aggregation best-effort-aggregation

Best effort aggregation works best for destinations which prefer fewer profiles per request and would rather take in more requests with less data than fewer requests with more data.

The example configuration below shows a best effort aggregation configuration. For an example of configurable aggregation, see the configurable aggregation section. The parameters applicable to best effort aggregation are documented in the table below.

"aggregation":{
   "aggregationType":"BEST_EFFORT",
   "bestEffortAggregation":{
      "maxUsersPerRequest":10,
      "splitUserById":false
   }
}

Configurable aggregation configurable-aggregation

Configurable aggregation works best if you’d rather take in large batches, with thousands of profiles on the same call. This option also allows you to aggregate the exported profiles based on complex aggregation rules.

The example configuration below shows a configurable aggregation configuration. For an example of best effort aggregation, see the best effort aggregation section. The parameters applicable to configurable aggregation are documented in the table below.

"aggregation":{
   "aggregationType":"CONFIGURABLE_AGGREGATION",
   "configurableAggregation":{
      "splitUserById":true,
      "maxBatchAgeInSecs":2400,
      "maxNumEventsInBatch":5000,
      "aggregationKey":{
         "includeSegmentId":true,
         "includeSegmentStatus":true,
         "includeIdentity":true,
         "oneIdentityPerGroup":true,
         "groups":[
            {
               "namespaces":[
                  "IDFA",
                  "GAID"
               ]
            },
            {
               "namespaces":[
                  "EMAIL"
               ]
            }
         ]
      }
   }
}

Next steps next-steps

After reading this article, you should have a better understanding of how you can configure aggregation policies for your destination.

To learn more about the other destination components, see the following articles: