Monitor dataflows for sources in the UI
In Adobe Experience Platform, data is ingested from a wide variety of sources, analyzed within Experience Platform, and activated to a wide variety of destinations. Platform makes the process of tracking this potentially non-linear flow of data easier by providing transparency with dataflows.
The monitoring dashboard provides you with a visual representation of the journey of a dataflow. You can use an aggregated monitoring view and navigate vertically from the source level, to a dataflow, and to a dataflow run, allowing you to view the corresponding metrics that contribute to a dataflow’s success or failure. You can also use the monitoring dashboard’s cross-service monitoring capacity to monitor a dataflow’s journey from a source, to Identity Service, and to Profile.
This tutorial provides steps to monitor your dataflow, using both aggregated monitoring view and cross-service monitoring.
Getting started getting-started
This tutorial requires a working understanding of the following components of Adobe Experience Platform:
- Dataflows: Dataflows are a representation of data jobs that move data across Platform. Dataflows are configured across different services, helping move data from source connectors to target datasets, to Identity and Profile, and to Destinations.
- Dataflow runs: Dataflow runs are the recurring scheduled jobs based on the frequency configuration of selected dataflows.
- Sources: Experience Platform allows data to be ingested from various sources while providing you with the ability to structure, label, and enhance incoming data using Platform services.
- Identity Service: Gain a better view of individual customers and their behavior by bridging identities across devices and systems.
- Real-Time Customer Profile: Provides a unified, real-time consumer profile based on aggregated data from multiple sources.
- Sandboxes: Experience Platform provides virtual sandboxes which partition a single Platform instance into separate virtual environments to help develop and evolve digital experience applications.
Aggregated monitoring view aggregated-monitoring-view
In the Platform UI, select Monitoring from the left navigation to access the Monitoring dashboard. The Monitoring dashboard contains metrics and information on all sources dataflows, including insights into the health of data traffic from a source to Identity Service, and to Profile.
At the center of the dashboard is the Source ingestion panel, which contains metrics and graphs that display data on records ingested and records failed.
By default, the data displayed contains ingestion rates from the last 24 hours. Select Last 24 hours to adjust the time frame of records displayed.
A calendar pop-up window appears, providing you options for alternative ingestion time frames. Select Last 30 days and then select Apply
The graphs are enabled by default and you can disable them to expand the list of sources below. Select the Metrics and graphs toggle to disable the graphs.
failed status.The source ingestion list displays all sources that contain at least one existing account. The list also includes information on each source’s ingestion rate, number of failed records, and total number of failed dataflows based on the time frame that you applied.
To sort through the list of sources, select My sources and then select your category of choice from the dropdown menu. For example, to focus on cloud storages, select Cloud storage
To view all existing dataflows across all sources, select Dataflows.
Alternatively, you can enter a source into the search bar to isolate a single source. Once you have your source identified, select the filter icon
A list of dataflows appears. To narrow down the list and focus on dataflows with errors, select Show failures only.
Locate the dataflow that you want to monitor and then select the filter icon
The dataflow run page displays information on your dataflow’s run start date, size of data, status, as well as its processing time duration. Select the filter icon
The Dataflow run details page displays information on the dataflow’s metadata, partial ingestion status, and error summary. The error summary contains the specific top-level error that shows at which step the ingestion process encountered an error.
Scroll down to see more specific information on the error that occurred.
The Dataflow run errors panel displays the specific error and error code that resulted in the dataflow’s ingestion failure. In this scenario, a mapper transformation error occurred, resulting in the failure of 24 records.
Select Files for more information.
The Files panel contains information on the file’s name and path.
For a more granular representation of the error, select Preview error diagnostics.
The Error diagnostics preview window appears, displaying a preview of up to 100 errors in the dataflow. You can select Download to retrieve a curl command, which then allows you to download the error diagnostics.
When you are finished, select Close
You can use the breadcrumb system at the top header to navigate your way back to the Monitoring dashboard. Select Run start: 2/14/2021, 9:47 PM to return to the previous page, and then select Dataflow: Loyalty Data Ingestion Demo - Failed to return to the dataflows page.
Next steps next-steps
By following this tutorial, you have successfully monitored the ingestion dataflow from the source-level using the Monitoring dashboard. You have also successfully identified errors that contributed to the failure of dataflows during the ingestion process. See the following documents for more details: