Architecture Overview

Data Flow

  • Source Data Collection: Data, potentially containing personally identifiable information (PII), is collected from various sources through the customer's instance of Airbyte.

  • PII Data Processing: The collected data is then routed to the PII Destination, which is specifically designed to handle and remove any PII to comply with privacy standards.

  • Secure Data Transmission: Post PII removal, the cleansed data is sent Anecdote infrastructure via a secure Injection Endpoint. This endpoint is protected by authentication through a unique credentials, ensuring that data transfer is both secure and reliable.

Data Integration and Storage

  • API Endpoint for Data Injection: The sanitized data is received at the API endpoint designated for data injection within the Anecdote infrastructure.

  • Storage in S3 Bucket: Once received, the data is stored in an S3 bucket, where it is held for further processing.

  • Data Pipeline: The data is then processed in a data pipeline, which could involve aggregation, transformation, and loading into various data analysis tools or databases.

Monitoring and Notifications

  • Failure and Success Notifications: The system is configured to send notifications in case of any failures or successful data collection. These notifications serve as a real-time monitoring tool to ensure the health and continuous operation of the data pipeline.

  • API Method for Notifications: An API method is in place to facilitate these notifications, allowing for automated alerts and updates on system status.

  • Notification Channels: All failure notifications are channeled to a designated Slack channel, providing immediate updates to the relevant teams for prompt action.

Last updated