> For the complete documentation index, see [llms.txt](https://docs.anecdoteai.com/docs/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.anecdoteai.com/docs/on-prem-setup/architecture-overview.md).

# Architecture Overview

<figure><img src="/files/9cNOrDwO8xs8ruhlPXlz" alt=""><figcaption></figcaption></figure>

**Data Flow**

* **Source Data Collection:** Data, potentially containing personally identifiable information (PII), is collected from various sources through the customer's instance of Airbyte.
* **PII Data Processing:** The collected data is then routed to the PII Destination, which is specifically designed to handle and remove any PII to comply with privacy standards.
* **Secure Data Transmission:** Post PII removal, the cleansed data is sent Anecdote infrastructure via a secure Injection Endpoint. This endpoint is protected by authentication through a unique credentials, ensuring that data transfer is both secure and reliable.

**Data Integration and Storage**

* **API Endpoint for Data Injection:** The sanitized data is received at the API endpoint designated for data injection within the Anecdote infrastructure.
* **Storage in S3 Bucket:** Once received, the data is stored in an S3 bucket, where it is held for further processing.
* **Data Pipeline:** The data is then processed in a data pipeline, which could involve aggregation, transformation, and loading into various data analysis tools or databases.

**Monitoring and Notifications**

* **Failure and Success Notifications:** The system is configured to send notifications in case of any failures or successful data collection. These notifications serve as a real-time monitoring tool to ensure the health and continuous operation of the data pipeline.
* **API Method for Notifications:** An API method is in place to facilitate these notifications, allowing for automated alerts and updates on system status.
* **Notification Channels:** All failure notifications are channeled to a designated Slack channel, providing immediate updates to the relevant teams for prompt action.