> For the complete documentation index, see [llms.txt](https://docs.anecdoteai.com/docs/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.anecdoteai.com/docs/on-prem-setup/why-airbyte-and-its-technical-details.md).

# Why Airbyte and Its Technical Details

Selecting Airbyte was pivotal for its ease of deployment and in-memory processing capabilities, which align with our requirements for a secure and efficient data handling solution.

**Airbyte Advantages:** Perfect for quick deployment and in-memory data processing.

**Understanding How Scraping Works:** Airbyte's role in data integration is vital, streamlining the collection and consolidation of information.

* **Data Collection:** Primarily driven by [**HTTP-API-based Sources**](https://docs.airbyte.com/connector-development/cdk-python/http-streams), constituting 90% of our data collection method. Airbyte processes HTTPS API requests, accumulating responses in a buffer before relaying them to the intended destination.
* **Source Code:** Discover more at the [Airbyte GitHub Repository](https://github.com/airbytehq/airbyte/tree/master/airbyte-integrations/connectors).

**Exclusive In-Memory Storing:** Emphasizing the security and integrity of data during processing through Airbyte's in-memory data management approach.

* **Temporary Storage:** Airbyte Sources store data in a buffer until collection completion, then transmit to the Destination. Post-PII removal, data is relayed to Anecdote’s infrastructure through the Injection End Point.
* **In-Memory Processing:** Ensures data security and integrity during record processing