System Architecture: Automated Email List Hygiene for Deliverability

System Overview#

This system provides a closed-loop, automated process for maintaining the health of our email marketing lists. Its primary function is to protect our sender reputation by systematically identifying and suppressing problematic email addresses based on real-time feedback and behavioral data. We built this to move away from reactive, manual list cleaning, which is inefficient and often occurs only after deliverability has already been damaged.

The core of this architecture is automation. We achieve this by capturing event webhooks directly from our email provider for critical events like bounces and spam complaints. Simultaneously, we run periodic checks for audience disengagement within our data warehouse. By combining these two data streams, we get a complete picture of list health. An email address that hard-bounces is a clear signal for immediate removal. A user who has not opened an email in three months represents a more subtle, but equally important, risk to our sender score.

The result of this implementation is a perpetually clean list. This is not a one-time project but a durable asset that continuously works in the background. A clean list directly improves campaign performance metrics, preserves our all-important sender score with inbox providers like Google and Microsoft, and significantly increases the probability of our messages reaching the primary inbox instead of the spam folder.

Prerequisites and Tools#

Before beginning the implementation, my team ensures that we have the necessary access and tooling in place. This preparation is critical for a smooth setup and prevents delays once the build process starts.

• Email Provider: I need administrative access to our email service provider (ESP) to configure webhooks. This is a non-negotiable requirement. This guide uses examples from common providers, but the principles apply to any ESP with a modern webhook system. We frequently work with SendGrid ↗, Postmark, and Mailgun ↗, all of which provide the necessary event-based triggers. I confirm that I can create, view, and edit webhook configurations.

• Data Warehouse: My team must have a data warehouse with write permissions to create and append to tables. This system is designed for scale, so a standard relational database may not be sufficient over the long term. I use platforms like Google BigQuery ↗, Snowflake ↗, or Amazon Redshift ↗ because of their ability to handle massive datasets and their native integration capabilities with other cloud services. Standard SQL knowledge is required to interact with the warehouse.

• Automation Platform: A platform capable of ingesting webhook data, running scheduled tasks, and making API calls is the engine of this system. My team uses no-code or low-code tools like Zapier and Make for their rapid development cycles. However, a custom serverless application (e.g., AWS Lambda, Google Cloud Functions) is an equally viable, and often more scalable, option for teams with engineering resources. The platform must be able to provide a stable, public URL to receive webhooks and execute conditional logic.

• API Credentials: I secure the necessary API keys for both the email provider and the data warehouse. These credentials grant our automation platform the authority to read and write data. It is a critical best practice to treat these keys as sensitive secrets. I store them as secure environment variables or in a dedicated secrets manager within the automation platform. They must never be stored in plain text ↗, committed to source code repositories, or shared insecurely ↗. Proper API key management ↗ is foundational to the security of our entire infrastructure.

Step-by-Step Implementation#

I follow these six steps methodically to build the complete suppression system. Each step builds upon the last, creating a robust data pipeline from event detection to final suppression.

Step 1: Configure Webhooks in the Email Provider

The first action is to instruct our email provider to send us real-time notifications for suppression events. I navigate to the webhook settings section within our email provider's admin panel. For example, in SendGrid, this is under Settings > Mail Settings > Event Webhooks ↗.

I create a new webhook and select only the specific events that indicate a permanent problem with an email address. The essential events are:

• Hard Bounce: Indicates a permanent delivery failure (e.g., the address does not exist). Mailgun calls this event permanent_fail.
• Spam Complaint: Occurs when a recipient marks our email as spam.
• Unsubscribe: Captures when a user clicks the native unsubscribe link from the provider.

I leave the destination URL field blank for a moment, as I will generate it in the next step. Once I have the URL, I will paste it here and activate the webhook.

Step 2: Establish the Data Ingestion Workflow

Within our automation platform (e.g., Zapier, Make), I build a new workflow. The trigger for this workflow must be a webhook listener. When I create this trigger, the platform provides a unique, public URL. This URL is the destination endpoint for the webhook.

I copy this unique URL and return to the email provider's webhook settings from Step 1. I paste the URL into the "Destination URL" or "Endpoint URL" field and save the configuration. At this point, any new bounce, complaint, or unsubscribe event in our email provider will send an HTTP POST request with a JSON payload to our automation workflow's listener. The connection is now live.

Step 3: Process and Standardize Incoming Data

The raw data arriving from the webhook is structured according to the provider's specific format. It needs to be parsed before it can be used. I add a step to my workflow immediately after the webhook trigger to handle the incoming JSON payload.

The goal is to extract the key pieces of information and standardize them into a consistent format. The payload structure varies by provider. For example:

• A SendGrid bounce event might look like this: [{"email": "user@example.com", "timestamp": 1678886400, "event": "bounce", "reason": "550 5.1.1 The email account that you tried to reach does not exist."}]
• A Mailgun complaint event might have a different structure: {"event-data": {"recipient": "user@example.com", "timestamp": 1678886400}, "event": "complained"}

In my workflow, I add a data transformation step. I extract the recipient's email, the event type, and the event timestamp. I map the provider-specific event names to our internal, standardized names (e.g., bounce, complaint, unsubscribe). My standardized output object for every event, regardless of source, looks like this:

{
  "email_address": "user@example.com",
  "event_type": "bounce",
  "event_timestamp": "2023-03-15T12:00:00Z"
}
This standardization ensures that the data entering our warehouse is clean and consistent.
 
#### Step 4: Load Suppression Data into the Data Warehouse
 
With the data parsed and standardized, I now configure a connector to our data warehouse. This step in the workflow executes a SQL `INSERT` statement to add the processed data from Step 3 into a dedicated `suppression_list` table.
 
Before I can insert data, I must first define the table schema with data integrity in mind. I run the following DDL statement in our warehouse:
 
```sql
CREATE TABLE suppression_list (
    id SERIAL PRIMARY KEY,
    email_address VARCHAR(255) NOT NULL,
    event_type VARCHAR(50) NOT NULL,
    event_timestamp TIMESTAMP WITH TIME ZONE,
    inserted_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP
);
The columns serve specific purposes: `email_address` is the user identifier, `event_type` stores our standardized event name, `event_timestamp` is the time the event occurred, and `inserted_at` is a metadata field that tells us when our system processed it.
 
The workflow then runs a parameterized `INSERT` statement for each incoming event:
 
```sql
INSERT INTO suppression_list (email_address, event_type, event_timestamp)
VALUES ('user@example.com', 'bounce', '2023-03-15T12:00:00Z');

Step 5: Automate Identification of Unengaged Segments

This is a separate, scheduled process that runs independently of the real-time webhook workflow. Its purpose is to identify subscribers who are no longer active, which is a proactive measure to protect our sender reputation.

I write a SQL query that joins our email_sends and email_opens tables to find users who have been sent emails but have not opened any in the last 90 days. The exact query depends on the schema, but it generally follows this logic:

INSERT INTO suppression_list (email_address, event_type, event_timestamp)
SELECT
    s.recipient_email,
    'unengaged',
    CURRENT_TIMESTAMP
FROM
    email_sends s
LEFT JOIN
    email_opens o ON s.recipient_email = o.recipient_email AND o.open_date >= NOW() - INTERVAL '90 DAY'
WHERE
    s.send_date >= NOW() - INTERVAL '90 DAY'
GROUP BY
    s.recipient_email
HAVING
    COUNT(o.id) = 0
ON CONFLICT (email_address) DO NOTHING;

This query identifies the unengaged segment and inserts their email addresses directly into our suppression_list table with the event type unengaged. I configure this query to run on a weekly schedule using the scheduling feature in our automation platform or a dedicated SQL runner like dbt.

Step 6: Build the Sync-Back Automation

Finally, I close the loop. The data is now in our warehouse, but the email provider is not yet aware that it should stop sending to these addresses. I create a new scheduled workflow that runs daily.

1. Query for New Suppressions: The workflow begins by querying the suppression_list table for entries added in the last 24 hours.

   SELECT email_address FROM suppression_list WHERE inserted_at >= NOW() - INTERVAL '1 DAY';

2. Loop and API Call: The workflow iterates through each email address returned by the query. In each iteration, it makes an API call to our email provider to add that email to its global suppression list. For SendGrid, this involves a POST request to the /v3/asm/suppressions/global ↗ endpoint with a payload like:

   {
     "recipient_emails": ["some.user@example.com", "another.user@example.com"]
   }

This final step ensures that our central suppression list is synchronized back to the email delivery tool, preventing any future sends to the problematic addresses.

Troubleshooting and Maintenance#

Even a well-designed automated system requires monitoring and occasional intervention. Here is how I address common issues.

• Webhook Failures: I periodically check the webhook logs in both the email provider and our automation platform. A common issue is a malformed URL or an authentication failure. I verify the endpoint URL is correct, active, and publicly accessible. Some providers, like Postmark, will retry failed webhooks but eventually disable them after persistent failures ↗, so it is important to resolve these issues promptly.

• Data Mismatches: If data appears incorrectly in the warehouse (e.g., a null email address), I re-examine the parsing step (Step 3) in the automation workflow. It is critical to consult the email provider's developer documentation for the exact structure of their webhook payloads, as these can be updated. I maintain a small set of test cases to validate my parsing logic whenever I make changes.

• API Rate Limits: During the initial setup, a large backfill of unengaged users might cause the sync-back automation (Step 6) to hit the email provider's API rate limits ↗. To manage this, I implement an exponential backoff retry strategy ↗ and add a small, fixed delay (e.g., 200 milliseconds) between API calls within my workflow's loop. This distributes the load and keeps our activity within acceptable limits.

• Query Performance: For very large datasets, the unengaged segment query from Step 5 can become slow and resource-intensive. To combat this, we ensure our database tables are properly indexed. An index acts like a book's index, allowing the database to find the data it needs without scanning the entire table. I place indexes on the columns used in JOIN conditions and WHERE clauses, such as recipient_email and the date columns. A simple CREATE INDEX statement can dramatically improve query speed ↗.

  CREATE INDEX idx_sends_recipient_date ON email_sends (recipient_email, send_date);
  CREATE INDEX idx_opens_recipient_date ON email_opens (recipient_email, open_date);

Expected Results#

The successful implementation of this system yields several tangible and highly valuable outcomes for our organization.

• A sustained hard bounce rate below 0.5 percent and a spam complaint rate consistently below 0.02 percent. These metrics are well within industry standards and signal to inbox providers that we are a responsible sender.
• A measurable improvement in open and click-through rates. By removing unengaged subscribers and invalid addresses, our audience becomes more concentrated with people who want to receive our communications, which naturally lifts engagement metrics.
• A durable, automated asset that protects our domain's sending reputation. This system eliminates the recurring, manual effort of list-cleaning tasks from my team, freeing them to focus on more strategic work.
• The creation of a centralized suppression_list table in our data warehouse. This table becomes the single source of truth for do-not-contact directives. We can then extend its use to other communication platforms, such as SMS or push notification systems, ensuring a consistent and compliant user experience across all channels.