Amazon Marketing Stream Setup: Hourly Ad Data via Firehose in Python

Here's the pattern most Amazon Ads developers fall into. You write a scheduled job that runs every hour, calls the Amazon Ads Reports API, waits for the report to generate, downloads it, and loads it into your database. It works — until it doesn't.

The Reports API has rate limits. When you're running the same job across multiple client profiles, the requests pile up. You start seeing throttling errors. You add retry logic. The job gets slower. And after all that, the data is still from yesterday — the Reports API gives you daily granularity, not hourly.

Amazon Marketing Stream (AMS) solves both problems. It's a push-based system. Amazon pushes your hourly ad data into an AWS queue or stream that you provision. No polling, no throttling, and genuine hourly granularity. Your data arrives within minutes of each hour closing.

Amazon Marketing Stream is an official Amazon Ads API feature. You subscribe a specific advertiser profile to a dataset — say, sp-traffic for Sponsored Products traffic metrics — and Amazon starts delivering that dataset's hourly snapshots to an AWS destination you own.

Two destination types are supported:

• Amazon SQS — messages arrive in a queue, consumed by a Lambda or worker
• Amazon Data Firehose — data is buffered and written directly to S3 (or Redshift, Snowflake, etc.)

This post covers the Firehose path. It's simpler operationally — no Lambda needed to fan out messages, no confirmation handshake. Data lands straight in S3, partitioned by year/month/day/hour.

Available datasets

• sp-traffic — Sponsored Products: impressions, clicks, spend by hour
• sp-conversion — Sponsored Products: conversion and sales data
• budget-usage — budget consumption for SP, SB, and SD campaigns
• sb-traffic — Sponsored Brands traffic (select markets)
• sd-traffic — Sponsored Display (added globally in late 2025)

Each dataset requires its own separate subscription call. You can run multiple subscriptions in parallel — one per dataset per advertiser profile.

Two prerequisites before touching the AWS console:

1. Amazon Ads API access — a registered app with a client_id, client_secret, and a valid refresh_token for the advertiser account you want to subscribe. If you don't have this yet, see the Ads API access setup guide.
2. An AWS account — logged into the correct region. This is critical: Marketing Stream only delivers data to three specific AWS regions.

You'll be collecting three ARNs as you go through setup. Open a text file now and keep it handy:

1. Firehose delivery stream ARN
2. FIREHOSE_SUBSCRIPTION_ROLE ARN
3. FIREHOSE_SUBSCRIBER_ROLE ARN

The Firehose stream is the pipe that receives data from Marketing Stream and writes it to S3. Start here.

1. AWS Console → search Amazon Data Firehose → open it
2. Click Create Firehose stream
3. Source → Direct PUT
4. Destination → Amazon S3
5. Give the stream a meaningful name, e.g. ams-sp-traffic-stream
6. Skip the "Transform and convert records" section
7. Under Destination settings → click Create to make a new S3 bucket
   • Confirm the region matches your advertiser region
   • Bucket name: something like ams-sp-traffic-data (lowercase only)
   • Leave everything else default → Create bucket
8. Back on the Firehose page → click Browse, reload, select the bucket
9. Scroll down → Create Firehose stream

Once the stream is created, open it and copy the ARN. It looks like:

arn:aws:firehose:eu-west-1:XXXXXXXXXXXX:deliverystream/ams-sp-traffic-stream

That's ARN #1 — paste it in your text file as deliveryStreamArn.

This role gives Amazon's SNS service permission to PUT records into your Firehose stream. It's not a role you'll assume yourself — Amazon's internal infrastructure assumes it on your behalf when it delivers data.

1. IAM → Roles → Create role
2. Trusted entity type → Custom trust policy
3. Replace the policy JSON with this:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": { "Service": "sns.amazonaws.com" },
            "Action": "sts:AssumeRole"
        },
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::926844853897:role/ReviewerRole"
            },
            "Action": "sts:AssumeRole"
        }
    ]
}

The account ID 926844853897 is Amazon's internal account — it's a fixed value, don't change it.

4. Click Next (skip Add permissions) → Next
5. Name: FIREHOSE_SUBSCRIPTION_ROLE → Create role
6. Click into the new role → Permissions → Add permissions → Create inline policy
7. Toggle to JSON editor, paste this (swap in your account ID and exact stream name):

{
    "Version": "2012-10-17",
    "Statement": [{
        "Action": [
            "firehose:DescribeDeliveryStream",
            "firehose:ListTagsForDeliveryStream",
            "firehose:ListDeliveryStreams",
            "firehose:PutRecord",
            "firehose:PutRecordBatch"
        ],
        "Resource": "arn:aws:firehose:eu-west-1:YOUR_ACCOUNT_ID:deliverystream/ams-sp-traffic-stream",
        "Effect": "Allow"
    }]
}

8. Name the inline policy FirehoseSubscriptionRolePolicy → Create policy
9. Copy the role ARN from the Summary panel → paste as subscriptionRoleArn (ARN #2)

This role allows Marketing Stream to create an SNS subscription on your behalf — it's the identity that AMS assumes when setting up the data flow from Amazon's side to yours.

1. IAM → Roles → Create role
2. Trusted entity type → Custom trust policy
3. Paste this trust policy:

{
    "Version": "2012-10-17",
    "Statement": [{
        "Effect": "Allow",
        "Principal": {
            "AWS": ["arn:aws:iam::926844853897:role/SubscriberRole"]
        },
        "Action": ["sts:AssumeRole", "sts:TagSession"]
    }]
}

4. Click Next (skip the managed permissions list entirely) → Next
5. Name: FIREHOSE_SUBSCRIBER_ROLE → Create role
6. Click into the role → Add permissions → Create inline policy → JSON editor
7. Paste this (swap in your account ID, and use the correct SNS ARN for your dataset + region from the table below):

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": "iam:PassRole",
            "Resource": "arn:aws:iam::YOUR_ACCOUNT_ID:role/FIREHOSE_SUBSCRIPTION_ROLE",
            "Effect": "Allow"
        },
        {
            "Action": ["sns:Subscribe", "sns:Unsubscribe"],
            "Resource": "arn:aws:sns:eu-west-1:668473351658:*",
            "Effect": "Allow"
        }
    ]
}

SNS ARN table — use the right one for your dataset and region

8. Name the policy Firehose_SubscriberRole_Policy → Create policy
9. Copy this role's ARN from the Summary panel → paste as subscriberRoleArn (ARN #3)

You now have all three ARNs. Time to make the API call.

With your three ARNs in hand, make a single POST request to the Marketing Stream subscriptions endpoint:

curl --location --request POST 'https://advertising-api-eu.amazon.com/streams/subscriptions' \
  --header 'Amazon-Advertising-API-ClientId: YOUR_CLIENT_ID' \
  --header 'Amazon-Advertising-API-Scope: YOUR_PROFILE_ID' \
  --header 'Content-Type: application/vnd.MarketingStreamSubscriptions.StreamSubscriptionResource.v1.0+json' \
  --header 'Authorization: Bearer YOUR_ACCESS_TOKEN' \
  --data-raw '{
     "clientRequestToken": "ams-myclient-sp-traffic-001",
     "dataSetId": "sp-traffic",
     "notes": "sp-traffic subscription via Firehose",
     "destination": {
        "firehoseDestination": {
            "deliveryStreamArn": "YOUR_FIREHOSE_DELIVERY_STREAM_ARN",
            "subscriptionRoleArn": "YOUR_FIREHOSE_SUBSCRIPTION_ROLE_ARN",
            "subscriberRoleArn": "YOUR_FIREHOSE_SUBSCRIBER_ROLE_ARN"
        }
     }
  }'

A successful response looks like:

{
    "subscriptionId": "amzn1.fead.cs1.xxxxxxxxxxxxxxxx",
    "clientRequestToken": "ams-myclient-sp-traffic-001"
}

No confirmation handshake needed. Unlike the SQS path, Firehose subscriptions are active immediately after the 200 response.

Verify the subscription is active

curl --location --request GET 'https://advertising-api-eu.amazon.com/streams/subscriptions' \
  --header 'Amazon-Advertising-API-ClientId: YOUR_CLIENT_ID' \
  --header 'Amazon-Advertising-API-Scope: YOUR_PROFILE_ID' \
  --header 'Authorization: Bearer YOUR_ACCESS_TOKEN'

Look for "status": "ACTIVE" in the response. If it's there, data will start landing within the hour.

After the subscription goes active, wait up to an hour. Marketing Stream drops the previous hour's data right at the start of the next hour.

Go to S3 → open your bucket. You'll see Firehose has created a folder structure automatically:

year=2026/month=05/day=04/hour=13/

Inside the hour folder are files with UUID-style names and no extension — these are the raw batches Firehose wrote. Download one and open it.

The format is newline-delimited JSON: one JSON record per line. Each record represents one keyword or placement for that hour window. Here's what an sp-traffic record looks like:

{
  "advertiserId": "ENTITY123456",
  "marketplaceId": "A1PA6795UKMFR9",
  "dataSetId": "sp-traffic",
  "date": "2026-05-04",
  "hour": 13,
  "campaignId": "111111111",
  "adGroupId": "222222222",
  "keywordText": "wireless earbuds",
  "matchType": "BROAD",
  "placement": "Top of Search (on-product)",
  "impressions": 4210,
  "clicks": 87,
  "cost": 34.56
}

From here you can load this straight into a database, run a Python parser over it, wire it into a dashboard, or feed it into automated bid logic. The next video in this series covers building hourly bid automation on top of this data.

These are the errors that appear most often when setting up Marketing Stream for the first time:

Once hourly ad data is landing in S3, the real work begins. Here's what teams typically build on top of a Marketing Stream pipeline:

• Intraday bid automation — adjust keyword bids based on the current hour's ACOS, before the day's budget is spent on bad traffic. The AMS automated bidding post covers this in detail.
• Budget pacing — track hourly spend against the daily budget and throttle bids or pause campaigns before the budget runs out mid-morning.
• Real-time performance dashboards — load each hour's S3 files into a database (Postgres, BigQuery, Redshift, Snowflake) and build live dashboards that don't rely on manually-pulled reports.
• Anomaly detection — flag campaigns whose spend or CPC jumps unexpectedly within the hour, before a daily report would catch it.
• Multi-client aggregation — run one subscription per advertiser profile, land all data in the same S3 bucket, and build a unified view across all your managed accounts.

Amazon also published a Snowflake integration guide in early 2025 — if your analytics stack is Snowflake, you can route Firehose directly into it instead of S3.

• Amazon Marketing Stream overview — official docs
• Marketing Stream onboarding guide
• Data guide — dataset schemas
• amzn/amazon-marketing-stream-examples — official CDK reference implementation (GitHub)
• AMS + Snowflake integration guide (Amazon, Jan 2025)
• Databaaba: How to Get Amazon Ads API Access
• Databaaba: Hourly Bid Automation with Amazon Marketing Stream

Do I need AWS CDK to set up Amazon Marketing Stream?

No. The official Amazon reference implementation uses AWS CDK, which automates the infrastructure provisioning. But you can do everything manually through the AWS console — that's exactly what this guide covers. CDK is a shortcut, not a requirement.

Which AWS regions does Amazon Marketing Stream support?

Only three: us-east-1 for North America, eu-west-1 for Europe, and us-west-2 for Far East. Any other region will either fail outright or accept the subscription but never deliver data.

Can I subscribe to multiple datasets at once?

Each dataset (sp-traffic, sp-conversion, budget-usage, etc.) needs its own POST call to /streams/subscriptions with a unique clientRequestToken. You can make these calls back to back — there's no limit to how many subscriptions one profile can have.

Why does my subscription call return "duplicate idempotent token"?

That means the subscription already exists from a previous call that used the same clientRequestToken. It's not an error — Amazon is telling you it already processed that request. Run a GET on /streams/subscriptions and you'll see the active subscription listed.

Does Amazon Marketing Stream cover Sponsored Brands and Sponsored Display?

Yes. sb-traffic (Sponsored Brands) and sd-traffic (Sponsored Display) are available — SD was added globally in late 2025. Each needs its own subscription call. The SNS ARNs differ by dataset, so use the table in Phase 3 to get the right one.