How to Run an App Store A/B Test You Can Actually Trust

Changing your screenshots and watching downloads afterward is not an A/B test.

Downloads may change because of seasonality, featuring, an app release, Apple Ads, a creator post, or a different mix of countries. A before-and-after comparison cannot separate all of that from the new creative.

Product Page Optimization gives you a better comparison. Apple shows the original product page and one or more treatments during the same period, then reports how each version converts.

The tool is easy to start. The difficult part is deciding what to test and knowing when the result is strong enough to use.

What Product Page Optimization does

Product Page Optimization, often shortened to PPO, is Apple's native product-page testing system.

The original page is the control. A treatment changes supported creative assets and receives a share of App Store traffic. Apple compares each treatment with the original page.

Apple's current treatment-configuration documentation supports testing:

• screenshots
• app previews
• app icons already included correctly in the current app binary

Apple currently allows up to three treatments in one test. A test runs for 90 days unless you stop it earlier. More treatments split the available traffic and can make a useful conclusion take longer. Apple's current test-creation guide documents the treatment limit, traffic split, duration estimate, and 90-day limit.

This is not the same as a custom product page.

Use PPO when the audience and question should stay the same. Use a custom page when the intended audience or promise changes.

The custom product pages guide covers that second case; do not use PPO to mix separate audiences into one creative test.

Apple's current setup does not run PPO tests on custom product pages, Apple Watch product pages, or iMessage product pages. Apple's PPO overview also warns that releasing a version containing assets or metadata under test may affect the result.

Do not start with an asset

“I want to test a new blue design” is not enough.

Start with a problem supported by evidence.

Useful evidence can come from:

• a short comprehension test where people misunderstand the first screenshot
• product-page views that do not turn into downloads
• reviews that reveal a missing expectation
• support questions about a feature the page hides
• competitor pages that make an important category convention clearer
• Apple Ads terms whose intent does not match the default screenshot sequence
• product research showing that users value something the listing barely mentions

Now write the hypothesis:

Because [evidence], I believe [specific audience] does not understand [specific value or proof]. Changing [asset and message] will improve [primary metric] without materially hurting [guardrail].

Hypothetical example:

Because new users repeatedly ask whether lists sync between two phones, I believe the current first screenshot makes the app look like a personal grocery list. Leading with shared editing and showing a live change will improve first-time-download conversion without reducing first-list creation.

This gives the treatment a job. It also tells you what to look at after the App Store result.

Pick one coherent hypothesis

“One hypothesis” does not always mean changing one image file.

If the idea is that collaboration is unclear, the first three screenshots may need to change together:

1. name the shared-list outcome
2. show a live update
3. show either person checking off an item

That is still one hypothesis.

Changing the audience, claim, color system, type style, screenshot order, app preview, and icon at once creates several explanations for the result. You may find a winning set, but you will learn less about why it worked.

Keep an asset-change log:

If a changed asset has no connection to the hypothesis, leave it alone.

Check whether the app has enough traffic

Low-traffic apps can run PPO tests. They may not collect enough evidence to reach a clear result within 90 days.

When you create a test, Apple estimates duration and required impressions using existing daily impressions, new downloads, and the improvement target you select. The estimate is guidance, not a guarantee.

Before investing in several treatments, ask:

• How many impressions and first-time downloads does this storefront receive?
• How much of the traffic can be assigned to treatments?
• Is the expected change large enough to detect?
• Will a product launch, seasonal event, or campaign alter the traffic soon?
• Can one treatment answer the question more efficiently than three?

For a smaller app, one meaningfully different treatment is usually more useful than three subtle alternatives. If the estimate already approaches the 90-day limit, splitting traffic further works against you.

Test results appear in Analytics after five first-time downloads are associated with the test. Five downloads do not make the result reliable. Apple's run-test instructions separate this reporting threshold from the evidence needed to reach a result.

Choose the treatment count

Use one treatment when:

• you have one strong hypothesis
• traffic is limited
• you want the simplest decision
• the new sequence is meaningfully different from the original

Use more treatments when:

• traffic can support them
• each treatment represents a distinct, documented version of the same hypothesis or clearly separated hypotheses
• you are prepared for a longer test
• the additional comparison is worth delaying the answer

Do not add a second treatment because the interface allows it.

Name treatments by the idea:

• shared-list-first
• outcome-captions
• ui-led-sequence

“Treatment B” will not help when you read the history six months later.

Choose the traffic proportion

The traffic proportion is the percentage of users who can be shown a treatment instead of the original page.

If you select three treatments and assign 30% of traffic to treatments, Apple divides that treatment traffic across them. Each treatment receives 10%, while the original receives the remaining 70%.

The trade-off is straightforward:

• More treatment traffic can produce evidence faster.
• More original-page traffic reduces exposure to an uncertain treatment.

Use the business risk and traffic level to decide. A small copy clarification with a strong pre-test review may justify a larger treatment share. A radical change to a high-volume page may deserve more control traffic and close monitoring.

Do not choose the split by habit. Record why it fits this test.

Keep localizations interpretable

Apple lets you select existing localizations for the test.

A visual-only change may be comparable across several markets. A caption or positioning change usually is not.

If the treatment contains language, start with one localization unless:

• the same hypothesis has been researched in every included market
• the copy was rewritten and reviewed for each language
• category expectations are sufficiently similar
• you will still be able to understand market-level differences

Combining several localizations can help traffic volume while hiding opposite results. A treatment can help in one market and hurt in another.

The Appfigures testing guide recommends keeping the first test to one localization. That is a sound default for interpretability, not a universal rule.

Prepare the assets and approval state

By default, treatments copy the original product page.

You can then change supported icons, screenshots, and previews. If you leave an asset unchanged for a device size, Apple uses the original-page asset.

New screenshots and previews need approval before the test can run. Apple lets you submit this metadata without submitting a new app version.

There are two useful exceptions:

• Reordering screenshots or previews already approved on the App Store does not require another metadata submission.
• App icons need to be included in the current binary using supported alternate-icon assets. The icon shown in the treatment follows the download and appears on the user's device.

Review every device size and localization in the treatment. A polished iPhone treatment with missing iPad assets can create a comparison you did not intend.

Apple's treatment configuration page has the current icon and asset requirements.

Create the test in App Store Connect

The app must be live on the App Store with a Pre-Order Ready for Distribution or Ready for Distribution status.

Current setup:

1. Open the app in App Store Connect.
2. Select Product Page Optimization in the sidebar.
3. Create a test.
4. Enter a reference name that describes the hypothesis.
5. Choose one to three treatments.
6. Choose the traffic proportion.
7. Select the localizations.
8. Review Apple's estimated duration and required impressions.
9. Create the test.
10. Rename and configure each treatment.
11. Submit new treatment metadata for approval where required.
12. Confirm the final assets, device sizes, and localization selection.
13. Start the test.

All supported localizations are selected by default. A user shown an excluded localization is not included in the test, so verify this setting instead of clicking through it.

Save a screenshot or export of every setting. The result is hard to interpret if nobody remembers the traffic split or which localizations were included.

Apple documents the current creation flow here.

Do not change the baseline casually

An A/B test is useful because treatments run during the same period as the original page.

That does not make it immune to operational changes.

Record:

• app releases during the test
• changes to the default screenshots or previews
• major Apple Ads budget changes
• featuring or promotional events
• price and subscription changes
• outages, crashes, or onboarding changes
• unusual seasonal traffic
• changes in country or source mix

Some events affect every variant. Others change which users enter the test or what happens after download.

Avoid unrelated product-page changes while the test runs. If a critical issue requires a change, make it, record it, and decide whether the test still answers the original question.

Read the current Apple result correctly

Apple's current Analytics uses Bayesian analysis for Product Page Optimization.

The dashboard reports:

• unique impressions
• estimated conversion rate
• estimated relative lift from the original
• confidence
• a performance status

The current statuses are:

Collecting Data

Apple does not yet have enough evidence to say the treatment performs differently from the original.

Performing Better

The treatment performs better than the baseline with at least 90% confidence.

Performing Worse

The treatment performs worse than the baseline with at least 90% confidence.

Likely to be Inconclusive

Based on current traffic and results, Apple expects the test may not reach enough evidence within 90 days.

Apple also shows a credible interval: a probable range for the conversion rate or lift. A wide range means the likely effect is still uncertain, even if the center estimate looks attractive.

Do not reduce the result to one green percentage. Read the status, confidence, interval, raw exposure, and duration together. Apple explains the current Analytics interpretation here.

Do not use a calendar as the decision rule

“Run it for a week” is an operating reminder, not a statistical conclusion.

The Appfigures guide correctly warns that looking too early is misleading. Its recommendation to wait at least a week or often a month is safer than declaring a winner after two days, but elapsed time alone does not make the result reliable.

Use Apple's confidence and status. A high-traffic app may reach an answer quickly. A low-traffic app may still be collecting data after a month.

Do not stop a treatment the first time its estimated lift moves above zero. Early estimates often move as the sample changes.

Set review points for operational health, not winner selection:

• Is the test running?
• Did the correct assets appear?
• Is traffic being allocated?
• Did a release or campaign change occur?
• Is a treatment clearly creating unacceptable harm?

Let the evidence determine the final decision.

Add business guardrails before the test

PPO is designed to compare App Store conversion. A higher download conversion rate is useful, but it is not the entire business result.

Before starting, choose one or two downstream guardrails:

• first-value completion
• onboarding completion
• trial start
• purchase or subscription conversion
• early retention
• refund or cancellation signal

Use the ASO-to-activation guide when the team still needs to define first value or separate acquisition cohorts from the overall product funnel.

Be precise about attribution. Apple's PPO dashboard reports variant conversion, but your product analytics may not identify the exact treatment each user saw. Overall activation or revenue movement during the test is not a randomized variant comparison unless your instrumentation can connect it correctly.

Use downstream data in one of three ways:

1. Direct variant attribution, if the platform and consented data genuinely support it.
2. Overall safety monitoring, while clearly labeling the limitation.
3. A follow-up rollout check after applying the treatment to the full page.

Do not claim that a treatment produced better subscribers when the data only shows more downloads.

Use a decision table

Write the decision rule before reading the result.

“Unknown” is a valid result. It is better than calling a small, uncertain lift a win.

What to do with an inconclusive test

An inconclusive result does not prove the treatment and original are identical.

It may mean:

• traffic was too low
• the treatment difference was too small
• several markets moved in different directions
• the expected improvement was unrealistic
• the test ended before enough evidence accumulated
• the new message was neither clearly better nor worse

Next options:

1. Keep the original and move to a higher-priority question.
2. Create a more meaningful treatment based on the same evidence.
3. Run the idea in one better-matched localization.
4. Wait for a higher-traffic period that is still representative.
5. Use qualitative research when the app cannot support a reliable live test.

Do not repeatedly rerun the same subtle treatment until one result looks positive. Repetition without a new reason increases the chance of selecting noise.

Apply a treatment carefully

Apple lets you apply one treatment from a test to the current original page or selected app versions.

Applying a treatment while the test is running stops the test. The action cannot be undone through the apply flow.

Apple applies screenshots and app previews from the treatment. A tested app icon does not become the default through this action; set it as the default in a future app version if you want to keep it.

Before applying:

• save the final result and test settings
• confirm the treatment assets for every device and localization
• check downstream guardrails and known limitations
• record the decision and owner
• plan the post-rollout observation window

Apple's current apply-treatment instructions describe the operational behavior.

Keep a test log

For every test, save:

• test name
• question and evidence
• hypothesis
• original and treatment assets
• exact asset changes
• traffic proportion
• localizations
• start and stop dates
• app releases and external events
• primary metric and guardrails
• final status, confidence, interval, and lift
• apply, reject, redesign, or inconclusive decision
• what the team learned

The rejected treatment is useful. It prevents another person from proposing the same idea six months later as if it has never been tested.

A good first test

Choose a problem you can explain in one sentence.

For many apps, that means the first screenshot:

• the audience is unclear
• the main result is generic
• a valued feature is buried
• the visible UI does not prove the caption
• the page attracts the wrong expectation

Create one treatment that fixes that problem. Keep the localization narrow. Give the treatment enough traffic to have a reasonable chance of concluding. Define a conversion decision and one downstream guardrail.

Then wait until the evidence is useful.

The point is not to run many tests. It is to make one decision with less guessing than before.

Return to the complete App Store ranking guide to decide whether the next constraint is creative, metadata, traffic quality, or the product itself.