Create a pricing experiment in the Smart Pricing app

You can create A/B pricing experiments to test changes to a product price over a trial period. These changes to product prices can come from AI-driven price tips, or can be manually set by you. You can then implement the lessons learned during the experiment to choose the best prices for your products on your online store.

Before creating a pricing experiment, review the Smart Pricing requirements and considerations.

When you create a pricing experiment, the catalog on your online store is temporarily split into 2 versions: a control version which contains no pricing changes, and an Experimental version which contains the adjusted prices. Your customers are split evenly between the catalogs at random, with each receiving 50% of the store traffic. At the end of the experiment, all store traffic is directed back to the control version. If your experimental catalog performed better, then you can choose to apply the new prices using a CSV file exported from the experiment's results page. Learn more about how price tips are generated.

Approximately 4 days after the start of a pricing experiment, the active experiment page displays metrics about the current results of the experiment, which refresh daily.

Pricing experiments have no set duration and can continue for as long as you require to develop statistically significant results. You can choose to manually end an experiment at any time.

A/B price testing

An A/B test (also known as a split test or bucket test) is a common business practice where you change a single element of some content and then compare the performance of the two versions (the control or unchanged version, and the altered version) to measure which one is more successful based on your chosen criteria.

In the case of an A/B test for pricing, such as those set up using the Smart Pricing app, you display the regular price of a product to 50% of your customers and a test price to the other 50% of your customers. After a set amount of time, you can confirm whether the test version of the price increased or decreased your conversion rate and sales. From this result, you can then make a decision about whether to make the price adjustment permanent.

To ensure that your experiment focusses on only the impact of a single change, always keep every other element consistent between the two versions of the product you choose to test. For example, if you want to A/B test the impact of changing the price of a product, then don't also change the product description for the test version at the same time.

Requirements for running A/B experiments in the Smart Pricing app

To create A/B pricing experiments in the Smart Pricing app, you must meet the following requirements:

• Your store must be on the Grow plan or higher, with sufficient visitor session volume for inclusion in early access.
• Your store must be based in the United States, selling to customers based in the United States.
• Your store must be using the new Shopify Markets:
  • Your store must have only 1 catalog for the United States market.
  • The United States market must be a top-level market. It can't be nested beneath any parent markets.
• You must have a minimum of 10 products, each with more than 25 sales per month.

Considerations about A/B pricing experiments

If your store is eligible for A/B pricing experiments, then review the following considerations:

• There can be only 1 active pricing experiment at a time.
• You can include up to 250 products in a pricing experiment.
• A maximum of 2 prices are used for A/B testing a product price: the original price and the test price. You can't create multiple test prices for the same product in an experiment.
• The Smart Pricing app doesn't adjust your product prices for you after an experiment concludes. Any pricing adjustments need to be directly implemented by you based on experiment results using a CSV file that you can download from the results page. Learn more about A/B pricing experiment results.
• There's no visual indication elsewhere in the Shopify admin that an experiment is running. You must communicate with any staff members to let them know not to change the affected products’ pricing themselves from the product details page in the admin.
• If you change the price of a product included in the experiment during the experiment duration, then it invalidates the experiment results, as product prices are tested and optimized as a set.
• Experiments aren't stopped automatically when store profit is significantly negatively impacted. If you determine your store profit is being significantly negatively impacted, then you must manually stop the experiment.
• The Smart Pricing app tests prices only on your Shopify online store. Other sales channels, such as Facebook and Instagram by Meta or Google & YouTube, don't receive test price data. This means that during an active experiment, a customer might receive the test price on your online store, but the original price when shopping for the same product on another sales channel.

Elements of a pricing experiment with price tips

When creating a pricing experiment, you can set the following elements to define the parameters of your experiment.

Create a pricing experiment with price tips in Smart Pricing

You can create a pricing experiment with Smart Pricing in the Shopify admin. Before creating a pricing experiment, review the Smart Pricing requirements and considerations.

There can be only 1 active pricing experiment at a time, and you can include up to 250 products in a pricing experiment. There is no set duration for a pricing experiment, and you can determine how long you want to run the experiment for based on the daily results of the experiment. Testing more products or running your pricing experiment for a longer time period leads to more accurate results.

Steps:

1. From your Shopify admin, go to Apps > Shopify Smart Pricing.

2. Click Review and experiment. If you'd previously dismissed the new price tips by clicking Review later, then click Review prices.

3. Optional: Set the price rounding rule for price tips:

   1. Click next to Price rounding.
   2. In the Edit price rounding rules dialog, select a whole number or set a decimal value to round price tips to.
   3. Click Update.

4. Next to Product preset, click .

5. In the Choose a preset dialog, review your preset options, and then click Select for the preset that you want to use.

6. Optional: In the In test product list, you can do any of the following actions:

   • To add or estimate any missing Cost per unit values, click Add cost to manually add individual values, or click Estimate margin to apply bulk values based on your store profit margin. Including this value improves the price tip and the accuracy of results. Learn more about the definition and formula for the Cost of Goods Sold on the Shopify Blog.
   • To manually override the generated price tip for a product, click the Test price field and set a different price to use for the duration of the experiment. To return to the original price tip value, click Revert.
   • To remove a product from the experiment, click the ⊖ icon. You can re-add the product by clicking ⊕.

7. Optional: In the Add more test products product list, you can do any of the following actions:

   • To add or estimate any missing Cost per unit values, click Add cost or Estimate margin.
   • To add a product to the experiment, click the ⊕ icon. If there's no price tip associated with the product, then you're prompted to manually set a test price.

8. Click Start experiment.

9. In the Before you start: Read carefully dialog, review the experiment summary, and then click Start experiment.

You can access the details of your active pricing experiment, including the products and prices being tested, at any time by returning to the Smart Pricing app in your Shopify admin.

Create a manual price experiment in Smart Pricing

If your store doesn't yet have access to AI-powered price tips, then you can set up a manual price experiment by selecting products and setting test prices yourself. Most A/B experiment features, including daily results, detailed reports, and price updating, are available for manually-created experiments. Only test price presets and price rounding aren't available, as they require access to AI-powered price tips.

There can be only 1 active pricing experiment at a time, and you can include up to 250 products in a pricing experiment. There is no set duration for a pricing experiment, and you can determine how long you want to run the experiment for based on the daily results of the experiment. Testing more products or running your pricing experiment for a longer time period leads to more accurate results.

Steps:

1. From your Shopify admin, go to Apps > Shopify Smart Pricing.

2. Click Setup manual experiment.

3. Click Add products to begin adding products to your experiment.

4. Select the products that you want to add to your experiment:

   1. Optional: Click to search and filter your product list.
   2. Click to add an individual product to your test.
   3. In the Include product in test dialog, set a test price for your product.
   4. Click Set price.

5. Optional: To add multiple products to your experiment and adjust prices in bulk:

   1. Click the checkboxes to select multiple products, and then click Add to experiment.
   2. Set the bulk price adjustment that you want to make to all selected product's prices. You can increase or decrease the price by a percentage or by a fixed amount.
   3. Click Apply and add to experiment.

6. To review your product list and the test prices you've set, click the In test tab:

   • To add or estimate any missing Cost per unit values, click Review cost to manually add individual values. Including this value improves the accuracy of results. Learn more about the definition and formula for the Cost of Goods Sold on the Shopify Blog.
   • To change your test price for a product, click the New price field and set a different price to use for the duration of the experiment.
   • To remove a product from the experiment, click ⊖ > Remove. You can re-add the product from the Add more test products tab.

7. Click Start experiment.

8. In the Before you start: Read carefully dialog, review the experiment summary, and then click Start experiment.

Remove products from an active experiment

You can't remove products from an active experiment, as it would invalidate the experiment results. You must stop the experiment.

Review the daily results of an active experiment

Four days after starting a pricing experiment, results data will generate and display in the active experiment page. This data refreshes daily based on the results of the ongoing experiment. Additionally, the active experiment's metrics display a Conclusive indicator when statistical significance has been reached at 95% confidence.

You can use these daily results to help you determine when to stop a pricing experiment.

Steps:

1. From your Shopify admin, go to Apps > Shopify Smart Pricing.

2. Click the active experiment that you want to access results for.

End an active experiment

You can manually end an active pricing experiment at any time, for any reason.

Experiments that stop before the 4-day mark won't generate results and can't be restarted.

For best results, you should conclude experiments when results have reached statistical significance, based on the daily results of the experiment displayed on the active experiment page.

Up to 2 days after a pricing experiment is concluded, the final results of the experiment generate and display on the experiment results page.

Steps:

1. From your Shopify admin, go to Apps > Shopify Smart Pricing.

2. Click the active experiment that you want to end.

3. End the experiment:

   • If the experiment has been running for fewer than 4 days, then click Stop experiment.
   • If the experiment has been running for 4 days or more, then click Conclude experiment.

4. In the dialog, click Stop experiment or Conclude experiment to confirm that you want to end the experiment.