A/B Testing (Beta)

Overview

Welcome to Pack's journey toward enhancing your Hydrogen storefront and shop experience! As part of our commitment to providing a comprehensive digital solution, we are excited to introduce Pack A/B testing. This feature gives you a straightforward method to manage and create content variants for testing purposes across both storefronts and shops.

Pack’s A/B testing brings the same ease of content management from our customizer into your testing workflow for both storefronts and shops. Additionally, leverage our APIs to ensure flexibility in defining and executing your tests according to your specific needs.

Our team is continuously dedicated to improving our A/B testing product and your experience across all platforms. Keep an eye out for ongoing updates as we introduce new features and optimize your experience.

How to opt-in to Pack A/B Testing

Pack A/B testing is available to everyone as a public opt-in. To opt-in to the beta, follow these steps:

1. Go to app.packdigital.com
2. Navigate to your storefront or shop
3. Go to your Settings found in the left sidebar
4. Under Enable Experimental Features, toggle on the A/B Testing (Beta) option
5. You should now see A/B Tests in the sidebar. If not, refresh the admin panel.

Please follow the setup requirements below for some additional development steps needed to get A/B testing working in your storefront or shop.

A/B Testing Storefront and Shop Setup Requirements

1. A Hydrogen storefront or shop on Pack
   • Follow our guide here to add Pack to your existing Hydrogen storefront or shop
2. Update Pack packages to A/B testing beta
   • @pack/hydrogen": "0.2.0-ab-beta.1
3. Update your storefront or shop's code to support Pack A/B testing. Follow our guide here.

Managing A/B Tests

In the admin panel, you can view a list of all your A/B tests. These tests can be in a draft, running, paused, or ended state. To view and manage your tests:

1. Navigate to your admin panel
2. Click A/B Tests in the sidebar
3. Create, update, delete, and run tests

You can also delete tests from this view. However, a test needs to be in a paused or ended state before it can be deleted.

Creating a test

1. Navigate to your Pack admin panel
2. Go to A/B Tests
3. Click Create test in the top right
4. Enter details about your test:
   • Title: The human-readable name for your test
   • Test Handle: A unique key for your test, useful for analytics or feature flag testing
   • Goal: A high-level description of the overarching objective of your test, grounded in a hypothesis reflecting your desired outcome
5. Click Create test in the top right corner to save.

Managing test variants

Once a test is created, you can edit and manage its variants. By default, when a test is created, we provide two variants: a control variant and variant B. These variants have a 50/50 traffic split by default.

Editing a variant

From the test view, click the action menu on the right side of the variant and select Update variant details.

From here, you can update information about the variant:

1. Title: The name for the variant, which can be customized
2. Variant Handle: A unique key for the variant, used for analytics or feature flag testing and as the identifier when users are bucketed into a test
3. Description: An optional field to describe the intent or changes the variant is supposed to have
4. Traffic allocation: Control how traffic is distributed between variants in this test

Adding additional test variants

Pack’s A/B testing allows you to add additional variants to an A/B test, enabling you to run A/B/n tests. To add additional test variants:

1. Visit your test
2. In the Test Variants table, click Add Variant in the top right

Deleting test variants

If you want to delete a variant from a test:

1. Visit your test
2. In the Test Variants table
3. Click the action menu to the right of the variant you want to delete
4. Click Delete variant
5. Confirm your variant deletion

If you delete a variant from an active test, users who were bucketed into this variant will be assigned to another variant in the test. Again, it’s crucial to understand how this change might impact your test results before deleting.

Editing content for your test

Once you've defined your test objective and configured your test and its variants, you're ready to edit the content for this test. Start by heading to the Customizer to begin editing your tests.

Enter Test mode

Once you are in the Customizer, enable test mode by clicking the flask icon in the top left corner of the toolbar.

In test mode, you can manage content for each test and its variants. Editing content in test mode assigns changes to the currently active variant. When visitors come to your site and are assigned a variant, they'll see the content saved for that specific test and variant.

Editing a variant’s content

At the top of the toolbar in the customizer, in test mode, select the test and the variant you want to start editing content for.

Changes made to this content will be saved to the variant:

• Changes on a page level
  • Adding new sections
  • Removing sections
  • Section templates
• Changes on a global level
  • Site settings
  • Linked sections or changes to section templates will update wherever they are referenced

Starting, pausing, and ending tests

Starting a test

When you’re ready to start your test, you can do it in the admin panel or the customizer.

• To start a test from the admin panel:
  • Go to your admin panel
  • Select A/B Tests in the sidebar
  • Select the test you want to start
  • Click Run test in the top right
• To start a test from the customizer:
  • Go to your admin panel
  • Select Customizer in the sidebar
  • Enter test mode
  • Select the test you want to start in the toolbar
  • Click Run test in the top right

Pausing a test

If there is an issue or an adjustment needed and you need to pause the test, you can do so in the admin panel or customizer.

• To pause a test from the admin panel:
  • Go to your admin panel
  • Select A/B Tests in the sidebar
  • Select the test you want to pause
  • Click Pause test in the top right
• To pause a test from the customizer:
  • Go to your admin panel
  • Select Customizer in the sidebar
  • Enter test mode
  • Select the test you want to pause in the toolbar
  • Click Pause test in the top right

Ending a test

When you want to end a test, you can do so in the admin panel or customizer.

• To end a test from the admin panel:
  • Go to your admin panel
  • Select A/B Tests in the sidebar
  • Select the test you want to end
  • Click the action menu

in the top right

• Select End test
• To end a test from the customizer:
  • Go to your admin panel
  • Select Customizer in the sidebar
  • Enter test mode
  • Select the test you want to end in the toolbar
  • Click the dropdown arrow next to the Pause test button in the top right
  • Select End test

Promoting a winner

Once you've completed your test and identified a winning variant, you can promote this successful version to become the new content displayed to your users.

• To promote a test from the admin panel:
  • Go to your admin panel
  • Select A/B Tests in the sidebar
  • Select the test you want to promote
  • Click the action menu in the top right
  • Select End test
• To promote a test from the customizer:
  • Go to your admin panel
  • Select Customizer in the sidebar
  • Enter test mode
  • Select the test you want to promote in the toolbar
  • Click the dropdown arrow next to the Pause test button in the top right
  • Select End test & promote variant

Manually bucketing yourself into a test

If you want to bucket yourself into a test to validate that everything is working, you can add these query parameters to the URL.

Note: The only requirement is that you include both a test_* and variant_* parameter so we know which test and variant to display.

You can use different combinations of the parameters, such as:

• ?test_handle=ZZZ&test_variant_handle=YYY
• ?test_id=ZZZ&test_variant_id=YYY
• ?test_handle=ZZZ&test_variant_id=YYY
• ?test_id=ZZZ&test_variant_handle=YYY