Using Your Own Testing Tool

If your team already runs experiments in a tool such as Kameleoon, Amplitude or Optimizely, that tool can decide which variant each visitor sees while Pack serves the variant's content. Your tool owns assignment, targeting, traffic split and reporting. Pack owns the content: you build each variant in Pack as usual, and Pack renders it on the server with no flicker.

This needs @pack/hydrogen 5.0 or later. Coming from an earlier version, see Upgrading @pack/hydrogen.

How it works

  1. Create the test in Pack and edit its variants, as for any A/B test.
  2. Create the matching experiment in your tool. Its variations need to map to the Pack test's variant handles, such as control and variant-b.
  3. Start the test in Pack, then start the experiment in your tool.
  4. On every request, your storefront asks your tool which variations the visitor is in, and passes them to createPackClient as externalTestAssignments.
  5. Pack serves each variant's content on the queries it changes. Your storefront reads back what Pack served and reports exposures to your tool.

Passing assignments

Pass every assignment for the request, not only the ones the current page uses. Name each test by testHandle or testId, and its variant by testVariantHandle or testVariantId. When both are given, the id wins.

server.ts

// Ask your testing tool's server SDK which variations this visitor is in.
const assignments = await getVisitorAssignments(request);

const pack = createPackClient({
  cache,
  waitUntil,
  storeId: env.PACK_STOREFRONT_ID,
  token: env.PACK_SECRET_TOKEN,
  session: packSession,
  testSession: packTestSession,
  externalTestAssignments: assignments.map((assignment) => ({
    testHandle: assignment.experimentKey,
    testVariantHandle: assignment.variationKey,
  })),
});

getVisitorAssignments stands in for your tool's API. If your tool's experiment and variation keys differ from Pack's handles, map them here.

The option's type isn't exported. To build the list ahead of time, annotate it as:

type ExternalTestAssignments = Parameters<
  typeof createPackClient
>[0]['externalTestAssignments'];

Assignments must be known on the server. Pack renders variant content during the server render, so the assignment has to be available when createPackClient runs: from your tool's server-side SDK, or from a cookie your tool sets. A tool that assigns only in the browser after the page loads can't tell Pack in time, and the visitor sees the base content on that page.

What Pack stops doing

Passing externalTestAssignments, even as an empty list, turns off Pack's own assignment for that request:

  • Pack doesn't place the visitor into any test, and doesn't evaluate the test's targeting or traffic split. Your tool decides who is in which variant.
  • Pack doesn't resolve personalizations for the request.
  • The __pack_test cookie is neither read nor written.
  • Pack reports no exposures. packTestInfo stays empty, so PackTestRoute and PackTestProvider send nothing to testExposureCallback. You can leave them in place.

To let Pack assign a request after all, leave the option out (pass undefined) rather than an empty list.

Reading back what Pack served

pack.abTests lists the tests Pack serves for the request, each with its id, handle and testVariant (id and handle). It includes every assignment Pack accepted, whether or not the current page shows the content that test changes. Assignments Pack dropped (see Gotchas) are not in it.

If your root loader spreads ...context.pack.getPackContextData(), the same list reaches the browser as packAbTests, and useAbTests() returns it:

app/components/ExternalTestReporter.tsx

import { useEffect } from 'react';
import { useAbTests } from '@pack/hydrogen';

export function ExternalTestReporter() {
  const tests = useAbTests();

  useEffect(() => {
    for (const test of tests) {
      // Replace with your tool's exposure or activation call.
      trackExposure({
        experimentKey: test.handle,
        variationKey: test.testVariant.handle,
      });
    }
  }, [tests]);

  return null;
}

Report one exposure per entry, and check how your tool counts repeats: most count a visitor once per experiment however often the call is made.

If your tool already records an exposure when it assigns, compare its assignments with useAbTests(). An assignment missing from the list was not served, so the visitor saw the base content.

Consent is up to you here. Pack's own exposure reporting waits for the visitor's analytics consent, but this code runs outside it. Gate it the same way your tool's other tracking is gated.

Gotchas

The test must be running in Pack. Pack serves an assignment only if it matches a running test in the storefront's content environment. An assignment for a test that is draft, paused or ended, or for a handle Pack doesn't recognize, is dropped: the visitor sees the base content, and nothing in the response says so. Your tool, meanwhile, still counts that visitor in the variant, which skews the results. So:

  • Start the test in Pack before you start the experiment in your tool, and stop your tool's experiment before you pause or end the test in Pack.
  • Allow about 30 seconds after starting a test in Pack before it is served.
  • Check that each handle matches exactly, for both the test and the variant.
  • Check the dropped-assignment list described in Debugging before launch.

Tests that change the same content. Assignments to tests that change different content are served together, up to three per query. Of two tests that change the same content, only one is served: the one whose Pack test id sorts first, whatever order you list them in. The other is dropped. Make sure your tool doesn't put a visitor into two experiments that edit the same section or setting.

One assignment per test. A second assignment to the same test is dropped.

control serves the base content. The visitor is still listed in abTests with the control variant, so it can be reported.

Preview and QA links still win. In the Customizer, or when the URL has test parameters, Pack serves the previewed test and ignores externalTestAssignments.

Don't pass a test option to pack.query. A query's own test option overrides external assignments for that query. Remove any test: {} wrapper added to stop Pack assigning; externalTestAssignments already does that.

If Pack's test list can't be loaded. Pack can't check the assignments, so it sends the first complete one as given on every query, and reports it in abTests as given. An entry named only by handle then has an empty id and testVariant.id, so key your reporting on handles, or skip entries without ids.

Debugging

  • Log useAbTests() to see what Pack served. window.packDebug.getTestInfos() doesn't help here: it reads the __pack_test cookie, which isn't written while external assignments are in use.
  • With the environment variable DEBUG=pack:* set, the storefront's server log names each dropped assignment and the reason: not a running test, not a variant of the test, a second assignment to one test, or a conflict with a test already served.
  • To look at one variant's content without your tool, use the test URL parameters.

Testing against the API directly

To check variant content outside the storefront, query the Pack API with the test headers described in Selecting a test variant with headers. In a storefront, use externalTestAssignments rather than setting these headers yourself: when several tests serve on one query, @pack/hydrogen sends additional headers that the edge cache depends on.

Was this page helpful?