We are very pleased to introduce a highly sought-after feature: A/B Testing, or more accurately Multivariate Testing. You can now configure one or more variations of a category merchandising campaign and run them alongside each other, to see which one performs the best.
For example, you could compare Klevu’s internal AI and Machine Learning versus your own manual merchandising, or see if boosting a particular product brand yields more conversions than another, or any other rules you wish to configure via our visual merchandising tool.
You can even configure the customer split percentage within the Klevu, specifying how many customers should see variation A vs B (or C, D or E), allowing you to try out smaller experiments for a few days on a small portion of your customers, or run larger tests with a head to head 50:50 split over the course of a few months.
In this article, we will introduce how you can do it and the most commonly asked questions (FAQs).
- Klevu A/B Testing is a part of Category Merchandising, so you must have an active Klevu subscription which includes the Category Merchandising solution.
- Currently, we only support merchants using our Klevu Template JS, as this is where we have implemented the customer-facing A/B test logic. If you are on JSv1 and wish to discuss your options, please contact support.
Step 1: Create an A/B Test
You can find A/B testing within Smart Category Merchandising > Visual Merchandising inside the Klevu Merchant Center (KMC). Alongside Default Rules and Scheduled Campaigns, you will see a new tab labeled A/B Test.
After clicking on this tab, the next page is where you would see any draft, completed or running tests you have already configured. Click on Create a New Test (or New A/B Test) to get started.
Objective / Goal / Test Name
The first thing to do is decide why you are running this test, by selecting one of the objectives available. If you’re trying this for the first time then you can select “I'm just playing around” and move on, but in general you should give your test an objective and a meaningful name, and bear those things in mind whilst configuring your test variants.
We will also use your chosen objective to highlight which variant is the strongest, although you will still be able to view all analytics data regardless of what you choose here.
Next select the category or categories you would like this test to apply to. Each A/B test can only contain up to 5 categories, and each category can only belong to one active A/B test at any given time. You can, however, schedule multiple A/B tests on the same category, as long as they do not run at the same time.
Next, define when your test will run. You can configure a test to start immediately by setting the start date as Today, or schedule to start automatically on some date in the future. You can also select any duration between 1 and 90 days.Use the quick links on the right to select the most common time ranges.
Click Proceed once you have completed your scheduling, then once you are happy with everything on this page, click on Save & Next to continue.
Step 2: Setup Variants
This is where you can configure up to five variants to run alongside each other, with their own merchandising rules, allowing you to analyze and compare which of your strategies is the strongest.
You cannot modify the base variant on this screen, as this variant represents your default rules which are already configured elsewhere in Klevu, via the Default Rules and Scheduled Campaigns tabs. Your base variant will always track whatever merchandising takes place in those tabs.
Next you can add additional variants, and even rename them and add a description to remember what that variation is focusing on. Each test can have up to five variants, including the base variant.
Step 3: Setup Merchandising Strategies for Each Variants
There are two types of strategy available for your variants:
- Manually Merchandise
- Self / Machine Learning
The first allows you to manually configure rules which will apply only to this particular variant, for example boosting high margin products, de-boosting certain brands, or pinning top products. These rules will take precedence over the Klevu AI self/machine learning, which will also apply.
Secondly you can choose a single variant to use only Klevu self/machine learning. In this case you will not be able to configure any manual rules at all, which can be useful to compare how Klevu AI performs and whether you should be spending time on merchandising or just letting Klevu AI run on its own.
If you choose manual merchandising, you will then select Click Here to Merchandise to continue.
Variant-level Visual Merchandising
Those of you who have used our Smart Category Merchandising tool will already be familiar with this screen. The only difference being you are configuring rules which will only apply to the variant you are editing. For more information on how to use our visual merchandising tool, please click here.
When you are happy with your changes, click on Save Variant in the top right to return to your test.
Step 4: Traffic Split between Variants
Once you have named your variants and given them a description, next you should decide which percentage of your customers should see which of your campaigns. All values must add up to 100. For a pure A/B test you can leave the traffic split as 50% base and 50% variant B. For a smaller experiment, you may allocate 90% to base and just 10% to variant B.
You cannot delete the base variant, but if you want to exclude it from your test you can allocate a Traffic Split of 0%, meaning that for the duration of your test no customers will see those rules.
Step 5: Schedule/Start A/B Test
In the bottom right you will notice two buttons:
- Save as Draft
- Start Test
Use Save as Draft, if you are not quite ready to start your test and come back to make changes later. What happens after you click the second button, Start Test, will depend on your schedule.
If your test start date is today, then the test will start immediately, and you will see the test in a running state. If your test was scheduled, the test will enter a scheduled state. You can still edit the test in this state, but once it enters the running state you can no longer edit it.
Interpreting the Results
Whilst a test is running, Klevu is constantly collecting the interactions customers are making in terms of clicks and conversions. The analytics data you see on the following screen is aggregated once per hour and presented so you can actively monitor your tests in close to real time. You can even stop a test if you don’t like what you see!
For any Running, Stopped or Completed tests, you can access the analytics of a test by clicking the analytics button:
Klevu will automatically highlight the winning variant depending on the objective you defined, but you can also select a different objective to see which variant would have won in that scenario, eg. switching between Click Through Rate or Conversion Rate.
The data available for each test is as follows:
- Category Views: The number of times the category(s) have been viewed.
- Product Clicks: The number of product clicks generated from those categories.
- CTR: The click through rate, calculated from the above two metrics.
- Product Conversions: The number of sales generated after interactions with those categories.
- Sales: The total of the sales generated from those product conversions.
- CR: The number of clicks that lead to purchases, the conversion rate.
You can also switch between variant and category view, to get an idea on how your variant rules performed on a category by category basis, as you may find that something works particularly well on one category but not on another, and you may wish to pick and choose which rules you take away from the experiment.
Applying the Results
Now that the test is complete and you have found your winning variant(s), the next step is to decide what to do with those results. Klevu will not apply anything automatically, so you can take your time and analyze things per category.
The action you will take next depends on the outcome of the test, detailed below. You can look at this from a variant level or category by category by using the available tabs, and decide what to do next:
If the base variant wins
Your existing scheduled campaign or default rules are already tuned better than your test. There is no action to be taken. You may like to tweak something and try another test.
If Klevu Self / Machine Learning Wins
This outcome suggests that you should trust Klevu AI, and remove any manual merchandising rules you have configured. If you tested against the base variant, you may want to go and edit your default or scheduled campaign to remove manual rules.
If Manual Merchandising Wins
In this scenario, the manual merchandising configured for this particular variant were the strongest, so you may want to export them so you can apply them to one or more categories. You will find a … on the right hand side where you can export the rules. Once you have them downloaded, you can follow the same process for importing Klevu visual merchandising rules, detailed here
Yes, and in fact this is a good way to compare your default merchandising rules against the Klevu AI, where you specify one variant as “Self/Machine Learning”, then continue to merchandise manually throughout the test. You can even add another variant to compare against a fixed set of rules too. Note that whilst it is possible to merchandise the default rules, you will see a warning whilst any tests are running, just so you are aware that your changes could impact any running tests. It is not possible to modify the merchandising rules of any A/B Test variants once the test is running.
When a test completes, the category(s) you have tested will revert back to their state as if there was no test running. You must manually decide whether to apply the rules of one of your variations to that category after a completed test. Therefore, at the end of an A/B test, one of the following will happen:
- Another A/B test may begin, if one was scheduled for the same category.
- A scheduled campaign may begin, if one was schedule.
- Otherwise, the default rules will apply for that category.
Once a test is running it cannot be edited. We recommend you stop the test and create a new one, omitting or modifying the poorly performing variant, or reducing the customer allocation percentage to run a smaller test for that variant.
Yes, you can delete a stopped or completed test, but please be aware that you will lose the view of analytics data associated to that particular test. Klevu will still keep the analytics data it needs, but you wont be able to see the A/B test results.