VWO : A/B test for Klevu Recommendations

In this guide, we will walk you through the steps to set up an A/B test for Klevu Recommendations using VWO. 

For the awareness of readers of this article, VWO is a platform for A/B testing, conversion optimization, and user behavior analysis. It offers tools like heatmaps, session recordings, and personalization to improve website performance.

Here are a few prerequisites before we proceed:

1. You need a VWO account. If you do not have one, please let us know by writing to our support. As part of our technology partnership deal with them, we will be happy to connect you with them to get a discounted price.
2. The Klevu template JS is delivering the Recommendations solutions on your webstore.

Implementation

This guide covers how to set up A/B tests for Klevu recommendations. For example, if you want to compare the effectiveness of Trending Products Vs. Newest Arrivals on home page, you'll need to create two variations: one displaying a Trending Products banner and the other showing Newest Arrivals. This implementation is client-side, with VWO managing the variations using their JavaScript. Finally, we will use Klevu Recommendations analytics to evaluate the performance of each variation.

We'll walk you through:

1. Getting Recommendation IDs: Learn how to identify the recommendation IDs needed for the test.
2. Creating Campaign & Variations: Step-by-step instructions to create and implement the A/B test variations & get familiar to JS code editor.
3. Retrieving VWO JS: Learn how to obtain and set up the VWO JavaScript at the platform level to enable VWO functionality.
4. Platform Integration: Detailed integration steps for popular e-commerce platforms such as BigCommerce, Shopify, and Magento.
5. Analyzing Performance: How to use Klevu Recommendations analytics to assess the results of your A/B tests.

By following this guide, you'll be able to optimize your Klevu recommendations for better engagement and sales across various pages on your e-commerce site.

Getting Recommendation Banner ID

Please follow our guide to create recommendation banners. In the guide, if you look at the step 6, Integration, here, you'll receive an HTML snippet that needs to be added to your page. This snippet includes the id field, which contains the unique ID for each recommendation banner (Screenshot below). Klevu uses these id to identify the banners. 

Ensure you retrieve the recommendation banner ID for each recommendation you want to A/B test. For instance, if you're testing Trending Products Vs. Newest Arrivals on your home page, you'll need to get the recommendation banner ID for both the respective banners.

Creating Campaign & Variations

The next step is to create variations in VWO. The detailed instructions are in the VWO guide but here, we highlight the steps which cover the example we are working with i.e. Trending Products Vs. Newest Arrivals

1. Login into the VWO using link https://app.vwo.com/#/login 
2. Once you are logged in, go to Testing > A/B > Create. This will be the page where you can create Campaign and setup the configuration. 
3. You need to setup the store URL where you want to perform the A/B test. It allows you the option to setup URL based on multiple matches e.g. URL matches and more as shown in below screenshot.
4. The next step is to set up the variations. Once you click on the Variations tab, you can add multiple variations by clicking on the Add Variation button. In our example, we added two variations: Trending Products and Newest Arrivals, as shown below. You can also manage the traffic split. For example, in this case, I allocated 50% to both Trending Products and Newest Arrivals.
5. After clicking the Edit button at the Variations level, the site will open in a new browser window where VWO allows you to configure different settings, code, etc. We will use the code editor functionality to add the Klevu JS to the page. This will be discussed in the next sections.
6. Select the tab Metrices as shown in the below screenshot, this does not include in our use-case as we would be tracking analytics at Klevu Merchant Center, it is a mandatory step from VWO to start a campaign
7. Click on Add Primary Metrices to create it for Click Event as shown below:
8. This step is optional. You can set up segmentation while creating the campaign with VWO. For more information, refer to this guide.

Retrieving VWO JS

In this section, we will obtain the VWO JavaScript needed to implement the A/B test on your platform. This script will enable VWO features. Here are the steps:

1. Go to https://app.vwo.com/ and on left tab, click on Websites and apps.
2. Click on Add new and choose Website and choose Next as shown below:
3. On the next section, you need to add domain name. Once this details are added. Proceed to the <b>code</b> tab, where you will be able to see the JS code, here is the example screenshot:

Platform Integration

After extracting the necessary details, the next step is to set up VWO on your chosen platform and begin the A/B test implementation. Each platform has its own URL structure and architecture, so we will cover the following platforms separately.

BigCommerce

1. Go to BigCommerce Admin > Themes > Edit Theme Files > Templates > Layout > base.html file.
2. Put the VWO JS code we extracted from above section Retrieving VWO JS into <head> section.
3. Put the below code in the <body> section, this will help us to determine what page we are on:

   <script type="text/javascript">
       {{#if page_type '===' 'category'}}
           var klevu_page_type = "category";
       {{/if}}
       {{#if page_type '===' 'product'}}
           var klevu_page_type = "product";
       {{/if}}
     {{#if page_type '===' 'cart'}}
           var klevu_page_type = "cart";
       {{/if}}
       {{#if page_type '===' 'default'}}
           var klevu_page_type = "home";
       {{/if}}
   </script>

   Using klevu_page_type variable, we would identify the page type.

4. Open the VWO code editor and select the specific variations e.g. Trending products and put the below code in JAVASCRIPTtab:

   /* CUSTOM CODE */
   if(klevu_page_type == "home"){
   var newDivElement = document.createElement("div");
   newDivElement.className = "klevu-recs";
   newDivElement.id = "k-abc123-def456-trending-products";
   var parentElement = document.querySelector(".page-main");
   if(typeof newDivElement.recsSource === "undefined" && typeof klevu.recs !== "undefined"){
     // Append the new div element to the parent element
     var klevuRectAbTest = klevu.recs.clone({
             recId: newDivElement.id,
             apiKey: klevu.getGlobalSetting( "recs.apiKey",klevu.getGlobalSetting( "global.apiKey" ) ),
             element: newDivElement
           });
     klevuRectAbTest.powerUp();
   }
   parentElement.appendChild(newDivElement);
   }

   Please note that you need to change the values for newDivElement.id = "k-abc123-def456-trending-products"; and var parentElement = document.querySelector(".page-main");. The newDivElement.id should be the Recommendation Banner ID for the trending products banner. The parentElement should be a class or ID-based selector indicating where you want to display the banner. You can create your own DIV in the theme where you want to render the recommendation banner. Additionally, the klevu_page_type variable condition can be adjusted based on the page where you want to set up the A/B test, such as category, product, or cart pages.
5. After setting up the JavaScript, click the Done button and modify the Variation as shown below to include the JavaScript code. Please note that you will need to set newDivElement.id for the other recommendation banner (e.g., k-abc123-def456-newest-arrivals in our case).
6. Once the above is completed, please Click on Save & Continue Button and it will save the changes you made for the variations.
   Now, you are ready to start the A/B test campaign you just created.

Magento

1. On the Magento Admin sidebar, go to Content > Design > Configuration. Find the store view that you want to proceed and click Edit in the Action column. Under Other Settings, expand Expansion selector the HTML Head section. 
2. Put the VWO JS code we extracted from above section Retrieving VWO JS into in Scripts and Style Sheets text Area.
3. Open the VWO code editor and select the specific variations e.g. Trending products and put the below code in JAVASCRIPTtab:

   /* CUSTOM CODE */ 
   if (document.querySelector('.cms-home')) {
   var newDivElement = document.createElement("div");
   newDivElement.className = "klevu-recs";
   newDivElement.id = "k-abc123-def456-trending-products";
   var parentElement = document.querySelector(".page-main");
   if(typeof newDivElement.recsSource === "undefined" && typeof klevu.recs !== "undefined"){
     // Append the new div element to the parent element
     var klevuRectAbTest = klevu.recs.clone({
             recId: newDivElement.id,
             apiKey: klevu.getGlobalSetting( "recs.apiKey",klevu.getGlobalSetting( "global.apiKey" ) ),
             element: newDivElement
           });
     klevuRectAbTest.powerUp();
   }
   parentElement.appendChild(newDivElement);
   }

   Please note that you need to change the values for newDivElement.id = "k-abc123-def456-trending-products"; and var parentElement = document.querySelector(".page-main");. The newDivElement.id should be the Recommendation Banner ID for the trending products banner. The parentElement should be a class or ID-based selector indicating where you want to display the banner. You can create your own DIV in the theme where you want to render the recommendation banner. Additionally, the initial if condition in the code will vary for category, product, and cart pages, as they use the classes catalog-category-view, catalog-product-view, and checkout-cart-index, respectively.
4. After setting up the JavaScript, click the Done button and modify the Variation as shown below to include the JavaScript code. Please note that you will need to set newDivElement.id for the other recommendation banner (e.g., k-abc123-def456-newest-arrivals in our case).
5. Once the above is completed, please Click on Save & Continue Button and it will save the changes you made for the variations.

   Now, you are ready to start the A/B test campaign you just created.

Shopify

1. Go to Shopify admin > Sales Channel > Online Store > Themes > Edit Code. 
2. Go to Layout > theme.liquid, Put the VWO JS code we extracted from above section Retrieving VWO JS into <head> section.
3. Since we are setting up an A/B test for the homepage, select the Campaign URL option as URL matches and enter the homepage URL.
4. Open the VWO code editor and select the specific variations e.g. Trending products and put the below code in JAVASCRIPTtab:

   /* CUSTOM CODE */
   var newDivElement = document.createElement("div");
   newDivElement.className = "klevu-recs";
   newDivElement.id = "k-abc123-def456-trending-products";
   var parentElement = document.querySelector(".page-main");
   if(typeof newDivElement.recsSource === "undefined" && typeof klevu.recs !== "undefined"){
     // Append the new div element to the parent element
     var klevuRectAbTest = klevu.recs.clone({
             recId: newDivElement.id,
             apiKey: klevu.getGlobalSetting( "recs.apiKey",klevu.getGlobalSetting( "global.apiKey" ) ),
             element: newDivElement
           });
     klevuRectAbTest.powerUp();
   }
   parentElement.appendChild(newDivElement);

   Please note that you need to change the values for newDivElement.id = "k-abc123-def456-trending-products"; and var parentElement = document.querySelector(".page-main");. The newDivElement.id should be the Recommendation Banner ID for the trending products banner. The parentElement should be a class or ID-based selector indicating where you want to display the banner. You can create your own DIV in the theme where you want to render the recommendation banner.

5. After setting up the JavaScript, click the Done button and modify the Variation as shown below to include the JavaScript code. Please note that you will need to set newDivElement.id for the other recommendation banner (e.g., k-abc123-def456-newest-arrivals in our case).
6. Once the above is completed, please Click on Save & Continue Button and it will save the changes you made for the variations.
   Now, you are ready to start the A/B test campaign you just created.

Note

The above example explains how to set up an A/B test on the homepage. However, the table below provides details on the URL settings (as shown in step 3) needed to set up A/B tests on product, category, or cart pages.

Analyzing Performance

Klevu features built-in analytics reporting that can be used to compare the performance of different recommendation banners. For instance, the banners you set up on your store (such as Trending Products vs. Newest Arrivals on the homepage) will be listed on the Klevu Recommendation Analytics page, allowing you to assess their performance. Further details on reporting can be found in this guide.