AI-powered content recommendations with a headless CMS

Wait... what happened to the Smart Recommendations API?

You may remember (especially if you read all of my blog posts!) that Kontent.ai once had its own Smart Recommendations API right within the product, which we used when we built the previous version of this site. The Smart Recommendations API actually represented a tight integration with Recombee behind the scenes. Rather than maintain the integration layer between the two products and potentially restrict customers from getting the full power of Recombee, Kontent.ai has sunset their own recommendations API in favour of full first-party support for Recombee directly, and maintaining open source integration code. This is a win for all involved!

This is the story of how we implemented that very integration, between Kontent.ai and Recombee, to power the smart recommendations on this very website.

How does it work?

Every time we publish new or updated content of a certain type (such as blogs, case studies, team member profiles, and landing pages), Kontent.ai sends the data to Recombee. Recombee then consumes every drop of juicy structured content that it can get into its magical machine learning algorithms – plain text, rich text, taxonomies, metadata, linked items (including IDs of them), alt text of images, we send it all.

With all those pages and their rich content sitting in Recombee's AI model, we then also track page visits to all those relevant pages and send that over too. A cookie is created in the visitor's browser so we can track sets of pages visited by unique visitors, just like Google Analytics would (don't worry, this does not include any personally identifiable information!).

Recombee is then able to work its magic and recommend/suggest similar pages for us to display at the bottom of the page. This happens via a JavaScript call after the page has loaded, so it won't slow down the page, and doesn't require us to implement any server-side code.

When a visitor clicks on one of these recommended links (in other words, a recommendation was a good one!) that is also sent to Recombee, which tracks that and learns that it was a good recommendation, feeding that into the machine learning algorithm. (Even without that click event being tracked, clever Recombee is able to infer from sets of pages visited by the same users, that certain recommendations work well together, even if it doesn't have that extra info of exactly which recommendation was followed.)

While that 'users who viewed this page also viewed this other page' logic isn't the only thing driving the recommendations of course – that would hardly qualify as Artificial Intelligence! The recommendation could be based on any number of factors, from the content of the current article, to patterns in the pages you've been viewing, or people with similar behavioural patterns viewed. For example, if you seem to like articles by Emmanuel (or if it doesn't know much about you, even if other people who viewed this article tended to), it's likely to recommend more of them.

How is it implemented?

🤓 Tech content warning! Feel free to skip to the results below.

This is just a quick summary – it's not rocket science thanks to the extensive integration guide and open source code repository provided by the legends at Kontent.ai.

How Kontent.ai, Netlify Functions, Recombee, and the website are integrated

In a nutshell:

• We use webhooks for our selected content types to send published item data to serverless Netlify functions. 
• Our serverless functions process the webhook data, and massage it into the right format to interface directly with Recombee's APIs (this means we can also trigger it manually using a Custom Element which talks to the same middleware).
• On the front end of the website, we use JavaScript, and an SDK provided by Recombee. This makes it easy to interface directly with Recombee's API endpoint to get our recommendations, after the page has loaded.

What was the result?

It works! 🙌

We're using it to provide the "Read More" section for our blog posts (scroll down and see!), our case studies, team member profiles, and even for certain sections of our home page, where it can recommend interesting posts or case studies and keep content fresh and relevant for new and returning visitors.

Even without diving deep into how effective the individual recommendations are, even at its most simple, just not having the same 'recent' or 'popular' article recommended to someone over and over again is a huge improvement (as Recombee knows if a visitor has already read a blog post)!

The biggest benefit by far however, is one of efficiency, and saving time for our marketing team. We simply do not need to maintain the myriad relationships between our hundreds and hundreds of content items (although we can still influence it if we want to). Content editors frequently underestimate just how much maintenance is required to keep 'related articles' up to date, as it's not just a matter of adding a few related articles to each new post – you also need to go back and add that new post as a related article to any existing content that might benefit from it. In reality, if left to a manual process, this very rarely happens.

Considering these relationships are never updated manually, the default often ends up being chronological recommendations – either 'next' or 'latest'. That's generally not how a visitor will view your website content however. We want to surface the highest performing relevant content, not just the next cab off the rank.

Our early quantitative results are hard to quantify exactly, but Recombee is letting us know that around five percent of its recommendations are apparently being followed (clicked on). Considering we often provide four recommendation options at a time (so the highest possible click-through rate could be 25 percent in that case), that's not bad at all!

What's next?

At the moment, to be honest, one of the best parts of this is that it's 'set and forget', so we don't have to do anything next! In other words, the biggest benefit we're enjoying is that this does NOT require constant maintenance. 

But of course, we're considering a few enhancements, such as:

• We could set up 'series' or 'collections' of content (and teach Recombee about it), to encourage people to view more of a curated set of related content once they're looking at part of it.
• It's fairly straightforward to 'boost' certain known high-value or high-conversion content (sometimes known as 'lead magnets'), even if they're not getting as many page views as some other pages less likely to 'convert' in a marketing sense.
• Configuring complex 'scenarios' and 'business rules' to override certain situations with our preference on recommendations – where we're happy to take the wheel off the AI and drive it ourselves. For example, we might want to encourage visitors down some particular paths toward a defined goal once they've read certain key pages.
• Is recommending four posts the most effective way to keep people reading? Would two be better? Should we just give them one very strong 'read next' recommendation perhaps? A/B testing different numbers of recommendations would remove any speculation and simply provide us with that answer.

A real-world use case for AI, today

Artificial Intelligence doesn't have to be rocket science. You don't have to try to match Amazon or Google, you might just want to save yourself a little time managing your website content. Services such as Recombee are available today for you to harness its power in simple and effective ways.