Guests to the Frankfurt roadshow of the 2023 Dataiku Everyday AI Conferences heard from an industry leader who went into detail about designing, testing, and leveraging AI models to optimize seasonal print ad campaigns. Dr. Stefan Mayer, senior data scientist at Marc O’Polo, shared details about the projects that he and his team executed using uplift modeling.
As part of an industry that can change so rapidly year-over-year and season-over-season, Mayer and his team understand that being able to leverage the benefits of data science is critical. With a business presence in over 60 countries and 2,000 employees with loads of industry knowledge, augmenting their work with uplift modeling has shown benefits in their current campaigns.
Their data science team is small but mighty. “We have five data analysts and data scientists,” he said. “We started with reporting analysis for the e-shop and the most data-heavy units. Now we’re actually supporting marketing, logistics, and have even started working with product and design.” Mayer mentioned that teams at Marc O’Polo by nature are very creative individuals, and it takes some effort to shift them into a more data-driven mindset, but he believes that this is critical to build this foundational knowledge across the organization.
About three years ago, we started the data science program at Marc O’Polo and established Dataiku as our data science platform.
— Dr. Stefan Mayer, Senior Data Scientist, Marc O’Polo
As part of an organization with high seasonality, Mayer understands how important making the best use of marketing spend is, especially with the different advertising channels available. “Twice a year we have quite a big sales campaign that we play in different countries,” he explained. They reach out to customers through various channels including social media, but an important form of media for them is print. “We have a lot of offline customers, and they still value print as a high-value, haptic touchpoint,” he said, “but as you all know, print comes with high cost, so we have to think carefully about who we should send mailing to.”
Mayer explained that before they could take advantage of modeling, they would use simple selection rules that weren’t the most cost-effective or efficient. “Just sending the mailing to our top customers, for example,” wasn’t the best option. Additionally, “if you think about whom you want to send mail to, you’ll soon end up with a treatment purchase matrix.”
“On the horizontal axis, we see the mention whether or not a purchase takes place without the mailing. Then, we have four different customer segments starting at the top right corner,” he explained. Dataiku’s customized segmentation solution can help your industry leverage AI and ML to find high-value customers, too.
Sure things: “These are more or less our most loyal customers,” Mayer explained. Marc O’Polo can almost rely on these customers to make a purchase if they receive a mailer or not. In other words, sending them a mailer doesn’t influence their purchase decisions.
Lost causes: At the other end of the spectrum, these customers are highly unlikely to purchase even if you send them a mailer.
Sleeping dogs: The third segment that he highlighted contains customers who are less of a presence at Marc O’Polo but need to be recognized. Per Mayer, these are individuals who are almost frightened when they receive a mailer. “For example,” he said, “in the telecommunications sector, where if you remind a customer of their contract, they want to cancel it instead.”
Persuadables: Mayer calls this the most interesting segment to study. These are customers who, “can actually be convinced to make a purchase if you send them the mailing, so our goal is to identify them and send mailings to only these customers to increase the overall conversion rate.”
With a clear customer segment in mind, Mayer and his team began to develop models to help them target the persuadables — work that began about two years ago. In the uplift modeling approach, they’ve trained two separate models. One is for customers who have received mailers in the past, and the other is for customers who have not. In other words, Mayer explained, “if we applied both models to our customers right now, we actually receive two different purchase probabilities that only distinguish between the case that the contact received some mailing in the first model, and provided the customer doesn’t receive mailing in the second purchase probability.”
By subtracting these two purchase probabilities, we end up with the predicted uplift, and that’s what we want to maximize.
— Dr. Stefan Mayer, Senior Data Scientist, Marc O’Polo
“We actually wanted to go one step further as we had the possibility to work with both the University of Cologne and a company called Valantic,” he explained. The team wanted to further develop the existing models they had by leveraging state-of-the-art industry research. They partnered with uplift modeling experts to develop custom algorithms and splitting criteria, which augmented their own work with that of others in the industry. After some custom feature engineering and model training, Mayer was proud to say that they were able to apply their uplift model to their latest campaign, which at the time of this writing was summer of 2023. He was even more proud to share the results, as well as what they had learned from this lengthy exercise.
“Most of the time was spent doing data preparation model training,” he explained. Taking data from their 2020 campaign, they performed an independent A/B test on those who received a mailer and those who did not. This formed the basis of the model training. “We only dealt with data strategies to come up with good data sets,” he said. “In this phase of the project, we didn’t really change anything, we just used standard settings and standard hyperparameters for the algorithms.”
Mayer spoke at length about the challenges that designing a strong data strategy presented. “We used different strategies to treat missing values,” he said. “To give you an example, if you have age as an attribute, we of course don’t have the age of all of our customers, then we have to think about what we do about these missing values. You can either replace it with the mean, or the maximum or minimum, or some other complicated strategy.”
The team also had to make decisions about different coding strategies. “We were dealing with outliers, so we had to look at correlations between different attributes, and the different strategies for feature selection.” In the end the team developed 18 different datasets, ranging from about 30 attributes to 300 attributes. To learn more, see how Dataiku can deliver custom applications for your organization.
After designing the datasets, the team had to decide how to use them. “We had about 17 different algorithms in place,” he said. “Some were the established ones for uplift modeling, but some had specific custom features. We had about 2,000 combinations of hyperparameters, and all of the models were able to identify an uplift.”
After his team had found a strong foundational data strategy, they combined them to get stronger datasets. From this point, it became an exercise in fine tuning.
We had the best dataset and the right hyperparameters, and were able to choose the best algorithm.
— Dr. Stefan Mayer, Senior Data Scientist, Marc O’Polo
The team was very excited to begin using their tailored model for their latest campaign, and decided to pit the new model against the established ones they had previously used. “We came up with an evaluation design where we split our customer base into five different randomized segments. The control group didn’t receive any mailing, and in the treatment group, every customer did. In the remaining three groups, two used the established model, and the final group used the new model.” All groups were independent from each other so Mayer’s team could compare results by predicted uplift without bias.
“We saw that in the control group, we had a conversion rate of 10.8%, meaning 10.8% of these customers made a purchase, and we saw 12% in the treatment group,” he revealed. “Sending out mailings, again the key difference between these groups, increased the conversion rate by 1.2%.” Our first model had a conversion rate of 10.9%, so this is an uplift of around 0.2% or 14% in relative terms compared to the upper lift bound. Per Mayer, the new model performed similarly with also a 10.9% conversion rate, but they could further increase the conversion rate by 0.4%.
Increasing the conversion rate by 0.4% is almost ⅓ of the upper limit bound, which is a really good value in uplift modeling. It doesn’t sound big in absolute terms, but in terms of customer base, it really makes a difference.
– Dr. Stefan Mayer, Senior Data Scientist, Marc O’Polo
Their team spent a long time conducting post-campaign analysis to investigate the difference between the models, comparing them to the initial segmentation that they were using.
The team sought to understand the accuracy and reliability of the original segmentation by having their model perform a similar distribution. He explained that not only were the vast majority classified correctly by both the old and new models his team developed, the new model “was really performing best overall,” he said.
Among the findings that Mayer called out was how model performance correlates to print ad spend. “We found that we could definitely increase the uplift conversion rate, but we also noticed that the new model implies a larger error in “sure things.” This isn’t a big problem, because it means that we send mailings to loyal customers who aren’t likely to complain. This is of course wasted money, though.”
They’ve also learned from the evaluation design that the maximum uplift bound is actually rather small, and they expected it to be greater. “It’s not something we can influence right now,” he began, “but it’s something we’ve learned from this use case. As a consequence the economic benefits really depend on the mailing costs. So if the mailing costs are beyond a certain threshold, and it makes sense to use the uplift model. If the mailing costs are too high, then there isn’t a benefit.”
For their next campaign (Winter 2023 as of this writing), they plan to keep the same evaluation design with a control and treatment group to more accurately measure the amount of “persuadables” they have. “We’ll use the new model to target the persuadables because it’s definitely performing best, but for some countries with low mailing costs, we won’t use the model and just send the mailing to everyone because it’s economically beneficial,” he said.
In closing, Mayer said that there’s still more to do. “We think about scaling to get the most out of the model,” he said, “and we’re also thinking about new use cases with more personalized content in the mailings. There are other avenues that are still ripe for experimentation and future projects, including other marketing channels, where Mayer believes that uplift modeling can be of great help.
Discover how LVMH centralized and customized deployment of AI algorithms for its luxury goods houses.
Read moreShowroomprivé leverages Dataiku to innovate across their business, including for machine learning-based targeting to build marketing campaigns that are 2.5x more effective.
Learn MoreDataiku scientists at Action worked with Capgemini and Dataiku to develop more accurate and transparent forecasting models, faster, ultimately achieving a 900% improvement on forecast runtime.
Learn MoreMandM uses Dataiku’s deployment infrastructure and MLOps capabilities to amplify the impact of their modest data science team, delivering impactful use cases across the company.
Learn MoreEach of Heraeus’s 20 operating companies has its own lead identification and qualification process. See how Heraeus uses Large Language Models (LLMs) in Dataiku to support these processes, ultimately saving time and increasing sales conversion.
Learn More
