Machine learning at cure.fit

So you finally beat laziness and decide to book a cult/mind class, perhaps your very first one, but, you see that the class is full. What a bummer, right? Well, in reality, eventually the class is not full. How you ask? Because life happens… and a few members are unable to make it to class. Sometimes this cancellation is planned but quite often it’s those last minute hiccups.

Simply put, even though your app suggested that the class is full — in reality when the time comes a few members don’t show up and the class still has space to accommodate others.

This causes 2 problems

1. It impacts your experience because you didn’t get to workout at your preferred time
2. It affects our revenue — we are not able to optimise the space we have

While we introduced features such as waitlist and drop out only recently to manage this situation better, there’s another solution that runs behind the scenes that helps predict eventual class strength. This is something you can’t physically see as a feature but can most definitely experience the benefits of.

Let’s simplify some of the terms we may use in the rest of the article, so you can fully understand the problem & the solution. (you can skip the next part if you know how cult works)

Terminology — let’s make it simple

Every cult/mind Class has a certain Capacity i.e. maximum number of members it can accommodate without hampering any customer’s experience. A Class is conducted for a particular Workout format ( Dance Fitness, Yoga, HRX etc.). A member expresses intent to workout by Booking a class. A member can later, Cancel a booking if they no longer wish to attend the class. This Cancellation should happen before the Cancellation Window closes (upto 1 hour before the class begins). If a member doesn’t show up for a booked class, they get a No-Show. If a class, for whatever reason, is attended by more members than it’s capacity, it is called Overcrowded.

The Problem Statement

The problem was clear -

We could not successfully optimise utilisation of class capacity because we could never tell how many people would eventually miss the class.

The only way to solve this problem was to be able to accurately predict whether every booking will lead to attendance, cancellation or no-show.

As with any prediction system, 100% accuracy is not practically possible. Even the user booking a class wouldn’t know, at that time, of an upcoming unavoidable circumstance. Hence, a better goal would be to maximise the accuracy with some allowance for error in prediction.

Our tolerance for the errors in prediction is also not the same in both directions. Underestimating cancellations and no-shows will result in an under utilised class. i.e. more members don’t show up than what we expected. Which means more members don’t get classes, they are left unhappy and spots go unused.

On the other hand Overestimating cancellations will result in an overcrowded class. I.e We expected more people to miss the class but only a few did as a result the class becomes overcrowded.This too will leave members unhappy.

Of both evils, overcrowding is the much less preferred one. Hence we prefer to err on the side of underestimating cancellations/no shows than overestimating.

All of this leads us to the following problem statement

Maximize Class Utilisation, by predicting member cancellations/no-shows, while restricting the Error Percentage to a small number

To some of you this problem might sound familiar. Well, airlines face a very similar problem while trying to optimally use their volatile inventory. Volatile because once a flight takes-off every empty seat is just lost revenue. There is a good amount of existing research on a solution for this problem. However we couldn’t adapt these approaches for our problem because of some key differences.

• Airlines often penalize every cancellation, they balance this cost with the cost of arranging alternate flights or upgrading customers to a different class/tier in the unlikely event of an extra customer turning up. We, however, neither charge the customer for cancelling a class nor have multiple tiers (business/first class) of bookings.
• The frequency of bookings is much higher for cult/mind.fit. A person, might make one airline booking a month. While a regular member of cult/mind.fit makes at least 10 bookings a month. We have a lot more data to model user behavior.

These and a few more reasons led us to deciding to build a custom in-house solution.

Solution Approach

Predicting a single member’s cancellation/no-show behaviour is highly error prone. Statistics tells us that attempting to predict aggregate cancellation/no-show percentages over a group of members is more accurate.

We decided to go this route — Predict cancellation and no-show percentages for a group of members.

To do this, we needed to identify a set of signals or attributes or features ( as they are called in Machine Learning domain). We then needed to choose an algorithm that uses these features to predict cancellations/no-shows

Solution 1: Heuristic Solution

One of the first approaches we implemented was based on simple heuristic.

Features

Four features of the class were used

• Center
• Workout
• Start time
• Day of the week

Algorithm

For every class, we found classes in the past which had the exact same feature values i.e. classes from the same center, classes with the same workout, same start time and same day of the week. Then, we predict cancellation % using the following formula

Here recency factor determines the relative weight of a class in prediction. The closer in time this class is, to the class we are trying to predict , the more it’s weight.

This approach is illustrated in the following graphic.

Limitations

1. With schedules changing frequently, it wasn’t guaranteed that a similar class necessarily existed in the past weeks.
2. This approach was too rigid for defining a similar class. Only a class in the past which had the same center, workout, time and day of the week was considered. Although these classes indeed captured the dominant trend, disregarding all the other classes which shared some features but differed in others, led to a huge loss of information
3. The features we were using were also pretty limited. No features of the users or the booking context were taken into account.
4. The accuracy of this algorithm was also low.

Owing to these limitations we moved to a more scientific approach.

Solution 2: Machine Learning Solution

Unlike the heuristic algorithm, we added an extensive list of features to the prediction algorithm. The features used and the rationale for using some of them are explained below

Features

• Center, Workout, Start Time and Day of the week: These were the strongest signals. For example Friday evening classes typically have very high cancellation/no-show %. Also very popular workouts like Dance Fitness had much lesser cancellation/no-show %
• Member cancellation and no-show %: Member’s previous cancellation and no-show behavior was also another important signal for predicting these for the future class
• Member affinity for class: An interesting feature we discovered while working on recommending classes for members (a different project) is that, the more the affinity of the member for a class, the less likely that they will cancel/no-show
• Time at which the class was booked: The earlier a member booked a class, the more likely they were to cancel/no-show. This is intuitive since the closer you book to the class start time, the smaller the chance of any conflicts in personal schedules

Additionally we also considered other features like

• Is Trial Booking (True/False) — Check whether the booking made was for a trial class
• User Booking count — How many classes have they booked & attended before this one

Algorithm

To decide on an algorithm we considered the following aspects

1. The ML algorithm had to be a regression algorithm as cancellation and no-show % were real numbers
2. Cancellation and no-show % were very tightly coupled with each other — Missing the cancellation window causes a no-show. The prediction of cancellation and no-show % had to agree with each other. Hence we needed a multi-dimensional prediction algorithm over two independent models
3. We wanted to pick an algorithm whose accuracy increases as the scale of data increased

After building and evaluating the results of a few algorithms we finally converged on Neural Networks.

Few Implementation Details

• Categorical variables like Center and Workout were one-hot encoded
• It was not possible to include member specific attributes directly because the number of outstanding bookings for a class kept varying. Hence user behavior was captured using aggregates
• The architecture of the network including the number of neurons, layers and the structure were tuned using a validation set

Future Work

• As discussed above, we do not have the same level of tolerance for underestimating and overestimating cancellation and no-shows. Additionally, we have different levels of risk tolerance for different classes owing to the nuances of spaces and workouts. This could be handled by, either designing an asymmetric loss function or building guardrails around the prediction of this network.
• Centers and Workouts are one-hot encoded. This led the input to linearly grow as a function of the number of centers and workouts. A possible direction is to replace them with a lower dimensional vector which captures the key aspects of a centers/workouts.

Results

This project significantly increased class availability for members. This enables a couple of thousand extra members to workout everyday at a cult.fit/mind.fit center near them.

Written by Satwik Gokina

Edited by Esha Savla