KPIs to successfully measure AI

I can imagine that any kind of foresight into future events that will impact travel can have a direct impact on the bottom line at Trip.com. And given the company’s visibility into global hotel chains and flight demand, it would seem tempting to create as many predictive models as possible? Given that, could you describe the current AI landscape that you’re responsible for in managing such a highly diverse set of entities at Travix?  

The deployment of any machine learning or AI model is extremely costly from an engineering perspective. You need the data engineer and machine learning engineer to support the deployment, and obviously a data scientist plays a crucial role in this process. Within deployment you work with multiple engineering teams, and it can take several teams a whole year to execute the front-end deployment, so there are high levels of investment involved. At trip.com we try to be as mindful as possible to obtain the best result from our investment, so we typically try to align with our commercial strategy so we can prioritize the most valuable items.

Within the data science team, we also have a phase where we use raw data to understand the feasibility of success, so together these exercises helps us to forecast the potential business value of a machine learning and data science solution. At trip.com we have a large portfolio of product domains such as the user experience, revenue management, and marketing. And we typically put these important areas together and decide at this juncture what domain needs some intervention from the data science team. We also see data science as a tool for the repetitive work our colleagues are doing so that they can focus on more value-added work for our customers and company.

Could you describe for us the process you go through in adding new AI projects to your catalogue? Do you inherit many projects from the parent company? Do you discover, explore, and vet projects centrally at Travix, or do you enable any of the 43 properties to operate independently in exploring new use cases and opportunities?  

For us, a new AI or machine learning product is a huge investment. A successful ML or AI product needs to involve the data engineering and data science team, and once the model is built, we also need to work with other front end and back-end teams to put that model  into production. It’s a huge investment that typically takes a lot of collaboration, so we need to be careful that these investments have a high ROI as sometimes the development cycle can takes a year, so it’s important to be able to prioritise.

I always engage with other business users early as they can provide historical performance data for certain workflows, and based on this data, coupled with assumptions or market research data, we can forecast the growth in revenue or other metrics. Based on this we can come up and analyze business value assuming there is an ML and AI solution and that we are able to forecast and prioritise the problem from the commercial side. From a technical standpoint, our data science team also spends a few weeks looking at the data, asking questions, understanding the data and, the problem; they then go through a feasibility study and see if there is a good use case for ML. These rigorous processes have helped me, and the organization helped to launch some successful products in the last few quarters.

On a more pragmatic level, I’m curious about the flow of data at Travix in supporting AI outcomes. Given your title, it seems that you are responsible for the creation of a unified data architecture in support of those outcomes. If so, could you describe for us that architecture in terms of how you collect data, where you store and process it, and how practitioners access that data? 

For a company our size our data assets are very diverse, and from an architectural level we do have a data lake that’s built on a google cloud platform. In terms of the types of data, we use event-based data, and we also receive external data from APIs, our financial systems, and different sources of connections. To support AI and ML it really depends on the subject matter domain, for example, sometimes the ideal data feed would be an API or event-based data, but that’s not always going to be the case, and that’s where you need a good data engineering team to figure out the best data environment for the ML team to be successful. There are times we find out we just don’t have a sustainable data source for certain machine learning solutions, and sometimes initiatives get delayed or put on hold simply because we need to wait for other teams to collect data better.

In creating this AI-supportive data architecture, have you explored concepts such as data fabrics as a means of turning data assets into APIs or perhaps even more ambitious endeavours such as the creation of data meshes, where domain experts retain ownership of data, working alongside IT? 

We still maintain a data lake architecture, although the ideal state would be a data mesh. However, Trip.com has not reached that point of maturity in terms of data ownership and there are a few things we are putting in place:

• Encouraging the front and back-end teams to migrate to an event driven data ingestion, meaning they can send us data through an API from an event, and they would own the design and data model of the event.
• Internal alignment with the data engineering team on the project. Some teams have done that successfully, so our idea is to start with a focus group and gradually advance this collaboration model, and for business teams to start adopting this strategy.

What are the top three challenges you face day in and day out in helping Travix to realize the full potential of its data through AI? For example, are you concerned over oncoming AI regulations, do you struggle to find skilled practitioners, or do you find current AI tooling too complex?  

A successful ML or AI product needs to be 100% aligned with the commercial strategy so that there is input and engagement with the business teams. Not all departments understand this collaboration model, and not all of them are data savvy, so it can be hard to distribute our knowledge evenly across different departments. You find that some departments have very advanced solutions, whereas others are missing out on opportunities because the employees or departments traditionally don’t have that kind of talent and expertise.

Secondly, the launching of a machine learning model on a large scale needs a lot of experimentation and testing. I think that’s a struggle for many data scientist practitioners as there is no easy way to expedite this, for example, sometimes we do have to wait for it to be in production and test for a few weeks to see the commercial performance. Then if performance is not in line with your expectations it does take more time to go back to development and start testing again, which is time consuming. Some of these tests are not easy to carry out to begin with so it can be a big challenge for us to move quickly enough to scale some of the data science solutions. Finally, there is a challenge around finding skilled practitioners. It’s not just about having a strong data science team, it’s also about communicating with other people that need to support data science initiatives, such as the MLOPs engineers, the front and back-end teams that need to push these models live, and that typically requires a different skill set. Most importantly there might be some cultural differences between data science teams and software development teams – that’s something we need to be mindful of as it can be challenging at times.

You can join Gan Zhang at The AI Summit London this June, where he will be a presenting a keynote on the Practitioners’ Stage on day one at 12-12:25pm. Make sure you secure your place on and be part of the action. Find out more.