The challenge

A young company growing fast

Quantiful is a young company that is growing fast. Having developed the QU product and behind the scenes processes on the go, the team also faces the challenge of onboarding enterprise customers with very large datasets that then need analysing. To support the businesses growth ambitions, the team needed to solve the challenge of scalability around dataset leaning, validation and feature engineering to enable a significantly larger number of SKUs (stock-keeping units) to be processed.

Rebecca Kemp, Chief Product Officer at Quantiful explains: “We need to make our solution more globally accessible and in order to do that we need to better leverage our processes and ways of working.”

Speeding up the onboarding process 

The second challenge was around standardising the AI/ML (artificial intelligence/machine learning) workflow across Quantiful’s different customers and use cases. This means striving for faster and more accurate implementation with the ability to use complex and differing models across a generic platform for each new instance of QU. With enterprise customers joining, the team wanted to ensure they could be onboarded quickly, efficiently and in a way that minimises the cost of integration and acquisition.

Rebecca explains: “We need to reduce our cost to serve, so that when we start ingesting customer data and serving it back to them on the QU Platform it’s done quickly, with maximum efficiency”.

In search of consistency  

Growing rapidly, the team at Quantiful had found themselves with a range of customers and a vast amount of data which demanded that common processes be used across all existing and future clients.

Gonzalo Naveira, Technical Lead for Quantiful explains: “We’d gotten to a point where we had a market-leading solution but the range of data and customers had become a challenge”.

The solution

AWS Glue and parallel processing 

After reviewing Quantiful’s infrastructure we recommended moving workflows to leverage AWS Glue to perform massive parallel processing to allow for a significantly larger amount of SKUs to be processed. AWS Glue is a serverless ETL (extraction, transformation, loading) tool that provides an environment to prepare and load large amounts of datasets from a variety of sources for analytics and data processing.

AWS Glue and data engineering  

Quantiful holds a significant amount of in-house structured and semi-structured data that is useful when performing machine learning. This data is used to augment the customer's bespoke dataset to enhance Quantiful's predictions. To ensure we can manage the variety of data, we leverage AWS Glue to maintain data catalogues which allows data scientists to pick and choose which data is applicable during the ETL process. AWS Glue data catalogues is a tool that allows you to leverage crawlers that will crawl your data automatically and identify the columns and data types, allowing much easier management of various data volumes.

Standardising the AI/ML workflow 

Our solution involved recommending Quantiful standardise their AI/ML data engineering workflow. We created standardised interfaces for each step of the pipeline, working alongside the Quantiful team. We also created a framework to leverage these interfaces for data wrangling which would help automate repetitive tasks, allowing Quantiful to focus on the bespoke requirements of each customer. Finally, we also recommended storing common functionality in a shareable repository.

The results

Increased processing power 

Our team helped Quantiful rearchitect components of the QU platform to make it a more efficient application. Having implemented AWS Glue, Quantiful can now process a significantly larger number of SKUs and benefit from near-linear scalability. This means that if Quantiful chooses to double their compute power, their processing times are halved, whilst the cost remains the same. Because AWS Glue is a cloud service, the team didn’t need to invest in on-prem infrastructure. The solution is cost-effective and fast. Final processed data can also be stored in many different places giving the team greater flexibility.

70% faster setup of new customers 

The result of standardising the AI/ML workflow is that we created a data wrangling tool in python. The tool is a series of scripts that automates the business-as-usual processes for Quantiful. Repetitive tasks like the intake of data, interactions with the AWS ecosystem and optimised data processing are all now automated. The Quantiful team estimate the new tooling will make it 70% faster to set up new customers. With the BAU taken care of, the Quantiful team can spend more time on the things that make each of their customers unique so they can deliver a seamless customer experience. Also, the team can now better collaborate on code in a secure and highly scalable ecosystem.

Augmenting the in-house team

Partnering with Lancom meant Quantiful were able to benefit from an influx of talent to seamlessly address their specific challenges. Whilst the in-house team knew what needed doing, they simply didn’t have the capacity. Lancom was able to buy into Quantiful’s purpose of ‘waste not, want not’ and augment the in-house team to deliver the results whilst the Quantiful team remained focused on their delivery schedule.

Leveraging the latest AWS technologies  

The solution designed by our team leverages the latest AWS technologies. The depth and breadth of tools available in AWS has enabled the build of a modern, cloud-based application that has been designed and developed as a server-less cloud platform using elastic technology to allow the application to automatically grow as Quantiful’s needs expand in the future.