A Framework for Assessing Enterprise ML Maturity Levels

Companies across every industry are including machine learning and AI as a core component of both short- and long-term strategies for their business. If they’re not, their competitors likely are. One of the most common challenges that I’ve seen is the struggle for organizations to construct teams that match their intent, desired level of investment, and alignment to milestones against their strategic goals. Even organizations that are leveraging machine learning today don’t always have a plan for scaling up and maturing.

As with most multi-faceted technology challenges, there is not a silver bullet solution. But there are ways to frame the ML journey for any company, no matter its stage in the process. After reading this, I hope that executives and tech leaders will be able to apply that framework to their organization. And the next time the board asks “Where are we with AI and ML?,” there will be no need to lay on the buzzwords or skirt the question; you’ll have a thoughtful status on the highest impact areas.

One important aspect of forming an organization around machine learning is to level set roles or titles as best you can. There are several terms used in this space and some certainly have overlap in skillset and responsibilities. Again, I don’t proclaim that the following is irrefutable, but this is typically what I would use as a baseline.

Data Engineer - Responsible for ETL and ELT workloads, typically at scale. Data engineers are strong computer scientists and understand the optimization of moving, transforming, and storing data. Great cloud knowledge with some experience in automated DevOps rounds out the job. Python, Go, or Scala would be common languages used and expertise in distributed computational libraries like Spark or Dask would be expected.

Data Scientist - Focusing on the “science,” hypothesis testing is a major differentiator here. A data scientist will usually be using open Python libraries (Pandas, NumPy, Scikit-Learn) in exploratory notebooks, looking to unlock insights in data or improve performance of existing models.

Data Analyst - Primarily concerned with what is happening in the business, looking at historics and generating reports or dashboards. These functions will always remain key to decision making at any company. SQL and Tableau (or other visualization tools) are common and many are modernizing to Snowflake as a cloud warehousing solution.

Machine Learning Engineer - Has a skillset blending data science & engineering. Understands the full ML lifecycle and can take a model from proof of concept, all the way through to production. The underlying mathematics of the model might be less important than would be for a data scientist, but that’s not universal. One distinction that we use at Capital One is that an MLE would report to a technology organization and a data scientist reports into a line of business directly.

The following levels can serve as a framework for evaluating progress and what benchmarks can be expected along the way.  Additionally, it can help determine what can be expected in the next phase and how the organization structure and culture will need to change to be successful as it evolves.

Level 0 - Nascent

Every organization starts their journey somewhere. The “nascent” level is the starting block and where a commitment toward AI and ML is established. The company most likely already has a data analytics practice and has made progress by better leveraging their data. One of the most important things to get right early in the transition is data architecture. A staggering amount of companies just start dumping raw data extracts from their warehouse (or worse, their transactional database) into buckets with no rhyme or reason. A data lake wants structure, metadata and intentional design. Get those Parquet files organized and get off on the right foot! If data science experiments cannot be reliably repeated, there’s very little point. Most of that is on the data strategy.

A challenge here will be making hires. The landscape is extremely competitive and shelling out a top tier salary for someone to establish and lead a practice might not be easy to do. Some try to hire a moderately experienced first data scientist just because price tags can be daunting, but this usually doesn’t work out. It’s a little bit like handing the keys to the Porsche to your 16 year old. Safe and responsible decisions come with experience. There’s really no replacement. I might be a little biased here because of my background, but leveraging a consulting firm can often be the right path. If they are a strategic partner, they could even help you interview and build up your practice as a piece of the engagement.

Level 1 - Early Successes

This phase is probably the most exciting. Your data is available and you are able to start looking for leverage. Often companies find low hanging fruit that can yield significant value for the business. Team makeup becomes more of a focus. The secret here? Data engineers are your best friends. This should be your number one focus out of the gate. The ratio between data engineers and scientists is discussed often. I’ve seen that ratio as small as 1:1, but would recommend 5:1 upwards of 10:1!. Applied machine learning at the end of the day is a data problem and very seldom an algorithms problem. You will want to keep your data scientists hungry and then well fed. Opening up data is the way to do that.

In this phase, a few high profile models probably make their way to production. Early results will be reported to the executives and leaders of the company and the case for further investment will be an easy sell. Stack up these wins because things will soon get more challenging.

Level 2 - Maturing

Alright! We are modeling. We are making predictions and decisions. Everyone is hyped. Then things will start to get unruly. It’s a natural point in the progression and efforts in ML practices, maturation will be required. The focus here will be governance. How are we thinking about security? Model monitoring and versioning? Data lineage through various transformations? Have we been thinking about explainability or bias?

Probably not. At least not the level that we should have been. Some of you might be saying, “You should have launched the program with these practices in place!” I would partially challenge that. In the early days, I think agility is preferred to rigorous process. However, it doesn’t hurt to think about these things earlier on, especially if you are a regulated entity or are using protected data classifications. In some mature tech organizations, you would probably be able to leverage existing best practices. But it’s also completely normal to learn lessons along the way. You don’t grow up without scrapes and bruises from the playground.

The team here should be augmented with more operationally focused roles. You’ll want to prioritize more and more infrastructure and DevOps. Machine learning is also beginning to penetrate architecture functions as it becomes woven into the fabric of the technical strategy.

Level 3 - Transformed

Machine learning has become a staple for the business. Members of your workforce that are not even in the job family are actively thinking about it and can speak to several core applications where it has paid serious dividends. At this point, you have a flourishing ecosystem. Gone are the days of one-off projects. There is now a well defined life cycle by which every new model will follow. This ensures that our models are compliant by default. All boxes have been checked and both the data and the model itself are monitored in production for anomalies that signal that something is happening outside of expected behavior.

At this point, a company can expect to be leading the charge in their industry, pushing the bleeding edge even further. Auto ML and models building models is probably starting to be explored. At a minimum, automatic model refit triggers are probably in place. I would expect an organization that has reached this state to have a dedicated research function, paying attention to the literature being published in the academic community. Deep learning, transfer learning and reinforcement learning have all made their way into the fold. We shan’t content ourselves with gradient boosting without cutting edge challenger models in consideration! We have reached the top of the mountain and the view is sublime.

I hope that this post has been informative and helpful. Or I hope you liked one of the metaphors. At the end of the day, as a leader, it will be up to you to come up with the right approach. Your strategy should be consistent with the rest of your organizational structure and culture. Do you operate with horizontal or centralized organizations? Or would machine learning practitioners be embedded directly with teams focusing on specific intent? What is your geolocation strategy? Will you be investing in large markets? If you are pre-ML, how can you be positioning your data today to apply it tomorrow?

Finally, when considering the cost of the investment that you will be making, ask yourself one last question: what do we risk if we don’t!