What's keeping you from managing your data better?

Building and maintaining data centers is costly in the face of growing volumes of data. There is also significant overhead and required staffing that can be outside of a company’s core competency. With exponential data growth, data management decision-makers will need to implement new ways to manage data in the cloud and get the most value from it. But while operating in the cloud reaps many benefits, there will be challenges along the way that require responsible data management. 

A commissioned study conducted by Forrester Consulting on behalf of Capital One shared that new data management models are necessary to operate in the cloud. The survey of 157 data management decision-makers in North America showed that while many organizations have yet to manage most of their data on the cloud, these organizations will need to adopt new ways to manage their data and overcome the common challenges they will face on this journey.

Forecasting and controlling costs is a top data ecosystem challenge. Most cloud costs come down to storage and compute, and in the on-prem data warehouse world, organizations know how much to budget for such capabilities. It is all capital expenditure (CapEx) – you budget for a fixed amount of server capacity, maintenance, and support staff. But in the cloud, everything becomes an operating expense (OpEx). Organizations pay cloud service providers like Snowflake and Amazon Web Services (AWS) for the data and compute services they use, and consumption tends to vary quite a bit between billing cycles.

While paying only for what you use is good business, the increase in data volume, variable usage, and near infinite elasticity can lead to unpredictable costs. The Forrester survey revealed that 82% of data management decision-makers view forecasting and controlling costs as a crucial data ecosystem challenge, making cloud cost optimization strategies more important than ever.

Another challenge organizations face when shifting to the cloud is the inability to see, understand, and use all their data. Nearly 80% of decision-makers surveyed by Forrester cited a lack of data cataloging–an organized inventory of data–as a top challenge, and close to 75% of respondents saw a lack of data observability as a problem.

Challenges navigating your data can lead to a host of other issues. If you can’t see who published the data and when, it is almost impossible to determine if the information is current, relevant, or redundant and it becomes difficult to use the data confidently. In short, you have a data quality problem, another challenge identified by 80% of respondents in the Forrester survey. If critical data resides on numerous on-premises and cloud-based servers, but isn’t integrated or observable, leaders cannot consider that information to make well-advised decisions, something 76% of those surveyed by Forrester cited as a challenge.

A third common challenge organizations face with cloud migration is data governance. The idea here is that to meet governance needs, which can differ by industry, organizations must know where their customer and financial data resides, ensure it is protected and monitor who is accessing it at any given time. 

Data governance can be a real headache for decision-makers because policies can be confusing, according to 82% of Forrester survey respondents. It can also be difficult to track and enforce adherence to data governance policies in cloud infrastructures because of their dispersed nature. Forrester cited that 80% of data management decision-makers identified they have difficulty governing data at scale, and lack entitlements and the ability to enforce role-based access to specific data.

Read more about cloud governance: Essential cloud governance disciplines and best practices

At Capital One, we’ve experienced many of these challenges on our own data journey. We exited our last on-premises data center in 2020 and became the first U.S. financial institution to go all-in on the public cloud. Doing this was no small undertaking. It involved shuttering eight data centers and moving our applications to the cloud. We built more than 80% of our applications to be cloud native so they would run more smoothly and efficiently. 

We weren’t immune to the challenges data management decision-makers face, including cost predictability, observability and data governance. To tackle these challenges head on, the first step we took was to federate our data management responsibility. With a federated model, the myriad data stakeholders across the organization can manage their own data while respecting individual and enterprise-wide data priorities and policies. We wanted a workflow-based model that automated as many back-end data management processes as possible. The goal was to make it easy to create a data set, manage access and build a data pipeline. 

We initially tried to address these issues with individual point solutions. We filled in the gaps with our own tools, but this led to tool sprawl, where our stack became so extensive that it hindered our efforts. For us, the solution was obvious. 

We built our own software that integrated some of these capabilities into a single data management solution, Slingshot. We built Slingshot to streamline our Snowflake provisioning processes while adhering to governance requirements, provide detailed visibility into cost drivers, and help optimize future spend. 

In today’s market, organizations need to make better use of data to maximize operational efficiency and deliver the kinds of experiences customers expect. But doing that, especially in the cloud, requires a solid and effective data management foundation. By using better software tools, companies can unleash the power of the cloud, achieve true business success and make the lives of data management professionals easier along the way. 

Download the “New data management models are essential to operate In the cloud” study now.