Giving a Fast-Changing Data Ecosystem Room to Grow

I’ve just had my home wired to be smart so that I can manage it from a distance. As a customer, I like how the application helps me manage my life. But as a data engineer, I appreciate what went into building the complex system that lies behind its simple dashboard.

At Capital One we’ve gone a long way towards creating a single, internal ecosystem that lets us find the data we need to answer our customers’ questions accurately and instantly — and an interface that makes it easy for them to communicate with us.

But as a society we’re just at the beginning of what we can do with data.

Several factors are making it easier to use data to inform and shape our experiences and improve our environment. For a start, we’re gathering much more of it. Twenty years ago, the average car’s data gathering capabilities didn’t run much beyond how much gas it had used or how far it had traveled. Now a Tesla, for example, can use external cameras to gather data about the street signs and the position of traffic lights.

It’s also much cheaper to store data than it was just a few years ago and easier to automatically analyze it. In the future, advances in artificial intelligence and machine learning will deliver us data analysis tools we can’t even imagine today.

Given the pace of change, it’s important to design flexibility into a data ecosystem now, so that we’ll be able to drive service innovation for our customers in the future.

The best ecosystem puts the right tools in the right people’s hands and gets out of their way — whether it’s customers, advisors, or internal teams who are analyzing and using data to create better customer experiences. And it has the ability to evolve as new tools and resources become available.

There is no margin for error when it comes to data management. Customers count on us to give them accurate, real-time, personal information to make sound financial decisions.

Operationally that means being able to access the right data instantly, while keeping it safe and compliant on systems that are resilient, accurate, and secure. Companies looking at modernizing their data ecosystems need to make sure they:

• Have access to large quantities of clean, relevant data - Algorithms learn from data, so the more relevant data there is available the better they perform. One approach we have worked with is to pool data in a lake that is fed with streams of information delivered by microservices.
• Put in place strong data protection - Data security breaches can damage everything from relationships with customers and suppliers to product development and brand. Security is integral to your data ecosystem and companies are advised to build it in from the outset while taking advantage of the security features of the cloud.
• Develop a storage strategy and catalogue data carefully so that systems can automatically find it - Organizations need to be able to access the data that is useful for processing real-time experiences occurring on data streams in the present, while identifying and storing historical data depending on how likely they are to need it again.
• Synchronize data - There’s little point in developing real-time services based on the most up-to-date customer information if different versions of the same customer data exist at any one time across various devices and storage systems. Synchronization involves continuously replicating huge amounts of constantly changing, and sometimes private data without losing any. It’s a complex feat of engineering, but it’s essential to the maintenance of a healthy data ecosystem that delivers useful, accurate services to customers.
• Flexibility - A viable data ecosystem needs the agility to evolve quickly to new developments in the market. Organizations can draw on the speed and scalability of the public cloud to create an environment in which developers and operational teams can rapidly create and wind down services as required.

Data ecosystems are in evolution, but by taking a long-term, strategic approach to their design, companies can create an environment that fosters the development of customer experiences without compromising security and flexibility.

This blog post originally appeared on Forbes.