According to a recent McKinsey Technology report, “Yesterday’s data architecture can’t meet today’s need for speed, flexibility, and innovation.” Indeed, the demands on today’s data infrastructure, systems, and apps is greater than ever as companies have rapidly deployed new data-driven solutions. This includes predictive analytics solutions aimed at improving marketing and sales results, data mining solutions that help drive customer personalization, and automation solutions that assist with repetitive tasks to increase workforce efficiency.

While most executives report added value and improved business outcomes resulting from data analytics initiatives, the benefits have not come without a cost. According the McKinsey Technology report, “These technical additions—from data lakes to customer analytics platforms to stream processing—have increased the complexity of data architectures enormously, often significantly hampering an organization’s ongoing ability to deliver new capabilities, maintain existing infrastructures, and ensure the integrity of artificial intelligence (AI) models.”

If this sounds familiar, you’re not alone. The problem is that a growing number of emerging and more agile companies are launching newer data technologies, such as serverless data platforms, that enable them to develop and launch products to market more quickly than traditional players. One needs to look no further than Amazon and Google for leading examples, but the threat to established companies can come from any direction, including from start-ups and smaller companies.

A New Approach to Data Stacks

According to the McKinsey Technology report, “For companies to build a competitive edge—or even to maintain parity, they will need a new approach to defining, implementing, and integrating their data stacks, leveraging both cloud (beyond infrastructure as a service) and new concepts and components.” The report goes on to offer the “six shifts” that any company can make to their data architecture to enable more rapid delivery of new capabilities, while simplifying existing approaches. These include moving:

  1. From on-premise to cloud-based platforms: Today’s cloud-based data platforms have unleashed a “radically new data architecture approach,” according to the report. This shift is enabling business of all sizes to deploy and run data infrastructure, platforms, and applications at scale. Enabling technology includes serverless data platforms, such as Amazon S3 and Google BigQuery, and containerized data solutions.
  2. From batch to real-time data processing: With lower costs for real-time data messaging services, innovative companies, including those in transportation and manufacturing, are offering more personalized services and alerts powered by real-time streaming. This shift is enabled by technologies such as messaging and alerting platforms and streaming processing and analytics solutions.
  3. From pre-integrated commercial solutions to modular, best-of-breed platforms: According to McKinsey, companies are “moving toward a highly modular data architecture that uses best-of-breed and, frequently, open-source components that can be replaced with new technologies as needed without affecting other parts of the data architecture.” This new and more flexible approach is being enabled by data pipeline and API-based interfaces and analytics workbenches.
  4. From point-to-point to decoupled data access: Companies today can allow employees and data teams to access more up-to-date data via APIs, while also ensuring greater control and security. This improves collaboration and speeds the adoption of data-driven technologies, like AI. Technologies, including an API management platform and “a data platform to buffer transactions outside of core systems,” are enabling this shift.
  5. From an enterprise warehouse to domain-based architecture: According to the McKinsey report, “Many data-architecture leaders have pivoted from a central enterprise data lake toward ‘domain-driven’ designs that can be customized and fit for purpose to improve time to market of new data products and services.” This allows product owners to better manage their data sets, making it easier to consume for team members and other parts of the business that may need them. Data as a Service platforms, data virtualization, and data cataloging tools are helping companies with this shift.
  6. From rigid data models to flexible, extensible data schemas: Today’s most innovative companies are shifting from pre-defined and proprietary data models to “schema-light” approaches that drive more flexibility into data analytics approaches and initiatives. Benefits include agile data exploration, greater flexibility in data storage, and reduced complexity, according to the McKinsey report.Data vault and graph database technologies have helped bring about these changes.

To learn more, check out the complete McKinsey Technology report. If you’d like to explore this topic further, see this related APEX of Innovation post on why you need a sound data management strategy.