In a recent InformationWeek article, John Edwards wrote, “Back in late 2017, Gartner analyst Nick Heudecker estimated that the failure rate of big data projects was 85%. Move the calendar forward two years and there’s no solid evidence proving that the failure rate has improved in any meaningful way.”

But the tide may be turning.  Many experts believe that artificial intelligence (AI) and machine learning (ML) can be used to drive better big data project planning which, in turn, will result in increased project success.

A key part of this success lies in applying AI and ML algorithms early to help companies determine whether the existing data architecture can support the big data program objectives. If it cannot, the technology can identify gaps that must first be addressed, thereby ensuring that the project only begins once the underlying architecture is prepared to support the initiative.

Another important distinction that makes AI an attractive technology is that it provides greater flexibility when organizations are grappling with numerous data variables. As Sorenson Ventures’ Ken Elefant told Edwards, “Instead of using a structured, rules-based approach, artificial intelligence as part of a big data project treats data much like a human would, by predicting how new data is affecting past trends.”

Of course, AI isn’t an out-of-the-box technology that can instantly solve companies’ big data woes. When designing AI algorithms for big data initiatives, it’s important that organizations examine both past project successes and failures so that the technology can better understand the qualifying factors that are intrinsically linked to success.

It may seem like an obvious best practice, but clearly defining the problem you wish to solve and the anticipated project goals are critical steps to realizing returns on big data investments. A granular understanding of these factors will help data scientists determine which AI application is best suited to the specific problem and types of data involved.

One of the experts with whom Edwards spoke likened a good data scientist to a skilled carpenter, in that both use the best combinations of tools to address the problem at hand. This analogy underscores that data science is as much an art as it is a technology discipline—something we’ve written about before on the APEX.

So, will AI be the game changer that finally helps companies improve the success rate of big data initiatives? While most companies have yet to deploy the technology at scale, the climate is positive for the technology to drive big data project gains.