Search may be the next big thing in "big data." To that end, enterprise Apache Hadoop provider Hortonworks has formed a strategic alliance with open source search developer LucidWorks that may pave the way for more people to access and analyze data faster.
Before we get into the details of the news, let’s get a handle on the players:
Apache Hadoop is an open source framework for distributed storing and processing of large sets of data on commodity hardware. The Hortonworks Data Platform (HDP) is an open platform that integrates with existing IT systems so enterprises can build and deploy Hadoop-based applications.
Currently, LucidWorks claims to be the only company delivering an enterprise-grade search development platform built on top of the Apache Solr open source search. LucidWorks’ platform offers an interface that delivers advanced search capabilities to non-technical users.
Analyzing Big Data Faster
“Because Apache Solr doesn’t require complex script development to do searches or access and identify patterns in data, our partnership with LucidWorks will dramatically accelerate the time to value for companies that are ready to make a major investment in big data,” said Tim Hall, vice president of Product Management at Hortonworks. “The use of Solr will also democratize access to big data, which is critical to enabling companies to realize the full value of their information.”
Under the terms of the alliance, Hortonworks will include Solr in HDP, and Hortonworks’ customers will receive support and technology from both Hortonworks and LucidWorks. Combining Apache Solr and HDP promises to make big data accessible to more people within an organization. In other words, it’s not just developers, analysts and data scientists that can tap the data anymore.
“Our customers have been selecting these two technologies on their own and asking Avalon to integrate them,” said Tony Jewitt, vice president of Big Data Solutions at Avalon Consulting LLC. “But now, this combination of LucidWorks and Hortonworks provides our clients with a seamless platform for deploying search-based applications running on HDP. It’s the perfect solution for analyzing social media, chat sessions, e-mails, documents and many other sources of big data.”
The Democratization of Big Data
We caught up with Charles King, principal analyst at Pund-IT, to get his take on the alliance. He told us when we started seeing big data platforms coming to the fore a couple of years ago -- and companies like Pivotal and Cloudera and Horntonworks started leveraging Hadoop -- there was an explosion of the idea around data scientists. But there was also a secondary issue related to how companies could best leverage big data across the broader workforces.
“Not everybody can be a data scientist,” King said. “The notion was that eventually big data would require search capabilities that were analogous to a Web search engine like Google or Bing where a person could input a simple question and have that question directed against big data resources a company holds.”
That, he said, is the basic concept Hortonworks and LucidWorks is forwarding. Essentially, it’s the democratization of big data. Although big data assets can be formed in great depth and great value can be achieved through the work of a sophisticated data scientist, he said, getting those assets into the hands of the broader workforce delivers stronger value.
“We’ve seen this with more traditional database tools and platforms as well,” King said. “The more people that can access and leverage information the more value the company can derive from that investment. The more the merrier.”
Apache Solr 4.7 certified with Hortonworks Data Platform 2.0 is available now.