Lucid Imagination Combines Search, Analytics and Big Data to Tackle the Problem of Dark Data

SAN MATEO, Calif., May 9, 2012 /PRNewswire/ -- Organizations today have little to no idea how much lost opportunity is hidden in the vast amounts of data they've collected and stored.  They have entered the age of total data overload driven by the sheer amount of unstructured information, also called "dark" data, which is contained in their stored audio files, text messages, e-mail repositories, log files, transaction applications, and various other content stores.  And this dark data is continuing to grow, far outpacing the ability of the organization to track, manage and make sense of it.

Lucid Imagination, a developer of search, discovery and analytics software based on Apache Lucene and Apache Solr technology, today unveiled LucidWorks Big Data™. LucidWorks Big Data is the industry's first fully integrated development stack that combines the power of multiple open source projects including Hadoop, Mahout, R and Lucene/Solr to provide search, machine learning, recommendation engines and analytics for structured and unstructured content in one complete solution available in the cloud.

Tweet This: Lucid Imagination combines #search, analytics and #BigData in complete stack. Beta now open http://ow.ly/aMHef

With LucidWorks Big Data, Lucid Imagination equips technologists and business users with the ability to initially pilot Big Data projects utilizing technologies such as Apache Lucene/Solr, Mahout and Hadoop, in a cloud sandbox. Once satisfied, the project can remain in the cloud, be moved on premise or executed within a hybrid configuration.  This means they can avoid the staggering overhead costs and long lead times associated with infrastructure and application development lifecycles prior to placing their Big Data solution into production.

The product is now available in beta. To sign up for inclusion in the beta program, visit http://www.lucidimagination.com/products/lucidworks-search-platform/lucidworks-big-data.

Dark Data Problem Is Real

How big is the problem of dark data? The total amount of digital data in the world will reach 2.7 zettabytes in 2012, a 48 percent increase from 2011.* 90 percent of this data will be unstructured or "dark" data. Worldwide, 7.5 quintillion bytes of data, enough to fill over 100,000 Libraries of Congress get generated every day. Conversely, that deep volume of data can serve to help predict the weather, uncover consumer buying patterns or even ease traffic problems – if discovered and analyzed proactively.

"We see a strong opportunity for search to play a key role in the future of data management and analytics," said Matthew Aslett, research manager, data management and analytics, 451 Research. "Lucid's Big Data offering, and its combination of large-scale data storage in Hadoop with Lucene/Solr-based indexing and machine-learning capabilities, provides a platform for developing new applications to tackle emerging data management challenges."

LucidWorks Big Data

Data analytics has traditionally been the domain of business intelligence technologies. Most of these tools, however, have been designed to handle structured data such as SQL, and cannot easily tap into the broad range of data types that can be used in a Big Data application. With the announcement of LucidWorks Big Data, organizations will be able to utilize a single platform for their Big Data search, discovery and analytics needs. LucidWorks Big Data is the only complete platform that:

• Combines the real time, ad hoc data accessibility of LucidWorks (Lucene/Solr) with compute and storage capabilities of Hadoop
• Delivers commonly used analytic capabilities along with Mahout's proven, scalable machine learning algorithms for deeper insight into both content and users
• Tackles data, both big and small with ease, seamlessly scaling while minimizing the impact of provisioning Hadoop, LucidWorks and other components
• Supplies a single, coherent, secure and well documented REST API for both application integration and administration
• Offers fault tolerance with data safety baked in
• Provides choice and flexibility, via on premise, cloud hosted or hybrid deployment solutions
• Is tested, integrated and fully supported by the world's leading experts in open source search
• Includes powerful tools for configuration, deployment, content acquisition, security, and search experience that is packaged in a convenient, well-organized application

Lucid Imagination's Open Search Platform uncovers real-time insights from any enterprise data, whether structured in databases, unstructured in formats such as emails or social channels, or semi-structured from sources such as websites.  The company's rich portfolio of enterprise-grade solutions is based on the same proven open source Apache Lucene/Solr technology that powers many of the world's largest e-commerce sites. Lucid Imagination's on-premise and cloud platforms are quicker to deploy, cost less than competing products and are more easily tailored to specific needs than business intelligence solutions because they leverage innovation from the open source community.  

"We're allowing a broad set of enterprises to test and implement data discovery and analysis projects that have historically been the province of large multinationals with large data centers. Cloud computing and LucidWorks Big Data finally level the field," said Paul Doscher, CEO of Lucid Imagination. "Large companies, meanwhile, can use our Big Data stack to reduce the time and cost associated with evaluating and ultimately implementing big data search, discovery and analysis. It's their data – now they can actually benefit from it."

• Watch the future of open search unfold on the Lucid Imagination blog
• Follow Lucid Imagination on Twitter and Facebook.

*IDC Predicts 2012 Will Be the Year of Mobile and Cloud Platform Wars as IT Vendors Vie for Leadership While the Industry Redefines Itself, Dec.1, 2011

About Lucid Imagination

Lucid Imagination is the only company that delivers enterprise-grade search development platforms built on the power of Apache Lucene/Solr open source search. Out of the 35 Core Committers to the Apache Lucene/Solr project, 9 individuals work for Lucid Imagination, making the company the largest supporter of open source search in the industry. Customers include AT&T, Sears, Ford, Verizon, Cisco, Zappos, Raytheon, The Guardian, The Smithsonian Institution, Salesforce.com, The Motley Fool, Qualcomm, Taser, eHarmony and many other household names around the world. Lucid Imagination investors include Shasta Ventures, Granite Ventures, Walden International, and In-Q-Tel. Learn more about the company at www.lucidimagination.com.

SOURCE Lucid Imagination