Report to the President: Every Federal Agency Needs a 'Big Data' Strategy

The Zettabyte Age is upon us: Networking and Information Technology working group notes that the exponential growth of data volume is now 'a fast-growing concern'

News provided by

Teradata

Mar 22, 2011, 11:30 ET

WASHINGTON, March 22, 2011 /PRNewswire/ -- A report from the President's Council of Advisors on Science and Technology (PCAST) covering many areas of technology has concluded that "every Federal agency needs to have a 'Big Data' strategy." The report, "Designing a Digital Future," adds that "the collection, management and analysis of data is a fast-growing concern of NIT research" at a time when "data volumes are growing exponentially" due to the "proliferation of sensors and new data sources." The report's contributor community included the PCAST Federal Networking and Information Technology Research and Development (NITRD) Program Review Working Group.

(Logo: http://photos.prnewswire.com/prnh/20090909/TERADATALOGO )

The 119-page report was characterized by five broad themes, with the information explosion and big data addressed first – while also noting the rising importance of data mining, described as "the transformation of data into knowledge, and of knowledge into action."

The NITRD working group, with fourteen members, includes Stephen Brobst, chief technology officer for Teradata Corporation, a company focused on data warehousing. Brobst is recognized for his expertise with global business and government clients on the identification and development of opportunities for the strategic use of technology. He is also a popular and frequent teacher-presenter at The Data Warehousing Institute events.

"We are entering the Zettabyte Age, and it is not hyperbole to say 'every Federal agency needs to have a big data strategy.' My colleagues and I in the Federal NITRD working group agree that this is an imperative, and leading commercial organizations are already refining such strategies," Brobst said.

"However, it is vital that government agencies, in particular, act now – because the increasing Federal commitment to U.S. citizens is openness and transparency – which rely on more powerful data management systems that can handle many simultaneous complex queries, extensive dashboarding, and capacities of speed and workload optimization that together must analyze masses of data from a broadening variety of data sources," Brobst said.

Brobst speaks from extensive experience as CTO of the world's largest company focused on data warehousing and analytics, with a significant number of its customers managing "big data" on the scale of many petabytes. Several of these engage in data-mining the global web, including Teradata customer eBay. Under Brobst's thought leadership, Teradata has also led innovation by being the first appliance vendor to introduce commercial database platforms with 100 percent solid state drive configurations for processing massive data volumes to deliver real time analysis and decision making.

Big Data is a term applied to data sets whose size is beyond the ability of commonly used software tools to capture, manage, and process the data within a tolerable elapsed time. Examples of big data sources include web logs, sensor networks, social media, detailed data captured from telecommunications networks, astronomical observations, biological systems, military surveillance, medical records, photographic archives and video archives.

Big Data requires exceptional technologies to efficiently process large quantities of information. Advances in Massively Parallel Processing (MPP) frameworks such as shared nothing relational databases, MapReduce programming frameworks, and cloud infrastructure are essential for harnessing the value from Big Data.

"Virtually all the objectives of government today require better, faster decisions backed by key facts and insights – which must often be discovered by analyzing mountains of data," said Richard Winter, president of Winter Corporation, a consultancy that specializes in very large data solutions. "The stakes are now too high for the outdated approaches that got us to this point. Most agencies will need to deploy a range of analytic database tools, including highly parallel data warehouse platforms, MapReduce and other technologies. It is crucial that agencies use the right tool for a given requirement. Only a well-defined, forward looking analytic data strategy will position them to do that."

Relevant news links