• <More on Intel.com

Big Data Analytics Stories: Consumer Energy Management

Pecan Street uses data-driven analytics to empower consumers

Bert Haskell, Technology Director, Pecan Street Inc.


Pecan Street Inc. is a nonprofit consortium of universities, technology companies, and utility providers collaborating on testing, piloting, and commercializing smart grid technologies. “Smart grid” is a term used to define the utility industry efforts to modernize the electrical grid system with advanced metering systems, home energy management systems (HEMS), sensors, solar photovoltaic (PV) systems, smart appliances, electric vehicles, and Internet services that can empower consumers to be more energy efficient. Today, Pecan Street has gathered almost two years of energy consumption data from sensor systems in more than 200 households in the Mueller community of Austin, Texas.

One of the primary goals of Pecan Street is to drive new products, services, and economic opportunities around consumer energy management. Our research has the potential to provide people with the knowledge and tools to manage and reduce their energy consumption, as well as make their homes more comfortable to live in. In addition, the utility companies will be able to use this data to better manage the grid and invest in relevant infrastructure modernization.

At the heart of Pecan Street’s research is a device-to-cloud architecture that captures data from multiple sources and stores it for analysis and visualization. We collect electrical data from systems that measure the amount of power flowing through six to eight circuits every six seconds. We also collect gas and water data from utility meters using a wireless gateway. Plus, a radio collector on an old airport tower in the community streams data from Landis+Gyr* advanced smart meters to the University of Texas supercomputing facility and into our database via a high-reliability utility network.

We log consumers’ actions such as modifications to the environmental controls in their home or adjustments to the way their energy information is viewed. We also plan to collect data from advanced thermostats, home automation systems, home security systems, motion detectors, and new energy technologies such as solar panels and electric vehicle charge stations.

It’s critical for us to understand how consumers engage with the data we collect. So, as part of our research, some of the participants will have access to their data via a web-based portal or smart phone app.

During two years, we’ve captured a huge amount of data—to the tune of well over 80 gigabytes of information—and we expect that to increase to a terabyte of data over the course of the program. While a terabyte may not seem like a lot of data, as individual data points it’s actually a tremendous amount. Each data point represents a unique event, so the processing of this data is extremely complex. The big data challenge is to aggregate and backhaul this steady stream of unstructured data from multiple, disparate sources to the University of Texas where it is analyzed and visualized.

We’re on our third iteration of database architectures trying to find the best way to store and analyze this much data. After the first month, we realized that our MySQL* approach wasn’t able to cope with complex queries, such as, “What is the aggregated refrigerator usage over a 24-hour period, and is there an opportunity for a demand-response or peak-shaving program for the utility?” Since then we’ve migrated to other database architectures and are now evaluating an EMC Greenplum* big data solution. Greenplum offers an integrated big data analytics system with a massively parallel processing (MPP) architecture without the complexity and constraints of proprietary hardware and a Hadoop* distribution that will help us to process and analyze our data with a modular solution for structured and unstructured data.

In addition to searching for the right big data analytics approach, one of our greatest challenges has been the integrity of the data we collect. Inoperative channels in the data system or a disruption of the residential broadband connection gave us unreliable values. We resolved this by producing qualified data sets of known good data. We tag those as very high quality and direct researchers to use those.

The organizations that make up the Pecan Street consortium, which includes Intel, are using the smart grid project as a development platform for new product concepts—a test bed for people to innovate. With big data analytics, we can develop a better understanding of how people consume energy and how they want to manage it. Plus, we can provide utility companies with insights that will help them make the best investments in modernizing the energy grid.

Pecan Street Inc. working with Intel: View the Video >


Bert Haskell

 Bert Haskell


“At the heart of Pecan Street’s research is a device-to-cloud architecture that captures data from multiple sources and stores it for analysis and visualization.”