U.S. Department of Energy

Pacific Northwest National Laboratory

Compressive Analysis

Today, most streaming data is collected and stored for future analysis using custom algorithms based on the data type, which does not support real-time analysis. The goal of compressive analysis is to identify events of interest in streaming data without the need for long term storage or custom algorithms.  


Our goal is to support rapid identification and classification of events of interest, speeding up the process of analyzing streaming data or pre-processing large volumes of static data. This analysis is performed in real or near real-time on streaming data by leveraging the metadata generated from highly optimized algorithms that were originally created for another purpose. For example, treating non-video data as video, we can use metadata generated by video compression algorithms to identify the “frame” or individual piece of data that has changed significantly from the previous “frame” or data point in the stream. The specific changes or patterns of change detected can be used to build models and combined with human feedback, further optimize data collection and either directly identify the specific event, or target the data of interest for further appropriate processing/analysis.


Our approach promises the benefit of low computational cost and rapid analysis of data streams, requiring minimal training, and little to no transformation of the data. This will speed analysis of large data sets and facilitate real-time or near real-time analysis of streaming data.  The ability to constantly monitor multiple streams of incoming data and only call human attention or use more computationally complex analysis, when needed, will be of value to multiple fields.

| Pacific Northwest National Laboratory