U.S. Department of Energy

Pacific Northwest National Laboratory

Modeling Continuous Human Information Processing

People operating in dynamic, streaming data environments must perform continuous, online decision making. Examples of continuous decisions include real-time alert monitoring, appropriately categorizing new data samples as they arrive in a display, or periodically adjusting sensor settings in response to changing environmental conditions.  The goal of this research effort is to develop human information processing modeling techniques that apply to a combination of measurable response behaviors. These behaviors include response choices, response times, eye or hand movement dynamics, vocal/speech patterns, and related psychophysiological measures (e.g., heart rate).  By leveraging the richness of multimodal behavior dynamics, we can make inferences about the mental activity demanded by continuous, online decision making.

The validation of models and metrics for continuous decision making will enable better assessment of user interfaces by developers and users. Users include, but are not limited to, marketing analysts monitoring news and social media sources, cyber defenders searching for threats in network traffic, operators of machine and sensor systems needing periodic adjustment and maintenance. With appropriate models and metrics in hand, we can improve our assessment of interactive visualizations, to ensure they support correct, useful, and timely decisions for streaming data.


Currently available information processing models are limited to discrete decision making instances. They do not capture well the mental mechanisms that support decision making tasks where novel information can arrive incomplete and/or at any time during the process. The latter are the situations of interest for AIM applications. Some of the current models are limited to only modeling response time or choice, and do not yet capture well the rich multimodal behaviors of interest in the current effort.


Our approach relies on the novel application of statistical and cognitive model measures, such as hazard functions and Gaussian processes, for functional analysis of multimodal behaviors during continuous decision making. We will use full dynamics of response, rather than the behaviors from only the end decision point in the task, to capture the ongoing mental activity. We particularly seek to characterize changes in the behavior dynamics that indicate shifts in cognitive mechanisms associated with changing task goals or demands. Metrics exist for response choices, eye and hand movement dynamics, response speeds, and verbal activity that capture well the component mental mechanisms that comprise continuous decision making. By leveraging a combination of these behaviors, we will advance our understanding of streaming analytics cognitive processes as an embodied combination of mental processes concurrently supporting the user’s broader goals.


This research affects all human-machine analytics systems, particularly those designed for dynamic, streaming environments. It influences the basic science of cognitive modeling by developing novel methods for characterizing multimodal behavioral data that can be applied in any perceptual or cognitive domain. We will also further our understanding of the cognitive mechanisms that support analytic sensemaking. In connecting these methods to streaming analytics, we will establish critical metrics for assessing streaming analytics applications. We will provide ways to assess human decision making from direct responses as well as motor and eye movement dynamics. This means we can flexibly assess the efficacy of novel interfaces and visual analytics techniques for any streaming analytics domain.

Second Workshop on Visualizing Eye Tracking Data (ETVIS 2016), Baltimore, Maryland. BEST PAPER AWARD. Balint, J. T., Arendt, D. L., & Blaha, L. M. (2016, October). Storyline visualizations of eye tracking of movie viewing.

| Pacific Northwest National Laboratory