Evaluating Multivariate Visualizations

Our studies of multivariate visualization techniques have yielded an assortment of observations.

  • Parameterized patterns of data-driven spots (DDS) and Oriented Slivers helped users perform critical point (maximum) detection more accurately and faster than glyph representations of Brush Strokes and Stick Figures and more accurately than Color Blending.
  • Some techniques were sensitive to monitor settings (brightness and contrast) and room lighting conditions.
  • On a trend detection task, DDS and a baseline case of separate grayscale visualizations outperformed Brush Strokes, Dimensional Stacking, Oriented Slivers, and Color Blending with respect to accuracy, but not with respect to response time.
  • A follow-up study found that the technique of Attribute Blocks improved greatly over Dimensional Stacking, but that adjustments to the DDS technique expected to improve performance (via improved contrast) worsened user performance. Previous exposure to techniques lowered response time and subjective workload, but not error on the trend localization task. With this task, the distance between the target and the distraction features was demonstrated to influence the error. Subjects were most accurate when a feature of similar value was not present; when such a distraction was present, the error decreased as the distraction was closer to the target (which we believe lowered the cognitive effort to compare the two regions).
  • On a task of maximum overlap, users were demonstrated to attune to texture density as a cue to identify the target (overlap region) amongst four to six variables presented with DDS, Oriented Slivers, and Attribute Blocks. The baseline visualization technique of juxtaposed presentation of grayscale images representing a single variable led to poor performance.
  • A follow-up study refined this observation that it was relative texture density that was a reliable predictor of user performance (demonstrated by a statistically significant correlation).

For more information, contact 5581web_info@nrl.navy.mil.

Multivariate visualization techniques used in a recent study: from top left in row-major order: Attribute Blocks, Temporal Attribute Blocks, Data-driven Spots (DDS), Oriented DDS, and Oriented Slivers. These images show a target just below and to the right of the image center.
Multivariate visualization techniques used in a recent study: from top left in row-major order: Attribute Blocks, Temporal Attribute Blocks, Data-driven Spots (DDS), Oriented DDS, and Oriented Slivers. These images show a target just below and to the right of the image center.

Publications

2013 Livingston, M. A., J. Decker, and Z. Ai, "Evaluating Multivariate Visualizations on Time-Varying Data", SPIE Visualization and Data Analysis, vol. 8654, Burlingame, California, 02/2013.  (404.88 KB)
2013 Livingston, M. A., A. Murugesan, D. Brock, W. K. Frost, and D. Perzanowski, "Visualization of Decision Processes Using a Cognitive Architecture", SPIE Visualization and Data Analysis, vol. 8654, Burlingame, California, 02/2013.  (313.43 KB)
2013 Decker, J., and M. A. Livingston, "Web Tools for Rapid Experimental Visualization Prototyping", SPIE Visualization and Data Analysis, vol. 8654, Burlingame, California, 02/2013.  (666.4 KB)
2012 Livingston, M. A., and J. Decker, "Evaluation of Multivariate Visualization on a Multivariate Task", IEEE Transactions on Visualization and Computer Graphics, vol. 18, pp. 2114-2121, 12/2012.  (391.29 KB)
2012 Decker, J., and M. A. Livingston, "Building a Large Tiled-Display Cluster: A Case Study", IEEE VisWeek Workshop on Large Data Analysis and Visualization, Seattle, Washington, 10/2012.  (123.5 KB)
2012 Livingston, M. A., and J. Decker, "Evaluation of Multi-Variate Visualizations: A Case Study of Refinements and User Experience", SPIE Visualization and Data Analysis, Burlingame, California, 01/2012.  (288.48 KB)
2011 Livingston, M. A., and J. Decker, "Evaluation of Trend Localization with Multi-Variate Visualization", IEEE Transactions on Visualization and Computer Graphics, vol. 17, issue 12, pp. 2053-2062, 11-12/2011.  (1.28 MB)
2011 Livingston, M. A., C. R. Garrett, and Z. Ai, "Image Processing for Human Understanding in Low Visibility", ASNE Human Systems Integration Symposium, Tysons Corner, Virginia, 10/2011.  (931.18 KB)
2011 Decker, J., M. A. Livingston, S. Russell, and P. Hyden, "Use Cases for Visualizing Uncertain Computer Networks", IEEE VisWeek Workshop on Working with Uncertainty, Providence, Rhode Island, 10/2011.  (282.41 KB)
2011 Livingston, M. A., J. Decker, and Z. Ai, "An Evaluation of Methods for Encoding Multiple, 2D Spatial Data", SPIE Visualization and Data Analysis, Burlingame, California, 01/2011.  (1000.5 KB)
2011 Livingston, M. A., A. M. Palepu, J. Decker, and M. Dermer, "A Randomized Framework for Discovery of Heterogeneous Mixtures", SPIE Visualization and Data Analysis, San Francisco, California, 01/2011.  (843.7 KB)
2010 Decker, J., and M. A. Livingston, "An Interactive, Visual Composite Tuner for Multi-layer Spatial Data Sets", IEEE Visualization Poster Session, Salt Lake City, Utah, 10/2010.  (1012.67 KB)
2009 Decker, J., A. Godwin, M. A. Livingston, and D. Royle, "A Scalable Architecture for Visual Data Exploration", IEEE Visual Analytics Science and Technology Poster Session, Atlantic City, New Jersey, 10/2009.  (409.49 KB)
2007 Julier, S. J., and R. Mittu, "Distributed Data Fusion and Maritime Domain Awareness For Harbor Protection", Aspects of Network and Information Security, NATO Advanced Studies Institute on Network Security and Intrusion Detection, Nork, Yerevan, Armenia, IOS Press, 10/2006, 2007.  (220.55 KB)
2007 Julier, S. J., and J. J. LaViola, "On Kalman Filtering With Nonlinear Equality Constraints", IEEE Transactions on Signal Processing, vol. 55, issue 6, pp. 2774-2784, 06/2007.  (873.77 KB)
2007 Schmidt, G. S., J. Goffeney, J. Dalton, and R. Willis, "Generating Imagery for Forecasting Terror Threats", SPIE Newsroom: Electronic Imaging and Signal Processing, 01/2007.  (334.13 KB)
2007 Schmidt, G. S., J. Goffeney, and R. Willis, "Impact of Uncertainty on Terror Forecasting", NRL Review, 2007.  (207.04 KB)
2006 Goffeney, J., G. S. Schmidt, J. Dalton, J. D'Archangelo, and R. Willis, "Forecast Visualizations for Terrorist Events", IEEE Visualization Conference, 10-11/2006.  (405.81 KB)
2006 Julier, S. J., "An Empirical Study into the Use of Chernoff Information for Robust, Distributed Fusion of Gaussian Mixture Models", The 9th International Conference on Information Fusion, Florence, Italy, 07/2006.  (214.56 KB)
2006 Borchardt, O., R. Calhoun, S. J. Julier, and J. Uhlmann, "Generalized Information Representation and Compression Using Covariance Union", The 9th International Conference on Information Fusion, Florence, Italy, 07/2006.  (161.64 KB)
2006 Julier, S. J., J. Uhlmann, J. Walters, R. Mittu, and K. Palaniappan, "The Challenge of Scalable and Distributed Fusion of Disparate Sources of Information", Multisensor, Multisource Information Fusion: Architectures, Algorithms, and Applications Conference, SPIE Defense and Security Symposium, Orlando, Florida, 04/2006.  (268 KB)
2005 Julier, S. J., and J. Uhlmann, "Fusion of Time Delayed Measurements with Uncertain Time Delays", American Control Conference, Portland, Oregon, 06/2005.  (109.3 KB)
2004 Schmidt, G. S., S-L. Chen, A. N. Bryden, M. A. Livingston, L. J. Rosenblum, and B. R. Osborn, "Multidimensional Visual Representations for Underwater Environmental Uncertainty", IEEE Computer Graphics & Applications, vol. 24, issue 5, pp. 56-65, 09/2004.  (6.78 MB)
2000 Swan II, J. E., M. Lanzagorta, D. B. Maxwell, E. Kuo, J. Uhlmann, W. Anderson, H-J. Shyu, and W. Smith, "A Computational Steering System for Studying Microwave Interactions with Space-Borne Bodies", IEEE Visualization 2000, Salt Lake City, Utah, IEEE Computer Society Press, pp. 441-444, 10/2000.  (571.63 KB)
1999 Chen, B., J. E. Swan II, E. Kuo, and A. Kaufman, "A Hybrid LOD-Sprite Technique for Accelerated Terrain Rendering", IEEE Visualization 1999, San Francisco, California, IEEE Computer Society Press, pp. 291-298, 10/1999.  (366.82 KB)
1998 Lanzagorta, M., M. V. Kral, J. E. Swan II, G. Spanos, R. Rosenberg, and E. Kuo, "Three-Dimensional Visualization of Microstructures", IEEE Visualization, Research Triangle Park, North Carolina, pp. 487-490, 10/1998.  (184.18 KB)
1998 Mueller, K., T. Moller, J. E. Swan II, R. Crawfis, N. Shareef, and R. Yagel, "Splatting Errors and Antialiasing", IEEE Transactions on Visualization and Computer Graphics, 2, vol. 4: IEEE Computer Society Press, pp. 178-191, 04-06/1998.  (1.05 MB)
1998 Kamgar-Parsi, B., L. J. Rosenblum, and E. O. Belcher, "Underwater Imaging With a Moving Acoustic Lens", IEEE Transactions on Image Processing, vol. 7, issue 1, pp. 91-99, 01/1998.  (211.9 KB)
1997 Swan II, J. E., K. Mueller, T. Moller, N. Shareef, R. Crawfis, and R. Yagel, "Anti-Aliasing Technique for Splatting", IEEE Visualization 1997, Phoenix, Arizona, IEEE Computer Society Press, 10/1997.  (243.17 KB)
1994 Rosenblum, L. J., "Research Issues in Scientific Visualization", IEEE Computer Graphics & Applications, vol. 14, issue 2, pp. 61-63, 03/1994.  (354.45 KB)
1994 Krueger, W., K. H. Hoehne, L. J. Rosenblum, and W. Schroeder, "Research Issues in Volume Visualization", IEEE Computer Graphics & Applications, vol. 14, issue 2, pp. 64-67, 03/1994.  (609.49 KB)