Evaluation of Electronic Documents for Preparing Naval Meteorological and Oceanographic Briefings

J.A. Ballas
Information Technology Division

W.C. Kooiman and R.T. Miyamoto
University of Washington

Introduction: Naval meteorological and oceanographic (MetOc) personnel provide weather information and forecasts to aircraft, ships, and land-based components of the Navy. They are called on frequently to prepare briefs that describe the impact of weather on naval operations. The MetOc personnel studied in this research estimated that they each gave an average of 144 briefings annually. With the work demands they face, MetOc personnel must have systems that are quick and easy to use. For decades, the Navy has produced a number of paper products that provide historical meteorological, oceanographic, and geophysical information about specific geographical areas. The documents include charts on winds, tides, rain, currents, temperature, sediments, and biology. In line with its goal of moving to computerized work processes, the Navy has been developing computer- based alternatives to these paper documents. This study evaluates two of these alternatives, comparing them to traditional paper documents. These two alternatives are web hosting a version of the paper document (WEB_PAP), and a user-centric electronic document (UC-CD) hosted on a CDROM.¹ The design objectives of this product (Fig. 1), the Digital METOC Acoustic Reference Manual, are to: (1) link all the parts of the product into one cohesive whole; (2) provide branching for quick access to any information; and (3) support the preparation of weather briefings by including digital cut/copy operations to move information into a briefing document.

Evaluation Approach: Our study compared the use of the traditional document (PAPER) to the two alternatives (WEB_PAP, UC-CD) in completing several tasks.² This included finding information about a location in the Persian Gulf, interpreting the information, and assembling the information into a briefing (Task1), as well as answering four specific questions (Tasks 2-5). Twelve personnel from MetOc units participated. They were familiarized with each document prior to using it for the tasks. Data included the time to complete the tasks, as well as time spent in the following subtasks: Browse for information; Interpret information; Compose/edit the briefing (Task 1 only); Copy/paste material from the MetOc document into the briefing (Task 1 only).

FIGURE 1
A typical DMARS screen with functional descriptions of the main sections. The section borders are highlighted in this illustration.

Results: The key result is that browsing for information to prepare a briefing (Task 1) was significantly slower with the WEB_PAP than with either the PAPER or the UC-CD (Fig. 2). It also took longer to answer a question about tides using the WEB_PAP. The main advantage of the PAPER and UC-CD documents on these two tasks was probably the ability to quickly scan the information in the entire document and glance at images. This was done by flipping pages in the PAPER, and with the menu and navigation bars in the UC-CD, as illustrated in Fig. 1.

However, answering a question about fishing activity (Task 3), by using the UC-CD was slower. This result was probably due to the UC-CD menu design, which placed Fishing Activity as an element on the menu entitled "Human Activities." In the other documents, fishing activity was listed as a separate topic in the table of contents. Thus, details of menu design are critical to performance on electronic documents. The results on Task 5, determining the duration of daylight at a particular location for a particular date, showed both the benefits and limitations of the UC-CD design. Using the PAPER or WEB_PAP documents, the subjects had to interpret the graph illustrated in Fig. 3. With the UC-CD, the subjects were significantly faster because they simply had to select a geographic location on the map, select a date from a calendar, and read the daylight duration. However, there were more errors with the UC-CD because the subjects sometimes neglected to select a location on the map. This type of error can be eliminated by a redesign of the interface to prompt for a map selection. Finally, we found that more images were incorporated into the briefing when the UC-CD was used, and the subjects preferred this document to the other two.

FIGURE 2
Average browse time in seconds by document type for Task 1, preparing a briefing. Standard error bars illustrate the variance of the average.

Fig3 Image

FIGURE 3
Illustration of how daylight data were presented in the PAPER and WEB_PAP documents.

The design approach used to produce the UCCD clearly paid off. The images were designed to be readable on the computer, to have a consistent map across different types of data, and to be informative and appealing. In addition, the interface and its controls were designed to be consistent and intuitive. Overall, our research provides a compelling case that electronic documents should be developed with a usercentric design approach.

Acknowledgments: Our thanks to LCDR Bill Nisley II, CDR Chris Gunderson, and LCDR Roy Ledesma for coordinating the subjects and facilities.

[Sponsored by ONR]

References
¹ R.T. Miyamoto, P.M. Hardisty, and M.W. Stoermer, "Transfer of Navy's Environmental Guides to CD-ROM," APL-UW Technical Memorandum TM 17-91, Applied Physics Laboratory, University of Washington, Seattle, WA, September 2000.
² J.A. Ballas, W.C. Kooiman, R.T. Miyamoto, and W.S. McBride, "Design and Evaluation of Digital METOC Documents to Support Retrieval and Use of Information: User-Centric CDROM Compared to an Equivalent Paper Document and Its Republished Web Version," NRL Formal Report, NRL/FR/5513--00-9963, October 2000.