The Naval Research Laboratory (NRL), in collaboration with the DOE's
Energy Sciences Network (ESnet), the
International Center for Advanced Internet Research (iCAIR) at Northwestern University, the
Center for Data Intensive Science (CDIS) at the University of Chicago, the
Open Cloud Consortium (OCC) and significant industry support, have conducted a 100 gigabits per second (100G) remote I/O demonstration at the
SC14 supercomputing conference in New Orleans, LA.
The remote I/O demonstration illustrates a pipelined distributed processing framework and software defined networking (SDN) between distant operating locations. The demonstration shows the capability to dynamically deploy a production quality 4K Ultra-High Definition Television (UHDTV) video workflow across a nationally distributed set of storage and computing resources that is relevant to emerging Department of Defense data processing challenges.
The remote I/O strategy allows data processing to start as soon as data begins to arrive at the compute location rather than waiting for bulk transfers to complete. This is achieved at near 100% efficiency over global distances and with low CPU loading using RDMA over Converged Ethernet (RoCE) and Open Fabrics Enterprise Distribution (OFED). The storage nodes generate multiple uncompressed, full quality 4K UHDTV video streams that fill a 100G link.
A single solid-state storage device at the Department of Energy's NERSC supercomputing facility in Oakland, CA, offers remote access to that data over the ESnet's 100G testbed via a connection to Chicago, IL, that brings in iCAIR's
StarLight International/National Communications Exchange Facility, then through New York City.
Video processing is accomplished in Oakland, Chicago, Washington, DC and New Orleans and shows the ability to process against remote data on the fly without first doing bulk data transfers. This demonstration showcases the remote process and management of ultra-capacity streams processing and networking over 100 Gbps WANs.
Future capabilities of the Pipelines processing framework application will automatically switch video sources and redistribute the processing. The Pipelines manager will dynamically redirect the data flow to line up with the processing options by using prearranged VLANs or by requesting the SDN controller to reconfigure the network such as the Software Defined Exchange capabilities at StarLight in Chicago.
Key participants in this demonstration also include
Orange Silicon Valley,
CenturyLink Government,
EchoStreams Innovative Solutions, and
EAI Design. Other significant capabilities were provided by
Mellanox Technologies,
Brocade,
NASA Goddard Space Flight Center (GSFC) and
Mid-Atlantic Crossroads.