Published Articles >> Table of Contents >> Abstract

Fifth IEEE/ACM International Workshop on Grid Computing (GRID'04)   pp. 243-250
High Performance Threaded Data Streaming for Large Scale Simulations
Viraj Bhat, Princeton University, NJ; Rutgers University, NJ
Scott Klasky, Princeton University, NJ
Scott Atchley, University of Tennessee, TN
Micah Beck, University of Tennessee, TN
Doug McCune, Princeton University, NJ
Manish Parashar, Rutgers University, NJ

Full Article Text:

DOI Bookmark: http://doi.ieeecomputersociety.org/10.1109/GRID.2004.36
Send link to a friend

Abstract
We have developed a threaded parallel data streaming approach using Logistical Networking (LN) to transfer multi-terabyte simulation data from computers at NERSC to our local analysis/visualization cluster, as the simulation executes, with negligible overhead. Data transfer experiments show that this concurrent data transfer approach is more favorable compared with writing to local disk and later transferring this data to be post-processed. Our algorithms are network aware, and can stream data at up to 97Mbs on a 100Mbs link from CA to NJ during a live simulation, using less than 5% CPU overhead at NERSC. This method is the first step in setting up a pipeline for simulation workflow and data management.
Back to Top Additional Information
Citation:  Viraj Bhat, Scott Klasky, Scott Atchley, Micah Beck, Doug McCune, Manish Parashar, "High Performance Threaded Data Streaming for Large Scale Simulations," grid, pp. 243-250,  Fifth IEEE/ACM International Workshop on Grid Computing (GRID'04),  2004