SC Conference - Activity Details

Wolfgang Frings  (Juelich Supercomputing Centre)

Felix Wolf  (Juelich Supercomputing Centre)

Ventsislav Petkov  (Technical University Munich)

High Performance Filesystems and I/O

Wednesday,  02:00PM - 02:30PM

Room PB251

Parallel applications often store data in multiple task-local files, for example, to remember checkpoints, to circumvent memory limitations, or to record performance data. When operating at very large processor configurations, such applications often experience scalability limitations when the simultaneous creation of thousands of files causes metadata-server contention or simply when large file counts complicate file management or operations on those files even destabilize the file system. SIONlib is a parallel I/O library that addresses this problem by transparently mapping a large number of task-local files onto a small number of physical files via internal metadata handling and block alignment to ensure high performance. While requiring only minimal source code changes, SIONlib significantly reduces file creation overhead and simplifies file handling without penalizing read and write performance. We evaluate SIONlib's efficiency with up to 288 K tasks and report significant performance improvements in two application scenarios.

The full paper can be found in the ACM Digital Library and IEEE Computer Society