SC Conference - Activity Details
Adaptive and Scalable Metadata Management to Support A Trillion Files
Authors:| Jing Xing (Chinese Academy of Sciences) |
| Jin Xiong (Chinese Academy of Sciences) |
| Ninghui Sun (Chinese Academy of Sciences) |
| Jie Ma (Chinese Academy of Sciences) |
| Metadata Management and Storage Cache Allocation |
| Thursday, 02:00PM - 02:30PM |
| Room PB251 |
How to provide high access performance to a single file system or directory with billions or more files is big challenge for cluster file systems. However, limited by a single directory index organization, exist file systems will be prohibitory slow for billions of files. In this paper, we present a scalable and adaptive metadata management system that aims to maintain trillions of files efficiently by an adaptive two-level directory partitioning based on extendible hashing. Moreover, our system utilizes fine-grained parallel processing within a directory to improve performance of concurrent updates, a multi-level metadata cache management to improve memory utilization, and a dynamic load-balance mechanism based on consistent hashing to improve scalability. Our performance tests on 32 metadata servers show that our system can create more than 74,000 files per second and can fstat more than 270,000 files per second in a single directory with 100 million files.
The full paper can be found in the
ACM Digital Library and IEEE Computer Society
   |