SC Conference - Activity Details

Mark Silberstein  (Technion)

Artyom Sharov  (Technion)

Dan Geiger  (Technion)

Assaf Schuster  (Technion)

Dynamic Task Scheduling

Thursday,  04:00PM - 04:30PM

Room PB251

We present a holistic approach for efficient execution of bags-of-tasks (BOTs) on multiple grids, clusters, and volunteer computing grids virtualized as a single computing platform. The challenge is twofold: to assemble this compound environment and to employ it for execution of a mixture of throughput- and performance-oriented BOTs, with a dozen to millions of tasks each. Our mechanism allows per BOT specification of dynamic arbitrary scheduling and replication policies as a function of the system state, BOT execution state or BOT priority. We implement our mechanism in the GridBot system and demonstrate its capabilities in a production setup. GridBot has executed hundreds of BOTs with total of over 9 million jobs during three months alone; these have been invoked on 25,000 hosts, 15,000 from the Superlink@Technion community grid and the rest from the Technion campus grid, local clusters, the Open Science Grid, EGEE, and the UW Madison pool.

The full paper can be found in the ACM Digital Library and IEEE Computer Society