Standard Template Library for Extra Large Data Sets (STXXL)

Hits: 0
Research Area: Algorithm Engineering for the Basic Toolbox
Status: In progress  
 
Project page
Description:

Standard Template Library for Extra Large Data Sets (STXXL)

The core of STXXL is an implementation of the C++ standard template library STL for external memory (out-of-core) computations, i.e., STXXL implements containers and algorithms that can process huge volumes of data that only fit on disks. While the compatibility to the STL supports ease of use and compatibility with existing applications, another design priority is high performance.

 

The following are some key features of STXXL:
· Transparent support of parallel disks. The library provides implementations of basic parallel disk algorithms. STXXL is the only external memory algorithm library supporting parallel disks.
· The library is able to handle problems of very large size (up to dozens of terabytes).
· Improved utilization of computer resources. STXXL supports explicitly overlapping between I/O and computation. STXXL implementations of external memory algorithms and data structures benefit from overlapping of I/O and computation.
· Small constant factors in I/O volume. A unique library feature "pipelining'' can save more than half the number of I/Os performed by many applications.
· Shorter development times due to well known STL-compatible interfaces for external memory algorithms and data structures. STL algorithms can be directly applied to STXXL containers; moreover the I/O complexity of the algorithms remains optimal in most of the cases.

[ Back ]