eXtremeDB Financial Edition FAQ
eXtremeDB Financial Edition is McObject’s database system for capital markets applications. It addresses the challenge to financial IT posed by growing volumes of market data that must be sorted and analyzed with the lowest possible latency, to support key application functions such as managing risk, identifying and acting on trading opportunities, and ensuring speed and reliability in financial exchanges.
eXtremeDB Financial Edition is a full-featured database management system (DBMS) with support for industry standards including SQL, ODBC and JDBC, for multi-process concurrency and for transactions that support the ACID (Atomic, Consistent, Isolated and Durable) properties. It is designed to be integrated within applications, with little or no maintenance by end-users. To support this developer orientation, eXtremeDB Financial Edition offers sophisticated developer tools such as multiple application programming interfaces (APIs), support for specialized database indexes, event notifications and more.
Specialized features include column-based handling of time series data (tick data), and a library of vector-based statistical functions that execute over sequences and can be pipelined to form an assembly line of processing on the market data. Used together, these features maximize the efficiency of the CPU cache in managing market data. Columnar handling ensures that the largest amount of time-critical data (sequence elements) is brought into the CPU with each fetch, and avoids flooding the CPU cache with irrelevant data. Pipelining of vector-based functions causes intermediate result sets used in processing to remain in CPU cache rather than being output as temporary tables. This eliminates the overhead of back-and-forth trips between the CPU cache and main memory.
To help handle large volumes of market data in trading applications, run-length encoding (RLE) compression can be applied to time series data (i.e. market data). In McObject’s tests, this feature reduced tick data to one-quarter of its pre-compression size, and increased speed in reading the database by 21%.
McObject launched the eXtremeDB product line more than 10 years ago. The initial product, a highly efficient in-memory database system with a C/C++ language interface, targeted embedded systems such as telecom and networking equipment, industrial control, aerospace, defense and consumer electronics. Since that introduction, eXtremeDB has gained features including optional persistent storage, high availability and clustering sub-systems, 64-bit support, new programming interfaces, and much more. eXtremeDB is used by hundreds of companies and deployed in millions of run-time applications and devices. These customers include many financial technology providers who, over the years, identified eXtremeDB as their ideal solution while researching high performance database systems. With feedback from these users – and from our own research identifying the significant challenges in handling market data – McObject developed eXtremeDB Financial Edition. It incorporates all capabilities and editions in the existing eXtremeDB product family, and adds new features such as the columnar data handling and vector-based statistical functions discussed above.
At its core, eXtremeDB Financial Edition is an in-memory database system (IMDS): records are stored in main memory, eliminating caching, disk and file I/O, and other significant storage-related latency found in “traditional” on-disk DBMSs. As a result, eXtremeDB operates many times faster than a conventional on-disk database management system, even if the disk-based DBMS is used with a RAM disk (this is discussed in detail in the white paper “Main Memory vs. RAM-disk Databases: A Linux-based Benchmark”, available for download here: White Papers).
However, eXtremeDB Financial Edition also enables the developer to selectively add persistent storage, so that a given database design can incorporate anywhere from zero to 100% persistent storage. When used with persistent storage, eXtremeDB Financial Edition provides caching to keep frequently-accessed records in memory. It also includes features such as cache prioritization to minimize retrieval overhead for objects used in time-sensitive tasks, as well as database striping to accelerate performance, and mirroring to support continuous backup, when working with two or more disks.
Yes. It also implements many SQL extensions to support eXtremeDB-specific features, including the ability to pipeline vector-based statistical functions from SQL. Compared to other SQL databases, eXtremeDB Financial Edition with SQL lends a performance advantage through its highly efficient and predictable rule-based SQL query optimizer. In contrast, most SQL DBMSs rely on a cost-based optimizer that must collect sample data, generate statistics and analyze thousands — sometimes hundreds of thousands — of possible execution plans; these requirements make cost-based optimizers’ operation CPU-intensive and unpredictable.
eXtremeDB Financial Edition also supports ODBC, which expands its ability to share data with systems ranging from mainframes to servers and desktop applications, and enables developers to create applications using an open and familiar database API.
Developers can access eXtremeDB Financial Edition’s most powerful capabilities – including pipelining vector-based statistical functions – from the industry standard SQL database programming language and Python scripting language. Using these languages alongside eXtremeDB Financial Edition’s dynamic database definition language (DDL) capability, developers can implement their ideas quickly, and optimize rapidly by testing changes to code, database tables and indexes.
Application development (including the use of pipelining and other specialized features) is supported in C/C++, Java, C# (.NET) and Python. The native, navigational C/C++ API is type-safe and specific to a given project/database design. When coding in C or C++, applications can access the database using this API as well as standard SQL/ODBC. eXtremeDB Financial Edition’s SQL implementation also provides a type-3, version 4 JDBC driver for development in Java. Its Java Native Interface (JNI) and C# native interface offer the fastest possible DBMS solutions in these languages, and afford developers the experience of working entirely with “plain old” Java and C# objects.
eXtremeDB Financial Edition is available for use on x32- and x64-bit versions of Windows and Linux, Linux on POWER8, Solaris Sparc, Solaris x86_64, HP-UX, and AIX.
eXtremeDB Financial Edition supports distributed query processing (aka horizontal partitioning) in which queries are divided between shards (instances of the eXtremeDB Financial Edition database server) that execute on multiple servers, CPUs and/or CPU cores. Performance is accelerated by parallel execution of database operations and by harnessing the capabilities of many host computers rather than just one.
eXtremeDB Financial Edition’s code path is very short: a “footprint” of approximately 150K for the core database system points to McObject’s unrelenting focus on eliminating even small sources of latency. If trading organizations will co-locate their IT with exchanges to eliminate network communication distance and its accompanying latency, they’d do well to consider another type of ‘distance’ – software code execution path – which imposes its own latency. (McObject’s CEO wrote an article on just this topic in Wall Street & Technology magazine.)
Many database systems include one or two of the performance-enhancing features mentioned so far, such as in-memory data storage, columnar data handling, fast native APIs as well as high performance SQL, and a short execution path. But you’d be hard-pressed to find all of them in one database system, and solutions offering some of these features may also lack “real” database capabilities such as ACID transactions. In contrast, eXtremeDB Financial Edition offers all of them plus a rich assortment of additional development and run-time features (our Key Features list is a good place to see the breadth of eXtremeDB Financial Edition’s capabilities).
Equally important, with new NoSQL solutions and databases cropping up so frequently, McObject’s eXtremeDB is a proven technology. McObject can point to a track record of 12+ years and millions of successful deployments of the database system, in demanding financial technology as well as in mission- and safety-critical applications ranging from aerospace systems to smart grid, defense, consumer electronics, industrial control and telecom/networking infrastructure.
eXtremeDB Financial Edition has been the subject of multiple STAC-M3 benchmark implementations, overseen and audited by the Securities Technology Analysis Center (STAC®). An overview of eXtremeDB Financial Edition’s record-setting performance on the benchmarks, and a link to download the STAC reports, is available here.
Another test of performance, from a partnership between McObject, Kove and Fultech Consulting, implemented complex risk analysis and reporting on a highly compressed timescale. It measured portfolio exposure of 50,000 accounts, each with 3,278 positions, using a 10-year historical value-at-risk (VaR) model, with a target processing time of ten minutes. The technology stack incorporating eXtremeDB Financial Edition performed the approximately 428 billion calculations in less than 8.5 minutes, handling approximately 13.5 terabytes of data, reduced to 3.5 terabytes of Kove® XPD™ L2 storage, to demonstrate continuous, sustained, deterministic and random financial model processing of approximately 950 million calculations per second and 65 terabytes per hour. Read more.
To evaluate eXtremeDB Financial Edition, please call us at +1 425 888 8505 or e-mail us. Evaluation software packages include complete documentation, the eXtremeDB Financial Edition runtime, schema compiler, and samples that illustrate features and programming techniques.
The Evaluation Software page includes documentation, including the eXtremeDB Financial Edition User’s Guide and Reference Guide. Separate addenda are offered for key functions such as Clustering, High Availability and SQL.
We strongly recommend you start the evaluation process by downloading and reading eXtremeDB – Getting Started, a 14-page PDF that walks through installation, basic concepts, and first steps in using the core product. Install the software using the provided installation program. Build the samples according to the readme file. Spend some time reading through the User’s Guide and, as different features are discussed, refer to the appropriate samples. Build some tests to measure the performance and memory usage. Contact us if you need assistance with the evaluation process. We will make sure that you get answers to all your questions in a timely manner, and will gladly assist you with writing benchmark tests.
McObject offers two demo packages based on eXtremeDB Financial Edition. The Algorithmic Trading Demo Application simulates time-critical quantitative analysis and data management tasks used in automated trading, to show how a distributed in-memory database system (IMDS) for capital markets can process complex queries and maintain a high transaction rate, in part by distributing the database to traders’ workstations. The Sequences Demo illustrates the use of eXtremeDB Financial Edition’s ‘sequence’ data type for managing time series data, emulating a tick stream for a single symbol and performing several common analytical queries, including Sum, Covariance and Correlation.
McObject offers an annual technical support and software maintenance contract. This entitles the developer to unlimited technical support through any channel (telephone, email, etc.), free updates (version number updates to the right of the decimal point, e.g. from 2.0 to 2.1) and free upgrades (version number updates to the left of the decimal point, e.g. from 2.0 to 3.0).
Yes. While eXtremeDB is not open source, McObject’s philosophy is to make the source code available and affordable for developers.