eXtremeDB Financial Edition FAQ
eXtremeDB Financial Edition is McObject’s database system for capital markets applications. It addresses the challenge to financial IT posed by growing volumes of market data that must be sorted and analyzed with the lowest possible latency, to support key application functions such as managing risk, identifying and acting on trading opportunities, and ensuring speed and reliability in financial exchanges.
eXtremeDB Financial Edition is a full-featured database management system (DBMS) with support for industry standards including SQL, ODBC and JDBC, for multi-process concurrency and for transactions that support the ACID (Atomic, Consistent, Isolated and Durable) properties. It is an embedded or in-process database system, i.e. it runs within the application process rather than as separate client and server modules, for maximum efficiency. It is designed to be integrated within applications, with little or no maintenance by end-users. To support this developer orientation, eXtremeDB Financial Edition offers sophisticated developer tools such as multiple application programming interfaces (APIs), support for specialized database indexes, event notifications and more.
Specialized features include an optional “sequence” data type that implements column-based handling of time series data (typically trades, quotes, etc.), and a library of vector-based statistical functions that execute over sequences and can be pipelined to form an assembly line of processing on the market data. Used together, these features maximize the efficiency of the CPU cache in managing market data. Columnar handling ensures that the largest amount of time-critical data (sequence elements) is brought into the CPU with each fetch, and avoids flooding the CPU cache with irrelevant data. Pipelining of vector-based functions causes intermediate result sets used in processing to remain in CPU cache rather than being output as temporary tables. This eliminates the overhead of back-and-forth trips between the CPU cache and main memory.
McObject launched the eXtremeDB product line more than 10 years ago. The initial product, a highly efficient in-memory database system with a C/C++ language interface, targeted embedded systems such as telecom and networking equipment, industrial control, aerospace, defense and consumer electronics. Since that introduction, eXtremeDB has gained features including optional persistent storage, high availability and clustering sub-systems, 64-bit support, new programming interfaces, and much more. eXtremeDB is used by hundreds of companies and deployed in millions of run-time applications and devices. These customers include many financial technology providers who, over the years, identified eXtremeDB as their ideal solution while researching high performance database systems. With feedback from these users – and from our own research identifying the significant challenges in handling market data – McObject developed eXtremeDB Financial Edition. It incorporates all capabilities and editions in the existing eXtremeDB product family, and adds new features such as the columnar data handling and vector-based statistical functions discussed above.
At its core, eXtremeDB Financial Edition is an in-memory database system (IMDS): records are stored in main memory, eliminating caching, disk and file I/O, and other significant storage-related latency found in “traditional” on-disk DBMSs. As a result, eXtremeDB operates many times faster than a conventional on-disk database management system, even if the disk-based DBMS is used with a RAM disk (this is discussed in detail in the white paper “Main Memory vs. RAM-disk Databases: A Linux-based Benchmark”, available for download here: White Papers).
However, eXtremeDB Financial Edition’s “Fusion” sub-system enables the developer to selectively add persistent storage, so that a given database design can incorporate anywhere from zero to 100% persistent storage. When used with persistent storage, eXtremeDB Financial Edition uses caching to keep frequently-accessed records in memory, for faster access. It also includes features such as cache prioritization to minimize retrieval overhead for objects used in time-sensitive tasks, as well as database striping to accelerate performance, and mirroring to support continuous backup, when working with two or more disks.
Yes. eXtremeDB Financial Edition’s SQL API implements much of the SQL-89 standard for SELECT, INSERT, UPDATE and DELETE statements, and implements many extensions to support eXtremeDB-specific features. Compared to other SQL databases, eXtremeDB Financial Edition with SQL lends a performance advantage through its highly efficient and predictable rule-based SQL query optimizer. In contrast, most SQL DBMSs rely on a cost-based optimizer that must collect sample data, generate statistics and analyze thousands — sometimes hundreds of thousands — of possible execution plans; these requirements make cost-based optimizers’ operation CPU-intensive and unpredictable.
eXtremeDB Financial Edition also supports ODBC, which expands eXtremeDB’s ability to share data with systems ranging from mainframes to servers and desktop applications, and enables developers to create applications using an open and familiar database API.
The most popular interface for working with eXtremeDB is its fast, native navigational API consisting of C/C++ functions. This is provided as both a type-safe and intuitive application-specific API (functions are generated based on the data design), and as a uniform data access (UDA) API for a consistent interface across all projects.
In addition, eXtremeDB’s Java Native Interface (JNI), C# native interface, and Python interface offer the fastest possible DBMS solutions in these languages, and the ability to access eXtremeDB while working entirely with “plain old” Java, C# and Python objects. The C# interface includes support for Language Integrated Query (LINQ), a Microsoft .NET Framework component that adds SQL-like data querying capabilities.
As mentioned above, eXtremeDB Financial Edition also supports SQL and ODBC. In addition, it includes a Java Database Connectivity (JDBC) driver compatible with JDK 7.
eXtremeDB Financial Edition’s code path is very short: a “footprint” of approximately 150K for the core database system points to McObject’s unrelenting focus on eliminating even small sources of latency. This short path speeds execution by reducing the number of CPU cycles required per database operation and by increasing the likelihood that code needed for an operation is already in the CPU cache.
If trading organizations will co-locate their IT with exchanges to eliminate network communication distance and its accompanying latency, they’d do well to consider another type of ‘distance’ – software code execution path – which imposes its own latency. (McObject’s CEO wrote an article on just this topic in Wall Street & Technology magazine.) Short execution path is an important reason eXtremeDB has been proven two to three times faster than a popular database often used for real-time and historical market data. The efficiency lent by this feature was also evident in eXtremeDB Financial Edition’s record-setting performance in independent, audited STAC-M3 benchmark tests.
Many database systems include one or two of the performance-enhancing features mentioned so far, such as in-memory data storage, columnar data handling, in-process architecture, fast native APIs as well as high performance SQL, and a short execution path. But you’d be hard-pressed to find all of them in one database system, and solutions offering some of these features may also lack “real” database capabilities such as ACID transactions. In contrast, eXtremeDB Financial Edition offers all of them plus a rich assortment of additional development and run-time features (our Key Features list is a good place to see the breadth of eXtremeDB Financial Edition’s capabilities).
Equally important, with NoSQL solutions and databases purporting to be “in-memory” cropping up so frequently today, McObject’s eXtremeDB is a proven technology. McObject can point to a track record of 10+ years and millions of successful deployments of the database system, in demanding financial technology as well as in mission- and safety-critical applications ranging from aerospace systems to smart grid, defense, consumer electronics, industrial control and telecom/networking infrastructure.
In an audited, independent benchmark, the Securities Technology Analysis Center (STAC®) documented a technology stack based on eXtremeDB Financial Edition achieving the lowest mean response times and lowest mean latency ever reported for 15 of 17 STAC-M3™ tests of time-series data management. The STAC-M3 tests consist of complex queries designed by trading firms to reflect real-world capital markets computing demands, and are considered the gold standard for measuring time-series data management (tick database) performance. Download the report from STAC.
Another test of performance, from a partnership between McObject, Kove and Fultech Consulting, implemented complex risk analysis and reporting on a highly compressed timescale. It measured portfolio exposure of 50,000 accounts, each with 3,278 positions, using a 10-year historical value-at-risk (VaR) model, with a target processing time of ten minutes. The technology stack incorporating eXtremeDB Financial Edition performed the approximately 428 billion calculations in less than 8.5 minutes, handling approximately 13.5 terabytes of data, reduced to 3.5 terabytes of Kove® XPD™ L2 storage, to demonstrate continuous, sustained, deterministic and random financial model processing of approximately 950 million calculations per second and 65 terabytes per hour. Read more.
Yet another eXtremeDB benchmark measured performance managing a 1.17 Terabyte, 15.54 billion row in-memory database on a 160-core Linux-based SGI® Altix® 4700 server. In SELECT, JOIN and SUBQUERY tests, benchmark results ranged as high as 87.78 million query transactions per second. It compared performance of eXtremeDB’s SQL API vs. its native, navigational C/C++ interface, and examined efficiency in utilizing all of the test bed system’s 160 processors. McObject offers a 4-page benchmark synopsis of this benchmark as well as the full report with database schema and relevant application code.
Download one of the eXtremeDB Financial Edition evaluation packages – currently available for Windows and Linux (32- and 64-bit) – from Evaluation Software. These include complete documentation, the eXtremeDB Financial Edition runtime, schema compiler, and samples that illustrate features and programming techniques.
The Evaluation Software page includes documentation, including the eXtremeDB Financial Edition User’s Guide and Reference Guide. Separate addenda are offered for key functions such as Clustering, High Availability and SQL.
We strongly recommend you start the evaluation process by downloading and reading eXtremeDB – Getting Started, a 14-page PDF that walks through installation, basic concepts, and first steps in using the core product. Install the software using the provided installation program. Build the samples according to the readme file. Spend some time reading through the User’s Guide and, as different features are discussed, refer to the appropriate samples. Build some tests to measure the performance and memory usage. Contact us if you need assistance with the evaluation process. We will make sure that you get answers to all your questions in a timely manner, and will gladly assist you with writing benchmark tests.
McObject offers two demo packages based on eXtremeDB Financial Edition. The Algorithmic Trading Demo Application simulates time-critical quantitative analysis and data management tasks used in automated trading, to show how a distributed in-memory database system (IMDS) for capital markets can process complex queries and maintain a high transaction rate, in part by distributing the database to traders’ workstations. The Sequences Demo illustrates the use of eXtremeDB Financial Edition’s ‘sequence’ data type for managing time series data, emulating a tick stream for a single symbol and performing several common analytical queries, including Sum, Covariance and Correlation.
McObject offers an annual technical support and software maintenance contract. This entitles the developer to unlimited technical support through any channel (telephone, email, etc.), free updates (version number updates to the right of the decimal point, e.g. from 2.0 to 2.1) and free upgrades (version number updates to the left of the decimal point, e.g. from 2.0 to 3.0).
Yes. While eXtremeDB is not open source, McObject’s philosophy is to make the source code available and affordable for developers.