Key eXtremeDB Features
eXtremeDB Embedded Database Architecture
|Flexible Storage (In-Memory or On-Disk)||Embedded or Client/Server Database Architecture|
|Short Execution Path||Columnar Layout for Time Series Data|
Powerful Run-Time Features
Unmatched Developer Flexibility
The eXtremeDB Financial Edition Database System, Feature-By-Feature:
eXtremeDB Financial Edition is based on a core in-memory database system (IMDS) design that eliminates performance-draining I/O, cache management and data transfer. Persistent (on-disk) storage is optional and can be implemented selectively. The ability to combine in-memory and persistent storage is ideal for handling real-time quote data and historical data within a single database architecture. When using such hybrid storage, a simple notation in the database schema designates a record type for on-disk storage.
eXtremeDB Financial Edition provides the choice of configuring the database system in either embedded (DBMS runs entirely within the application process) or client/server designs, to support tiered application processing and performance optimization.
eXtremeDB Financial Edition’s code path is very short: a “footprint” of approximately 150K for the core database system points to McObject’s unrelenting focus on eliminating even small sources of latency. This short path speeds execution by reducing the number of CPU cycles required per database operation and by increasing the likelihood that code needed for an operation is already in the CPU cache. eXtremeDB is two to three times faster than a popular database often used for real-time and historical market data.
Traditional DBMSs bring rows of data into the CPU cache for processing. But financial data – such as trades and quotes – is naturally columnar, and handled more efficiently by a column-based layout. eXtremeDB Financial Edition stores time series data with a columnar layout, and “normal” data with a conventional row-based layout. The result is higher performance, resulting from a database system that best leverages the CPU cache speed and avoids costly (in performance terms) fetches from main memory.
Committed to 99.999% uptime or better? eXtremeDB Financial Edition’s high availability option ensures continuous database operation even in the face of hardware or software failure. eXtremeDB Financial Edition’s HA support includes both synchronous (2-safe) and asynchronous (1-safe) replication, with automatic or application-directed failover.
eXtremeDB Financial Edition’s clustering capability can dramatically increase available net processing power, lower system expansion costs (by enabling use of low-cost “commodity” servers), and maximize scalability and reliability. In a cluster deployment, all nodes serve as “master,” but a local tables option speeds processing at selected nodes by defining data sets to be managed at those nodes, without cluster-wide replication.
eXtremeDB Financial Edition is unique as the first clustering database system to offer an embedded architecture: the database system runs within the application process at every node. This design eliminates inter-process communication (IPC) overhead within database instances, accelerating performance. It also reduces complexity, contributing to higher reliability and dramatically simplified deployment and maintenance.
Run-length encoding (RLE) compression can be applied to columnar data (i.e. fields defined as the ‘sequence’ data type). In McObject’s tests, activating this feature reduced tick data to one-quarter of its pre-compression size, and increased speed in reading the database by 21%. (eXtremeDB Financial Edition also includes a feature for compressing non-columnar data.)
eXtremeDB Financial Edition’s graphical performance monitor shows the effect of changes to the database or application on transaction throughput, memory consumption and other key factors when fine-tuning to reduce latency. End-users can optimize software for efficiency while developers gain the ability to offer integrated performance monitoring under their own product’s branding. (See a screen shot of the performance monitor.)
eXtremeDB Financial Edition’s transactions support the ACID (Atomic, Consistent, Isolated and Durable) principles, which safeguard data integrity by guaranteeing that updates will complete together or the database will roll back to a pre-transaction state. This feature also enables the developer to prioritize transactions.
This capability enables eXtremeDB Financial Edition to partition a database and distribute query processing across multiple servers, CPUs and/or CPU cores. Performance is accelerated – dramatically, in some cases – via parallel execution of database operations and by harnessing the capabilities of many host computers.
eXtremeDB Financial Edition is 64-bit and scales to support in-memory databases that are terabytes in size and on-disk databases that are much larger. In a published benchmark, an eXtremeDB in-memory database scaled to 1.17 terabytes and 15.54 billion rows, with no apparent limits on further growth. Performance remained consistent as the database size increased, suggesting nearly linear scalability. How fast is an IMDS at scale? For a simple SELECT against the fully populated database, eXtremeDB processed 87.78 million query transactions per second using its native application programming interface (API) and 28.14 million transactions per second using its SQL API.
Another important tool for scalability is eXtremeDB Financial Edition’s optional Multi-Version Concurrency Control (MVCC) transaction manager, which is available as an alternative to the “pessimistic” model of the MURSIW (MUltiple Reader, SIngle Writer) transaction manager. MVCC can dramatically improve scalability and performance, especially in applications with a) on-disk or hybrid (in-memory and on-disk) database storage, b) many tasks or processes concurrently modifying the database (versus read-only) and, c) in multi-core systems.
eXtremeDB Financial Edition’s transaction logging adds recoverability of in-memory databases by appending database changes to a transaction log on persistent media. Logging may be set to different levels of transaction durability, allowing system designers to make intelligent trade-offs between performance and risk of unrecoverable transactions.
McObject’s eXtremeDB Data Relay technology facilitates seamless, fine-grained data replication from real-time systems based on eXtremeDB Financial Edition to external systems such as enterprise DBMSs.
eXtremeDB Financial Edition, in its persistent storage, improves on typical Least Recently Used (LRU) cache policies by allowing applications to influence how long certain pages remain in cache, to minimize retrieval overhead for objects used in time-sensitive tasks.
Similar to triggers in enterprise DBMSs, this feature enables eXtremeDB Financial Edition to notify an application when something “of interest” in the database changes. It supports synchronous and asynchronous modes.
Cost-based SQL optimizers’ practice of collecting meta data (such as row counts), generating statistics (such as index histograms) and analyzing potential execution plans make such optimizers CPU-intensive and unpredictable. In contrast, SQL in eXtremeDB Financial Edition uses a highly efficient and predictable rule-based optimizer.
eXtremeDB Financial Edition protects your database. Cyclic Redundancy Check (CRC) on the database page level detects any unauthorized modification to stored data, while RC4 encryption employs a user-provided cipher to prevent access or tampering. Secure Sockets Layer (SSL) is supported in all communications (client/server, HA and Cluster).
eXtremeDB can save a database as a binary image and then restore it with a changed schema. This can accomplish design changes quickly.
eXtremeDB is primarily an embedded DBMS but provides an RPC library and tools to develop an application-specific client/server database interface. In addition, the eXtremeDB Financial Edition SQL library enables the remote applications to connect to a central database, and provides an ODBC driver for standards-based connections to both local and remote databases. The product’s clustering capability, which distributes the database to multiple network nodes, can eliminate the need for nodes to establish a remote connection to a central database.
eXtremeDB Financial Edition’s persistent storage can exploit multi-disk (solid state or spinning) configurations with its support for RAID-like data striping and data mirroring. Striping accelerates performance when working with two or more disks by parallelizing access to data. Mirroring provides continuous backup by replicating data onto separate disks (if disk A fails, identical records are available from disk B).
eXtremeDB Financial Edition is available for use on x32- and x64-bit versions of Windows and Linux, Linux on POWER8, Solaris Sparc, Solaris x86_64, HP-UX (Itanium), and AIX.
Developers can access eXtremeDB Financial Edition’s most powerful capabilities – including pipelining vector-based statistical functions – from the industry standard SQL database programming language and Python scripting language. Using these languages alongside eXtremeDB Financial Edition’s dynamic database definition language (DDL) capability, developers can implement their ideas quickly, and optimize rapidly by testing changes to code, database tables and indexes.
Application development is supported in C/C++, Java, C# (.NET) and Python. When coding in C or C++, applications can access the database system using its fast, native navigational C/C++ API as well as standard SQL/ODBC. eXtremeDB Financial Edition’s SQL implementation also provides a type-3, version 4 JDBC driver for development in Java. Its Java Native Interface (JNI) and C# native interface offer the fastest possible DBMS solutions in these languages, and afford developers the experience of working entirely with “plain old” Java and C# objects.
eXtremeDB Financial Edition’s native C/C++ API is type-safe: errors in data-typing are caught at compile time, to eliminate database corruption. In addition, the database runtime implements many verification traps and consistency checks. After application debugging, the optimized version of the eXtremeDB Financial Edition runtime can be used, removing traps and internal checks, and restoring valuable clock cycles. In addition, cyclic redundancy check is built into the backup-and-restore feature for in-memory databases, to ensure the database was written in its entirety when saved, and has not been corrupted.
eXtremeDB provides a wide range of database indexes, to boost application performance and minimize footprint. These include standard B-trees, KD-trees for multi-dimensional data and Query-by-Example (QBE), custom indexes, hash indexes and more. For in-memory databases, rather than storing duplicate data, indexes contain only a reference to data, keeping memory requirements to an absolute minimum.
eXtremeDB Financial Edition supports a wide range of data types — including structures, arrays, vectors and BLOBs — for maximum coding efficiency. Data can be stored in the same complex form in which it is used in the application, or as normalized relations. Date, time and timestamp (down to the nanosecond) are fully supported by all eXtremeDB Financial Edition programming interfaces.
eXtremeDB Financial Edition provides an extensive library of vector-based statistical functions for financial calculations. Functions execute over one or more sequences of data (typically time series data, such as trades and quotes), cutting latency by maximizing the CPU cache use. Functions can be pipelined to form an assembly line of operations on sequences, to perform statistical/quantitative analysis.
Available source code provides the ultimate transparency for developers using eXtremeDB Financial Edition.
eXtremeDB has a 15+ year track record in some of the most demanding application categories, including both real-time embedded systems (aerospace, industrial control, etc.) and financial technology. Hundreds of organizations use McObject’s DBMS and have deployed millions of run-time eXtremeDB copies.
McObject technical support engineers are database and real-time application experts who answer customer questions in detail. Customers get prompt, informative answers and continued follow-up to speed development and get the most from eXtremeDB.