In Memory Data Grid vs. In Memory Database
The technologies that are most present in our lives are often ones we don’t realize exist, (even within the IT domain), or are poorly understood. In this post I’d like to provide a business-level explanation of the difference between an in-memory data grid and an in-memory database in order to drive a better understanding of why this is important to a non-technical reader. The key attributes are how important is scalability, and how critical is it that you keep your existing database?
The context for this is transactional use. If you use a credit card, if you’ve ever purchased from a major e-commerce site, if you play online games, etc., you’re already a user of in-memory technology, and the choice of in-memory technology affects the quality of your experience.
One of the ongoing broadly-based, but not well recognized, technology initiatives underway is to move compute and storage into memory in order to avoid the latency associated with traversing a network to access information stored in a database. A good example of this is the complexity associated with placing an online order: the e-commerce site has to sort through configuration options, inventory availability, shipping and taxes, updating your customer profile, not to mention validating the credit card, etc. In this example, multiple complex transactions execute so quickly you don’t even notice. This is the sort of thing where in-memory technology affects billions of people continuously. The two popular approaches to putting information where it can be accessed instantly are in-memory databases, and in-memory data grids. There are lots of low level technical differences which would be interesting to a developer, but surfacing the differences to an executive is more critical, since they are more closely aligned with how their business makes money, and understanding this technology can have a huge impact on your bottom line.
An In-Memory Data Grid (IMDG) is not a database, it is a technology that is designed to augment an existing database, rather than replace it. An In-Memory Database (IMDB) is a database, with all the infrastructure and overhead associated with a database that just happens to be in memory. An IMDG sits between an existing database and the application that needs access to it, such as a customer-facing product catalogue on an e-commerce site. The IMDG (like the back-end data it supports) is often distributed across multiple nodes and essentially serves as a cache of data (a copy of data that is accessed repeatedly and therefore needs to be reached quickly). Having this type of data cached in memory makes a huge difference in access time (think micro-seconds vs. minutes). The bottom line, however, is that an IMDG still requires a database in the ecosystem, even though they are often tightly integrated. There will be some changes required to the application that accesses the IMDG, since it is different than a database, but the technology to do this is mature, well-known, and easy to use. An IMDG is a good choice if scalability is a requirement, and most IMDGs (particularly Hazelcast, if your transactions are complex)) do not compromise on speed at all.
An In-Memory Database, by comparison, is a full-blown database. If you choose to implement an IMDB, you need to replace your existing database and since they are normally a core part of a legacy system, replacing them is non-trivial. On the upside, there will be limited changes required to the application layer. IMDBs offer good speed, but do not scale particularly well. So moving your database in-memory will deliver some speed improvements and won’t do you much good if you need to scale, but will only require limited changes to the apps that access the database.
As mentioned earlier, there are lots of technical nuances between the two options (which are quite easy to find). Bottom line? If your needs are modest and you don’t plan to grow, an IMDB will probably meet your needs. If you’re driven by complex transactions that require speed, have to scale to internet levels, and can’t or don’t want to swap out your database (keeping in mind all legacy systems include a database), then IMDG is probably the best choice.