Distributed Caching for Storage Acceleration : The Fundamentals

Distributed Caching for Storage Acceleration is a modus operandi for increased server performance with highest density of virtual machines balancing I/O. The topic Distributed Caching is elaborated discussed first quite elaborately on the Internet by Marc Leavitt, on DataCenterKnowledge. The other important source of ideas and knowledge on Distributed Caching for Storage Acceleration can be found on Google Scholar. This is a basic article on Distributed Caching for Storage Acceleration to provide an average user an idea about the topic. In depth article is beyond the scope of this website as we are more towards the usages for the end users. This article contains internal and public publications for academic purpose published by Microsoft, HP, ISRO, NASA as well. We have mentioned these resources on Distributed Caching for Storage Acceleration for academic interest of the reader. This article is itself not a republication of any of the source.

Distributed Caching for Storage Acceleration

For the system administrators, the increased server performance, highest density of virtual machines, improvements in network bandwidth and intensive enterprise workloads as a whole, creates an imbalance in the performance of I/O between the servers, networks and traditional storage systems. Storage I/O is the well known bottleneck in performance sensitive applications. While the performance of the processor and memory are growing maintaining the curve drawn from Moore’s Law, the storage however remained lagging far behind. This performance gap is further evidenced by the rapid growth of data volumes or the word – big data, this is the most organizations, companies, Governments, Institutions are experiencing today. It is expected to be increased by 44 times in the next 10 years.

Following are taken as the best practices in the industry; the storage is installed, the centrally storage networks which can be connected by Fibre Channel, Fibre Channel over Ethernet (FCoE), iSCSI, to increase the efficiency, the policy compliance, and the strategy for protection of data. In any case, the design of a networked storage introduces many new points in which the latency actually increases and this increases the response time and reduces the access of applications to the information and decreases the total performance.

This is the basis of the thought behind Distributed Caching for Storage Acceleration.

Distributed Caching for Storage Acceleration : The Approaches and Issues

In recent years, the flash memory has been used as a useful instrument to increase the storage performance, the flash technology in fact exceeds the performance of magnetic disks. To improve system performance, flash was primarily configured as a SSD drive, a format that simplifies and accelerates the adoption.  These State of art SSD equipped servers combined with usual caching, can be expensive to implement to achieve a good level of performance, it requires complex layers of caching software, which increase the risks of interoperability and consume the resources of the processors and memory. For this reason, we said that Reduce Cache to Reduce Operational Cost on Cloud Server for gross usage.

The storage industry has produced a new solution that simultaneously eliminates the risk of corruption of data due to the loss of coherence in the cache while allowing scalability and resource pooling SSD cache between servers.  The storage clustering creates a logical group, that provides a single point of management and maintains cache coherency with high availability and optimal allocation of the cache. When one creates the cluster, a cluster is assigned as the primary control cluster and another cluster as secondary.

These are basic points on Distributed Caching for Storage Acceleration, which is actually employed on mid to high end servers. It is quite obvious point that, for a rented Cloud, you need not to think about the longevity of the server due to increased workload but about the bill. But for the company or you as a bare metal owner, it also becomes a big issue.

What we have practically seen, using the method of partitioning of data as well as replication for database server (typically for MySQL) works better than simple database Caching. Such example usage on the frontend can be HyperDB plugin / dropin for WordPress.