M3 vs Mimir

M3 Overview

M3 is a distributed time series database written entirely in Go. It is designed to collect a high volume of monitoring time series data, distribute storage in a horizontally scalable manner, and efficiently leverage hardware resources. M3 was initially developed by Uber as a scalable remote storage backend for Prometheus and Graphite and later open-sourced for broader use.

Mimir Overview

Grafana Mimir is an open-source software project that provides a scalable long-term storage solution for Prometheus. Started at Grafana Labs and announced in 2022, Grafana Mimir aims to become the most scalable and performant open-source time series database for metrics. The project incorporates the knowledge and experience gained by Grafana Labs engineers from running Grafana Enterprise Metrics and Grafana Cloud Metrics at massive scale.

M3 for Time Series Data

M3 is specifically designed for time-series data. It is a distributed and scalable time-series database optimized for handling large volumes of high-resolution data points, making it an ideal solution for storing, querying, and analyzing time-series data.

M3’s architecture focuses on providing fast and efficient querying capabilities, as well as high ingestion rates, which are essential for working with time-series data. Its horizontal scalability and high availability ensure that it can handle the demands of large-scale deployments and maintain performance as data volumes grow.

Mimir for Time Series Data

Grafana Mimir is well-suited for handling time series data, making it a suitable choice for scenarios involving metric storage and analysis. It provides long-term storage capabilities for Prometheus, a popular open-source monitoring and alerting system. With Grafana Mimir, users can store and query time series metrics over extended periods, allowing for historical analysis and trend detection. It is especially useful for applications that require scalable and performant storage of time series data for metrics monitoring and observability purposes.

M3 Key Concepts

• Time Series Compression: M3 has the ability to compress time series data, resulting in significant memory and disk savings. It uses two compression algorithms, M3TSZ and protobuf encoding, to achieve efficient data compression.
• Sharding: M3 uses virtual shards that are assigned to physical nodes. Timeseries keys are hashed to a fixed set of virtual shards, making horizontal scaling and node management seamless.
• Consistency Levels: M3 provides variable consistency levels for read and write operations, as well as cluster connection operations. Write consistency levels include One (success of a single node), Majority (success of the majority of nodes), and All (success of all nodes). Read consistency level is One, which corresponds to reading from a single nod

Mimir Key Concepts

• Metrics: In Grafana Mimir, metrics represent the measurements or observations tracked over time. They can include various types of data, such as system metrics, application performance metrics, or sensor data.
• Long-term Storage: Grafana Mimir provides a storage solution specifically tailored for long-term retention of time series data, allowing users to store and query historical metrics over extended periods.
• Microservices: Grafana Mimir adopts a microservices-based architecture, where the system consists of multiple horizontally scalable microservices that can operate independently and in parallel.

M3 Architecture

M3 is designed to be horizontally scalable and handle high data throughput. It uses fileset files as the primary unit of long-term storage, storing compressed streams of time series values. These files are flushed to disk after a block time window becomes unreachable. M3 has a commit log, equivalent to the commit log or write-ahead-log in other databases, which ensures data integrity. Client Peer streaming is responsible for fetching blocks from peers for bootstrapping purposes. M3 also implements caching policies to optimize efficient reads by determining which flushed blocks are kept in memory.

Mimir Architecture

Grafana Mimir adopts a microservices-based architecture, where the system comprises multiple horizontally scalable microservices. These microservices can operate independently and in parallel, allowing for efficient distribution of workload and scalability. Grafana Mimir’s components are compiled into a single binary, providing a unified and cohesive system. The architecture is designed to be highly available and multi-tenant, enabling multiple users and applications to utilize the database concurrently. This distributed architecture ensures scalability and resilience in handling large-scale metric storage and retrieval scenarios.