Beyond that, this release expands TimeSeries and TDigest command coverage and tightens replication reliability — all while upgrading the C++ standard to C++20 and RocksDB to v10.10.1.

Highlights:

• Support SELECT database — Kvrocks can now emulate Redis's multi-database model, unblocking migrations from Redis deployments that use SELECT to switch between logical databases
• New TimeSeries commands — TS.MREVRANGE, TS.QUERYINDEX, and TS.ALTER round out the time-series API surface for analytics-heavy workloads
• New TDigest commands — TDIGEST.RANK, TDIGEST.REVRANK, TDIGEST.BYRANK, and TDIGEST.BYREVRANK enable percentile and rank queries on streaming data

Support SELECT Database​

Kvrocks uses a namespace mechanism as its multi-tenancy primitive, which differs from Redis's numbered logical databases. Before this release, issuing SELECT against Kvrocks was a no-op — the command was accepted but had no effect. v2.15.0 introduces the redis-databases option (#3294) to bridge this gap and make SELECT behave as it does in Redis.

The option takes a numeric value:

• redis-databases 0 — default; SELECT remains a no-op (existing namespace behavior is unchanged)
• redis-databases N (N > 0) — enables Redis-style database switching with N logical databases; requires that no namespaces are configured

# kvrocks.conf
redis-databases 16

Once enabled, clients can issue SELECT as they would against Redis:

127.0.0.1:6666> SELECT 3
# OK
127.0.0.1:6666[3]> SET mykey "hello"
# OK — stored in database 3

New TimeSeries Commands​

Kvrocks's RedisTimeSeries compatibility layer gains three commands that are essential for production time-series pipelines. TS.MREVRANGE lets you query multiple series in reverse chronological order in a single round-trip. TS.QUERYINDEX filters series by label predicates without pulling data. TS.ALTER modifies retention and label metadata on existing series without recreating them.

# Create series with labels
127.0.0.1:6666> TS.CREATE metrics:us-east:1 LABELS region us-east
127.0.0.1:6666> TS.CREATE metrics:us-east:2 LABELS region us-east
127.0.0.1:6666> TS.CREATE temperature:NYC LABELS sensor temperature location NYC

# Add samples
TS.ADD metrics:us-east:1 * 42
127.0.0.1:6666> TS.ADD metrics:us-east:2 * 87

# Query the last 10 samples across all "region=us-east" series, newest first
127.0.0.1:6666> TS.MREVRANGE - + FILTER region=us-east COUNT 10

# Find series matching a label filter (no data returned)
127.0.0.1:6666> TS.QUERYINDEX sensor=temperature location=NYC

# Change retention period on an existing series
127.0.0.1:6666> TS.ALTER temperature:NYC RETENTION 86400000

New TDigest Commands​

The TDigest sketch — useful for approximate quantile computation over streaming data — now supports rank-based queries. TDIGEST.RANK returns the estimated rank of a value within the digest. TDIGEST.BYRANK returns the value at a given rank. The REV variants work in descending order. Together these four commands make it practical to answer questions like "what percentile is this latency?" or "what is the p99 value?" directly from Kvrocks.

# Create a TDigest key
127.0.0.1:6666> TDIGEST.CREATE latency

# Insert observations
127.0.0.1:6666> TDIGEST.ADD latency 120 95 200 340 88 150

# What rank is the value 150?
127.0.0.1:6666> TDIGEST.RANK latency 150

# What value sits at rank 4?
127.0.0.1:6666> TDIGEST.BYRANK latency 4

# What value sits at rank 4 counting from the top?
127.0.0.1:6666> TDIGEST.BYREVRANK latency 4

End​

Except for that, this release also includes a number of bug fixes and performance improvements. For the full list of changes, please refer to changelog to see all the details. We encourage users to upgrade to v2.15.0 to take advantage of these new features and improvements. As always, we welcome feedback and contributions from the community!

TL;DR:

Pretty cool, right? Let's dive in!

(The full example is provided in the final section.)

Apache Kvrocks​

Apache Kvrocks is a Redis-compatible database built on RocksDB.

It supports the RESP protocol (version 2 and 3) and a wide range of Redis commands, encompassing core data structures like Strings, Sets, Hashes, Sorted Sets, Stream, GEO, as well as Lua Scripts, Transactions, Functions and even BloomFilter, JSON from the Redis Stack.

Unlike Redis which stores data in memory, Kvrocks persists data on disk for improved storage capabilities without being constrained by machine memory limit.

The capability to query​

In recent years, NoSQL databases have become more popular than traditional databases because they perform better, scale easily, and are more flexible for different industries.

However, many users are unwilling to abandon the essential features of SQL databases just for performance reasons. These include ACID transactions, expressive query capabilities inherent in SQL, as well as optimization and abstraction possibilities offered by structured data and relational algebra. Consequently, a new category of databases known as NewSQL has emerged gradually.

Kvrocks is a NoSQL database. While not classified as NewSQL, Kvrocks aims to strike a balance between NoSQL and NewSQL paradigms: It aims to maintain the high performance of NoSQL while also implementing transaction guarantees and supporting more complex queries.

RediSearch?​

RediSearch is a Redis module that enhances Redis with query, secondary indexing, and full-text search functionalities. While its Redis commands begin with FT. (i.e. full text), it goes beyond just full-text search.

In fact, it is Redis moving closer to SQL databases: RediSearch enables users to create structured schemas on existing Redis JSON or HASH data for index building. Its schema supports various field types such as numeric, tag, geo, text, and vector - the latter two are utilized for full-text and vector searches. Instead of SQL support, RediSearch provides a unique query syntax known as the RediSearch query language.

RediSearch finds applications in various fields. One recent application involves utilizing its vector search feature to develop retrieval-augmented generation (RAG). For instance, LangChain utilizes Redis as one of its vector database. If Kvrocks can be compatible with RediSearch, it could benefit from these ecosystem from RediSearch.

SQL?​

RediSearch uses a unique syntax for queries, but there are some issues to consider:

Firstly, RediSearch's schema (a.k.a. index, created with FT.CREATE) can be regarded as a table in an SQL database. Its query syntax also aligns semantically with SQL queries. Given this similarity, supporting SQL should not increase significant challenges; why not include SQL support as well?

Secondly, SQL enjoys broader usage and is familiar to more individuals. It is simpler to learn at the syntax level. While developers may need time to understand RediSearch query syntax, adapting to a new SQL database often requires less effort. Furthermore, SQL offers robust support for various query features, enhanced expressive capabilities (like JOINs, subqueries, aggregations).

Finally, RediSearch query syntax suffers from some historical designs. For example, the operator precedence of AND and OR (represented by space and | operator in RediSearch queries) varies across different dialect versions (dialect 1 vs. dialect 2). This tribal knowledge might lead users to prefer established query languages.

To sum up, we believe that supporting SQL as a querying language would be a good decision.

How we support both?​

To introduce SQL capability to Kvrocks, we need to design a robust architecture with scalability, maintainability, and strong query planning and optimization features.

We plan to accomplish this through KQIR. In the context of Kvrocks, KQIR stands for both:

1. The complete query engine, covering frontend language parsing, query optimization and execution, etc.
2. An intermediate language (IR) that traverses the entire query engine.

KQIR: a multiple-level IR​

To support both SQL and RediSearch queries, an intermediate language (IR) is necessary to handle them consistently in subsequent processes without concern for the user's input language.

We have developed parsers for a subset of MySQL syntax and RediSearch queries, converting the resulting syntax tree into KQIR.

And KQIR is a multi-level IR that can represent query structures at various levels during optimization. The initial transformation from the syntax tree results in Syntactical IR, a high-level representation of certain syntactic expressions. As it undergoes processing by an IR optimizer, KQIR evolves into Planning IR, a low-level representation used to express query execution plans within the query engine.

Additionally, we will conduct semantic checks on the IR before optimization to ensure that the query is semantically correct. This includes verifying that it does not include any undefined schemas or fields and uses the appropriate field types.

IR Optimizer​

The KQIR optimizer consists of multiple passes, a concept borrowed from LLVM. Each pass takes IR as input, conducts analysis and modifications, and generates a new IR.

Currently, the optimizer's passes are categorized into three main groups:

• expression passes for optimizing logical expressions like AND, OR, NOT operators;
• numeric passes for optimizing numerical comparisons with an interval analysis (i.e. analyze the mathematical properties of numerical comparisons in terms of intervals) to enhance query optimization by eliminating unnecessary comparisons or improving comparison expressions;
• planning passes for converting syntactical IR to planning IR and enhancing query plans through a cost model that selects optimal indexes and removes unnecessary sortings.

Pass execution order is controlled by the pass manager. A pass may run multiple times at different stages to simplify individual passes by combining them.

Plan Executor​

The KQIR plan executor is built on the Volcano model.

Once the IR optimizer finishes all optimizations, the resulting IR becomes a planning IR. This will then be passed to the plan executor to create execution logic based on certain context corresponding to the plan operator.

Subsequently, Kvrocks retrieves query results through iterative execution.

On-disk indexing​

Unlike Redis, which stores index data in memory, Kvrocks requires the construction of indexes on the disk. This means that for any field type (e.g. tag, numeric), we need an encoding to reduce such index to RocksDB's key-values.

Furthermore, we incrementally create indexes before and after JSON or HASH commands getting executed to guarantee that query results are in real-time.

Current status​

The KQIR functionality is currently available on the unstable branch, supporting commands like FT.CREATE, FT.SEARCH, and FT.SEARCHSQL (an extension for running SQL queries) to encourage user to test.

However, as KQIR is still in early development, compatibility cannot be guaranteed and many features remain incomplete. Thus the upcoming release (version 2.9.0) will not include any KQIR component.

Supported field types​

Currently, we only support two field types: tag and numeric.

Tag fields label each data record with multiple tags for filtering in queries. And numeric fields hold numerical data within double-precision floating-point ranges. They allow sorting and filtering by specific numerical ranges.

In the future, we plan to expand support to include vector search and full-text search capabilities alongside other field types.

Transaction guarantees​

Currently, the transaction guarantee of KQIR is weak, which may lead to unexpected issues during use.

Another project in the Kvrocks community aims to enhance Kvrocks' transaction support by establishing a structured framework. We will leverage these efforts to uphold the ACID properties of KQIR and release an official version incorporating KQIR after that.

Limitation on IR optimizer​

Currently, KQIR does not use the cost model when optimizing record sorting. Instead, it relies on specialized logic. This could be an area for improvement soon.

Furthermore, KQIR does not currently utilize optimizations based on runtime statistics. Our future focus will be on integrating runtime statistics into the cost model for more precise index selection.

Relationship with other features​

KQIR integrates well with the namespace feature. Any index created is restricted to the current namespace and cannot be accessed in other namespaces, aligning with how other data is accessed within the namespace.

Currently, KQIR cannot be enabled in the cluster mode. Cluster mode support may not be implemented in the short term, but we encourage anyone to submit discussions, design proposals, or suggestions.

Compliance​

While KQIR is designed to be compatible with RediSearch at the interface level, it does not include any code from RediSearch. As previously mentioned, KQIR features a completely new framework, and its query architecture (including parsing, optimization, execution) is independent of RediSearch.

This distinction is important due to the proprietary license under which RediSearch is released.

High experimental!​

The current implementation of KQIR is in its early experimental stage. We advise users to consider carefully when using KQIR functionalities in a production environment, as we do not guarantee compatibility, and there may be unexpected errors.

Future outlook​

KQIR is currently in development, and all mentioned aspects will continue to evolve. If you're interested, please stay updated on the progress.

Developers keen on KQIR are encouraged to get involved in the development process and join the Apache Kvrocks community.

Note that our community consists entirely of volunteers. As an ASF community, we strive to offer an open, inclusive, and vendor-neutral environment.

Vector search​

The design and implementation of vector search support are currently underway, which is very exciting.

In the Kvrocks community, some members have raised discussions and proposed an encoding design for implementing vector search on KQIR.

As per the plan, we will initially implement an on-disk HNSW index and introduce the vector field type.

Full-text search​

There is currently no design proposal for full-text search.

However, community members are exploring the potential of incorporating full-text indexing in KQIR via CLucene or PISA.

We encourage anyone interested to share their ideas or suggestions and get involved in the development and implementation.

More SQL features​

In the future, we aim to progressively broaden our support for SQL features, potentially encompassing subqueries (including common table expressions), JOIN operations, aggregation functions, and other functionalities.

Our primary focus will remain on transaction processing rather than analytical tasks.

Try it!​

First, we can easily set up a Kvrocks instance via Docker images. You also have the choice to manually build executable from the source code in the unstable branch.

docker run -it -p 6666:6666 apache/kvrocks:nightly --log-dir stdout

Then, we can connect to kvrocks locally using redis-cli, and create an index named testidx consisting a tag field a and numeric field b with the following command:

FT.CREATE testidx ON JSON PREFIX 1 'test:' SCHEMA a TAG b NUMERIC

Next, we can add some new data using Redis JSON commands: (Note that it is also possible to add data before running FT.CREATE.)

JSON.SET test:k1 $ '{"a": "x,y", "b": 11}'
JSON.SET test:k2 $ '{"a": "y,z", "b": 22}'
JSON.SET test:k3 $ '{"a": "x,z", "b": 33}'

Finally, we can execute some SQL queries to get the desired results:

FT.SEARCHSQL 'select * from testidx where a hastag "z" and b < 30'

Or an equivalent RediSearch query:

FT.SEARCH testidx '@a:{z} @b:[-inf (30]'

If you experience any issues with KQIR, please feel free to report them.

Enjoy it!

Before we start, we would like to thanks all the contributors and users who have been supporting Apache Kvrocks. We can't make it without you!

Project Retrospective​

We collected a lot of feedback from users and developers in the discussion: 2023 Planning(Want your feedback), some of them have been resolved in 2023, and some of them are still in the plan list.

The key progresses in 2023:

• JSON data structure
• BloomFilter data structure
• Watch/Unwatch command
• Lua Functions
• Replication with TLS
• Allow to enable 64 bit expiration time (precision is in milliseconds instead of seconds), and the length of List/Hash/Set/ZSET/Stream can be also extended from 32 to 64 bits.
• Allow to use LOAD RDB command to load Redis RDB file
• Allow to dynamically adjust the number of Worker threads
• Enable Namespace replication
• Support RESTORE command

What we are still working on:

• RESP3 protocol, more information can be found in Tracking issue: Support RESP3 in Kvrocks
• Kvrocks in Kubernetes, kvrocks-operator is ready to test now, but still need more work to make it production-ready
• Controller to manage cluster, the API part of kvrocks-controller is finished, but we belive it's not enough of users before UI part is ready

What still in the backlog:

• Allow to use the Raft protocol for replication
• HyperLogLog
• RedisGraph, no plan to support since it's deprecated by the Redis community
• Semi-sync replication
• Client-side caching

Community​

In the past year, in addition to operating based on the principle of "Community over Code" of the Apache Foundation, the community maintenance also adheres to the attitude of openness, transparency, openness, inclusiveness, freedom and equality towards everyone.

In 2023, the community has many things worth sharing:

• Successfully graduated from the incubator to become an Apache top-level project in June
• Released a total of 6 versions from 2.3.0 - 2.7.0
• Voted for 5 new Committers, they are: xiaobiaozhao / mwish / Aleks Lozovyuk / binbin / yangshixi
• The total number of contributors exceeded 100+, which increased by 41 compared to 2022
• Merged PR total number 500+
• In terms of community communication groups, the number of WeChat groups is 450+, and the number of Slack is 250+

New Logo​

The number of users in the community has increased by 7 organizations, including Opera / iFlytek / SHOPLAZZA / Netease Hangzhou Research Institute / ZTO Express / Coin Index and AHOY.

Thanks to the above users for their use and active feedback, and welcome more users to leave your usage scenarios in GitHub Issue: Who is using Kvrocks?

2024 Planning​

In the past few years, the annual plan of Kvrocks has also maintained the principle of openness and transparency, and I hope to hear the voices of users and developers. Therefore, the community is open to collect needs and feedback from everyone.

Please don't hesitate to leave comments in 2024 Planning(Want your feedback) if you have any ideas or suggestions.

Kvrocks Overview & Advantages​

Kvrocks is a distributed key-value NoSQL database that uses RocksDB as its storage engine and supports the Redis protocol. Compared to Redis, Kvrocks allows users to decrease memory costs and increase capacity.

Kvrocks Feature Highlights​

• Redis Compatible: Support common Redis data types and commands;
• Namespace: Similar to Redis DB but equipped with a token per namespace;
• Replication: Async replication using WAL of RocksDB;
• High Availability: Support Redis sentinel to failover when master failed; and
• Cluster: Centralized management but accessible via any Redis cluster client.

ADDITIONAL RESOURCES​

• GitHub: https://github.com/apache/kvrocks
• Download: https://kvrocks.apache.org/download
• Docs: https://kvrocks.apache.org/
• Contribute: https://kvrocks.apache.org/community/contributing

Since being open-sourced in 2019, Kvrocks has been serving as an alternative for Redis in massive data scenarios. Many companies are deploying and using Kvrocks in the production environment, such as Baidu, Circl.lu, Ctrip, Meitu, Opera, U-Next and Xueqiu, among others.

To serve users better, Kvrocks plans to add Kubernetes deployment support, the controller to make the cluster easier to maintain and operate, and add more data structures for fulfilling the user requirements.

Incubation processes​

Kvrocks joined the Apache Incubator at the end of April 2022 and officially became an Apache top-level project in June 2023. There has been a significant increase in community activity

1. The number of contributors has increased from 27 to 82
2. Released 4 minor versions during incubation: 2.1.0 / 2.2.0 / 2.3.0 / 2.4.0
3. Created 900+ Pull Requests
4. Created 300+ Issues
5. Nominated 4 Committers: PragmaTwice / Torwig / Ruixiang Tan / Xiaobiao Zhao, and PragmaTwice and Torwig are also PMC members now.

Our Users​

It's worth noting that the majority of Kvrocks community contributors and committers are from the community users. In addition to using, they also help make the community better in their own way. Huge thanks to every user, contributor and committer.

Mentor & PMC Words​

@Liang Chen (Champion, Apache Member, Apache Incubator Mentor): Congratulations to Kvrocks for becoming an ASF Top Project. Kvrocks community has become an outstanding NoSQL DB open source project in the Big Data ecosystem after more than 1 year of hard work and incubation in accordance with The Apache Way.

@Xiaoqiao He (Mentor, Apache Member, Apache Incubator Mentor): Congratulations to Kvrocks on its successful graduation from the Apache Incubator. It was a pleasure to participate and witness Kvrocks successfully apply The Apache Way to community operations and project evolution after more than a year of incubation run and graduation. Graduation means a new beginning, I wish Kvrocks continues to build and prosper the data ecosystem, and look forward to Kvrocks creating more value.

@tison (Mentor, Apache Member, Apache Incubator Mentor): Congratulations to Kvrocks, my first mentor-incubated project, for graduating from the incubator. I'm excited to help the Kvrocks community grow under the guidance of The Apache Way, and to see how quickly the diversity of Kvrocks users and developers has grown over the past year. Graduating from the incubator is a small step for Kvrocks, but also a big step for all community members. Thank you and congratulations to everyone who has contributed to the development of Kvrocks!

@Yaroslav Stepanchuk (Kvrocks PMC Member): It has been approximately a year since I first joined the Kvrocks community. From the very beginning, I was welcomed into a supportive environment, which made it incredibly easy for me, a complete newcomer to open-source, to make my initial contributions. Witnessing the project's journey and observing its current state fills me with pride. The fact that Kvrocks has successfully graduated from the Apache Incubator is a remarkable accomplishment that can be credited to all the contributors and users involved in the project. Reflecting on this milestone, I am truly amazed by the power of open-source collaboration. I firmly believe that the project's graduation will provide an additional boost, serving as a source of inspiration and leading to the expansion of our already exceptional community.

@Hulk Lin (Kvrocks PMC Member): As one of the community maintainers, I am very happy that Kvrocks has become a top project in Apache after more than a year of incubation, and I am honored to work with many excellent contributors from home and abroad to maintain the project and the community. Graduation is just a new starting point, and I'm looking forward to having more great people join us.

@Yuan Wang (Kvrocks PMC Member): We're excited to graduate from the incubator and are happy to be working on Kvrocks with great contributors in the community, and we'll continue to pay attention to the feedback from the community to polish and optimize the product to make Kvrocks more stable and easy to use.

@PragmaTwice (Kvrocks PMC Member): The ASF community's volunteer model and the community over code credo have helped me understand more about open source and make me feel very comfortable in the community. Coincidentally, Kvrocks graduated almost at the same time as me, so congratulations to me and the community!

@DongHui Liu (Kvrocks PMC Member): We are happy to witness the whole journey of Kvrocks from the initial open source to the incubation in Apache to the successful graduation, and we are honored to maintain the Kvrocks project with the excellent contributors in the community. Graduation is an important milestone and the beginning of a new journey, we will continue to focus on our product capabilities and make Kvrocks better for the community. Good luck with Kvrocks!

Thanks​

• Thanks to the great efforts of every contributor, to make it possible for Kvrocks becoming a top-level project;
• Thank you to every Release Manager and to those who have helped Kvrocks release process. It is a great opportunity for community members to learn and practice the Apache Way from the release process;
• Thanks to the incubator mentors for their guidance and help in helping community members understand Apache's philosophy and code of conduct. In addition to being heavily involved in code development and community building;
• Finally, thank you to all users, your feedback and suggestions have had a significant impact on the development of Kvrocks, and this is what keeps the community going!

The End​

Apache Kvrocks has significant potential for further development in terms of community influence and feature iteration. We work on maintaining an open and friendly atmosphere based on The Apache Way and attracting more excellent contributors. In the future, we plan to support more data types and optimize the experience of using Kvrocks in container environments. Additionally, we plan to simplify the operation and management of clusters.

What is Kvrocks?​

Apache Kvrocks is a distributed key-value NoSQL database that uses RocksDB as a storage engine and is compatible with Redis protocol.

You can use Kvrocks as a drop-in replacement for Redis to store data on SSD, reducing storage costs and increasing capacity. For example, imagine taking one of the many existing Redis-based job queues and using them with Kvrocks and SSD storage.

Kvrocks supports most Redis commands including the watch command starting from v2.4.0. Kvrocks Cluster and replication are available as well.

Getting started with Kvrocks​

You can launch Kvrocks using Docker:

docker run -it -p 6666:6666 apache/kvrocks

And start using it right away:

redis-cli -p 6666

127.0.0.1:6666> get foo
(nil)
127.0.0.1:6666> set foo bar
OK
127.0.0.1:6666> get foo
"bar"

You can also build Kvrocks with GCC yourself.

Connecting to Kvrocks from Go​

Since Kvrocks uses Redis-compatible protocol, you can use your favorite Redis client to work with Kvrocks, for example, Go Redis client:

package main

import (
    "context"
    "github.com/redis/go-redis/v9"
)

func main() {
    ctx := context.Background()

    rdb := redis.NewClient(&redis.Options{
        Addr: "localhost:6666",
    })

    err := rdb.Set(ctx, "key", "value", 0).Err()
    if err != nil {
        panic(err)
    }

    val, err := rdb.Get(ctx, "key").Result()
    if err != nil {
        panic(err)
    }
    fmt.Println("key", val)
}

Pipelines, pub/sub, and even Lua scripts are working as well:

var incrBy = redis.NewScript(`
local key = KEYS[1]
local change = ARGV[1]

local value = redis.call("GET", key)
if not value then
  value = 0
end

value = value + change
redis.call("SET", key, value)

return value
`)

You can then run the script like this:

keys := []string{"my_counter"}
values := []interface{}{+1}
num, err := incrBy.Run(ctx, rdb, keys, values...).Int()

Monitoring Kvrocks​

Monitoring Kvrocks performance is crucial to ensure optimal operation and identify any potential bottlenecks or issues.

OpenTelemetry provides instrumentation libraries and integrations for monitoring Kvrocks using its unified observability framework.

What is OpenTelemetry?​

OpenTelemetry is an open-source observability framework designed to standardize and simplify the collection, analysis, and export of telemetry data

OpenTelemetry allows developers to collect and export telemetry data in a vendor agnostic way. With OpenTelemetry, you can instrument your application once and then add or change vendors without changing the instrumentation, for example, here is a list DataDog competitors that support OpenTelemetry.

What is Uptrace?​

Uptrace is an OpenTelemetry APM that supports distributed tracing, metrics, and logs. You can use it to monitor applications and set up automatic alerts to receive notifications via email, Slack, Telegram, and more.

Uptrace stores telemetry data in ClickHouse database. ClickHouse is an open source column-oriented database management system that is designed to process large volumes of data in real-time and to provide fast analytics and reporting capabilities.

Monitoring Kvrocks client​

You can use OpenTelemetry and Uptrace together to monitor Kvrocks performance using the go-redis instrumentation:

import "github.com/redis/go-redis/extra/redisotel/v9"

rdb := redis.NewClient(&redis.Options{
    Addr: ":6666",
})

if err := redisotel.InstrumentTracing(rdb, redisotel.WithDBSystem("kvrocks")); err != nil {
    panic(err)
}
if err := redisotel.InstrumentMetrics(rdb, redisotel.WithDBSystem("kvrocks")); err != nil {
    panic(err)
}

Once the data reaches Uptrace, it will generate the following dashboard for you:

Because OpenTelemetry provides a vendor-agnostic approach, it allows you to choose an OpenTelemetry backend that best suits your requirements, for example, Jaeger or Zipkin.

Monitoring Kvrocks server​

You can also configure OpenTelemetry Redis receiver to monitor Kvrocks:

receivers:
  redis/kvrocks:
    endpoint: "kvrocks:6666"
    collection_interval: 10s

The receiver works by parsing the output of INFO command and produces a number of useful metrics:

See GitHub example for details.

You can also export the collected metrics to Prometheus using OpenTelemetry Prometheus bridge

Custom metrics​

Using OpenTelemetry Metrics API, you can even create custom Kvrocks metrics. For example, the following function parses used_disk_percent to create kvrocks.used_disk_percent metric:

var re = regexp.MustCompile(`used_disk_percent:\s(\d+)%`)

func monitorKvrocks(ctx context.Context, rdb *redis.Client) error {
    mp := global.MeterProvider()
    meter := mp.Meter("github.com/uptrace/uptrace/example/kvrocks")

    usedDiskPct, err := meter.AsyncFloat64().Gauge(
        "kvrocks.used_disk_percent",
        instrument.WithUnit("%"),
    )
    if err != nil {
        return err
    }

    return meter.RegisterCallback(
        []instrument.Asynchronous{
            usedDiskPct,
        },
        func(ctx context.Context) {
            pct, err := getUsedDiskPercent(ctx, rdb)
            if err != nil {
                otel.Handle(err)
            }
            usedDiskPct.Observe(ctx, pct, semconv.DBSystemKey.String("kvrocks"))
        },
    )
}

func getUsedDiskPercent(ctx context.Context, rdb *redis.Client) (float64, error) {
    info, err := rdb.Info(ctx, "keyspace").Result()
    if err != nil {
        return 0, err
    }

    m := re.FindStringSubmatch(info)
    if m == nil {
        return 0, errors.New("can't find used_disk_percent metric")
    }

    n, err := strconv.ParseInt(m[1], 10, 64)
    if err != nil {
        return 0, err
    }

    return float64(n) / 100, nil
}

The metric looks like this in Uptrace:

See OpenTelemetry Go Metrics API for details.

Useful links​

• Getting started with Kvrocks
• Golang Redis
• Get started with Uptrace

Background​

Let's have a look at how Kvrocks uses the RocksDB before introducing performance optimization. From the implementation side, Kvrocks would encode the Redis data structure into the key-values and write them into the different RocksDB's column families. There's five column family type in Kvrocks:

• Metadata Column Family: used to store the metadata(expired, size..) for complex data structures like Hash/Set/ZSet/List, also string key-value was stored in this column family
• Subkey Column Family: used to store key-values for complex data structures were mentioned before
• ZSetScore Column Family: only store the score of the sorted set
• PubSub Column Family: used to store and propagate pubsub messages between the master and replicas
• Propagated Column Family: used to propagate commands between the master and replicas

Also, Kvrocks uses the RocksDB WAL to implement the replication, for more detail can see:

• Kvrocks: An Open-Source Distributed Disk Key-Value Storage With Redis Protocol
• How to implement the Redis data structures on RocksDB

We can have a glance at the Kvrocks architecture from 10,000 feet view：

How to profile RocksDB​

Memtable Optimization​

Currently, Kvrocks was using the SkipList Memtable. Compared with the HashSkipList Memtable, it has better performance when searching across multiple prefixes and uses less memory. Kvrocks also enabled the whole_key_filtering the option which would create a bloom filter for the key in the memtable, it can reduce the number of comparisons, thereby reducing the CPU usage during point query. Related configuration:

metadata_opts.memtable_whole_key_filtering = true
metadata_opts.memtable_prefix_bloom_size_ratio = 0.1

Data Block Optimization​

Previously, Kvrocks used binary search when searching the data block, which may cause CPU cache miss and increase the CPU usage. As the point query was the most used scenario in the key-value service, so Kvrocks switched to the hash index to reduce the binary search comparisons. Official test data shows that this feature can reduce CPU utilization by 21.8% and increase throughput by 10%, but it will increase space usage by 4.6%. Compared with disk resources, CPU resources are more expensive. Under the trade-off, Kvrocks chose to enable the Hash index to improve the efficiency of point queries.

Related configuration:

BlockBasedTableOptions::data_block_index_type = DataBlockIndexType::kDataBlockBinaryAndHash
BlockBasedTableOptions::data_block_hash_table_util_ratio = 0.75

Filter/Index Block​

The old version of RocksDB used Bloom Filter of BlockBasedFilter type by default. The basic mechanism is to generate a Filter for every 2KB of Key-Value data, and finally form a Filter array. When searching, first check the Index Block, and for the Data Block that may have the Key, then use the corresponding Filter Block to determine whether the key exists or not.

The new version of RocksDB optimizes the original Filter mechanism by introducing Full Filter. Each SST has a Filter, which can check whether the Key exists or not in the SST and avoid reading the Index Block. However, in the scenario with a large key number in the SST, the Filter Block and Index Block will still be larger. For 256MB SST, the size of Index and Filter Block is about 0.5MB and 5MB, which is much larger than Data Block (usually 4–32KB). In the most ideal case, when the Index/Filter Block is completely stored in memory, it will only be read once per SST life cycle, but when it competes with the Data Block for the Block Cache, it is likely to be re-read from the disk due to being evicted. Do it many times, resulting in very serious read amplification.

Kvrocks' previous approach was to dynamically adjust the SST-related configuration so that the SST file will not be too large, thereby avoiding the Index/Filter Block from being too large. However, the problem with this mechanism is that when the amount of data is very large, too many SST files will take up more system resources and cause performance degradation. The new version of Kvrocks optimizes this and opens the related configuration of the Partitioned Block. The principle of the Partitioned Block is to add a secondary index to the Index/Filter Block. When reading the Index or Filter, the secondary index is first to read into the memory, and then Find the required partition Index Block according to the secondary index, and load it into the Block Cache.

The advantages of Partitioned Block are as follows:

• Increase the cache hit rate: Large Index/Filter Block will pollute the cache space. The large Block will be partitioned, allowing the Index/Filter Block to be loaded at a finer granularity, thereby effectively using the cache space
• Improve cache efficiency: The partition Index/Filter Block will become smaller, the lock competition in the Cache will be further reduced, and the efficiency under high concurrency will be improved
• Reduce IO utilization: When the cache Miss of the index/filter partition, only a small partition needs to be loaded from the disk. Compared with the Index/Filter Block that reads the entire SST file, this will make the load on the disk smaller

Related configuration:

format_version = 5
index_type = IndexType::kTwoLevelIndexSearch
BlockBasedTableOptions::partition_filters = true
cache_index_and_filter_blocks = true
pin_top_level_index_and_filter = true
cache_index_and_filter_blocks_with_high_priority = true
pin_l0_filter_and_index_blocks_in_cache = true
optimize_filters_for_memory = true

Data compression optimization​

RocksDB compresses the data when it's placed on the disk. We compared and tested different compression algorithms on Kvrocks and found that different compression algorithms have a great impact on performance, especially when CPU resources are tight, which will significantly increase latency.

The following figure shows the test data of compression speed and compression ratio of different compression algorithms:

In Kvrocks, compression is not set for the SST of the L0 and L1 layers, because these two layers have a small amount of data. Compressing the data at these levels cannot reduce a lot of disk space, but not compressing the data at these levels can save CPU. Each Compaction from L0 to L1 needs to access all files in L1. In addition, the range scan cannot use Bloom Filter, and it needs to find all files in L0. If you do not need to decompress when reading data in L0 and L1, and writing data in L0 and L1 do not need to be compressed, then these two frequent CPU-intensive operations will take up less CPU, compared to the disk space saved by compression, it is more profitable.

Considering the trade-off between compression speed and compression ratio, Kvrocks mainly chooses two algorithms, LZ4 and ZSTD. For other levels, LZ4 is used because the compression algorithm is faster and the compression ratio is higher. RocksDB officially recommends using LZ4. For scenes with large data volume and low QPS, the last layer will be set to ZSTD to further reduce storage space and reduce costs. The advantage of ZSTD is that the compression ratio is higher and the compression speed is faster.

Cache optimization​

For the simple data type (String type), the data is directly stored in Metadata CF, while for complex data types, only the metadata is stored in Metadata CF, and the actual data is stored in Subkey CF. Kvrocks previously allocated the same size of Block Cache to these two CFs by default. However, the online scene is complicated and the user's data type cannot be predicted, so it is not possible to allocate an appropriate Block Cache size to each CF in advance. If users use simple types and use complex types in different proportions, the Block Cache hit rate will decrease. Kvrocks shared the same large Block Cache to achieve a 30% improvement in the command rate of the Cache. In addition, Row Cache is also introduced to deal with the problem of hotkeys. RocksDB checks Row Cache first, then Block Cache. For scenes with hot spots, data will be stored in Row Cache first to further improve Cache utilization.

Key-Value Separation​

The LSM storage engine will store the Key and Value together. During the compaction process, both Key and Value will be rewritten. When the Value is large, it will cause serious write amplification problems. In response to this problem, the WiscKey paper proposed a Key-Value separation scheme. Based on this paper, the industry also realized the KV separation of LSM-type storage engines, such as RocksDB's BlobDB, PingCAP's Titan engine, Quantum engine used by Baidu's UNDB.

RocksDB 6.18 version re-implemented BlobDB (RocksDB's Key-Value separation scheme), integrated it into the main logic of RocksDB, and has been improving and optimizing BlobDB related features. Kvrocks introduced this feature in 2.0.5 to deal with scenarios with large values. Tests show that when Kvrocks turns on the KV separation switch, for the scenario where Value is 10KB, the write performance is increased by 2.5 times, and the read performance is not attenuated; the larger the value, the greater the write performance improvement, and the write performance is improved when the Value is 50KB. 5 times.

Related configuration:

ColumnFamilyOptions.enable_blob_files = config_->RocksDB.enable_blob_files;
min_blob_size = 4096
blob_file_size = 128M
blob_compression_type = lz4
enable_blob_garbage_collection = true
blob_garbage_collection_age_cutoff = 0.25
blob_garbage_collection_force_threshold = 0.8

Kvrocks Roadmap​

2021 is coming to an end, the related work of Kvrocks 2.0 has been basically completed, and the plan of Kvrocks 3.0 has also been listed on GitHub. This article lists the following two important features.

Why amplification occurs​

Amplification mainly comes from two aspects:

• The hardware-level requires the smallest reading and writing unit
• How we organize the data distribution on the software-level

Take SSD as an example, the smallest unit of reading/writing operation was the page (commonly 4KiB/8KiB/16KiB), and it would read or write one page even the request size was 1byte. Moreover, the way of SSD modification was Read-Modify-Write instead of in-place, which means SSD would read the page content and modify then write it out to another page, the old page would be reclaimed by GC. Similar to the following:

As we can see, a large random io was very unfriendly to the SSD disk, except for the performance issue, frequent erasing and writing will also seriously lead to the life of the SSD (random reads and writes are also unfriendly to HDDs, requiring constant seek and addressing). LSM-Tree alleviates such problems by changing random writes into sequential batch writes.

The read-write amplification at the software level mainly comes from the data organization method, and the degree of read-write amplification brought about by different organization methods will also vary greatly. Take RocksDB as an example here, RocksDB is Facebook based on Google LevelDB which enriches the multi-threading, Backup and Compaction, and many other very useful functions. To solve the problem of the disk write amplification, it also brings some space enlargement problems. Let's take a brief look at how LSM-Tree organizes data:

LSM-Tree would create a new entry per write. For example in the above picture, the variable X is written 4 times successively, which are 0, 1, 2, 3. From the single variable X side, it was caused 4 times space amplification, those old spaces would be reclaimed on the background compaction. Similarly, deletion is achieved by inserting a record whose value is empty. The space size of each layer of LSM-Tree increases layer by layer. When the capacity reaches the limit, it will trigger compaction to merge to the next layer, and so on. Assuming that the maximum storage size of Level 0 is M Bytes, it increases layer by layer in 10 times and the maximum is 7 layers. Theoretically, the space enlargement is about 1.111111 times. Calculated as follows:

amplification ratio = (1 + 10 + 100 +1000 + 10000 + 100000 + 1000000) * M / (1000000 * M)

However, the magnification space rate is much larger than this theoretical value since the last layer generally cannot reach the maximum value. It is also mentioned in the RocksDB documentation. For details see also RockDB's blog "Dynamic Level Size for Level-Based Compaction".

In addition, since RocksDB reads and writes are all based on key-value, the larger the value, the greater the read-write amplification may be. For example, suppose there is a JSON with a Value of 10 MiB. If you want to modify a field in this key, you need to read the entire JSON, modify and write it back again, which will cause huge read-write amplification. The paper "WiscKey: Separating Keys from Values in SSD-conscious Storage" that optimizes the large Key-Value of LSM-Tree by separating key-value to reduce the write amplification problem caused by Compaction. The titan project of TiKV is based on WiscKey paper to optimize RocksDB's write amplification in large key-value scenarios. RocksDB also implements this function in the community version, but it is still in an experimental stage.

Implement bitmap on RocksDB​

Kvrocks is disk storage compatible with the Redis protocol implemented on RocksDB. It needs to support the bitmap data structure, so needs to implement the bitmap on RocksDB. In most scenes, the bitmap is used as sparse arrays, which means the offset written should be random, for the first time maybe 1, and the next offset maybe 1000000000 or more. Therefore, the implementation will face the above-mentioned amplification issue.

A simple way is to regard the entire bitmap as a value, and read the value into the memory and then write it back when writing. Although this implementation is very simple, it would cause seriously amplification when the value was huge. In addition to the problem of effective space utilization, it may directly cause the entire service to be unavailable since we need to read and write back the entire value. Bitmap in Pika is such an implementation, but the maximum value is limit to 128 KiB. Limit the value size can avoid the above-mentioned extreme cases, but it will greatly affect the user scenes of bitmap.

Since we know that the core problem is caused by a single key-value that is too large, the most direct way is to split the bitmap into multiple key-values, and control the single key-value size within a reasonable range, so the amplification is relatively under control. In the current implementation of Kvrocks, each key-value is divided into 1 KiB(8192 bits). The algorithm diagram is as follows:

Take setbit foo 8192002 1 as an example, the implementation steps are:

• Calculate the key corresponding to the offset of 8192002, because Kvrocks uses a value of 1 KiB, so the number of the key is 8192002/(1024*8)=1000, so you can know the bit should be stored in the sub key foo1000.
• Then get the value corresponding to this key from RocksDB and calculate the offset in the segment, 8192002%8192 is equal to 2, and then set the bit with the offset of 2 to 1.
• Finally, write the entire value back to RocksDB.

A key point of this implementation is only read-write the limit part of the bitmap we need. Assuming that we have only executed setbit twice, setbit foo 1 1 and setbit foo 8192002 1, then there will only read and write two keys foo:0 and foo:1000 in RocksDB, and the actual read-value size is only 2 KiB in total. It can be perfectly adapted to the sparse array scene like the bitmap, and will also not cause the problem of space enlargement due to sparse writing.

This idea is also similar to Linux's virtual memory/physical memory mapping strategy. For example, we request to malloc for 1GiB, and the operating system only allocates a piece of virtual memory address space. The physical memory was allocated when it is actually written will it trigger a page fault interrupt. That is, if the memory page has not been written, read-only will not cause physical memory allocation.

GetBit is similar. It first calculates the key where the offset is located, and then reads the key from RocksDB.

• If not exist, means that segment has not been written, returns 0 directly.
• If exists, read the Value and return the value of the corresponding bit.

In addition, the actual key-value size is also determined by the largest offset currently written. It would NOT always create a 1024 KiB key-value when there is a write. This can also help to optimize the read-write amplification problem within a single key-value in some degree. You can read the source code of bitmap for more details.

Summary​

It can be seen that to achieve the same thing in memory and disk was entirely different, the challenges are completely different. For disk-type services, it MUST continuously optimize the random read and write and space amplification issues. Familiar with the software was not enough, also requires to understand the hardware internal.