# MinIO MemKV RELEASE.2026-10-04T22-42-25Z

A MinIO MemKV server can now serve on every RDMA NIC in the host, and clients place keys by node and drive each GPU's transfers through the NIC on its own PCIe switch. Upgrading re-homes the cache once on multi-node clusters unless `node_id` is set as described under Upgrade Instructions.

## Downloads

### linux-amd64

- [libmemkv_preload.so](https://dl.min.io/aistor/memkv/release/linux-amd64/archive/libmemkv_preload.so)
- [libmemkv_preload.so.RELEASE.2026-10-04T22-42-25Z](https://dl.min.io/aistor/memkv/release/linux-amd64/archive/libmemkv_preload.so.RELEASE.2026-10-04T22-42-25Z)
- [libplugin_MEMKV.so](https://dl.min.io/aistor/memkv/release/linux-amd64/archive/libplugin_MEMKV.so)
- [libplugin_MEMKV.so.RELEASE.2026-10-04T22-42-25Z](https://dl.min.io/aistor/memkv/release/linux-amd64/archive/libplugin_MEMKV.so.RELEASE.2026-10-04T22-42-25Z)
- [memkv](https://dl.min.io/aistor/memkv/release/linux-amd64/archive/memkv)
- [memkv.RELEASE.2026-10-04T22-42-25Z](https://dl.min.io/aistor/memkv/release/linux-amd64/archive/memkv.RELEASE.2026-10-04T22-42-25Z)
- [libmemkv_preload.so.RELEASE.2026-10-04T22-42-25Z.sha256sum](https://dl.min.io/aistor/memkv/release/linux-amd64/archive/libmemkv_preload.so.RELEASE.2026-10-04T22-42-25Z.sha256sum)
- [libmemkv_preload.so.sha256sum](https://dl.min.io/aistor/memkv/release/linux-amd64/archive/libmemkv_preload.so.sha256sum)
- [libplugin_MEMKV.so.RELEASE.2026-10-04T22-42-25Z.sha256sum](https://dl.min.io/aistor/memkv/release/linux-amd64/archive/libplugin_MEMKV.so.RELEASE.2026-10-04T22-42-25Z.sha256sum)
- [libplugin_MEMKV.so.sha256sum](https://dl.min.io/aistor/memkv/release/linux-amd64/archive/libplugin_MEMKV.so.sha256sum)
- [memkv-20261004224225.0.0-1.x86_64.rpm.sha256sum](https://dl.min.io/aistor/memkv/release/linux-amd64/archive/memkv-20261004224225.0.0-1.x86_64.rpm.sha256sum)
- [memkv.RELEASE.2026-10-04T22-42-25Z.sha256sum](https://dl.min.io/aistor/memkv/release/linux-amd64/archive/memkv.RELEASE.2026-10-04T22-42-25Z.sha256sum)
- [memkv.sha256sum](https://dl.min.io/aistor/memkv/release/linux-amd64/archive/memkv.sha256sum)
- [memkv_20261004224225.0.0_amd64.deb.sha256sum](https://dl.min.io/aistor/memkv/release/linux-amd64/archive/memkv_20261004224225.0.0_amd64.deb.sha256sum)
- [memkv_20261004224225.0.0_x86_64.apk.sha256sum](https://dl.min.io/aistor/memkv/release/linux-amd64/archive/memkv_20261004224225.0.0_x86_64.apk.sha256sum)
- [libmemkv_preload.so.RELEASE.2026-10-04T22-42-25Z.asc](https://dl.min.io/aistor/memkv/release/linux-amd64/archive/libmemkv_preload.so.RELEASE.2026-10-04T22-42-25Z.asc)
- [libmemkv_preload.so.RELEASE.2026-10-04T22-42-25Z.minisig](https://dl.min.io/aistor/memkv/release/linux-amd64/archive/libmemkv_preload.so.RELEASE.2026-10-04T22-42-25Z.minisig)
- [libmemkv_preload.so.asc](https://dl.min.io/aistor/memkv/release/linux-amd64/archive/libmemkv_preload.so.asc)
- [libmemkv_preload.so.minisig](https://dl.min.io/aistor/memkv/release/linux-amd64/archive/libmemkv_preload.so.minisig)
- [libplugin_MEMKV.so.RELEASE.2026-10-04T22-42-25Z.asc](https://dl.min.io/aistor/memkv/release/linux-amd64/archive/libplugin_MEMKV.so.RELEASE.2026-10-04T22-42-25Z.asc)
- [libplugin_MEMKV.so.RELEASE.2026-10-04T22-42-25Z.minisig](https://dl.min.io/aistor/memkv/release/linux-amd64/archive/libplugin_MEMKV.so.RELEASE.2026-10-04T22-42-25Z.minisig)
- [libplugin_MEMKV.so.asc](https://dl.min.io/aistor/memkv/release/linux-amd64/archive/libplugin_MEMKV.so.asc)
- [libplugin_MEMKV.so.minisig](https://dl.min.io/aistor/memkv/release/linux-amd64/archive/libplugin_MEMKV.so.minisig)
- [memkv.RELEASE.2026-10-04T22-42-25Z.asc](https://dl.min.io/aistor/memkv/release/linux-amd64/archive/memkv.RELEASE.2026-10-04T22-42-25Z.asc)
- [memkv.RELEASE.2026-10-04T22-42-25Z.minisig](https://dl.min.io/aistor/memkv/release/linux-amd64/archive/memkv.RELEASE.2026-10-04T22-42-25Z.minisig)
- [memkv.asc](https://dl.min.io/aistor/memkv/release/linux-amd64/archive/memkv.asc)
- [memkv.minisig](https://dl.min.io/aistor/memkv/release/linux-amd64/archive/memkv.minisig)
- [memkv-20261004224225.0.0-1.x86_64.rpm](https://dl.min.io/aistor/memkv/release/linux-amd64/archive/memkv-20261004224225.0.0-1.x86_64.rpm)
- [memkv.apk](https://dl.min.io/aistor/memkv/release/linux-amd64/archive/memkv.apk)
- [memkv.deb](https://dl.min.io/aistor/memkv/release/linux-amd64/archive/memkv.deb)
- [memkv.rpm](https://dl.min.io/aistor/memkv/release/linux-amd64/archive/memkv.rpm)
- [memkv_20261004224225.0.0_amd64.deb](https://dl.min.io/aistor/memkv/release/linux-amd64/archive/memkv_20261004224225.0.0_amd64.deb)
- [memkv_20261004224225.0.0_x86_64.apk](https://dl.min.io/aistor/memkv/release/linux-amd64/archive/memkv_20261004224225.0.0_x86_64.apk)

### linux-arm64

- [libmemkv_preload.so](https://dl.min.io/aistor/memkv/release/linux-arm64/archive/libmemkv_preload.so)
- [libmemkv_preload.so.RELEASE.2026-10-04T22-42-25Z](https://dl.min.io/aistor/memkv/release/linux-arm64/archive/libmemkv_preload.so.RELEASE.2026-10-04T22-42-25Z)
- [libplugin_MEMKV.so](https://dl.min.io/aistor/memkv/release/linux-arm64/archive/libplugin_MEMKV.so)
- [libplugin_MEMKV.so.RELEASE.2026-10-04T22-42-25Z](https://dl.min.io/aistor/memkv/release/linux-arm64/archive/libplugin_MEMKV.so.RELEASE.2026-10-04T22-42-25Z)
- [memkv](https://dl.min.io/aistor/memkv/release/linux-arm64/archive/memkv)
- [memkv.RELEASE.2026-10-04T22-42-25Z](https://dl.min.io/aistor/memkv/release/linux-arm64/archive/memkv.RELEASE.2026-10-04T22-42-25Z)
- [libmemkv_preload.so.RELEASE.2026-10-04T22-42-25Z.sha256sum](https://dl.min.io/aistor/memkv/release/linux-arm64/archive/libmemkv_preload.so.RELEASE.2026-10-04T22-42-25Z.sha256sum)
- [libmemkv_preload.so.sha256sum](https://dl.min.io/aistor/memkv/release/linux-arm64/archive/libmemkv_preload.so.sha256sum)
- [libplugin_MEMKV.so.RELEASE.2026-10-04T22-42-25Z.sha256sum](https://dl.min.io/aistor/memkv/release/linux-arm64/archive/libplugin_MEMKV.so.RELEASE.2026-10-04T22-42-25Z.sha256sum)
- [libplugin_MEMKV.so.sha256sum](https://dl.min.io/aistor/memkv/release/linux-arm64/archive/libplugin_MEMKV.so.sha256sum)
- [memkv-20261004224225.0.0-1.aarch64.rpm.sha256sum](https://dl.min.io/aistor/memkv/release/linux-arm64/archive/memkv-20261004224225.0.0-1.aarch64.rpm.sha256sum)
- [memkv.RELEASE.2026-10-04T22-42-25Z.sha256sum](https://dl.min.io/aistor/memkv/release/linux-arm64/archive/memkv.RELEASE.2026-10-04T22-42-25Z.sha256sum)
- [memkv.sha256sum](https://dl.min.io/aistor/memkv/release/linux-arm64/archive/memkv.sha256sum)
- [memkv_20261004224225.0.0_aarch64.apk.sha256sum](https://dl.min.io/aistor/memkv/release/linux-arm64/archive/memkv_20261004224225.0.0_aarch64.apk.sha256sum)
- [memkv_20261004224225.0.0_arm64.deb.sha256sum](https://dl.min.io/aistor/memkv/release/linux-arm64/archive/memkv_20261004224225.0.0_arm64.deb.sha256sum)
- [libmemkv_preload.so.RELEASE.2026-10-04T22-42-25Z.asc](https://dl.min.io/aistor/memkv/release/linux-arm64/archive/libmemkv_preload.so.RELEASE.2026-10-04T22-42-25Z.asc)
- [libmemkv_preload.so.RELEASE.2026-10-04T22-42-25Z.minisig](https://dl.min.io/aistor/memkv/release/linux-arm64/archive/libmemkv_preload.so.RELEASE.2026-10-04T22-42-25Z.minisig)
- [libmemkv_preload.so.asc](https://dl.min.io/aistor/memkv/release/linux-arm64/archive/libmemkv_preload.so.asc)
- [libmemkv_preload.so.minisig](https://dl.min.io/aistor/memkv/release/linux-arm64/archive/libmemkv_preload.so.minisig)
- [libplugin_MEMKV.so.RELEASE.2026-10-04T22-42-25Z.asc](https://dl.min.io/aistor/memkv/release/linux-arm64/archive/libplugin_MEMKV.so.RELEASE.2026-10-04T22-42-25Z.asc)
- [libplugin_MEMKV.so.RELEASE.2026-10-04T22-42-25Z.minisig](https://dl.min.io/aistor/memkv/release/linux-arm64/archive/libplugin_MEMKV.so.RELEASE.2026-10-04T22-42-25Z.minisig)
- [libplugin_MEMKV.so.asc](https://dl.min.io/aistor/memkv/release/linux-arm64/archive/libplugin_MEMKV.so.asc)
- [libplugin_MEMKV.so.minisig](https://dl.min.io/aistor/memkv/release/linux-arm64/archive/libplugin_MEMKV.so.minisig)
- [memkv.RELEASE.2026-10-04T22-42-25Z.asc](https://dl.min.io/aistor/memkv/release/linux-arm64/archive/memkv.RELEASE.2026-10-04T22-42-25Z.asc)
- [memkv.RELEASE.2026-10-04T22-42-25Z.minisig](https://dl.min.io/aistor/memkv/release/linux-arm64/archive/memkv.RELEASE.2026-10-04T22-42-25Z.minisig)
- [memkv.asc](https://dl.min.io/aistor/memkv/release/linux-arm64/archive/memkv.asc)
- [memkv.minisig](https://dl.min.io/aistor/memkv/release/linux-arm64/archive/memkv.minisig)
- [memkv-20261004224225.0.0-1.aarch64.rpm](https://dl.min.io/aistor/memkv/release/linux-arm64/archive/memkv-20261004224225.0.0-1.aarch64.rpm)
- [memkv.apk](https://dl.min.io/aistor/memkv/release/linux-arm64/archive/memkv.apk)
- [memkv.deb](https://dl.min.io/aistor/memkv/release/linux-arm64/archive/memkv.deb)
- [memkv.rpm](https://dl.min.io/aistor/memkv/release/linux-arm64/archive/memkv.rpm)
- [memkv_20261004224225.0.0_aarch64.apk](https://dl.min.io/aistor/memkv/release/linux-arm64/archive/memkv_20261004224225.0.0_aarch64.apk)
- [memkv_20261004224225.0.0_arm64.deb](https://dl.min.io/aistor/memkv/release/linux-arm64/archive/memkv_20261004224225.0.0_arm64.deb)

### Container Images

- `quay.io/minio/memkv:RELEASE.2026-10-04T22-42-25Z`
- `quay.io/minio/memkv:latest`

### Python Plugins

```bash
pip install -U memkv-lmcache memkv-sglang memkv-vllm
```

PyPI serves **1.0.10** for amd64 and arm64 (manylinux_2_28). Use it with this release. `memkv-lmcache` requires Python 3.10–3.13 and `lmcache>=0.5.4,<0.6`, `memkv-vllm` requires Python 3.10+, and `memkv-sglang` requires Python 3.8+.

1.0.10 fixes problems in the plugins packaged with this release:

- On a cluster with more than one server, batch lookups and batch transfers could report stored keys as missing. Single reads and writes were not affected.
- vLLM could report a block as cached after one rank's part of it had been evicted, or after the server evicted it, and then fail the transfer. In vLLM's direct offload path, a failed transfer stops the engine.
- vLLM rewrote blocks that were already stored.

It also adds `memkv_lookup_hit_ttl` (default 30 seconds): how long the vLLM plugin trusts a cached hit before checking the servers again. If your environment sets `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True`, start vLLM with `--enable-cumem-allocator`; vLLM refuses to start its offloading connector otherwise.

The deb/rpm packages install the plugins under `/usr/share/memkv/wheels/`, stamped **1.0.8**, and do not include these fixes. Install 1.0.10 from PyPI instead. For hosts without network access, copy the 1.0.10 wheels from PyPI.

## New Features

- **Serve on every RDMA NIC.** The new server setting `rdma.devices` lists the NICs to serve on, one rail each. A client on any rail of a rail-optimised fabric is then answered on its own rail, without crossing the CPU root complex. The rails share one buffer pool. `rdma.device` stays as the single-NIC setting. (#93)
- **Node identity for placement.** The new server setting `network.node_id` names the node; it defaults to the hostname. Clients place keys by node rather than by address, so a node reachable on several addresses is one placement target. (#93)
- **Each vLLM rank picks its own NIC.** At startup, the vLLM plugin reads the PCIe topology and drives each rank's RDMA through the NIC on its GPU's PCIe switch that reaches a configured server. `MEMKV_RDMA_DEVICES` still narrows the candidates. (#93)
- **`memkv_scratch_medium: auto`** (the new vLLM plugin default). It stages in GPU memory when every GPU on the host has a NIC on its own PCIe switch, and in pinned host memory otherwise. (#93)
- **`memkv net topo`** prints how far each RDMA NIC is from each GPU, and names the NIC each GPU should use. (#93)
- **Per-rail status.** `GET /v1/status` now reports `rdma.rails`: each rail's device, addresses and active connections. A rail that failed to come up does not appear. (#93)

## Bug Fixes

### Client

- A client with several RDMA NICs could stall when `servers` and `rdma_devices` were not listed in matching order. Each server is now reached through the NIC whose subnet contains its address, whatever the list order. (#93)

### Python Plugins

- The 1.0.10 plugins on PyPI fix batch lookups that missed stored keys on multi-server clusters, and vLLM lookups that reported evicted blocks as cached. Use 1.0.10 rather than the 1.0.8 wheels packaged with this release. (#96, #99)

## Placement and Default Changes

- **The cache re-homes once on multi-node clusters.** Clients used to place keys by the server's address; they now use its `node_id`. The first time clients restart against upgraded servers, most keys move to a new owner. No data is lost, but the cache stays cold until it refills. To avoid this, see Upgrade Instructions. (#93)
- **`memkv_scratch_medium` defaults to `auto` instead of `host`.** On hosts where every GPU has a NIC on its own PCIe switch, ranks now stage in GPU memory instead of pinned host memory. Set it to `host` to keep the previous memory footprint. (#93)
- **An empty `bind_addresses` entry (`~`) now means the matched NIC's address, not any address.** It defaults to the address of the NIC matched to that server, which keeps the control connection on the same rail as the data. (#93)

## Security & Compliance

Software bill of materials for this release, attached to the release and published alongside these notes:

- `sbom-RELEASE.2026-10-04T22-42-25Z.spdx.json`
- `sbom-RELEASE.2026-10-04T22-42-25Z.cyclonedx.json`
- `cargo-tree-RELEASE.2026-10-04T22-42-25Z.txt`
- `crates-RELEASE.2026-10-04T22-42-25Z.txt`

## Upgrade Instructions

1. **Keep placement unchanged (optional).** On each server, set `network.node_id` to the address clients already list for that node in `MEMKV_SERVERS`, for example:

   ```yaml
   network:
     address: 0.0.0.0:9900
     node_id: 10.0.0.1:9900
   ```

   With that set, keys keep their current owners. Without it, expect one cache refill after the upgrade.

2. **Upgrade servers first, then restart clients.** A client learns node identities once, at startup. During a rolling upgrade, clients started before and after a server upgrades disagree about where that server's keys live. That costs cache misses, not correctness. Mixed versions are otherwise compatible in both directions.

3. **Upgrade every client together** unless you set `node_id` as in step 1. Older clients, including PyPI 1.0.8 and earlier, still place keys by address. Running them next to upgraded clients means each group misses the other's keys. Install the 1.0.10 plugins from PyPI.

4. **To serve on several NICs**, set `rdma.devices`, keep `network.address` on the wildcard address (`0.0.0.0:9900`), and give each NIC an address on its own rail's subnet. The startup log names each rail and warns if either condition is not met.

5. **To keep staging in host memory** on hosts with a NIC per GPU, set `memkv_scratch_medium` to `host` in `kv_connector_extra_config`.

Documentation: <https://docs.min.io/memkv/> (`/operate/configuration` and `/operate/cli`). Licensed customers: contact MinIO support through SUBNET. Security disclosures: security@min.io.