Notational Conventions

The keywords “MUST”, “MUST NOT”, “REQUIRED”, “SHALL”, “SHALL NOT”, “SHOULD”, “SHOULD NOT”, “RECOMMENDED”, “NOT RECOMMENDED”, “MAY”, and “OPTIONAL” are to be interpreted as described in RFC 2119.

Terminology

Term Definition
VolumeID The identifier of the volume generated by the plugin.
CO Container Orchestration system that communicates with plugins using CSI service RPCs.
SP Storage Provider, the vendor of a CSI plugin implementation.
RPC Remote Procedure Call.

Objective

Define an extension to the CSI Specification that will enable storage vendors (SP) to develop controllers/plugins that can free unused storage allocations from existing volumes.

Goals in MVP

The new extension will define a procedure that

Non-Goals in MVP

Solution Overview

This specification defines an interface along with the minimum operational and packaging recommendations for a storage provider (SP) to implement a space reclaim operation for volumes. The interface declares the RPCs that a plugin MUST expose.

Architecture

A user instructs the CSI-Addons CO plugin to reclaim unused space from a volume. The CO plugin inspects the capabilities advertised by the CSI-driver (see the Identity ReclaimSpace capability) and translates the request into a gRPC call that is relayed to the CSI-driver. The operation runs either on the CSI Controller (OFFLINE, via ControllerReclaimSpace) or on the CSI NodePlugin (ONLINE, via NodeReclaimSpace).

Error Handling

All RPCs defined in this specification return a standard gRPC status. The status code MUST be a canonical gRPC error code; the CO MUST handle all valid error codes. When the code is not OK, the status message MUST contain a human-readable description of the error, which MAY be surfaced by the CO to end users. The status details MUST be empty.

Each RPC below documents, in its Errors table, the gRPC codes the CSI-driver returns for the listed conditions and the recovery behavior the caller is expected to implement.

The following error codes are common to all RPCs in this specification. In addition to the conditions below, each RPC MAY define further codes in its own Errors section.

Condition gRPC Code Description Recovery Behavior
Missing required field 3 INVALID_ARGUMENT Indicates that a required field is missing from the request. Caller MUST fix the request by adding the missing required field before retrying.
Invalid or unsupported field in the request 3 INVALID_ARGUMENT Indicates that one or more fields in the request is either not allowed by the CSI-driver or has an invalid value. Caller MUST fix the field before retrying.
Operation pending 10 ABORTED Indicates that there is already an operation pending for the specified resource. The CSI-driver SHOULD handle this as gracefully as possible, and MAY return this error code to reject secondary calls. Caller SHOULD ensure that there are no other calls pending for the specified resource, and then retry with exponential back off.
Call not implemented 12 UNIMPLEMENTED The invoked RPC is not implemented by the CSI-driver or disabled in its current mode of operation. Caller MUST NOT retry.
Not authenticated 16 UNAUTHENTICATED The invoked RPC does not carry secrets that are valid for authentication. Caller SHALL either fix the secrets provided in the RPC, or otherwise regalvanize said secrets such that they pass authentication by the CSI-driver for the attempted RPC, after which point the caller MAY retry the attempted RPC.
Error is Unknown 2 UNKNOWN Indicates that an unknown error occurred. Caller MUST study the logs before retrying.

RPC Interface

syntax = "proto3";
package reclaimspace;

import "github.com/container-storage-interface/spec/lib/go/csi/csi.proto";
import "google/protobuf/descriptor.proto";

option go_package = "github.com/csi-addons/spec/lib/go/reclaimspace";

// ReclaimSpaceController holds the RPC method for running discard operations
// on the inactive (not required to be staged/published) volume.
service ReclaimSpaceController {
  // ControllerReclaimSpace is a procedure that gets called on the CSI
  // Controller.
  rpc ControllerReclaimSpace (ControllerReclaimSpaceRequest)
  returns (ControllerReclaimSpaceResponse) {}
}

// ReclaimSpaceNode holds the RPC method for running discard operations on the
// active (staged/published) volume.
service ReclaimSpaceNode {
  // NodeReclaimSpace is a procedure that gets called on the CSI NodePlugin.
  rpc NodeReclaimSpace (NodeReclaimSpaceRequest)
  returns (NodeReclaimSpaceResponse) {}
}

ControllerReclaimSpace

A Controller Plugin MUST implement this RPC call if it has the ReclaimSpace OFFLINE capability (see the Identity service). This RPC is called by the CSI-Addons CO plugin to free unused storage allocations of a volume that does not need to be staged/published on a node. This operation MUST be idempotent. On success the CSI-driver replies 0 OK. The amount of space actually reclaimed is storage-specific and MAY be zero; the optional pre_usage and post_usage allow the CO to observe the effect. Because reclaiming space does not change the logical content of the volume, reclaiming a volume that has no unused space to free MUST still succeed. If the operation fails, or the CO does not know whether it succeeded, it MAY call ControllerReclaimSpace again.

// ControllerReclaimSpaceRequest contains the information needed to identify
// the volume by the SP and access any backend services so that space can be
// reclaimed.
message ControllerReclaimSpaceRequest {
  // The ID of the volume. This field is REQUIRED.
  string volume_id = 1;

  // Plugin specific parameters passed in as opaque key-value pairs.
  map<string, string> parameters = 2;
  // Secrets required by the plugin to complete the request.
  map<string, string> secrets = 3 [(csi.v1.csi_secret) = true];
}

// ControllerReclaimSpaceResponse holds the information about the result of the
// ControllerReclaimSpaceRequest call.
message ControllerReclaimSpaceResponse {
  // This field is OPTIONAL. This allows the SP to inform the CO about the
  // storage consumption before the ReclaimSpace operation was executed.
  StorageConsumption pre_usage = 1;

  // This field is OPTIONAL. This allows the SP to inform the CO about the
  // storage consumption after the ReclaimSpace operation was executed.
  StorageConsumption post_usage = 2;
}

// StorageConsumption contains the usage in bytes.
message StorageConsumption {
  // This field is REQUIRED. usage_bytes contains the consumed storage in
  // bytes.
  int64 usage_bytes = 1;
}

ControllerReclaimSpace Errors

If the CSI-driver is unable to complete the ControllerReclaimSpace call successfully, it MUST return a non-ok gRPC code in the gRPC status. If the conditions defined below are encountered, the CSI-driver MUST return the specified gRPC error code. The caller MUST implement the specified error recovery behavior when it encounters the gRPC error code.

Condition gRPC Code Description Recovery Behavior
Volume does not exist 5 NOT_FOUND Indicates that a volume corresponding to the specified volume_id does not exist. Caller MUST verify that the volume_id is correct and that the volume is accessible and has not been deleted before retrying with exponential back off.
Operation not supported 12 UNIMPLEMENTED Indicates that the operation is not supported. The filesystem may not support fstrim, or the block-device may lack support for DISCARD operations. Caller MUST NOT retry the request as the underlying storage does not support space reclamation.

NodeReclaimSpace

A Node Plugin MUST implement this RPC call if it has the ReclaimSpace ONLINE capability (see the Identity service). This RPC is called by the CSI-Addons CO plugin on the node where the volume is staged/published, to free unused storage allocations of the active volume using the volume_path at which the volume is available. This operation MUST be idempotent. On success the CSI-driver replies 0 OK. The amount of space actually reclaimed is storage-specific and MAY be zero; the optional pre_usage and post_usage allow the CO to observe the effect. Reclaiming a volume that has no unused space to free MUST still succeed. If the operation fails, or the CO does not know whether it succeeded, it MAY call NodeReclaimSpace again.

// NodeReclaimSpaceRequest contains the information needed to identify the
// location where the volume is mounted so that local filesystem or
// block-device operations to reclaim space can be executed.
message NodeReclaimSpaceRequest {
  // The ID of the volume. This field is REQUIRED.
  string volume_id = 1;

  // The path on which volume is available. This field is REQUIRED.
  // This field overrides the general CSI size limit.
  // SP SHOULD support the maximum path length allowed by the operating
  // system/filesystem, but, at a minimum, SP MUST accept a max path
  // length of at least 128 bytes.
  string volume_path = 2;

  // The path where the volume is staged, if the plugin has the
  // STAGE_UNSTAGE_VOLUME capability, otherwise empty.
  // If not empty, it MUST be an absolute path in the root
  // filesystem of the process serving this request.
  // This field is OPTIONAL.
  // This field overrides the general CSI size limit.
  // SP SHOULD support the maximum path length allowed by the operating
  // system/filesystem, but, at a minimum, SP MUST accept a max path
  // length of at least 128 bytes.
  string staging_target_path = 3;

  // Volume capability describing how the CO intends to use this volume.
  // This allows SP to determine if volume is being used as a block
  // device or mounted file system. For example - if volume is being
  // used as a block device the SP MAY choose to skip calling filesystem
  // operations to reclaim space, but still perform rest of the housekeeping
  // needed for reducing the size of the volume. If volume_capability is
  // omitted the SP MAY determine access_type from given volume_path for the
  // volume and perform space reduction. This is an OPTIONAL field.
  csi.v1.VolumeCapability volume_capability = 4;

  // Secrets required by plugin to complete the reclaim space operation.
  // This field is OPTIONAL.
  map<string, string> secrets = 5 [(csi.v1.csi_secret) = true];
}

// NodeReclaimSpaceResponse holds the information about the result of the
// NodeReclaimSpaceRequest call.
message NodeReclaimSpaceResponse {
  // This field is OPTIONAL. This allows the SP to inform the CO about the
  // storage consumption before the ReclaimSpace operation was executed.
  StorageConsumption pre_usage = 1;

  // This field is OPTIONAL. This allows the SP to inform the CO about the
  // storage consumption after the ReclaimSpace operation was executed.
  StorageConsumption post_usage = 2;
}

NodeReclaimSpace Errors

If the CSI-driver is unable to complete the NodeReclaimSpace call successfully, it MUST return a non-ok gRPC code in the gRPC status. If the conditions defined below are encountered, the CSI-driver MUST return the specified gRPC error code. The caller MUST implement the specified error recovery behavior when it encounters the gRPC error code.

Condition gRPC Code Description Recovery Behavior
Volume does not exist 5 NOT_FOUND Indicates that a volume corresponding to the specified volume_id does not exist. Caller MUST verify that the volume_id is correct and that the volume is accessible and has not been deleted before retrying with exponential back off.
Operation not supported 12 UNIMPLEMENTED Indicates that the operation is not supported. The filesystem may not support fstrim, or the block-device may lack support for DISCARD operations. Caller MUST NOT retry the request as the underlying storage does not support space reclamation.