Notational Conventions
The keywords “MUST”, “MUST NOT”, “REQUIRED”, “SHALL”, “SHALL NOT”, “SHOULD”, “SHOULD NOT”, “RECOMMENDED”, “NOT RECOMMENDED”, “MAY”, and “OPTIONAL” are to be interpreted as described in RFC 2119.
Terminology
| Term | Definition |
|---|---|
| VolumeID | The identifier of the volume generated by the plugin. |
| CO | Container Orchestration system that communicates with plugins using CSI service RPCs. |
| SP | Storage Provider, the vendor of a CSI plugin implementation. |
| RPC | Remote Procedure Call. |
Objective
Define an extension to the CSI Specification that will enable storage vendors (SP) to develop controllers/plugins that can free unused storage allocations from existing volumes.
Goals in MVP
The new extension will define a procedure that
- can be called for existing volumes
- functions for Filesystem and Block mode volumes
- interacts with the Controller and Node-Plugin
- makes it possible for the SP to free unused space from volumes
Non-Goals in MVP
- Freeing unused storage allocations for inactive volumes
Solution Overview
This specification defines an interface along with the minimum operational and packaging recommendations for a storage provider (SP) to implement a space reclaim operation for volumes. The interface declares the RPCs that a plugin MUST expose.
Architecture
A user instructs the CSI-Addons CO plugin to reclaim unused space from a volume.
The CO plugin inspects the capabilities advertised by the CSI-driver (see the
Identity ReclaimSpace capability) and translates the
request into a gRPC call that is relayed to the CSI-driver. The operation runs
either on the CSI Controller (OFFLINE, via ControllerReclaimSpace) or on the
CSI NodePlugin (ONLINE, via NodeReclaimSpace).
Error Handling
All RPCs defined in this specification return a standard
gRPC status.
The status code MUST be a
canonical gRPC error code;
the CO MUST handle all valid error codes. When the code is not OK, the status
message MUST contain a human-readable description of the error, which MAY be
surfaced by the CO to end users. The status details MUST be empty.
Each RPC below documents, in its Errors table, the gRPC codes the CSI-driver returns for the listed conditions and the recovery behavior the caller is expected to implement.
The following error codes are common to all RPCs in this specification. In addition to the conditions below, each RPC MAY define further codes in its own Errors section.
| Condition | gRPC Code | Description | Recovery Behavior |
|---|---|---|---|
| Missing required field | 3 INVALID_ARGUMENT | Indicates that a required field is missing from the request. | Caller MUST fix the request by adding the missing required field before retrying. |
| Invalid or unsupported field in the request | 3 INVALID_ARGUMENT | Indicates that one or more fields in the request is either not allowed by the CSI-driver or has an invalid value. | Caller MUST fix the field before retrying. |
| Operation pending | 10 ABORTED | Indicates that there is already an operation pending for the specified resource. The CSI-driver SHOULD handle this as gracefully as possible, and MAY return this error code to reject secondary calls. | Caller SHOULD ensure that there are no other calls pending for the specified resource, and then retry with exponential back off. |
| Call not implemented | 12 UNIMPLEMENTED | The invoked RPC is not implemented by the CSI-driver or disabled in its current mode of operation. | Caller MUST NOT retry. |
| Not authenticated | 16 UNAUTHENTICATED | The invoked RPC does not carry secrets that are valid for authentication. | Caller SHALL either fix the secrets provided in the RPC, or otherwise regalvanize said secrets such that they pass authentication by the CSI-driver for the attempted RPC, after which point the caller MAY retry the attempted RPC. |
| Error is Unknown | 2 UNKNOWN | Indicates that an unknown error occurred. | Caller MUST study the logs before retrying. |
RPC Interface
- Controller Service: The Controller plugin MAY implement this RPC.
- Node Service: The Node plugin MAY implement this RPC.
syntax = "proto3";
package reclaimspace;
import "github.com/container-storage-interface/spec/lib/go/csi/csi.proto";
import "google/protobuf/descriptor.proto";
option go_package = "github.com/csi-addons/spec/lib/go/reclaimspace";
// ReclaimSpaceController holds the RPC method for running discard operations
// on the inactive (not required to be staged/published) volume.
service ReclaimSpaceController {
// ControllerReclaimSpace is a procedure that gets called on the CSI
// Controller.
rpc ControllerReclaimSpace (ControllerReclaimSpaceRequest)
returns (ControllerReclaimSpaceResponse) {}
}
// ReclaimSpaceNode holds the RPC method for running discard operations on the
// active (staged/published) volume.
service ReclaimSpaceNode {
// NodeReclaimSpace is a procedure that gets called on the CSI NodePlugin.
rpc NodeReclaimSpace (NodeReclaimSpaceRequest)
returns (NodeReclaimSpaceResponse) {}
}
ControllerReclaimSpace
A Controller Plugin MUST implement this RPC call if it has the ReclaimSpace
OFFLINE capability (see the Identity service). This
RPC is called by the CSI-Addons CO plugin to free unused storage allocations of
a volume that does not need to be staged/published on a node. This operation
MUST be idempotent. On success the CSI-driver replies 0 OK. The amount of space
actually reclaimed is storage-specific and MAY be zero; the optional pre_usage
and post_usage allow the CO to observe the effect. Because reclaiming space does
not change the logical content of the volume, reclaiming a volume that has no
unused space to free MUST still succeed. If the operation fails, or the CO does
not know whether it succeeded, it MAY call ControllerReclaimSpace again.
// ControllerReclaimSpaceRequest contains the information needed to identify
// the volume by the SP and access any backend services so that space can be
// reclaimed.
message ControllerReclaimSpaceRequest {
// The ID of the volume. This field is REQUIRED.
string volume_id = 1;
// Plugin specific parameters passed in as opaque key-value pairs.
map<string, string> parameters = 2;
// Secrets required by the plugin to complete the request.
map<string, string> secrets = 3 [(csi.v1.csi_secret) = true];
}
// ControllerReclaimSpaceResponse holds the information about the result of the
// ControllerReclaimSpaceRequest call.
message ControllerReclaimSpaceResponse {
// This field is OPTIONAL. This allows the SP to inform the CO about the
// storage consumption before the ReclaimSpace operation was executed.
StorageConsumption pre_usage = 1;
// This field is OPTIONAL. This allows the SP to inform the CO about the
// storage consumption after the ReclaimSpace operation was executed.
StorageConsumption post_usage = 2;
}
// StorageConsumption contains the usage in bytes.
message StorageConsumption {
// This field is REQUIRED. usage_bytes contains the consumed storage in
// bytes.
int64 usage_bytes = 1;
}
ControllerReclaimSpace Errors
If the CSI-driver is unable to complete the ControllerReclaimSpace call successfully, it MUST return a non-ok gRPC code in the gRPC status. If the conditions defined below are encountered, the CSI-driver MUST return the specified gRPC error code. The caller MUST implement the specified error recovery behavior when it encounters the gRPC error code.
| Condition | gRPC Code | Description | Recovery Behavior |
|---|---|---|---|
| Volume does not exist | 5 NOT_FOUND | Indicates that a volume corresponding to the specified volume_id does not exist. |
Caller MUST verify that the volume_id is correct and that the volume is accessible and has not been deleted before retrying with exponential back off. |
| Operation not supported | 12 UNIMPLEMENTED | Indicates that the operation is not supported. The filesystem may not support fstrim, or the block-device may lack support for DISCARD operations. | Caller MUST NOT retry the request as the underlying storage does not support space reclamation. |
NodeReclaimSpace
A Node Plugin MUST implement this RPC call if it has the ReclaimSpace ONLINE
capability (see the Identity service). This RPC is
called by the CSI-Addons CO plugin on the node where the volume is
staged/published, to free unused storage allocations of the active volume using
the volume_path at which the volume is available. This operation MUST be
idempotent. On success the CSI-driver replies 0 OK. The amount of space actually
reclaimed is storage-specific and MAY be zero; the optional pre_usage and
post_usage allow the CO to observe the effect. Reclaiming a volume that has no
unused space to free MUST still succeed. If the operation fails, or the CO does
not know whether it succeeded, it MAY call NodeReclaimSpace again.
// NodeReclaimSpaceRequest contains the information needed to identify the
// location where the volume is mounted so that local filesystem or
// block-device operations to reclaim space can be executed.
message NodeReclaimSpaceRequest {
// The ID of the volume. This field is REQUIRED.
string volume_id = 1;
// The path on which volume is available. This field is REQUIRED.
// This field overrides the general CSI size limit.
// SP SHOULD support the maximum path length allowed by the operating
// system/filesystem, but, at a minimum, SP MUST accept a max path
// length of at least 128 bytes.
string volume_path = 2;
// The path where the volume is staged, if the plugin has the
// STAGE_UNSTAGE_VOLUME capability, otherwise empty.
// If not empty, it MUST be an absolute path in the root
// filesystem of the process serving this request.
// This field is OPTIONAL.
// This field overrides the general CSI size limit.
// SP SHOULD support the maximum path length allowed by the operating
// system/filesystem, but, at a minimum, SP MUST accept a max path
// length of at least 128 bytes.
string staging_target_path = 3;
// Volume capability describing how the CO intends to use this volume.
// This allows SP to determine if volume is being used as a block
// device or mounted file system. For example - if volume is being
// used as a block device the SP MAY choose to skip calling filesystem
// operations to reclaim space, but still perform rest of the housekeeping
// needed for reducing the size of the volume. If volume_capability is
// omitted the SP MAY determine access_type from given volume_path for the
// volume and perform space reduction. This is an OPTIONAL field.
csi.v1.VolumeCapability volume_capability = 4;
// Secrets required by plugin to complete the reclaim space operation.
// This field is OPTIONAL.
map<string, string> secrets = 5 [(csi.v1.csi_secret) = true];
}
// NodeReclaimSpaceResponse holds the information about the result of the
// NodeReclaimSpaceRequest call.
message NodeReclaimSpaceResponse {
// This field is OPTIONAL. This allows the SP to inform the CO about the
// storage consumption before the ReclaimSpace operation was executed.
StorageConsumption pre_usage = 1;
// This field is OPTIONAL. This allows the SP to inform the CO about the
// storage consumption after the ReclaimSpace operation was executed.
StorageConsumption post_usage = 2;
}
NodeReclaimSpace Errors
If the CSI-driver is unable to complete the NodeReclaimSpace call successfully, it MUST return a non-ok gRPC code in the gRPC status. If the conditions defined below are encountered, the CSI-driver MUST return the specified gRPC error code. The caller MUST implement the specified error recovery behavior when it encounters the gRPC error code.
| Condition | gRPC Code | Description | Recovery Behavior |
|---|---|---|---|
| Volume does not exist | 5 NOT_FOUND | Indicates that a volume corresponding to the specified volume_id does not exist. |
Caller MUST verify that the volume_id is correct and that the volume is accessible and has not been deleted before retrying with exponential back off. |
| Operation not supported | 12 UNIMPLEMENTED | Indicates that the operation is not supported. The filesystem may not support fstrim, or the block-device may lack support for DISCARD operations. | Caller MUST NOT retry the request as the underlying storage does not support space reclamation. |