This document and the Identity Service is based on the Container Storage Interface specification.
Notational Conventions
The keywords “MUST”, “MUST NOT”, “REQUIRED”, “SHALL”, “SHALL NOT”, “SHOULD”, “SHOULD NOT”, “RECOMMENDED”, “NOT RECOMMENDED”, “MAY”, and “OPTIONAL” are to be interpreted as described in RFC 2119.
Terminology
| Term | Definition |
|---|---|
| CSI | Container Storage Interface. |
| CSI-driver | A component that implements CSI service RPCs, the CSI Controller or NodePlugin. |
| CO | Container Orchestration system that communicates with users and CSI-Addons service RPCs. |
| SP | Storage Provider, the vendor of a CSI-driver implementation. |
| RPC | Remote Procedure Call. |
Objective
Define an API that makes it possible for the CSI-Addons plugin for a CO to communicate with CSI-driver components that can execute storage maintenance operations.
Goals in MVP
The API will
- make it possible for worker nodes (CSI NodePlugin) to announce themselves
- have the CSI-Addons plugin for the CO list all announced CSI-Addons capable CSI-drivers
- provide details about CSI-Addons features the CSI-driver supports
Non-Goals in MVP
This will be limited to node discovery functionality only. CSI-Addons features will be documented in their own specifications. The available capabilities will get extended when new CSI-Addons operations are proposed.
Solution Overview
The CSI-Addons plugin for a CO reacts on events generated by the CO on request of user interactions. Events are translated to CSI-Addons gRPC procedures which are relayed to the CSI-drivers (CSI Controller and NodePlugin).
Architecture
A user (someone that creates/consumes CSI volumes) instructs the CSI-Addons plugin of the CO to invoke a storage maintenance operation. This interaction is done in a CO specific way (e.g. a Custom Resourse will be provided for Kubernetes).
The CSI-Addons plugin for the CO inspects the instruction from the user, and formats it into an operation that is sent to the CSI-driver Controller and/or NodePlugin.
.------. .------------.
| User |-------| CSI-Addons |
'------' | CO-plugin |
'------------'
|
| gRPC
|
.------------.
| CSI-driver |
'------------'
Error Handling
All RPCs defined in this specification return a standard
gRPC status.
The status code MUST be a
canonical gRPC error code;
the CO plugin MUST handle all valid error codes. When the code is not OK, the
status message MUST contain a human-readable description of the error, which MAY
be surfaced by the CO to end users. The status details MUST be empty.
The per-RPC Errors subsections below describe the conditions under which the CSI-driver MUST return a non-ok gRPC code and the recovery behavior the CO plugin is expected to implement.
RPC Interface
Identity Service: The CSI-driver MUST implement this, the CO-plugin uses the Identity Service to inspect the capabilities of the CSI-driver.
syntax = "proto3";
package identity;
import "google/protobuf/wrappers.proto"; // for BoolValue in ProbeResponse
option go_package = ".;identity";
// Identity is the service that CSI-drivers MUST implement in order to be
// compatible with CSI-Addons specification. It provides the basic details that
// are needed by the CSI-Addons CO plugin to communicate with the CSI-driver
// and detects its supported features.
service Identity {
// GetIdentity returns basic information about the side-car and CSI-driver.
rpc GetIdentity(GetIdentityRequest)
returns (GetIdentityResponse) {}
// GetCapabilities returns the capabilities that the CSI-driver supports.
rpc GetCapabilities(GetCapabilitiesRequest)
returns (GetCapabilitiesResponse) {}
// Probe is called by the CO plugin to validate that the CSI-Addons Node is
// still healthy.
rpc Probe(ProbeRequest)
returns (ProbeResponse) {}
}
The general flow of the success case MAY be as follows:
sequenceDiagram
participant CO as CSI-Addons CO plugin
participant Driver as CSI-driver
Note over CO,Driver: 1. Obtain the CSI-driver identity
CO->>Driver: GetIdentity
Driver-->>CO: name, vendor_version
Note over CO,Driver: 2. Query the supported CSI-Addons capabilities
CO->>Driver: GetCapabilities
Driver-->>CO: capabilities (e.g. CONTROLLER_SERVICE)
Note over CO,Driver: 3. Check the readiness of the CSI-driver
CO->>Driver: Probe
Driver-->>CO: ready: true
GetIdentity
The CSI-driver needs to provide its driver-name so that the CSI-Addons CO plugin can invoke RPCs for volumes that use this driver. There is an option for the CSI-driver to provide details about its version and a key/value manifest.
// GetIdentityRequest is sent by the CSI-Addons CO plugin to obtain the
// drivername, version and optional details from the CSI-driver.
message GetIdentityRequest {
// Intentionally empty.
}
// GetIdentityResponse is returned by the CSI-driver as a response to a
// GetIdentityRequest.
message GetIdentityResponse {
// The name MUST follow domain name notation format
// (https://tools.ietf.org/html/rfc1035#section-2.3.1). It SHOULD include
// the CSI-drivers's host company name and the CSI-driver name, to minimize
// the possibility of collisions. It MUST be 63 characters or less, beginning
// and ending with an alphanumeric character ([a-z0-9A-Z]) with dashes (-),
// dots (.), and alphanumerics between. This field is REQUIRED.
string name = 1;
// This field is REQUIRED. Value of this field is opaque to the CO.
string vendor_version = 2;
// This field is OPTIONAL. Values are opaque to the CO.
map<string, string> manifest = 3;
}
GetIdentity Errors
If the CSI-driver is unable to complete the GetIdentity call successfully, it MUST return a non-ok gRPC code in the gRPC status.
GetCapabilities
The CSI-Addons plugin for the CO needs to detect the supported features for each CSI-driver. Features are described as capabilities, each with additional information on how the CSI-Addons CO plugin may invoke the operation.
There might be operations that need to run on the CSI Controller (provisioner) for storage system access. Other operations may require running on a volume that is attached/staged on a node. By specifying the type of Service that the CSI-driver provides, the CSI-Addons CO plugin can make informed decisions on where to invoke the RPCs for an operation.
// GetCapabilitiesRequest is sent by the CSI-Addons CO plugin to detect the
// features that a CSI-driver supports.
message GetCapabilitiesRequest {
// Intentionally empty.
}
// GetCapabilitiesResponse is returned by the CSI-driver as a response to a
// GetCapabilitiesRequest.
message GetCapabilitiesResponse {
// All the capabilities that the controller service supports. This
// field is OPTIONAL.
repeated Capability capabilities = 1;
}
// Specifies one or more capabilities of the CSI-driver.
message Capability {
// Service contains the type of CSI Service that the CSI-driver provides.
message Service {
// Type describes a CSI Service that CSI-drivers can support.
enum Type {
// UNKNOWN indicates that the CSI-driver does not neither provide the CSI
// ControllerService or CSI NodeService. The CSI-Addons CO plugin will
// most likely ignore the node providing this Identity Service.
UNKNOWN = 0;
// CONTROLLER_SERVICE indicates that the CSI-driver provides RPCs for the
// CSI ControllerService.
// The presence of this capability determines whether the CSI-Addons CO
// plugin can invoke RPCs that require access to the storage system,
// similar to the CSI Controller (provisioner).
CONTROLLER_SERVICE = 1;
// NODE_SERVICE indicates that the CSI-driver provides RPCs for the CSI
// NodeService.
// The presence of this capability determines whether the CSI-Addons CO
// plugin can invoke RPCs that require a volume to be staged/attached to
// the node.
NODE_SERVICE = 2;
}
// type contains the Type of CSI Service that the CSI-driver supports.
Type type = 1;
}
// ReclaimSpace contains the features of the ReclaimSpace operation that the
// CSI-driver supports.
message ReclaimSpace {
// Type describes a CSI Service that CSI-drivers can support.
enum Type {
// UNKNOWN indicates that the CSI-driver does not support the ReclaimSpace
// operation in the current mode. The CSI-driver may be able to support
// the operation when is it configured differently. The CSI-Addons CO
// plugin will most likely ignore this node for the ReclaimSpace
// operation.
UNKNOWN = 0;
// OFFLINE indicates that the CSI-driver provides RPCs for an offline
// ReclaimSpace operation.
// The presence of this capability determines whether the CSI-Addons CO
// plugin can invoke RPCs that require access to the storage system,
// similar to the CSI Controller (provisioner).
OFFLINE = 1;
// ONLINE indicates that the CSI-driver provides RPCs for an online
// ReclaimSpace operation.
// The presence of this capability determines whether the CSI-Addons CO
// plugin can invoke RPCs that require a volume to be staged/attached to
// the node.
ONLINE = 2;
}
// type contains the Type of CSI Service that the CSI-driver supports.
Type type = 1;
}
// NetworkFence contains the features of the NetworkFence operation that the
// CSI-driver supports.
message NetworkFence {
// Type describes a CSI Service that CSI-drivers can support.
enum Type {
// UNKNOWN indicates that the CSI-driver does not support the NetworkFence
// operation in the current mode. The CSI-Addons CO plugin will most
// likely ignore this node for the NetworkFence operation.
UNKNOWN = 0;
// NETWORK_FENCE indicates that the CSI-driver provides RPCs for a
// NetworkFence operation.
// The presence of this capability determines whether the CSI-Addons CO
// plugin can invoke RPCs that require access to the storage system,
// similar to the CSI Controller (provisioner).
NETWORK_FENCE = 1;
// GET_CLIENTS_TO_FENCE indicates that the CSI-driver provides RPCs for a
// GET_CLIENTS_TO_FENCE operation to get the clients to fence.
// The presence of this capability determines whether the CSI-Addons CO
// plugin can invoke RPCs that require access to the storage system,
// similar to the CSI Controller (provisioner).
GET_CLIENTS_TO_FENCE = 2;
}
// type contains the Type of CSI Service that the CSI-driver supports.
Type type = 1;
}
// VolumeReplication contains the features of the volumereplication operation
// that the CSI-driver supports.
message VolumeReplication {
// Type describes a CSI Service that CSI-drivers can support.
enum Type {
// UNKNOWN indicates that the CSI-driver does not support the
// VolumeReplication operation in the current mode. The CSI-Addons CO
// plugin will most likely ignore this node for the
// VolumeReplication operation.
UNKNOWN = 0;
// VOLUME_REPLICATION indicates that the CSI-driver provides RPCs for a
// VolumeReplication operation.
// The presence of this capability determines whether the CSI-Addons CO
// plugin can invoke RPCs that require access to the storage system,
// similar to the CSI Controller (provisioner).
VOLUME_REPLICATION = 1;
// GET_REPLICATION_DESTINATION_INFO indicates that the CSI-driver
// supports getting the destination volume or volume group details
// for a replication relationship. This capability is relevant for
// storage providers where source and destination volume or volume
// group identifiers differ.
GET_REPLICATION_DESTINATION_INFO = 2;
}
// type contains the Type of CSI Service that the CSI-driver supports.
Type type = 1;
}
// VolumeGroup contains the features of the volumegroup operation
// that the CSI-driver supports.
message VolumeGroup {
// Type describes a CSI Service that CSI-drivers can support.
enum Type {
// UNKNOWN indicates that the CSI-driver does not support the
// VolumeGroup operation in the current mode. The CSI-Addons CO
// plugin will most likely ignore this node for the
// VolumeGroup operation.
UNKNOWN = 0;
// VOLUME_GROUP indicates that the CSI-driver provides RPCs for a
// VolumeGroup operation.
// The presence of this capability determines whether the CSI-Addons CO
// plugin can invoke RPCs that require access to the storage system,
// similar to the CSI Controller (provisioner).
VOLUME_GROUP = 1;
// LIMIT_VOLUME_TO_ONE_VOLUME_GROUP indicates that the CSI-driver
// does not support that one volume will be in multiple VGs,
// so it can be in only one VG.
LIMIT_VOLUME_TO_ONE_VOLUME_GROUP = 2;
// DO_NOT_ALLOW_VG_TO_DELETE_VOLUMES indicates that the CSI-driver
// does not support that deletion of the VG
// won't delete the volumes under it.
DO_NOT_ALLOW_VG_TO_DELETE_VOLUMES = 3;
// MODIFY_VOLUME_GROUP indicates that the CSI-driver
// supports that modifying the VG.
MODIFY_VOLUME_GROUP = 4;
// GET_VOLUME_GROUP indicates that the CSI-driver
// supports getting the VG.
GET_VOLUME_GROUP = 5;
// LIST_VOLUME_GROUPS indicates that the CSI-driver
// supports listing the VGs.
LIST_VOLUME_GROUPS = 6;
}
// type contains the Type of CSI Service that the CSI-driver supports.
Type type = 1;
}
// EncryptionKeyRotation contains the features of the EncryptionKeyRotation
// operation that the CSI driver supports.
message EncryptionKeyRotation {
// Type describes a CSI Encryption Service that CSI driver can support.
enum Type {
// UNKNOWN indicates that the CSI driver does not support the
// EncryptionKeyRotation operation in the current mode.
// The CSI-Addons CO plugin will most likely ignore this node
// for the EncryptionKeyRotation operation.
UNKNOWN = 0;
// ENCRYPTIONKEYROTATION indicates that the CSI driver provides
// RPCs for an EncryptionKeyRotation operation.
// The presence of this capability determines whether the CSI-Addons CO
// plugin can invoke RPCs for rotating the encryption key.
ENCRYPTIONKEYROTATION = 1;
}
// type contains the Type of CSI Service that the CSI driver supports.
Type type = 1;
}
// Additional CSI-Addons operations will need to be added here.
oneof type {
// Service or operation that the CSI-driver supports.
Service service = 1;
// ReclaimSpace operation capabilities.
ReclaimSpace reclaim_space = 2;
// NetworkFence operation capabilities
NetworkFence network_fence = 3;
// VolumeReplication operation capabilities.
VolumeReplication volume_replication = 4;
// VolumeGroup operation capabilities.
VolumeGroup volume_group = 5;
// EncryptionKeyRotation operation capabilities.
EncryptionKeyRotation encryption_key_rotation = 6;
// Additional CSI-Addons operations need to be appended to this list.
}
}
GetCapabilities Errors
If the CSI-driver is unable to complete the GetCapabilities call successfully, it MUST return a non-ok gRPC code in the gRPC status.
Probe
The primary utility of the Probe RPC is to verify that the CSI-driver is in a healthy and ready state. If an unhealthy state is reported, via a non-success response, a CSI-Addons CO plugin MAY take action with the intent to bring the CSI-driver to a healthy state. Such actions MAY include, but SHALL NOT be limited to, the following:
- Restarting the CSI-driver container, or
- Notifying the CSI-driver supervisor.
The CSI-driver MAY verify that it has the right configurations, devices, dependencies and drivers in order to run and return a success if the validation succeeds. The CSI-Addons CO plugin MAY invoke this RPC at any time. The CSI-Addons CO plugin MAY invoke this call multiple times with the understanding that a CSI-driver’s implementation MAY NOT be trivial and there MAY be overhead incurred by such repeated calls. The SP SHALL document guidance and known limitations regarding a particular CSI-driver’s implementation of this RPC. For example, the SP MAY document the maximum frequency at which its Probe implementation SHOULD be called.
// ProbeRequest is sent to the CSI-driver to confirm that it can respond to
// requests from the CSI-Addons CO plugin.
message ProbeRequest {
// Intentionally empty.
}
// ProbeResponse is returned by the CSI-driver as a response to a ProbeRequest.
message ProbeResponse {
// Readiness allows a CSI-driver to report its initialization status back
// to the CSI-Addons CO plugin. Initialization for some CSI-drivers MAY be
// time consuming and it is important for a CO to distinguish between the
// following cases:
//
// 1) The CSI-driver is in an unhealthy state and MAY need restarting. In
// this case a gRPC error code SHALL be returned.
// 2) The CSI-driver is still initializing, but is otherwise perfectly
// healthy. In this case a successful response SHALL be returned
// with a readiness value of `false`. Calls to the CSI-driver's
// Controller and/or Node services MAY fail due to an incomplete
// initialization state.
// 3) The CSI-driver has finished initializing and is ready to service
// calls to its Controller and/or Node services. A successful
// response is returned with a readiness value of `true`.
//
// This field is OPTIONAL. If not present, the caller SHALL assume
// that the CSI-driver is in a ready state and is accepting calls to its
// Controller and/or Node services (according to the CSI-driver's reported
// capabilities).
.google.protobuf.BoolValue ready = 1;
}
Probe Errors
If the CSI-driver is unable to complete the Probe call successfully, it MUST return a non-ok gRPC code in the gRPC status.
If the conditions defined below are encountered, the CSI-driver MUST return the specified gRPC error code.
The CSI-Addons CO plugin MAY implement the specified error recovery behavior when it encounters the gRPC error code.
| Condition | gRPC Code | Description | Recovery Behavior |
|---|---|---|---|
| CSI-driver not healthy | 9 FAILED_PRECONDITION | Indicates that the CSI-driver is not in a healthy/ready state. | Caller SHOULD assume the CSI-driver is not healthy and that future RPCs MAY fail because of this condition. |
| Missing required dependency | 9 FAILED_PRECONDITION | Indicates that the CSI-driver is missing one or more required dependency. | Caller MUST assume the CSI-driver is not healthy. |