Notational Conventions

The keywords “MUST”, “MUST NOT”, “REQUIRED”, “SHALL”, “SHALL NOT”, “SHOULD”, “SHOULD NOT”, “RECOMMENDED”, “NOT RECOMMENDED”, “MAY”, and “OPTIONAL” are to be interpreted as described in RFC 2119.

Terminology

Term Definition
CO Container Orchestration system that communicates with plugins using CSI service RPCs.
SP Storage Provider, the vendor of a CSI plugin implementation.
RPC Remote Procedure Call.

Objective

Define a standard that will enable storage providers (SP) to perform node level fencing using corresponding CIDR blocks.

Goals in MVP

The new extension will define a procedure that allows SP to block access to any client using corresponding CIDR blocks, and unblock as per the requirement.

If multiple COs are accessing a shared resource provided by the SP, and a node failure takes place, this procedure will allow the SP to block access from any node of CO, and thus preventing further access and corruption of data.

Non-Goals in MVP

Solution Overview

This specification defines an interface along with the minimum operational and packaging recommendations for a storage provider (SP) to implement fencing operation. The interface declares the RPCs that a plugin MUST expose.

Since the fencing is network based, the fencing mechanism will be most aptly implemented by the CSI Controller.

Architecture

A user instructs the CSI-Addons CO plugin to fence or unfence a set of clients. The CO plugin inspects the capabilities advertised by the CSI-driver (see the Identity NetworkFence capability) and relays the request as a gRPC call to the CSI-driver. Because fencing is network based, the operation is served by the CSI Controller.

Error Handling

All RPCs defined in this specification return a standard gRPC status. The status code MUST be a canonical gRPC error code; the CO MUST handle all valid error codes. When the code is not OK, the status message MUST contain a human-readable description of the error, which MAY be surfaced by the CO to end users. The status details MUST be empty.

Each RPC below documents, in its Errors table, the gRPC codes the CSI-driver returns for the listed conditions and the recovery behavior the caller is expected to implement.

The following error codes are common to all RPCs in this specification. In addition to the conditions below, each RPC MAY define further codes in its own Errors section.

Condition gRPC Code Description Recovery Behavior
Missing required field 3 INVALID_ARGUMENT Indicates that a required field is missing from the request. Caller MUST fix the request by adding the missing required field before retrying.
Invalid or unsupported field in the request 3 INVALID_ARGUMENT Indicates that one or more fields in the request is either not allowed by the CSI-driver or has an invalid value. Caller MUST fix the field before retrying.
Operation pending 10 ABORTED Indicates that there is already an operation pending for the specified resource. The CSI-driver SHOULD handle this as gracefully as possible, and MAY return this error code to reject secondary calls. Caller SHOULD ensure that there are no other calls pending for the specified resource, and then retry with exponential back off.
Call not implemented 12 UNIMPLEMENTED The invoked RPC is not implemented by the CSI-driver or disabled in its current mode of operation. Caller MUST NOT retry.
Not authenticated 16 UNAUTHENTICATED The invoked RPC does not carry secrets that are valid for authentication. Caller SHALL either fix the secrets provided in the RPC, or otherwise regalvanize said secrets such that they pass authentication by the CSI-driver for the attempted RPC, after which point the caller MAY retry the attempted RPC.
Error is Unknown 2 UNKNOWN Indicates that an unknown error occurred. Caller MUST study the logs before retrying.

RPC Interface

syntax = "proto3";
package fence;

import "github.com/container-storage-interface/spec/lib/go/csi/csi.proto";
import "google/protobuf/descriptor.proto";

option go_package = "github.com/csi-addons/spec/lib/go/fence";

// FenceController holds the RPC method for performing fencing operations.
service FenceController {
  // FenceClusterNetwork RPC call to perform a fencing operation.
  rpc FenceClusterNetwork (FenceClusterNetworkRequest)
  returns (FenceClusterNetworkResponse) {}

  // UnfenceClusterNetwork RPC call to remove a list of CIDR blocks from the
  // list of blocklisted/fenced clients.
  rpc UnfenceClusterNetwork (UnfenceClusterNetworkRequest)
  returns (UnfenceClusterNetworkResponse) {}

  // ListClusterFence RPC call to provide a list of blocklisted/fenced clients.
  rpc ListClusterFence(ListClusterFenceRequest)
  returns (ListClusterFenceResponse){}

  // GetFenceClients RPC calls to get the client information to use in a
  // FenceClusterNetwork or UnfenceClusterNetwork RPC.
  rpc GetFenceClients(GetFenceClientsRequest)
  returns (GetFenceClientsResponse){}
}

FenceClusterNetwork

The Controller plugin MUST implement this RPC call if it has the NETWORK_FENCE capability. This RPC is called by the CO to block access to the storage cluster for the clients identified by the provided list of CIDR blocks. This operation MUST be idempotent: fencing a client that is already fenced MUST succeed and reply 0 OK. The operation affects only the clients identified by the provided CIDR blocks; clients that were already fenced remain fenced. If the operation fails, or the CO does not know whether it succeeded, it MAY call FenceClusterNetwork again.

// FenceClusterNetworkRequest contains the information needed to identify
// the storage cluster so that the appropriate fencing operation can be
// performed.
message FenceClusterNetworkRequest {
  // Plugin specific parameters passed in as opaque key-value pairs.
  map<string, string> parameters = 1;
  // Secrets required by the plugin to complete the request.
  map<string, string> secrets = 2 [(csi.v1.csi_secret) = true];
  // list of CIDR blocks on which the fencing operation is expected to be
  // performed.
  repeated CIDR cidrs = 3;
}

// FenceClusterNetworkResponse is returned by the CSI-driver as a result of
// the FenceClusterNetworkRequest call.
message FenceClusterNetworkResponse {
  // Intentionally empty.
}

FenceClusterNetwork Errors

If the CSI-driver is unable to complete the FenceClusterNetwork call successfully, it MUST return a non-ok gRPC code in the gRPC status. This RPC defines no error codes beyond those common to all RPCs; see the Error Handling section.

UnfenceClusterNetwork

The Controller plugin MUST implement this RPC call if it has the NETWORK_FENCE capability. This RPC is called by the CO to remove the provided list of CIDR blocks from the set of fenced/blocklisted clients, restoring their access to the storage cluster. This operation MUST be idempotent: unfencing a client that is not currently fenced MUST succeed and reply 0 OK. The operation restores access only for the clients identified by the provided CIDR blocks; clients not in the list are unaffected.

// UnfenceClusterNetworkRequest contains the information needed to identify
// the cluster so that the appropriate fence operation can be
// disabled.
message UnfenceClusterNetworkRequest {
  // Plugin specific parameters passed in as opaque key-value pairs.
  map<string, string> parameters = 1;
  // Secrets required by the plugin to complete the request.
  map<string, string> secrets = 2 [(csi.v1.csi_secret) = true];
  // list of CIDR blocks on which the fencing operation is expected to be
  // performed.
  repeated CIDR cidrs = 3;
}

// UnfenceClusterNetworkResponse is returned by the CSI-driver as a result of
// the UnfenceClusterNetworkRequest call.
message UnfenceClusterNetworkResponse {
  // Intentionally empty.
}

UnfenceClusterNetwork Errors

If the CSI-driver is unable to complete the UnfenceClusterNetwork call successfully, it MUST return a non-ok gRPC code in the gRPC status. This RPC defines no error codes beyond those common to all RPCs; see the Error Handling section.

ListClusterFence

The Controller plugin MUST implement this RPC call if it has the NETWORK_FENCE capability. This RPC is called by the CO to retrieve the list of CIDR blocks that are currently fenced/blocklisted by the CSI-driver. This is a read-only operation. The response contains the CIDR blocks currently fenced by the CSI-driver, and is empty when no client is fenced.

// ListClusterFenceRequest contains the information needed to identify
// the cluster so that the appropriate fenced clients can be listed.
message ListClusterFenceRequest {
  // Plugin specific parameters passed in as opaque key-value pairs.
  map<string, string> parameters = 1;
  // Secrets required by the plugin to complete the request.
  map<string, string> secrets = 2 [(csi.v1.csi_secret) = true];
}

// ListClusterFenceResponse holds the information about the result of the
// ListClusterFenceResponse call.
message ListClusterFenceResponse {
  // list of IPs that are blocklisted by the SP.
  repeated CIDR cidrs = 1;
}

ListClusterFence Errors

If the CSI-driver is unable to complete the ListClusterFence call successfully, it MUST return a non-ok gRPC code in the gRPC status. This RPC defines no error codes beyond those common to all RPCs; see the Error Handling section.

CIDR blocks

// CIDR holds a CIDR block.
message CIDR {
  // CIDR block
  string cidr = 1;
}

GetFenceClients

The Controller plugin MUST implement this RPC call if it has the GET_CLIENTS_TO_FENCE capability. This RPC is called by the CO to discover the clients, along with their addresses, that need to be fenced. The returned ClientDetails can then be used to build the list of CIDR blocks passed to a subsequent FenceClusterNetwork or UnfenceClusterNetwork call. This is a read-only operation.

// GetFenceClientsRequest contains the necessary information to identify
// the clients that need fencing.
message GetFenceClientsRequest {
  // Plugin-specific parameters passed in as opaque key-value pairs.
  map<string, string> parameters = 1;
  // Secrets required by the plugin to complete the request.
  map<string, string> secrets = 2 [(csi.v1.csi_secret) = true];
}

// GetFenceClientsResponse holds the information about clients that require
// fencing.
message GetFenceClientsResponse {
  // List of clients that need to be fenced.
  repeated ClientDetails clients = 1;
}

ClientDetails

ClientDetails describes a single client returned by GetFenceClients. It carries the client’s unique identifier and the list of addresses (as CIDR blocks) used to fence or unfence that client.

// ClientDetails holds the information about the client that requires fencing.
message ClientDetails {
  // id represents the unique identifier of the client.
  // Required field.
  string id = 1;
  // list of IP addresses that represent the client's local addresses.
  // Required field.
  repeated CIDR addresses = 2;
}

GetFenceClients Errors

If the CSI-driver is unable to complete the GetFenceClients call successfully, it MUST return a non-ok gRPC code in the gRPC status. This RPC defines no error codes beyond those common to all RPCs; see the Error Handling section.