[RFCs/IDs] [Plain Text] [Tracker] [WG] [Email] [Diff1] [Diff2] [Nits]
Versions: 00 01 02 03 04 05 06 RFC 5046
INTERNET DRAFT Mike Ko
draft-ietf-ips-iser-06.txt IBM Corporation
Mallikarjun Chadalapaka
Hewlett-Packard Company
John Hufferd
Brocade, Inc.
Uri Elzur
Hemal Shah
Patricia Thaler
Broadcom Corporation
Expires: May, 2007
iSCSI Extensions for RDMA Specification
Status of this Memo
By submitting this Internet-Draft, each author represents that any
applicable patent or other IPR claims of which he or she is aware
have been or will be disclosed, and any of which he or she becomes
aware will be disclosed, in accordance with Section 6 of BCP 79.
Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF), its areas, and its working groups. Note that
other groups may also distribute working documents as Internet-
Drafts.
Internet-Drafts are draft documents valid for a maximum of six
months and may be updated, replaced, or obsoleted by other documents
at any time. It is inappropriate to use Internet-Drafts as
reference material or to cite them other than as "work in progress."
The list of current Internet-Drafts can be accessed at
http://www.ietf.org/1id-abstracts.html.
The list of Internet-Draft Shadow Directories can be accessed at
http://www.ietf.org/shadow.html.
Abstract
iSCSI Extensions for RDMA provides the RDMA data transfer capability
to iSCSI by layering iSCSI on top of an RDMA-Capable Protocol such
as the iWARP protocol suite. An RDMA-Capable Protocol provides RDMA
Read and Write services, which enable data to be transferred
directly into SCSI I/O Buffers without intermediate data copies.
This document describes the extensions to the iSCSI protocol to
Ko et al. Expires May 2007 1
Internet-Draft iSER Specification November 2006
support RDMA services as provided by an RDMA-Capable Protocol such
as the iWARP protocol suite.
Ko et al. Expires May 2007 2
Internet-Draft iSER Specification November 2006
Table of Contents
1 Definitions and Acronyms....................................7
1.1 Definitions.................................................7
1.2 Acronyms...................................................13
1.3 Conventions................................................15
2 Introduction...............................................16
2.1 Motivation.................................................16
2.2 Architectural Goals........................................17
2.3 Protocol Overview..........................................18
2.4 RDMA services and iSER.....................................19
2.4.1 STag......................................................19
2.4.2 Send......................................................20
2.4.3 RDMA Write................................................20
2.4.4 RDMA Read.................................................20
2.5 SCSI Read Overview.........................................21
2.6 SCSI Write Overview........................................21
2.7 iSCSI/iSER Layering........................................22
3 Upper Layer Interface Requirements.........................23
3.1 Operational Primitives offered by iSER.....................23
3.1.1 Send_Control..............................................24
3.1.2 Put_Data..................................................24
3.1.3 Get_Data..................................................24
3.1.4 Allocate_Connection_Resources.............................25
3.1.5 Deallocate_Connection_Resources...........................25
3.1.6 Enable_Datamover..........................................25
3.1.7 Connection_Terminate......................................26
3.1.8 Notice_Key_Values.........................................26
3.1.9 Deallocate_Task_Resources.................................26
3.2 Operational Primitives used by iSER........................27
3.2.1 Control_Notify............................................27
3.2.2 Data_Completion_Notify....................................27
3.2.3 Data_ACK_Notify...........................................28
3.2.4 Connection_Terminate_Notify...............................28
3.3 iSCSI Protocol Usage Requirements..........................28
4 Lower Layer Interface Requirements.........................30
4.1 Interactions with the RCaP Layer...........................30
4.2 Interactions with the Transport Layer......................31
5 Connection Setup and Termination...........................32
5.1 iSCSI/iSER Connection Setup................................32
5.1.1 Initiator Behavior........................................33
5.1.2 Target Behavior...........................................35
5.1.3 iSER Hello Exchange.......................................37
5.2 iSCSI/iSER Connection Termination..........................38
5.2.1 Normal Connection Termination at the Initiator............38
5.2.2 Normal Connection Termination at the Target...............38
5.2.3 Termination without Logout Request/Response PDUs..........39
Ko et al. Expires May 2007 3
Internet-Draft iSER Specification November 2006
6 Login/Text Operational Keys................................41
6.1 HeaderDigest and DataDigest................................41
6.2 MaxRecvDataSegmentLength...................................41
6.3 RDMAExtensions.............................................41
6.4 TargetRecvDataSegmentLength................................42
6.5 InitiatorRecvDataSegmentLength.............................43
6.6 OFMarker and IFMarker......................................43
6.7 MaxOutstandingUnexpectedPDUs...............................44
7 iSCSI PDU Considerations...................................45
7.1 iSCSI Data-Type PDU........................................45
7.2 iSCSI Control-Type PDU.....................................46
7.3 iSCSI PDUs.................................................46
7.3.1 SCSI Command..............................................46
7.3.2 SCSI Response.............................................48
7.3.3 Task Management Function Request/Response.................49
7.3.4 SCSI Data-out.............................................51
7.3.5 SCSI Data-in..............................................51
7.3.6 Ready To Transfer (R2T)...................................54
7.3.7 Asynchronous Message......................................56
7.3.8 Text Request & Text Response..............................56
7.3.9 Login Request & Login Response............................56
7.3.10 Logout Request & Logout Response........................56
7.3.11 SNACK Request...........................................57
7.3.12 Reject..................................................57
7.3.13 NOP-Out & NOP-In........................................57
8 Flow Control and STag Management...........................58
8.1 Flow Control for RDMA Send Message Types...................58
8.1.1 Flow Control for Control-Type PDUs from the Initiator.....58
8.1.2 Flow Control for Control-Type PDUs from the Target........61
8.2 Flow Control for RDMA Read Resources.......................62
8.3 STag Management............................................62
8.3.1 Allocation of STags.......................................63
8.3.2 Invalidation of STags.....................................63
9 iSER Control and Data Transfer.............................65
9.1 iSER Header Format.........................................65
9.2 iSER Header Format for iSCSI Control-Type PDU..............65
9.3 iSER Header Format for iSER Hello Message..................67
9.4 iSER Header Format for iSER HelloReply Message.............68
9.5 SCSI Data Transfer Operations..............................69
9.5.1 SCSI Write Operation......................................69
9.5.2 SCSI Read Operation.......................................70
9.5.3 Bidirectional Operation...................................70
10 iSER Error Handling and Recovery...........................71
10.1 Error Handling............................................71
10.1.1 Errors in the Transport Layer...........................71
10.1.2 Errors in the RCaP Layer................................72
10.1.3 Errors in the iSER Layer................................72
Ko et al. Expires May 2007 4
Internet-Draft iSER Specification November 2006
10.1.4 Errors in the iSCSI Layer...............................74
10.2 Error Recovery............................................76
10.2.1 PDU Recovery............................................76
10.2.2 Connection Recovery.....................................77
11 Security Considerations....................................78
12 IANA Considerations........................................79
13 References.................................................80
13.1 Normative References......................................80
13.2 Informative References....................................80
14 Appendix A.................................................82
14.1 iWARP Message Format for iSER.............................82
14.1.1 iWARP Message Format for iSER Hello Message.............82
14.1.2 iWARP Message Format for iSER HelloReply Message........83
14.1.3 iWARP Message Format for SCSI Read Command PDU..........84
14.1.4 iWARP Message Format for SCSI Read Data.................85
14.1.5 iWARP Message Format for SCSI Write Command PDU.........86
14.1.6 iWARP Message Format for RDMA Read Request..............87
14.1.7 iWARP Message Format for Solicited SCSI Write Data......88
14.1.8 iWARP Message Format for SCSI Response PDU..............89
15 Appendix B.................................................90
15.1 Architectural discussion of iSER over InfiniBand..........90
15.2 The Host side of the iSCSI & iSER connections in Infiniband90
15.3 The Storage side of iSCSI & iSER mixed network environment91
15.4 Discovery processes for an InfiniBand Host................91
15.5 IBTA Connection specifications............................92
16 Author's Address...........................................93
17 Acknowledgments............................................94
18 Full Copyright Statement...................................95
Ko et al. Expires May 2007 5
Internet-Draft iSER Specification November 2006
Table of Figures
Figure 1 Example of iSCSI/iSER Layering in Full Feature Phase...22
Figure 2 iSER Header Format.....................................65
Figure 3 iSER Header Format for iSCSI Control-Type PDU..........66
Figure 4 iSER Header Format for iSER Hello Message..............67
Figure 5 iSER Header Format for iSER HelloReply Message.........68
Figure 6 SendSE Message containing an iSER Hello Message........82
Figure 7 SendSE Message containing an iSER HelloReply Message...83
Figure 8 SendSE Message containing a SCSI Read Command PDU......84
Figure 9 RDMA Write Message containing SCSI Read Data...........85
Figure 10 SendSE Message containing a SCSI Write Command PDU....86
Figure 11 RDMA Read Request Message.............................87
Figure 12 RDMA Read Response Message containing SCSI Write Data.88
Figure 13 SendInvSE Message containing SCSI Response PDU........89
Figure 14 iSCSI and iSER on IB..................................90
Figure 15 Storage Controller with TCP, iWARP, and IB Connections91
Ko et al. Expires May 2007 6
Internet-Draft iSER Specification November 2006
1 Definitions and Acronyms
1.1 Definitions
Advertisement (Advertised, Advertise, Advertisements, Advertises) -
The act of informing a remote iSER Layer that a local node's
buffer is available to it. A Node makes a buffer available for
incoming RDMA Read Request Message or incoming RDMA Write
Message access by informing the remote iSER Layer of the Tagged
Buffer identifiers (STag, TO, and buffer length). Note that
this Advertisement of Tagged Buffer information is the
responsibility of the iSER Layer on either end and is not
defined by the RDMA-Capable Protocol. A typical method would be
for the iSER Layer to embed the Tagged Buffer's STag, TO, and
buffer length in a Send Message destined for the remote iSER
Layer.
Completion (Completed, Complete, Completes) - Completion is defined
as the process by the RDMA-Capable Protocol layer to inform the
iSER Layer, that a particular RDMA Operation has performed all
functions specified for the RDMA Operation.
Connection - A connection is a logical circuit between the initiator
and the target, e.g., a TCP connection. Communication between
the initiator and the target occurs over one or more
connections. The connections carry control messages, SCSI
commands, parameters, and data within iSCSI Protocol Data Units
(iSCSI PDUs).
Connection Handle - An information element that identifies the
particular iSCSI connection and is unique for a given iSCSI-iSER
pair. Every invocation of an Operational Primitive is qualified
with the Connection Handle.
Data Sink - The peer receiving a data payload. Note that the Data
Sink can be required to both send and receive RCaP Messages to
transfer a data payload.
Data Source - The peer sending a data payload. Note that the Data
Source can be required to both send and receive RCaP Messages to
transfer a data payload.
Datamover Interface (DI) - The interface between the iSCSI Layer and
the Datamover Layer as described in [DA].
Datamover Layer - A layer that is directly below the iSCSI Layer and
above the underlying transport layers. This layer exposes and
Ko et al. Expires May 2007 7
Internet-Draft iSER Specification November 2006
uses a set of transport independent Operational Primitives for
the communication between the iSCSI Layer and itself. The
Datamover layer, operating in conjunction with the transport
layers, moves the control and data information on the iSCSI
connection. In this specification, the iSER Layer is the
Datamover layer.
Datamover Protocol - A Datamover protocol is the wire-protocol that
is defined to realize the Datamover layer functionality. In
this specification, the iSER protocol is the Datamover protocol.
Event - An indication provided by the RDMA-Capable Protocol layer to
the iSER Layer to indicate a Completion or other condition
requiring immediate attention.
Inbound RDMA Read Queue Depth (IRD) - The maximum number of incoming
outstanding RDMA Read Requests that the RDMA-Capable Controller
can handle on a particular RCaP Stream at the Data Source. For
some RDMA-Capable Protocol layers, the term "IRD" may be known
by a different name. For example, for InfiniBand, the
equivalent for IRD is the Responder Resources.
Invalidate STag - A mechanism used to prevent the Remote Peer from
reusing a previous explicitly Advertised STag, until the iSER
Layer at the local node makes it available through a subsequent
explicit Advertisement.
I/O Buffer - A buffer that is used in a SCSI Read or Write operation
so SCSI data may be sent from or received into that buffer.
iSCSI - The iSCSI protocol as defined in [RFC3720] is a mapping of
the SCSI Architecture Model of SAM-2 over TCP.
iSCSI control-type PDU - Any iSCSI PDU that is not an iSCSI data-
type PDU and also not a SCSI Data-out PDU carrying solicited
data is defined as an iSCSI control-type PDU. Specifically, it
is to be noted that SCSI Data-out PDUs for unsolicited data are
defined as iSCSI control-type PDUs.
iSCSI data-type PDU - An iSCSI data-type PDU is defined as an iSCSI
PDU that causes data transfer, transparent to the remote iSCSI
Layer, to take place between the peer iSCSI nodes on a full
feature phase iSCSI connection. An iSCSI data-type PDU, when
requested for transmission by the sender iSCSI Layer, results in
the associated data transfer without the participation of the
remote iSCSI Layer, i.e. the PDU itself is not delivered as-is
Ko et al. Expires May 2007 8
Internet-Draft iSER Specification November 2006
to the remote iSCSI Layer. The following iSCSI PDUs constitute
the set of iSCSI data-type PDUs - SCSI Data-In PDU and R2T PDU.
iSCSI Layer - A layer in the protocol stack implementation within an
end node that implements the iSCSI protocol and interfaces with
the iSER Layer via the Datamover Interface.
iSCSI PDU (iSCSI Protocol Data Unit) - The iSCSI Layer at the
initiator and the iSCSI Layer at the target divide their
communications into messages. The term "iSCSI protocol data
unit" (iSCSI PDU) is used for these messages.
iSCSI/iSER Connection - An iSER-assisted iSCSI connection.
iSCSI/iSER Session - An iSER-assisted iSCSI session.
iSCSI-iSER Pair - The iSCSI Layer and the underlying iSER Layer.
iSER - iSCSI Extensions for RDMA, the protocol defined in this
document.
iSER-assisted - A term generally used to describe the operation of
iSCSI when the iSER functionality is also enabled below the
iSCSI Layer for the specific iSCSI/iSER connection in question.
iSER-IRD - This variable represents the maximum number of incoming
outstanding RDMA Read Requests that the iSER Layer at the
initiator declares on a particular RCaP Stream.
iSER-ORD - This variable represents the maximum number of
outstanding RDMA Read Requests that the iSER Layer can initiate
on a particular RCaP Stream. This variable is maintained only
by the iSER Layer at the target.
iSER Layer - The layer that implements the iSCSI Extensions for RDMA
(iSER) protocol.
iWARP - A suite of wire protocols comprising of [RDMAP], [DDP], and
[MPA] when layered above [TCP]. [RDMAP] and [DDP] may be
layered above SCTP or other transport protocols.
Local Mapping - A task state record maintained by the iSER Layer
that associates the Initiator Task Tag to the Local STag(s).
The specifics of the record structure are implementation
dependent.
Ko et al. Expires May 2007 9
Internet-Draft iSER Specification November 2006
Local Peer - The implementation of the RDMA-Capable Protocol on the
local end of the connection. Used to refer to the local entity
when describing protocol exchanges or other interactions between
two Nodes.
Node - A computing device attached to one or more links of a
network. A Node in this context does not refer to a specific
application or protocol instantiation running on the computer.
A Node may consist of one or more RDMA-Capable Controllers
installed in a host computer.
Operational Primitive - An Operational Primitive is an abstract
functional interface procedure that requests another layer to
perform a specific action on the requestor's behalf or notifies
the other layer of some event. The Datamover Interface between
an iSCSI Layer and a Datamover layer within an iSCSI end node
uses a set of Operational Primitives to define the functional
interface between the two layers. Note that not every
invocation of an Operational Primitive may elicit a response
from the requested layer. A full discussion of the Operational
Primitive types and request-response semantics available to
iSCSI and iSER can be found in [DA].
Outbound RDMA Read Queue Depth (ORD) - The maximum number of
outstanding RDMA Read Requests that the RDMA-Capable Controller
can initiate on a particular RCaP Stream at the Data Sink. For
some RDMA-Capable Protocol layer, the term "ORD" may be known by
a different name. For example, for InfiniBand, the equivalent
for ORD is the Initiator Depth.
Phase-Collapse - Refers to the optimization in iSCSI where the SCSI
status is transferred along with the final SCSI Data-in PDU from
a target. See section 3.2 in [RFC3720].
RCaP Message - One or more packets of the network layer comprising a
single RDMA operation or a part of an RDMA Read Operation of the
RDMA-Capable Protocol. For iWARP, an RCaP Message is known as
an RDMAP Message.
RCaP Stream - A single bidirectional association between the peer
RDMA-Capable Protocol layers on two Nodes over a single
transport-level stream. For iWARP, an RCaP Stream is known as
an RDMAP Stream, and the association is created when the
connection transitions to iSER-assisted mode following a
successful Login Phase during which iSER support is negotiated.
Ko et al. Expires May 2007 10
Internet-Draft iSER Specification November 2006
RDMA-Capable Protocol (RCaP) - The protocol or protocol suite that
provides a reliable RDMA transport functionality, e.g., iWARP,
InfiniBand, etc.
RDMA-Capable Controller - A network I/O adapter or embedded
controller with RDMA functionality. For example, for iWARP,
this could be an RNIC, and for InfiniBand, this could be a HCA
(Host Channel Adapter) or TCA (Target Channel Adapter).
RDMA-enabled Network Interface Controller (RNIC) - A network I/O
adapter or embedded controller with iWARP functionality.
RDMA Operation - A sequence of RCaP Messages, including control
Messages, to transfer data from a Data Source to a Data Sink.
The following RDMA Operations are defined - RDMA Write
Operation, RDMA Read Operation, Send Operation, Send with
Invalidate Operation, Send with Solicited Event Operation, Send
with Solicited Event and Invalidate Operation, and Terminate
Operation.
RDMA Protocol (RDMAP) - A wire protocol that supports RDMA
Operations to transfer ULP data between a Local Peer and the
Remote Peer as described in [RDMAP].
RDMA Read Operation - An RDMA Operation used by the Data Sink to
transfer the contents of a Data Source buffer from the Remote
Peer to a Data Sink buffer at the Local Peer. An RDMA Read
operation consists of a single RDMA Read Request Message and a
single RDMA Read Response Message.
RDMA Read Request - An RCaP Message used by the Data Sink to request
the Data Source to transfer the contents of a buffer. The RDMA
Read Request Message describes both the Data Source and the Data
Sink buffers.
RDMA Read Response - An RCaP Message used by the Data Source to
transfer the contents of a buffer to the Data Sink, in response
to an RDMA Read Request. The RDMA Read Response Message only
describes the Data Sink buffer.
RDMA Write Operation - An RDMA Operation used by the Data Source to
transfer the contents of a Data Source buffer from the Local
Peer to a Data Sink buffer at the Remote Peer. The RDMA Write
Message only describes the Data Sink buffer.
Remote Direct Memory Access (RDMA) - A method of accessing memory on
a remote system in which the local system specifies the remote
Ko et al. Expires May 2007 11
Internet-Draft iSER Specification November 2006
location of the data to be transferred. Employing an RDMA-
Capable Controller in the remote system allows the access to take
place without interrupting the processing of the CPU(s) on the
system.
Remote Mapping - A task state record maintained by the iSER Layer
that associates the Initiator Task Tag to the Advertised STag(s).
The specifics of the record structure are implementation
dependent.
Remote Peer - The implementation of the RDMA-Capable Protocol on the
opposite end of the connection. Used to refer to the remote
entity when describing protocol exchanges or other interactions
between two Nodes.
SCSI Layer - This layer builds/receives SCSI CDBs (Command
Descriptor Blocks) and sends/receives them with the remaining
command execute [SAM2] parameters to/from the iSCSI Layer.
Send - An RDMA Operation that transfers the contents of a Buffer
from the Local Peer to a Buffer at the Remote Peer.
Send Message Type - A Send Message, Send with Invalidate Message,
Send with Solicited Event Message, or Send with Solicited Event
and Invalidate Message.
SendInvSE Message - A Send with Solicited Event and Invalidate
Message.
SendSE Message - A Send with Solicited Event Message
Sequence Number (SN) - DataSN for a SCSI Data-in PDU and R2TSN for
an R2T PDU. The semantics for both types of sequence numbers
are as defined in [RFC3720].
Session, iSCSI Session - The group of Connections that link an
initiator SCSI port with a target SCSI port form an iSCSI
session (equivalent to a SCSI I-T nexus). Connections can be
added to and removed from a session even while the I-T nexus is
intact. Across all connections within a session, an initiator
sees one and the same target.
Solicited Event (SE) - A facility by which an RDMA Operation sender
may cause an Event to be generated at the recipient, if the
recipient is configured to generate such an Event, when a Send
with Solicited Event or Send with Solicited Event and Invalidate
Message is received.
Ko et al. Expires May 2007 12
Internet-Draft iSER Specification November 2006
Steering Tag (STag) - An identifier of a Tagged Buffer on a Node
(Local or Remote) as defined in [RDMAP] and [DDP]. For other
RDMA-Capable Protocols, the Steering Tag may be known by
different names but will be herein referred to as STags. For
example, for Infiniband, a Remote STag is known as an R-Key, and
a Local STag is known as an L-Key, and both will be considered
STags.
Tagged Buffer - A buffer that is explicitly Advertised to the iSER
Layer at the remote node through the exchange of an STag, Tagged
Offset, and length.
Tagged Offset (TO) - The offset within a Tagged Buffer.
Traditional iSCSI - Refers to the iSCSI protocol as defined in
[RFC3720] (i.e. without the iSER enhancements).
Untagged Buffer - A buffer that is not explicitly Advertised to the
iSER Layer at the remode node.
1.2 Acronyms
Acronym Definition
--------------------------------------------------------------
AHS Additional Header Segment
BHS Basic Header Segment
CO Connection Only
CRC Cyclic Redundancy Check
DDP Direct Data Placement Protocol
DI Datamover Interface
HCA Host Channel Adapter
IANA Internet Assigned Numbers Authority
IB Infiniband
IETF Internet Engineering Task Force
I/O Input - Output
Ko et al. Expires May 2007 13
Internet-Draft iSER Specification November 2006
IO Initialize Only
IP Internet Protocol
IPoIB IP over Infiniband
IPsec Internet Protocol Security
iSER iSCSI Extensions for RDMA
ITT Initiator Task Tag
LO Leading Only
MPA Marker PDU Aligned Framing for TCP
NOP No Operation
NSG Next Stage (during the iSCSI Login Phase)
OS Operating System
PDU Protocol Data Unit
R2T Ready To Transfer
R2TSN Ready To Transfer Sequence Number
RDMA Remote Direct Memory Access
RDMAP Remote Direct Memory Access Protocol
RFC Request For Comments
RNIC RDMA-enabled Network Interface Controller
SAM2 SCSI Architecture Model - 2
SCSI Small Computer Systems Interface
SNACK Selective Negative Acknowledgment - also
Sequence Number Acknowledgement for data
STag Steering Tag
SW Session Wide
Ko et al. Expires May 2007 14
Internet-Draft iSER Specification November 2006
TCA Target Channel Adapter
TCP Transmission Control Protocol
TMF Task Management Function
TTT Target Transfer Tag
TO Tagged Offset
ULP Upper Level Protocol
1.3 Conventions
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
document are to be interpreted as described in RFC 2119.
Ko et al. Expires May 2007 15
Internet-Draft iSER Specification November 2006
2 Introduction
2.1 Motivation
The iSCSI protocol ([RFC3720]) is a mapping of the SCSI Architecture
Model (see [SAM2]) over the TCP protocol. SCSI commands are carried
by iSCSI requests and SCSI responses and status are carried by iSCSI
responses. Other iSCSI protocol exchanges and SCSI Data are also
transported in iSCSI PDUs.
Out-of-order TCP segments in the Traditional iSCSI model have to be
stored and reassembled before the iSCSI protocol layer within an end
node can place the data in the iSCSI buffers. This reassembly is
required because not every TCP segment is likely to contain an iSCSI
header to enable its placement and TCP itself does not have a built-
in mechanism for signaling ULP message boundaries to aid placement
of out-of-order segments. This TCP reassembly at high network
speeds is quite counter-productive for the following reasons: wasted
memory bandwidth in data copying, need for reassembly memory, wasted
CPU cycles in data copying, and the general store-and-forward
latency from an application perspective. TCP reassembly was
recognized as a serious issue in [RFC3720], and the notion of a
"sync and steering layer" was introduced that is optional to
implement and use. One specific sync and steering mechanism, called
"markers", was defined in [RFC3720] which provides an application-
level way of framing iSCSI PDUs within the TCP data stream even when
the TCP segments are not yet reassembled to be in-order.
With these defined techniques in [RFC3720], a Network Interface
Controller customized for iSCSI (SNIC) could offload the TCP/IP
processing and support direct data placement, but most iSCSI
implementations do not support iSCSI "markers", making SNIC
marker-based direct data placement unusable in practice.
The iWARP protocol stack provides direct data placement
functionality that is usable in practice, and in addition, there is
also interest in using iSCSI with other RDMA protocol stacks that
support direct data placement, such as the one provided by
InfiniBand. The generic term RDMA-Capable Protocol (RCaP) is used
to refer to the RDMA functionality provided by such protocol stacks.
With the availability of RDMA-Capable Controllers within a host
system, which does not have SNICs, it is appropriate for iSCSI to be
able to exploit the direct data placement function of the RDMA-
Capable Controller like other applications.
Ko et al. Expires May 2007 16
Internet-Draft iSER Specification November 2006
iSCSI Extensions for RDMA (iSER) is designed precisely to take
advantage of generic RDMA technologies - iSER's goal is to permit
iSCSI to employ direct data placement and RDMA capabilities using a
generic RDMA-Capable Controller. In summary, iSCSI/iSER protocol
stack is designed to enable scaling to high speeds by relying on a
generic data placement process and RDMA technologies and products,
which enable direct data placement of both in-order and out-of-order
data.
This document describes iSER as a protocol extension to iSCSI, both
for convenience of description and also because it is true in a very
strict protocol sense. However, it is to be noted that iSER is in
reality extending the connectivity of the iSCSI protocol defined in
[RFC3720], and the name iSER reflects this reality.
When the iSCSI protocol as defined in [RFC3720] (i.e. without the
iSER enhancements) is intended in the rest of the document, the term
"Traditional iSCSI" is used to make the intention clear.
2.2 Architectural Goals
This section summarizes the architectural goals that guided the
design of iSER.
1. Provide an RDMA data transfer model for iSCSI that enables direct
in order or out of order data placement of SCSI data into pre-
allocated SCSI buffers while maintaining in order data delivery.
2. Not require any major changes to SCSI Architecture Model [SAM2]
and SCSI command set standards.
3. Utilize existing iSCSI infrastructure (sometimes referred to as
"iSCSI ecosystem") including but not limited to MIB,
bootstrapping, negotiation, naming & discovery, and security.
4. Require a session to operate in the Traditional iSCSI data
transfer mode if iSER is not supported by either the initiator or
the target (not require iSCSI full feature phase interoperability
between an end node operating in Traditional iSCSI mode, and an
end node operating in iSER-assisted mode).
5. Allow initiator and target implementations to utilize generic
RDMA-Capable Controllers such as RNICs, or implement iSCSI and
iSER in software (not require iSCSI or iSER specific assists in
the RCaP implementation or RDMA-Capable Controller).
Ko et al. Expires May 2007 17
Internet-Draft iSER Specification November 2006
6. Require full and only generic RCaP functionality at both the
initiator and the target.
7. Implement a light weight Datamover protocol for iSCSI with minimal
state maintenance.
2.3 Protocol Overview
Consistent with the architectural goals stated in section 2.2, the
iSER protocol does not require changes in the iSCSI ecosystem or any
related SCSI specifications. iSER protocol defines the mapping of
iSCSI PDUs to RCaP Messages in such a way that it is entirely
feasible to realize iSCSI/iSER implementations that are based on
generic RDMA-Capable Controllers. The iSER protocol layer requires
minimal state maintenance to assist an iSCSI full feature phase
connection, besides being oblivious to the notion of an iSCSI
session. The crucial protocol aspects of iSER may be summarized
thus:
1. iSER-assisted mode is negotiated during the iSCSI login for each
session, and an entire iSCSI session can only operate in one mode
(i.e. a connection in a session cannot operate in iSER-assisted
mode if a different connection of the same session is already in
full feature phase in the Traditional iSCSI mode).
2. Once in iSER-assisted mode, all iSCSI interactions on that
connection use RCaP Messages.
3. A Send Message Type is used for carrying an iSCSI control-type
PDU preceded by an iSER header. See section 7.2 for more details
on iSCSI control-type PDUs.
4. RDMA Write, RDMA Read Request, and RDMA Read Response Messages
are used for carrying control and all data information associated
with the iSCSI data-type PDUs. See section 7.1 for more details
on iSCSI data-type PDUs.
5. Target drives all data transfer (with the exception of iSCSI
unsolicited data) for SCSI writes and SCSI reads, by issuing RDMA
Read Requests and RDMA Writes respectively.
6. RCaP is responsible for ensuring data integrity. (For example,
iWARP includes a CRC-enhanced framing layer called MPA on top of
TCP; and for Infiniband, the CRCs are included in the Reliable
Connection mode). For this reason, iSCSI header and data digests
are negotiated to "None" for iSCSI/iSER sessions.
Ko et al. Expires May 2007 18
Internet-Draft iSER Specification November 2006
7. The iSCSI error recovery hierarchy defined in [RFC3720] is fully
supported by iSER. (However, see section 7.3.11 on the handling
of SNACK Request PDUs.)
8. iSER requires no changes to iSCSI authentication, security, and
text mode negotiation mechanisms.
Note that Traditional iSCSI implementations may have to be adapted
to employ iSER. It is expected that the adaptation when required is
likely to be centered around the upper layer interface requirements
of iSER (section 3).
2.4 RDMA services and iSER
iSER is designed to work with software and/or hardware protocol
stacks providing the protocol services defined in RCaP documents
such as [RDMAP], [IB], etc. The following subsections describe the
key protocol elements of RCaP services that iSER relies on.
2.4.1 STag
An STag is the identifier of an I/O Buffer unique to an RDMA-Capable
Controller that the iSER Layer Advertises to the remote iSCSI/iSER
node in order to complete a SCSI I/O.
In iSER, Advertisement is the act of informing the target by the
initiator that an I/O Buffer is available at the initiator for RDMA
Read or RDMA Write access by the target. The initiator Advertises
the I/O Buffer by including the STag in the header of an iSER
Message containing the SCSI Command PDU to the target. The base
Tagged Offset is not explicitly specified, but the target must
always assume it as zero. The buffer length is as specified in the
SCSI Command PDU.
The iSER Layer at the initiator Advertises the STag for the I/O
Buffer of each SCSI I/O to the iSER Layer at the target in the iSER
header of the SendSE Message containing the SCSI Command PDU, unless
the I/O can be completely satisfied by unsolicited data alone.
The iSER Layer at the target provides the STag for the I/O Buffer
that is the Data Sink of an RDMA Read Operation (section 2.4.4) to
the RCaP layer on the initiator node - i.e. this is completely
transparent to the iSER Layer at the initiator.
The iSER protocol is defined so that the Advertised STag is
automatically invalidated upon a normal completion of the associated
task. This automatic invalidation is realized via the SendInvSE
Ko et al. Expires May 2007 19
Internet-Draft iSER Specification November 2006
Message carrying the SCSI Response PDU. There are two exceptions to
this automatic invalidation - bidirectional commands, and abnormal
completion of a command. The iSER Layer at the initiator is
required to explicitly invalidate the STag in these cases, in
addition to sanity checking the automatic invalidation even when
that does happen.
2.4.2 Send
Send is the RDMA Operation that is not addressed to an Advertised
buffer by the sending side, and thus uses Untagged buffers on the
receiving side.
The iSER Layer at the initiator uses the Send Operation to transmit
any iSCSI control-type PDU to the target. As an example, the
initiator uses Send Operations to transfer iSER Messages containing
SCSI Command PDUs to the iSER Layer at the target.
An iSER layer at the target uses the Send Operation to transmit any
iSCSI control-type PDU to the initiator. As an example, the target
uses Send Operations to transfer iSER Messages containing SCSI
Response PDUs to the iSER Layer at the initiator.
2.4.3 RDMA Write
RDMA Write is the RDMA Operation that is used to place data into an
Advertised buffer on the receiving side. The sending side addresses
the Message using an STag and a Tagged Offset that are valid on the
Data Sink.
The iSER Layer at the target uses the RDMA Write Operation to
transfer the contents of a local I/O Buffer to an Advertised I/O
Buffer at the initiator. The iSER Layer at the target uses the RDMA
Write to transfer whole or part of the data required to complete a
SCSI Read command.
The iSER Layer at the initiator does not employ RDMA Writes.
2.4.4 RDMA Read
RDMA Read is the RDMA Operation that is used to retrieve data from
an Advertised buffer on a remote node. The sending side of the RDMA
Read Request addresses the Message using an STag and a Tagged Offset
that are valid on the Data Source in addition to providing a valid
local STag and Tagged Offset that identify the Data Sink.
Ko et al. Expires May 2007 20
Internet-Draft iSER Specification November 2006
The iSER Layer at the target uses the RDMA Read Operation to
transfer the contents of an Advertised I/O Buffer at the initiator
to a local I/O Buffer at the target. The iSER Layer at the target
uses the RDMA Read to fetch whole or part of the data required to
complete a SCSI Write Command.
The iSER Layer at the initiator does not employ RDMA Reads.
2.5 SCSI Read Overview
The iSER Layer at the initiator receives the SCSI Command PDU from
the iSCSI Layer. The iSER Layer at the initiator generates an STag
for the I/O Buffer of the SCSI Read and Advertises the buffer by
including the STag as part of the iSER header for the PDU. The iSER
Message is transferred to the target using a SendSE Message.
The iSER Layer at the target uses one or more RDMA Writes to
transfer the data required to complete the SCSI Read.
The iSER Layer at the target uses a SendInvSE Message to transfer
the SCSI Response PDU back to the iSER Layer at the initiator. The
iSER Layer at the initiator notifies the iSCSI Layer of the
availability of the SCSI Response PDU.
2.6 SCSI Write Overview
The iSER Layer at the initiator receives the SCSI Command PDU from
the iSCSI Layer. If solicited data transfer is involved, the iSER
Layer at the initiator generates an STag for the I/O Buffer of the
SCSI Write and Advertises the buffer by including the STag as part
of the iSER header for the PDU. The iSER Message is transferred to
the target using a SendSE Message.
The iSER Layer at the initiator may optionally send one or more non-
immediate unsolicited data PDUs to the target using Send Message
Types.
If solicited data transfer is involved, the iSER Layer at the target
uses one or more RDMA Reads to transfer the data required to
complete the SCSI Write.
The iSER Layer at the target uses a SendInvSE Message to transfer
the SCSI Response PDU back to the iSER Layer at the initiator. The
iSER Layer at the initiator notifies the iSCSI Layer of the
availability of the SCSI Response PDU.
Ko et al. Expires May 2007 21
Internet-Draft iSER Specification November 2006
2.7 iSCSI/iSER Layering
iSCSI Extensions for RDMA (iSER) is layered between the iSCSI layer
and the RCaP layer. Note that the RCaP layer may be composed of one
or more distinct protocol layers depending on the specifics of the
RCaP. Figure 1 shows an example of the relationship between SCSI,
iSCSI, iSER, and the different RCaP layers. For TCP, the RCaP is
iWARP. For Infiniband, the RCaP is the Reliable Connected Transport
Service. Note that the iSCSI layer as described here supports the
RDMA Extensions as used in iSER.
+-------------------------------------+
| SCSI |
+-------------------------------------+
| iSCSI |
DI ------> +-------------------------------------+
| iSER |
+---------+--------------+------------+
| RDMAP | | |
+---------+ Infiniband | |
| DDP | Reliable | Other |
+---------+ Connected | RDMA- |
| MPA | Transport | Capable |
+---------+ Service | Protocol |
| TCP | | |
+---------+--------------+------------+
| | Infiniband | Other |
| IP | Network | Network |
| | Layer | Layer |
+---------+--------------+------------+
Figure 1 Example of iSCSI/iSER Layering in Full Feature Phase
Ko et al. Expires May 2007 22
Internet-Draft iSER Specification November 2006
3 Upper Layer Interface Requirements
This section discusses the upper layer interface requirements in the
form of an abstract model of the required interactions between the
iSCSI Layer and the iSER Layer. The abstract model used here is
derived from the architectural model described in [DA]. [DA] also
provides a functional overview of the interactions between the iSCSI
Layer and the datamover layer as intended by the Datamover
Architecture.
The interface requirements are specified by Operational Primitives.
An Operational Primitive is an abstract functional interface
procedure between the iSCSI Layer and the iSER Layer that requests
one layer to perform a specific action on behalf of the other layer
or notifies the other layer of some event. Whenever an Operational
Primitive in invoked, the Connection_Handle qualifier is used to
identify a particular iSCSI connection. For some Operational
Primitives, a Data_Descriptor is used to identify the iSCSI/SCSI
data buffer associated with the requested or completed operation.
The abstract model and the Operational Primitives defined in this
section facilitate the description of the iSER protocol. In the
rest of the iSER specification, the compliance statements related to
the use of these Operational Primitives are only for the purpose of
the required interactions between the iSCSI Layer and the iSER
Layer. Note that the compliance statements related to the
Operational Primitives in the rest of this specification only
mandate functional equivalence on implementations, but do not put
any requirements on the implementation specifics of the interface
between the iSCSI Layer and the iSER Layer.
Each Operational Primitive is invoked with a set of qualifiers which
specify the information context for performing the specific action
being requested of the Operational Primitive. While the qualifiers
are required, the method of realizing the qualifiers (e.g., by
passing synchronously with invocation, or by retrieving from task
context, or by retrieving from shared memory, etc.) is
implementation dependent.
3.1 Operational Primitives offered by iSER
The iSER protocol layer MUST support the following Operational
Primitives to be used by the iSCSI protocol layer.
Ko et al. Expires May 2007 23
Internet-Draft iSER Specification November 2006
3.1.1 Send_Control
Input qualifiers: Connection_Handle, BHS and AHS (if any) of
the iSCSI PDU, PDU-specific qualifiers
Return results: Not specified
This is used by the iSCSI Layers at the initiator and the target to
request the outbound transfer of an iSCSI control-type PDU (see
section 7.2). Qualifiers that only apply for a particular control-
type PDU are known as PDU-specific qualifiers, e.g.,
ImmediateDataSize for a SCSI Write command. For details on PDU-
specific qualifiers, see section 7.3. The iSCSI Layer can only
invoke the Send_Control Operational Primitive when the connection is
in iSER-assisted mode.
3.1.2 Put_Data
Input qualifiers: Connection_Handle, content of a SCSI Data-in
PDU header, Data_Descriptor, Notify_Enable
Return results: Not specified
This is used by the iSCSI Layer at the target to request the
outbound transfer of data for a SCSI Data-in PDU from the buffer
identified by the Data_Descriptor qualifier. The iSCSI Layer can
only invoke the Put_Data Operational Primitive when the connection
is in iSER-assisted mode.
The Notify_Enable qualifier is used to indicate to the iSER Layer
whether or not it should generate an eventual local completion
notification to the iSCSI Layer. See section 3.2.2 on
Data_Completion_Notify for details.
3.1.3 Get_Data
Input qualifiers: Connection_Handle, content of an R2T PDU,
Data_Descriptor, Notify_Enable
Return results: Not specified
This is used by the iSCSI Layer at the target to request the inbound
transfer of solicited data requested by an R2T PDU into the buffer
identified by the Data_Descriptor qualifier. The iSCSI Layer can
only invoke the Get_Data Operational Primitive when the connection
is in iSER-assisted mode.
Ko et al. Expires May 2007 24
Internet-Draft iSER Specification November 2006
The Notify_Enable qualifier is used to indicate to the iSER Layer
whether or not it should generate the eventual local completion
notification to the iSCSI Layer. See section 3.2.2 on
Data_Completion_Notify for details.
3.1.4 Allocate_Connection_Resources
Input qualifiers: Connection_Handle, Resource_Descriptor
(optional)
Return results: Status
This is used by the iSCSI Layers at the initiator and the target to
request the allocation of all connection resources necessary to
support RCaP for an operational iSCSI/iSER connection. The iSCSI
Layer may optionally specify the implementation-specific resource
requirements for the iSCSI connection using the Resource_Descriptor
qualifier.
A return result of Status=success means the invocation succeeded,
and a return result of Status=failure means that the invocation
failed. If the invocation is for a Connection_Handle for which an
earlier invocation succeeded, the request will be ignored by the
iSER Layer and the result of Status=success will be returned. Only
one Allocate_Connection_Resources Operational Primitive invocation
can be outstanding for a given Connection_Handle at any time.
3.1.5 Deallocate_Connection_Resources
Input qualifiers: Connection_Handle
Return results: Not specified
This is used by the iSCSI Layers at the initiator and the target to
request the deallocation of all connection resources that were
allocated earlier as a result of a successful invocation of the
Allocate_Connection_Resources Operational Primitive.
3.1.6 Enable_Datamover
Input qualifiers: Connection_Handle,
Transport_Connection_Descriptor, Final Login_Response_PDU
(optional)
Return results: Not specified
Ko et al. Expires May 2007 25
Internet-Draft iSER Specification November 2006
This is used by the iSCSI Layers at the initiator and the target to
request that a specified iSCSI connection be transitioned to iSER-
assisted mode. The Transport_Connection_Descriptor qualifier is
used to identify the specific connection associated with the
Connection_Handle. The iSCSI layer can only invoke the
Enable_Datamover Operational Primitive when there was a
corresponding prior resource allocation.
The Final_Login_Response_PDU input qualifier is applicable only for
a target, and contains the final Login Response PDU that concludes
the iSCSI Login Phase. If the underlying transport is TCP, the
final Login Response PDU must be sent as a byte stream as expected
by the iSCSI Layer at the initiator. When this qualifier is used,
the iSER Layer at the target MUST transmit this final Login Response
PDU before transitioning to iSER-assisted mode.
3.1.7 Connection_Terminate
Input qualifiers: Connection_Handle
Return results: Not specified
This is used by the iSCSI Layers at the initiator and the target to
request that a specified iSCSI/iSER connection be terminated and all
associated connection and task resources be freed. When this
Operational Primitive invocation returns to the iSCSI layer, the
iSCSI layer may assume full ownership of all iSCSI-level resources,
e.g. I/O Buffers, associated with the connection.
3.1.8 Notice_Key_Values
Input qualifiers: Connection_Handle, number of keys, list of
Key-Value pairs
Return results: Not specified
This is used by the iSCSI Layers at the initiator and the target to
request the iSER Layer to take note of the specified Key-Value pairs
which were negotiated by the iSCSI peers for the connection.
3.1.9 Deallocate_Task_Resources
Input qualifiers: Connection_Handle, ITT
Return results: Not specified
Ko et al. Expires May 2007 26
Internet-Draft iSER Specification November 2006
This is used by the iSCSI Layers at the initiator and the target to
request the deallocation of all RCaP-specific resources allocated by
the iSER Layer for the task identified by the ITT qualifier. The
iSER Layer may require a certain number of RCaP-specific resources
associated with the ITT for each new iSCSI task. In the normal
course of execution, these task-level resources in the iSER Layer
are assumed to be transparently allocated on each task initiation
and deallocated on the conclusion of each task as appropriate. In
exception scenarios where the task does not conclude with a SCSI
Response PDU, the iSER Layer needs to be notified of the individual
task terminations to aid its task-level resource management. This
Operational Primitive is used for this purpose, and is not needed
when a SCSI Response PDU normally concludes a task. Note that RCaP-
specific task resources are deallocated by the iSER Layer when a
SCSI Response PDU normally concludes a task, even if the SCSI Status
was not success.
3.2 Operational Primitives used by iSER
The iSER layer MUST use the following Operational Primitives offered
by the iSCSI protocol layer when the connection is in iSER-assisted
mode.
3.2.1 Control_Notify
Input qualifiers: Connection_Handle, an iSCSI control-type PDU
Return results: Not specified
This is used by the iSER Layers at the initiator and the target to
notify the iSCSI Layer of the availability of an inbound iSCSI
control-type PDU. A PDU is described as "available" to the iSCSI
Layer when the iSER Layer notifies the iSCSI Layer of the reception
of that inbound PDU, along with an implementation-specific
indication as to where the received PDU is.
3.2.2 Data_Completion_Notify
Input qualifiers: Connection_Handle, ITT, SN
Return results: Not specified
This is used by the iSER Layer to notify the iSCSI Layer of the
completion of outbound data transfer that was requested by the iSCSI
Layer only if the invocation of the Put_Data Operational Primitive
(see section 3.1.2) was qualified with Notify_Enable set. SN refers
to the DataSN associated with the SCSI Data-In PDU.
Ko et al. Expires May 2007 27
Internet-Draft iSER Specification November 2006
This is used by the iSER Layer to notify the iSCSI Layer of the
completion of inbound data transfer that was requested by the iSCSI
Layer only if the invocation of the Get_Data Operational Primitive
(see section 3.1.3) was qualified with Notify_Enable set. SN refers
to the R2TSN associated with the R2T PDU.
3.2.3 Data_ACK_Notify
Input qualifier: Connection_Handle, ITT, DataSN
Return results: Not specified
This is used by the iSER Layer at the target to notify the iSCSI
Layer of the arrival of the data acknowledgement (as defined in
[RFC3720]) requested earlier by the iSCSI Layer for the outbound
data transfer via an invocation of the Put_Data Operational
Primitive where the A-bit in the SCSI Data-in PDU is set to 1. See
section 7.3.5. DataSN refers to the expected DataSN of the next
SCSI Data-in PDU which immediately follows the SCSI Data-in PDU with
the A-bit set to which this notification corresponds, with semantics
as defined in [RFC3720].
3.2.4 Connection_Terminate_Notify
Input qualifiers: Connection_Handle
Return results: Not specified
This is used by the iSER Layers at the initiator and the target to
notify the iSCSI Layer of the unsolicited termination or failure of
an iSCSI/iSER connection. The iSER Layer MUST deallocate the
connection and task resources associated with the terminated
connection before the invocation of this Operational Primitive.
Note that the Connection_Terminate_Notify Operational Primitive is
not invoked when the termination of the connection was earlier
requested by the local iSCSI Layer.
3.3 iSCSI Protocol Usage Requirements
To operate in an iSER-assisted mode, the iSCSI Layers at both the
initiator and the target MUST negotiate the RDMAExtensions key (see
section 6.3) to "Yes" on the leading connection. If the
RDMAExtensions key is not negotiated to "Yes", then iSER-assisted
mode MUST NOT be used. If the RDMAExtensons key is negotiated to
"Yes" but the invocation of the Allocate_Connection_Resources
Operational Primitive to the iSER layer fails, the iSCSI layer MUST
Ko et al. Expires May 2007 28
Internet-Draft iSER Specification November 2006
fail the iSCSI Login process or terminate the connection as
appropriate. See section 10.1.3.1 for details.
If the RDMAExtensions key is negotiated to "Yes", the iSCSI Layer
MUST satisfy the following protocol usage requirements from the iSER
protocol:
1. The iSCSI Layer at the initiator MUST set ExpDataSN to 0 in Task
Management Function Requests for Task Allegiance Reassignment
for read/bidirectional commands, so as to cause the target to
send all unacknowledged read data.
2. The iSCSI Layer at the target MUST always return the SCSI status
in a separate SCSI Response PDU for read commands, i.e., there
MUST NOT be a "phase collapse" in concluding a SCSI Read
Command.
3. The iSCSI Layers at both the initiator and the target MUST
support the keys as defined in section 6 on Login/Text
Operational Keys. If used as specified, these keys MUST NOT be
answered with NotUnderstood and the semantics as defined MUST be
followed for each iSER-assisted connection.
4. The iSCSI Layer at the initiator MUST NOT issue SNACKs for PDUs.
Ko et al. Expires May 2007 29
Internet-Draft iSER Specification November 2006
4 Lower Layer Interface Requirements
4.1 Interactions with the RCaP Layer
The iSER protocol layer is layered on top of an RCaP layer (see
Figure 1) and the following are the key features that are assumed to
be supported by any RCaP layer:
* The RCaP layer supports all basic RDMA operations, including RDMA
Write Operation, RDMA Read Operation, Send Operation, Send with
Invalidate Operation, Send with Solicited Event Operation, Send
with Solicited Event & Invalidate Operation, and Terminate
Operation.
* The RCaP layer provides reliable, in-order message delivery and
direct data placement.
* When the iSER Layer initiates an RDMA Read Operation following an
RDMA Write Operation on one RCaP Stream, the RDMA Read Response
Message processing on the remote node will be started only after
the preceding RDMA Write Message payload is placed in the memory
of the remote node.
* The RCaP layer encapsulates a single iSER Message into a single
RCaP Message on the Data Source side. The RCaP layer
decapsulates the iSER Message before delivering it to the iSER
Layer on the Data Sink side.
* When the iSER Layer provides the STag to be remotely invalidated
to the RCaP layer for a SendInvSE Message, the RCaP layer uses
this STag as the STag to be invalidated in the SendInvSE Message.
* The RCaP layer uses the STag and Tagged Offset provided by the
iSER Layer for the RDMA Write and RDMA Read Request Messages.
* When the RCaP layer delivers the content of an RDMA Send Message
Type to the iSER Layer, the RCaP layer provides the length of the
RDMA Send message. This ensures that the iSER Layer does not
have to carry a length field in the iSER header.
* When the RCaP layer delivers the SendSE or SendInvSE Message to
the iSER Layer, it notifies the iSER Layer with the mechanism
provided on that interface.
* When the RCaP layer delivers a SendInvSE Message to the iSER
Layer, it passes the value of the STag that was invalidated.
Ko et al. Expires May 2007 30
Internet-Draft iSER Specification November 2006
* The RCaP layer propagates all status and error indications to the
iSER Layer.
* For a transport layer that operates in byte stream mode such as
TCP, the RCaP implementation supports the enabling of the RDMA
mode after Connection establishment and the exchange of Login
parameters in byte stream mode. For a transport layer that
provides message delivery capability such as [IB], the RCaP
implementation supports the use of the messaging capability by
the iSCSI Layer directly for the Login phase after connection
establishment before enabling iSER-assisted mode.
* Whenever the iSER Layer terminates the RCaP Stream, the RCaP
layer terminates the associated Connection.
4.2 Interactions with the Transport Layer
The iSER Layer does not directly setup the transport layer
connection (e.g., TCP, or [IB]). During Connection setup, the iSCSI
Layer is responsible for setting up the Connection. If the login is
successful, the iSCSI Layer invokes the Enable_Datamover Operational
Primitive to request the iSER Layer to transition to the iSER-
assisted mode for that iSCSI connection. See section 5.1 on
iSCSI/iSER Connection Setup. After transitioning to iSER-assisted
mode, the RCaP layer and the underlying transport layer are
responsible for maintaining the Connection and reporting to the iSER
Layer any Connection failures.
Ko et al. Expires May 2007 31
Internet-Draft iSER Specification November 2006
5 Connection Setup and Termination
5.1 iSCSI/iSER Connection Setup
During connection setup, the iSCSI Layer at the initiator is
responsible for establishing a connection with the target. After
the connection is established, the iSCSI Layers at the initiator and
the target enter the Login Phase using the same rules as outlined in
[RFC3720]. Transition to iSER-assisted mode occurs when the
connection transitions into the iSCSI full feature phase following a
successful login negotiation between the initiator and the target in
which iSER-assisted mode is negotiated and the connection resources
necessary to support RCaP have been allocated at both the initiator
and the target. The same connection MUST be used for both the iSCSI
Login phase and the subsequent iSER-assisted full feature phase.
iSER-assisted mode MUST be enabled only if it is negotiated on the
leading connection during the LoginOperationalNegotiation Stage of
the iSCSI Login Phase. iSER-assisted mode is negotiated using the
RDMAExtensions=<boolean-value> key. Both the initiator and the
target MUST exchange the RDMAExtensions key with the value set to
"Yes" to enable iSER-assisted mode. If both the initiator and the
target fail to negotiate the RDMAExtensions key set to "Yes", then
the connection MUST continue with the login semantics as defined in
[RFC3720]. If the RDMAExtensions key is not negotiated to Yes, then
for some RCaP implementation (such as [IB]), the connection may need
to be re-established in TCP capable mode. (For InfiniBand this will
require an [IPoIB] type connection.)
iSER-assisted mode is defined for a Normal session only and the
RDMAExtensions key MUST NOT be negotiated for a Discovery session.
Discovery sessions are always conducted using the transport layer as
described in [RFC3720].
An iSER enabled node is not required to initiate the RDMAExtensions
key exchange if its preference is for the Traditional iSCSI mode.
The RDMAExtensions key, if offered, MUST be sent in the first
available Login Response or Login Request PDU in the
LoginOperationalNegotiation stage. This is due to the fact that the
value of some login parameters might depend on whether iSER-assisted
mode is enabled or not.
iSER-assisted mode is a session-wide attribute. If both the
initiator and the target negotiated RDMAExtensions="Yes" on the
leading connection of a session, then all subsequent connections of
the same session MUST enable iSER-assisted mode without having to
exchange RDMAExtensions key during the iSCSI Login Phase.
Ko et al. Expires May 2007 32
Internet-Draft iSER Specification November 2006
Conversely, if both the initiator and the target failed to negotiate
RDMAExtensions to "Yes" on the leading connection of a session, then
the RDMAExtensions key MUST NOT be negotiated further on any
additional subsequent connection of the session.
When the RDMAExtensions key is negotiated to "Yes", the HeaderDigest
and the DataDigest keys MUST be negotiated to "None" on all
iSCSI/iSER connections participating in that iSCSI session. This is
because, for an iSCSI/iSER connection, RCaP is responsible for
providing error detection that is at least as good as a 32-bit CRC
for all iSER Messages. Furthermore, all SCSI Read data are sent
using RDMA Write Messages instead of the SCSI Data-in PDUs, and all
solicited SCSI write data are sent using RDMA Read Response Messages
instead of the SCSI Data-out PDUs. HeaderDigest and DataDigest
which apply to iSCSI PDUs would not be appropriate for RDMA Read and
RDMA Write operations used with iSER.
5.1.1 Initiator Behavior
If the outcome of the iSCSI negotiation is to enable iSER-assisted
mode, then on the initiator side, prior to sending the Login Request
with the T (Transit) bit set to 1 and the NSG (Next Stage) field set
to FullFeaturePhase, the iSCSI Layer MUST request the iSER Layer to
allocate the connection resources necessary to support RCaP by
invoking the Allocate_Connection_Resources Operational Primitive.
The connection resources required are defined by implementation and
are outside the scope of this specification. The iSCSI Layer may
invoke the Notice_Key_Values Operational Primitive before invoking
the Allocate_Connection_Resources Operational Primitive to request
the iSER Layer to take note of the negotiated values of the iSCSI
keys for the Connection. The specific keys to be passed in as input
qualifiers are implementation dependent. These may include, but not
limited to, MaxOutstandingR2T, ErrorRecoveryLevel, etc.
To minimize the potential for a denial of service attack, the iSCSI
Layer MUST NOT request the iSER Layer to allocate the connection
resources necessary to support RCaP until the iSCSI layer is
sufficiently far along in the iSCSI Login Phase that it is
reasonably certain that the peer side is not an attacker. In
particular, if the Login Phase includes a SecurityNegotiation stage,
the iSCSI Layer MUST defer the connection resource allocation (i.e.
invoking the Allocate_Connection_Resources Operational Primitive) to
the LoginOperationalNegotiation stage ([RFC3720]) so that the
resource allocation occurs after the authentication phase is
completed.
Ko et al. Expires May 2007 33
Internet-Draft iSER Specification November 2006
Among the connection resources allocated at the initiator is the
Inbound RDMA Read Queue Depth (IRD). As described in section 9.5.1,
R2Ts are transformed by the target into RDMA Read operations. IRD
limits the maximum number of simultaneously incoming outstanding
RDMA Read Requests per an RCaP Stream from the target to the
initiator. The required value of IRD is outside the scope of the
iSER specification. The iSER Layer at the initiator MUST set IRD to
1 or higher if R2Ts are to be used in the connection. However, the
iSER Layer at the initiator MAY set IRD to 0 based on implementation
configuration which indicates that no R2Ts will be used on that
connection. Initially, the iSER-IRD value at the initiator SHOULD
be set to the IRD value at the initiator and MUST NOT be more than
the IRD value.
On the other hand, the Outbound RDMA Read Queue Depth (ORD) MAY be
set to 0 since the iSER Layer at the initiator does not issue RDMA
Read Requests to the target.
Failure to allocate the requested connection resources locally
results in a login failure and its handling is described in section
10.1.3.1.
If the iSER Layer at the initiator is successful in allocating the
connection resources necessary to support RCaP, the following events
MUST occur in the specified sequence:
1. The iSER Layer MUST return a success status to the iSCSI Layer
in response to the Allocate_Connection_Resources Operational
Primitive.
2. After the target returns the Login Response with the T bit set
to 1 and the NSG field set to FullFeaturePhase, and a status
class of 0 (Success), the iSCSI Layer MUST request the iSER
Layer to transition to iSER-assisted mode by invoking the
Enable_Datamover Operational Primitive with the following
qualifiers. (See section 10.1.4.6 for the case when the status
class is not Success.):
a. Connection_Handle that identifies the iSCSI connection.
b. Transport_Connection_Descriptor which identifies the
specific transport connection associated with the
Connection_Handle.
3. If necessary, the iSER Layer should enable RCaP and transition
the connection to iSER-assisted mode. When the RCaP is iWARP,
Ko et al. Expires May 2007 34
Internet-Draft iSER Specification November 2006
then this step MUST be done. Not all RCaPs may need it
depending on the RCaP Stream start-up state.
4. The iSER Layer MUST send the iSER Hello Message as the first
iSER Message. See Section 5.1.3 on iSER Hello Exchange.
5.1.2 Target Behavior
If the outcome of the iSCSI negotiation is to enable iSER-assisted
mode, then on the target side, prior to sending the Login Response
with the T (Transit) bit set to 1 and the NSG (Next Stage) field set
to FullFeaturePhase, the iSCSI Layer MUST request the iSER Layer to
allocate the resources necessary to support RCaP by invoking the
Allocate_Connection_Resources Operational Primitive. The connection
resources required are defined by implementation and are outside the
scope of this specification. Optionally, the iSCSI Layer may invoke
the Notice_Key_Values Operational Primitive before invoking the
Allocate_Connection_Resources Operational Primitive to request the
iSER Layer to take note of the negotiated values of the iSCSI keys
for the Connection. The specific keys to be passed in as input
qualifiers are implementation dependent. These may include, but not
limited to, MaxOutstandingR2T, ErrorRecoveryLevel, etc.
To minimize the potential for a denial of service attack, the iSCSI
Layer MUST NOT request the iSER Layer to allocate the connection
resources necessary to support RCaP until the iSCSI layer is
sufficiently far along in the iSCSI Login Phase that it is
reasonably certain that the peer side is not an attacker. In
particular, if the Login Phase includes a SecurityNegotiation stage,
the iSCSI Layer MUST defer the connection resource allocation (i.e.
invoking the Allocate_Connection_Resources Operational Primitive) to
the LoginOperationalNegotiation stage ([RFC3720]) so that the
resource allocation occurs after the authentication phase is
completed.
Among the connection resources allocated at the target is the
Outbound RDMA Read Queue Depth (ORD). As described in section
9.5.1, R2Ts are transformed by the target into RDMA Read operations.
The ORD limits the maximum number of simultaneously outstanding RDMA
Read Requests per RCaP Stream from the target to the initiator.
Initially, the iSER-ORD value at the target SHOULD be set to the ORD
value at the target.
On the other hand, the IRD at the target MAY be set to 0 since the
iSER Layer at the target does not expect RDMA Read Requests to be
issued by the initiator.
Ko et al. Expires May 2007 35
Internet-Draft iSER Specification November 2006
Failure to allocate the requested connection resources locally
results in a login failure and its handling is described in section
10.1.3.1.
If the iSER Layer at the target is successful in allocating the
connection resources necessary to support RCaP, the following events
MUST occur in the specified sequence:
1. The iSER Layer MUST return a success status to the iSCSI Layer
in response to the Allocate_Connection_Resources Operational
Primitive.
2. The iSCSI Layer MUST request the iSER Layer to transition to
iSER-assisted mode by invoking the Enable_Datamover Operational
Primitive with the following qualifiers:
a. Connection_Handle that identifies the iSCSI connection.
b. Transport_Connection_Descriptor which identifies the
specific transport connection associated with the
Connection_Handle.
c. The final transport layer (e.g. TCP) message containing the
Login Response with the T bit set to 1 and the NSG field set
to FullFeaturePhase
3. The iSER Layer MUST send the final Login Response PDU in the
native transport mode to conclude the iSCSI Login Phase. If the
underlying transport is TCP, then the iSER Layer MUST send the
final Login Response PDU in byte stream mode.
4. After sending the final Login Response PDU, the iSER Layer
should enable RCaP if necessary and transition the connection to
iSER-assisted mode. When the RCaP is iWARP, then this step MUST
be done. Not all RCaPs may need it depending on the RCaP Stream
start-up state.
5. After receiving the iSER Hello Message from the initiator, the
iSER Layer MUST respond with the iSER HelloReply Message to be
sent as the first iSER Message. See section 5.1.3 on iSER Hello
Exchange for more details.
Note: In the above sequence, the operations as described in bullets
3 and 4 MUST be performed atomically for iWARP connections. Failure
to do this may result in race conditions.
Ko et al. Expires May 2007 36
Internet-Draft iSER Specification November 2006
5.1.3 iSER Hello Exchange
After the connection transitions into the iSER-assisted mode, the
first iSER Message sent by the iSER Layer at the initiator to the
target MUST be the iSER Hello Message. The iSER Hello Message is
used by the iSER Layer at the initiator to declare iSER parameters
to the target. See section 9.3 on iSER Header Format for iSER Hello
Message.
In response to the iSER Hello Message, the iSER Layer at the target
MUST return the iSER HelloReply Message as the first iSER Message
sent by the target. The iSER HelloReply Message is used by the iSER
Layer at the target to declare iSER parameters to the initiator.
See section 9.4 on iSER Header Format for iSER HelloReply Message.
In the iSER Hello Message, the iSER Layer at the initiator declares
the iSER-IRD value to the target.
Upon receiving the iSER Hello Message, the iSER Layer at the target
MUST set the iSER-ORD value to the minimum of the iSER-ORD value at
the target and the iSER-IRD value declared by the initiator. The
iSER Layer at the target MAY adjust (lower) its ORD value to match
the iSER-ORD value if the iSER-ORD value is smaller than the ORD
value at the target in order to free up the unused resources.
In the iSER HelloReply Message, the iSER Layer at the target
declares the iSER-ORD value to the initiator.
Upon receiving the iSER HelloReply Message, the iSER Layer at the
initiator MAY adjust (lower) its IRD value to match the iSER-ORD
value in order to free up the unused resources, if the iSER-ORD
value declared by the target is smaller than the iSER-IRD value
declared by the initiator.
It is an iSER level negotiation failure if the iSER parameters
declared in the iSER Hello Message by the initiator are unacceptable
to the target. This includes the following:
* The initiator-declared iSER-IRD value is greater than 0 and the
target-declared iSER-ORD value is 0.
* The initiator-supported and the target-supported iSER protocol
versions do not overlap.
See section 10.1.3.2 on the handling of the error situation.
Ko et al. Expires May 2007 37
Internet-Draft iSER Specification November 2006
5.2 iSCSI/iSER Connection Termination
5.2.1 Normal Connection Termination at the Initiator
The iSCSI Layer at the initiator terminates an iSCSI/iSER connection
normally by invoking the Send_Control Operational Primitive
qualified with the Logout Request PDU. The iSER Layer at the
initiator MUST use a SendSE Message to send the Logout Request PDU
to the target. After the iSER Layer at the initiator receives the
SendSE Message containing the Logout Response PDU from the target,
it MUST notify the iSCSI Layer by invoking the Control_Notify
Operational Primitive qualified with the Logout Response PDU.
After the iSCSI logout process is complete, the iSCSI layer at the
target is responsible for closing the iSCSI/iSER connection as
described in Section 5.2.2. After the RCaP layer at the initiator
reports that the Connection has been closed, the iSER Layer at the
initiator MUST deallocate all connection and task resources (if any)
associated with the connection, invalidate the Local Mapping(s) (if
any) that associate the ITT(s) used on that connection to the local
STag(s) before notifying the iSCSI Layer by invoking the
Connection_Terminate_Notify Operational Primitive.
5.2.2 Normal Connection Termination at the Target
Upon receiving the SendSE Message containing the Logout Request PDU,
the iSER Layer at the target MUST notify the iSCSI Layer at the
target by invoking the Control_Notify Operational Primitive
qualified with the Logout Request PDU. The iSCSI Layer completes
the logout process by invoking the Send_Control Operational
Primitive qualified with the Logout Response PDU. The iSER Layer at
the target MUST use a SendSE Message to send the Logout Response PDU
to the initiator. After the iSCSI logout process is complete, the
iSCSI Layer at the target MUST request the iSER Layer at the target
to terminate the RCaP Stream by invoking the Connection_Terminate
Operational Primitive.
As part of the termination process, the RCaP layer MUST close the
Connection. When the RCaP layer notifies the iSER Layer after the
RCaP Stream and the associated Connection are terminated, the iSER
Layer MUST deallocate all connection and task resources (if any)
associated with the connection, and invalidate the Local and Remote
Mapping(s) (if any) that associate the ITT(s) used on that
connection to the local STag(s) and the Advertised STag(s)
respectively.
Ko et al. Expires May 2007 38
Internet-Draft iSER Specification November 2006
5.2.3 Termination without Logout Request/Response PDUs
5.2.3.1 Connection Termination Initiated by the iSCSI Layer
The Connection_Terminate Operational Primitive MAY be invoked by the
iSCSI Layer to request the iSER Layer to terminate the RCaP Stream
without having previously exchanged the Logout Request and Logout
Response PDUs between the two iSCSI/iSER nodes. As part of the
termination process, the RCaP layer will close the Connection. When
the RCaP layer notifies the iSER Layer after the RCaP Stream and the
associated Connection are terminated, the iSER Layer MUST perform
the following actions.
If the Connection_Terminate Operational Primitive is invoked by the
iSCSI Layer at the target, then the iSER Layer at the target MUST
deallocate all connection and task resources (if any) associated
with the connection, and invalidate the Local and Remote Mappings
(if any) that associate the ITT(s) used on the connection to the
local STag(s) and the Advertised STag(s) respectively.
If the Connection_Terminate Operational Primitive is invoked by the
iSCSI Layer at the initiator, then the iSER Layer at the initiator
MUST deallocate all connection and task resources (if any)
associated with the connection, and invalidate the Local Mapping(s)
(if any) that associate the ITT(s) used on the connection to the
local STag(s).
5.2.3.2 Connection Termination Notification to the iSCSI Layer
If the iSCSI/iSER connection is terminated without the invocation of
Connection_Terminate from the iSCSI Layer, the iSER Layer MUST
notify the iSCSI Layer that the iSCSI/iSER connection has been
terminated by invoking the Connection_Terminate_Notify Operational
Primitive.
Prior to invoking Connection_Terminate_Notify, the iSER Layer at the
target MUST deallocate all connection and task resources (if any)
associated with the connection, and invalidate the Local and Remote
Mappings (if any) that associate the ITT(s) used on the connection
to the local STag(s) and the Advertised STag(s) respectively.
Prior to invoking Connection_Terminate_Notify, the iSER Layer at the
initiator MUST deallocate all connection and task resources (if any)
associated with the connection, and invalidate the Local Mappings
(if any) that associate the ITT(s) used on the connection to the
local STag(s).
Ko et al. Expires May 2007 39
Internet-Draft iSER Specification November 2006
If the remote iSCSI/iSER node initiated the closing of the
Connection (e.g., by sending a TCP FIN or TCP RST), the iSER Layer
MUST notify the iSCSI Layer after the RCaP layer reports that the
Connection is closed by invoking the Connection_Terminate_Notify
Operational Primitive.
Another example of a Connection termination without a preceding
logout is when the iSCSI Layer at the initiator does an implicit
logout (connection reinstatement).
Ko et al. Expires May 2007 40
Internet-Draft iSER Specification November 2006
6 Login/Text Operational Keys
Certain iSCSI login/text operational keys have restricted usage in
iSER, and additional keys are used to support the iSER protocol
functionality. All other keys defined in [RFC3720] and not
discussed in this section may be used on iSCSI/iSER connections with
the same semantics.
6.1 HeaderDigest and DataDigest
Irrelevant when: RDMAExtensions=Yes
Negotiations resulting in RDMAExtensions=Yes for a session implies
HeaderDigest=None and DataDigest=None for all connections in that
session and overrides both the default and an explicit setting.
6.2 MaxRecvDataSegmentLength
For an iSCSI connection belonging to a session in which
RDMAExtensions=Yes was negotiated on the leading connection of the
session, MaxRecvDataSegmentLength need not be declared in the Login
Phase. Instead InitiatorRecvDataSegmentLength (as described in
section 6.5) and TargetRecvDataSegmentLength (as described in
section 6.4) keys are negotiated. The values of the local and
remote MaxRecvDataSegmentLength are derived from the
InitiatorRecvDataSegmentLength and TargetRecvDataSegmentLength keys
even if the MaxRecvDataSegmentLength was declared during the login
phase.
In the full feature phase, the initiator MUST consider the value of
its local MaxRecvDataSegmentLength (that it would have declared to
the target) as having the value of InitiatorRecvDataSegmentLength,
and the value of the remote MaxRecvDataSegmentLength (that would
have been declared by the target) as having the value of
TargetRecvDataSegmentLength. Similarly, the target MUST consider
the value of its local MaxRecvDataSegmentLength (that it would have
declared to the initiator) as having the value of
TargetRecvDataSegmentLength, and the value of the remote
MaxRecvDataSegmentLength (that would have been declared by the
initiator) as having the value of InitiatorRecvDataSegmentLength.
The MaxRecvDataSegmentLength key is applicable only for iSCSI
control-type PDUs.
6.3 RDMAExtensions
Use: LO (leading only)
Ko et al. Expires May 2007 41
Internet-Draft iSER Specification November 2006
Senders: Initiator and Target
Scope: SW (session-wide)
RDMAExtensions=<boolean-value>
Irrelevant when: SessionType=Discovery
Default is No
Result function is AND
This key is used by the initiator and the target to negotiate the
support for iSER-assisted mode. To enable the use of iSER-assisted
mode, both the initiator and the target MUST exchange
RDMAExtensions=Yes. iSER-assisted mode MUST NOT be used if either
the initiator or the target offers RDMAExtensions=No.
An iSER-enabled node is not required to initiate the RDMAExtensions
key exchange if it prefers to operate in the Traditional iSCSI mode.
However, if the RDMAExtensions key is to be negotiated, an initiator
MUST offer the key in the first Login Request PDU in the
LoginOperationalNegotiation stage of the leading connection, and a
target MUST offer the key in the first Login Response PDU with which
it is allowed to do so (i.e., the first Login Response PDU issued
after the first Login Request PDU with the C bit set to 0) in the
LoginOperationalNegotiation stage of the leading connection. In
response to the offered key=value pair of RDMAExtensions=yes, an
initiator MUST respond in the next Login Request PDU with which it
is allowed to do so, and a target MUST respond in the next Login
Response PDU with which it is allowed to do so.
Negotiating the RDMAExtensions key first enables a node to negotiate
the optimal value for other keys. Certain iSCSI keys such as
MaxBurstLength, MaxOutstandingR2T, ErrorRecoveryLevel, InitialR2T,
ImmediateData, etc., may be negotiated differently depending on
whether connection is in Traditional iSCSI mode or iSER-assisted
mode.
6.4 TargetRecvDataSegmentLength
Use: IO (Initialize only)
Senders: Initiator and Target
Scope: CO (connection-only)
Ko et al. Expires May 2007 42
Internet-Draft iSER Specification November 2006
Irrelevant when: RDMAExtensions=No
TargetRecvDataSegmentLength=<numerical-value-512-to-(2**24-1)>
Default is 8192 bytes
Result function is minimum
This key is relevant only for the iSCSI connection of an iSCSI
session if RDMAExtensions=Yes was negotiated on the leading
connection of the session. It is used by the initiator and the
target to negotiate the maximum size of the data segment that an
initiator may send to the target in an iSCSI control-type PDU in the
full feature phase. For SCSI Command PDUs and SCSI Data-out PDUs
containing non-immediate unsolicited data to be sent by the
initiator, the initiator MUST send all non-Final PDUs with a data
segment size of exactly TargetRecvDataSegmentLength whenever the
PDUs constitute a data sequence whose size is larger than
TargetRecvDataSegmentLength.
6.5 InitiatorRecvDataSegmentLength
Use: IO (Initialize only)
Senders: Initiator and Target
Scope: CO (connection-only)
Irrelevant when: RDMAExtensions=No
InitiatorRecvDataSegmentLength=<numerical-value-512-to-(2**24-1)>
Default is 8192 bytes
Result function is minimum
This key is relevant only for the iSCSI connection of an iSCSI
session if RDMAExtensions=Yes was negotiated on the leading
connection of the session. It is used by the initiator and the
target to negotiate the maximum size of the data segment that a
target may send to the initiator in an iSCSI control-type PDU in the
full feature phase.
6.6 OFMarker and IFMarker
Irrelevant when: RDMAExtensions=Yes
Ko et al. Expires May 2007 43
Internet-Draft iSER Specification November 2006
Negotiations resulting in RDMAExtensions=Yes for a session implies
OFMarker=No and IFMarker=No for all connections in that session and
overrides both the default and an explicit setting.
6.7 MaxOutstandingUnexpectedPDUs
Use: LO (leading only), Declarative
Senders: Initiator and Target
Scope: SW (session-wide)
Irrelevant when: RDMAExtensions=No
MaxOutstandingUnexpectedPDUs=<numerical-value-from-2-to-(2**32-1) |
0>
Default is 0
This key is used by the initiator and the target to declare the
maximum number of outstanding "unexpected" iSCSI control-type PDUs
that it can receive in the full feature phase. It is intended to
allow the receiving side to determine the amount of buffer resources
needed beyond the normal flow control mechanism available in iSCSI.
An initiator or target should select a value such that it would not
impose an unnecessary constraint on the iSCSI Layer under normal
circumstances. The value of 0 is defined to indicate that the
declarer has no limit on the maximum number of outstanding
"unexpected" iSCSI control-type PDUs that it can receive. See
sections 8.1.1 and 8.1.2 for the usage of this key. Note that iSER
Hello and HelloReply Messages are not iSCSI control-type PDUs and
are not affected by this key.
Ko et al. Expires May 2007 44
Internet-Draft iSER Specification November 2006
7 iSCSI PDU Considerations
When a connection is in the iSER-assisted mode, two types of message
transfers are allowed between the iSCSI Layer at the initiator and
the iSCSI Layer at the target. These are known as the iSCSI data-
type PDUs and the iSCSI control-type PDUs and these terms are
described in the following sections.
7.1 iSCSI Data-Type PDU
An iSCSI data-type PDU is defined as an iSCSI PDU that causes data
transfer, transparent to the remote iSCSI layer, to take place
between the peer iSCSI nodes in the full feature phase of an
iSCSI/iSER connection. An iSCSI data-type PDU, when requested for
transmission by the iSCSI Layer in the sending node, results in the
data being transferred without the participation of the iSCSI Layers
at the sending and the receiving nodes. This is due to the fact
that the PDU itself is not delivered as-is to the iSCSI Layer in the
receiving node. Instead, the data transfer operations are
transformed into the appropriate RDMA operations which are handled
by the RDMA-Capable Controller. The set of iSCSI data-type PDUs
consists of SCSI Data-in PDUs and R2T PDUs.
If the invocation of the Operational Primitive by the iSCSI Layer to
request the iSER Layer to process an iSCSI data-type PDU is
qualified with Notify_Enable set, then upon completing the RDMA
operation, the iSER Layer at the target MUST notify the iSCSI Layer
at the target by invoking the Data_Completion_Notify Operational
Primitive qualified with ITT and SN. There is no data completion
notification at the initiator since the RDMA operations are
completely handled by the RDMA-Capable Controller at the initiator
and the iSER Layer at the initiator is not involved with the data
transfer associated with iSCSI data-type PDUs.
If the invocation of the Operational Primitive by the iSCSI Layer to
request the iSER Layer to process an iSCSI data-type PDU is
qualified with Notify_Enable cleared, then upon completing the RDMA
operation, the iSER Layer at the target MUST NOT notify the iSCSI
Layer at the target and MUST NOT invoke the Data_Completion_Notify
Operational Primitive.
If an operation associated with an iSCSI data-type PDU fails for any
reason, the contents of the Data Sink buffers associated with the
operation are considered indeterminate.
Ko et al. Expires May 2007 45
Internet-Draft iSER Specification November 2006
7.2 iSCSI Control-Type PDU
Any iSCSI PDU that is not an iSCSI data-type PDU and also not a SCSI
Data-out PDU carrying solicited data is defined as an iSCSI control-
type PDU. The iSCSI Layer invokes the Send_Control Operational
Primitive to request the iSER Layer to process an iSCSI control-type
PDU. iSCSI control-type PDUs are transferred using Send Message
Types of RCaP. Specifically, it is to be noted that SCSI Data-Out
PDUs carrying unsolicited data are defined as iSCSI control-type
PDUs. See section 7.3.4 on the treatment of SCSI Data-out PDUs.
When the iSER Layer receives an iSCSI control-type PDU, it MUST
notify the iSCSI Layer by invoking the Control_Notify Operational
Primitive qualified with the iSCSI control-type PDU.
7.3 iSCSI PDUs
This section describes the handling of each of the iSCSI PDU types
by the iSER Layer. The iSCSI Layer requests the iSER Layer to
process the iSCSI PDU by invoking the appropriate Operational
Primitive. A Connection_Handle MUST qualify each of these
invocations. In addition, BHS and the optional AHS of the iSCSI PDU
as defined in [RFC3720] MUST qualify each of the invocations. The
qualifying Connection_Handle, the BHS and the AHS are not explicitly
listed in the subsequent sections.
7.3.1 SCSI Command
Type: control-type PDU
PDU-specific qualifiers (for SCSI Write or bidirectional
command): ImmediateDataSize, UnsolicitedDataSize,
DataDescriptorOut
PDU-specific qualifiers (for SCSI Read or bidirectional
command): DataDescriptorIn
The iSER Layer at the initiator MUST send the SCSI command in a
SendSE Message to the target.
For a SCSI Write or bidirectional command, the iSCSI Layer at the
initiator MUST invoke the Send_Control Operational Primitive as
follows:
* If there is immediate data to be transferred for the SCSI write
or bidirectional command, the qualifier ImmediateDataSize MUST be
used to define the number of bytes of immediate unsolicited data
Ko et al. Expires May 2007 46
Internet-Draft iSER Specification November 2006
to be sent with the write or bidirectional command, and the
qualifier DataDescriptorOut MUST be used to define the
initiator's I/O Buffer containing the SCSI Write data.
* If there is unsolicited data to be transferred for the SCSI Write
or bidirectional command, the qualifier UnsolicitedDataSize MUST
be used to define the number of bytes of immediate and non-
immediate unsolicited data for the command. The iSCSI Layer will
issue one or more SCSI Data-out PDUs for the non-immediate
unsolicited data. See Section 7.3.4 on SCSI Data-out.
* If there is solicited data to be transferred for the SCSI Write
or bidirectional command, as indicated by the Expected Data
Transfer Length in the SCSI Command PDU exceeding the value of
UnsolicitedDataSize, the iSER Layer at the initiator MUST do the
following:
a. It MUST allocate a Write STag for the I/O Buffer defined by
the qualifier DataDescriptorOut. DataDescriptorOut
describes the I/O buffer starting with the immediate
unsolicited data (if any), followed by the non-immediate
unsolicited data (if any) and solicited data. This means
that the BufferOffset for the SCSI Data-out for this command
is equal to the TO. This implies zero TO for this STag
points to the beginning of this I/O Buffer.
b. It MUST establish a Local Mapping that associates the
Initiator Task Tag (ITT) to the Write STag.
c. It MUST Advertise the Write STag to the target by sending it
as the Write STag in the iSER header of the iSER Message
(the payload of the SendSE Message of RCaP) containing the
SCSI Write or bidirectional command PDU. See section 9.2 on
iSER Header Format for iSCSI Control-Type PDU.
For a SCSI Read or bidirectional command, the iSCSI Layer at the
initiator MUST invoke the Send_Control Operational Primitive
qualified with DataDescriptorIn which defines the initiator's I/O
Buffer for receiving the SCSI Read data. The iSER Layer at the
initiator MUST do the following:
a. It MUST allocate a Read STag for the I/O Buffer.
b. It MUST establish a Local Mapping that associates the
Initiator Task Tag (ITT) to the Read STag.
Ko et al. Expires May 2007 47
Internet-Draft iSER Specification November 2006
c. It MUST Advertise the Read STag to the target by sending it
as the Read STag in the iSER header of the iSER Message (the
payload of the SendSE Message of RCaP) containing the SCSI
Read or bidirectional command PDU. See section 9.2 on iSER
Header Format for iSCSI Control-Type PDU.
If the amount of unsolicited data to be transferred in a SCSI
Command exceeds TargetRecvDataSegmentLength, then the iSCSI Layer at
the initiator MUST segment the data into multiple iSCSI control-type
PDUs, with the data segment length in all PDUs generated except the
last one having exactly the size TargetRecvDataSegmentLength. The
data segment length of the last iSCSI control-type PDU carrying the
unsolicited data can be up to TargetRecvDataSegmentLength.
When the iSER Layer at the target receives the SCSI Command, it MUST
establish a Remote Mapping that associates the ITT to the Advertised
Write STag and the Read STag if present in the iSER header. The
Write STag is used by the iSER Layer at the target in handling the
data transfer associated with the R2T PDU(s) as described in section
7.3.6. The Read STag is used in handling the SCSI Data-in PDU(s)
from the iSCSI Layer at the target as described in section 7.3.5.
7.3.2 SCSI Response
Type: control-type PDU
PDU-specific qualifiers: DataDescriptorStatus
The iSCSI Layer at the target MUST invoke the Send_Control
Operational Primitive qualified with DataDescriptorStatus which
defines the buffer containing the sense and response information.
The iSCSI Layer at the target MUST always return the SCSI status for
a SCSI command in a separate SCSI Response PDU. "Phase collapse"
for transferring SCSI status in a SCSI Data-in PDU MUST NOT be used.
The iSER Layer at the target sends the SCSI Response PDU according
to the following rules:
* If no STags were Advertised by the initiator in the iSER Message
containing the SCSI command PDU, then the iSER Layer at the
target MUST send a SendSE Message containing the SCSI Response
PDU.
* If the initiator Advertised a Read STag in the iSER Message
containing the SCSI Command PDU, then the iSER Layer at the
target MUST send a SendInvSE Message containing the SCSI Response
PDU. The header of the SendInvSE Message MUST carry the Read
STag to be invalidated at the initiator.
Ko et al. Expires May 2007 48
Internet-Draft iSER Specification November 2006
* If the initiator Advertised only the Write STag in the iSER
Message containing the SCSI command PDU, then the iSER Layer at
the target MUST send a SendInvSE Message containing the SCSI
Response PDU. The header of the SendInvSE Message MUST carry the
Write STag to be invalidated at the initiator.
When the iSCSI Layer at the target invokes the Send_Control
Operational Primitive to send the SCSI Response PDU, the iSER Layer
at the target MUST invalidate the Remote Mapping that associates the
ITT to the Advertised STag(s) before transferring the SCSI Response
PDU to the initiator.
Upon receiving the SendInvSE Message containing the SCSI Response
PDU from the target, the RCaP layer at the initiator will invalidate
the STag specified in the header. The iSER Layer at the initiator
MUST ensure that the correct STag is invalidated. If both the Read
and the Write STags were Advertised earlier by the initiator, then
the iSER Layer at the initiator MUST explicitly invalidate the Write
STag upon receiving the SendInvSE Message because the header of the
SendInvSE Message can only carry one STag (in this case the Read
STag) to be invalidated.
The iSER Layer at the initiator MUST ensure the invalidation of the
STag(s) used in a command before notifying the iSCSI Layer at the
initiator by invoking the Control_Notify Operational Primitive
qualified with the SCSI Response. This precludes the possibility of
using the STag(s) after the completion of the command thereby
causing data corruption.
When the iSER Layer at the initiator receives the SendSE or the
SendInvSE Message containing the SCSI Response PDU, it SHOULD
invalidate the Local Mapping that associates the ITT to the local
STag(s). The iSER Layer MUST ensure that all local STag(s)
associated with the ITT are invalidated before notifying the iSCSI
Layer of the SCSI Response PDU by invoking the Control_Notify
Operational Primitive qualified with the SCSI Response PDU.
7.3.3 Task Management Function Request/Response
Type: control-type PDU
PDU-specific qualifiers (for TMF Request): DataDescriptorOut,
DataDescriptorIn
The iSER Layer MUST use a SendSE Message to send the Task Management
Function Request/Response PDU.
Ko et al. Expires May 2007 49
Internet-Draft iSER Specification November 2006
For the Task Management Function Request with the TASK REASSIGN
function, the iSER Layer at the initiator MUST do the following:
* It MUST use the ITT as specified in the Referenced Task Tag from
the Task Management Function Request PDU to locate the existing
STag(s), if any, in the Local Mapping(s) that associates the ITT
to the local STag(s).
* It MUST invalidate the existing STag(s), if any, and the Local
Mapping(s) that associates the ITT to the local STag(s).
* It MUST allocate a Read STag for the I/O Buffer as defined by the
qualifier DataDescriptorIn if the Send_Control Operational
Primitive invocation is qualified with DataDescriptorIn.
* It MUST allocate a Write STag for the I/O Buffer as defined by
the qualifier DataDescriptorOut if the Send_Control Operational
Primitive invocation is qualified with DataDescriptorOut.
* If STags are allocated, it MUST establish new Local Mapping(s)
that associate the ITT to the allocated STag(s).
* It MUST Advertise the STags, if allocated, to the target in the
iSER header of the SendSE Message carrying the iSCSI PDU, as
described in section 9.2.
For the Task Management Function Request with the TASK REASSIGN
function for a SCSI Read or bidirectional command, the iSCSI Layer
at the initiator MUST set ExpDataSN to 0 since the data transfer and
acknowledgements happen transparently to the iSCSI Layer at the
initiator. This provides the flexibility to the iSCSI Layer at the
target to request transmission of only the unacknowledged data as
specified in [RFC3720].
When the iSER Layer at the target receives the Task Management
Function Request with the TASK REASSIGN function, it MUST do the
following:
* It MUST use the ITT as specified in the Referenced Task Tag from
the Task Management Function Request PDU to locate the mappings
that associate the ITT to the Advertised STag(s) and the local
STag(s), if any.
* It MUST invalidate the local STaq(s), if any, associated with the
ITT.
Ko et al. Expires May 2007 50
Internet-Draft iSER Specification November 2006
* It MUST replace the Advertised STag(s) in the Remote Mapping that
associates the ITT to the Advertised STag(s) with the Write STag
and the Read STag if present in the iSER header. The Write STag
is used in the handling of the R2T PDU(s) from the iSCSI Layer at
the target as described in section 7.3.6. The Read STag is used
in the handling of the SCSI Data-in PDU(s) from the iSCSI Layer
at the target as described in section 7.3.5.
7.3.4 SCSI Data-out
Type: control-type PDU
PDU-specific qual