/
githubmirror
/
rdma-core
Обзор
Документация
Войти
/
githubmirror
/
rdma-core
Код
Запросы
0
Пакеты
0
Релизы
0
Аналитика
Безопасность
master
librdmacm/docs/rsocket
280 строк
11 KB
Benjamin Drung
Fix spelling mistake of transferred
26 фев 2024, 20:32
26 фев 2024, 20:32
88f659c
Код
Авторство
О чём код?
.. Licensed under the OpenIB.org BSD license (FreeBSD Variant) - See COPYING.md rsocket Protocol and Design Guide 11/11/2012 Data Streaming (TCP) Overview ----------------------------- Rsockets is a protocol over RDMA that supports a socket-level API for applications. For details on the current state of the implementation, readers should refer to the rsocket man page. This document describes the rsocket protocol, general design, and some implementation details. Rsockets exchanges data by performing RDMA write operations into exposed data buffers. In addition to RDMA write data, rsockets uses small, 32-bit messages for internal communication. RDMA writes are used to transfer application data into remote data buffers and to notify the peer when new target data buffers are available. The following figure highlights the operation. host A host B remote SGL target SGL <------------- [ ] [ ] ------ [ ] -- ------ receive buffer(s) -- -----> +--+ -- | | -- | | -- | | -- +--+ -- ---> +--+ | | | | +--+ The remote SGL contains the address, size, and rkey of the target SGL. As receive buffers become available on host B, rsockets will issue an RDMA write against one of the entries in the target SGL on host A. The updated entry will reference an available receive buffer. Immediate data included with the RDMA write will indicate to host A that a target SGE has been updated. When host A has data to send, it will check its target SGL. The current target SGE will contain the address, size, and rkey of the next receive buffer on host B. If the data transfer is smaller than the size of the remote receive buffer, host A will update its target SGE to reflect the remaining size of the receive buffer. That is, once a receive buffer has been published to a remote peer, it will be fully consumed before a second buffer is used. Rsockets relies on immediate data to notify the remote peer when data has been transferred or when a target SGL has been updated. Because immediate data requires that the remote QP have a posted receive, rsockets also uses a credit based flow control mechanism. The number of credits is based on the size of the receive queue, with initial credits exchanged during connection setup. In order to transfer data, rsockets requires both available receive buffers (published via the target SGL) and data credits. Since immediate data is limited to 32-bits, messages may either indicate the arrival of application data or may be an internal message, but not both. To avoid credit deadlock, rsockets reserves a small number of available credits for control messages only, with the protocol relying on RNR NAKs and retries to make forward progress. Connection Establishment ------------------------ rsockets uses the RDMA CM for connection establishment. Struct rs_conn_data is exchanged during the connection exchange as private data in the request and reply messages. struct rs_sge { uint64_t addr; uint32_t key; uint32_t length; }; #define RS_CONN_FLAG_NET 1 struct rs_conn_data { uint8_t version; uint8_t flags; uint16_t credits; uint32_t reserved2; struct rs_sge target_sgl; struct rs_sge data_buf; }; Version - current version is 1 Flags RS_CONN_FLAG_NET - Set to 1 if host is big Endian. Determines byte ordering for RDMA write messages Credits - number of initial receive credits Reserved2 - set to 0 Target SGL - Address, size (# entries), and rkey of target SGL. Remote side will copy this into their remote SGL. Data Buffer - Initial receive buffer address, size (in bytes), and rkey. Remote side will copy this into their first target SGE. Message Format -------------- Rsocket uses RDMA writes with immediate data for all message exchanges. RDMA writes of 0 length are used if no additional data beyond the message needs to be exchanged. Immediate data is limited to 32-bits. Rsockets defines the following format for messages. The upper 3 bits are used to define the type of message being exchanged, with the meaning of the lower 29 bits determined by the upper bits. Bits Message Meaning of 31:29 Type Bits 28:0 000 Data Transfer bytes transferred 001 reserved 010 reserved - used internally, available for future use 011 reserved 100 Credit Update received credits granted 101 reserved 110 Iomap Updated index of updated entry 111 Control control message type Data Transfer Indicates that application data has been written into the next available receive buffer. The size of the transfer, in bytes, is carried in the lower bits of the message. Credit Update Used to indicate that additional receive buffers and credits are available. The number of available credits is carried in the lower bits of the message. A credit update message is also used to indicate that a target SGE has been updated, in which case the number of additional credits may be 0. The receiver of a credit update message must check for updates to the target SGL by inspecting the contents of the SGL. The rsocket implementation must take care not to modify a remote target SGL while it may be in use. This is done by tracking when a receive buffer referenced by a remote target SGL has been filled. Iomap Updated Used to indicate that a remote iomap entry was updated. The updated entry contains the offset value associated with an address, length, and rkey. Once an iomap has been updated, the local application can issue directed IO transfers against the corresponding remote buffer. Control Message - DISCONNECT Indicates that the rsocket connection has been fully disconnected and will no longer send or receive data. Data received before the disconnect message was processed may still be available for reading. Control Message - SHUTDOWN Indicates that the remote rsocket has shutdown the send side of its connection. The recipient of a shutdown message will no longer accept incoming data, but may still transfer outbound data. Iomapped Buffers ---------------- Rsockets allows for zero-copy transfers using what it refers to as iomapped buffers. Iomapping and direct data placement (zero-copy) transfers are done using rsocket specific extensions. The general operation is similar to that used for normal data transfers described above. host A host B remote iomap target iomap <----------- [ ] [ ] ------ [ ] -- ------ iomapped buffer(s) -- -----> +--+ -- | | -- | | -- | | -- +--+ -- ---> +--+ | | | | +--+ The remote iomap contains the address, size, and rkey of the target iomap. As the applicaton maps buffers host B to a given rsocket, rsockets will issue an RDMA write against one of the entries in the target iomap on host A. The updated entry will reference an available iomapped buffer. Immediate data included with the RDMA write will indicate to host A that a target iomap has been updated. When host A wishes to transfer directly into an iomapped buffer, it will check its target iomap for an offset corresponding to a remotely mapped buffer. A matching iomap entry will contain the address, size, and rkey of the target buffer on host B. Host A will then issue an RDMA operation against the registered remote data buffer. From host A's perspective, the transfer appears as a normal send/write operation, with the data stream redirected directly into the receiving application's buffer. Datagram Overview ----------------- The rsocket API supports datagram sockets. Datagram support is handled through an entirely different protocol and internal implementation. Unlike connected rsockets, datagram rsockets are not necessarily bound to a network (IP) address. A datagram socket may use any number of network (IP) addresses, including those which map to different RDMA devices. As a result, a single datagram rsocket must support using multiple RDMA devices and ports, and a datagram rsocket references a single UDP socket, plus zero or more UD QPs. Rsockets uses headers inserted before user data sent over UDP sockets to resolve remote UD QP numbers. When a user first attempts to send a datagram to a remote address (IP and UDP port), rsockets will take the following steps: 1. Store the destination address into a lookup table. 2. Resolve which local network address should be used when sending to the specified destination. 3. Allocate a UD QP on the RDMA device associated with the local address. 4. Send the user's datagram to the remote UDP socket. A header is inserted before the user's datagram. The header specifies the UD QP number associated with the local network address (IP and UDP port) of the send. A service thread is used to process messages received on the UDP socket. This thread updates the rsocket lookup tables with the remote QPN and path record data. The service thread forwards data received on the UDP socket to an rsocket QP. After the remote QPN and path records have been resolved, datagram communication between two nodes are done over the UD QP. UDP Message Format ------------------ Rsockets uses messages exchanged over UDP sockets to resolve remote QP numbers. If a user sends a datagram to a remote service and the local rsocket is not yet configured to send directly to a remote UD QP, the user data is sent over a UDP socket with the following header inserted before the user data. struct ds_udp_header { uint32_t tag; uint8_t version; uint8_t op; uint8_t length; uint8_t reserved; uint32_t qpn; /* lower 8-bits reserved */ union { uint32_t ipv4; uint8_t ipv6[16]; } addr; }; Tag - Marker used to help identify that the UDP header is present. #define DS_UDP_TAG 0x55555555 Version - IP address version, either 4 or 6 Op - Indicates message type, used to control the receiver's operation. Valid operations are RS_OP_DATA and RS_OP_CTRL. Data messages carry user data, while control messages are used to reply with the local QP number. Length - Size of the UDP header. QPN - UD QP number associated with sender's IP address and port. The sender's address and port is extracted from the received UDP datagram. Addr - Target IP address of the sent datagram. Once the remote QP information has been resolved, data is sent directly between UD QPs. The following header is inserted before any user data that is transferred over a UD QP. struct ds_header { uint8_t version; uint8_t length; uint16_t port; union { uint32_t ipv4; struct { uint32_t flowinfo; uint8_t addr[16]; } ipv6; } addr; }; Verion - IP address version Length - Size of the header Port - Associated source address UDP port Addr - Associated source IP address