DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 05/23/2025 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Claim Objections
Claims 3-10, 13-20 and 24 are objected to because of the following informalities:
“the address space” in line 4 of claim 3 should read as “an address space”.
“the send queue” in line 3 of claim 6 and “the doorbell register” in line 4 of claim 6 should read as “a send queue” and “a doorbell register”, respectively.
“the address space” in line 5 of claim 9 should read as “an address space”.
“remote direct memory access RDMA)” in line 3 of claim 13 should read as “remote direct memory access (RDMA)”
“the address space” in line 5 of claim 15 should read as “an address space”.
“or a send a command” in line 8 of claim 16 should read as “or send a command”.
“to perform the method claim 21” in lines 3-4 of claim 24 should read as “to perform the method of claim 21”.
Claims 4-5, 7-8, 10, 14, and 17-20 are objected to because they are dependent on the objected claims. Appropriate correction is required.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-4, 6-11, and 21 are rejected under 35 U.S.C. 103 as being unpatentable over Hefty (US 2013/0262613) in view of Chang (US 2009/0125604) and further in view of Nagasubramaniam (US 2017/0034269).
Regarding claim 1, Hefty teaches a first node (Fig. 1, Subnet manager 116), which is connected to a second node (Fig. 1, Node 102A) and one or more third nodes (Fig. 1, Node 102B) by remote direct memory access (RDMA) connection (Fig. 1, Network 126 uses RDMA; Paragraph 0012, Each network node element 102A, 102B, 102C, . . . , 102N and the SM 116 may communicate with each other, via network 126, using an remote direct memory access-capable (RDMA-capable) communications protocol), the first node comprising: a processor (Fig. 1, Network controller 118 of subnet manager 116); and a memory coupled to the processor and having processor-executable instruction stored thereon, which upon execution by the processor (Fig. 1, Subnet manager 116 has system memory; Paragraph 0029, System memory, e.g., system memory 106 and/or memory associated with the network controller, e.g., network controller 110, 118), cause the first node to: receive, from the second node, a read request for requested data (Fig. 2, Node 102A sends SA data query 206 related to read I/O transaction to subnet manager 116; Paragraph 0018, an SA data query 206 may be sent by the node 102A to the SM 116… Paragraph 0021, SA data query may be generated, for example, by an application being executed on the network node element. The application may use the SA data, for example, to communicate with another network node element to facilitate an I/O transaction (e.g., data read, data write, etc.) between the network node elements, and thus, the SA data may include, for example, the address of the other network node element); and wherein the first node is connected to the second node by a first RDMA connection (Fig. 1, Subnet manager 116 (i.e. first node) is connected to node 102A (i.e. second node) via a first RDMA connection between the 116 and 102A using shared/overlapping network 126), wherein the first node is connected to the one or more third nodes by a second RDMA connection (Fig. 1, Subnet manager 116 (i.e. first node) is connected to node 102B (i.e. third node) via second RDMA connection between 116 and 102B using shared/overlapping network 126); and wherein the second node is connected to the one or more third nodes by the third RDMA connection (Fig. 1, Node 102A (i.e. second node) is connected to node 102B (i.e. third nodes) via third RDMA connection between 102A and 102B using shared/overlapping network 126).
Hefty does not teach the first node comprising: based on the requested data not being available at the first node, determine at which of the third or more nodes the requested data is available; cause the determined third node to send the requested data.
Chang teaches the first node (Fig. 14, Managing node) comprising: determine at which of the third or more nodes the requested data is available (Fig. 14, Managing node “B” (i.e. first node) receives read request from data target “A” (i.e. second node), determines read data is on data source “C” (i.e. third node); Paragraph 0224, A node "A" in the cluster may make a file system read/write request. The request gets forwarded to one of the DLM table manager nodes "B." The DLM table manager handles the request, checks its tables and determines that Node "C" owns the blocks of data that node "A" requests access to); cause the determined third node to send the requested data over a third RDMA connection to the second node (Fig. 14, Managing node “B” (i.e. first node) causes a direct RDMA connection to be established to transfer the read data between the data target “A” and the data source “C” (i.e. third RDMA connection); Paragraph 0224, Node "B" can now send a third party RDMA transfer request to node "C" asking node "C" to send the requested data to node "A" and to notify node "B" when the transfer is complete).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Hefty’s first node to incorporate the teachings of Chang and allow for the subnet manager of Hefty to determine where data is located and allow for transfer of the data between the source and target via a direct RDMA connection.
One of ordinary skill in the art would be motivated to make the modifications in order to reduce the number of handshake and intermediary transfer operations required for data request operations (See Chang: Paragraphs 0224 and 0229).
Neither Hefty nor Chang teaches the first node comprising: based on the requested data not being available at the first node: determine at which of the one or more third nodes the requested data is available.
Nagasubramaniam teaches the first node (Fig. 3C, Point of contact node 12) comprising: based on the requested data not being available at the first node (Fig. 3C, Data is not available in volume 16 of point of contact node 12; Paragraph 0045, Upon determining that the requested data is not located at the local storage volume 16), determine at which of the third or more nodes the requested data is available (Fig. 3BC Point of contact node 12 (i.e. first node) determines data is available at remote node 14 (i.e. third node); Paragraph 0045, Upon determining that the requested data… is located at a remote storage volume 18).
Hefty, Chang, and Nagasubramaniam are analogous arts because they are in the same field of endeavor of performing RDMA data transfers between different network nodes.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Hefty/Chang’s first node to incorporate the teachings of Nagasubramaniam and enable the subnet manager to determine if the data is present in the memory of the subnet manager and to forward the read request if the data is not present.
One of ordinary skill in the art would be motivated to make the modifications in order to yield the obvious result of efficiently performing data querying and retrieval amongst a distributed network, thus improving storage capabilities and increasing scalability of memory systems.
Regarding claim 2, the combination of Hefty/Chang/Nagasubramaniam teaches the first node of claim 1. Hefty teaches the first node comprising wherein the read request (Fig. 2, Node 102A sends SA data query 206 related to read I/O transaction to subnet manager 116; Paragraph 0018, an SA data query 206 may be sent by the node 102A to the SM 116… to communicate with another network node element to facilitate an I/O transaction (e.g., data read) comprises information that identifies and grants access to a memory in the first node and a memory in the second node (Fig. 2, SA query 206 associated with read I/O transaction includes a key that grants access to the memory; Paragraph 0019, node 102A may update the SA data query 206 with RDMA write target buffer address, encryption key), an address space of the first node, and an address space of the second node (Fig. 1, System memory 106 of node 102A is an address space of a second node and system memory of subnet manager 116 is an address space of the first node; Paragraph 0014, system memory 106 may be controlled to provide an RDMA target write buffer 105 to enable an RDMA transfer from the SM 116 to the node… Paragraph 0029, memory associated with the network controller, e.g., network controller 110, 118).
Regarding claim 3, the combination of Hefty/Chang/Nagasubramaniam teaches the first node of claim 1.
Chang teaches the first node comprising wherein for causing the determined third node to provide the requested data to the second node, the first node is configured to trigger a write of the requested data by the determined third node into the address space of the second node (Fig. 14, Managing node (i.e. first node) sends third party RDMA read request to data source (i.e. third node) which causes data source to write data to data target (i.e. second node), wherein the data is stored in an address space; Paragraph 0052, packet transfers made directly from the memory address space of one data processing unit to the memory address space of one or more other data processing units… Paragraph 0226, When the data source adapter receives the Third Party RDMA Read Request packet and verifies the rCxt key, tid, etc., it swaps the node ids for the source and the data target, and initiates the operation to send the data to the data target adapter… All RDMA data packets sent contain the packet type of Third Party RDMA Write Request).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Hefty’s first node to incorporate the teachings of Chang and allow for the subnet manager of Hefty to determine where data is located and allow for transfer of the data between the source and target via a direct RDMA connection.
One of ordinary skill in the art would be motivated to make the modifications in order to reduce the number of handshake and intermediary transfer operations required for data request operations (See Chang: Paragraphs 0224 and 0229).
Regarding claim 4, the combination of Hefty/Chang/Nagasubramaniam teaches the first node of claim 3.
Chang teaches wherein the processor further executes the processor-executable instructions to cause the first node to: control the third RDMA connection from the determined third node to the second node (Fig. 14, Managing node (i.e. first node) sends third party RDMA read request to data source (i.e. third node) which controls data source to write data to data target (i.e. second node) over the RDMA connection between data source and data target (i.e. third RDMA connection); Paragraph 0225, this process involves controlling or managing a node so that it "knows" the available rCxt ids, current tids and the memory structure of the data source and data target nodes. The controlling or managing node constructs a Third Party RDMA Read Request packet).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Hefty’s first node to incorporate the teachings of Chang and allow for the subnet manager of Hefty to determine where data is located and allow for transfer of the data between the source and target via a direct RDMA connection.
One of ordinary skill in the art would be motivated to make the modifications in order to reduce the number of handshake and intermediary transfer operations required for data request operations (See Chang: Paragraphs 0224 and 0229).
Regarding claim 6, the combination of Hefty/Chang/Nagasubramaniam teaches the first node of claim 3.
Chang teaches the first node comprising wherein for triggering the write of the requested data by the third node, the first node is configured to send at least one RDMA write request to the address of the send queue at the third node and/or to the address of the doorbell register of this send queue (Fig. 23, Managing node has identifiers for send queue 203 of adapter 207 of node; Paragraph 0225, this process involves controlling or managing a node so that it "knows" the available rCxt ids, current tids and the memory structure of the data source and data target nodes… Paragraph 0128, submits an RDMA request with respect to HAL FIFO send queue 203).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Hefty’s first node to incorporate the teachings of Chang and allow for the subnet manager of Hefty to determine where data is located and allow for transfer of the data between the source and target via a direct RDMA connection.
One of ordinary skill in the art would be motivated to make the modifications in order to reduce the number of handshake and intermediary transfer operations required for data request operations (See Chang: Paragraphs 0224 and 0229).
Regarding claim 7, the combination of Hefty/Chang/Nagasubramaniam teaches the first node of claim 6. Hefty teaches the first node comprising wherein the at least one RDMA write request (Fig. 2, Node 102A sends SA data query 206 related to read I/O transaction to subnet manager 116; Paragraph 0018, an SA data query 206 may be sent by the node 102A to the SM 116… to communicate with another network node element to facilitate an I/O transaction (e.g., data read) comprises information that identifies and grants access to a memory in the first node and a memory in the second node (Fig. 2, SA query 206 associated with read I/O transaction includes a key that grants access to the memory; Paragraph 0019, node 102A may update the SA data query 206 with RDMA write target buffer address, encryption key), an address space of the first node, and an address space of the second node (Fig. 1, System memory 106 of node 102A is an address space of a second node and system memory of subnet manager 116 is an address space of the first node; Paragraph 0014, system memory 106 may be controlled to provide an RDMA target write buffer 105 to enable an RDMA transfer from the SM 116 to the node… Paragraph 0029, memory associated with the network controller, e.g., network controller 110, 118).
Regarding claim 8, the combination of Hefty/Chang/Nagasubramaniam teaches the first node of claim 6.
Chang teaches the first node comprising: wherein at least one RDMA write request comprises one of: a RDMA write-with-immediate request for the requested data and for a notification; a RDMA write request for the requested data and a send request for a notification; a RDMA write request for the requested data and a send-with-immediate request for a notification (Fig. 14, Managing node “B” (i.e. first node) causes a direct RDMA connection to be established to transfer the read data between the data target “C” and the data source “A” (i.e. third RDMA connection) and to further transfer a notification of the RDMA write request back to node B; Paragraph 0224, Node "B" can now send a third party RDMA transfer request to node "C" asking node "C" to send the requested data to node "A" and to notify node "B" when the transfer is complete).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Hefty’s first node to incorporate the teachings of Chang and allow for the subnet manager of Hefty to determine where data is located and allow for transfer of the data between the source and target via a direct RDMA connection and enable a notification to be sent between the nodes and the managing node.
One of ordinary skill in the art would be motivated to make the modifications in order to reduce the number of handshake and intermediary transfer operations required for data request operations (See Chang: Paragraphs 0224 and 0229).
Regarding claim 9, the combination of Hefty/Chang/Nagasubramaniam teaches the first node of claim 1.
Chang teaches the first node comprising wherein for causing the determined third node to provide the requested data to the second node, the first node is configured to send a read request to the determined third node, wherein the read request indicates the requested data is to be written by the determined third node into the address space of the second node (Fig. 14, Managing node (i.e. first node) sends third party RDMA read request to data source (i.e. third node) which causes data source to write data to data target (i.e. second node), wherein the data is to be stored in memory address space of data target (i.e. second node); Paragraph 0052, packet transfers made directly from the memory address space of one data processing unit to the memory address space of one or more other data processing units… Paragraph 0226, When the data source adapter receives the Third Party RDMA Read Request packet and verifies the rCxt key, tid, etc., it swaps the node ids for the source and the data target, and initiates the operation to send the data to the data target adapter… All RDMA data packets sent contain the packet type of Third Party RDMA Write Request).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Hefty’s first node to incorporate the teachings of Chang and allow for the subnet manager of Hefty to determine where data is located and allow for transfer of the data between the source and target via a direct RDMA connection.
One of ordinary skill in the art would be motivated to make the modifications in order to reduce the number of handshake and intermediary transfer operations required for data request operations (See Chang: Paragraphs 0224 and 0229).
Regarding claim 10, the combination of Hefty/Chang/Nagasubramaniam teaches the first node of claim 9. Hefty teaches the first node comprising wherein the read request (Fig. 2, Node 102A sends SA data query 206 related to read I/O transaction to subnet manager 116; Paragraph 0018, an SA data query 206 may be sent by the node 102A to the SM 116… to communicate with another network node element to facilitate an I/O transaction (e.g., data read) comprises information that identifies and grants access to a memory in the first node and a memory in the second node (Fig. 2, SA query 206 associated with read I/O transaction includes a key that grants access to the memory; Paragraph 0019, node 102A may update the SA data query 206 with RDMA write target buffer address, encryption key), an address space of the first node, and an address space of the second node (Fig. 1, System memory 106 of node 102A is an address space of a second node and system memory of subnet manager 116 is an address space of the first node; Paragraph 0014, system memory 106 may be controlled to provide an RDMA target write buffer 105 to enable an RDMA transfer from the SM 116 to the node… Paragraph 0029, memory associated with the network controller, e.g., network controller 110, 118).
Regarding claim 11, the combination of Hefty/Chang/Nagasubramaniam teaches the first node of claim 1. Hefty teaches the first node comprising wherein the processor further executes the processor-executable instructions to cause the first node to: based on the requested data being available at the first node, send the requested data to the second node over the first RDMA connection to the second node (Fig. 2, RC RDMA SA response 210 from subnet manager 116 (i.e. first node) sends requested data to node 102A (i.e. second node) over RDMA connection between 116 and 102A (i.e. first RDMA connection); Paragraph 0019, node 102A may also be capable of allocating an RDMA write target buffer 105 to receive and store an RDMA response from the SM 116… SM 116 may determine if the SA data 109 may be sent as a reliable connected RDMA SA data response (RC RDMA SA response 210)).
Regarding claim 21, Hefty teaches a method for a first node (Fig. 1, Subnet manager 116), which is connected to a second node (Fig. 1, Node 102A) and one or more third nodes (Fig. 1, Node 102B) by remote direct memory access (RDMA) connection (Fig. 1, Network 126 uses RDMA; Paragraph 0012, Each network node element 102A, 102B, 102C, . . . , 102N and the SM 116 may communicate with each other, via network 126, using an remote direct memory access-capable (RDMA-capable) communications protocol), the method comprising: receiving, from the second node, a read request for requested data (Fig. 2, Node 102A sends SA data query 206 related to read I/O transaction to subnet manager 116; Paragraph 0018, an SA data query 206 may be sent by the node 102A to the SM 116… Paragraph 0021, SA data query may be generated, for example, by an application being executed on the network node element. The application may use the SA data, for example, to communicate with another network node element to facilitate an I/O transaction (e.g., data read, data write, etc.) between the network node elements, and thus, the SA data may include, for example, the address of the other network node element); and wherein the first node is connected to the second node by a first RDMA connection (Fig. 1, Subnet manager 116 (i.e. first node) is connected to node 102A (i.e. second node) via a first RDMA connection between the 116 and 102A using shared/overlapping network 126), wherein the first node is connected to the one or more third nodes by a second RDMA connection (Fig. 1, Subnet manager 116 (i.e. first node) is connected to node 102B (i.e. third node) via second RDMA connection between 116 and 102B using shared/overlapping network 126); and wherein the second node is connected to the one or more third nodes by the third RDMA connection (Fig. 1, Node 102A (i.e. second node) is connected to node 102B (i.e. third nodes) via third RDMA connection between 102A and 102B using shared/overlapping network 126).
Hefty does not teach the method comprising: based on the requested data not being available at the first node, determining at which of the third or more nodes the requested data is available; causing the determined third node to send the requested data.
Chang teaches the method comprising: determining at which of the third or more nodes the requested data is available (Fig. 14, Managing node “B” (i.e. first node) receives read request from data target “A” (i.e. second node), determines read data is on data source “C” (i.e. third node); Paragraph 0224, A node "A" in the cluster may make a file system read/write request. The request gets forwarded to one of the DLM table manager nodes "B." The DLM table manager handles the request, checks its tables and determines that Node "C" owns the blocks of data that node "A" requests access to); causing the determined third node to send the requested data over a third RDMA connection to the second node (Fig. 14, Managing node “B” (i.e. first node) causes a direct RDMA connection to be established to transfer the read data between the data target “A” and the data source “C” (i.e. third RDMA connection); Paragraph 0224, Node "B" can now send a third party RDMA transfer request to node "C" asking node "C" to send the requested data to node "A" and to notify node "B" when the transfer is complete).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Hefty’s method to incorporate the teachings of Chang and allow for the subnet manager of Hefty to determine where data is located and allow for transfer of the data between the source and target via a direct RDMA connection.
One of ordinary skill in the art would be motivated to make the modifications in order to reduce the number of handshake and intermediary transfer operations required for data request operations (See Chang: Paragraphs 0224 and 0229).
Neither Hefty nor Chang teaches the method comprising: based on the requested data not being available at the first node: determining at which of the one or more third nodes the requested data is available.
Nagasubramaniam teaches the method comprising: based on the requested data not being available at the first node (Fig. 3C, Data is not available in volume 16 of point of contact node 12; Paragraph 0045, Upon determining that the requested data is not located at the local storage volume 16), determining at which of the third or more nodes the requested data is available (Fig. 3BC Point of contact node 12 (i.e. first node) determines data is available at remote node 14 (i.e. third node); Paragraph 0045, Upon determining that the requested data… is located at a remote storage volume 18).
Hefty, Chang, and Nagasubramaniam are analogous arts because they are in the same field of endeavor of performing RDMA data transfers between different network nodes.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Hefty/Chang’s method to incorporate the teachings of Nagasubramaniam and enable the subnet manager to determine if the data is present in the memory of the subnet manager and to forward the read request if the data is not present.
One of ordinary skill in the art would be motivated to make the modifications in order to yield the obvious result of efficiently performing data querying and retrieval amongst a distributed network, thus improving storage capabilities and increasing scalability of memory systems.
Claim 5 is rejected under 35 U.S.C. 103 as being unpatentable over Hefty (US 2013/0262613) in view of Chang (US 2009/0125604) in view of Nagasubramaniam (US 2017/0034269) and further in view of Wong (US 2009/0083392).
Regarding claim 5, the combination of Hefty/Chang/Nagasubramaniam teaches the first node of claim 3.
Chang teaches the first node comprising wherein the first node comprises an address of a send queue at the third node, for sending from the third node to the second node (Fig. 23, Managing node has identifiers for send queue 203 of adapter 207 of node; Paragraph 0225, this process involves controlling or managing a node so that it "knows" the available rCxt ids, current tids and the memory structure of the data source and data target nodes… Paragraph 0128, submits an RDMA request with respect to HAL FIFO send queue 203).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Hefty’s first node to incorporate the teachings of Chang and allow for the subnet manager of Hefty to determine where data is located and allow for transfer of the data between the source and target via a direct RDMA connection.
One of ordinary skill in the art would be motivated to make the modifications in order to reduce the number of handshake and intermediary transfer operations required for data request operations (See Chang: Paragraphs 0224 and 0229).
The combination of Hefty/Chang/Nagasubramaniam does not teach the first node comprising an address of a doorbell register of the send queue.
Wong teaches the first node comprising an address of a doorbell register of the send queue (Fig. 1, Nodes 12 write to doorbell registers 28 associated with queues 18 to initiate RDMA operations; Paragraph 0026, Software writes the address of the descriptor into the RDMA doorbell register to initiate the RDMA… Paragraph 0029, Software on 12.sup.1 then creates an RDMA descriptor 34.sup.1 that includes the address of buffer 18.sup.1 and the address of buffer 18.sup.2 (sent over by software from server 12.sup.2 earlier). Software on 12.sup.1 then writes the address and size of the descriptor into the RDMA doorbell register 28).
Hefty, Chang, Nagasubramaniam, and Wong are analogous arts because they are in the same field of endeavor of performing RDMA data transfers between different network nodes.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Hefty/Chang/Nagasubramaniam’s first node to incorporate the teachings of Wong and include doorbell registers associated with the send buffers.
One of ordinary skill in the art would be motivated to make the modifications in order to improve messaging between nodes while reducing latency of data transfer operations (See Wong: Paragraph 0010).
Claims 13-14, 16-18, 20, and 22-24 are rejected under 35 U.S.C. 103 as being unpatentable over Hefty (US 2013/0262613) in view of Chang (US 2009/0125604) and further in view of Nagasubramaniam (US 2017/0034269).
Regarding claim 13, Hefty teaches a second node (Fig. 1, Node 102A), which is connected to a first node (Fig. 1, Subnet manager 116) and to one or more third nodes (Fig. 1, Node 102B) by remote direct memory access (RDMA) connection (Fig. 1, Network 126 uses RDMA; Paragraph 0012, Each network node element 102A, 102B, 102C, . . . , 102N and the SM 116 may communicate with each other, via network 126, using an remote direct memory access-capable (RDMA-capable) communications protocol), the second node comprising: a processor (Fig. 1, Host CPU 104); and a memory coupled to the processor and having processor-executable instruction stored thereon, which upon execution by the processor (Fig. 1, System memory 106), cause the second node to: send a read request for requested data to the first node Fig. 2, Node 102A sends SA data query 206 related to read I/O transaction to subnet manager 116; Paragraph 0018, an SA data query 206 may be sent by the node 102A to the SM 116… Paragraph 0021, SA data query may be generated, for example, by an application being executed on the network node element. The application may use the SA data, for example, to communicate with another network node element to facilitate an I/O transaction (e.g., data read, data write, etc.) between the network node elements, and thus, the SA data may include, for example, the address of the other network node element)); and wherein the first node is connected to the second node by a first RDMA connection (Fig. 1, Subnet manager 116 (i.e. first node) is connected to node 102A (i.e. second node) via a first RDMA connection between the 116 and 102A using shared/overlapping network 126), wherein the first node is connected to the one or more third nodes by a second RDMA connection (Fig. 1, Subnet manager 116 (i.e. first node) is connected to node 102B (i.e. third node) via second RDMA connection between 116 and 102B using shared/overlapping network 126); and wherein the second node is connected to the one or more third nodes by the third RDMA connection (Fig. 1, Node 102A (i.e. second node) is connected to node 102B (i.e. third nodes) via third RDMA connection between 102A and 102B using shared/overlapping network 126).
Hefty does not teach the second node comprising: receive the requested data from one of the one or more third nodes.
Chang teaches the second node (Fig. 14, Data target “A”) comprising: receive the requested data from one of the one or more third nodes (Fig. 14, Managing node “B” (i.e. first node) causes a direct RDMA connection to be established to transfer the read data between the data target “A” and the data source “C”; Paragraph 0224, Node "B" can now send a third party RDMA transfer request to node "C" asking node "C" to send the requested data to node "A").
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Hefty’s second node to incorporate the teachings of Chang and allow for the subnet manager of Hefty to determine where data is located and allow for transfer of the data between the source and target via a direct RDMA connection.
One of ordinary skill in the art would be motivated to make the modifications in order to reduce the number of handshake and intermediary transfer operations required for data request operations (See Chang: Paragraphs 0224 and 0229).
Regarding claim 14, Hefty in view of Chang teaches the second node of claim 13. Hefty teaches the second node comprising wherein the read request (Fig. 2, Node 102A sends SA data query 206 related to read I/O transaction to subnet manager 116; Paragraph 0018, an SA data query 206 may be sent by the node 102A to the SM 116… to communicate with another network node element to facilitate an I/O transaction (e.g., data read) comprises information that identifies and grants access to a memory in the first node and a memory in the second node (Fig. 2, SA query 206 associated with read I/O transaction includes a key that grants access to the memory; Paragraph 0019, node 102A may update the SA data query 206 with RDMA write target buffer address, encryption key), an address space of the first node, and an address space of the second node (Fig. 1, System memory 106 of node 102A is an address space of a second node and system memory of subnet manager 116 is an address space of the first node; Paragraph 0014, system memory 106 may be controlled to provide an RDMA target write buffer 105 to enable an RDMA transfer from the SM 116 to the node… Paragraph 0029, memory associated with the network controller, e.g., network controller 110, 118).
Regarding claim 16, Hefty teaches a third node, which is connected to a first node and a second node by remote direct memory access (RDMA) connection (Fig. 1, Node 102B (i.e. third node) connected to subnet manager 116 (i.e. first node) and node 102A (i.e. second node) via RDMA network 126; Paragraph 0012, Each network node element 102A, 102B, 102C, . . . , 102N and the SM 116 may communicate with each other, via network 126, using an remote direct memory access-capable (RDMA-capable) communications protocol), the third node comprising: a processor (Fig. 1, Node 102A has a host CPU 104); and a memory coupled to the processor and having processor-executable instruction stored thereon, which upon execution by the processor (Fig. 1, Node 102B has a system memory 106); and wherein the first node is connected to the second node by a first RDMA connection (Fig. 1, Subnet manager 116 (i.e. first node) is connected to node 102A (i.e. second node) via a first RDMA connection between the 116 and 102A using shared/overlapping network 126), wherein the first node is connected to the one or more third nodes by a second RDMA connection (Fig. 1, Subnet manager 116 (i.e. first node) is connected to node 102B (i.e. third node) via second RDMA connection between 116 and 102B using shared/overlapping network 126); and wherein the second node is connected to the one or more third nodes by the third RDMA connection (Fig. 1, Node 102A (i.e. second node) is connected to node 102B (i.e. third nodes) via third RDMA connection between 102A and 102B using shared/overlapping network 126).
Hefty does not teach the third node comprising: receive a RDMA write request or send a command from the first node; implement one of the following: providing data to the second node by performing a RDMA write operation in response to the command written by the first node directly to a send queue at the third node; or providing the data to the second node by performing a send operation in response to the command written by the first node directly to the send queue at the third node.
Chang teaches the third node comprising: receive a RDMA write request or send a command from the first node (Fig. 14, Managing node “B” (i.e. first node) receives read/write request from data target “A” (i.e. second node) and sends read/write request to data source “C” (i.e. third node); Paragraph 0224, A node "A" in the cluster may make a file system read/write request. The request gets forwarded to one of the DLM table manager nodes "B." The DLM table manager handles the request, checks its tables and determines that Node "C" owns the blocks of data that node "A" requests access to); implement one of the following: providing data to the second node by performing a RDMA write operation in response to the command written by the first node directly to a send queue at the third node; or providing the data to the second node by performing a send operation in response to the command written by the first node directly to the send queue at the third node (Fig. 14, Data source “C” (i.e. third node) provides data to data target “A” (i.e. second node), wherein managing node “B” (i.e. first node) sends the write request to a queue of the adapter of data source “C” (i.e. third node); Paragraph 0224, Node "B" can now send a third party RDMA transfer request to node "C" asking node "C" to send the requested data to node "A" and to notify node "B" when the transfer is complete… Paragraph 0225, this process involves controlling or managing a node so that it "knows" the available rCxt ids, current tids and the memory structure of the data source and data target nodes… Paragraph 0128, submits an RDMA request with respect to HAL FIFO send queue 203).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Hefty’s third node to incorporate the teachings of Chang and allow for the subnet manager of Hefty to determine where data is located and allow for transfer of the data between the source and target via a direct RDMA connection.
One of ordinary skill in the art would be motivated to make the modifications in order to reduce the number of handshake and intermediary transfer operations required for data request operations (See Chang: Paragraphs 0224 and 0229).
Regarding claim 17, Hefty in view of Chang teaches the third node of claim 17.
Chang teaches the third node comprising wherein the processor further executes the processor-executable instructions to cause the third node to: perform, as the RDMA write operation, a write-with-immediate, or a write and send operations, or a write and send-with-immediate operations (Fig. 14, Managing node “B” (i.e. first node) causes a direct RDMA connection to be established to transfer the read data between the data target “C” and the data source “A” (i.e. third RDMA connection) and to further transfer a notification of the RDMA write request back to node B; Paragraph 0224, Node "B" can now send a third party RDMA transfer request to node "C" asking node "C" to send the requested data to node "A" and to notify node "B" when the transfer is complete).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Hefty’s third node to incorporate the teachings of Chang and allow for the subnet manager of Hefty to determine where data is located and allow for transfer of the data between the source and target via a direct RDMA connection and enable a notification to be sent between the nodes and the managing node.
One of ordinary skill in the art would be motivated to make the modifications in order to reduce the number of handshake and intermediary transfer operations required for data request operations (See Chang: Paragraphs 0224 and 0229).
Regarding claim 18, Hefty in view of Chang teaches the third node of claim 16.
Chang teaches the third node comprising wherein the processor further executes the processor-executable instructions to cause the third node to: execute the command written to the send queue at the third node that provides the data into an address space of the second node (Fig. 14, Managing node (i.e. first node) sends third party RDMA request to data source (i.e. third node) which causes data source to execute the third party RDMA request in the queue of the adapter of data source to perform data transfer with data target (i.e. second node), wherein the data is stored in an address space; Paragraph 0052, packet transfers made directly from the memory address space of one data processing unit to the memory address space of one or more other data processing units… Paragraph 0226, When the data source adapter receives the Third Party RDMA Read Request packet and verifies the rCxt key, tid, etc., it swaps the node ids for the source and the data target, and initiates the operation to send the data to the data target adapter… All RDMA data packets sent contain the packet type of Third Party RDMA Write Request).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Hefty’s first node to incorporate the teachings of Chang and allow for the subnet manager of Hefty to determine where data is located and allow for transfer of the data between the source and target via a direct RDMA connection.
One of ordinary skill in the art would be motivated to make the modifications in order to reduce the number of handshake and intermediary transfer operations required for data request operations (See Chang: Paragraphs 0224 and 0229).
Regarding claim 20, Hefty in view of Chang teaches the third node of claim 16.
Chang teaches the third node comprising wherein, based on the third node receiving the command via s end from the first node, the third node is configured to: provide the data to the second node by initiating the execution of the operation indicated by the RDMA command to the second node, and/or by controlling a receive queue at the third node, for sending from the third node to the second node (Fig. 14, Data source (i.e. third node) to execute the third party RDMA request in the queue of the adapter of data source to perform data transfer with data target (i.e. second node), wherein the data is stored in an address space; Paragraph 0052, packet transfers made directly from the memory address space of one data processing unit to the memory address space of one or more other data processing units… Paragraph 0226, When the data source adapter receives the Third Party RDMA Read Request packet and verifies the rCxt key, tid, etc., it swaps the node ids for the source and the data target, and initiates the operation to send the data to the data target adapter… All RDMA data packets sent contain the packet type of Third Party RDMA Write Request).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Hefty’s first node to incorporate the teachings of Chang and allow for the subnet manager of Hefty to determine where data is located and allow for transfer of the data between the source and target via a direct RDMA connection.
One of ordinary skill in the art would be motivated to make the modifications in order to reduce the number of handshake and intermediary transfer operations required for data request operations (See Chang: Paragraphs 0224 and 0229).
Regarding claim 22, Hefty teaches a method for a second node (Fig. 1, Node 102A), which is connected to a first node (Fig. 1, Subnet manager 116) and to one or more third nodes (Fig. 1, Node 102B) by remote direct memory access (RDMA) connection (Fig. 1, Network 126 uses RDMA; Paragraph 0012, Each network node element 102A, 102B, 102C, . . . , 102N and the SM 116 may communicate with each other, via network 126, using an remote direct memory access-capable (RDMA-capable) communications protocol), the method comprising: sending a read request for requested data to the first node Fig. 2, Node 102A sends SA data query 206 related to read I/O transaction to subnet manager 116; Paragraph 0018, an SA data query 206 may be sent by the node 102A to the SM 116… Paragraph 0021, SA data query may be generated, for example, by an application being executed on the network node element. The application may use the SA data, for example, to communicate with another network node element to facilitate an I/O transaction (e.g., data read, data write, etc.) between the network node elements, and thus, the SA data may include, for example, the address of the other network node element)); and wherein the first node is connected to the second node by a first RDMA connection (Fig. 1, Subnet manager 116 (i.e. first node) is connected to node 102A (i.e. second node) via a first RDMA connection between the 116 and 102A using shared/overlapping network 126), wherein the first node is connected to the one or more third nodes by a second RDMA connection (Fig. 1, Subnet manager 116 (i.e. first node) is connected to node 102B (i.e. third node) via second RDMA connection between 116 and 102B using shared/overlapping network 126); and wherein the second node is connected to the one or more third nodes by the third RDMA connection (Fig. 1, Node 102A (i.e. second node) is connected to node 102B (i.e. third nodes) via third RDMA connection between 102A and 102B using shared/overlapping network 126).
Hefty does not teach the method comprising: receiving the requested data from one of the one or more third nodes.
Chang teaches the method comprising: receiving the requested data from one of the one or more third nodes (Fig. 14, Managing node “B” (i.e. first node) causes a direct RDMA connection to be established to transfer the read data between the data target “A” and the data source “C”; Paragraph 0224, Node "B" can now send a third party RDMA transfer request to node "C" asking node "C" to send the requested data to node "A").
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Hefty’s method to incorporate the teachings of Chang and allow for the subnet manager of Hefty to determine where data is located and allow for transfer of the data between the source and target via a direct RDMA connection.
One of ordinary skill in the art would be motivated to make the modifications in order to reduce the number of handshake and intermediary transfer operations required for data request operations (See Chang: Paragraphs 0224 and 0229).
Regarding claim 23, Hefty teaches a method for a third node, which is connected to a first node and a second node by remote direct memory access (RDMA) connection (Fig. 1, Node 102B (i.e. third node) connected to subnet manager 116 (i.e. first node) and node 102A (i.e. second node) via RDMA network 126; Paragraph 0012, Each network node element 102A, 102B, 102C, . . . , 102N and the SM 116 may communicate with each other, via network 126, using an remote direct memory access-capable (RDMA-capable) communications protocol), the third node comprising: wherein the first node is connected to the second node by a first RDMA connection (Fig. 1, Subnet manager 116 (i.e. first node) is connected to node 102A (i.e. second node) via a first RDMA connection between the 116 and 102A using shared/overlapping network 126), wherein the first node is connected to the one or more third nodes by a second RDMA connection (Fig. 1, Subnet manager 116 (i.e. first node) is connected to node 102B (i.e. third node) via second RDMA connection between 116 and 102B using shared/overlapping network 126); and wherein the second node is connected to the one or more third nodes by the third RDMA connection (Fig. 1, Node 102A (i.e. second node) is connected to node 102B (i.e. third nodes) via third RDMA connection between 102A and 102B using shared/overlapping network 126).
Hefty does not teach the method comprising: receiving a RDMA write request or send a command from the first node; implementing one of the following: providing data to the second node by performing a RDMA write operation in response to the command written by the first node directly to a send queue at the third node; or providing the data to the second node by performing a send operation in response to the command written by the first node directly to the send queue at the third node.
Chang teaches the method comprising receiving a RDMA write request or send a command from the first node (Fig. 14, Managing node “B” (i.e. first node) receives read/write request from data target “A” (i.e. second node) and sends read/write request to data source “C” (i.e. third node); Paragraph 0224, A node "A" in the cluster may make a file system read/write request. The request gets forwarded to one of the DLM table manager nodes "B." The DLM table manager handles the request, checks its tables and determines that Node "C" owns the blocks of data that node "A" requests access to); implementing one of the following: providing data to the second node by performing a RDMA write operation in response to the command written by the first node directly to a send queue at the third node; or providing the data to the second node by performing a send operation in response to the command written by the first node directly to the send queue at the third node (Fig. 14, Data source “C” (i.e. third node) provides data to data target “A” (i.e. second node), wherein managing node “B” (i.e. first node) sends the write request to a queue of the adapter of data source “C” (i.e. third node); Paragraph 0224, Node "B" can now send a third party RDMA transfer request to node "C" asking node "C" to send the requested data to node "A" and to notify node "B" when the transfer is complete… Paragraph 0225, this process involves controlling or managing a node so that it "knows" the available rCxt ids, current tids and the memory structure of the data source and data target nodes… Paragraph 0128, submits an RDMA request with respect to HAL FIFO send queue 203).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Hefty’s method to incorporate the teachings of Chang and allow for the subnet manager of Hefty to determine where data is located and allow for transfer of the data between the source and target via a direct RDMA connection.
One of ordinary skill in the art would be motivated to make the modifications in order to reduce the number of handshake and intermediary transfer operations required for data request operations (See Chang: Paragraphs 0224 and 0229).
Regarding claim 24, Hefty in view of Chang teaches a non-transitory computer readable medium comprising instructions which, upon being executed by a computer, cause the computer to perform the method of claim 21 (Fig. 1, Subnet manager 116 comprises a processor and a system memory; Paragraph 0030, operations according to the methods described herein may be distributed across a plurality of physical devices, such as processing structures at several different physical locations. The storage medium may include any type of tangible, non-transitory storage medium).
Claim 19 is rejected under 35 U.S.C. 103 as being unpatentable over Hefty (US 2013/0262613) in view of Chang (US 2009/0125604) and further in view of Wong (US 2009/0083392).
Regarding claim 19, Hefty in view of Chang teaches the third node of claim 16.
Chang teaches the third node comprising wherein based on the first node directly writing the command to the send queue of the third node, the third node configured to: provide the data to the second node without processing at the third node, and/or without controlling a receive queue at the third node, for sending from the third node to the second node (Fig. 14, Data source sends third party RDMA data to data target using RDMA thus performing data transfer without processing; Paragraph 0216, RDMA operations allow the communicating task to stripe and transfer data in parallel across multiple network interfaces without impacting the tasks running on the other CPUs).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Hefty’s third node to incorporate the teachings of Chang and allow for the subnet manager of Hefty to determine where data is located and allow for transfer of the data between the source and target via a direct RDMA connection.
One of ordinary skill in the art would be motivated to make the modifications in order to reduce the number of handshake and intermediary transfer operations required for data request operations (See Chang: Paragraphs 0224 and 0229).
Neither Hefty nor Chang teaches the third node comprising a doorbell register of the send queue.
Wong teaches the third node comprising a doorbell register of the send queue (Fig. 1, Nodes 12 write to doorbell registers 28 associated with queues 18 to initiate RDMA operations; Paragraph 0026, Software writes the address of the descriptor into the RDMA doorbell register to initiate the RDMA… Paragraph 0029, Software on 12.sup.1 then creates an RDMA descriptor 34.sup.1 that includes the address of buffer 18.sup.1 and the address of buffer 18.sup.2 (sent over by software from server 12.sup.2 earlier). Software on 12.sup.1 then writes the address and size of the descriptor into the RDMA doorbell register 28).
Hefty, Chang, Nagasubramaniam, and Wong are analogous arts because they are in the same field of endeavor of performing RDMA data transfers between different network nodes.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Hefty/Chang’s third node to incorporate the teachings of Wong and include doorbell registers associated with the send buffers.
One of ordinary skill in the art would be motivated to make the modifications in order to improve messaging between nodes while reducing latency of data transfer operations (See Wong: Paragraph 0010).
Allowable Subject Matter
Claims 12 and 15 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims, and if the Claim Objection for claim 15 is overcome.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
US PGPUB 2020/0401551 to Joshua discloses a plurality of hosts coupled with a plurality of RDMA NICs.
US PGPUB 2021/0342071 to Puhov discloses a distributed network of nodes coupled with RDMA interfaces.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to HARRY Z WANG whose telephone number is (571)270-1716. The examiner can normally be reached 9 am - 3 pm (Monday-Friday).
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Henry Tsai can be reached at 571-272-4176. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/H.Z.W./Examiner, Art Unit 2184
/HENRY TSAI/Supervisory Patent Examiner, Art Unit 2184
Prosecution Insights