Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
This office action is in response to Applicant’s Amendment filed 02/05/2026. Claims 1-20 are pending. Claims 1, 3, 8, 10, 15, and 17 have been amended. Any examiner’s note, objection, or rejection not repeated is withdrawn due to Applicant’s amendment.
Information Disclosure Statement
The information disclosure statements (IDS) submitted on 02/28/2022 and 04/26/2023 are in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statements are being considered by the examiner.
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 02/05/2026 has been entered.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1-20 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 1 recites the limitation "wherein the available GPU memory is a portion of total GPU that is configured for running the inference model" on Lines 10-11. There is insufficient antecedent basis for this limitation in the claim, as “total GPU” is not previously defined in the Claim. For examination, it is being interpreted by the Examiner as "wherein the available GPU memory is a portion of a total memory of the GPU that is configured for running the inference model".
Claim 8 recites the limitation "wherein the available GPU memory is a portion of total GPU that is configured for running the inference model" on Lines 12-13. There is insufficient antecedent basis for this limitation in the claim, as “total GPU” is not previously defined in the Claim. For examination, it is being interpreted by the Examiner as "wherein the available GPU memory is a portion of a total memory of the GPU that is configured for running the inference model".
Claim 15 recites the limitation "wherein the available GPU memory is a portion of total GPU that is configured for running the inference model" on Lines 14-15. There is insufficient antecedent basis for this limitation in the claim, as “total GPU” is not previously defined in the Claim. For examination, it is being interpreted by the Examiner as "wherein the available GPU memory is a portion of a total memory of the GPU that is configured for running the inference model".
Appropriate correction is required.
Any claims not specifically mentioned are rejected by virtue of their dependency to rejected claims.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-3, 5, 7-10, 12, 14 -17, and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Choudhury et al. (US 20200125926 A1) in view of Zhu et al. (US 20210374518 A1), and further in view of Ma et al. (US 20200012924 A1), hereinafter referred to as Choudhury, Zhu, and Ma, respectively.
Regarding Claim 1, Choudhury discloses A computer-implemented method ([0004] cause a computer to carry out a plurality of method steps, as described herein) for running an inference model with a graphical processing unit (GPU) having a restrained resource ([0014] inferencing using one or more models used for inferencing and resource constraints (such as total available memory, maximum latency for inferencing, maximum energy for inferencing, etc.). Please note that total available memory corresponds to Applicant’s available GPU memory, as GPU memory is a variant of memory as is known in the art, resource constraints correspond to Applicant’s restrained resource, and the models used for inferencing with resource constraints such as total available memory corresponds to Applicant’s available GPU memory on which the model is run.), the method comprising:
in response to an edge device request, receiving by an inference program from a model provisioning program, a trained inference model, wherein the trained inference model is configured to execute intended objectives of the edge device ([0002] Deep neural networks are used for a variety of artificial intelligence applications such as computer vision, speech recognition, natural language processing, etc. Additionally, such deep learning models can be used on mobile phones and other edge devices in the context of Internet of Things (IoT). Thus, inferencing can be carried out either on the cloud or the edge device itself. Inferencing, as used herein, refers to the stage wherein a trained network predicts and/or classifies input test samples. Please note that models being used on edge devices for AI applications corresponds to a trained inference model being configured to execute intended objectives of an edge device, which makes an edge device request, i.e., a request to execute an AI application. Furthermore, since inferencing is done with a trained network, this corresponds to receiving a trained inference model by an inference program from a model provisioning program, as it is known in the art that there must be means for provisioning the trained inference model to be used in the requested application of the edge device.);
based on a configuration of the trained inference model, loading by the inference program, a subset of total number of layers in the inference model ([0014] determining individual layer batch sizes for inferencing using one or more models used for inferencing and resource constraints. [0018] a batch size optimizer. In such an embodiment, L.sub.1, L.sub.2, . . . , L.sub.n represent the n layers of the network. A simple path network, for example, can include an output of layer L.sub.i being fed only into its successor layer L.sub.i+1. Please note that the resource constraints correspond to Applicant’s configuration of the trained inference model, and determining individual layer batch sizes for inferencing, where a batch size optimizer that operates with representations of n layers of the network, corresponds to Applicant’s loading a subset of total number of layers in the inference model, as it processes a subset of the inference model based on resource constraints and operates with n layers.) wherein the subset of layers is a count that is fewer than the total number of layers in the inference model and is based on available GPU memory, wherein the available GPU memory is a portion of total GPU that is configured for running the inference model ([0014] determining individual layer batch sizes for inferencing using one or more models used for inferencing and resource constraints (such as total available memory); [0016] given memory availability […] a memory requirement of layer L.sub.2 104 can restrict the batch size that can be processed for the network; Fig 1. Please note that in each example of batch sizes for each layer corresponding to Applicant’s count of the subset of the layers, the batch size is less than the available memory, corresponding to Applicant’s count being a value less than the total number of layers of the inference model, and since it is based on memory availability, this corresponds to being based on available GPU memory. Additionally, as the resource constraints for inferencing using the models include total available memory, this corresponds to the available GPU memory being a portion of total GPU that is configured for running the inference model.);
Choudhury does not explicitly disclose until all of the subset of layers is processed, loading and running, by the inference program into a corresponding GPU memory allocation, a current layer of the subset of layers along with a corresponding step input, giving a step output; based on there being more layers of the subset of layers, releasing, by the inference program, a portion of the GPU memory allocation corresponding to the current layer of the subset of layers; loading, by the inference program, a next layer of the subset of layers along with the corresponding step input, giving a corresponding step output; and based on all of the subset of layers being processed, outputting by the inference program a prediction
However, Zhu discloses until all of the subset of layers is processed, loading and running, by the inference program into a corresponding GPU memory allocation, a current layer of the subset of layers along with a corresponding step input, giving a step output ([0075] processor causes additional copies of output from precedent layer to be provided to additional subsequent layers that are to be performed on same […] GPUs.; [0092] inference […] may include […] input/output data corresponding to neurons or layers of a neural network; [0093] causes the loading of weight or other parameter information into processor ALUs based on an architecture of a neural network; [0090] weight parameters […] used in conjunction with […] inferencing. Please note that loading weight parameters and using them in conjunction with inferencing corresponds to Applicant’s loading and running step. Please note that in this case, since the GPU is processing the data that is copied, this corresponds to the available GPU memory allocations, as the layers being performed on GPUs would be in the available GPU memory allocations as they are processed. Since the processor causes copies of output from precedent layers to be provided to subsequent layers after using input data, this corresponds to running the current layer of the subset of layers along with a corresponding step input, giving a step output.);
based on there being more layers of the subset of layers, releasing, by the inference program, a portion of the GPU memory allocation corresponding to the current layer of the subset of layers (([0075] processor causes additional copies of output from precedent layer to be provided to additional subsequent layers that are to be performed on same […] GPUs.; [0100] different layers of a neural network, such that resulting activation from one storage/computational pair 801/802 of code […] is provided as an input to a next storage/computational pair 805/806. Please note that, as is common knowledge to a person having ordinary skill in the art, the generation of an activation that can be provided as an input to a next layer in a neural network requires the release of the first layer, corresponding to Applicant’s releasing the portion of the GPU memory allocation corresponding to the current layer of the subset of layers. In this case, since the GPU is processing the data that is copied, this corresponds to the available GPU memory allocations, as the layers being performed on GPUs would be in the available GPU memory allocations as they are processed.);
loading, by the inference program, a next layer of the subset of layers along with the corresponding step input, giving a corresponding step output ( [0092] inference […] may include […] input/output data corresponding to neurons or layers of a neural network; [0093] causes the loading of weight or other parameter information into processor ALUs based on an architecture of a neural network; [0090] weight parameters […] used in conjunction with […] inferencing. Please note that loading weight parameters and using them in conjunction with inferencing corresponds to Applicant’s loading and running step. Please note that since the processor causes copies of output from precedent layers to be provided to subsequent layers after using input data, proceeding to the next layer, this corresponds to loading a next layer of the subset of layers along with a corresponding step input, giving a corresponding step output.);
and based on all of the subset of layers being processed, outputting by the inference program a prediction ([0113] In at least one embodiment, trained machine learning models corresponding to one or more neural networks may be used to infer or predict information using resources described above with respect to data center 1000 by using weight parameters calculated through one or more training techniques described herein. Please note that the machine learning models using weight parameters calculated through the described training techniques to predict or infer information corresponds to Applicant’s outputting the prediction by the inference program. It is known in the art that a prediction of a model is output once its layers have been processed; therefore, this would include all of the subset of layers as well.).
Choudhury and Zhu are both considered to be analogous to the claimed invention because they are in the same field of creating systems for inference models in computers. Therefore, it would have been obvious to someone of ordinary skill in the art prior to the effective filing date of the claimed invention to have modified Choudhury to incorporate the teachings of Zhu to modify the inference model system using dynamic batch sizes to utilize a sequential inference model to generate a prediction, allowing for more efficient use of available resources and improved performance from pipelining, as described in Zhu.
Choudhury-Zhu does not explicitly disclose dividing, by the inference program, the available GPU memory into a number of allocations comprising a layers allocation, an input/output allocation, and an intermediate information allocation, wherein a size of allocations is determined based on a size of the subset of layers, a size of a step input for a given layer, a size of a step output for a given layer, and a size of intermediate information for a given layer,
However, Ma discloses dividing, by the inference program, the available GPU memory into a number of allocations comprising a layers allocation ([0006] Neural connections among the artificial neurons are formed by interconnect circuitry coupled to control lines of the memory array to subdivide the memory array into a plurality of layers of the artificial neural network. Control circuitry is configured to transmit a plurality of iterations of an input value on input control lines of a first layer of the artificial neural network for inference operations by at least one or more additional layers.; [0019] A graphics processing unit (GPU), which has started to gain favor over CPUs, uses a parallel architecture and can handle many sets of very simple instructions. Please note that subdividing the memory array into a plurality of layers of the ANN that carries out inference operations, where the GPU is handling instructions, corresponds to Applicant’s dividing the available GPU memory into a number of allocations comprising a layers allocation by the inference program.), an input/output allocation ([0037] Memory 131 can also be configured to store input data and output data. Please note that storing input and output data in the memory corresponds to Applicant’s input/output allocation, as in order to store this data in the memory, it is obvious to one of ordinary skill in the art that a portion of memory is allocated. ), and an intermediate information allocation ([0037] memory 131 can store intermediate and final values for an ANN pipeline. Please note that storing intermediate values for an ANN pipeline in the memory corresponds to Applicant’s intermediate information allocation, as in order to store this data in the memory, it is obvious to one of ordinary skill in the art that a portion of memory is allocated. ), wherein a size of allocations is determined based on a size of the subset of layers, a size of a step input for a given layer, a size of a step output for a given layer, and a size of intermediate information for a given layer, ([0037] In one example, memory 131 is employed as an output buffer to store synaptic weights for artificial neurons of an ANN. Control system 130 can load these synaptic weights into NVM elements of memory array 110 prior to introduction of input data to the ANN layers. Memory 131 can also be configured to store input data and output data. [...] In the examples below, an ANN pipelining technique is discussed, and memory 131 can store intermediate and final values for an ANN pipeline. [0038] Memory array 110 can be used to implement a single layer of an artificial neural network, or instead might implement a multi-layer ANN. Please note that the memory array implementing a multi-layer ANN corresponds to the size of the subset of layers, the memory being configured to store the input data corresponds to a size of a step input for a given layer, the memory being configured to store the output data corresponds to a size of a step output for a given layer, the memory storing the intermediate values corresponds to a size of intermediate information for a given layer, as in order to store this data in the memory, it is obvious to one of ordinary skill in the art that a correspondingly sized portion of memory is allocated after determining the appropriate respective size for storing the data.)
Choudhury-Zhu and Ma are both considered to be analogous to the claimed invention because they are in the same field of creating systems for inference models in computers. Therefore, it would have been obvious to someone of ordinary skill in the art prior to the effective filing date of the claimed invention to have modified Choudhury-Zhu to incorporate the teachings of Ma to modify the inference model system using dynamic batch sizes and utilizing a sequential inference model to generate a prediction to have allocations in available GPU memory with allocated sizes for data of each step of each layer, allowing for improved inference accuracy and improved performance from pipelining via reduced runtime, as described in Ma.
Regarding Claim 2, Choudhury-Zhu-Ma as described in Claim 1 discloses from Zhu wherein, as a result of calculating the step output for the each of the subset of layers ([0095] data storage 801 stores weight parameters and/or input/output data of each layer of a neural network trained or used in conjunction with one or more embodiments during forward propagation of input/output data […] during training and/or inferencing. Please note that weight parameters and input/output data of each layer during forward propagation of input/output data corresponds to Applicant’s result of calculating the step output for the each of the subset of layers.), the method further comprises: releasing the portion of the GPU memory allocation corresponding to the current layer of the subset of layers (([0075] processor causes additional copies of output from precedent layer to be provided to additional subsequent layers that are to be performed on same […] GPUs.; [0100] different layers of a neural network, such that resulting activation from one storage/computational pair 801/802 of code […] is provided as an input to a next storage/computational pair 805/806. Please note that, as is common knowledge to a person having ordinary skill in the art, the generation of an activation that can be provided as an input to a next layer in a neural network requires the release of the first layer, corresponding to Applicant’s releasing the portion of the memory allocation corresponding to the current layer of the subset of layers. In this case, since the GPU is processing the data that is copied, this corresponds to the available GPU memory allocations, as the layers being performed on GPUs would be in the available GPU memory allocations as they are processed.); and loading and running a next layer of the inference model to create further layers for a subsequent step ([0093] causes the loading of weight or other parameter information into processor ALUs based on an architecture of a neural network; [0090] data storage 801 stores weight parameters and/or input/output data of each layer of a neural network trained or used in conjunction with one or more embodiments during forward propagation of input/output data and/or weight parameters during training and/or inferencing Please note that loading weights/parameter information into processor ALUs based on an architecture of a neural network corresponds to Applicant’s loading a next layer of the inference model. Furthermore, forward propagation of weight parameters during inferencing corresponds to Applicant’s running a next layer of the inference model to create further layers for a subsequent step.).
Regarding Claim 3, Choudhury-Zhu-Ma as described in Claim 2 discloses from Zhu wherein the released one of the subset of layers is a lowest level layer among the subset of layers and the loaded one of the subset of layers is a next higher level layer relative to the subset of layers ([0072] neural networks are modified 204 so that output from a first layer of a neural network […] is provided to a second layer. Please note that a first layer corresponds to Applicant’s lowest level layer among the subset of layers and a second layer corresponds to Applicant’s next higher level relative to the subset of layers, as layer 1 would be the lowest level layer, and layer 2 would be the next higher level relative to the subset of layers in a situation where there are at least 2 layers. Furthermore, as layer 2 is to be loaded as part of the operation of a neural network as is known in the art and is dependent on output from layer 1, layer 1 must first be released to obtain its output.).
Regarding Claim 5, Choudhury-Zhu-Ma as described in Claim 1 discloses from Zhu wherein memory identification information indicates a location within the available GPU memory allocations where the step output is stored ([0075] processor causes additional copies of output from precedent layer to be provided to additional subsequent layers that are to be performed on same […] GPUs. Please note that the copies of output from the precedent layer for subsequent layers corresponds to Applicant’s memory identification information indicating a location within the allocations where the step output is stored, as it is known to a person of ordinary skill in the art that accessing a copy of data uses a reference to its location in memory, corresponding to Applicant’s memory identification information indicating a location within the allocations where the step output is stored. In this case, since the GPU is processing the data that is copied, this corresponds to the available GPU memory allocations, as the layers being performed on GPUs would be in the available GPU memory allocations as they are processed.).
Regarding Claim 7, Choudhury-Zhu-Ma as described in Claim 1 discloses from Zhu wherein the step output is retained in the allocations until a last layer of the layers of the inference model is processed ([0075] additional subsequent layers also receive a copied output from a precedent layer. Please note that additional subsequent layers receiving copied output from a precedent corresponds to Applicant’s step output being retained in the allocations until a last layer of the layers of the inference model is processed, as a copy of output corresponding to Applicant’s step output will have to be retained in allocations until the process completes execution, i.e., reaches the last layer after which there is no subsequent layer and thus no more copied step output to be retained).
Regarding Claim 8, Choudhury discloses A non-transitory computer-readable storage media ([0045] A computer readable storage medium, as used herein, is not to be construed as being transitory) that configures a computer to perform program instructions stored on the non-transitory computer-readable storage media ([0035] computer-readable recordable storage medium with code adapted to be executed to carry out at least one method step described herein ) for running an inference model with a graphical processing unit (GPU) having a restrained resource ([0014] inferencing using one or more models used for inferencing and resource constraints (such as total available memory, maximum latency for inferencing, maximum energy for inferencing, etc.). Please note that total available memory corresponds to Applicant’s available GPU memory, as GPU memory is a variant of memory as is known in the art, resource constraints correspond to Applicant’s restrained resource, and the models used for inferencing with resource constraints such as total available memory corresponds to Applicant’s available GPU memory on which the model is run.), the program instructions comprising:
in response to an edge device request, receiving by an inference program from a model provisioning program, a trained inference model, wherein the trained inference model is configured to execute intended objectives of the edge device ([0002] Deep neural networks are used for a variety of artificial intelligence applications such as computer vision, speech recognition, natural language processing, etc. Additionally, such deep learning models can be used on mobile phones and other edge devices in the context of Internet of Things (IoT). Thus, inferencing can be carried out either on the cloud or the edge device itself. Inferencing, as used herein, refers to the stage wherein a trained network predicts and/or classifies input test samples. Please note that models being used on edge devices for AI applications corresponds to a trained inference model being configured to execute intended objectives of an edge device, which makes an edge device request, i.e., a request to execute an AI application. Furthermore, since inferencing is done with a trained network, this corresponds to receiving a trained inference model by an inference program from a model provisioning program, as it is known in the art that there must be means for provisioning the trained inference model to be used in the requested application of the edge device.);
based on a configuration of the trained inference model, loading by the inference program, a subset of total number of layers in the inference model ([0014] determining individual layer batch sizes for inferencing using one or more models used for inferencing and resource constraints. [0018] a batch size optimizer. In such an embodiment, L.sub.1, L.sub.2, . . . , L.sub.n represent the n layers of the network. A simple path network, for example, can include an output of layer L.sub.i being fed only into its successor layer L.sub.i+1. Please note that the resource constraints correspond to Applicant’s configuration of the trained inference model, and determining individual layer batch sizes for inferencing, where a batch size optimizer that operates with representations of n layers of the network, corresponds to Applicant’s loading a subset of total number of layers in the inference model, as it processes a subset of the inference model based on resource constraints and operates with n layers.) wherein the subset of layers is a count that is fewer than the total number of layers in the inference model and is based on available GPU memory, wherein the available GPU memory is a portion of total GPU that is configured for running the inference model ([0014] determining individual layer batch sizes for inferencing using one or more models used for inferencing and resource constraints (such as total available memory); [0016] given memory availability […] a memory requirement of layer L.sub.2 104 can restrict the batch size that can be processed for the network; Fig 1. Please note that in each example of batch sizes for each layer corresponding to Applicant’s count of the subset of the layers, the batch size is less than the available memory, corresponding to Applicant’s count being a value less than the total number of layers of the inference model, and since it is based on memory availability, this corresponds to being based on available GPU memory. Additionally, as the resource constraints for inferencing using the models include total available memory, this corresponds to the available GPU memory being a portion of total GPU that is configured for running the inference model.);
Choudhury does not explicitly disclose until all of the subset of layers is processed, loading and running, by the inference program into a corresponding GPU memory allocation, a current layer of the subset of layers along with a corresponding step input, giving a step output; based on there being more layers of the subset of layers, releasing, by the inference program, a portion of the GPU memory allocation corresponding to the current layer of the subset of layers; loading, by the inference program, a next layer of the subset of layers along with the corresponding step input, giving a corresponding step output; and based on all of the subset of layers being processed, outputting by the inference program a prediction
However, Zhu discloses until all of the subset of layers is processed, loading and running, by the inference program into a corresponding GPU memory allocation, a current layer of the subset of layers along with a corresponding step input, giving a step output ([0075] processor causes additional copies of output from precedent layer to be provided to additional subsequent layers that are to be performed on same […] GPUs.; [0092] inference […] may include […] input/output data corresponding to neurons or layers of a neural network; [0093] causes the loading of weight or other parameter information into processor ALUs based on an architecture of a neural network; [0090] weight parameters […] used in conjunction with […] inferencing. Please note that loading weight parameters and using them in conjunction with inferencing corresponds to Applicant’s loading and running step. Please note that in this case, since the GPU is processing the data that is copied, this corresponds to the available GPU memory allocations, as the layers being performed on GPUs would be in the available GPU memory allocations as they are processed. Since the processor causes copies of output from precedent layers to be provided to subsequent layers after using input data, this corresponds to running the current layer of the subset of layers along with a corresponding step input, giving a step output.);
based on there being more layers of the subset of layers, releasing, by the inference program, a portion of the GPU memory allocation corresponding to the current layer of the subset of layers (([0075] processor causes additional copies of output from precedent layer to be provided to additional subsequent layers that are to be performed on same […] GPUs.; [0100] different layers of a neural network, such that resulting activation from one storage/computational pair 801/802 of code […] is provided as an input to a next storage/computational pair 805/806. Please note that, as is common knowledge to a person having ordinary skill in the art, the generation of an activation that can be provided as an input to a next layer in a neural network requires the release of the first layer, corresponding to Applicant’s releasing the portion of the GPU memory allocation corresponding to the current layer of the subset of layers. In this case, since the GPU is processing the data that is copied, this corresponds to the available GPU memory allocations, as the layers being performed on GPUs would be in the available GPU memory allocations as they are processed.);
loading, by the inference program, a next layer of the subset of layers along with the corresponding step input, giving a corresponding step output ( [0092] inference […] may include […] input/output data corresponding to neurons or layers of a neural network; [0093] causes the loading of weight or other parameter information into processor ALUs based on an architecture of a neural network; [0090] weight parameters […] used in conjunction with […] inferencing. Please note that loading weight parameters and using them in conjunction with inferencing corresponds to Applicant’s loading and running step. Please note that since the processor causes copies of output from precedent layers to be provided to subsequent layers after using input data, proceeding to the next layer, this corresponds to loading a next layer of the subset of layers along with a corresponding step input, giving a corresponding step output.);
and based on all of the subset of layers being processed, outputting by the inference program a prediction ([0113] In at least one embodiment, trained machine learning models corresponding to one or more neural networks may be used to infer or predict information using resources described above with respect to data center 1000 by using weight parameters calculated through one or more training techniques described herein. Please note that the machine learning models using weight parameters calculated through the described training techniques to predict or infer information corresponds to Applicant’s outputting the prediction by the inference program. It is known in the art that a prediction of a model is output once its layers have been processed; therefore, this would include all of the subset of layers as well.).
Choudhury and Zhu are both considered to be analogous to the claimed invention because they are in the same field of creating systems for inference models in computers. Therefore, it would have been obvious to someone of ordinary skill in the art prior to the effective filing date of the claimed invention to have modified Choudhury to incorporate the teachings of Zhu to modify the inference model system using dynamic batch sizes to utilize a sequential inference model to generate a prediction, allowing for more efficient use of available resources and improved performance from pipelining, as described in Zhu.
Choudhury-Zhu does not explicitly disclose dividing, by the inference program, the available GPU memory into a number of allocations comprising a layers allocation, an input/output allocation, and an intermediate information allocation, wherein a size of allocations is determined based on a size of the subset of layers, a size of a step input for a given layer, a size of a step output for a given layer, and a size of intermediate information for a given layer,
However, Ma discloses dividing, by the inference program, the available GPU memory into a number of allocations comprising a layers allocation ([0006] Neural connections among the artificial neurons are formed by interconnect circuitry coupled to control lines of the memory array to subdivide the memory array into a plurality of layers of the artificial neural network. Control circuitry is configured to transmit a plurality of iterations of an input value on input control lines of a first layer of the artificial neural network for inference operations by at least one or more additional layers.; [0019] A graphics processing unit (GPU), which has started to gain favor over CPUs, uses a parallel architecture and can handle many sets of very simple instructions. Please note that subdividing the memory array into a plurality of layers of the ANN that carries out inference operations, where the GPU is handling instructions, corresponds to Applicant’s dividing the available GPU memory into a number of allocations comprising a layers allocation by the inference program.), an input/output allocation ([0037] Memory 131 can also be configured to store input data and output data. Please note that storing input and output data in the memory corresponds to Applicant’s input/output allocation, as in order to store this data in the memory, it is obvious to one of ordinary skill in the art that a portion of memory is allocated. ), and an intermediate information allocation ([0037] memory 131 can store intermediate and final values for an ANN pipeline. Please note that storing intermediate values for an ANN pipeline in the memory corresponds to Applicant’s intermediate information allocation, as in order to store this data in the memory, it is obvious to one of ordinary skill in the art that a portion of memory is allocated. ), wherein a size of allocations is determined based on a size of the subset of layers, a size of a step input for a given layer, a size of a step output for a given layer, and a size of intermediate information for a given layer, ([0037] In one example, memory 131 is employed as an output buffer to store synaptic weights for artificial neurons of an ANN. Control system 130 can load these synaptic weights into NVM elements of memory array 110 prior to introduction of input data to the ANN layers. Memory 131 can also be configured to store input data and output data. [...] In the examples below, an ANN pipelining technique is discussed, and memory 131 can store intermediate and final values for an ANN pipeline. [0038] Memory array 110 can be used to implement a single layer of an artificial neural network, or instead might implement a multi-layer ANN. Please note that the memory array implementing a multi-layer ANN corresponds to the size of the subset of layers, the memory being configured to store the input data corresponds to a size of a step input for a given layer, the memory being configured to store the output data corresponds to a size of a step output for a given layer, the memory storing the intermediate values corresponds to a size of intermediate information for a given layer, as in order to store this data in the memory, it is obvious to one of ordinary skill in the art that a correspondingly sized portion of memory is allocated after determining the appropriate respective size for storing the data.)
Choudhury-Zhu and Ma are both considered to be analogous to the claimed invention because they are in the same field of creating systems for inference models in computers. Therefore, it would have been obvious to someone of ordinary skill in the art prior to the effective filing date of the claimed invention to have modified Choudhury-Zhu to incorporate the teachings of Ma to modify the inference model system using dynamic batch sizes and utilizing a sequential inference model to generate a prediction to have allocations in available GPU memory with allocated sizes for data of each step of each layer, allowing for improved inference accuracy and improved performance from pipelining via reduced runtime, as described in Ma.
Regarding Claim 9, Choudhury-Zhu-Ma as described in Claim 8 discloses from Zhu wherein, as a result of calculating the step output for the each of the subset of layers ([0095] data storage 801 stores weight parameters and/or input/output data of each layer of a neural network trained or used in conjunction with one or more embodiments during forward propagation of input/output data […] during training and/or inferencing. Please note that weight parameters and input/output data of each layer during forward propagation of input/output data corresponds to Applicant’s result of calculating the step output for the each of the subset of layers.), the program instructions further comprise: releasing the portion of the GPU memory allocation corresponding to the current layer of the subset of layers (([0075] processor causes additional copies of output from precedent layer to be provided to additional subsequent layers that are to be performed on same […] GPUs.; [0100] different layers of a neural network, such that resulting activation from one storage/computational pair 801/802 of code […] is provided as an input to a next storage/computational pair 805/806. Please note that, as is common knowledge to a person having ordinary skill in the art, the generation of an activation that can be provided as an input to a next layer in a neural network requires the release of the first layer, corresponding to Applicant’s releasing the portion of the memory allocation corresponding to the current layer of the subset of layers. In this case, since the GPU is processing the data that is copied, this corresponds to the available GPU memory allocations, as the layers being performed on GPUs would be in the available GPU memory allocations as they are processed.); and loading and running a next layer of the inference model to create further layers for a subsequent step ([0093] causes the loading of weight or other parameter information into processor ALUs based on an architecture of a neural network; [0090] data storage 801 stores weight parameters and/or input/output data of each layer of a neural network trained or used in conjunction with one or more embodiments during forward propagation of input/output data and/or weight parameters during training and/or inferencing Please note that loading weights/parameter information into processor ALUs based on an architecture of a neural network corresponds to Applicant’s loading a next layer of the inference model. Furthermore, forward propagation of weight parameters during inferencing corresponds to Applicant’s running a next layer of the inference model to create further layers for a subsequent step.).
Regarding Claim 10, Choudhury-Zhu-Ma as described in Claim 9 discloses from Zhu wherein the released one of the subset of layers is a lowest level layer among the subset of layers and the loaded one of the subset of layers is a next higher level layer relative to the subset of layers ([0072] neural networks are modified 204 so that output from a first layer of a neural network […] is provided to a second layer. Please note that a first layer corresponds to Applicant’s lowest level layer among the subset of layers and a second layer corresponds to Applicant’s next higher level relative to the subset of layers, as layer 1 would be the lowest level layer, and layer 2 would be the next higher level relative to the subset of layers in a situation where there are at least 2 layers. Furthermore, as layer 2 is to be loaded as part of the operation of a neural network as is known in the art and is dependent on output from layer 1, layer 1 must first be released to obtain its output.).
Regarding Claim 12, Choudhury-Zhu-Ma as described in Claim 8 discloses from Zhu wherein memory identification information indicates a location within the available GPU memory allocations where the step output is stored ([0075] processor causes additional copies of output from precedent layer to be provided to additional subsequent layers that are to be performed on same […] GPUs. Please note that the copies of output from the precedent layer for subsequent layers corresponds to Applicant’s memory identification information indicating a location within the allocations where the step output is stored, as it is known to a person of ordinary skill in the art that accessing a copy of data uses a reference to its location in memory, corresponding to Applicant’s memory identification information indicating a location within the allocations where the step output is stored. In this case, since the GPU is processing the data that is copied, this corresponds to the available GPU memory allocations, as the layers being performed on GPUs would be in the available GPU memory allocations as they are processed.).
Regarding Claim 14, Choudhury-Zhu-Ma as described in Claim 8 discloses from Zhu wherein the step output is retained in the allocations until a last layer of the layers of the inference model is processed ([0075] additional subsequent layers also receive a copied output from a precedent layer. Please note that additional subsequent layers receiving copied output from a precedent corresponds to Applicant’s step output being retained in the allocations until a last layer of the layers of the inference model is processed, as a copy of output corresponding to Applicant’s step output will have to be retained in allocations until the process completes execution, i.e., reaches the last layer after which there is no subsequent layer and thus no more copied step output to be retained).
Regarding Claim 15, Choudhury discloses A computer system ([0004] a computer to carry out a plurality of method steps […] implemented in the form of a system) for running an inference model with a graphical processing unit (GPU) having a restrained resource ([0014] inferencing using one or more models used for inferencing and resource constraints (such as total available memory, maximum latency for inferencing, maximum energy for inferencing, etc.). Please note that total available memory corresponds to Applicant’s available GPU memory, as GPU memory is a variant of memory as is known in the art, resource constraints correspond to Applicant’s restrained resource, and the models used for inferencing with resource constraints such as total available memory corresponds to Applicant’s available GPU memory on which the model is run.), the computer system comprising: one or more computer processors ([0004] form of a system including [...] at least one processor), one or more computer-readable storage media ([0004] tangible computer-readable storage medium), and program instructions stored on the one or more of the computer-readable storage media for execution ([0004] computer readable instructions which, when implemented, cause a computer to carry out a plurality of method steps) by at least one of the one or more processors capable of performing a method ([0004] processor […] configured to perform noted method steps), the method comprising:
in response to an edge device request, receiving by an inference program from a model provisioning program, a trained inference model, wherein the trained inference model is configured to execute intended objectives of the edge device ([0002] Deep neural networks are used for a variety of artificial intelligence applications such as computer vision, speech recognition, natural language processing, etc. Additionally, such deep learning models can be used on mobile phones and other edge devices in the context of Internet of Things (IoT). Thus, inferencing can be carried out either on the cloud or the edge device itself. Inferencing, as used herein, refers to the stage wherein a trained network predicts and/or classifies input test samples. Please note that models being used on edge devices for AI applications corresponds to a trained inference model being configured to execute intended objectives of an edge device, which makes an edge device request, i.e., a request to execute an AI application. Furthermore, since inferencing is done with a trained network, this corresponds to receiving a trained inference model by an inference program from a model provisioning program, as it is known in the art that there must be means for provisioning the trained inference model to be used in the requested application of the edge device.);
based on a configuration of the trained inference model, loading by the inference program, a subset of total number of layers in the inference model ([0014] determining individual layer batch sizes for inferencing using one or more models used for inferencing and resource constraints. [0018] a batch size optimizer. In such an embodiment, L.sub.1, L.sub.2, . . . , L.sub.n represent the n layers of the network. A simple path network, for example, can include an output of layer L.sub.i being fed only into its successor layer L.sub.i+1. Please note that the resource constraints correspond to Applicant’s configuration of the trained inference model, and determining individual layer batch sizes for inferencing, where a batch size optimizer that operates with representations of n layers of the network, corresponds to Applicant’s loading a subset of total number of layers in the inference model, as it processes a subset of the inference model based on resource constraints and operates with n layers.) wherein the subset of layers is a count that is fewer than the total number of layers in the inference model and is based on available GPU memory, wherein the available GPU memory is a portion of total GPU that is configured for running the inference model ([0014] determining individual layer batch sizes for inferencing using one or more models used for inferencing and resource constraints (such as total available memory); [0016] given memory availability […] a memory requirement of layer L.sub.2 104 can restrict the batch size that can be processed for the network; Fig 1. Please note that in each example of batch sizes for each layer corresponding to Applicant’s count of the subset of the layers, the batch size is less than the available memory, corresponding to Applicant’s count being a value less than the total number of layers of the inference model, and since it is based on memory availability, this corresponds to being based on available GPU memory. Additionally, as the resource constraints for inferencing using the models include total available memory, this corresponds to the available GPU memory being a portion of total GPU that is configured for running the inference model.);
Choudhury does not explicitly disclose until all of the subset of layers is processed, loading and running, by the inference program into a corresponding GPU memory allocation, a current layer of the subset of layers along with a corresponding step input, giving a step output; based on there being more layers of the subset of layers, releasing, by the inference program, a portion of the GPU memory allocation corresponding to the current layer of the subset of layers; loading, by the inference program, a next layer of the subset of layers along with the corresponding step input, giving a corresponding step output; and based on all of the subset of layers being processed, outputting by the inference program a prediction
However, Zhu discloses until all of the subset of layers is processed, loading and running, by the inference program into a corresponding GPU memory allocation, a current layer of the subset of layers along with a corresponding step input, giving a step output ([0075] processor causes additional copies of output from precedent layer to be provided to additional subsequent layers that are to be performed on same […] GPUs.; [0092] inference […] may include […] input/output data corresponding to neurons or layers of a neural network; [0093] causes the loading of weight or other parameter information into processor ALUs based on an architecture of a neural network; [0090] weight parameters […] used in conjunction with […] inferencing. Please note that loading weight parameters and using them in conjunction with inferencing corresponds to Applicant’s loading and running step. Please note that in this case, since the GPU is processing the data that is copied, this corresponds to the available GPU memory allocations, as the layers being performed on GPUs would be in the available GPU memory allocations as they are processed. Since the processor causes copies of output from precedent layers to be provided to subsequent layers after using input data, this corresponds to running the current layer of the subset of layers along with a corresponding step input, giving a step output.);
based on there being more layers of the subset of layers, releasing, by the inference program, a portion of the GPU memory allocation corresponding to the current layer of the subset of layers (([0075] processor causes additional copies of output from precedent layer to be provided to additional subsequent layers that are to be performed on same […] GPUs.; [0100] different layers of a neural network, such that resulting activation from one storage/computational pair 801/802 of code […] is provided as an input to a next storage/computational pair 805/806. Please note that, as is common knowledge to a person having ordinary skill in the art, the generation of an activation that can be provided as an input to a next layer in a neural network requires the release of the first layer, corresponding to Applicant’s releasing the portion of the GPU memory allocation corresponding to the current layer of the subset of layers. In this case, since the GPU is processing the data that is copied, this corresponds to the available GPU memory allocations, as the layers being performed on GPUs would be in the available GPU memory allocations as they are processed.);
loading, by the inference program, a next layer of the subset of layers along with the corresponding step input, giving a corresponding step output ( [0092] inference […] may include […] input/output data corresponding to neurons or layers of a neural network; [0093] causes the loading of weight or other parameter information into processor ALUs based on an architecture of a neural network; [0090] weight parameters […] used in conjunction with […] inferencing. Please note that loading weight parameters and using them in conjunction with inferencing corresponds to Applicant’s loading and running step. Please note that since the processor causes copies of output from precedent layers to be provided to subsequent layers after using input data, proceeding to the next layer, this corresponds to loading a next layer of the subset of layers along with a corresponding step input, giving a corresponding step output.);
and based on all of the subset of layers being processed, outputting by the inference program a prediction ([0113] In at least one embodiment, trained machine learning models corresponding to one or more neural networks may be used to infer or predict information using resources described above with respect to data center 1000 by using weight parameters calculated through one or more training techniques described herein. Please note that the machine learning models using weight parameters calculated through the described training techniques to predict or infer information corresponds to Applicant’s outputting the prediction by the inference program. It is known in the art that a prediction of a model is output once its layers have been processed; therefore, this would include all of the subset of layers as well.).
Choudhury and Zhu are both considered to be analogous to the claimed invention because they are in the same field of creating systems for inference models in computers. Therefore, it would have been obvious to someone of ordinary skill in the art prior to the effective filing date of the claimed invention to have modified Choudhury to incorporate the teachings of Zhu to modify the inference model system using dynamic batch sizes to utilize a sequential inference model to generate a prediction, allowing for more efficient use of available resources and improved performance from pipelining, as described in Zhu.
Choudhury-Zhu does not explicitly disclose dividing, by the inference program, the available GPU memory into a number of allocations comprising a layers allocation, an input/output allocation, and an intermediate information allocation, wherein a size of allocations is determined based on a size of the subset of layers, a size of a step input for a given layer, a size of a step output for a given layer, and a size of intermediate information for a given layer,
However, Ma discloses dividing, by the inference program, the available GPU memory into a number of allocations comprising a layers allocation ([0006] Neural connections among the artificial neurons are formed by interconnect circuitry coupled to control lines of the memory array to subdivide the memory array into a plurality of layers of the artificial neural network. Control circuitry is configured to transmit a plurality of iterations of an input value on input control lines of a first layer of the artificial neural network for inference operations by at least one or more additional layers.; [0019] A graphics processing unit (GPU), which has started to gain favor over CPUs, uses a parallel architecture and can handle many sets of very simple instructions. Please note that subdividing the memory array into a plurality of layers of the ANN that carries out inference operations, where the GPU is handling instructions, corresponds to Applicant’s dividing the available GPU memory into a number of allocations comprising a layers allocation by the inference program.), an input/output allocation ([0037] Memory 131 can also be configured to store input data and output data. Please note that storing input and output data in the memory corresponds to Applicant’s input/output allocation, as in order to store this data in the memory, it is obvious to one of ordinary skill in the art that a portion of memory is allocated. ), and an intermediate information allocation ([0037] memory 131 can store intermediate and final values for an ANN pipeline. Please note that storing intermediate values for an ANN pipeline in the memory corresponds to Applicant’s intermediate information allocation, as in order to store this data in the memory, it is obvious to one of ordinary skill in the art that a portion of memory is allocated. ), wherein a size of allocations is determined based on a size of the subset of layers, a size of a step input for a given layer, a size of a step output for a given layer, and a size of intermediate information for a given layer, ([0037] In one example, memory 131 is employed as an output buffer to store synaptic weights for artificial neurons of an ANN. Control system 130 can load these synaptic weights into NVM elements of memory array 110 prior to introduction of input data to the ANN layers. Memory 131 can also be configured to store input data and output data. [...] In the examples below, an ANN pipelining technique is discussed, and memory 131 can store intermediate and final values for an ANN pipeline. [0038] Memory array 110 can be used to implement a single layer of an artificial neural network, or instead might implement a multi-layer ANN. Please note that the memory array implementing a multi-layer ANN corresponds to the size of the subset of layers, the memory being configured to store the input data corresponds to a size of a step input for a given layer, the memory being configured to store the output data corresponds to a size of a step output for a given layer, the memory storing the intermediate values corresponds to a size of intermediate information for a given layer, as in order to store this data in the memory, it is obvious to one of ordinary skill in the art that a correspondingly sized portion of memory is allocated after determining the appropriate respective size for storing the data.)
Choudhury-Zhu and Ma are both considered to be analogous to the claimed invention because they are in the same field of creating systems for inference models in computers. Therefore, it would have been obvious to someone of ordinary skill in the art prior to the effective filing date of the claimed invention to have modified Choudhury-Zhu to incorporate the teachings of Ma to modify the inference model system using dynamic batch sizes and utilizing a sequential inference model to generate a prediction to have allocations in available GPU memory with allocated sizes for data of each step of each layer, allowing for improved inference accuracy and improved performance from pipelining via reduced runtime, as described in Ma.
Regarding Claim 16, Choudhury-Zhu-Ma as described in Claim 15 discloses from Zhu wherein, as a result of calculating the step output for the each of the subset of layers ([0095] data storage 801 stores weight parameters and/or input/output data of each layer of a neural network trained or used in conjunction with one or more embodiments during forward propagation of input/output data […] during training and/or inferencing. Please note that weight parameters and input/output data of each layer during forward propagation of input/output data corresponds to Applicant’s result of calculating the step output for the each of the subset of layers.), the method further comprises: releasing the portion of the GPU memory allocation corresponding to the current layer of the subset of layers (([0075] processor causes additional copies of output from precedent layer to be provided to additional subsequent layers that are to be performed on same […] GPUs.; [0100] different layers of a neural network, such that resulting activation from one storage/computational pair 801/802 of code […] is provided as an input to a next storage/computational pair 805/806. Please note that, as is common knowledge to a person having ordinary skill in the art, the generation of an activation that can be provided as an input to a next layer in a neural network requires the release of the first layer, corresponding to Applicant’s releasing the portion of the memory allocation corresponding to the current layer of the subset of layers. In this case, since the GPU is processing the data that is copied, this corresponds to the available GPU memory allocations, as the layers being performed on GPUs would be in the available GPU memory allocations as they are processed.); and loading and running a next layer of the inference model to create further layers for a subsequent step ([0093] causes the loading of weight or other parameter information into processor ALUs based on an architecture of a neural network; [0090] data storage 801 stores weight parameters and/or input/output data of each layer of a neural network trained or used in conjunction with one or more embodiments during forward propagation of input/output data and/or weight parameters during training and/or inferencing Please note that loading weights/parameter information into processor ALUs based on an architecture of a neural network corresponds to Applicant’s loading a next layer of the inference model. Furthermore, forward propagation of weight parameters during inferencing corresponds to Applicant’s running a next layer of the inference model to create further layers for a subsequent step.).
Regarding Claim 17, Choudhury-Zhu-Ma as described in Claim 16 discloses from Zhu wherein the released one of the subset of layers is a lowest level layer among the subset of layers and the loaded one of the subset of layers is a next higher level layer relative to the subset of layers ([0072] neural networks are modified 204 so that output from a first layer of a neural network […] is provided to a second layer. Please note that a first layer corresponds to Applicant’s lowest level layer among the subset of layers and a second layer corresponds to Applicant’s next higher level relative to the subset of layers, as layer 1 would be the lowest level layer, and layer 2 would be the next higher level relative to the subset of layers in a situation where there are at least 2 layers. Furthermore, as layer 2 is to be loaded as part of the operation of a neural network as is known in the art and is dependent on output from layer 1, layer 1 must first be released to obtain its output.).
Regarding Claim 19, Choudhury-Zhu-Ma as described in Claim 15 discloses from Zhu wherein memory identification information indicates a location within the available GPU memory allocations where the step output is stored ([0075] processor causes additional copies of output from precedent layer to be provided to additional subsequent layers that are to be performed on same […] GPUs. Please note that the copies of output from the precedent layer for subsequent layers corresponds to Applicant’s memory identification information indicating a location within the allocations where the step output is stored, as it is known to a person of ordinary skill in the art that accessing a copy of data uses a reference to its location in memory, corresponding to Applicant’s memory identification information indicating a location within the allocations where the step output is stored. In this case, since the GPU is processing the data that is copied, this corresponds to the available GPU memory allocations, as the layers being performed on GPUs would be in the available GPU memory allocations as they are processed.).
Claims 4, 6, 11, 13, 18, and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Choudhury et al. (US 20200125926 A1) in view of Zhu et al. (US 20210374518 A1), and further in view of Ma et al. (US 20200012924 A1), as applied to Claims 2, 8, 9, and 15 above, and further in view of Fuchs et al. (US 20190295252 A1), hereinafter referred to as Choudhury, Zhu, Ma, and Fuchs, respectively.
Regarding Claim 4, Choudhury-Zhu-Ma as described in Claim 2 discloses from Zhu loading and running , by the inference program into the corresponding GPU memory allocation, the current layer of the subset of layers along with the corresponding step input, giving the step output ([0075] processor causes additional copies of output from precedent layer to be provided to additional subsequent layers that are to be performed on same […] GPUs.; [0092] inference […] may include […] input/output data corresponding to neurons or layers of a neural network; [0093] causes the loading of weight or other parameter information into processor ALUs based on an architecture of a neural network; [0090] weight parameters […] used in conjunction with […] inferencing. Please note that loading weight parameters and using them in conjunction with inferencing corresponds to Applicant’s loading and running step. Please note that in this case, since the GPU is processing the data that is copied, this corresponds to the available GPU memory allocations, as the layers being performed on GPUs would be in the available GPU memory allocations as they are processed. Since the processor causes copies of output from precedent layers to be provided to subsequent layers after using input data, this corresponds to running the current layer of the subset of layers along with the corresponding step input, giving the step output.), outputting by the inference program the prediction ([0113] In at least one embodiment, trained machine learning models corresponding to one or more neural networks may be used to infer or predict information using resources described above with respect to data center 1000 by using weight parameters calculated through one or more training techniques described herein. Please note that the machine learning models using weight parameters calculated through the described training techniques to predict or infer information corresponds to Applicant’s outputting the prediction by the inference program.).
Choudhury-Zhu-Ma does not explicitly disclose based on all of the subset of layers being processed;
until all of the subset of layers is processed .
However, Fuchs discloses based on all of the subset of layers being processed ([0172] The model applier 3218 may repeat the feeding of the output of one transform layer into the input of the subsequent transform layer in the inference model 3212 until the last transform layer. Please note that repeating the feeding of the output of one transform layer into the input of a subsequent transform layer in the inference model 3212 until the last transform layer corresponds to Applicant’s all of the subset of layers being processed, as it involves ceasing the repetition of the subsequent layers once the last layer is reached, i.e., the whole subset has been processed.);
until all of the subset of layers is processed ([0172] The model applier 3218 may repeat the feeding of the output of one transform layer into the input of the subsequent transform layer in the inference model 3212 until the last transform layer. Please note that repeating the feeding of the output of one transform layer into the input of a subsequent transform layer in the inference model 3212 until the last transform layer corresponds to Applicant’s until all of the subset of layers is processed, as it involves ceasing the repetition of the subsequent layers once the last layer is reached, i.e., the whole subset has been processed.).
Choudhury-Zhu-Ma and Fuchs are both considered to be analogous to the claimed invention because they are in the same field of creating systems for inference models in computers. Therefore, it would have been obvious to someone of ordinary skill in the art prior to the effective filing date of the claimed invention to have modified Choudhury-Zhu-Ma to incorporate the teachings of Fuchs to modify the sequential inference model system to repeat the steps until all of the subset of layers are processed, allowing for improved efficacy of resource usage as described in Fuchs.
Regarding Claim 6, Choudhury-Zhu-Ma as described in Claim 1 does not explicitly disclose wherein the step input for a first step of a sequential inference process corresponds to the input and the step output for a last step of the sequential inference process corresponds to the output.
However, Fuchs discloses wherein the step input for a first step of the sequential inference process corresponds to the input ([0172] The input of the first transform layer may be the set of tiles 3236. Please note that the set of tiles being input in the first transform layer corresponds to Applicant’s step input for a first step of the sequential inference process corresponding to the input.) and the step output for a last step of the sequential inference process corresponds to the output ([0172] identify the output of the last transform layer in the inference model 3212. Please note that output of the last transform layer in the inference model corresponds to Applicant’s step output for a last step of the sequential inference process corresponding to the output, as it is being identified as the output of the process.).
Choudhury-Zhu-Ma and Fuchs are both considered to be analogous to the claimed invention because they are in the same field of creating systems for inference models in computers. Therefore, it would have been obvious to someone of ordinary skill in the art prior to the effective filing date of the claimed invention to have modified Choudhury-Zhu-Ma to incorporate the teachings of Fuchs to modify the sequential inference model system to have the first step’s step input correspond to the input and the last step’s step output correspond to the output, allowing for an efficient pipelined system as described in Fuchs.
Regarding Claim 11, Choudhury-Zhu-Ma as described in Claim 9 discloses from Zhu loading and running , by the inference program into the corresponding GPU memory allocation, the current layer of the subset of layers along with the corresponding step input, giving the step output ([0075] processor causes additional copies of output from precedent layer to be provided to additional subsequent layers that are to be performed on same […] GPUs.; [0092] inference […] may include […] input/output data corresponding to neurons or layers of a neural network; [0093] causes the loading of weight or other parameter information into processor ALUs based on an architecture of a neural network; [0090] weight parameters […] used in conjunction with […] inferencing. Please note that loading weight parameters and using them in conjunction with inferencing corresponds to Applicant’s loading and running step. Please note that in this case, since the GPU is processing the data that is copied, this corresponds to the available GPU memory allocations, as the layers being performed on GPUs would be in the available GPU memory allocations as they are processed. Since the processor causes copies of output from precedent layers to be provided to subsequent layers after using input data, this corresponds to running the current layer of the subset of layers along with the corresponding step input, giving the step output.), outputting by the inference program the prediction ([0113] In at least one embodiment, trained machine learning models corresponding to one or more neural networks may be used to infer or predict information using resources described above with respect to data center 1000 by using weight parameters calculated through one or more training techniques described herein. Please note that the machine learning models using weight parameters calculated through the described training techniques to predict or infer information corresponds to Applicant’s outputting the prediction by the inference program.).
Choudhury-Zhu-Ma does not explicitly disclose based on all of the subset of layers being processed;
until all of the subset of layers is processed .
However, Fuchs discloses based on all of the subset of layers being processed ([0172] The model applier 3218 may repeat the feeding of the output of one transform layer into the input of the subsequent transform layer in the inference model 3212 until the last transform layer. Please note that repeating the feeding of the output of one transform layer into the input of a subsequent transform layer in the inference model 3212 until the last transform layer corresponds to Applicant’s all of the subset of layers being processed, as it involves ceasing the repetition of the subsequent layers once the last layer is reached, i.e., the whole subset has been processed.);
until all of the subset of layers is processed ([0172] The model applier 3218 may repeat the feeding of the output of one transform layer into the input of the subsequent transform layer in the inference model 3212 until the last transform layer. Please note that repeating the feeding of the output of one transform layer into the input of a subsequent transform layer in the inference model 3212 until the last transform layer corresponds to Applicant’s until all of the subset of layers is processed, as it involves ceasing the repetition of the subsequent layers once the last layer is reached, i.e., the whole subset has been processed.).
Choudhury-Zhu-Ma and Fuchs are both considered to be analogous to the claimed invention because they are in the same field of creating systems for inference models in computers. Therefore, it would have been obvious to someone of ordinary skill in the art prior to the effective filing date of the claimed invention to have modified Choudhury-Zhu-Ma to incorporate the teachings of Fuchs to modify the sequential inference model system to repeat the steps until all of the subset of layers are processed, allowing for improved efficacy of resource usage as described in Fuchs.
Regarding Claim 13, Choudhury-Zhu-Ma as described in Claim 8 does not explicitly disclose wherein the step input for a first step of a sequential inference process corresponds to the input and the step output for a last step of the sequential inference process corresponds to the output.
However, Fuchs discloses wherein the step input for a first step of the sequential inference process corresponds to the input ([0172] The input of the first transform layer may be the set of tiles 3236. Please note that the set of tiles being input in the first transform layer corresponds to Applicant’s step input for a first step of the sequential inference process corresponding to the input.) and the step output for a last step of the sequential inference process corresponds to the output ([0172] identify the output of the last transform layer in the inference model 3212. Please note that output of the last transform layer in the inference model corresponds to Applicant’s step output for a last step of the sequential inference process corresponding to the output, as it is being identified as the output of the process.).
Choudhury-Zhu-Ma and Fuchs are both considered to be analogous to the claimed invention because they are in the same field of creating systems for inference models in computers. Therefore, it would have been obvious to someone of ordinary skill in the art prior to the effective filing date of the claimed invention to have modified Choudhury-Zhu-Ma to incorporate the teachings of Fuchs to modify the sequential inference model system to have the first step’s step input correspond to the input and the last step’s step output correspond to the output, allowing for an efficient pipelined system as described in Fuchs.
Regarding Claim 18, Choudhury-Zhu-Ma as described in Claim 15 discloses from Zhu loading and running , by the inference program into the corresponding GPU memory allocation, the current layer of the subset of layers along with the corresponding step input, giving the step output ([0075] processor causes additional copies of output from precedent layer to be provided to additional subsequent layers that are to be performed on same […] GPUs.; [0092] inference […] may include […] input/output data corresponding to neurons or layers of a neural network; [0093] causes the loading of weight or other parameter information into processor ALUs based on an architecture of a neural network; [0090] weight parameters […] used in conjunction with […] inferencing. Please note that loading weight parameters and using them in conjunction with inferencing corresponds to Applicant’s loading and running step. Please note that in this case, since the GPU is processing the data that is copied, this corresponds to the available GPU memory allocations, as the layers being performed on GPUs would be in the available GPU memory allocations as they are processed. Since the processor causes copies of output from precedent layers to be provided to subsequent layers after using input data, this corresponds to running the current layer of the subset of layers along with the corresponding step input, giving the step output.), outputting by the inference program the prediction ([0113] In at least one embodiment, trained machine learning models corresponding to one or more neural networks may be used to infer or predict information using resources described above with respect to data center 1000 by using weight parameters calculated through one or more training techniques described herein. Please note that the machine learning models using weight parameters calculated through the described training techniques to predict or infer information corresponds to Applicant’s outputting the prediction by the inference program.).
Choudhury-Zhu-Ma does not explicitly disclose based on all of the subset of layers being processed;
until all of the subset of layers is processed .
However, Fuchs discloses based on all of the subset of layers being processed ([0172] The model applier 3218 may repeat the feeding of the output of one transform layer into the input of the subsequent transform layer in the inference model 3212 until the last transform layer. Please note that repeating the feeding of the output of one transform layer into the input of a subsequent transform layer in the inference model 3212 until the last transform layer corresponds to Applicant’s all of the subset of layers being processed, as it involves ceasing the repetition of the subsequent layers once the last layer is reached, i.e., the whole subset has been processed.);
until all of the subset of layers is processed ([0172] The model applier 3218 may repeat the feeding of the output of one transform layer into the input of the subsequent transform layer in the inference model 3212 until the last transform layer. Please note that repeating the feeding of the output of one transform layer into the input of a subsequent transform layer in the inference model 3212 until the last transform layer corresponds to Applicant’s until all of the subset of layers is processed, as it involves ceasing the repetition of the subsequent layers once the last layer is reached, i.e., the whole subset has been processed.).
Choudhury-Zhu-Ma and Fuchs are both considered to be analogous to the claimed invention because they are in the same field of creating systems for inference models in computers. Therefore, it would have been obvious to someone of ordinary skill in the art prior to the effective filing date of the claimed invention to have modified Choudhury-Zhu-Ma to incorporate the teachings of Fuchs to modify the sequential inference model system to repeat the steps until all of the subset of layers are processed, allowing for improved efficacy of resource usage as described in Fuchs.
Regarding Claim 20, Choudhury-Zhu-Ma as described in Claim 15 does not explicitly disclose wherein the step input for a first step of a sequential inference process corresponds to the input and the step output for a last step of the sequential inference process corresponds to the output.
However, Fuchs discloses wherein the step input for a first step of the sequential inference process corresponds to the input ([0172] The input of the first transform layer may be the set of tiles 3236. Please note that the set of tiles being input in the first transform layer corresponds to Applicant’s step input for a first step of a sequential inference process corresponding to the input.) and the step output for a last step of the sequential inference process corresponds to the output ([0172] identify the output of the last transform layer in the inference model 3212. Please note that output of the last transform layer in the inference model corresponds to Applicant’s step output for a last step of the sequential inference process corresponding to the output, as it is being identified as the output of the process.).
Choudhury-Zhu-Ma and Fuchs are both considered to be analogous to the claimed invention because they are in the same field of creating systems for inference models in computers. Therefore, it would have been obvious to someone of ordinary skill in the art prior to the effective filing date of the claimed invention to have modified Choudhury-Zhu-Ma to incorporate the teachings of Fuchs to modify the sequential inference model system to have the first step’s step input correspond to the input and the last step’s step output correspond to the output, allowing for an efficient pipelined system as described in Fuchs.
Response to Arguments
Applicant’s arguments filed 02/05/2026 have been fully considered but they are not persuasive.
Applicant’s arguments are summarized as follows:
Claims 1, 3, 8, 10, 15, and 17 have been sufficiently amended to overcome their objections.
Regarding Claims 1, 8, and 15 rejected under 35 U.S.C. 103 as being unpatentable over Choudhury in view of Zhu, Choudhury discloses a batch, which is not the same as a layer, and does not disclose a subset of layers as calculated in [0048] of the Specification. Additionally, the “resource constraints” do not correspond to “configuration of the trained inference model.” Choudhury also does not disclose more than one layer per batch size, whereas the “subset of layers” as disclosed is more than one layer and is the number of layers of the inference model that can be loaded into the available GPU memory.
Zhu teaches that the layers can be dispatched to different distinct computing devices, but not which of many circuits within a processor a process executes, as corresponds to the limitation “dividing, by the inference program, the available GPU memory into a layers allocation, an input/output allocation, and an intermediate information allocation, wherein a size of allocations is determined based on the available GPU memory.” Zhu additionally does not teach “an input/output allocation, and an intermediate information allocation, wherein a size of allocations is determined based on the available GPU memory,” or “until all of the subset of layers is processed, loading and running, by the inference program into a corresponding GPU memory allocation,” as Applicant does not copy data, and though Zhu relies on weight parameters, Applicant does not disclose or claim weight parameters.
Therefore, the limitations of the Claims are not disclosed. The rejections under 35 U.S.C. 103 should be withdrawn.
All objections and rejections have been overcome, all pending claims are in a form suitable for allowance, and the application is in condition for allowance.
Regarding A), the amended claims have been corrected sufficiently to overcome the basis for objection; therefore, the objections are withdrawn.
Regarding B), the examiner respectfully disagrees. The Applicant’s arguments are moot, as the rejections of the Claim now relies on a new grounds of rejection, Zhu-Choudhury-Ma, which discloses the limitations stated by the Applicant via the combination of references, as stated above. Therefore, the recited features can be found in the cited combination of references, and independent Claims 1, 8, and 15 remain rejected under 35 U.S.C. 103 for the reasons stated above, and the combinations cited would have been obvious to a person of ordinary skill in the art prior to the effective filing date of the application. The rejections under 35 U.S.C. 103 are maintained.
Regarding C), the examiner respectfully disagrees. As previously stated, independent claims 1, 8, and 15 remain rejected under 35 U.S.C. 103 for the reasons stated above, and the combinations cited would have been obvious to a person of ordinary skill in the art prior to the effective filing date of the application. Therefore, contrary to Applicant’s arguments, because the dependent claims depend on unpatentable claims and do not add limitations that overcome the rejection, they likewise remain rejected, and the application is not in condition for allowance.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Mathuriya et al. (US 20190057300 A1) discloses memory circuitry storing intermediate input/data values generated by some of the hidden layers of a neural network, dynamic allocation based on the size/number of layers, utilizing GPUs, and performing inference calculations (see [0044-0046, 0049]).
Any inquiry concerning this communication or earlier communications from the examiner should be directed to FARAZ T AKBARI whose telephone number is (571)272-4166. The examiner can normally be reached Monday-Thursday 9:30am-7:30pm ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, April Blair can be reached at (571)270-1014. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/FARAZ T AKBARI/ Examiner, Art Unit 2196
/APRIL Y BLAIR/ Supervisory Patent Examiner, Art Unit 2196
Prosecution Insights