Detailed Action
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Objections
Claim 1 is objected to because of the following informalities: “detecting a sub-graph within the computing graph matches at least one defined sub-structure” should read as “detecting a sub-graph within the computing graph matching at least one defined sub-structure” or “detecting a sub-graph within the computing graph that matches at least one defined sub-structure” or “detecting a sub-graph within the computing graph, wherein the sub-graph matches at least one defined sub-structure”. In addition, “combing” should read as “combining”. Appropriate correction is required.
Further, claim 8 is objected for the following informality: “combing” should read as “combining”. Appropriate correction is required.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Regarding claim 20, the claim states, among its limitations, “…connecting a first set of directed edges of vertices of the computing graph to the single vertex, the first set of directed edges corresponding to inputs to the sub-graph; connecting a second set of directed edges from the single vertex to vertices of the computing graph, the first set of directed edges corresponding to outputs to the sub- graph…”. According to the limitation as written, the first set of directed edges corresponds to both the inputs and outputs to the subgraph, while the second set of directed edges seemingly is arbitrarily connected to the computing graph with no correspondence. There does not seem to be anything written in the specification to clarify this, and it is the examiner’s interpretation that the second set of directed edges are meant to be the ones corresponding to the outputs to the sub-graph while the first set of directed edges is meant to only correspond to the inputs to the sub-graph and thus, for the sake of further examination, it will be interpreted this way herein.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Regarding claim 1, in Step 1 of the 101-analysis set forth in MPEP 2106, the claim recites “A method”. A method is within one of the four statutory categories of invention.
In Step 2a Prong 1 of the 101-analysis set forth in the MPEP 2106, the examiner has determined that the following limitations recite a process that, under the broadest reasonable interpretation, covers an abstract idea but for recitation of generic computer components:
“A method comprising: generating a computing graph based on a neural network, the computing graph including a set of vertices representing operators and a set of directed edges representing a computing order and data dependency” (A person can mentally evaluate a neural network and make a judgement to draw out or “generate” a computing graph based on it (MPEP 2106).)
“detecting a sub-graph within the computing graph matches at least one defined sub-structure” (This recites a mental process. A person can mentally evaluate a computing graph and make a judgement to detect that a sub-graph within it matches another defined sub-structure. A mental process recites an abstract idea (MPEP 2106).)
“combing a subset of vertices of the set of vertices corresponding to the sub- graph into a vertex by at least connecting a subset of directed edges of the set of directed edges to the vertex to generate an optimized graph, where the subset of directed edges correspond to inputs and outputs associated with the subset of vertices” (A person can mentally evaluate a computing graph and make a judgement to optimize it by fusing vertices and edges (MPEP 2106).)
“generating a net-list object and a weight object based on the optimized graph and the set of parameters” (A person can mentally evaluate the optimized graph and set of parameters and make a judgement to “generate” corresponding lists based on that (MPEP 2106).)
If claim limitations, under their broadest reasonable interpretation, covers performance of the limitations as an abstract idea but for the recitation of generic computer components, then it “recites” an abstract idea.
In Step 2a Prong 2 of the 101-analysis set forth in MPEP 2106, the examiner has
determined that the following additional elements do not integrate this judicial exception into a
practical application:
“extracting a set of parameters from the optimized graph” (Adding insignificant extra-solution activity (mere data gathering) to the judicial exception (MPEP 2106.05(g)).)
“providing the net-list object and the weight object to an inference framework to enable the inference framework to perform inferencing” (Adding insignificant extra-solution activity (mere data output) to the judicial exception (MPEP 2106.05(g)).)
Since the claim as a whole, looking at the additional elements individually and in combination, does not contain any other additional elements that are indicative of integration into a practical application, the claim is “directed” to an abstract idea.
In Step 2b of the 101-analysis set forth in the 2019 PEG, the examiner has determined that the claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
As discussed above, additional elements (v) & (vi) recite insignificant extra-solution activities. Further, these elements recite steps of receiving/transmitting data via a network, which has been determined by the courts to recite a well-understood, routine, and conventional activity, which is not indicative of significantly more (Receiving or transmitting data over a network, e.g., using the Internet to gather data, Symantec, 838 F.3d at 1321, 120 USPQ2d at 1362). Considering the additional elements individually and in combination, and the claim as a whole, the additional elements do not provide significantly more than the abstract idea. Therefore, the claim is not patent eligible.
Regarding claim 2, it is dependent upon claim 1, and thereby incorporates the limitations of, and corresponding analysis applied to claim 1. Further, claim 2 recites “wherein the sub-graph includes an isomorphic sub-graph” (In step 2a, prong 2, this recites generally linking the use of the judicial exception to a particular technological environment or field of use (MPEP 2106.05(h).) In step 2B, generally linking the use of the judicial exception to a particular technological environment or field of use is not indicative of significantly more.)
Since the claim does not recite additional elements that either integrate the judicial exception into a practical application, nor provide significantly more than the judicial exception, the claim is not patent eligible.
Regarding claim 3, it is dependent upon claim 2, and thereby incorporates the limitations of, and corresponding analysis applied to claim 2. Further, claim 3 recites the following additional abstract idea:
“wherein detecting the sub-graph within the computing graph further comprises using a graph isomorphism algorithm to determine the sub-graph is equivalent to the defined sub-structure” (This limitation seems to be explicitly directed to the use of a graph isomorphism algorithm, which is a mathematical process. A mathematical process recites an abstract idea (MPEP 2106).)
Since the claim does not recite additional elements that either integrate the judicial exception into a practical application, nor provide significantly more than the judicial exception, the claim is not patent eligible.
Regarding claim 4, it is dependent upon claim 1, and thereby incorporates the limitations of, and corresponding analysis applied to claim 1. Further, claim 4 recites the following additional abstract idea:
“wherein the method further comprises merging at least two parameters of the set of parameters” (This limitation seems to be explicitly directed to the combination of two or more numerical values, which is a mathematical process. A mathematical process recites an abstract idea (MPEP 2106).)
Since the claim does not recite additional elements that either integrate the judicial exception into a practical application, nor provide significantly more than the judicial exception, the claim is not patent eligible.
Regarding claim 5, it is dependent upon claim 4, and thereby incorporates the limitations of, and corresponding analysis applied to claim 4. Further, claim 5 recites the following additional abstract idea:
“wherein merging the at least two parameters further comprises merging a subset of parameters of the set of parameters into a weight value and a bias value, where the subset of parameters correspond to a batch normalization operation” (This limitation seems to be explicitly directed to the combination of two or more numerical values, which now must correspond to a batch normalization operation, which is a mathematical process. A mathematical process recites an abstract idea (MPEP 2106).)
Since the claim does not recite additional elements that either integrate the judicial exception into a practical application, nor provide significantly more than the judicial exception, the claim is not patent eligible.
Regarding claim 6, it is dependent upon claim 1, and thereby incorporates the limitations of, and corresponding analysis applied to claim 1. Further, claim 6 recites “wherein the at least one defined sub-structure is stored in a macro library and indicates a set of operations of the neural network that can be performed by an operation of a kernel” (In step 2A, prong 2, this recites insignificant extra-solution activity (mere data storage) to the judicial exception (MPEP 2106.05(g).) In step 2B, the courts have found steps that store and retrieve information in memory to be a well-understood, routine, and conventional activity, which is not indicative of significantly more (Storing and retrieving information in memory, Versata Dev. Group, Inc. v. SAP Am., Inc., 793 F.3d 1306, 1334, 115 USPQ2d 1681, 1701 (Fed. Cir. 2015)).)
Since the claim does not recite additional elements that either integrate the judicial exception into a practical application, nor provide significantly more than the judicial exception, the claim is not patent eligible.
Regarding claim 7, it is dependent upon claim 1, and thereby incorporates the limitations of, and corresponding analysis applied to claim 1. Further, claim 7 recites “wherein the computing graphs further comprises a state space representation” (In step 2a, prong 2, this recites generally linking the use of the judicial exception to a particular technological environment or field of use (MPEP 2106.05(h).) In step 2B, generally linking the use of the judicial exception to a particular technological environment or field of use is not indicative of significantly more.)
Since the claim does not recite additional elements that either integrate the judicial exception into a practical application, nor provide significantly more than the judicial exception, the claim is not patent eligible.
Regarding claim 8, in Step 1 of the 101-analysis set forth in MPEP 2106, the claim recites “A non-transitory computer-readable medium”. A non-transitory medium is within one of the four statutory categories of invention.
Further, claim 8 comprises similar limitations as claim 1 and is rejected under the same rationale, with the following additional limitation: “A non-transitory computer-readable medium storing executable instructions embodied thereon, which, when executed by a processing device, cause the processing device to perform operations comprising…” (In step2A, prong 2, this recites using a computer as a tool to perform an abstract idea (MPEP 2106.05(f).) In step 2B, using a computer as a tool to perform an abstract idea is not indicative of significantly more.)
Since the claim does not recite additional elements that either integrate the judicial exception into a practical application, nor provide significantly more than the judicial exception, the claim is not patent eligible.
Regarding claim 9, it is dependent upon claim 8, and thereby incorporates the limitations of, and corresponding analysis applied to claim 8. Further, claim 9 recites the following additional mental process:
“wherein the net-list indicates connectivity between layers of the machine learning model and a set of kernel operations associated with the layers, where the set of kernel operations are included in a software kernel of the computing device” (A person can mentally evaluate the generated list and make a judgement to include the specified details (MPEP 2106).)
Since the claim does not recite additional elements that either integrate the judicial exception into a practical application, nor provide significantly more than the judicial exception, the claim is not patent eligible.
Regarding claim 10, it is dependent upon claim 9, and thereby incorporates the limitations of, and corresponding analysis applied to claim 9. Further, claim 10 recites the following additional abstract idea:
“wherein the single vertex corresponds to a kernel operation of the set of kernel operations” (A person can mentally evaluate the consolidated vertex and make a judgement to ensure it corresponds to a specific operation (MPEP 2106).)
Since the claim does not recite additional elements that either integrate the judicial exception into a practical application, nor provide significantly more than the judicial exception, the claim is not patent eligible.
Regarding claim 11, it is dependent upon claim 10, and thereby incorporates the limitations of, and corresponding analysis applied to claim 10. Further, claim 11 recites the following additional abstract idea:
“wherein the set of vertices represent operators and the set of directed edges represent a computing order corresponding to the operators and data dependency between vertices of the set of vertices.” (A person can mentally evaluate the set of vertices and directed edges and make a judgement to ensure they represent corresponding operators and data dependencies (MPEP 2106).)
Since the claim does not recite additional elements that either integrate the judicial exception into a practical application, nor provide significantly more than the judicial exception, the claim is not patent eligible.
Regarding claim 12, it is dependent upon claim 11, and thereby incorporates the limitations of, and corresponding analysis applied to claim 11. Further, claim 12 recites the following additional abstract idea:
“wherein the weight object indicates a set of weights assigned to the vertices of the set of vertices” (A person can mentally evaluate the generated list and make a judgement to include the specified details (MPEP 2106).)
Since the claim does not recite additional elements that either integrate the judicial exception into a practical application, nor provide significantly more than the judicial exception, the claim is not patent eligible.
Regarding claim 13, it is dependent upon claim 8, and thereby incorporates the limitations of, and corresponding analysis applied to claim 8. Further, claim 13 recites the following additional abstract idea:
“wherein optimizing the computing graph to generate the optimized computing graph further comprises connecting a first subset of directed edges of the set of directed edges from a first subset of vertices of the set of vertices to the single vertex and connecting a second subset of directed edges of the set of directed edges from the single vertex to a second subset of vertices of the set of vertices, where the first subset of directed edges associated with the subset of vertices correspond to inputs and the second subset of directed edges correspond to outputs” (A person can mentally evaluate the graph and make a judgement to connect edges and vertices based on corresponding inputs and outputs of the neural network (MPEP 2106).)
Since the claim does not recite additional elements that either integrate the judicial exception into a practical application, nor provide significantly more than the judicial exception, the claim is not patent eligible.
Regarding claim 14, it is dependent upon claim 8, and thereby incorporates the limitations of, and corresponding analysis applied to claim 8. Further, claim 14 recites “wherein the machine learning model is a neural network” (In step 2a, prong 2, this recites generally linking the use of the judicial exception to a particular technological environment or field of use (MPEP 2106.05(h).) In step 2B, generally linking the use of the judicial exception to a particular technological environment or field of use is not indicative of significantly more.)
Since the claim does not recite additional elements that either integrate the judicial exception into a practical application, nor provide significantly more than the judicial exception, the claim is not patent eligible.
Regarding claim 15, it is dependent upon claim 8, and thereby incorporates the limitations of, and corresponding analysis applied to claim 8. Further, claim 15 recites “wherein providing the net-list object and the weight object to the computing device further comprises providing the net-list object and weight object to an inference framework executed by the computing device” (In step 2A, prong 2, this recites insignificant extra-solution activity (mere data output) to the judicial exception (MPEP 2106.05(g).) In step 2B, the courts have found steps that present output of data to be a well-understood, routine, and conventional activity, which is not indicative of significantly more (Presenting offers and gathering statistics, OIP Techs., 788 F.3d at 1362-63, 115 USPQ2d at 1092-93).)
Regarding claim 16, in Step 1 of the 101-analysis set forth in MPEP 2106, the claim recites “A system comprising: a memory component; and a processing device coupled to the memory component”. A system of the described configuration is within one of the four statutory categories of invention.
Further, claim 16 comprises similar limitations as claim 1 and is rejected under the same rationale, with the following additional limitation: “A system comprising: a memory component; and a processing device coupled to the memory component, the processing device to perform operations comprising…” (In step2A, prong 2, this recites using a computer as a tool to perform an abstract idea (MPEP 2106.05(f).) In step 2B, using a computer as a tool to perform an abstract idea is not indicative of significantly more.)
Since the claim does not recite additional elements that either integrate the judicial exception into a practical application, nor provide significantly more than the judicial exception, the claim is not patent eligible.
Regarding claim 17, it is dependent upon claim 16, and thereby incorporates the limitations of, and corresponding analysis applied to claim 16. Further, claim 17 recites “wherein sub-structures of the set of sub-structures define operations of a software kernel executed by the computing device” (In step 2a, prong 2, this recites generally linking the use of the judicial exception to a particular technological environment or field of use (MPEP 2106.05(h).) In step 2B, generally linking the use of the judicial exception to a particular technological environment or field of use is not indicative of significantly more.)
Since the claim does not recite additional elements that either integrate the judicial exception into a practical application, nor provide significantly more than the judicial exception, the claim is not patent eligible.
Regarding claims 18-20, they are dependent upon claim 16, and thereby incorporates the limitations of, and corresponding analysis applied to claim 16. Further, claims 18-20 recite similar additional limitations as claims 7, 14, & 13, respectively, and are rejected under the same rationale.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1-8, 13-16, & 18-20 are rejected under 35 U.S.C. 103 as being unpatentable over Sui, L. et al. US PG Pub No: US 2019/0303762 A1, published 3 October 2019 (hereafter, SUI), and further in view of Yeh, J. et al. “OpenVINO.” Available on 14 December 2021 (hereafter, YEH)
Regarding claim 1, SUI teaches “A method comprising: generating a computing graph based on a neural network, the computing graph including a set of vertices representing operators and a set of directed edges representing a computing order and data dependency”:
([0103] “In step S1510, the topology of the neural network is analyzed to obtain a neural network computational graph…”) Thus, a computing graph based on a neural network is generated at step S1510. Further, node/operator representation, graph/edges/topology are inherently part of the “computational graph” analysis, as defined at https://www.sciencedirect.com/topics/computer-science/computation-graph ([1. Introduction to Computation Graphs] “A computation graph is a directed graph in which nodes correspond to mathematical operations (operators), variables, or equations, and edges represent the flow of information or data dependencies between these entities.”) Further, the figures 4-7 show some examples of this, such as figure 4 which shows nodes/vertices like “Conv 1x1” & “Conv 3x3” which represent operators, and theyre connected by edges that show computing order and data dependencies.
Further, SUI teaches “detecting a sub-graph within the computing graph matches at least one defined sub-structure”:
([0013] “…setting a subgraph template according to which we can perform layer fusion…”) Detecting a sub-graph within the computing graph
And further:
([0012] “…searching a topology conforming the preset rules in the neural network computational graph and reconstructing the computational graph.”) This is equivalent to detecting sub-graphs matching defined structures (via template/rule matching)
Further, SUI teaches “combing a subset of vertices of the set of vertices corresponding to the sub-graph into a vertex by at least connecting a subset of directed edges of the set of directed edges to the vertex to generate an optimized graph, where the subset of directed edges correspond to inputs and outputs associated with the subset of vertices”:
([Claim 1] “…fusing at least two adjacent layers (layer fusion) in the computational graph according to the selected layer objects…”) This citation simply shows that the patent is entirely about fusing layers/operations in the computational graph. Fusing layers combines vertices and their edges into a single higher-level node, as can be seen in FIGs. 4-7, which show the transition from “naive” graph to optimized (fused) graph where separate nodes/edges become merged. For example, figure 4 shows (Conv 1x1) and (Conv 3x3) being combined into a single node, in addition to the edges from “input” to them, being combined into a single edge.
SUI fails to explicitly teach “…extracting a set of parameters from the optimized graph; generating a net-list object and a weight object based on the optimized graph and the set of parameters; and providing the net-list object and the weight object to an inference framework to enable the inference framework to perform inferencing.” However, analogous art, YEH, does teach “extracting a set of parameters from the optimized graph”:
([Slide 20, Model Optimization] “Model Optimizer loads a model into memory, reads it, builds the internal representation of the model (a graph), optimizes it, and produces the Intermediate Representation”) The slide further explains that the optimization includes fusion and that a .xml file is produced which describes the network topology, explicitly including the parameters. Loading a model into memory, reading it, building an internal representation of it, optimizing it, and producing a new representation including a file which explicitly contains the parameters is a step-by-step description of a process in which parameters are extracted from an optimized graph.
Further, YEH teaches “generating a net-list object and a weight object based on the optimized graph and the set of parameters”:
([Slide 20, Model Optimization] The .xml file describes the network topology including dimensions, connectivity, and parameters, making it equivalate to a net-list object. The .bin file that is also generated describes the weights and biases binary data, making it equivalate to the “weight object”)
Further, YEH teaches “providing the net-list object and the weight object to an inference framework to enable the inference framework to perform inferencing”:
([Slide 20, Model Optimization] “The slide explicitly shows the following figure:
PNG
media_image1.png
103
440
media_image1.png
Greyscale
which shows at the end, the intermediate representation (including the net-list object and weight object as cited above, are provided for inferencing, as it explicitly states “Read, Load, Infer”)”)
It would be obvious to one of ordinary skill in the art, prior to the effective filing date of the claimed invention, to combine the base reference of SUI with the teachings of YEH because both references involve the use of optimized graphs in relation to neural networks.
One of ordinary skill in the art would be motivated to do so because, as pointed out by YEH on slide 11, “The usage of OpenVINO for computer vision related solution is almost unlimited, such as smart manufacturing, healthcare, retail, surveillance.., and so on. You can use it to improve application performance as well as accelerate deployment time.”
Regarding claim 2, SUI in view of YEH teaches the limitations of claim 1. Further, SUI teaches “wherein the sub-graph includes an isomorphic sub-graph”:
([Abstract] “…The method to optimize a computational graph of the present invention can be automatically carried out based on rules or through isomorphic subgraph matching…”)
Regarding claim 3, SUI in view of YEH teaches the limitations of claim 2. Further, SUI teaches “wherein detecting the sub-graph within the computing graph further comprises using a graph isomorphism algorithm to determine the sub-graph is equivalent to the defined sub-structure”:
([0123] “…high efficient automatic isomorphic subgraph matching can be realized, and the possibilities of all potential layer fusions are searched in a self-adaptive mode. The strategy is open, through a performance simulator in cooperation with instructions, an optimal solution of layer fusion can be automatically found by just adding a template of a subgraph”) This citation explicitly describes using isomorphic subgraph matching to determine the sub-graph is equivalent by searching all potential layer fusions.
Regarding claim 4, SUI in view of YEH teaches the limitations of claim 1. Further, SUI teaches “herein the method further comprises merging at least two parameters of the set of parameters”:
([0086] “FIG. 10 shows that a BatchNorm (batch normalization) layer and a Scale layer is fused... In the calculation process, the BatchNorm layer and the Scale layer can directly merge operations and parameters into a previous CONV layer.”)
Regarding claim 5, SUI in view of YEH teaches the limitations of claim 4. Further, SUI teaches “wherein merging the at least two parameters further comprises merging a subset of parameters of the set of parameters into a weight value and a bias value, where the subset of parameters correspond to a batch normalization operation”:
([0086] “FIG. 10 shows that a BatchNorm (batch normalization) layer and a Scale layer is fused... In the calculation process, the BatchNorm layer and the Scale layer can directly merge operations and parameters into a previous CONV layer.”) Here, SUI explicitly describes using batch normalization to merge parameters into preceding layers, which necessarily results in new weights and biases. Batch normalization has parameters including scale, shift, running mean, and running variance. When this is folded into a preceding layer, the original layer’s weights are rescaled and a bias term is generated or modified.
Regarding claim 6, SUI in view of YEH teaches the limitations of claim 1. Further, SUI teaches “wherein the at least one defined sub-structure is stored in a macro library and indicates a set of operations of the neural network that can be performed by an operation of a kernel”:
([0013] “… a method to automatically optimize a neural network computational graph is provided, comprising: analyzing the topology of the neural network to obtain a neural network computational graph; setting a subgraph template according to which we can perform layer fusion, here, the layer fusion is at least partially used for reducing frequency of data exchange with the off-chip memory; according to the preset rules, acquiring at least one subgraph matching strategy for the computational graph; and based on the subgraph matching strategy, reconstructing the computational graph to form a computational graph through layer fusion.”) A set of predefined templates/rules used by the optimizer/compiler (the DPU of [0045-0046]) to determine which subgraphs can be replaced by a fused implementation is a macro library (a stored collection of patterns, each pattern corresponding to a single executable kernel/instruction.) Further, the operations are performed by a “convolution kernel” as defined at [0056-0062].)
Regarding claim 7, SUI in view of YEH teaches the limitations of claim 1. Further, claim 7 recites “wherein the computing graphs further comprises a state space representation”. Under BRI, the computational graphs of SUI and YEH, cited above, inherently represent the “state space” of a neural network model, which can be interpreted as a “state space representation”.
Regarding claim 8, SUI teaches “A non-transitory computer-readable medium storing executable instructions embodied thereon, which, when executed by a processing device, cause the processing device to perform operations”:
([0131] “…the present invention may also be implemented as a non-transitory machine-readable storage medium (or a computer readable storage medium, or a machine-readable storage medium), on which an executable code (or a computer program, or a computer instruction code) is stored, when the executable code ( or a computer program, or a computer instruction code) is executed by a processor of an electronic equipment ( or a computational equipment, a server, etc.), the processor executes the steps of the above method according to the present invention”)
Further, claim 8 comprises similar additional limitations as claim 1, and is rejected under the same rationale.
Regarding claim 13, SUI in view of YEH teaches the limitations of claim 8. Further, SUI teaches “wherein optimizing the computing graph to generate the optimized computing graph further comprises connecting a first subset of directed edges of the set of directed edges from a first subset of vertices of the set of vertices to the single vertex and connecting a second subset of directed edges of the set of directed edges from the single vertex to a second subset of vertices of the set of vertices, where the first subset of directed edges associated with the subset of vertices correspond to inputs and the second subset of directed edges correspond to outputs”:
([0013] a method to automatically optimize a neural network computational graph is provided, comprising: analyzing the topology of the neural network to obtain a neural network computational graph; setting a subgraph template according to which we can perform layer fusion, here, the layer fusion is at least partially used for reducing frequency of data exchange with the off-chip memory; according to the preset rules, acquiring at least one subgraph matching strategy for the computational graph; and based on the subgraph matching strategy, reconstructing the computational graph to form a computational graph through layer fusion.) A computational graph that undergoes layer fusion will result in inputs to the subgraph becoming inputs to the fused node, outputs of the subgraph becoming outputs of the fused node (as shown in figures 4-7; For example, in figure 5, the inputs to “Conv 1x1” and “Conv 3x3” become fused into the new node, and in Figure 6, you can see the outputs from CONV, ReLU, and POOL going to MEM are still output from the newly fused node into the newly fused MEM node), and edges between fused layers being removed (also shown in figs 4-7), which is exactly what this claim’s limitations describe happening. This is merely a description of the process of “layer fusion”.
Regarding claim 14, SUI in view of YEH teaches the limitations of claim 8. Further, SUI teaches “wherein the machine learning model is a neural network”:
([Abstract, Sentence 1] “The present invention discloses a method to optimize a neural network computational graph.”)
Regarding claim 15, SUI in view of YEH teaches the limitations of claim 8. Further, YEH teaches “wherein providing the net-list object and the weight object to the computing device further comprises providing the net-list object and weight object to an inference framework executed by the computing device”:
([Slide 29] This slide explicitly shows the “read, load, infer” step cited above on slide 20 being deployed to an “inference engine” described as a “common API that abstracts low-level programming for each hardware” before being sent to various hardware pieces such as GPU, CPU, and/or FGPA.)
Regarding claim 16, SUI teaches “A system comprising: a memory component; and a processing device coupled to the memory component, the processing device to perform operations”:
([0124] “In the neural network computational implementation system of the present invention, part or all of the functions of a computational executing device used to execute neural network calculation can be realized by a digital circuit. In an embodiment, the computational system of the present invention may comprise a general processor, a storage, and a system on a chip (SOC) implemented with digital circuit. FIG. 16 shows an example which can be used for realizing SOC of an inference system of the present invention.”)
Further, claim 16 comprises similar additional limitations as claim 1, and is rejected under the same rationale.
Regarding claims 18-20, SUI in view of YEH teaches the limitations of claim 16. Further, claims 18-20 comprise similar additional limitations as claims 7, 14, & 13 respectively, and are rejected under the same rationale.
Claims 9-12, & 17 are rejected under 35 U.S.C. 103 as being unpatentable over SUI in view of YEH, as applied to claims above, and further in view of Kim, E. et al. “GPUPluginDebugUtils” Available on 28 September 2022 (hereafter, KIM)
Regarding claim 9, SUI in view of YEH teaches the limitations of claim 8. Further, YEH teaches “wherein the net-list indicates connectivity between layers of the machine learning model”:
([Slide 20, Model Optimization] “The .xml file describes the network topology including dimensions, connectivity, and parameters.”)
SUI in view of YEH fails to explicitly teach “wherein the net-list indicates… a set of kernel operations associated with the layers, where the set of kernel operations are included in a software kernel of the computing device.” However, analogous art, KIM, does teach this:
([Dump execution graph] “The execution graph… is a device specific graph after all transformations applied by the plugin…”) KIM describes an optional plugin for OpenVINO which can produce an “execution graph” to an .xml file (a different type of net-list object)
And further:
([Dump execution graph, paragraph 2] “The capability to retrieve execution graph and store it on the disk is integrated into benchmark_app. The execution graph can be simply dumped by setting additional parameter exec_graph_path exec_graph.xml for benchmark_app. Output xml file has a format similar to usual IR (Here, we explicitly see it is similar to standard IR as cited above), but contains execution nodes with some runtime info such as…
Mapping between nodes in final device specific graph and original input graph operations
…
Primitive type”)
In the above citation, we explicitly see that the execution graph .xml file (the net-list object) contains runtime info including a mapping between nodes in final device specific graph and original graph operations and “primitive type”. This primitive type field correlates to a kernel operation associated with the layer.
And further:
The above citation goes on to show code examples that explicitly show the layers in addition to more following text including “…for each node you can check that… The node used expected kernel for execution…”
It would be obvious to one of ordinary skill in the art, prior to the effective filing date of the claimed invention, to combine the base reference of SUI in view of YEH with the teachings of KIM because KIM is an optional addon built explicitly for the invention disclosed in YEH.
One of ordinary skill in the art would be motivated to do so because, as pointed out in KIM in the “Dump execution graph” section, “…It's a very useful feature for performance analysis and it allows to find a source of performance regressions quickly.”
Regarding claim 10, SUI in view of YEH & KIM teaches the limitations of claim 9. Further, KIM teaches “wherein the single vertex corresponds to a kernel operation of the set of kernel operations”:
([KIM, Dump execution graph] The cited section, as cited above, is for an optional plugin (dubbed GPU plugin debug utils) for the previously cited method of YEH, (dubbed OpenVINO), which does consolidate multiple graph nodes into a single vertex, by employing methods such as layer fusion [YEH, slide 20], and by referencing this section of KIM, we see “…for each node you can check that… The node used expected kernel for execution…” )
Regarding claim 11, SUI in view of YEH & KIM teaches the limitations of claim 10. Further, SUI teaches “wherein the set of vertices represent operators and the set of directed edges represent a computing order corresponding to the operators and data dependency between vertices of the set of vertices”:
([0013] a method to automatically optimize a neural network computational graph is provided, comprising: analyzing the topology of the neural network to obtain a neural network computational graph; setting a subgraph template according to which we can perform layer fusion, here, the layer fusion is at least partially used for reducing frequency of data exchange with the off-chip memory; according to the preset rules, acquiring at least one subgraph matching strategy for the computational graph; and based on the subgraph matching strategy, reconstructing the computational graph to form a computational graph through layer fusion.) A “computational graph” necessarily includes “operator execution order” and “data dependencies”. This claim is merely describing the necessary structure for a computational graph in order to perform “layer fusion”.
Regarding claim 12, SUI in view of YEH & KIM teaches the limitations of claim 11. Further, YEH teaches “wherein the weight object indicates a set of weights assigned to the vertices of the set of vertices”:
([Slide 20] the slide explicitly shows that the .bin file (the weight object) “describes the weights and biases binary data”)
Regarding claim 17, SUI in view of YEH teaches the limitations of claim 16. Further, SUI in view of YEH fails to explicitly teach “wherein sub-structures of the set of sub-structures define operations of a software kernel executed by the computing device.” However, analogous art, KIM, does teach this:
([Dump execution graph] “The execution graph… is a device specific graph after all transformations applied by the plugin…”) KIM describes an optional plugin for OpenVINO which can produce an “execution graph” to an .xml file (a different type of net-list object)
And further:
([Dump execution graph, paragraph 2] “The capability to retrieve execution graph and store it on the disk is integrated into benchmark_app. The execution graph can be simply dumped by setting additional parameter exec_graph_path exec_graph.xml for benchmark_app. Output xml file has a format similar to usual IR (Here, we explicitly see it is similar to standard IR as cited above), but contains execution nodes with some runtime info such as…
Mapping between nodes in final device specific graph and original input graph operations
…
Primitive type”)
In the above citation, we explicitly see that the execution graph .xml file (the sub-structures) contains runtime info including a mapping between nodes in final device specific graph and original graph operations and “primitive type”. This primitive type field correlates to a kernel operation associated with the layer.
And further:
The above citation goes on to show code examples that explicitly show the layers in addition to more following text including “…for each node you can check that… The node used expected kernel for execution…”
It would be obvious to one of ordinary skill in the art, prior to the effective filing date of the claimed invention, to combine the base reference of SUI in view of YEH with the teachings of KIM because KIM is an optional addon built explicitly for the invention disclosed in YEH.
One of ordinary skill in the art would be motivated to do so because, as pointed out in KIM in the “Dump execution graph” section, “…It's a very useful feature for performance analysis and it allows to find a source of performance regressions quickly.”
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MATTHEW LEE LEWIS whose telephone number is (571)272-1906. The examiner can normally be reached Monday: 12:00PM - 4:00PM and Tuesday - Friday: 12:00PM - 9PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Tamara Kyle can be reached at (571)272-4241. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Matthew Lee Lewis/Examiner, Art Unit 2144
/TAMARA T KYLE/Supervisory Patent Examiner, Art Unit 2144