Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Amendments
This action is in response to amendments filed February 27th, 2026, in which Claims 1, 10, and 20 have been amended. Claim 18 has been cancelled. The amendments have been entered, and Claims 1-17, 19, and 20 are currently pending
Claim Objections
Claim 1 is objected to because of the following informalities: The final line of the claim recites a graph editing technique of computational graph. This appears to be a typographical error, and will be interpreted as if it had read a graph editing technique of a computational graph. Appropriate correction is required.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claims 1-17, 19, and 20 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Abadi et al., “TensorFlow: A System of Large Scale Machine Learning.”
Regarding Claim 1, Abadi teaches a model processing method (Abadi, title, “A system for large-scale machine learning”) implemented by a distributed training framework (Abadi, pg. 265, 2nd column, 1st paragraph, “TensorFlow supports both large-scale training and inference: it efficiently uses hundreds of powerful GPU-enabled servers for fast training”) comprising: acquiring a first computational graph corresponding to a model to be trained (Abadi, pg. 267, Fig. 1, the user provides code to TensorFlow’s API that describes a computational graph corresponding to a model to be trained, see Fig. 1, “Construct a graph” & “Train iteratively for NUM_STEPS”, also see pg. 269, Fig. 2) and a parallelization strategy for the model to be trained (Abadi, pg. 265, 2nd column, 2nd paragraph, “allows programmers to experiment with different parallelization schemes” denotes that the user can select or indicate the parallelization strategy/scheme”), wherein the parallelization strategy for the model to be trained comprises at least one of the following: pipeline parallelism (Abadi, pg. 269, “a training pipeline” & pg. 270, 2nd column, 2nd-to-last paragraph, “Fig. 2 shows a typical training application, with multiple subgraphs that execute concurrently”); model parallelism (Abadi, pg. 278, Fig. 9, “parallelizing the softmax computation” within a single model & pg. 274, Fig. 5 where each worker trains their own model), data parallelism (Abadi, pg. 270, 2nd-to-last paragraph, “to implement data-parallel training”), and operator split (Abadi, pg. 267, 2nd paragraph, “a Dist-Belief model comprises relatively few complex ‘layers’ whereas the corresponding TensorFlow model represents individual mathematical operators as nodes in the data flow graph” denotes splitting complex operators into simple ones); adding, according to the parallelization strategy for the model to be trained, parallelization information into the first computational graph, to obtain a second computational graph (Abadi, pg. 271, 1st column, 1st paragraph, “By default, concurrent executions of a TensorFlow subgraph run asynchronously with respect to one another … TensorFlow also provides the primitives needed to synchronize workers during training” & final paragraph, “expert users and optimize performance by manually placing operations”); determining, according to the second computational graph and a computing resource, a distributed computational graph (Abadi, pg. 217, 1st column, 2nd -4th paragraphs, “Dataflow simplifies distributed execution … The TensorFlow runtime places operations on devices, subject to implicit or explicit constraints in the graph”); and training, according to the distributed computational graph, the model to be trained (Abadi, pg. 268, 1st column, 1st paragraph, “executes an optimized version of the program on the set of available devices” which is training, see pg. 269, Fig. 2, “a training pipeline”), wherein the determining, according to the second computational graph and the computing resource, the distributed computational graph comprises: dividing the computing resources to obtain one or a plurality of virtual devices, wherein the virtual device comprises one or a plurality of physical devices, wherein the virtual device comprises one or a plurality of physical devices (Abadi, pg. 277, 1st column, 2nd paragraph, “we use Google Compute Engine virtual machines running on Intel Xeon e5 servers … configured with 8 vGPUs … and one GPU per VM [and] 7 PS tasks running on separate VMs with no GPU”), and obtaining the distributed computational graph based on the parallelization information, the second computational graph, and the physical devices (Abadi, pg. 271, 1st column, final paragraph, “expert users and optimize performance by manually placing operations”); wherein the distributed training framework comprises a model and parallel intermediate representation layer, a parallel representation layer in the model and parallel intermediate representation layer is configured to analyze the parallelization strategy for the model to be trained to obtain a corresponding abstraction (Abadi, pg. 267, 2nd column, final paragraph, “A typical TensorFlow application has two distinct phases: the first phase defines the program as a symbolic dataflow graph with placeholders for the input data and variables the represent the state … optimized version” denotes an the model with a representation layer obtaining an abstraction while pg. 271, 1st column, last two paragraphs, “The TensorFlow runtime places operations on devices, subject to implicit or explicit constraints on the graph. The placement algorithm computes a feasible set of devices for each operation” including ”the user may specify partial device preferences and the runtime will respect these constraints” where user constraints are parallelization strategy, denotes analyzing the parallelization strategy to obtain the abstraction) wherein the abstraction is configured to unify and express different parallelization strategies (all the parallelization strategies noted, i.e. pg. 269, 2nd column, final paragraph, “our users have been able to experiment with different … parallelization strategies” will be unified and expressed/used together as specified in the dataflow abstraction) and the distribution training framework implements the parallelization strategy for the model to be trained ((Abadi, pg. 267, 2nd column, final paragraph, “A typical TensorFlow application has two distinct phases: … the second phase executes an optimized version”) further based on a graph editing technique of a computational graph (Abadi, pg. 274, 2nd column, 2nd paragraph, “The TensorFlow graph enables uses to change how parameters are read and written when training a model, and we implement three alternatives” where each of the three alternatives correspond to a different computational graph displayed in Figure 5, thus changing strategies is a graph editing technique of the computational graphs of Figure 5).
Regarding Claim 2, Abadi teaches the method according to Claim 1 (and thus the rejection of Claim 1 is incorporated). Abadi further teaches wherein the adding, according to the parallelization strategy for the model to be trained, the parallelization information into the first computational graph, to obtain the second computational graph, comprises: dividing, according to the parallelization strategy for the model to be trained, the first computational graph into a plurality of subgraphs; and adding, according to the parallelization strategy for the model to be trained, the parallelization information into each first subgraph in the plurality of first subgraphs, to obtain the second computational graph (Abadi, pg. 269, Fig. 2, “a training pipeline, containing subgraphs for …” & pg. 270, 2nd column, “Partial and concurrent execution” of sub-graphs & pg. 271, 1st column, final paragraph, “expert users and optimize performance by manually placing operations” is adding parallelization information).
Regarding Claim 3, Abadi teaches the method according to Claim 2 (and thus the rejection of Claim 2 is incorporated). Abadi further teaches wherein the parallelization information comprises the parallelization information between different first subgraphs (Abadi, pg. 271, 1st column, 2nd -4th paragraphs, “Dataflow simplifies distributed execution … The TensorFlow runtime places operations on devices, subject to implicit or explicit constraints in the graph” such as Fig. 2, the order requirements are between different subgraphs, see pg. 271, 2nd column, “blocks until the value … is available locally” & pg. 274, Fig. 5 ) and the parallelization information in each first subgraph (Abadi, pg. 271, 2nd column, 2nd paragraph, “per device sub-graphs“ with pg. 275, 1st column, last paragraph, “The dataflow executor dispatches kernels to local devices and runs kernels in parallel when possible, for example by using multiple CPU cores or GPU streams”).
Regarding Claim 4, Abadi teaches the method according to Claim 3 (and thus the rejection of Claim 3 is incorporated). Abadi further teaches a parallelization strategy adopted between different first subgraphs (Abadi, Abadi, pg. 269, Fig. 2 & pg. 274, Fig. 5, where each device/worker is assigned its own subgraph).
Regarding Claim 5, Abadi teaches the method according to Claim 4 (and thus the rejection of Claim 4 is incorporated). Abadi further teaches wherein the parallelization information between different first subgraphs comprises: parameter information of the parallelization strategy adopted between the different first subgraphs (Abadi, pg. 271, 1st column, 1st paragraph, “By default, concurrent executions of a TensorFlow subgraph run asynchronously with respect to one another … TensorFlow also provides the primitives needed to synchronize workers during training” where indicating synchronization is a parallelization parameter).
Regarding Claim 6, Abadi teaches the method according to Claim 4 (and thus the rejection of Claim 4 is incorporated). Abadi further teaches wherein the parallelization strategy adopted between different first subgraphs comprises: the pipeline parallelism or the model parallelism (Abadi, pg. 269, Fig. 2 shows pipeline parallelism & pg. 274, Fig. 5 shows model parallelism).
Regarding Claim 7, Abadi teaches the method according to Claim 3 (and thus the rejection of Claim 3 is incorporated). Abadi further teaches wherein the parallelization information in each first subgraph comprises: a parallelization strategy in each first subgraph (Abadi, pg. 271, 2nd column, 2nd paragraph, “per device sub-graphs“ with pg. 275, 1st column, last paragraph, “The dataflow executor dispatches kernels to local devices and runs kernels in parallel when possible, for example by using multiple CPU cores or GPU streams”).
Regarding Claim 8, Abadi teaches the method according to Claim 7 (and thus the rejection of Claim 7 is incorporated). Abadi further teaches wherein the parallelization information in each first subgraph comprises: parameter information of the parallelization strategy in each first subgraph (Abadi, pg. 271, 2nd column, 2nd paragraph, “per device sub-graphs“ with pg. 275, 1st column, last paragraph, “The dataflow executor dispatches kernels to local devices and runs kernels in parallel when possible, for example by using multiple CPU cores or GPU streams” where kernels comprise parameters).
Regarding Claim 9, Abadi teaches the method according to Claim 7 (and thus the rejection of Claim 7 is incorporated). Abadi further teaches wherein the parallelization strategy in each first subgraph comprises: the data parallelism and or the operator split (Abadi, pg. 271, 2nd column, 2nd paragraph, “per device sub-graphs“ with pg. 275, 1st column, last paragraph, “The dataflow executor dispatches kernels to local devices and runs kernels in parallel when possible, for example by using multiple CPU cores or GPU streams” where different “GPU streams” denotes data parallelism).
Regarding Claim 10, Abadi teaches the method according to Claim 2 (and thus the rejection of Claim 2 is incorporated). Abadi further teaches converting, according to the parallelization information, the second computational graph into a third computational graph; and mapping the third computational graph to the physical device, to obtain the distributed computational graph (Abadi, pg. 273, Fig. 4 & 2nd column, 2nd paragraph, “Users writing a TensorFlow model do not typically construct graphs like Figure 4 manually. Instead, TensorFlow includes libraries that expose the abstraction of a sharded parameter, and build appropriate graphs of primitive operations based on the desired degree of distribution” & 1st column, 2nd-to-last paragraph, “too large even to store in RAM on a single host” & 2nd column, 1st paragraph “each of the shards”).
Regarding Claim 11, Abadi teaches the method according to Claim 10 (and thus the rejection of Claim 10 is incorporated). Abadi further teaches dividing, according to the parallelization information, the computing resource, to obtain one or the plurality of virtual devices (Abadi, pg. 277, 1st column, 2nd paragraph, “we use Google Compute Engine virtual machines running on Intel Xeon e5 servers, configured with 8 vCPUs … and one GPU per VM”).
Regarding Claim 12, Abadi teaches the method according to Claim 10 (and thus the rejection of Claim 10 is incorporated). Abadi further teaches converting, according to the parallelization information of each first subgraph in the plurality of first subgraphs, the first subgraph to a distributed second subgraph; and connecting, according to a connection relationship between the plurality of first subgraphs, the distributed second subgraph corresponding to each first subgraph, to obtain the third computational graph (Abadi, pg. 271, 2nd column, 2nd paragraph, “Once the operations in a graph has been placed, and the partial subgraph has been computed for a step, TensorFlow partitions the operations into per-device subgraphs. A per-device subgraph for device d contains all of the operations that were assigned to d, with additional Send and Recv operations that replace edges across device boundaries”).
Regarding Claim 13, Abadi teaches the method according to Claim 12 (and thus the rejection of Claim 12 is incorporated). Abadi further teaches mapping each second subgraph in the third computational device to the physical device (Abadi, pg. 271, 2nd column, 2nd paragraph, “Once the operations in a graph has been placed, and the partial subgraph has been computed for a step, TensorFlow partitions the operations into per-device subgraphs. A per-device subgraph for device d contains all of the operations that were assigned to d, with additional Send and Recv operations that replace edges across device boundaries”).
Regarding Claim 14, Abadi teaches the method according to Claim 13 (and thus the rejection of Claim 13 is incorporated). Abadi further teaches mapping each first subgraph to one virtual device; and mapping each second subgraph corresponding to the first subgraph to a physical device included in the virtual device corresponding to the first subgraph (Abadi, pg. 277, 1st column, 2nd paragraph, “we use Google Compute Engine virtual machines running on Intel Xeon e5 servers, configured with 8 vCPUs … and one GPU per VM” & pg. 271, 2nd column, 2nd paragraph, “Once the operations in a graph has been placed, and the partial subgraph has been computed for a step, TensorFlow partitions the operations into per-device subgraphs. A per-device subgraph for device d contains all of the operations that were assigned to d, with additional Send and Recv operations that replace edges across device boundaries”).
Regarding Claim 15, Abadi teaches the method according to Claim 1 (and thus the rejection of Claim 1 is incorporated). Abadi further teaches determining, according to the first computational graph corresponding to the model to be trained, the parallelization strategy for the model to be trained (Abadi, pg. 268, 1st column, 1st paragraph, “TensorFlow can optimize the execution stage by using global information about the computation” & pg. 271, 1st column, last paragraph, “TensorFlow thus permits great flexibility in how operations in the dataflow graph are mapped to devices … simple heuristics yield adequate performance for novice users”).
Regarding Claim 16, Abadi teaches the method according to Claim 1 (and thus the rejection of Claim 1 is incorporated). Abadi further teaches acquiring a user-selected parallelization strategy for the model to be trained (Abadi, pg. 268, 1st column, 1st paragraph, “TensorFlow can optimize the execution stage by using global information about the computation” & pg. 271, 1st column, last paragraph, “TensorFlow thus permits great flexibility in how operations in the dataflow graph are mapped to devices …expert users can optimize performance by manually placing operations to balance the computation, memory, and network requirements across multiple tasks and multiple devices within those tasks”).
Regarding Claim 17, Abadi teaches the method according to Claim 1 (and thus the rejection of Claim 1 is incorporated). Abadi further teaches converting the distributed computational graph to a distributed computational graph which is recognizable by a training engine (Abadi, pg. 270, 1st column, 4th paragraph, teaches that their graph models are compiled, “at compile-time” where compiling is converting to a graph which is recognizable by a training engine) and inputting the distributed computational graph which is recognizable by the training engine into the training engine, and training, by the training engine, the model to be trained (Abadi, pg. 268, 1st column, 1st paragraph, “executes an optimized version of the program on the set of available devices” which is training, see pg. 269, Fig. 2, “a training pipeline” also see pg. 271, 1st column, 3rd paragraph, “TensorFlow allows multiple kernels to be registered for a single operation, which specialized implementations for a particular device or data type”).
Claim 19 recites a device to perform the method of Claim 1. As Abadi implements their method on a computer (TensorFlow is a software application with an API), Claim 19 is rejected for reasons set forth in the rejection of Claim 1. Similarly, Claim 20 recites a computer-readable storage medium which stores a computer program to implement the method of Claim 1, which is also inherent in the computer, and thus the claim is also rejected for reasons set forth in the rejection of Claim 1.
Response to Arguments
Applicant’s arguments filed February 25th, 2026 have been fully considered, but are not fully persuasive.
Applicant’s amendments have overcome the 35 U.S.C. 101 and 35 U.S.C. 112(b) rejections of the previous office action.
Applicant’s arguments regarding the 35 U.S.C. 102(a)(1) rejection of the previous office action have been fully considered, but are not persuasive, as Abadi teaches the amended limitations on which the applicant’s arguments rely.
Applicant argues that the disclosed invention “integrates various multiple parallelization strategies into one system”. However, Abadi also explicitly contains this features (pg. 269, 2nd column, final paragraph, “our users have been able to experiment with different … parallelization strategies”). Applicant argues that their recited features of a model comprising a layer that can analyze the strategies to unify and express different parallelization strategies. However, Abadi also contains these features – TensorFlow is a model, which the capability (i.e. a layer in the model) to express different parallelization strategies, and as many different parallelization strategies as a user applies in a model (i.e. unify the different required device placements specified by the user).
Applicant argues that “as recited in claim 1, the distributed training framework is provided with a user interface, and a user may use the user interface to select a parallelization strategy for the model to be trained”. However, no user interface is recited in Claim 1. Even so, Abadi includes a user interface to allow the users to select their parallelization strategy (Abadi, pg. 265, 2nd column, 2nd paragraph, “allows programmers to experiment with different parallelization schemes” denotes that the user can select or indicate the parallelization strategy/scheme”) – allowing the users program TensorFlow is a user interface (see Abadi, pg. 267, Fig. 1 & 2nd column, 1st paragraph, “A high level scripting interface” & pg. 271, 1st column, “explicit constraints” on operation placement on devices, “the user may specify partial device preferences … and the runtime will respect these constraints”). Thus, despite the applicant’s assertion, the parallelization strategies of TensorFlow “allow the user to flexibly make their own selection”.
Applicant next argues that “Abadi does not involve dividing the computing resource to obtain one or a plurality of user devices, in which the virtual devices comprises one or a plurality of physical devices.” The plain text of Abadi clearly contradicts the applicant’s assertion. (Abadi, pg. 277, 1st column, 2nd paragraph, “we use Google Compute Engine virtual machines running on Intel Xeon e5 servers … configured with 8 vGPUs … and one GPU per VM [and] 7 PS tasks running on separate VMs with no GPU”). Applicant’s assertion that “Abadi does not involve how to achieve a unifying training framework that can support various hybrid parallelization strategies in any combination” is also contradicted by the explicit text of Abadi (pg. 265, 2nd column, 2nd-to-last paragraph, “By unifying the computation and state management in a single programming model, TensorFlow allows programmers to experiment with different parallelization schemes” & pg. 268, 2nd column, last paragraph, “our uses have been able to experiment with different optimization algorithms, consistency schemes, and parallelization strategies” & pg. 271, 1st column, last paragraph, “expert users and optimize performance by manually placing operations” & pg. 274, Fig. 5 further demonstrates “various parallelization strategies” that TensorFlow has unified into their model).
Conclusion
Applicant’s amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to BRIAN M SMITH whose telephone number is (469)295-9104. The examiner can normally be reached Monday - Friday, 8:00am - 4pm Pacific.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kakali Chaki can be reached at (571) 272-3719. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/BRIAN M SMITH/Primary Examiner, Art Unit 2122
Prosecution Insights