DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-11 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more. Claim 1 recites, “dividing nodes in the topological structure diagram” and “performing space distribution on the target device block…”. The limitations of “dividing” and “performing” as drafted are functions that, under their broadest reasonable interpretation, recite the abstract idea of a mental process. The limitations encompass a human mind carrying out the function through observation, evaluation, judgment and /or opinion, or even with the aid of pen and paper. Thus, this limitation recites and falls within the “Mental Processes” grouping of abstract ideas under Prong 1.
Under Prong 2, this judicial exception is not integrated into a practical application. The claim recites the following additional elements “obtaining a topological structure…” and “deploying the deep learning network to the heterogenous platform”. The additional elements of “obtaining” and “deploying” are insignificant pre and post solution activities. Accordingly, the additional elements do not integrate the recited judicial exception into a practical application and the claim is therefore directed to the judicial exception.
Under Step 2B, the claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception. Regarding the additional elements of “obtaining” and “deploying”, receiving or transmitting data over a network are well-understood, routine and conventional activities. See MPEP 2106.05(d). Accordingly, claim 1 is not patent eligible under 35 USC 101.
Claim 2, The steps of “dividing the nodes…”, “arranging the candidate device blocks…”, and “determining a input node and an output node…” are additional limitations of the abstract idea “Mental Process”. Nothing in the claimed limitations prevent this limitation form being performed in the mind.
Claim 3, limitation, “wherein the information of the node comprise a node type of the node…”. The additional elements of this limitation are neither a practical application under prong 2, nor an inventive concept under step 2B. Regarding the limitations of “determining…” and “dividing…”, these are additional limitations of the abstract idea “Mental Process”. Nothing in these claimed limitations preclude them from being performed in the mind.
Claim 4, the limitation of “dividing the i-th node…”, is an additional limitation of the abstract idea “Mental Process”. Nothing in the claimed limitation precludes it from being performed in the mind.
Claim 5, The additional elements of this limitation are neither a practical application under prong 2, nor an inventive concept under step 2B.
Claim 6, the limitation of “calculation a space size…” is an additional limitation of the abstract idea “Mental Process”. Nothing in the claimed limitation prevents it from being performed in the mind. Regarding the limitation, “obtaining a maximum storage space”, The courts have identified mere data gathering as a well-understood, routine and conventional activity. See MPEP 2106.05(d) and MPEP 2106.05(f). The additional elements of this limitation are neither a practical application under prong 2, nor an inventive concept under step 2B.
Claim 7, regarding the limitations of “sorting an output node…”, “searching a target node space…”, “determining whether there is the output node satisfying…”, “releasing the node space” and “determining the maximum storage space”. These limitations are additional limitations of the abstract idea “Mental Process”. Nothing in the claimed limitations prevent this limitation form being performed in the mind. Regarding the multiple limitation of “distributing”, receiving or transmitting data over a network are well-understood, routine and conventional activities. See MPEP 2106.05(d). The additional elements of this limitation are neither a practical application under prong 2, nor an inventive concept under step 2B.
Claim 8, the limitation “determining whether the deployment information…” is an additional limitation of the abstract idea “Mental Process”. Nothing in the claimed limitation prevents it from being performed in the mind. Regarding the limitations of “obtaining deployment information…” and “sending the deployment information…”, receiving or transmitting data over a network are well-understood, routine and conventional activities. See MPEP 2106.05(d).
Claim 9, Regarding the limitations of “sending the deployment information…”, receiving or transmitting data over a network are well-understood, routine and conventional activities. See MPEP 2106.05(d).
Claims 10-11, contain similar limitations to claim 1 and are therefore rejected for the same reasons.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1-2 and 8-11 are rejected under 35 U.S.C. 102a2 as being anticipated by Brady et al. (US 2019/0391796 A1).
As per claim 1, Brady et al. teaches the invention as claimed including, “A deployment method of a heterogeneous platform based on a TVM compiler comprising:
obtaining a topological structure diagram, a network parameter and function information of each function of a deep learning network generated by the TVM compiler;”
A compiler accesses a neural network model, together with information (e.g., target descriptor file) concerning the application and the target hardware and generate an improved intermediate representation (IR)(topological structure diagram) from which the binary will be generated. The intermediate representation can be sub-models that include an operator model, a data model, and a control model. The intermediate representation may also be provided with data (e.g., structural data) describing attributes of the target hardware device) (0051-0052).
“dividing nodes in the topological structure diagram into target device blocks corresponding to device types, based on information of the nodes in the topological structure diagram and the device types of the heterogeneous platform;”
performing space distribution on the target device block, to obtain space information of the target device block; and”
Compilation passes may be performed using the operator model and/or control model of the IR to determine which operations are to be performed by which compute resources and in what order. Additional passes may be performed using the data model of the IR to determine how best to allocate memory (space distribution) to enable fast and efficient use of the tensors to thereby optimize performance of the operations of the neural network by the particular target hardware (0080). The compiler is to allocate specific physical memory address to data structures (tensors) in the memory regions specified in the target descriptor file. These memory regions may be dependent on the resources of the target device. The specific region of the memory that a specific data structure is assigned to reside in its typically determined during compilation passes that determine the order of execution of the operations and/or map the execution of each operation to a particular compute resource (0091).
Additional compilation passes may be performed to allocate hardware barrier resources (divide) for the optimized compute graph. For instance, nodes in the transformed intermediate representation (e.g., the operation and/or control graph sub-models) may be traversed and opportunities may be identified for using hardware barrier resources (e.g., counters) on the target computing device (device type). For instance, the compiler may determine whether a given operation represented in the transformed graph model is to be synchronized with a barrier (dividing node/operation from others) (and represented by a corresponding barrier task object). For instance one or more rules, conditions or algorithms may be defined (e.g., by the target descriptor or compilation descriptor, or from other data or in logic of the compiler) to determine whether a barrier should be inserted into the graph. For instance, barriers may be inserted before each direct memory access and processor operation/task that has a data dependency (on another operation/task), such as when a task (e.g., mathematical or data movement operation) requires an output of a preceding operation/task before being able to successfully proceed. Barriers may be inserted before every task/operation that exceed the number of parallel barriers designated for use during the corresponding processing and implementation of the corresponding neural network (e.g., a group of eight hardware barriers (e.g., a subset of the overall barriers provided on the hardware) may be designated for a particular compute resource (target device type) or the aggregate collection of compute resources on the target device, etc). Barriers may also be inserted before operations that have a control dependency graph edge which forces serail operation, without data dependency (0102).
With barriers inserted into the graph (e.g., within the control model graph), graph theory-based analysis may be performed, among other optimization techniques, by the compiler, to identify opportunities to reduce the number of or otherwise optimize the barrier tasks. For instance, redundant barrier tasks may be combined (e.g., when two or more operation rely on the same preceding dependencies, they may share the same barrier (rather than each requiring their own distinct barrier)). In other instance changes may be made to the underlying control flow or data flow represented in the intermediate representation based on limited hardware barrier resources (e.g., to serialize operations when the number of parallel control flow paths outnumber the number of hardware barrier devices available on the target computing device) (0103).
“deploying the deep learning network to the heterogeneous platform, based on the topological structure diagram, the network parameter, the function information, the target device block and the space information of the target device block.”
A binary is generated capable of being executed by the target hardware to implement the neural network (0076). The intermediate representation is used by the compiler to generate the binary code to facilitate task scheduling using hardware barrier resources (0103). The compiler may utilize hardware barrier information for a target machine learning device and generate one or more hardware barrier tasks to generate a binary that utilize the hardware barrier resources to realize optimized scheduling using these resources. The barrier tasks may be generated in association with one or more compilation passes and inserted in a graph of the intermediate representation (0053). Also see figure 1.
As per claim 2, Brady et al. further teaches, “The deployment method as claimed in claim 1, wherein the step of obtaining the topological structure diagram, the network parameter and the function information of each function of the deep learning network generated by the TVM compiler, comprises:
dividing the nodes into candidate device blocks corresponding to each device type, to obtain target nodes comprised in each of the candidate device blocks, based on the information of the nodes and the device types, each device type comprising at least one candidate device block;”
Additional compilation passes may be performed to allocate hardware barrier resources for the optimized compute graph. For instance, nodes in the transformed intermediate representation (e.g., the operation and/or control graph sub-models) may be traversed and opportunities may be identified for using hardware barrier resources (e.g., counters) on the target computing device. For instance, the compiler may determine whether a given operation represented in the transformed graph model is to be synchronized with a barrier (dividing node/operation from others) (and represented by a corresponding barrier task object). For instance one or more rules, conditions or algorithms may be defined (e.g., by the target descriptor or compilation descriptor, or from other data or in logic of the compiler) to determine whether a barrier should be inserted into the graph. For instance, barriers may be inserted before each direct memory access and processor operation/task that has a data dependency (on another operation/task), such as when a task (e.g., mathematical or data movement operation) requires an output of a preceding operation/task before being able to successfully proceed. Barriers may be inserted before every task/operation that exceed the number of parallel barriers designated for use during the corresponding processing and implementation of the corresponding neural network (e.g., a group of eight hardware barriers (e.g., a subset of the overall barriers provided on the hardware) may be designated for a particular compute resource (target device type) or the aggregate collection of compute resources on the target device, etc). Barriers may also be inserted before operations that have a control dependency graph edge which forces serail operation, without data dependency (0102).
“arranging the candidate device blocks according to a preset rule, and merging the candidate device blocks that are continuous and belong to the same device type, to obtain the target device block; and
determining an input node and an output node of each target device block, according to information of the target node in the target device block.”
With barriers inserted into the graph (e.g., within the control model graph), graph theory-based analysis may be performed, among other optimization techniques, by the compiler, to identify opportunities to reduce the number of or otherwise optimize the barrier tasks. For instance, redundant barrier tasks may be combined (merged)(e.g., when two or more operation rely on the same preceding dependencies, they may share the same barrier (rather than each requiring their own distinct barrier)). In other instance changes may be made to the underlying control flow or data flow represented in the intermediate representation based on limited hardware barrier resources (e.g., to serialize operations when the number of parallel control flow paths outnumber the number of hardware barrier devices available on the target computing device). The intermediate representation is used by the compiler to generate the binary code to facilitate task scheduling using hardware barrier resources (0103). The examiner states that if nodes of the intermediate represent being traversed and barriers are being inserted into the graph based on certain criteria including dependencies and the intermediate representation is used to generated the binary code, then it would be inherent that inputs and outputs are determined in order for the generated binary to allow the neural network to execute correctly.
As per claim 8, Brady et al. further teaches, “The deployment method as claimed in claim 2, wherein the step of deploying the deep learning network to the heterogeneous platform, based on the topological structure diagram, the network parameter, the function information, the target device block and the space information of the target device block, comprises:
obtaining deployment information of an m-th target device block, based on the input data of the input node, the output data of the output node, the space information, the function information, and the network parameter that are in the m-th target device block arranged according to the preset rule;
determining whether the deployment information of the m-th target device block satisfies a deployment condition, wherein the deployment condition is that the m-th target device block does not need to wait for output data of a (m-1)-th target device block to serve as input data; and
if the deployment condition of the m-th target device block satisfies the deployment condition, sending the deployment information of the m-th target device block to a device with the same device type as that of the target device block of the m-th target device block in the heterogeneous platform.”
The control model implements a portion of the resource sub-model of the intermediate representation. The control model models the order and dependencies of the collection of operations determined to implement the neural network. The ordering may be determined not only from the nodes of the neural network graph, but also form the attributed and resource contains of the target hardware system, as identified in a target descriptor file (0064). The control model may define that operation 1110 is to begin after (and is dependent on) completion of operation 1105, that operation 1115 is to begin after (and is dependent on) completion of operation 1115. As operation 1125 is in a parallel branch as operations 1120 and 1120, operation 1125 is not dependent on operations 1120 or 1120and operations 1120 and 1130 may be performed before, after, or in parallel with operations (0065). Compilation passes configured to optimally determine buffers for the various tensors defined in the model , as well as allocate and assign address to memory of the target hardware for these buffer and determine addressing of the allocated memory (0075). Compilation passes may be performed using the operator model and./or control model of the IR to determine which operations are to be performed by which compute resources and in what order(0080). Also see 0091.
Additional compilation passes may be performed to allocate hardware barrier resources for the optimized compute graph. For instance, nodes in the transformed intermediate representation (e.g., the operation and/or control graph sub-models) may be traversed and opportunities may be identified for using hardware barrier resources (e.g., counters) on the target computing device. For instance, the compiler may determine whether a given operation represented in the transformed graph model is to be synchronized with a barrier (dividing node/operation from others) (and represented by a corresponding barrier task object). For instance one or more rules, conditions or algorithms may be defined (e.g., by the target descriptor or compilation descriptor, or from other data or in logic of the compiler) to determine whether a barrier should be inserted into the graph. For instance, barriers may be inserted before each direct memory access and processor operation/task that has a data dependency (on another operation/task), such as when a task (e.g., mathematical or data movement operation) requires an output of a preceding operation/task before being able to successfully proceed. Barriers may be inserted before every task/operation that exceed the number of parallel barriers designated for use during the corresponding processing and implementation of the corresponding neural network (e.g., a group of eight hardware barriers (e.g., a subset of the overall barriers provided on the hardware) may be designated for a particular compute resource (target device type) or the aggregate collection of compute resources on the target device, etc). Barriers may also be inserted before operations that have a control dependency graph edge which forces serail operation, without data dependency (0102-0103).
A binary is generated capable of being executed by the target hardware to implement the neural network (0076). The intermediate representation is used by the compiler to generate the binary code to facilitate task scheduling using hardware barrier resources (0103). The compiler may utilize hardware barrier information for a target machine learning device and generate one or more hardware barrier tasks to generate a binary that utilize the hardware barrier resources to realize optimized scheduling using these resources. The barrier tasks may be generated in association with one or more compilation passes and inserted in a graph of the intermediate representation (0053). Also see figure 1.
The examiner states that as shown above the intermediate representation with barriers is used to generate the binary code to facilitate task scheduling using hardware barrier resources. Therefore, when the binary is executed/deployed, the binary will perform the highlighted functions as shown above such as waiting for the output of a preceding operation/task before being able to successfully proceed based on task scheduling.
As per claim 9, Brady et al. further teaches, “The deployment method as claimed in claim 8, wherein after determining whether the deployment information of the m-th target device block satisfies the deployment condition, the method further comprises:
if the deployment information of the m-th target device block does not satisfy the deployment condition, after obtaining output data of the (m-1)-th target device block, sending the deployment information of the m-th target device block to the device with the same device type as that of the target device block of the m-th target device block in the heterogeneous platform. “
The control model implements a portion of the resource sub-model of the intermediate representation. The control model models the order and dependencies of the collection of operations determined to implement the neural network. The ordering may be determined not only from the nodes of the neural network graph, but also form the attributed and resource contains of the target hardware system, as identified in a target descriptor file (0064). The control model may define that operation 1110 is to begin after (and is dependent on) completion of operation 1105, that operation 1115 is to begin after (and is dependent on) completion of operation 1115. As operation 1125 is in a parallel branch as operations 1120 and 1120, operation 1125 is not dependent on operations 1120 or 1120and operations 1120 and 1130 may be performed before, after, or in parallel with operations (0065). Compilation passes configured to optimally determine buffers for the various tensors defined in the model , as well as allocate and assign address to memory of the target hardware for these buffer and determine addressing of the allocated memory (0075). Compilation passes may be performed using the operator model and./or control model of the IR to determine which operations are to be performed by which compute resources and in what order(0080). Also see 0091.
Additional compilation passes may be performed to allocate hardware barrier resources for the optimized compute graph. For instance, nodes in the transformed intermediate representation (e.g., the operation and/or control graph sub-models) may be traversed and opportunities may be identified for using hardware barrier resources (e.g., counters) on the target computing device. For instance, the compiler may determine whether a given operation represented in the transformed graph model is to be synchronized with a barrier (dividing node/operation from others) (and represented by a corresponding barrier task object). For instance one or more rules, conditions or algorithms may be defined (e.g., by the target descriptor or compilation descriptor, or from other data or in logic of the compiler) to determine whether a barrier should be inserted into the graph. For instance, barriers may be inserted before each direct memory access and processor operation/task that has a data dependency (on another operation/task), such as when a task (e.g., mathematical or data movement operation) requires an output of a preceding operation/task before being able to successfully proceed. Barriers may be inserted before every task/operation that exceed the number of parallel barriers designated for use during the corresponding processing and implementation of the corresponding neural network (e.g., a group of eight hardware barriers (e.g., a subset of the overall barriers provided on the hardware) may be designated for a particular compute resource (target device type) or the aggregate collection of compute resources on the target device, etc). Barriers may also be inserted before operations that have a control dependency graph edge which forces serail operation, without data dependency (0102-0103).
A binary is generated capable of being executed by the target hardware to implement the neural network (0076). The intermediate representation is used by the compiler to generate the binary code to facilitate task scheduling using hardware barrier resources (0103). The compiler may utilize hardware barrier information for a target machine learning device and generate one or more hardware barrier tasks to generate a binary that utilize the hardware barrier resources to realize optimized scheduling using these resources. The barrier tasks may be generated in association with one or more compilation passes and inserted in a graph of the intermediate representation (0053). Also see figure 1.
The examiner states that as shown above the intermediate representation with barriers is used to generate the binary code to facilitate task scheduling using hardware barrier resources. Therefore, when the binary is executed/deployed, the binary will perform the highlighted functions as shown above such as waiting for the output of a preceding operation/task before being able to successfully proceed based on task scheduling.
As per claims 10-11, claims 10-11 contain similar limitations to claim 1 and are therefore rejected for the same reasons.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Tianqi Chen et al, “TVM: An Automated End-to-End Optimization Compiler for Deep Learning” 2018, Teaches a TCM compiler that exposes graph-level and operator-level optimizations to provide performance portability to deep learning workloads across diverse hardware back-ends (abstract).
Nishant Malpani, “A Brief Review of TVM: An Automated End-to-End Optimizing Compiler for Deep Learning” 2018, teaches a TVM compiler that optimizes a deep learning model for deployment (Abstract).
Kuo et al. (WO 2019/103999 A1), teaches generating and deploying packages to implement machine learning at connected devices (abstract).
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MARK A GOORAY whose telephone number is (571)270-7805. The examiner can normally be reached Monday - Friday 10:00am - 6:00pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Lewis Bullock can be reached at 571-272-3759. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MARK A GOORAY/ Examiner, Art Unit 2199
/DUY KHUONG T NGUYEN/Primary Examiner, Art Unit 2199