Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Applicant's claim for the benefit of a prior-filed application under 35 U.S.C. 119(e) or under 35 U.S.C. 120, 121, 365(c), or 386(c) is acknowledged. The present application claims foreign priority based on Korean Patent Application No. KR10-2022-0135259 filed on October 19, 2022. Receipt is acknowledged of certified copies of papers required by 37 CFR 1.55.
Information Disclosure Statement
Acknowledgment is made of the information disclosure statements filed August 16, 2023, which comply with 37 CFR 1.97. As such, the information disclosure statements have been placed in the application file and the information referred to therein has been considered by the examiner.
Acknowledgment is made of the information disclosure statements filed January 30, 2026, which comply with 37 CFR 1.97. As such, the information disclosure statements have been placed in the application file and the information referred to therein has been considered by the examiner.
Allowable Subject Matter
Claims 10, 11, 12, 17, 18, and 19 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims. Examiner’s Note: Subject matter ineligibility as a judicial exception under 35 U.S.C. § 101 still stands.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-23 are rejected under 35 U.S.C. 101 because the claimed invention is directed to non-statutory subject matter.
According to the USPTO guidelines, a claim is directed to non-statutory subject matter if:
Step 1: The claim does not fall within one of the four statutory categories of invention (process, machine, manufacture, or composition of matter), or,
Step 2: The claim recites a judicial exception, e.g. an abstract idea, without reciting additional elements that amount to significantly more than the judicial exception, as determined using the following analysis:
Step 2A, Prong 1: Does the claim recite an abstract idea, law of nature, or natural phenomenon?
Step 2A, Prong 2: Does the claim recite additional elements that integrate the judicial exception into a practical application?
Step 2B: Does the claim recite additional elements that amount to significantly more than the judicial exception?
MPEP 2106.04(a)(2)(I) states: "The mathematical concepts grouping is defined as
mathematical relationships, mathematical formulas or equations, and mathematical
calculations."
MPEP 2106.04(a)(2)(III) states: "Accordingly, the "mental processes" abstract idea grouping is defined as concepts performed in the human mind, and examples of mental
processes include observations, evaluations, judgements, and opinions.
Further, the MPEP states: "The courts do not distinguish between mental processes that
are performed entirely in the human mind and mental processes that require a human to use a
physical aid (e.g. pen and paper or a slide run) to perform the claim limitation.
Using the two-step inquiry, it is clear that Claims 1-23 are each directed to non-statutory
subject matter as shown below:
Please note the following:
The following groups of claims are expressed in different statutory categories:
Claims 1-12 are directed to a system/apparatus/device comprising of a plurality of processors and memories configured to carry out a process for providing optimization of training-scheduling and hardware-resource-allocation techniques for transformer models.
Claims 13-21 are directed to a method for providing optimization of training-scheduling and hardware-resource-allocation techniques for transformer models by making them more efficient.
Claims 22-23 are directed to a system/apparatus/device comprising of a memory and a processor containing executable instructions configured to carry out a process for providing resource consumption analysis of transformer models.
With respect to Claim 1, which is an independent claim:
Step 1: Claim 1 is directed to a system for providing dynamically optimized training of transformer models by dividing training data into independent micro-batches, corresponding to an article of manufacture, which is one of the four statutory categories of patentable subject matter.
Step 2A, Prong 1: A judicial exception is recited in the claims as they recite mental processes, which are abstract ideas:
“by configuring the processors to: identify batches of training data into a plurality of micro-batches;” ; Identifying batches of training data as a plurality of micro-batches is an abstract idea of a mental process that can practically be performed in the human mind, with or without the use of a physical aid such as pen and paper (including an observation, evaluation, judgment, opinion) -See MPEP § 2106.04(a)(2)(III).
“select layer pairs for the plurality of micro-batches;”; Selecting layer pairs for a plurality of micro-batches is an abstract idea of a mental process that can practically be performed in the human mind, with or without the use of a physical aid such as pen and paper (including an observation, evaluation, judgment, opinion) -See MPEP § 2106.04(a)(2)(III).
”assemble a processing order of the layer pairs;”; Assembling a processing order of the layer pairs is an abstract idea of a mental process that can practically be performed in the human mind, with or without the use of a physical aid such as pen and paper (including an observation, evaluation, judgment, opinion) -See MPEP § 2106.04(a)(2)(III).
“determine resource information to be allocated to the layer pairs;”; Determining resource information to be allocated to layer pairs is an abstract idea of a mental process that can practically be performed in the human mind, with or without the use of a physical aid such as pen and paper (including an observation, evaluation, judgment, opinion) -See MPEP § 2106.04(a)(2)(III).
Step 2A, Prong 2: The claims do not recite additional elements that integrate the judicial exception into a practical application:
“and allocating resources to the layer pairs based on the determined resource information to be allocated to the layer pairs, dependent on the processing order of the layer pairs.”; Allocating resources to the layer pairs based on determined resource information dependent on the processing order is considered insignificant extra-solution activity (post-solution activity) - see MPEP 2106.05(g).
Step 2B: The claims do not recite additional elements that amount to significantly more than the judicial exception. The act of allocating resources to the layer pairs based on resource information and initial dependency on the processing order already being determined constitutes as an insignificant extra-solution activity, specifically a post-solution activity. - see MPEP 2106.05(g).
Therefore, Claim 1 is directed to non-statutory subject matter and rejected.
With respect to Claim 2, which is dependent on Claim 1 respectively:
Step 2A, Prong 1: A judicial exception is recited in the claims as they recite mental processes, which are abstract ideas:
“divide the batches into a plurality of micro-batches having no dependency on each other.”; Dividing batches into a plurality of micro-batches having no dependency on each other is an abstract idea of a mental process that can practically be performed in the human mind, with or without the use of a physical aid such as pen and paper (including an observation, evaluation, judgment, opinion) -See MPEP § 2106.04(a)(2)(III).
Step 2A, Prong 2: The claims do not recite additional elements that integrate the judicial exception into a practical application:
Step 2B: The claims do not recite additional elements that amount to significantly more than the judicial exception.
Therefore, Claim 2 is directed to non-statutory subject matter and rejected.
With respect to Claim 3, which is dependent on Claim 1 respectively:
Step 2A, Prong 1: A judicial exception is recited in the claims as they recite mental processes, which are abstract ideas:
“calculate an idle time in response to resources being allocated to the layer pairs.”; Calculating an idle time in response to resources being allocated to the layer pairs is an abstract idea of a mathematical calculation, as directed to “a claim that recites a mathematical calculation, when the claim is given its broadest reasonable interpretation in light of the specification, will be considered as falling within the "mathematical concepts" grouping. A mathematical calculation is a mathematical operation (such as multiplication) or an act of calculating using mathematical methods to determine a variable or number”. See MPEP § 2106.04(a)(2)(I)(C).
Step 2A, Prong 2: The claims do not recite additional elements that integrate the judicial exception into a practical application:
Step 2B: The claims do not recite additional elements that amount to significantly more than the judicial exception.
Therefore, Claim 3 is directed to non-statutory subject matter and rejected.
With respect to Claim 4, which is dependent on Claim 3 respectively:
Step 2A, Prong 1: A judicial exception is recited in the claims as they recite mental processes, which are abstract ideas:
“calculate the idle time until the calculated idle time is minimized for the layer pairs.”; Calculating an idle time until the calculated idle time is minimized for the layer pairs is an abstract idea of a mathematical calculation, as directed to “a claim that recites a mathematical calculation, when the claim is given its broadest reasonable interpretation in light of the specification, will be considered as falling within the "mathematical concepts" grouping. A mathematical calculation is a mathematical operation (such as multiplication) or an act of calculating using mathematical methods to determine a variable or number”. See MPEP § 2106.04(a)(2)(I)(C).
Step 2A, Prong 2: The claims do not recite additional elements that integrate the judicial exception into a practical application:
Step 2B: The claims do not recite additional elements that amount to significantly more than the judicial exception.
Therefore, Claim 4 is directed to non-statutory subject matter and rejected.
With respect to Claim 5, which is dependent on Claim 1 respectively:
Step 2A, Prong 1: A judicial exception is recited in the claims as they recite mental processes, which are abstract ideas:
“for the selecting, the one or more processors are configured to assign layers of the plurality of micro-batches as the layer pairs based on an idle time.”; Selecting/Assigning layers of the plurality of micro-batches as layer pairs based on idle time is an abstract idea of a mental process that can practically be performed in the human mind, with or without the use of a physical aid such as pen and paper (including an observation, evaluation, judgment, opinion) -See MPEP § 2106.04(a)(2)(III).
Step 2A, Prong 2: The claims do not recite additional elements that integrate the judicial exception into a practical application:
Step 2B: The claims do not recite additional elements that amount to significantly more than the judicial exception.
Therefore, Claim 5 is directed to non-statutory subject matter and rejected.
With respect to Claim 6, which is dependent on Claim 1 respectively:
Step 2A, Prong 1: A judicial exception is recited in the claims as they recite mental processes, which are abstract ideas:
“calculate a total operation execution time in response to resources being allocated to the layer pairs.”; Calculating a total operation execution time in response to resources being allocated to the layer pairs is an abstract idea of a mathematical calculation, as directed to “a claim that recites a mathematical calculation, when the claim is given its broadest reasonable interpretation in light of the specification, will be considered as falling within the "mathematical concepts" grouping. A mathematical calculation is a mathematical operation (such as multiplication) or an act of calculating using mathematical methods to determine a variable or number”. See MPEP § 2106.04(a)(2)(I)(C).
Step 2A, Prong 2: The claims do not recite additional elements that integrate the judicial exception into a practical application:
Step 2B: The claims do not recite additional elements that amount to significantly more than the judicial exception.
Therefore, Claim 6 is directed to non-statutory subject matter and rejected.
With respect to Claim 7, which is dependent on Claim 6 respectively:
Step 2A, Prong 1: A judicial exception is recited in the claims as they recite mental processes, which are abstract ideas:
“to minimize the total operation execution time, used in the identifying, until the calculated total operation execution time is minimized.”; Minimizing the total operation execution time until the calculated total operation execution time is minimized is an abstract idea of a mathematical calculation, as directed to “a claim that recites a mathematical calculation, when the claim is given its broadest reasonable interpretation in light of the specification, will be considered as falling within the "mathematical concepts" grouping. A mathematical calculation is a mathematical operation (such as multiplication) or an act of calculating using mathematical methods to determine a variable or number”. See MPEP § 2106.04(a)(2)(I)(C).
Step 2A, Prong 2: The claims do not recite additional elements that integrate the judicial exception into a practical application:
Step 2B: The claims do not recite additional elements that amount to significantly more than the judicial exception.
Therefore, Claim 7 is directed to non-statutory subject matter and rejected.
With respect to Claim 8, which is dependent on Claim 1 respectively:
Step 2A, Prong 1: A judicial exception is recited in the claims as they recite mental processes, which are abstract ideas:
“assign layers of the plurality of micro-batches as the layer pairs based on a total operation execution time.”; Assigning layers of the plurality of micro-batches as layer pairs based on idle time is an abstract idea of a mental process that can practically be performed in the human mind, with or without the use of a physical aid such as pen and paper (including an observation, evaluation, judgment, opinion) -See MPEP § 2106.04(a)(2)(III).
Step 2A, Prong 2: The claims do not recite additional elements that integrate the judicial exception into a practical application:
Step 2B: The claims do not recite additional elements that amount to significantly more than the judicial exception.
Therefore, Claim 8 is directed to non-statutory subject matter and rejected.
With respect to Claim 13 (independent claim) and claim 21, with inherited limitations:
Step 1: Claim 13 is directed to a method, also known as a process, which is one of the four statutory categories of patentable subject matter.
Step 2A, Prong 1: A judicial exception is recited in the claims as they recite mental processes, which are abstract ideas:
“dividing batches of data into a plurality of micro-batches;”; Dividing batches into a plurality of micro-batches is an abstract idea of a mental process that can practically be performed in the human mind, with or without the use of a physical aid such as pen and paper (including an observation, evaluation, judgment, opinion) -See MPEP § 2106.04(a)(2)(III).
“forming layer pairs from layers of the plurality of micro-batches;”; Forming layer pairs for a plurality of micro-batches is an abstract idea of a mental process that can practically be performed in the human mind, with or without the use of a physical aid such as pen and paper (including an observation, evaluation, judgment, opinion) -See MPEP § 2106.04(a)(2)(III).
“generating an operation processing order of the layer pairs;”; Generating an operation processing order of the layer pairs is an abstract idea of a mental process that can practically be performed in the human mind, with or without the use of a physical aid such as pen and paper (including an observation, evaluation, judgment, opinion) -See MPEP § 2106.04(a)(2)(III).
“determining resource information to be allocated to the layer pairs;”; Determining resource information to be allocated to the layer pairs is an abstract idea of a mental process that can practically be performed in the human mind, with or without the use of a physical aid such as pen and paper (including an observation, evaluation, judgment, opinion) -See MPEP § 2106.04(a)(2)(III).
Step 2A, Prong 2: The claims do not recite additional elements that integrate the judicial exception into a practical application:
“and allocating resources to the layer pairs based on the resource information.”; Allocating resources to the layer pairs based on the resource information is considered insignificant extra-solution activity (post-solution activity) - see MPEP 2106.05(g).
Step 2B: The claims do not recite additional elements that amount to significantly more than the judicial exception. Allocating resources to the layer pairs based on the resource information constitutes as an insignificant extra-solution activity, specifically a post-solution activity. - see MPEP 2106.05(g).
Therefore, Claims 13 and 21 are directed to non-statutory subject matter and rejected.
With respect to Claims 9 and 16, which are dependent on Claims 1 and 13 respectively:
Step 2A, Prong 1: A judicial exception is recited in the claims as they recite mental processes, which are abstract ideas:
“respectively classify layers of each of the layer pairs into a corresponding layer type, among predefined layer types, according to an operation per byte ratio of an operation performed on each of the layers;”; Classifying layers in their corresponding pre-defined layer type as a result of an operation per byte ratio calculation being performed on said layers is an abstract idea of a mathematical calculation, as directed to “a claim that recites a mathematical calculation, when the claim is given its broadest reasonable interpretation in light of the specification, will be considered as falling within the "mathematical concepts" grouping. A mathematical calculation is a mathematical operation (such as multiplication) or an act of calculating using mathematical methods to determine a variable or number”. See MPEP § 2106.04(a)(2)(I)(C).
“and determine resource information to be allocated to the layer types where each layer belongs to a separate type of layer.”; Determining resource information to be allocated to the layer types where each layer belongs to a separate type of layer is an abstract idea of a mental process that can practically be performed in the human mind, with or without the use of a physical aid such as pen and paper (including an observation, evaluation, judgment, opinion) -See MPEP § 2106.04(a)(2)(III).
Step 2A, Prong 2: The claims do not recite additional elements that integrate the judicial exception into a practical application:
Step 2B: The claims do not recite additional elements that amount to significantly more than the judicial exception.
Therefore, Claims 9 and 16 are directed to non-statutory subject matter and rejected.
With respect to Claims 10 and 17, which are dependent on Claims 9 and 16 respectively:
Step 2A, Prong 1: A judicial exception is not recited in the claims as they do not recite an abstract idea (mathematical concepts, certain methods of organizing human activity, or mental processes), law of nature, or natural phenomenon.
Step 2A, Prong 2: The claims do not recite additional elements that integrate the judicial exception into practical application:
"a first layer type including layers having a first operation per byte ratio of the operation performed on each of the layers that is greater than or equal to a predetermined first reference operation per byte ratio;”; Performing a per byte operation on a first layer type that is greater than or equal to a predetermined first reference operation per byte ratio only amounts to "apply it" and the mere instructions to implement an abstract idea on a computer - see MPEP 2106.05(f)(1) in addition to generally links the use of the abstract idea to a particular technological environment or field of use - See MPEP § 2106.05(h).
“a second layer type including layers having a second operation per byte ratio of the operation performed on each of the layers that is less than the predetermined first reference operation per byte ratio and that is greater than or equal to a predetermined second reference operation per byte ratio;”; Performing a second per byte operation on a second layer type that is less than a predetermined first reference operation per byte ratio but greater than or equal to a predetermined second reference operation only amounts to "apply it" and the mere instructions to implement an abstract idea on a computer - see MPEP 2106.05(f)(1) in addition to generally links the use of the abstract idea to a particular technological environment or field of use - See MPEP § 2106.05(h).
“and a third layer type including layers having a third operation per byte ratio of the operation performed on each of the layers that is less than the predetermined second reference operation per byte ratio.”; Performing a third per byte operation on a third layer type that is less than a predetermined second reference operation per byte ratio only amounts to "apply it" and the mere instructions to implement an abstract idea on a computer - see MPEP 2106.05(f)(1) in addition to generally links the use of the abstract idea to a particular technological environment or field of use - See MPEP § 2106.05(h).
Step 2B: Performing a per byte operation on a first layer type that is greater than or equal to a predetermined first reference operation per byte ratio, performing a second per byte operation on a second layer type that is less than a predetermined first reference operation per byte ratio but greater than or equal to a predetermined second reference operation, and performing a third per byte operation on a third layer type that is less than a predetermined second reference operation per byte ratio amounts to "apply it" and mere instructions to implement an abstract idea on a computer. The claim fails to recite details of how a solution or outcome to a problem is accomplished because it is unclear how the "AI system" or "machine learning" is used nor does the specification make it clear how these actions are performed - see MPEP 2106.05(f)(1)). The usage of a machine learning model layers and per byte ratio operations is generally linked to a particular technological environment or field of use (AI/ML/DL/GPU Performance) - see MPEP 2106.05(h).
Therefore, Claims 10 and 17 are directed to non-statutory subject matter and rejected.
With respect to Claims 11 and 18, which are dependent on Claims 9 and 16 respectively:
Step 2A, Prong 1: A judicial exception is not recited in the claims as they do not recite an abstract idea (mathematical concepts, certain methods of organizing human activity, or mental processes), law of nature, or natural phenomenon.
Step 2A, Prong 2: The claims do not recite additional elements that integrate the judicial exception into practical application:
"allocate respective layers that form a respective layer pair to a resource core responsive to the respective layers belonging to a first layer type and a second layer type;”; Allocating respective layers that form a pair to a resource core responsive to the respective layers belonging to a first layer and second layer type only amounts to "apply it" and the mere instructions to implement an abstract idea on a computer - see MPEP 2106.05(f)(1) in addition to generally links the use of the abstract idea to a particular technological environment or field of use - See MPEP § 2106.05(h).
“and allocate a first layer type of the respective layer pair to the resource core and a third layer type of the respective layer pair to a unified vector unit (VU) responsive to the respective layers belonging to the first layer type and the third layer type.”; Allocating a first layer type to a resource core and a third layer type to a unified vector unit responsive to the respective layers belonging to a first layer and third layer type only amounts to "apply it" and the mere instructions to implement an abstract idea on a computer - see MPEP 2106.05(f)(1) in addition to generally links the use of the abstract idea to a particular technological environment or field of use - See MPEP § 2106.05(h).
Step 2B: Allocating respective layers that form a pair to a resource core responsive to the respective layers belonging to a first layer and second layer type and allocating a first layer type to a resource core and a third layer type to a unified vector unit responsive to the respective layers belonging to a first layer and third layer type amounts to "apply it" and mere instructions to implement an abstract idea on a computer. The claim fails to recite details of how a solution or outcome to a problem is accomplished because it is unclear how the "AI system" or "machine learning" is used nor does the specification make it clear how these actions are performed - see MPEP 2106.05(f)(1)). The usage of machine learning model layers and a resource core is generally linked to a particular technological environment or field of use (AI/ML/DL/GPU Performance) - see MPEP 2106.05(h).
Therefore, Claims 11 and 18 are directed to non-statutory subject matter and rejected.
With respect to Claims 12 and 19, which are dependent on Claims 11 and 17 respectively:
Step 2A, Prong 1: A judicial exception is recited in the claims as they recite mental processes, which are abstract ideas:
“wherein the first layer type comprises a layer on which a general matrix multiply operation is performed, wherein the second layer type comprises a layer on which a batched general matrix multiply operation is performed, and wherein the third layer type comprises a layer on which a normalization operation is performed.”; Performing generalized matrix multiplication, batched matrix multiplication, and normalization on model layers is an abstract idea of a mathematical calculation, as directed to “a claim that recites a mathematical calculation, when the claim is given its broadest reasonable interpretation in light of the specification, will be considered as falling within the "mathematical concepts" grouping. A mathematical calculation is a mathematical operation (such as multiplication) or an act of calculating using mathematical methods to determine a variable or number”. See MPEP § 2106.04(a)(2)(I)(C).
Step 2A, Prong 2: The claims do not recite additional elements that integrate the judicial exception into a practical application:
Step 2B: The claims do not recite additional elements that amount to significantly more than the judicial exception.
Therefore, Claims 12 and 19 are directed to non-statutory subject matter and rejected.
With respect to Claim 14, which is dependent on Claim 13 respectively:
Step 2A, Prong 1: A judicial exception is recited in the claims as they recite mental processes, which are abstract ideas:
“wherein the dividing of the batches into the plurality of micro-batches comprises dividing the batches into the plurality of micro-batches to minimize a total operation execution time.”; Dividing batches into a plurality of micro-batches to minimize total operation execution time is an abstract idea of a mental process that can practically be performed in the human mind, with or without the use of a physical aid such as pen and paper (including an observation, evaluation, judgment, opinion) -See MPEP § 2106.04(a)(2)(III).
Step 2A, Prong 2: The claims do not recite additional elements that integrate the judicial exception into a practical application:
Step 2B: The claims do not recite additional elements that amount to significantly more than the judicial exception.
Therefore, Claim 14 is directed to non-statutory subject matter and rejected.
With respect to Claim 15, which is dependent on Claim 13 respectively:
Step 2A, Prong 1: A judicial exception is recited in the claims as they recite mental processes, which are abstract ideas:
“wherein the forming of the layer pairs comprises assigning layers from the plurality of micro-batches into the layer pairs to minimize an idle time.”; Forming layer pairs from a plurality of micro-batches by assigning them as pairs to minimize an idle time is an abstract idea of a mental process that can practically be performed in the human mind, with or without the use of a physical aid such as pen and paper (including an observation, evaluation, judgment, opinion) -See MPEP § 2106.04(a)(2)(III).
Step 2A, Prong 2: The claims do not recite additional elements that integrate the judicial exception into a practical application:
Step 2B: The claims do not recite additional elements that amount to significantly more than the judicial exception.
Therefore, Claim 15 is directed to non-statutory subject matter and rejected.
With respect to Claim 20, which is dependent on Claim 13 respectively:
Step 2A, Prong 1: A judicial exception is not recited in the claims as they do not recite an abstract idea (mathematical concepts, certain methods of organizing human activity, or mental processes), law of nature, or natural phenomenon.
Step 2A, Prong 2: The claims do not recite additional elements that integrate the judicial exception into practical application:
"performing training of a transformer or an inference operation of a trained transformer, using the allocated resources.”; Using allocated resources to perform training of a transformer or an inference operation of a trained transformer only amounts to "apply it" and the mere instructions to implement an abstract idea on a computer - see MPEP 2106.05(f)(1) in addition to generally links the use of the abstract idea to a particular technological environment or field of use - See MPEP § 2106.05(h).
Step 2B: Using allocated resources to perform training of a transformer or an inference operation of a trained transformer amounts to "apply it" and mere instructions to implement an abstract idea on a computer. The claim fails to recite details of how a solution or outcome to a problem is accomplished because it is unclear how the "AI system" or "machine learning" is used nor does the specification make it clear how these actions are performed - see MPEP 2106.05(f)(1)). The usage of an inference operation and trained/training transformer model is generally linked to a particular technological environment or field of use (AI/ML/DL) - see MPEP 2106.05(h).
Therefore, Claim 20 is directed to non-statutory subject matter and rejected.
With respect to Claim 22, which is an independent claim:
Step 1: Claim 22 is directed to a system for providing dynamically optimized training of transformer models by dividing training data into independent micro-batches, corresponding to an article of manufacture, which is one of the four statutory categories of patentable subject matter.
Step 2A, Prong 1: A judicial exception is recited in the claims as they recite mental processes, which are abstract ideas:
“identify a plurality of batches of input data into a plurality of micro-batches, wherein each micro-batch of the plurality of micro-batches has no dependency to other micro-batches of the plurality of micro-batches;” ; Identifying batches of training data as a plurality of micro-batches is an abstract idea of a mental process that can practically be performed in the human mind, with or without the use of a physical aid such as pen and paper (including an observation, evaluation, judgment, opinion) -See MPEP § 2106.04(a)(2)(III).
Step 2A, Prong 2: The claims do not recite additional elements that integrate the judicial exception into a practical application:
“and assign a layer pair to each micro-batch of the plurality of micro-batch according to a resource consumption indicator, dependent on an analysis of layers of micro-batches for the consumption indicator, and a layer type of each layer of the layer pair.”; Assigning a layer pair to each micro-batch according to a resource consumption indicator dependent on an analysis of layers of micro-batches and a layer type of each layer only amounts to "apply it" and the mere instructions to implement an abstract idea on a computer - see MPEP 2106.05(f)(1) in addition to generally links the use of the abstract idea to a particular technological environment or field of use - See MPEP § 2106.05(h).
Step 2B: The claims do not recite additional elements that amount to significantly more than the judicial exception. Assigning a layer pair to each micro-batch according to a resource consumption indicator dependent on an analysis of layers of micro-batches and a layer type of each layer amounts to "apply it" and mere instructions to implement an abstract idea on a computer. The claim fails to recite details of how a solution or outcome to a problem is accomplished because it is unclear how the "AI system" or "machine learning" is used nor does the specification make it clear how these actions are performed - see MPEP 2106.05(f)(1)). The usage of machine learning model layers and a resource consumption indicator is generally linked to a particular technological environment or field of use (AI/ML/DL/GPU Performance) - see MPEP 2106.05(h).
Therefore, Claim 22 is directed to non-statutory subject matter and rejected.
With respect to Claim 23, which is dependent on Claim 22 respectively:
Step 2A, Prong 1: A judicial exception is not recited in the claims as they do not recite an abstract idea (mathematical concepts, certain methods of organizing human activity, or mental processes), law of nature, or natural phenomenon.
Step 2A, Prong 2: The claims do not recite additional elements that integrate the judicial exception into practical application:
"allocate resources to the layer pair based on the resource information to be allocated to a plurality of layer pairs, dependent on a processing order of the layer pairs.”; Allocating resources to the layer pairs based on determined resource information dependent on the processing order is considered insignificant extra-solution activity (post-solution activity) - see MPEP 2106.05(g).
Step 2B: The act of allocating resources to the layer pairs based on resource information and initial dependency on the processing order already being determined constitutes as an insignificant extra-solution activity, specifically a post-solution activity. - see MPEP 2106.05(g).
Therefore, Claim 23 is directed to non-statutory subject matter and rejected.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or non-obviousness.
Claims 1-8, 13, 14, 15, 20 and 21 are rejected under 35 U.S.C. 103 as being unpatentable over Lamy-Poirier, (U.S Patent Application Publication No. US 20220383084 A1, filed on February 9, 2022, hereinafter "Lamy"), in view of Lee et. Al, (U.S Patent Application Publication No. US 20220114015 A1, filed on March 9, 2021, hereinafter "Lee"). Both these dates are before the effective filing date of this application, i.e., August 16, 2023 as well as the foreign priority filing date of October 19, 2022 where it applies.
With respect to independent Claim 1:
Lamy teaches:
“by configuring the processors to: identify batches of training data into a plurality of micro-batches;” (Paragraph [0127] teaches identifying batches of training data into micro-batches, “…with respect to the model parameters is estimated from a micro-batch of the training data. Such a micro-batch can be as small as a single sample of training data, or multiple samples, all of which are processed by the same computational node or set of computational nodes…”)
“select layer pairs for the plurality of micro-batches;” (Paragraph [0152] discloses the selecting of specific layer(s) associated with micro-batches, “In the "Layered Gradient Accumulation," the input batch was split into micro-batches the same as for the non-layered gradient accumulation, but all of the micro batches are processed (forward or backward) for a given layer before proceeding to the next layer. To save on memory use, such layers can be selected as the intervals between activation checkpoints…”)
“assemble a processing order of the layer pairs;” (Paragraph [0164] denotes how the layers will be ordered (sequentially), “The example embodiment of FIG. 9 includes assigning layers of a machine learning model to a plurality of compute nodes, wherein each of the layers is assigned to exactly one of the compute nodes, wherein the layers have a sequential ordering (910).”)
Lamy does not appear to explicitly disclose:
“determine resource information to be allocated to the layer pairs;”
“and allocating resources to the layer pairs based on the determined resource information to be allocated to the layer pairs, dependent on the processing order of the layer pairs.”
However, Lee teaches:
“determine resource information to be allocated to the layer pairs;” (Paragraph [0083] teaches that resource information that is allocated to the layers varies/requires determination, “…models and/or layers included in each of the models may have different workload characteristics, and thus the operation resource and the memory access resource that would be used for each model or layer may differ for each model or layer. Thus, by performing scheduling based on the workload characteristics, to increase or maximally overlap times for which the memory and computation/operation resources in the accelerator 210 are used and to reduce or minimize idle times, various examples may improve an overall system performance.”)
“and allocating resources to the layer pairs based on the determined resource information to be allocated to the layer pairs, dependent on the processing order of the layer pairs.” (Paragraph [0115] teaches allocating previously determined resources to the layer pairs that is also relevant/dependent on the processing order of the layer pairs, “For example, while one layer included in the first model is being allocated to the operation resource of the accelerator, the scheduler may allocate, to the memory access resource of the accelerator, a subsequent layer of the first model or a layer of the second model needed to be subsequently processed.”)
Lamy and Lee are analogous art and in the same field of invention because both references pertain to optimizing the computational execution and parallel processing of complex, multi-layered machine learning architectures (such as transformers). While Lamy teaches the pipelined training implementation of multiple compute nodes for the purpose of processing different chunks of data concurrently, Lee teaches layer-wise scheduling across different models based on estimated idle times. It would have been obvious to a person having ordinary skill in the art (PHOSITA) before the effective filing date of the claimed invention to implement the base reference of Lamy (sequentially ordered forward/backward propagation of micro-batched data) with the teachings of Lee (sustainable inference-phase scheduling) in order to maximize hardware computing capacity and minimize processing bottlenecks by smartly scheduling operations, accelerating training and inference speeds, and making the deployment of massive resource-heavy models more cost-effective. One of ordinary skill in the art would be motivated to do so because by integrating Lamy's framework into the methods of Lee one would be able to, "improve training efficiency while also reducing memory usage and network requirements…,{[0123] of Lamy}."
Therefore, Claim 1 is rejected.
With respect to Claim 2:
The combination of Lamy and Lee teaches:
“divide the batches into a plurality of micro-batches having no dependency on each other.” (Paragraph [0164] of Lamy teaches dividing the batches of training data into micro-batches, “The example embodiment of FIG. 9 additionally includes dividing training data for the machine learning model into a plurality of micro-batches (912).” Paragraph [0129] further discloses that these micro-batches have no dependency on each other, “In data parallelism, a batch of training samples is split between the devices, each independently processing a single micro-batch…” Paragraph [0091] further discloses no dependency, “In an example set of models, there may be no data dependency between the models…”)
Therefore, Claim 2 is rejected.
With respect to Claim 3:
The combination of Lamy and Lee teaches:
“calculate an idle time in response to resources being allocated to the layer pairs.” (Paragraph [0114] of Lee discloses resources being allocated to layer pairs, “When the memory access operation is terminated, the layer L11 is allocated to an operation resource and a computation operation associated with the layer L11 is performed, and a first layer L21 of the second model is allocated to the memory access resource and a memory access operation associated with the layer L21 is performed. When the memory access operation associated with the layer L21 is terminated during the computation operation associated with the layer L11, a second layer L12 of the first model is subsequently allocated to the memory access resource.” Paragraph [0103] further discloses that upon resource allocation to a pair of layers an idle time can be calculated, “In the example of FIG . 4, a scheduler may determine an idle time of the operation resource based on a difference between a point in time t2 at which the operation resource is most recently) executed for a previous layer and a point in time t, at which the memory access resource is most recently executed, and on an execution time for which the memory access resource is executed for a candidate layer, which is a target for scheduling. For example, the scheduler may calculate the idle time of the operation resource…”)
Therefore, Claim 3 is rejected.
With respect to Claim 4:
The combination of Lamy and Lee teaches:
“calculate the idle time until the calculated idle time is minimized for the layer pairs.” (Paragraph [0116] of Lee describes calculation and analysis of idle times for layer pairs until eventual minimization is achieved, “the scheduler may perform layer wise scheduling on the first model and the second model based on a workload characteristic of each layer of the first model and the second model and a hardware resource of the accelerator. Thus, idle times may be minimized or reduced from occurring in each resource and the utilization of an accelerator.”)
Therefore, Claim 4 is rejected.
With respect to Claim 5:
The combination of Lamy and Lee teaches:
“for the selecting, the one or more processors are configured to assign layers of the plurality of micro-batches as the layer pairs based on an idle time.” (Paragraph [0158] of Lamy discloses assigning micro-batch reliant layer pairs, “The computation is done in the expected way, with a given instance processing all micro-batches for a given layer, then beginning the next assigned layer, the input for which should be ready and received from the instance assigned the immediately preceding layer, etc.”)
Lamy does not appear to explicitly disclose:
“…based on an idle time.”
However, Lee teaches:
“…based on an idle time.” (Paragraph [0022] teaches that specified operations are performed on selected/assigned layer pairs based on idle times, “for execution of a plurality of models to be independently executed in an accelerator, and a layer-wise scheduling of the plurality of models, for execution by the accelerator, based on estimated idle times for respective candidate layers of each of the plurality of models corresponding to the queued respective requests.”)
Therefore, Claim 5 is rejected.
With respect to Claim 6:
The combination of Lamy and Lee teaches:
“calculate a total operation execution time in response to resources being allocated to the layer pairs.” (Paragraph [0012] of Lee discloses determining/acquiring a total execution time as a result of resources being allocated to the layer pairs, “…with respect to each of the respective candidate layers, may occur when an execution time of the corresponding memory access resource is respectively greater than an execution time of an operation resource for a previous layer that is most recently scheduled.”)
Therefore, Claim 6 is rejected.
With respect to Claim 7:
The combination of Lamy and Lee teaches:
“to minimize the total operation execution time, used in the identifying, until the calculated total operation execution time is minimized.” (Paragraph [0122] of Lamy discloses the intentional reduction of total operation execution time, “implement a variety of parallel training configurations and strategies for large and dense transformers or other large, multi-layered machine learning model types, with the goal of reducing training time…” Paragraph [0177] further describes specific scenarios in which total operation execution time is calculated and undergoes optimization in order to be minimized, “The resulting configurations are summarized in Table 1, together with the expected computational efficiency and approximate predicted training time. We found both data and tensor parallelism to be necessary (and together sufficient) to train in a reasonable amount of time. Modular pipeline parallelism stood out with both a high GPU count and near-optimal efficiency, allowing the model to be trained in a week. It also out-performed the baseline in the absence of tensor parallelism, but in that case the training time remained above three months.”)
Therefore, Claim 7 is rejected.
With respect to Claim 8:
The combination of Lamy and Lee teaches:
“assign layers of the plurality of micro-batches as the layer pairs based on a total operation execution time.” (Paragraph [0158] of Lamy discloses assigning micro-batch reliant layer pairs, “The computation is done in the expected way, with a given instance processing all micro-batches for a given layer, then beginning the next assigned layer, the input for which should be ready and received from the instance assigned the immediately preceding layer, etc.”
Lamy does not appear to explicitly disclose”
“…based on a total operation execution time.”
However, Lee teaches:
“…based on a total operation execution time.” (Paragraph [0105] teaches that total operation execution time is taken into consideration when specified operations need to be performed on selected/assigned layer pairs, “In the example of FIG. 5, when a time for the operation on the first layer is greater than an execution time of the memory access resource for a subsequent layer (e.g., the second layer and the third layer)…”)
Therefore, Claim 8 is rejected.
With respect to Claims 13 (independent claim) and 21:
Lamy teaches:
“dividing batches of data into a plurality of micro-batches;” (Paragraph [0164] teaches this limitation verbatim, “The example embodiment of FIG. 9 additionally includes dividing training data for the machine learning model into a plurality of micro-batches (912).”)
“forming layer pairs from layers of the plurality of micro-batches;” (Paragraph [0164] discloses performance of operations as a result of predetermined layer pairs already being formed relative to a plurality of micro-batches, “…of the micro-batches through the machine learning model, wherein each of the compute nodes operates in parallel to generate respective error states for the micro-batches with each of its assigned layers, wherein the respective error states are communicated between the compute nodes in accordance with a reversal of the sequential ordering of the layers, and wherein each of the compute nodes completes the backward-propagation of all of the micro-batches through a given layer prior to performing backward-propagation through any layer that precedes the given layer in the sequential ordering (916).”
“generating an operation processing order of the layer pairs;” (Paragraph [0151] of Lamy exhibits the pre-existing compliance with an operation processing order (sequential ordering) of the layer pairs, “In the "Non-Layered Gradient Accumulation" pipeline parallelism depicted in FIG. 6, layers of the model were split into contiguous chunks. The micro-batches were then streamed through the pipeline in the forward pass, then streamed in the reverse order during the backward pass.”)
Lamy does not appear to explicitly disclose:
“determining resource information to be allocated to the layer pairs;”
“and allocating resources to the layer pairs based on the resource information.”
However, Lee teaches:
“determining resource information to be allocated to the layer pairs;” (Paragraph [0083] teaches that resource information that is allocated to the layers varies/requires determination, “…models and/or layers included in each of the models may have different workload characteristics, and thus the operation resource and the memory access resource that would be used for each model or layer may differ for each model or layer. Thus, by performing scheduling based on the workload characteristics, to increase or maximally overlap times for which the memory and computation/operation resources in the accelerator 210 are used and to reduce or minimize idle times, various examples may improve an overall system performance.”)
“and allocating resources to the layer pairs based on the resource information.” (Paragraph [0114] teaches allocating resources to the layer pairs based on resource information/conditions, “When the memory access operation is terminated, the layer L11 is allocated to an operation resource and a computation operation associated with the layer L11 is performed, and a first layer L21 of the second model is allocated to the memory access resource and a memory access operation associated with the layer L21 is performed.” Paragraph [0115] further teaches resource allocation dependent on the circumstance, “In this example, the layer of the first model to be allocated to the operation resource may have a workload characteristic different from that of the subsequent layer of the first model or the layer of the second model to be allocated to the memory access resource.”)
*Lee also teaches:
“forming layer pairs from layers…” (Paragraph [0112] discloses forming layer pairs/combinations, “For example, in a case in which the size of a window corresponds to three layers, candidate layers may be layers 1, 2, and 3 of the model A and layers 1, 2, and 3 of the model B.”)
Therefore, Claims 13 and 21 are rejected.
With respect to Claim 14:
The combination of Lamy and Lee teaches:
“wherein the dividing of the batches into the plurality of micro-batches comprises dividing the batches into the plurality of micro-batches to minimize a total operation execution time.” (Paragraph [0164] of Lamy teaches dividing the batches of training data into micro-batches, “The example embodiment of FIG. 9 additionally includes dividing training data for the machine learning model into a plurality of micro-batches (912).” Paragraph [0122] of Lamy discloses the intentional reduction of total operation execution time, “implement a variety of parallel training configurations and strategies for large and dense transformers or other large, multi-layered machine learning model types, with the goal of reducing training time…” Paragraph [0177] further describes specific scenarios in which total operation execution time is calculated and undergoes optimization in order to be minimized, “The resulting configurations are summarized in Table 1, together with the expected computational efficiency and approximate predicted training time. We found both data and tensor parallelism to be necessary (and together sufficient) to train in a reasonable amount of time. Modular pipeline parallelism stood out with both a high GPU count and near-optimal efficiency, allowing the model to be trained in a week. It also out-performed the baseline in the absence of tensor parallelism, but in that case the training time remained above three months.”)
Therefore, Claim 14 is rejected.
With respect to Claim 15:
The combination of Lamy and Lee teaches:
“wherein the forming of the layer pairs comprises assigning layers from the plurality of micro-batches into the layer pairs to minimize an idle time.” (Paragraph [0158] of Lamy discloses assigning micro-batch reliant layer pairs, “The computation is done in the expected way, with a given instance processing all micro-batches for a given layer, then beginning the next assigned layer, the input for which should be ready and received from the instance assigned the immediately preceding layer, etc.”)
Lamy does not appear to explicitly disclose:
“…to minimize an idle time.”
However, Lee teaches:
“…to minimize an idle time.” (Paragraph [0091] discloses assigning/scheduling layer pairs to minimize idle time, “The scheduler may be called each time execution of a scheduled layer is completed in the accelerator device 320 and the scheduler may perform scheduling for a layer of that model or another model that minimizes an idle time of the accelerator device 320 at a corresponding time.”)
Therefore, Claim 15 is rejected.
With respect to Claim 20:
The combination of Lamy and Lee teaches:
“performing training of a transformer or an inference operation of a trained transformer, using the allocated resources.” (Paragraph [0122] of Lamy discloses training a transformer model using allocated resources, “implement a variety of parallel training configurations and strategies for large and dense transformers or other large, multi-layered machine learning model types, with the goal of reducing training time when using existing commercially-available hardware.” Paragraph [0136] further discloses the specific operational capacities of these allocated resources, “The threshold for a compute-bound operation is described through the concept of arithmetic intensity. For an operation which requires both computation and data transfer, the arithmetic intensity is defined as the ratio between the amount of computation and data transfer. In the case of perfect overlap, the operation is compute-bound as long as the arithmetic intensity is higher than what the hardware can support.”)
Lee also teaches:
“…or an inference operation of a trained transformer, using the allocated resources.” (Paragraph [0096] teaches performing an inference operation of a trained transformer, (executed scheduling of a model layer) using allocated resources (accelerator), “The accelerator device 320 may then execute a layer according to the accelerator instruction and return an inference result of a model (or layer of the model) for which the layer execution is completed to the host device 310.”)
Therefore, Claim 20 is rejected.
Claims 9 and 16 are rejected under 35 U.S.C. 103 as being unpatentable over Lamy-Poirier, (U.S Patent Application Publication No. US 20220383084 A1, filed on February 9, 2022, hereinafter "Lamy"), in view of Lee et. Al, (U.S Patent Application Publication No. US 20220114015 A1, filed on March 9, 2021, hereinafter "Lee"), further in view of Kumar Jha et. Al, (Data-type Aware Arithmetic Intensity for Deep Neural Networks, published November, 2019, hereinafter "Kumar"). All three of these dates are before the effective filing date of this application, i.e., August 16, 2023 as well as the foreign priority filing date of October 19, 2022 where it applies.
With respect to Claims 9 and 16:
The combination of Lamy and Lee does not appear to explicitly disclose:
“respectively classify layers of each of the layer pairs into a corresponding layer type, among predefined layer types, according to an operation per byte ratio of an operation performed on each of the layers; and determine resource information to be allocated to the layer types where each layer belongs to a separate type of layer.”
However, Kumar teaches:
“respectively classify layers of each of the layer pairs into a corresponding layer type, among predefined layer types, according to an operation per byte ratio of an operation performed on each of the layers; and determine resource information to be allocated to the layer types where each layer belongs to a separate type of layer.” ([Page 2, Col. 1] discloses operations being performed on layers that have been classified into predefined layer types, “We first explore the intricacies of data reuse patterns in DNNs (§II) and perform a detailed analysis of layer wise data reuse patterns in DNNs (§III). We include all the possible variations of data reuse patterns arising from (A) different types of convolutions such as standard, group, pointwise, depth wise, etc., (B) different types of layers such as Conv, FC and others, and (C) different design heuristics such as feed-forward/skip connections. Through our comprehensive experiment and analysis, we show that data reuse estimated by arithmetic intensity is not tightly-coupled with the energy efficiency of MAC operations in DNNs.” [Page 3, Col. 2] further discloses that the operations encompass a per byte ratio basis, “Figure 2 shows arithmetic intensity of a layer (Conv and FC) defined as the ratio of “number of MACs performed in that layer” to “the sum of total number of weights and activations in that layer”. [Page 7, Col. 1] further discloses that resource information to be allocated to the layer types varies and is dependent on the layer type, “Our model estimates the data reuse available in DNNs with the assumption that underlying platforms have sufficient compute/memory resources to exploit the available data reuse in DNNs. However, different hardware platforms are optimized for contrasting design goals and have dissimilar memory hierarchy with non-identical number of layers and capacity. We now discuss whether our model is applicable to hardware platforms such as CPU, FPGA, etc., or do we need to re calibrate the value of α on them?”)
Lamy, Lee, and Kumar are analogous art and in the same field of invention because all three references pertain to computing machinery-level optimization, distributed processing, and energy efficiency required to train and run large transformer models and Deep Neural Networks (DNNs). While Lamy teaches splitting training data into micro-batches to solve memory and computational bottlenecks, Lee teaches monitoring accelerator pipeline “bubbles” to maximize hardware utilization and ensure high availability. Similarly, Kumar describes the mathematical frameworks used to estimate energy consumption and operational efficiency during the designing phase of a transformer model. It would have been obvious to a person having ordinary skill in the art (PHOSITA) before the effective filing date of the claimed invention to implement the base reference of Lamy (sequentially ordered forward/backward propagation of micro-batched data) with the teachings of Lee (sustainable inference-phase scheduling) and the teachings of Kumar (precision and data-type weighted arithmetic intensity based estimation of energy consumption) in order to train complex models faster by reducing latency, overcoming memory limits, preventing processor starvation, and quantizing architectures to reduce energy-draining data movement. One of ordinary skill in the art would be motivated to do so because by integrating Lee and Kumar's frameworks into the methods of Lamy one would be able to recognize that, "by performing scheduling based on the workload characteristics, to increase or maximally overlap times for which the memory and computation/operation resources in the accelerator 210 are used and to reduce or minimize idle times, various examples may improve an overall system performance, {[0083] of Lee}."
Therefore, Claims 9 and 16 are rejected.
Claim 22 is rejected under 35 U.S.C. 103 as being unpatentable over Lamy-Poirier, (U.S Patent Application Publication No. US 20220383084 A1, filed on February 9, 2022, hereinafter "Lamy"), in view of Huang et. Al, (GPipe: Easy Scaling with Micro-Batch Pipeline Parallelism, published July, 2019, hereinafter "Huang"). Both of these dates are before the effective filing date of this application, i.e., August 16, 2023 as well as the foreign priority filing date of October 19, 2022 where it applies.
With respect to independent Claim 22:
Lamy teaches:
“identify a plurality of batches of input data into a plurality of micro-batches, wherein each micro-batch of the plurality of micro-batches has no dependency to other micro-batches of the plurality of micro-batches;” (Paragraph [0129] discloses that input data is divided into a plurality of micro-batches that do not depend on one another, “In data parallelism, a batch of training samples is split between the devices, each independently processing a single micro-batch…” Paragraph [0091] further discloses no dependency, “In an example set of models, there may be no data dependency between the models…”)
Lamy does not appear to explicitly disclose:
“and assign a layer pair to each micro-batch of the plurality of micro-batch according to a resource consumption indicator, dependent on an analysis of layers of micro-batches for the consumption indicator, and a layer type of each layer of the layer pair.”
However, Huang teaches:
“and assign a layer pair to each micro-batch of the plurality of micro-batch according to a resource consumption indicator, dependent on an analysis of layers of micro-batches for the consumption indicator, and a layer type of each layer of the layer pair.” ([Pg. 3 Sec. 2.1] discloses the composition of the GPipe interface that encompasses tagging layers to specified micro-batches that are in collaboration with a resource consumption indicator (computation cost estimation function) as well as classification types (definitions) of layers in the model, “Any deep neural network can be defined as a sequence of L layers. Each layer Li is composed of a forward computation function fi, and a corresponding set of parameters wi. GPipe additionally allows the user to specify an optional computation cost estimation function, ci…The GPipe interface is extremely simple and intuitive, requiring the user to specify: (i) the number of model partitions K, (ii) the number of micro-batches M, and (iii) the sequence and definitions of L layers that define the model.” [Pg. 3 Sec. 2.2] further discloses the importance of the resource consumption indicator in determining the layered structure of the model, “The partitioning algorithm minimizes the variance in the estimated costs of all cells in order to maximize the efficiency of the pipeline by syncing the computation time across all partitions.” [Pg. 6 Sec. 5] further depicts varying layer types (L encoder layers and L decoder layers) that can be altered, manipulated, and scaled depending on performance discrepancies (such as computational cost) and testing, “Our comparison is based on the performance of a single Transformer [15] trained on all language pairs in this corpus. We scale the architecture along two dimensions to stress the flexibility of GPipe: (i) along the depth by increasing the number of layers in the model and (ii) along the width by increasing the hidden dimension in the feed-forward layers and the number of attention heads (as well as # attention channels) in multi-head attention layers.”)
Lamy and Huang are analogous art and in the same field of invention because both references focus on scaling and optimizing the computational execution of complex, multi-layered neural network architectures (like transformers). While Lamy teaches partitioning to enable the parallel processing of sequential data while maintaining the exact forward and backward dependencies required to update the model correctly, Huang teaches maintaining gradient consistency so that training outcomes and model quality remain identical regardless of how many accelerators are used. It would have been obvious to a person having ordinary skill in the art (PHOSITA) before the effective filing date of the claimed invention to implement the base reference of Lamy (solving memory constraints and hardware bottlenecks) with the teachings of Huang (reducing idle times “bubbles” and drastically lowering memory footprints) in order to overcome the single-accelerator memory limits that traditionally restrict deep learning model sizes, allowing developers to scale neural networks to gigantic capacities as well as providing a task-independent, highly efficient pipelining infrastructure that achieves almost linear speedup without requiring architecture-specific modifications. One of ordinary skill in the art would be motivated to do so because by integrating Lamy's framework into the methods of Huang one would be able to develop a machine learning model that encompasses, "Efficiency: Using a novel batch-splitting pipelining algorithm, GPipe achieves almost linear speedup with the number of devices. 2) Flexibility: GPipe supports any deep network that can be represented as a sequence of layers. 3) Reliability: GPipe utilizes synchronous gradient descent and guarantees consistent training regardless of the number of partitions, {[Pg. 9, Sec. 7] of Huang}."
Therefore, Claim 22 is rejected.
Claims 23 is rejected under 35 U.S.C. 103 as being unpatentable over Lamy-Poirier, (U.S Patent Application Publication No. US 20220383084 A1, filed on February 9, 2022, hereinafter "Lamy"), in view of Huang et. Al, (GPipe: Easy Scaling with Micro-Batch Pipeline Parallelism, published July, 2019, hereinafter "Huang"), further in view of Lee et. Al, (U.S Patent Application Publication No. US 20220114015 A1, filed on March 9, 2021, hereinafter "Lee"). All three of these dates are before the effective filing date of this application, i.e., August 16, 2023 as well as the foreign priority filing date of October 19, 2022 where it applies.
With respect to Claim 23:
The combination of Lamy and Huang does not appear to explicitly disclose:
“allocate resources to the layer pair based on the resource information to be allocated to a plurality of layer pairs, dependent on a processing order of the layer pairs.”
However, Lee teaches:
“allocate resources to the layer pair based on the resource information to be allocated to a plurality of layer pairs, dependent on a processing order of the layer pairs.” (Paragraph [0085] discloses that the processing order is relevant to the overall resource allocation and eventual performance of the model layers, “The data dependency may indicate a computation order of sets of data intended by a design or a compiler to obtain a desired result, and a plurality of layers included in one model may be sequentially processed in a preset order.” Paragraph [0092] further discloses obtaining relevant/specified resource information that influences the allocation of resources to the layers upon carrying out the scheduling operation, “In an example, a subsequent accelerator state may be tracked and recorded, and the scheduler may perform scheduling based on the accelerator state. Such as discussed above with respect to the memory and operation resources, an accelerator state described herein may include at least one of usage information of a memory included in an accelerator a (e.g., an entire capacity, a used capacity, and/or a remaining capacity of an on-chip memory in MB units of measure), a difference between a point in time at which an operation resource of the accelerator is most recently used and a point in time at which a memory access resource of the accelerator starts being used (e.g., in cycles unit of measure), or a state of a progression of each of models (e.g., represented by an n - th layer, considering the presence of data dependency among layers included in a same model).” Paragraph [0114] further showcases resource allocation, “When the memory access operation is terminated, the layer L11 is allocated to an operation resource and a computation operation associated with the layer L11 is performed.”)
Lamy, Huang, and Lee are analogous art and in the same field of invention because all three references pertain to tuning layer-to-hardware mapping to scale computational execution, leveraging balanced stage-aware scheduling and tensor parallelism to optimize multi-layered architectures. While Lamy teaches strategically partitioning model layers across compute nodes to prevent resource bottlenecks, Huang teaches overcoming the trainability issues associated with deeper models as well as enhancing transfer learning capabilities. Similarly, Lee describes maximizing accelerator efficiency by overlapping compute and communication phases. It would have been obvious to a person having ordinary skill in the art (PHOSITA) before the effective filing date of the claimed invention to implement the base reference of Lamy (sequentially ordered forward/backward propagation of micro-batched data) with the teachings of Huang (increasing model depth by improving generalization) and the teachings of Lee (sustainable inference-phase scheduling) in order to maximize resource utilization by keeping processors constantly engaged, preventing memory overloads, and drastically accelerating the training of complex transformer architectures. One of ordinary skill in the art would be motivated to do so because by integrating Huang and Lee's frameworks into the methods of Lamy one would be able to, "improve training efficiency while also reducing memory usage and network requirements…,{[0123] of Lamy}."
Therefore, Claim 23 is rejected.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to NOOR F CHEEMA whose telephone number is (571)272-9642. The examiner can normally be reached Monday-Friday 7:30am-5:00pm alternative Fridays off.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Mariela Reyes can be reached at (571) 270-1006. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/N.F.C./Examiner, Art Unit 2142
/HAIMEI JIANG/Primary Examiner, Art Unit 2142
Prosecution Insights