DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
This office action is in response to submission of application on 11/17/2023.
Claims 1-20 are presented for examination.
Information Disclosure Statement
The information disclosure statement(s) submitted on 2/22/2024 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement(s) are considered by the examiner.
Drawings
The Drawings filed on 10/17/2023 are acceptable for examination purposes.
Specification
The Specification filed on 10/17/2023 is acceptable for examination purposes.
The disclosure is objected to because of the following informalities:
¶0021 labels the “model identification phase” as both 104 [Line 1] and 103 [Line 4]. This is assumed to be a typographical error as 103 does not appear in the drawings.
"Fine-tuning" is labeled as both 308 [¶0030 Line 4] and 310 [¶0031 Line 4]. Figure 3 labels 310 as "Fine-Tuning" and 308 as "Full or Fine-Tune"
Appropriate correction is required.
The disclosure is objected to because it contains an embedded hyperlink and/or other form of browser-executable code [¶0054, ¶0055, and ¶0060]. Applicant is required to delete the embedded hyperlink and/or other form of browser-executable code; references to websites should be limited to the top-level domain name without any prefix such as http:// or other browser-executable code. See MPEP § 608.01.
Claim Objections
Claims 6 and 16 are objected to because of the following informalities: Typographical error "fine tuning" . Appropriate correction is required.
Claims 14 - 20 are objected to because of the following informalities: Typographical error "The method of claim" should be "The non-transitory storage medium of claim". For the purposes of examination, claims 14-20 will be interpreted as dependent on the non-transitory storage medium introduced in claim 13. Appropriate correction is required.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 8-10, 17, and 18 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 8 and 17 recites the limitation "the number of graphical processing units" in lines 5 & 6. There is insufficient antecedent basis for this limitation in the claim.
Claims 9. 10, and 18 are rejected for being dependent on a rejected base claim.
Claim 10 recites the limitation "a recommended estimate" in line 1. This limitation is undefined in the claims and for the purpose of examination will be interpreted as the estimated cost calculated in claim 9.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-20 rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more. The analysis of the claims will follow the 2019 Revised Patent Subject Matter Eligibility Guidelines (“2019 PEG”).
Step 1: Is the claim directed to a process, machine, manufacture or composition of matter?
Claims 1-12 recite a process (a method), and claims 13-20 recite a manufacture (non-transitory storage medium). Therefore, claims 1-20 are directed to one of the four statutory categories of invention, i.e., process, machine, manufacture, or composition of matter.
Claim 1
Step 2A, Prong 1: The claim recites, inter alia:
performing a model identification phase by identifying a model to be trained using the dataset; - Model identification based on a dataset is a mental process: observation, evaluation, judgement, something that can be done with the human mind, or with the aid of pen and paper (MPEP 2106.04(a)(2)(III));
estimating at least computing resources and time required to train the identified model using the dataset; - Estimating resources is a mental process: observation, evaluation, judgement, something that can be done with the human mind, or with the aid of pen and paper (MPEP 2106.04(a)(2)(III));
Step 2A, Prong 2: This judicial exception is not integrated into a practical application. The additional elements of the claim are as follows:
receiving a dataset in preparation for performing a training operation; - Receiving a dataset represents insignificant extra-solution activity of data gathering. (MPEP 2106.05(g))
training the identified model with the dataset. - Training a model is mere recitation that a judicial exception is to be performed using generic computer equipment running general class of computer algorithms in their ordinary capacity. (MPEP 2106.05(f))
Step 2B: The additional elements, taken alone or in combination, do not represent significantly more than the abstract idea itself. As discussed above with respect to integration of the abstract idea into a practical application, the additional elements do not amount to significantly more than the abstract idea itself. The steps of collecting information, storing, electronically sending requests, electronically receiving feedback, and presenting are routine data gathering and output steps are well-understood, routine and conventional activities recognized by the Courts (MPEP 2106.05(d)(II)). Using a generic computer and/or general class of computer algorithms with well-understood, routine, conventional activities to implement the abstract idea does not amount to significantly more than the abstract idea itself.
Claim 2
Step 2A, Prong 1: The claim recites inter alia:
performing a data exploration phase to prepare the dataset for training the identified model, wherein the data exploration phase includes cleaning the dataset and/or augmenting the dataset. - Cleaning and/or augmenting a dataset is a mental process: observation, evaluation, judgement, something that can be done with the human mind, or with the aid of pen and paper (MPEP 2106.04(a)(2)(III));
Step 2A, Prong 2 and Step 2B: There are no additional elements recited, as such the claim does not provide a practical application and is not considered to be significantly more.
Claim 3
Step 2A, Prong 1: No additional abstract idea limitations.
Step 2A, Prong 2: This judicial exception is not integrated into a practical application. The additional elements of the claim are as follows:
receiving the dataset from a user and receiving a use case from the user... – Receiving a dataset or use case represents the insignificant extra-solution activity of data gathering. (MPEP 2106.05(g))
Step 2B: There are no additional elements recited, as such the claim does not provide a practical application and is not considered to be significantly more.
Claim 4
Step 2A, Prong 1: No additional abstract idea limitations.
Step 2A, Prong 2: This judicial exception is not integrated into a practical application. The additional elements of the claim are as follows:
the identified model is associated with model metadata and model weights, wherein the model metadata includes at least one of a number of hidden layers, a historical batch size, model size, recommended task, and a tokenizer associated with the model. - Associating additional data and defining the characteristics of a model as such represents insignificant extra-solution activity of data gathering. (MPEP 2106.05(g)).
Corresponding analysis of the parent claim maintained here.
Step 2B: Corresponding analysis of the parent claim maintained here.
Claim 5
Step 2A, Prong 1: No additional abstract idea limitations.
Step 2A, Prong 2: This judicial exception is not integrated into a practical application. The additional elements of the claim are as follows:
tokenizing the dataset using the tokenizer. - Tokenizing a dataset is mere recitation that a judicial exception is to be performed using generic computer equipment running general class of computer algorithms in their ordinary capacity. (MPEP 2106.05(f))
Corresponding analysis of the parent claim maintained here.
Step 2B: Corresponding analysis of the parent claim maintained here.
Claim 6
Step 2A, Prong 1: The claim recites inter alia:
Determining whether to perform a full fine tuning of the model or an optimized fine-tuning of the model - As described in the specification, this can be determined through user input or by identifying if the model is not pre-trained and is determined to be a mental process: observation, evaluation, judgement, something that can be done with the human mind, or with the aid of pen and paper (MPEP 2106.04(a)(2)(III));
Step 2A, Prong 2 and Step 2B: There are no additional elements recited, as such the claim does not provide a practical application and is not considered to be significantly more. Corresponding analysis of parent claim maintained here.
Claim 7
Step 2A, Prong 1: The claim recites inter alia:
Reducing a number of parameters of the identified model for training… - Parameter reduction is a mental process: observation, evaluation, judgement, something that can be done with the human mind, or with the aid of pen and paper (MPEP 2106.04(a)(2)(III));
Step 2A, Prong 2: This judicial exception is not integrated into a practical application. The additional elements of the claim are as follows:
...prior to training the model when performing an optimized fine-tuning. - training a model is mere recitation that a judicial exception is to be performed using generic computer equipment running general class of computer algorithms in their ordinary capacity. (MPEP 2106.05(f))
Corresponding analysis of parent claim maintained here.
Step 2B: The additional elements, taken alone or in combination, do not represent significantly more than the abstract idea itself. As discussed above with respect to integration of the abstract idea into a practical application, the additional elements do not amount to significantly more than the abstract idea itself.
Claim 8
Step 2A, Prong 1: The claim recites inter alia:
Estimating the time and computing resources based on a relationship of:
E
n
d
-
t
o
-
e
n
d
t
r
a
i
n
i
n
g
t
i
m
e
≈
8
T
P
n
X
- Calculating a relationship is a mathematical concept: mathematical relationships, mathematical formulas or equations, or mathematical calculations. (MPEP 2106.04(a)(2)(I));
Step 2A, Prong 2 and Step 2B: There are no additional elements recited, as such the claim does not provide a practical application and is not considered to be significantly more. Corresponding analysis of parent claim maintained here.
Claim 9
Step 2A, Prong 1: The claim recites inter alia:
Adjusting the number of graphical processing units to change the estimated time and/or computing resources… - Adjusting a variable within a formula is a mental process: observation, evaluation, judgement, something that can be done with the human mind, or with the aid of pen and paper (MPEP 2106.04(a)(2)(III));
…estimating a cost based on the time and/or the number of graphical processing units - Calculating an estimation is a mental process: observation, evaluation, judgement, something that can be done with the human mind, or with the aid of pen and paper (MPEP 2106.04(a)(2)(III));
Step 2A, Prong 2 and Step 2B: There are no additional elements recited, as such the claim does not provide a practical application and is not considered to be significantly more. Corresponding analysis of parent claim maintained here.
Claim 10
Step 2A, Prong 1: No additional abstract idea limitations.
Step 2A, Prong 2: This judicial exception is not integrated into a practical application. The additional elements of the claim are as follows:
instantiating a training instance once a recommended estimate is approved by a user. - This limitation is recited at a high level of generality. Mere recitation that a judicial exception is to be performed using generic computer equipment running general class of computer algorithms in their ordinary capacity. (MPEP 2106.05(f))
Corresponding analysis of the parent claim maintained here.
Step 2B: Corresponding analysis of the parent claim maintained here.
Claim 11
Step 2A, Prong 1: The claim recites inter alia:
Estimating the computing resources for the full fine-tuning based on model state, activation memory and model state working memory – Estimating resources is a mental process: observation, evaluation, judgement, something that can be done with the human mind, or with the aid of pen and paper (MPEP 2106.04(a)(2)(III));
Step 2A, Prong 2: This judicial exception is not integrated into a practical application. The additional elements of the claim are as follows:
...when training the model, wherein the model is not pre-trained. - Training a model is mere recitation that a judicial exception is to be performed using generic computer equipment running general class of computer algorithms in their ordinary capacity. (MPEP 2106.05(f))
Step 2B: The additional elements, taken alone or in combination, do not represent significantly more than the abstract idea itself. As discussed above with respect to integration of the abstract idea into a practical application, the additional elements do not amount to significantly more than the abstract idea itself.
Claim 12
Step 2A, Prong 1: The claim recites inter alia:
Automatically configuring parameters of the training operation without user input - Parameter configuration is a mental process: observation, evaluation, judgement, something that can be done with the human mind, or with the aid of pen and paper (MPEP 2106.04(a)(2)(III));
Step 2A, Prong 2 and Step 2B: There are no additional elements recited, as such the claim does not provide a practical application and is not considered to be significantly more. Corresponding analysis of parent claim maintained here.
Claim 13 is substantially similar in scope and spirit to claim 1. Therefore, it would be rejected under similar analysis.
Claim 14 is substantially similar in scope and spirit to claims 2 and 3. Therefore, it would be rejected under similar analysis.
Claim 15 is substantially similar in scope and spirit to claims 4 and 5. Therefore, it would be rejected under similar analysis.
Claim 16 is substantially similar in scope and spirit to claim 6. Therefore, it would be rejected under similar analysis.
Claim 17 is substantially similar in scope and spirit to claims 7 and 8. Therefore, it would be rejected under similar analysis.
Claim 18 is substantially similar in scope and spirit to claims 9 and 10. Therefore, it would be rejected under similar analysis.
Claim 19 is substantially similar in scope and spirit to claim 11. Therefore, it would be rejected under similar analysis.
Claim 20 is substantially similar in scope and spirit to claim 12. Therefore, it would be rejected under similar analysis.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claims 1 and 13 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Li et all (US Publication No US 2019/0325307), hereinafter Li.
Regarding Claim 1, Li discloses a method comprising: receiving a dataset in preparation for performing a training operation. [Item 220 Fig. 2, Step 630 Fig. 6, and [¶0063] Lines 6-10 "it may require the user to define the needed input dataset, such as the type and size of the input dataset”]; performing a model identification phase by identifying a model to be trained using the dataset [ [¶0007] Lines 3-5 “obtaining a structure of a deep neural network model of a user-defined deep learning application based on the deep neural network model” reasonably corresponds to a model identification phase with a dataset]; estimating at least computing resources and time required to train the identified model using the dataset [Abstract discloses resource configuration and training time being included in parameter dimensions of the training dataset used to create an estimation model for estimating resources utilized by deep learning applications, including training of large language models.]; and training the identified model with the dataset [ [¶0068] “the resource scheduler 630 my deploy the user-defined deep learning application to the allocated hardware resources”, reasonably corresponds to training a model using the dataset].
Claim 13 is substantially similar in scope and spirit to claim 1. Therefore, it would be rejected under similar analysis.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 2, 3, 12, 14, and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Li in view of Skerry-Ryan et al. (US Publication US 2024/0104394 A1) hereinafter Skerry-Ryan.
Regarding Claim 2, Li discloses the method of claim 1 as from above.
Li does not specifically teach performing a data exploration phase to prepare the dataset for training the identified model, wherein the data exploration phase includes cleaning the dataset and/or augmenting the dataset.
However, Skerry-Ryan in the same field of endeavor discloses the above limitation. [¶0034] “…performs automated feature engineering to generate a clean set of feature data from raw training data”. The paragraph teaches a cleaning and augmenting process as part of a machine learning pipeline that is reasonably comparable to the claimed limitation.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the automated feature engineering introduced by Skerry-Ryan to the method of training deep learning models of Li. The motivation is to enable [Skerry-Ryan: ¶0034] automatic semantic type detection for each feature included in the inputted data, [Skerry-Ryan: ¶0035] automatically label the features with the detected semantic type, [Skerry-Ryan: ¶0036] understand correlations between different features that may enable the generation of additional feature data, and [Skerry-Ryan: ¶0037] enabling the user the user to unlock additional levels of data insight, understanding and interpretability.
Regarding Claim 3¸ Li discloses the method of claim 1 as from above. Further, Li teaches receiving the dataset from a user, [[¶0063] lines 1-10 discloses receiving a user’s input including “... information about user-defined deep learning application… may require the user to define the needed input dataset”].
Li does not specifically teach receiving a user case from the user, wherein the model identification phase identifies the model based at least on the use case.
However, Skerry-Ryan in the same field of endeavor discloses the above limitation. [¶0043] “a user can supply a set of training data… can include and/or constitute a structured training dataset having data associated with a number of labels. The user can select one of the features as a label…, which may start the search for the best machine learning model… the user may also specify other ‘advanced’ settings from the US, such as… details of the ML task (e.g., corresponding to a problem statement).” Selecting a model based on a problem statement or featured label reasonably corresponds to a model selection based on use case. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Li’s method of cost estimation and model training with Skerry-Ryan’s method of model selection. Motivation to adopt this method is that there is a [Skerry-Ryan: ¶0023] “large value to entry-level users because many product teams, infrastructure teams, and analysts want to deploy machine learning for their use cases, but do not have the necessary infrastructure expertise to facilitate development of a full deployment machine learning pipeline.”
Regarding Claim 12, Li discloses the method of claim 1 as from above.
Li does not specifically teach automatically configuring parameters of the training operation without user input.
However, Skerry-Ryan in the same field of endeavor discloses the above limitation. [¶0032] “the meta-learning system for automated machine learning systems can include a machine-learned model that is trained to predict parameter or hyperparameter for the automated machine learning system to be applied with respect to generation of a model for a new dataset.” It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Li’s method of determining cost and training a deep learning model with the automated machine learning pipeline of Skerry-Ryan. Motivation to adopt this automated machine learning pipeline is that [Skerry-Ryan: ¶0025] model developers do not need to spend significant amounts of time and computational resources training and evaluating different model architecture.
Claim 14 is substantially similar in scope and spirit to claims 2 and 3. Therefore, it would be rejected under similar analysis.
Claim 20 is substantially similar in scope and spirit to claim 12. Therefore, it would be rejected under similar analysis.
Claims 4-6, 15, and 16 are rejected under 35 U.S.C. 103 as being unpatentable over Li in view of Skerry-Ryan, further in view of BERT: Pre-Training of Deep Bidirectional Transformers for Language Understanding to Devlin et al. hereinafter Devlin.
Regarding Claim 4, the combination of Li and Skerry-Ryan discloses the method of claim 3, wherein the identified model is associated with model metadata and model weights. [Li: ¶0005] “…the performance benchmark database includes at least structural data of one or more deep neural network models…” as interpreted above, the deep neural network models are reasonably comparable to an ‘identified model’, as such structural data of the deep neural network models is reasonably comparable to general model metadata.
The combination of Li and Skerry-Ryan does not disclose specifics of the model metadata includes at least one of a number of hidden layers, a historical batch size, model size, recommended task, and a tokenizer associated with the model.
However, Devlin in the same field of endeavor discloses a language representation model called BERT (Bidirectional Encoder Representations from Transformers) that is a well-known deep neural network model architecture. BERT’s model architecture includes a number of hidden layers, model sizes [Devlin: Section 3 Model Architecture] as well as a tokenizer, WordPiece [Devlin: Section 3 Input/Output Representations] for pre-training and fine-tuning tasks. It would have been obvious to one of ordinary skill in the art prior to the effective filing date to apply the known tokenization and preprocessing techniques of Devlin to the models within the deep-learning training framework disclosed by Li. Devlin writes that rich, unsupervised pre-training is an integral part of many language understanding systems. In particular, these results enable even low-resource tasks to benefit from deep unidirectional architectures [Devlin: Section 6 Conclusion].
Regarding Claim 5, the combination of Li, Skerry-Ryan, and Devlin teaches the method of claim 4, additionally Devlin teaches tokenizing the dataset using the tokenizer. By tokenizing textual input datasets using a WordPiece tokenizer as part of pre-training. [Devlin: 3. Input/Output Representations & A.2 Pre-Training Procedure]. The rationale to combine Li’s method of determining cost and training a deep learning model with the tokenization and preprocessing techniques of Devlin are the same as with the parent claim.
Regarding Claim 6, the combination of Li, Skerry-Ryan, and Devlin teaches the method of claim 5, additionally Li teaches determining whether to perform a full fine-tuning of the model or an optimized fine-tuning of the model. Through “estimating resources utilized by deep learning applications” [Li: Abstract] determining “different combinations of various different parameter dimension variables” [Li: ¶0059] and “allocating system available resources reasonably” [Li: ¶0031] based on the estimation model, Li’s methods are reasonably comparable to determining a fine-tuning configuration based on estimated resource utilization.
Claim 15 is substantially similar in scope and spirit to claims 4 and 5. Therefore, it would be rejected under similar analysis.
Claim 16 is substantially similar in scope and spirit to claim 6. Therefore, it would be rejected under similar analysis.
Claim 7 is rejected under 35 U.S.C. 103 as being unpatentable over Li in view of Skerry-Ryan and Devlin, further in view of LoRA: Low-Rank Adaptation of Large Language Models to Hu et al. hereinafter Hu.
The combination of Li, Skerry-Ryan, and Devlin teaches the method of Claim 6, as above. They do not teach reducing a number of parameters of the identified model for training prior to training the model when performing an optimized fine-tuning.
However, Hu in the same field of endeavor teaches a parameter-efficient method of fine tuning known as LoRA, Low-Rank Adaptation, which freezes model parameters in order to reduce the number of trainable parameters for the model. When compared to other methods, “LoRA can reduce the number of trainable parameters by 10,000 times and the GPU memory requirement by 3 times” [Hu: Abstract]. It would have been obvious to one of ordinary skill in the art prior to the effective filing date to apply the known optimization techniques of LoRA to the resource estimation framework disclosed by Li. Motivation to combine comes from the noted reduction of GPU memory as a result of reduced parameters, furthermore Hu states “The most significant benefit comes from the reduction in memory and storage usage” [Hu: 4.2 Practical Benefits and Limitations] and notes “LoRA can be combined with other efficient adaptation methods, potentially providing orthogonal improvement.” [Hu: 8. Conclusion and Future Work]
Claims 8, 9, and 17 are rejected under 35 U.S.C. 103 as being unpatentable over Li in view of Skerry-Ryan, Devlin, Hu, and further in view of Efficient Large-Scale Language model Training on GPU Clusters Using Megatron-LM to Narayanan et al. hereinafter Narayanan.
Regarding Claim 8, the combination of Li, Skerry-Ryan, Devlin, and Hu disclose the method of claim 7 including methods of estimating time and resources, as above.
What is not disclosed is estimating the time and resources based on a relationship of:
E
n
d
-
t
o
-
e
n
d
t
r
a
i
n
i
n
g
t
i
m
e
≈
8
T
P
n
X
Wherein T is the number of tokens, P is the number of parameters, n is the number of graphical processing units, and X is flops/seconds.
However, Narayanan in the same field of endeavor teaches mathematical throughput and scaling relationships in order to estimate time, resources, and end-to-end training time, and deriving the formula:
E
n
d
-
t
o
-
e
n
d
t
r
a
i
n
i
n
g
t
i
m
e
≈
8
T
P
n
X
[Narayanan: 5.1 End-to-End Performance, Formula 4]
It would have been obvious to one of ordinary skill in the art prior to the effective filing date to integrate Narayanan’s known training and scaling estimation techniques into Li’s resource estimation framework. Motivation to do in order to account for distributed-training scaling behavior when estimating training time and computational resource utilization [Narayanan: 5.1 End-to-End Performance: Training Time Estimates].
Regarding Claim 9, the combination of Li, Skerry-Ryan, Devlin, Hu, and Narayanan disclose the method of claim 8 as above, additionally Narayanan teaches adjusting the number of graphical processing units to change the estimated time and/or computing resources [Narayanan: Table 1, Table 2, and Figure 10 show scaling behavior and throughput under varying GPU configurations and training times]. Further, Li teaches estimating a cost based on the time and/or the number of graphical processing units. [Li: Abstract discloses resource configuration, reasonably comprising graphical processing units, and training time being used to create an estimation model for estimating resources utilized by deep learning applications, including training of large language models.] As stated above, the combination of Narayanan’s training and scaling estimation techniques and Li’s resource estimation framework would have been obvious to one of ordinary skill in the art prior to the effective filing date.
Claim 17 is substantially similar in scope and spirit to claims 7 and 8. Therefore, it would be rejected under similar analysis.
Claims 10 and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Li in view of Skerry-Ryan, Devlin, Hu, and Narayanan as applied to claims 1-9 and 13-17 above.
Regarding Claim 10, the combination of Li, Skerry-Ryan, Devlin, Hu, and Narayanan disclose the method of claim 9, further Li teaches instantiating a training instance once a recommended estimate is approved by a user. “The resource model network server issues a resource estimation response to the resource scheduler to notify the estimation result for the user’s input” [Li: ¶0067] and “the resource scheduler may allocate required computing resources to the user-defined deep learning application based on the estimation result” [Li: ¶0068].
Claim 18 is substantially similar in scope and spirit to claims 9 and 10. Therefore, it would be rejected under similar analysis.
Claims 11 and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Lin in view of Skerry-Ryan, Devlin, and Hu as applied to claims 1-7 and 13-16 above, and further in view of Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model to Smith et al. hereinafter Smith.
Regarding Claim 11, the combination of Li, Devlin, Hu, and Skerry-Ryan disclose the method of claim 6.
While the combination of Li, Devlin, Hu, and Skerry-Ryan teaches estimating the computing resources for full fine-tuning it is not disclosed that this training is based on model state, activation memory and model state working memory when training the model, wherein the model is not pre-trained.
However, Smith in the same field of endeavor teaches that memory consumed by model weights, gradients, optimizer states, activations, and working GPU memory are major contributors to the resource requirements of large-scale deep learning training operations [Smith: 2.1.1 Memory Efficiency]. Noting that achieving high compute efficiency as large scale is challenging. Smith further teaches optimization techniques such as partitioning and checkpointing for reducing resource consumption of these factors. It would have been obvious to one of ordinary skill in the art prior to the effective filing date to base Li’s computing-resource estimation for training a model, wherein the model is not pre-trained and will require the most resources, on the memory resource contributors identified by Smith. Motivation for this comes from Smith teaching that adjusting and reducing these memory categories improves the scalability of large-scale deep learning training operations [Smith: 2.1.2 and 2.2 memory efficiency notes].
Claim 19 is substantially similar in scope and spirit to claim 11. Therefore, it would be rejected under similar analysis.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Appel et al. US 20220188663 A1 Automated Machine Learning Model Selection.
Teaches automatic architectures and model selection based on resource constraints. Discloses methods of resource/cost prediction and automated selection and training.
Watson et al. US 20200320379 A1 Training Transfer-Focused Models for Deep Learning.
Teaches optimized and full fine-tuning of neural networks based on datasets, cost prediction and automatic training and tuning as well as user authorized methods.
Xie et al. US 20220197901 A1 AI Model Optimization Method and Apparatus.
Teaches model training using an estimator for resource consumption, model optimization and resource scheduling based off the estimation.
Yakovlev et al. US 11429895 B2 Predicting Machine Learning or Deep Learning Model Training Time.
Teaches methods of estimating time for training models, both full training and optimization based on dataset. Teaches cost prediction and model identification.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to GRANT F FITCH whose telephone number is (571)270-0621. The examiner can normally be reached Monday-Thursday 7-4.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Miranda Huang can be reached at (571) 270-7092. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
GRANT F. FITCH
Examiner
Art Unit 2124
/MIRANDA M HUANG/Supervisory Patent Examiner, Art Unit 2124
Prosecution Insights