DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
The text of those sections of Title 35, U.S. Code not included in this action can be found in a prior Office action.
Response to Amendments and Arguments
Regarding an outstanding rejection under 35 U.S.C. §103, applicant amended independent claim 1 by adding new limitations. Applicant argued (Remarks, pages 9-11) that previously cited references (Ouyang in view of Maor) fail to teach the newly added limitations.
After performing an update search, the examiner discovered a reference to Bakhov et al. (US PG Pub. 2022/0188280). Bakhov discloses monitoring data obtained from different data sources (Bakhov, [0007-0010], [0066-00680], heterogeneous sources with wide variety of types and formats). Bakhov further discloses transforming the obtained data from different data sources with variety formats into a common internal format (Bakhov, [0066-0068], transformed data from different data sources into a universal format or a common internal format). All transformed data are saved as total samples (Bakhov, Abstract, [0012], [0067-0068]). Bakhov further shows different data types (Bakhov, [0078], [0091], Fig. 4).
Bahkov discloses the system receives data from one heterogeneous source and converts the data into a common internal format (Bahkov, [0066-0068], the common internal format corresponds to a claimed “a first format”). This converted dataset corresponds to a claimed “a first labeled dataset having a first format”. When the system further receives another dataset from a different data source with a different format, this dataset corresponds to a claimed “a second labelled dataset having a second format”. The total samples in a common internal format correspond to a claimed “a unified fine-tuning dataset” and “aggregating …”. Bahkov meets the newly added / argued limitations.
The examiner further notices that the cited Ouyang references disclose using multiple datasets from a user feedbacks for fine-tuning GPT-3 large language models (Ouyang, Abstract, pages 12-14, using various publicly available datasets). Since Ouyang discloses training GPT-3 model is performed by a computer, the datasets must be in “binary format”. Please notes, all data stored in a computer are in a binary format.
The secondary reference to Maor discloses creating a new dataset by combining a primary dataset with a secondary dataset for training a machine learning model (Maor, [0040-0046], Fig. 2). Since all data are processed using a computer, all data are in a binary format.
In the following section, the examiner combines the newly discovered Bakhov reference to reject the amended claims. Applicant’s arguments are considered but are moot because the arguments do not apply the rejection over the combined teaching necessitated by the amendments.
Claim Rejections - 35 USC § 103
Claims 1-9 are rejected under 35 U.S.C. §103 as being unpatentable over Ouyang et al. (“Training language models to follow instructions with human feedback”, published in 2022, a reference submitted by the applicant in an IDS, referred to as Ouyang) in view of Maor et al. (US PG Pub. 2017/0017899, referred to as Maor) and further in view of Bakhov et al. (US PG Pub. 2022/0188280, referred to as Bakhov)
Ouyang discloses fine-tuning a pre-trained GPT-3 large language model (LLM) to generate a fine-turned large language model (named as “InstructGPT”) based on collected various training datasets containing human feedbacks (Ouyang, Abstract, section 3.2, using three different datasets; section 3.3, a dataset by our labelers and a dataset submitted by GPT users).
Maor discloses combining two training datasets, i.e., a primary dataset and a secondary dataset, to create an enhanced training dataset for training a machine learning model (Maor, Abstract, [0005], [0042-0043] Fig. 2). Maor further discloses processing the primary dataset into a format suitable for training a machine learning model. The secondary dataset is also processed to a format for training the machine learning model (Maor, [0077-0078], [0088-0089]).
Bakhov discloses transforming data obtained from different data sources with variety formats into a common internal format (Bakhov, [0066-0068], transforming data into a universal format / a common internal format). All transformed data are saved as total samples (Bakhov, Abstract, [0068]). Bakhov further shows different data types (Bakhov, [0078], [0091], Fig. 4).
Regarding claim 1, Ouyang discloses a non-transitory computer-readable medium storing executable instructions, which when executed by a processing device, cause the processing device to perform operations (Ouyang, Section 3.6, evaluation experiments for fine-turning a large language model, a GPT-3 model, to generate a fine-tuned InstructGPT model, the training was performed using computers), comprising:
accessing a first labeled dataset having a first format of human feedback associated with performance of a pre-trained language model (Ouyang, section 3.2, pages 2-7, obtaining a dataset labeled by our contractors from human feedback. For example, a SFT dataset is one of the three different datasets, a labelled SFT dataset could be a claimed “a first labeled dataset”, see three different datasets in Table 6), wherein the first format of human feedback comprises ONE of a binary format, a numerical format, OR a multi-axis numerical format (Ouyang, section 3, Ouyang discloses using a computer to fine-tune GPT-3 model, all human received feedback data are in “a binary format”);
accessing a second labeled dataset having a second format of human feedback associated with performance of the pre-trained large language model (Ouyang, sections 3.2, accessing a RM dataset, “a second labelled dataset”, to further fine-turn the LLM using rewarding algorithm, see three datasets Table 6), wherein the second format of human feedback comprises one of the binary format, the numerical format, or the multi-axis numerical format that is different from the first format (Ouyang, section 3, Ouyang discloses using a computer to fine-tune GPT-3 model, all human received feedback data are in “a binary format”);
fine-tuning the pre-trained language model using datasets (Ouyang, Abstract, Section 3.5, starting with pre-trained GPT-3 mode, using supervised fine-tune technique, then applying reward modeling; also see page 14, using public NLP datasets to fine-tune our model); and
outputting the fine-tuned language model (Ouyang, Fig. 2, Section 4.1, providing and evaluating a fine-tuned large language model: “InstructGPT”).
Ouyang discloses fine-tuning a pre-trained GPT-3 large language models using many labelled training datasets (Ouyang, sections 3.2 and 3.4). Although Ouyang discloses using different datasets for fine-tuning the pre-trained GPT-3 model, Ouyang does not explicitly disclose: “generating a unified fine-tuning dataset by converting the second labeled dataset to a refined labeled dataset having the first format of human feedback and aggregating the first labeled dataset having the first format of human feedback with the refined labeled dataset having the first format of human feedback;”
Maor discloses combining two training datasets to create an enhanced training dataset for training a machine learning model (Maor, Abstract, [0002], [0005], [0042-0043] Fig. 2). Maor further discloses processing the primary dataset into a format suitable for training a machine learning model. The secondary dataset is also processed and in a format for training (Maor, [0077-0078], [0088-0089]). Maor further discloses processing the secondary data set by selecting and linking statistically significant data fields to the primary dataset to form an enhanced dataset (Maor, [0038-0042]).
Ouyang does not explicitly discloses “converting the second labeled dataset into a refined labeled dataset” and “aggregating the first labeled dataset…”.). Bakhov discloses transforming data obtained from different data sources into a common internal format to format total samples (Bakhov, [0066-00680]). All transformed data are saved as a total sample (Bakhov, Abstract, [0068], the total sample corresponds to a claimed “aggregating the first labeled dataset”).
Ouyang and Maor are related to training machine learning models using different datasets. Bakhov is related to using machine learning model to analyze data from heterogeneous data sources with variety types and formats. It would have been obvious to a person having ordinary skill in the art at the time the invention was filed to modify Ouyang’s teaching with Maor and Bakhov teaching to generate an enhanced dataset by combining the primary dataset and the secondary dataset by converting data from different data sources into a common format. One having ordinary skill in the art would have been motivated to make such a modification to improve the performance of training a machine learning model (Maor, [0045], [0062]). One having ordinary skill in the art would have been motivated to make such a modification to solve a problem that data from heterogeneous data sources having variety data types / formats (Bakhov, [0066-0068]). In addition, all the claimed elements were known in the prior art and one skilled in the art could have combined the elements as claimed by known methods, and in the combination each element merely would have performed the same function as it did separately. “A combination of familiar elements according to known methods is likely to be obvious when it does no more than yield predictable results.” KSR, 550 U.S. ___, 82 USPQ2d at 1395 (2007). One of ordinary skill in the art would have recognized that the results of the combination were predictable.
Regarding claim 2, the combined teaching of Ouyang in view of Maor and Bakhov further discloses: the first format of human feedback comprises a binary format, and the second format of human feedback comprises the numerical format (Ouyang, Table 3; Maor, [0002-0003], training data sets can have different formats; Bakhov, [0091], Fig. 4).
Regarding claim 3, the combined teaching of Ouyang in view of Maor and Bakhov further discloses: obtaining the first labeled dataset, wherein the human feedback in the first format is provided by a feedback provider indicating interest or preference associated with corresponding responses (Ouyang, section 1, page 2, fine-tuning a LLM using labeled datasets with human feedback indicating human preferences, such as less harmful).
Regarding claim 4, the combined teaching of Ouyang in view of Maor and Bakhov further discloses: the first labeled dataset comprises a set of prompts, a set of responses corresponding with the prompts, and a set of feedback indicating the performance of the pre-trained language model in relation to the set of responses (Ouyang, Abstract, section 1, fine-turning pre-trained GPT-3 model using datasets from human feedback to minimize performance regressions).
Regarding claim 5, the combined teaching of Ouyang in view of Maor and Bakhov further discloses: the pre-trained language model comprises a large language model (Ouyang, Abstract, fine-turning a pre-trained GPT-3 large language model, which has 175 billion parameters).
Regarding claim 6, the combined teaching of Ouyang in view of Maor and Bakhov further discloses: the first labeled dataset is obtained from a first data source, and the second labeled dataset is obtained from a second data source (Ouyang, section 3.2, our dataset and dataset submitted by customers).
Regarding claim 7, the combined teaching of Ouyang in view of Maor and Bakhov further discloses: the first format of human feedback comprises a binary format indicating a preference of a first response compared to a second response associated with a prompt, and wherein the second labeled dataset is converted to the binary format (Ouyang, Table 3, fig. 24, page 45 and page 49, binary multiple-choice questions).
Regarding claim 8, the combined teaching of Ouyang in view of Maor and Bakhov further discloses: the pre-trained language model is fine-tuned using supervised fine-tuning or reinforcement learning fine-tuning to align the pre-trained language model with human preferences (Ouyang, Abstract, section 3.5, fine-turning a pre-trained GPT-3 model by using a supervised fine-turning algorithm, then using enforcement learning).
Regarding claim 9, the combined teaching of Ouyang in view of Maor and Bakhov further discloses: using the fine-tuned language model to perform a task (Ouyang, Abstract, Section 3.3 and 4.2, evaluating performance of the fine-turned LLM, an InstructGPT model, for different natural language tasks).
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Jialong He, whose telephone number is (571) 270-5359. The examiner can normally be reached on Monday – Friday, 8:00AM – 4:30PM, EST.
If attempts to reach the examiner by telephone are unsuccessful, the examiner's supervisor, Pierre Desir can be reached on (571) 272-7799. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JIALONG HE/Primary Examiner, Art Unit 2659