DETAILED ACTION
This office action is in response to Applicant’s Request for Continued Examination (RCE), received on 04/16/2026. Claims 1, 2, 6, 9-20 have been amended. Claims 1-20 are pending and have been considered.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 04/16/2026 has been entered.
Response to Arguments
Applicant’s arguments, see pgs. 11-13, filed 04/16/2026, with respect to the rejection(s) of claim(s) 1 under 35 U.S.C. 103 (Chen in view of Lev) have been fully considered and are persuasive. Therefore, the rejection has been withdrawn. However, upon further consideration, a new ground(s) of rejection is made in view of Feng et al. (“GenAug: Data Augmentation for Finetuning Text Generators”), hereinafter Feng. Feng discloses “various augmentation methods, including some that incorporate external knowledge, for finetuning GPT-2 on a subset of Yelp Reviews. We also examine the relationship between the amount of augmentation and the quality of the generated text. We utilize several metrics that evaluate important aspects of the generated text including its diversity and fluency” (abstract). The diversified texts of Feng and their evaluations will be taken in view of the fine-tuned language model of Chen. Feng will be replacing Lev. See updated rejections below.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1, 4-5 is/are rejected under 35 U.S.C. 103 as being unpatentable over Chen et al. (US-20250077792-A1), hereinafter Chen, in view of Feng et al. (“GenAug: Data Augmentation for Finetuning Text Generators”), hereinafter Feng.
Regarding claim 1, Chen discloses: a computer-implemented method ([0129] implemented using computer software, hardware, or software and hardware) comprising:
generating, via a generative language model ([0040] the training data generator 156 may be a generative pre-trained transformer (GPT)), a set of natural language text snippets based on a corresponding set of data ([Fig. 1, Content Items 108, Input-Output Pairs 104/132], [0040] manual input-output pairs 104 and content items 160 are sampled such that the training component 152 trains the training data generator 156 to generate outputs associated with the content item 160, [0036] Content items 160 can include any digital content items such as job postings, comments, resumes, and articles [Generating training data based on received content items, i.e. input, wherein the content items will contain natural language, indicating the content items represent a set of data with corresponding generated natural language text snippets, i.e. the output of input-output pairs. The input-output pairs (plurality emphasized) indicates a set of outputs, i.e. snippets, associated with a set of content items, i.e. data]);
fine-tuning a language model into a fine-tuned language model using the set of natural language text snippets and the corresponding set of data as training data ([0033] fine-tuning manager 158 to fine-tune a pretrained machine learning model 120 to obtain the fine-tuned model 110 using the input-output pairs 132 determined, at least in part, by the training data generator 156, [0042] the pretrained machine learning model 120 is an LLM [Wherein the training data is representative of text snippets and corresponding training data as previously disclosed, see above “generating…” element]);
generating, via the generative language model ([In view of the previously disclosed generative language model of Chen]), a set of independent natural language text snippets based on the corresponding set of data ([Fig. 1, “Generate Training Data 114”], [Wherein this generation is performed by the training data generator, a generative language model, as previously disclosed, and the data is in the form of input-output pairs, tracking to text snippets, i.e. outputs, associated with data, i.e. inputs]).
Chen does not disclose:
the set of independent natural language text snippets corresponding to a natural language variation of the set of natural language text snippets;
generating an evaluation metric that evaluates the fine-tuned language model with respect to the natural language variation based on applying the set of independent natural language text snippets to the fine-tuned language model to generate output and
evaluating the output of the fine-tuned language model with respect to the corresponding set of data used to generate both the set of independent natural language text snippets and the set of natural language text snippets.
Feng discloses:
the set of independent natural language text snippets corresponding to a natural language variation of the set of natural language text snippets ([Table 1, Methods used for augmentation based on an original review text], [The examiner asserts that the operation shown in Table 1 of Feng could be extended to the multiple training datum of Chen without a change in functionality to the augmentation of Feng as Feng discloses multiple natural language text snippets, see Tables 6, 7, 12, 13, and Yelp review dataset size as disclosed in section 2.2]);
generating an evaluation metric that evaluates the fine-tuned language model with respect to the natural language variation based on applying the set of independent natural language text snippets to the fine-tuned language model to generate output ([pg. 31, Section 2.5] We evaluate generated continuations using various metrics assessing major aspects of the text including its diversity, fluency, semantic content preservation, and sentiment consistency, [pg. 32, Section 2.5.2] We finetune GPT-2 on a two-million review subset of YR (with a 500K additional validation split) and use this finetuned model for PPL evaluation., [The examiner asserts that diversity of responses tracks to variation of responses, wherein the generated continuations are independent natural language text snippets being evaluated via metric as would be performed within the model (SBLEU of 2.5.1). The SBLEU score tracks to an evaluation metric which evaluates the fine-tuned language model based on the output produced by said model. Further, Feng discloses using a fine-tuned model for metric evaluation relating to fluency. The examiner asserts that a change in fluency between a natural language snippet and an independent natural language snippet indicates a varied independent natural language snippet. This also further indicates a model could be used for the determining the diversity metric]) and
evaluating the output of the fine-tuned language model with respect to the corresponding set of data used to generate both the set of independent natural language text snippets and the set of natural language text snippets ([Fig, 1(a), SBLUE by Variation], [A comparison between natural language text snippets and varied, i.e. independent, natural language text snippets indicates a comparison between the two with respect to the previously disclosed sets of snippets, wherein the snippets are gathered from the set of data, indicating the evaluation to be with respect to the corresponding set of data from which the snippets are gathered]).
Chen and Feng are considered analogous art within generative model fine-tuning using augmented data. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Chen to incorporate the teachings of Feng, because of the novel way to augment existing data using a GPT model and a plurality of augmentation methods, improving the ability of generative models to generate meaningful responses for low-data regimes through determining an amount of augmented data for producing the highest quality generations (Feng, Abstract).
Regarding claim 4, Chen in view of Feng discloses: the computer-implemented method of claim 1.
Chen further discloses:
wherein generating the set of independent natural language text snippets further comprises:
receiving a prompt that is different than an initial prompt used to generate the set of natural language text snippets ([Fig. 1, Obtain Content Items 108], [0043] the pre-processing operation 151 transforms content items 160 into a prompt that the training data generator 156 can subsequently complete, [0043] a resume content item can be mapped to a prompt, an article can be mapped to a prompt, etc. [Defining a plurality of prompts in view of the plurality of content items indicates at least a second prompt that is different, i.e. independent, from an initial prompt, i.e. resume vs. article, used to generate natural language text snippets, i.e. training data]); and
generating each independent natural language text snippet of the set of independent natural language text snippets based on the prompt ([Fig. 6, 604], [0082] the fine-tuning manager 430 can fine-tune the pretrained machine learning model 408 using extra prompts [Fine-tuning using extra prompts indicates the extra prompt is a prompt different from an initial prompt, wherein the output 604 is clearly a natural language text snippet based on a resume (previously determined to be able to be mapped to a prompt, see [0043)]).
Regarding claim 5, Chen in view of Feng discloses: the computer-implemented method of claim 1.
Chen further discloses:
wherein each natural language text snippet of the set of natural language text snippets and each independent natural language text snippet of the set of independent natural language text snippets are natural language queries ([Fig. 1, 106, 170], [0014] The model input includes a task description, also referred to as a prompt. The task description can include instructions and/or examples of digital content. A task description can be in the form of natural language text, such as a question [In view of the previously disclosed two distinct sets of text snippet training data 104 and 136 to be retrieved separately for different purposes, indicating the data within these sets to be “natural language” and “independent natural language” text snippets, wherein they are all disclosed to be questions, i.e. queries]).
Claim(s) 2-3, 7-12, 14-17, 19-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Chen in view of Feng, further in view of Zha et al. (US-20240202458-A1), hereinafter Zha.
Regarding claim 2, Chen in view of Feng discloses: the computer-implemented method of claim 1.
Chen further discloses:
wherein the corresponding set of data comprises a corresponding set of documents ([0036] Content items 160 can include any digital content items such as job postings, comments, resumes, and articles [Pluralizing content items indicates a set of content items/documents]) and generating the set of natural language text snippets further comprises:
receiving a set of keywords and the corresponding set of documents ([Fig. 7, Content Item Data Store 754], [0115] The queries are designed to find information that matches specified criteria, such as keywords and phrases. For example, search engine 744 is used to retrieve data by executing queries on various data stores of data storage system 740 [Connecting a search engine 744 to answer queries with the content item data store 754 through network 722 indicates the application software system can receive both keywords (as would be gathered from input through the user interface 712) and documents (as would be gathered from the content item data store 754)]), each keyword of the set of keywords corresponding to each document of the corresponding set of documents ([Retrieving data, i.e. content items tracking to documents, based on keyword matching indicates each keyword corresponds to each document retrieved]).
Chen in view of Feng does not disclose:
generating each natural language text snippet of the set of natural language text snippets based on the set of keywords and the corresponding set of documents.
Zha discloses:
generating each natural language text snippet of the set of natural language text snippets based on the set of keywords and the corresponding set of documents ([0041] Discovery request(s) 400 may include various features, such as description 411. Description 411 may be key words, terms, or other search criteria that can be used to discover prompts. In some embodiments, discovery request(s) 400 may include a sample input/output 412. For instance, sample input may be sample document, file, or other text that may be analyzed using the NLP ML model, and the sample output may an example of expected results (e.g., The most frequent sentiment of commenters on the post is [sentiment]”) [Wherein the output is clearly a natural language snippet, i.e. “The most frequent sentiment…”, generated based on a description, i.e. keywords, and documents]).
Chen, Feng, and Zha are considered analogous art within prompt response. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Chen in view of Feng to incorporate the teachings of Zha, because of the novel way to collect, aggregate, and process large amounts of fine-grained data in the context of specific domains, improving access to increasingly sophisticated machine learning models that can analyze complex data (Zha, [0001]).
Regarding claim 3, Chen in view of Feng discloses: the computer-implemented method of claim 1.
Chen in view of Feng does not disclose:
wherein generating the set of natural language text snippets further comprises:
receiving a prompt, the prompt comprising a set of exemplars, each exemplar in the set of exemplars comprising a keyword, a document corresponding to the keyword, and a natural language text snippet corresponding to the keyword and the document; and,
generating each natural language text snippet of the set of natural language text snippets based on the prompt.
Zha discloses:
wherein generating the set of natural language text snippets further comprises:
receiving a prompt ([Fig. 4, Prompt Task Classification 410], [Classification of a prompt indicates a required reception]), the prompt comprising a set of exemplars ([Fig. 4, Discovery Request(s) 410], [Defining a plurality of requests indicates that the requests, i.e. exemplars, form a set]), each exemplar in the set of exemplars comprising a keyword ([Fig. 4, Description 411], [Wherein it has been previously disclosed that the description contains keywords, see [0041] of Zha]), a document corresponding to the keyword ([Fig. 4, Sample input/output], [0041] In some embodiments, discovery request(s) 400 may include a sample input/output 412. For instance, sample input may be sample document, [0042] key word searches or comparisons on description 411 or sample output 413, [0044] A similar technique could be performed using sample input (e.g., comparing sample input text with sample input text for prompts) [In view of the document selection based on keyword matching of Chen which could be used to select the sample input of Zha without a change in functionality to Zha as the discovery request is dependent upon both keywords and input documents]), and a natural language text snippet corresponding to the keyword and the document ([Fig. 4, Task 419], [0038] NLP processing task 313 (e.g., as indicated by a code, label, category, or natural language description of the NLP task) [The task of Fig. 3 could be used as the task of Fig. 4 without a change in functionality to Fig. 4. Defining a task to be a natural language description indicates that it is a text snippet, corresponding to the keyword and document as shown through the data structure of the discovery request containing all required information]); and,
generating each natural language text snippet of the set of natural language text snippets based on the prompt ([0047] Prompt recommendation generation 440 may generate prompt recommendation 450 and include various information, such as prompt(s) 451, sample output 453 for the prompt(s), [Generating a sample output for a prompt tracks to generation of a text snippet based on the prompt]).
Chen are considered analogous art within prompt response. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Chen in view of Feng to incorporate the teachings of Zha, because of the novel way to collect, aggregate, and process large amounts of fine-grained data in the context of specific domains, improving access to increasingly sophisticated machine learning models that can analyze complex data (Zha, [0001]).
Regarding claim 7, Chen in view of Feng discloses: the computer-implemented method of claim 1.
Chen in view of Feng does not disclose:
wherein the evaluation metric comprising at least one of accuracy, precision, recall, Hit@K, Mean Reciprocal Rank (MRR), and Normalized Discounted Cumulative Gain (NDCG).
Zha discloses:
wherein the evaluation metric comprising at least one of accuracy ([0062] performance metrics may be used for evaluation (e.g., when labeled or ground truth data for test data 672 is used to score the accuracy of the prompt and tuned NLP ML models, [In view of the comparator loss evaluation metric of Chen, which could perform the same accuracy comparison operation as disclosed in Zha without a change in functionality to Zha]), precision, recall, Hit@K, Mean Reciprocal Rank (MRR), and Normalized Discounted Cumulative Gain (NDCG), ([The examiner would like to note that not all evaluation metrics require a mapping due to the disjunctive nature of the claim]).
Chen are considered analogous art within prompt response. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Chen in view of Feng to incorporate the teachings of Zha, because of the novel way to collect, aggregate, and process large amounts of fine-grained data in the context of specific domains, improving access to increasingly sophisticated machine learning models that can analyze complex data (Zha, [0001]).
Regarding claim 8, Chen in view of Feng discloses: the computer-implemented method of claim 1.
Chen in view of Feng does not disclose:
wherein the set of independent natural language text snippets and the corresponding set of data is a test set of data.
Zha discloses:
wherein the set of independent natural language text snippets and the corresponding set of data is a test set of data ([0039] Using test data for an NLP processing task, like task 313, inferences may be made using the prompt 311 and obtain for validation, as indicated at 353. These inferences may then be compared with the sample output 317 and ground truth labels for the test data to determine whether the prompt's claimed sample output 317 is achieved [Comparing test data to a sample output for validation indicates the test data to be independent natural language, i.e. existing after an original training data generation step]).
Chen are considered analogous art within prompt response. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Chen in view of Feng to incorporate the teachings of Zha, because of the novel way to collect, aggregate, and process large amounts of fine-grained data in the context of specific domains, improving access to increasingly sophisticated machine learning models that can analyze complex data (Zha, [0001]).
Regarding claim 9, Chen discloses: a non-transitory computer-readable medium storing executable instructions ([0157] non-transitory machine-readable storage medium (e.g., a non-transitory computer readable medium)… suitable for storing electronic instructions), which when executed by a processing device ([0031] instructions run or executed on a processing device), cause the processing device to perform operations comprising:
prompting a generative language model ([0040] the training data generator 156 may be a generative pre-trained transformer (GPT)).
Chen does not disclose:
the set of independent natural language text snippets corresponding to a natural language variation of the set of natural language text snippets;
generating an evaluation metric that evaluates the fine-tuned language model with respect to the natural language variation based on applying the set of independent natural language text snippets to the fine-tuned language model to generate output and
evaluating the output of the fine-tuned language model with respect to the corresponding set of data used to generate both the set of independent natural language text snippets and the set of natural language text snippets.
Feng discloses:
the set of independent natural language text snippets corresponding to a natural language variation of the set of natural language text snippets ([Table 1, Methods used for augmentation based on an original review text], [The examiner asserts that the operation shown in Table 1 of Feng could be extended to the multiple training datum of Chen without a change in functionality to the augmentation of Feng as Feng discloses multiple natural language text snippets, see Tables 6, 7, 12, 13, and Yelp review dataset size as disclosed in section 2.2]);
generating an evaluation metric that evaluates the fine-tuned language model with respect to the natural language variation based on applying the set of independent natural language text snippets to the fine-tuned language model to generate output ([pg. 31, Section 2.5] We evaluate generated continuations using various metrics assessing major aspects of the text including its diversity, fluency, semantic content preservation, and sentiment consistency, [pg. 32, Section 2.5.2] We finetune GPT-2 on a two-million review subset of YR (with a 500K additional validation split) and use this finetuned model for PPL evaluation., [The examiner asserts that diversity of responses tracks to variation of responses, wherein the generated continuations are independent natural language text snippets being evaluated via metric as would be performed within the model (SBLEU of 2.5.1). The SBLEU score tracks to an evaluation metric which evaluates the fine-tuned language model based on the output produced by said model. Further, Feng discloses using a fine-tuned model for metric evaluation relating to fluency. The examiner asserts that a change in fluency between a natural language snippet and an independent natural language snippet indicates a varied independent natural language snippet. This also further indicates a model could be used for the determining the diversity metric]) and
evaluating the output of the fine-tuned language model with respect to the corresponding set of data used to generate both the set of independent natural language text snippets and the set of natural language text snippets ([Fig, 1(a), SBLUE by Variation], [A comparison between natural language text snippets and varied, i.e. independent, natural language text snippets indicates a comparison between the two with respect to the previously disclosed sets of snippets, wherein the snippets are gathered from the set of data, indicating the evaluation to be with respect to the corresponding set of data from which the snippets are gathered]).
Chen and Feng are considered analogous art within generative model fine-tuning using augmented data. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Chen to incorporate the teachings of Feng, because of the novel way to augment existing data using a GPT model and a plurality of augmentation methods, improving the ability of generative models to generate meaningful responses for low-data regimes through determining an amount of augmented data for producing the highest quality generations (Feng, Abstract).
Chen in view of Feng does not disclose:
prompting a generative language model to generate a set of natural language text snippets based on a set of keywords and a corresponding set of documents, each keyword of the set of keywords corresponding to each document of the set of documents; and,
prompting a generative language model to generate a set of independent natural language text snippets based on the set of keywords and the corresponding set of documents.
Zha discloses:
prompting a generative language model to generate a set of natural language text snippets based on a set of keywords and a corresponding set of documents ([0025] NLP ML models and may be based on various different ML model architectures (e.g., Generative Pre-trained Transformer (GPT)-based ML models)… [0041] Discovery request(s) 400 may include various features, such as description 411. Description 411 may be key words, terms, or other search criteria that can be used to discover prompts. In some embodiments, discovery request(s) 400 may include a sample input/output 412. For instance, sample input may be sample document, file, or other text that may be analyzed using the NLP ML model, and the sample output may an example of expected results (e.g., The most frequent sentiment of commenters on the post is [sentiment]”) [Wherein the output is clearly a natural language snippet, i.e. “The most frequent sentiment…”, generated based on a description, i.e. keywords, and documents]), each keyword of the set of keywords corresponding to each document of the set of documents ([Fig. 4, Discovery Request(s) 410], [In view of the data structure of the discovery request, it is clear that each description, i.e. keyword, corresponds to a sample input, i.e. document]); and,
prompting a generative language model ([In view of the previously disclosed generative language model of Zha]), to generate a set of independent natural language text snippets based on the set of keywords and the corresponding set of documents ([Any of the plurality of prompts generated using the plurality of discovery requests can be considered to be an “independent natural language text snippet”. Further, consider the development/deployment distinction made in Fig. 1 indicating new prompts generated during prompt discovery are independent]).
Chen are considered analogous art within prompt response. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Chen in view of Feng to incorporate the teachings of Zha, because of the novel way to collect, aggregate, and process large amounts of fine-grained data in the context of specific domains, improving access to increasingly sophisticated machine learning models that can analyze complex data (Zha, [0001]).
Chen further discloses:
fine-tuning a language model into a fine-tuned language model using the set of natural language text snippets and the corresponding set of documents as training data ([0033] fine-tuning manager 158 to fine-tune a pretrained machine learning model 120 to obtain the fine-tuned model 110 using the input-output pairs 132 determined, at least in part, by the training data generator 156, [0042] the pretrained machine learning model 120 is an LLM [Wherein the input/output pairs of Chen could be substituted for the discovery request/prompt recommendation of Zha without a change in functionality to Chen as they both have a dependence upon keywords and associated documents, see [0115] of Chen]); and,
generating an evaluation metric of the fine-tuned language model based on the set of independent natural language text snippets and the corresponding set of documents ([0035] an input could be a domain-specific document, [0073] The error (represented by error signal 412) is determined by comparing the predicted output 406 (e.g., generated text of a content type) to the pseudo label 418 (e.g., a content type determined by the training data generator 256) using the comparator 410 [Wherein comparing a predicted output from a fine-tuned model to a ground-truth pseudo label indicates a required comparison of the generated independent natural language text (which will inherently have a content type) to the set of data, i.e. classified by content type, to determine a loss. Further, this comparison indicates that the error, i.e. signal, is a metric reflecting differences between the output and label]).
Regarding claim 10, Chen in view of Feng, further in view of Zha discloses: the non-transitory computer-readable medium of claim 9.
Zha further discloses:
wherein prompting the generative language model to generate a set of natural language text snippets further comprises:
generating a prompt ([Fig. 4, Prompt Task Classification 410], [Classification of a prompt indicates a required generation]), the prompt comprising a set of exemplars ([Fig. 4, Discovery Request(s) 410], [Defining a plurality of requests indicates that the requests, i.e. exemplars, form a set]), each exemplar in the set of exemplars comprising a keyword ([Fig. 4, Description 411], [Wherein it has been previously disclosed that the description contains keywords, see [0041] of Zha]), a document corresponding to the keyword ([Fig. 4, Sample input/output], [0041] In some embodiments, discovery request(s) 400 may include a sample input/output 412. For instance, sample input may be sample document, [0042] key word searches or comparisons on description 411 or sample output 413, [0044] A similar technique could be performed using sample input (e.g., comparing sample input text with sample input text for prompts) [In view of the document selection based on keyword matching of Chen which could be used to select the sample input of Zha without a change in functionality to Zha as the discovery request is dependent upon both keywords and input documents]), and a natural language text snippet corresponding to the keyword and the document ([Fig. 4, Task 419], [0038] NLP processing task 313 (e.g., as indicated by a code, label, category, or natural language description of the NLP task) [The task of Fig. 3 could be used as the task of Fig. 4 without a change in functionality to Fig. 4. Defining a task to be a natural language description indicates that it is a text snippet, corresponding to the keyword and document as shown through the data structure of the discovery request containing all required information]); and,
prompting the generative language model to generate each natural language text snippet of the set of natural language text snippets based on the prompt ([0047] Prompt recommendation generation 440 may generate prompt recommendation 450 and include various information, such as prompt(s) 451, sample output 453 for the prompt(s), [Generating a sample output for a prompt tracks to generation of a text snippet based on the prompt]).
Regarding claim 11, Chen in view of Feng, further in view of Zha discloses: the non-transitory computer-readable medium of claim 9.
Chen further discloses:
wherein prompting the generative language model to generate the set of independent natural language text snippets further comprises:
generating a prompt that is different than an initial prompt used to generate the set of natural language text snippets ([Fig. 1, Obtain Content Items 108], [0043] the pre-processing operation 151 transforms content items 160 into a prompt that the training data generator 156 can subsequently complete, [0043] a resume content item can be mapped to a prompt, an article can be mapped to a prompt, etc. [Defining a plurality of prompts in view of the plurality of content items indicates at least a second prompt that is different from an initial prompt, i.e. resume vs. article, used to generate natural language text snippets, i.e. training data]); and
prompting the generative language model to generate each independent natural language text snippet of the set of independent natural language text snippets based on the prompt ([Fig. 6, 604], [0082] the fine-tuning manager 430 can fine-tune the pretrained machine learning model 408 using extra prompts [Fine-tuning using extra prompts indicates the extra prompt is a prompt different, i.e. independent, from an initial prompt, wherein the output 604 is clearly a natural language text snippet based on a resume (previously determined to be able to be mapped to a prompt, see [0043)]).
Regarding claim 12, Chen in view of Feng, further in view of Zha discloses: the non-transitory computer-readable medium of claim 9.
Chen further discloses:
wherein each natural language text snippet of the set of natural language text snippets and each independent natural language text snippet of the set of independent natural language text snippets are natural language queries ([Fig. 1, 106, 170], [0014] The model input includes a task description, also referred to as a prompt. The task description can include instructions and/or examples of digital content. A task description can be in the form of natural language text, such as a question [In view of the previously disclosed two distinct sets of text snippet training data 104 and 136 to be retrieved separately for different purposes, indicating the data within these sets to be “natural language” and “independent natural language” text snippets, wherein they are all disclosed to be questions, i.e. queries]).
Regarding claim 14, Chen in view of Feng, further in view of Zha discloses: the non-transitory computer-readable medium of claim 9.
Zha further discloses:
wherein the evaluation metric comprising at least one of accuracy ([0062] performance metrics may be used for evaluation (e.g., when labeled or ground truth data for test data 672 is used to score the accuracy of the prompt and tuned NLP ML models, [In view of the comparator loss evaluation metric of Chen, which could perform the same accuracy comparison operation as disclosed in Zha without a change in functionality to Zha]), precision, recall, Hit@K, Mean Reciprocal Rank (MRR), and Normalized Discounted Cumulative Gain (NDCG), ([The examiner would like to note that not all evaluation metrics require a mapping due to the disjunctive nature of the claim]).
Regarding claim 15, Chen discloses: a computing system ([0097] FIG. 7 is a block diagram of a computing system) comprising:
a processor ([0146] Processing device 902); and,
a non-transitory computer-readable medium having stored thereon instructions that when executed by the processor ([0157] non-transitory machine-readable storage medium (e.g., a non-transitory computer readable medium)… suitable for storing electronic instructions), cause the processor to perform operations including:
trigger-generating, via a generative language model ([0040] the training data generator 156 may be a generative pre-trained transformer (GPT)).
Chen does not disclose:
the set of independent natural language text snippets corresponding to a natural language variation of the set of natural language text snippets;
generating an evaluation metric that evaluates the fine-tuned language model with respect to the natural language variation based on applying the set of independent natural language text snippets to the fine-tuned language model to generate output and
evaluating the output of the fine-tuned language model with respect to the corresponding set of data used to generate both the set of independent natural language text snippets and the set of natural language text snippets.
Feng discloses:
the set of independent natural language text snippets corresponding to a natural language variation of the set of natural language text snippets ([Table 1, Methods used for augmentation based on an original review text], [The examiner asserts that the operation shown in Table 1 of Feng could be extended to the multiple training datum of Chen without a change in functionality to the augmentation of Feng as Feng discloses multiple natural language text snippets, see Tables 6, 7, 12, 13, and Yelp review dataset size as disclosed in section 2.2]);
generating an evaluation metric that evaluates the fine-tuned language model with respect to the natural language variation based on applying the set of independent natural language text snippets to the fine-tuned language model to generate output ([pg. 31, Section 2.5] We evaluate generated continuations using various metrics assessing major aspects of the text including its diversity, fluency, semantic content preservation, and sentiment consistency, [pg. 32, Section 2.5.2] We finetune GPT-2 on a two-million review subset of YR (with a 500K additional validation split) and use this finetuned model for PPL evaluation., [The examiner asserts that diversity of responses tracks to variation of responses, wherein the generated continuations are independent natural language text snippets being evaluated via metric as would be performed within the model (SBLEU of 2.5.1). The SBLEU score tracks to an evaluation metric which evaluates the fine-tuned language model based on the output produced by said model. Further, Feng discloses using a fine-tuned model for metric evaluation relating to fluency. The examiner asserts that a change in fluency between a natural language snippet and an independent natural language snippet indicates a varied independent natural language snippet. This also further indicates a model could be used for the determining the diversity metric]) and
evaluating the output of the fine-tuned language model with respect to the corresponding set of data used to generate both the set of independent natural language text snippets and the set of natural language text snippets ([Fig, 1(a), SBLUE by Variation], [A comparison between natural language text snippets and varied, i.e. independent, natural language text snippets indicates a comparison between the two with respect to the previously disclosed sets of snippets, wherein the snippets are gathered from the set of data, indicating the evaluation to be with respect to the corresponding set of data from which the snippets are gathered]).
Chen and Feng are considered analogous art within generative model fine-tuning using augmented data. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Chen to incorporate the teachings of Feng, because of the novel way to augment existing data using a GPT model and a plurality of augmentation methods, improving the ability of generative models to generate meaningful responses for low-data regimes through determining an amount of augmented data for producing the highest quality generations (Feng, Abstract).
Chen in view of Feng does not disclose:
trigger-generating, via a generative language model, a set of natural language queries based on a set of keywords and a corresponding set of documents, each keyword of the set of keywords corresponding to each document of the set of documents; and,
trigger-generating a fine-tuned language model from a language model using the set of natural language queries and the corresponding set of documents as training data.
Zha discloses:
trigger-generating, via a generative language model ([0025] NLP ML model(s) 124 may be pre-trained or custom NLP ML models and may be based on various different ML model architectures (e.g., Generative Pre-trained Transformer (GPT)-based ML models)) to generate a set of natural language queries based on a set of keywords and a corresponding set of documents ([0041] Discovery request(s) 400 may include various features, such as description 411. Description 411 may be key words, terms, or other search criteria that can be used to discover prompts. In some embodiments, discovery request(s) 400 may include a sample input/output 412. For instance, sample input may be sample document, file, or other text that may be analyzed using the NLP ML model, and the sample output may an example of expected results (e.g., The most frequent sentiment of commenters on the post is [sentiment]”) [Wherein there is clearly a required natural language query generated to answer, i.e. “What is the most frequency sentiment of the commentors?”, generated based on a description, i.e. keywords, and documents]), each keyword of the set of keywords corresponding to each document of the set of documents ([Fig. 4, Discovery Request(s) 410], [In view of the data structure of the discovery request, it is clear that each description, i.e. keyword, corresponds to a sample input, i.e. document]); and,
trigger-generating, via the generative language model ([In view of the previously disclosed generative language model of Chen]), a set of independent natural language queries based on the set of keywords and the corresponding set of documents ([Any of the plurality of prompts generated using the plurality of discovery requests can be considered to be an “independent natural language query”. Further, consider the development/deployment distinction made in Fig. 1 indicating new prompts generated during prompt discovery are independent]).
Chen are considered analogous art within prompt response. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Chen in view of Feng to incorporate the teachings of Zha, because of the novel way to collect, aggregate, and process large amounts of fine-grained data in the context of specific domains, improving access to increasingly sophisticated machine learning models that can analyze complex data (Zha, [0001]).
Chen further discloses:
trigger-generating a fine-tuned language model from a language model using the set of natural language queries and the corresponding set of documents as training data ([0033] fine-tuning manager 158 to fine-tune a pretrained machine learning model 120 to obtain the fine-tuned model 110 using the input-output pairs 132 determined, at least in part, by the training data generator 156, [0042] the pretrained machine learning model 120 is an LLM [Wherein the input/output pairs of Chen could be substituted for the discovery request/prompt recommendation of Zha as they both have a dependence upon keywords and associated documents, see [0115] of Chen]); and,
trigger-generating an evaluation metric of the fine-tuned language model based on the set of independent natural language queries and the corresponding set of documents ([0035] an input could be a domain-specific document, [0073] The error (represented by error signal 412) is determined by comparing the predicted output 406 (e.g., generated text of a content type) to the pseudo label 418 (e.g., a content type determined by the training data generator 256) using the comparator 410 [Wherein comparing a predicted output from a fine-tuned model to a ground-truth pseudo label indicates a required comparison of the generated independent natural language text (which will inherently have a content type) to the set of data, i.e. classified by content type, to determine a loss. Further, this comparison indicates that the error, i.e. signal, is a metric reflecting differences between the output and label]).
Zha further discloses:
causing display of the evaluation metric ([0047] Prompt recommendations 450 may be presented in various ways, using various visualization techniques, including, but not limited to, ranking, charts, graphs, output displays, etc. FIG. 5B, discussed below, provides an example of a prompt recommendation, [0046] Other performance metrics may be used for evaluation (e.g., when labeled or ground truth data for test data 435 is used to score the accuracy of the candidate prompt and candidate NLP ML model combination [Providing a prompt recommendation with performance information indicates a display of an evaluation/performance metric]).
Regarding claim 16, Chen in view of Feng, further in view of Zha discloses: the computing system of claim 15.
Zha further discloses:
wherein trigger-generating the set of natural language queries further comprises:
receiving a prompt ([Fig. 4, Prompt Task Classification 410], [Classification of a prompt indicates a required reception]), the prompt comprising a set of exemplars ([Fig. 4, Discovery Request(s) 410], [Defining a plurality of requests indicates that the requests, i.e. exemplars, form a set]), each exemplar in the set of exemplars comprising a keyword ([Fig. 4, Description 411], [Wherein it has been previously disclosed that the description contains keywords, see [0041] of Zha]), a document corresponding to the keyword ([Fig. 4, Sample input/output], [0041] In some embodiments, discovery request(s) 400 may include a sample input/output 412. For instance, sample input may be sample document, [0042] key word searches or comparisons on description 411 or sample output 413, [0044] A similar technique could be performed using sample input (e.g., comparing sample input text with sample input text for prompts) [In view of the document selection based on keyword matching of Chen which could be used to select the sample input of Zha without a change in functionality to Zha as the discovery request is dependent upon both keywords and input documents]), and a natural language query corresponding to the keyword and the document ([Fig. 4, Task 419], [0038] NLP processing task 313 (e.g., as indicated by a code, label, category, or natural language description of the NLP task) [The task of Fig. 3 could be used as the task of Fig. 4 without a change in functionality to Fig. 4. Defining a task to be a natural language description indicates that it is a natural language query in the context of the usage of Zha, corresponding to the keyword and document as shown through the data structure of the discovery request containing all required information]); and,
generating each natural language query of the set of natural language queries based on the prompt ([0047] Prompt recommendation generation 440 may generate prompt recommendation 450 and include various information, such as prompt(s) 451, sample output 453 for the prompt(s), [Generating a sample output for a prompt tracks to generation of a query based on the prompt]).
Regarding claim 17, Chen in view of Feng, further in view of Zha discloses: the computing system of claim 15.
Chen further discloses:
wherein trigger-generating the set of independent natural language queries further comprises:
receiving a prompt that is different than an initial prompt used to generate the set of natural language queries ([Fig. 1, Obtain Content Items 108], [0043] the pre-processing operation 151 transforms content items 160 into a prompt that the training data generator 156 can subsequently complete, [0043] a resume content item can be mapped to a prompt, an article can be mapped to a prompt, etc. [Defining a plurality of prompts in view of the plurality of content items indicates at least a second prompt that is different from an initial prompt, i.e. resume vs. article, used to generate natural language queries, i.e. training data]); and
generating each independent natural language query of the set of independent natural language queries based on the prompt ([Fig. 6, 604], [0082] the fine-tuning manager 430 can fine-tune the pretrained machine learning model 408 using extra prompts [Fine-tuning using extra prompts indicates the extra prompt is a prompt different from an initial prompt, wherein the output 604 is clearly a natural language query based on a resume (previously determined to be able to be mapped to a prompt, see [0043)]).
Regarding claim 19, Chen in view of Feng, further in view of Zha discloses: the computing system of claim 15.
Zha further discloses:
wherein the evaluation metric comprising at least one of accuracy ([0062] performance metrics may be used for evaluation (e.g., when labeled or ground truth data for test data 672 is used to score the accuracy of the prompt and tuned NLP ML models, [In view of the comparator loss evaluation metric of Chen, which could perform the same accuracy comparison operation as disclosed in Zha without a change in functionality to Zha]), precision, recall, Hit@K, Mean Reciprocal Rank (MRR), and Normalized Discounted Cumulative Gain (NDCG), ([The examiner would like to note that not all evaluation metrics require a mapping due to the disjunctive nature of the claim]).
Regarding claim 20, Chen in view of Feng, further in view of Zha discloses: the computing system of claim 15.
Zha further discloses:
wherein the set of independent natural language queries and the corresponding set of data is a test set of data ([0039] Using test data for an NLP processing task, like task 313, inferences may be made using the prompt 311 and obtain for validation, as indicated at 353. These inferences may then be compared with the sample output 317 and ground truth labels for the test data to determine whether the prompt's claimed sample output 317 is achieved [Comparing test data to a sample output for validation indicates the test data to be independent natural language, i.e. existing after an original training data generation step]).
Claim(s) 6 is/are rejected under 35 U.S.C. 103 as being unpatentable over Chen in view of Feng, further in view of Shmuel et al. (US-20230401389-A1), hereinafter Shmuel.
Regarding claim 6, Chen in view of Feng discloses: the computer-implemented method of claim 1.
Chen in view of Feng does not disclose:
wherein fine-tuning the language model into the fine-tuned language model uses Sentence Bidirectional Encoder Representations from Transformers (SBERT).
Shmuel discloses:
wherein fine-tuning the language model into the fine-tuned language model uses Sentence Bidirectional Encoder Representations from Transformers (SBERT) ([0020] BERT includes various techniques for pretraining general purpose language representation model. The general purpose pretrained models can then be fine-tuned on smaller task-specific datasets. SBERT is a modification of the pretrained BERT networks that use Siamese and triplet network structures to derive semantically meaningful sentence embeddings, which can then be compared using cosine-similarity for example).
Chen, Feng, and Shmuel are considered analogous art within natural language processing for data searching/retrieval. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Chen in view of Feng to incorporate the teachings of Shmuel, because of the novel way to generate an embedding for each new search to be compared for similarity to previously generated, existing document embeddings, providing a list of most similar documents to a user, wherein if a selection is made from the search result, the selection is used to retrain the model, improving media content searches with ingested metadata and user input from successful searches (Shmuel, [0023]).
Claim(s) 13, 18 is/are rejected under 35 U.S.C. 103 as being unpatentable over Chen in view of Feng, further in view of Zha, further in view of Shmuel.
Regarding claim 13, Chen in view of Feng, further in view of Zha discloses: the non-transitory computer-readable medium of claim 9.
Chen in view of Feng, further in view of Zha does not disclose:
wherein fine-tuning the language model into the fine-tuned language model uses Sentence Bidirectional Encoder Representations from Transformers (SBERT).
Shmuel discloses:
wherein fine-tuning the language model into the fine-tuned language model uses Sentence Bidirectional Encoder Representations from Transformers (SBERT) ([0020] BERT includes various techniques for pretraining general purpose language representation model. The general purpose pretrained models can then be fine-tuned on smaller task-specific datasets. SBERT is a modification of the pretrained BERT networks that use Siamese and triplet network structures to derive semantically meaningful sentence embeddings, which can then be compared using cosine-similarity for example).
Chen, Feng, Zha, and Shmuel are considered analogous art within natural language processing for data searching/retrieval. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Chen in view of Feng, further in view of Zha to incorporate the teachings of Shmuel, because of the novel way to generate an embedding for each new search to be compared for similarity to previously generated, existing document embeddings, providing a list of most similar documents to a user, wherein if a selection is made from the search result, the selection is used to retrain the model, improving media content searches with ingested metadata and user input from successful searches (Shmuel, [0023]).
Regarding claim 18, Chen in view of Feng, further in view of Zha discloses: the computing system of claim 15.
Chen in view of Feng, further in view of Zha does not disclose:
wherein the fine-tuned language model is generated using Sentence Bidirectional Encoder Representations from Transformers (SBERT).
Shmuel discloses:
wherein the fine-tuning uses Sentence Bidirectional Encoder Representations from Transformers (SBERT) ([0020] BERT includes various techniques for pretraining general purpose language representation model. The general purpose pretrained models can then be fine-tuned on smaller task-specific datasets. SBERT is a modification of the pretrained BERT networks that use Siamese and triplet network structures to derive semantically meaningful sentence embeddings, which can then be compared using cosine-similarity for example).
Chen, Feng, Zha, and Shmuel are considered analogous art within natural language processing for data searching/retrieval. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Chen in view of Feng, further in view of Zha to incorporate the teachings of Shmuel, because of the novel way to generate an embedding for each new search to be compared for similarity to previously generated, existing document embeddings, providing a list of most similar documents to a user, wherein if a selection is made from the search result, the selection is used to retrain the model, improving media content searches with ingested metadata and user input from successful searches (Shmuel, [0023]).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Liu et al. (US-20240346254-A1) discloses “The techniques described herein enhance the operations of natural language generation systems through training and/or augmentation by a large language model. In a first example, the large language model can execute training operations by processing a training dataset to produce a natural language output. The natural language generation system can analyze the training dataset and the natural language output to generate a natural language output mimicking the output of the large language model. The large language model can then evaluate the output of the natural language generation system to iteratively adjust and improve the quality of natural language outputs. In a second example, the large language can augment a small language model in executing natural language tasks. This is accomplished by retrieving external information using the large language model to generate an augmentation input to provide context and a language framework to the small language model to enhance overall outputs.” (abstract). See entire document.
Shrivastava et al. (US-11593556-B2) discloses “predicting current word of the text summary, by iteratively performing: generating first probability distribution of first set of words using first decoder based on word embedding vectors, generating second probability distribution of second set of words using second decoder based on word embedding vectors, and ensembling first and second probability distributions using configurable weight parameter for determining current word. First probability distribution indicates selection probability of each word being selected as current word. Method includes providing custom reward score as feedback to second decoder based on custom reward model and modifying second probability distribution of words for text summary based on feedback” (abstract). See entire document.
Rennie et al. (US-20240265041-A1) discloses “a method for document processing (particularly for training of a machine learning question answering platform, and for ingestion of documents). The method includes obtaining a question dataset (e.g., either from public or private repositories of questions) comprising one or more source questions for document processing by a machine learning question-and-answer system that provides answer data in response to question data submitted by a user, modifying a source question from the question dataset to generate one or more augmented questions with equivalent semantic meanings as that of the source question, and processing a document with the one or more augmented questions” (abstract). See entire document.
Fernandez Musoles et al. (US-20240079000-A1) discloses “a computer-implemented process including obtaining a corpus of natural-language text documents, automatically generating questions about information in corresponding portions of the documents, and associating the questions with the corresponding portions of the documents. The process further includes storing the questions and the associations with the corresponding portions of the documents in memory to form an index of automatically-generated questions to corresponding portions of documents that answer the questions” (abstract). See entire document.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to THEODORE JOHN WITHEY whose telephone number is (703)756-1754. The examiner can normally be reached Monday - Friday, 8am-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Andrew Flanders can be reached at (571) 272-7516. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/THEODORE WITHEY/Examiner, Art Unit 2655
/JESSE S PULLIAS/Primary Examiner, Art Unit 2655 07/29/26
Prosecution Insights