DETAILED ACTION
This non-final office action is responsive to application 18/750,742 as submitted on June 21, 2024.
Claim status is currently pending and under examination for claims 1-20 of which independent claims are 1, 12 and 20.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Independent Claims 1, 12 and 20
Step 2A Prong One: Does the claim recite an abstract idea, law of nature, or natural phenomenon?
Yes, independent claim 1, under the broadest reasonable interpretation, recites the following limitations that are abstract ideas:
determining one or more candidate immutable tokens from the dataset; (mental process)
determining one or more immutable tokens from the one or more candidate immutable tokens, based on a predetermined rule or a subject matter expert analysis; (mental process)
generating synthetic data, using a large language model, (mental process)
and filtering the generated synthetic data based on compliance with the one or more immutable tokens (mental process)
generating synthetic data, using a generative large language model, based on the one or more determined immutable tokens; (claim 20) (mental process)
and filtering, using the generative large language model, the generated synthetic data based on one or more rules associated with the one or more immutable tokens. (claim 20) (mental process)
The “determining one or more candidate immutable tokens” step involves identifying from a dataset which tokens are candidate immutable tokens which amounts to no more than observations, evaluations, and judgments that can be performed in the human mind or with the use of a physical aid (e.g., pen and paper). The claim recites the step of determining candidate immutable tokens at a high degree of generality, thus the step is not required to have any specific level of complexity that would preclude the step from being mental processes. Therefore, the step is considered to be mental processes, see MPEP § 2106.04(a)(2)(III).
The “determining one or more immutable tokens” step involves using a predetermined rule or subject matter expert analysis to identify which candidate immutable tokens are immutable tokens which amounts to no more than observations, evaluations, and judgments that can be performed in the human mind or with the use of a physical aid (e.g., pen and paper). The claim recites the step of determining immutable tokens at a high degree of generality, thus the step is not required to have any specific level of complexity that would preclude the step from being mental processes. Therefore, the step is considered to be mental processes, see MPEP § 2106.04(a)(2)(III).
The “generating synthetic data” step involves determining new data samples which amounts to no more than observations, evaluations, and judgments that can be performed in the human mind or with the use of a physical aid (e.g., pen and paper). The claim recites the step of generating synthetic data at a high degree of generality, thus the step is not required to have any specific level of complexity that would preclude the step from being mental processes. Therefore, the “generating synthetic data” step is considered to be mental processes, see MPEP § 2106.04(a)(2)(III).
The “filtering … based on compliance” step involves identifying which synthetic data to keep or remove based on compliance with immutable tokens which amounts to no more than observations, evaluations, and judgments that can be performed in the human mind or with the use of a physical aid (e.g., pen and paper). The claim recites the step of filtering generated synthetic data at a high degree of generality, thus the step is not required to have any specific level of complexity that would preclude the step from being mental processes. Therefore, the step is considered to be mental processes, see MPEP § 2106.04(a)(2)(III).
The “generating synthetic data … based on … determined immutable tokens” step involves using immutable tokens to determine new data samples which amounts to no more than observations, evaluations, and judgments that can be performed in the human mind or with the use of a physical aid (e.g., pen and paper). The claim recites the step of generating synthetic data at a high degree of generality, thus the step is not required to have any specific level of complexity that would preclude the step from being mental processes. Therefore, the step is considered to be mental processes, see MPEP § 2106.04(a)(2)(III).
The “filtering … based on one or more rules” step involves identifying which synthetic data to keep or remove based on rules which amounts to no more than observations, evaluations, and judgments that can be performed in the human mind or with the use of a physical aid (e.g., pen and paper). The claim recites the step of filtering generated synthetic data at a high degree of generality, thus the step is not required to have any specific level of complexity that would preclude the step from being mental processes. Therefore, the step is considered to be mental processes, see MPEP § 2106.04(a)(2)(III).
Therefore, the independent claim recites a judicial exception. Independent claims 12 and 20 recite similar limitations corresponding to claim 1, therefore the same subject matter eligibility analysis is applied.
Step 2A Prong Two: Does the claim recite additional elements that integrate the judicial exception into a practical application?
No, the judicial exception recited above is not integrated into a practical application. The claims recite the following additional elements, but these additional elements are not sufficient to integrate the judicial exception into a practical application:
receiving a dataset comprising a plurality of text entities; (MPEP § 2106.05(g) necessary data gathering and insignificant extra-solution activity to the judicial exception)
receiving a dataset, the dataset comprising a plurality of classes; (claim 20) (MPEP § 2106.05(g) necessary data gathering and insignificant extra-solution activity to the judicial exception)
The “receiving” steps amount to mere data gathering and are recited at a high level of generality, thus adding insignificant extra-solution activity to the judicial exception – see MPEP § 2106.05(g). Under MPEP § 2106.05(d), such additional elements have been found by the courts to not integrate a judicial exception into a practical application.
generating synthetic data, using a large language model, (MPEP § 2106.05(f) mere instructions to implement an abstract idea on a computer, or generally links exception to a technological environment)
The “generating synthetic data, using a large language model” step requires a “large language model” to generate synthetic data. The large language model is used to apply the recited judicial exception without placing any limitation on how the large language model operates. The limitation amounts to mere instructions to “apply” the judicial exception on a computer. It can also be viewed as nothing more than an attempt to generally link the use of the judicial exception to the technological environment of computers, see MPEP § 2106.05(f).
wherein the large language model is instructed to maintain the one or more immutable tokens; (MPEP § 2106.05(f) mere instructions to implement an abstract idea on a computer, or generally links exception to a technological environment)
The “wherein the large language model” step is recited at a high-level of generality such that the limitation amounts to no more than mere instructions to “apply” the judicial exception on a computer. It can also be viewed as nothing more than an attempt to generally link the use of the judicial exception to the technological environment of computers, see MPEP § 2106.05(f).
a datastore having stored therein a dataset comprising a plurality of text entities; (claim 12) (MPEP § 2106.05(f) mere instructions to implement an abstract idea on a computer, or generally links exception to a technological environment)
The “a datastore” step is recited at a high-level of generality such that the limitation amounts to no more than mere instructions to “apply” the judicial exception on a computer. It can also be viewed as nothing more than an attempt to generally link the use of the judicial exception to the technological environment of computers, see MPEP § 2106.05(f).
generating synthetic data, using a generative large language model, based on the one or more determined immutable tokens; (claim 20) (MPEP § 2106.05(f) mere instructions to implement an abstract idea on a computer, or generally links exception to a technological environment)
The “generating synthetic data … based on … determined immutable tokens” step requires a “generative large language model” to generate synthetic data. The generative large language model is used to apply the recited judicial exception without placing any limitation on how the generative large language model operates. The limitation amounts to mere instructions to “apply” the judicial exception on a computer. It can also be viewed as nothing more than an attempt to generally link the use of the judicial exception to the technological environment of computers, see MPEP § 2106.05(f).
and filtering, using the generative large language model, the generated synthetic data based on one or more rules associated with the one or more immutable tokens. (claim 20) (MPEP § 2106.05(f) mere instructions to implement an abstract idea on a computer, or generally links exception to a technological environment)
The “filtering … based on one or more rules” step requires a “generative large language model” to filter generated data. The generative large language model is used to apply the recited judicial exception without placing any limitation on how the generative large language model operates. The limitation amounts to mere instructions to “apply” the judicial exception on a computer. It can also be viewed as nothing more than an attempt to generally link the use of the judicial exception to the technological environment of computers, see MPEP § 2106.05(f).
a computing node comprising a computer readable storage medium having program instructions embodied therewith, (claim 12) (MPEP § 2106.05(f) mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea)
the program instructions executable by a processor of the computing node to cause the processor to perform a method comprising: (claim 12) (MPEP § 2106.05(f) mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea)
the computer program product comprising a computer readable storage medium having program instructions embodied therewith, (claim 20) (MPEP § 2106.05(f) mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea)
the program instructions executable by a processor to cause the processor to perform a method comprising: (claim 20) (MPEP § 2106.05(f) mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea)
The remaining additional elements are recited at a high-level of generality such that they amount to no more than mere instructions to “apply” an exception using a generic component. Adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea, see MPEP § 2106.05(f).
Therefore, the above limitations do not integrate the judicial exception into a practical application.
Step 2B: Does the claim recite additional elements that amount to significantly more than the judicial exception?
No. The claims do not include additional elements that are sufficient for the claims to amount to significantly more than the judicial exception.
In regards to the “receiving” steps, these steps add insignificant extra-solution activity. An extra-solution activity is a well-understood, routine and conventional (WURC) activity per MPEP § 2106.05(d)(II), “the courts have recognized the following computer functions as well‐understood, routine, and conventional functions when they are claimed in a merely generic manner (e.g., at a high level of generality) or as insignificant extra-solution activity. i. Receiving or transmitting data over a network, e.g., using the Internet to gather data.” The “receiving” steps do not integrate the judicial exception into a practical application and do not amount to significantly more.
In regards to the “large language model” in the “generating synthetic data, using a large language model” step, the limitations are recited so generically such that they amount to no more than mere instructions to “apply” the judicial exception on a computer using generic computer components. Mere instructions to apply a judicial exception cannot provide an inventive concept. See MPEP § 2106.05(f).
In regards to the “generative large language model” in the “generating synthetic data … based on … determined immutable tokens” step, the limitations are recited so generically such that they amount to no more than mere instructions to “apply” the judicial exception on a computer using generic computer components. Mere instructions to apply a judicial exception cannot provide an inventive concept. See MPEP § 2106.05(f).
In regards to the “generative large language model” in the “filtering … based on one or more rules” step, the limitations are recited so generically such that they amount to no more than mere instructions to “apply” the judicial exception on a computer using generic computer components. Mere instructions to apply a judicial exception cannot provide an inventive concept. See MPEP § 2106.05(f).
In regards to the “wherein the large language model” and “a datastore” steps and the remaining additional elements, the limitations are recited so generically such that they amount to no more than mere instructions to “apply” the judicial exception on a computer using generic computer components. Mere instructions to apply a judicial exception cannot provide an inventive concept. See MPEP § 2106.05(f).
Therefore, independent claims 1, 12 and 20 are not patent eligible.
Dependent Claims 2-11 and 13-19
The remaining dependent claims being rejected do not recite additional elements, whether considered individually or in combination, that are sufficient to integrate the judicial exception into a practical application or amount to significantly more than a judicial exception.
Claim limitation
Examiner analysis
2 and 13. The method of claim 1, wherein said filtering is performed using a large language model.
The step is recited at a high-level of generality such that the limitations amount to no more than mere instructions to “apply” the judicial exception on a computer. They can also be viewed as nothing more than an attempt to generally link the use of the judicial exception to the technological environment of computers, see MPEP § 2106.05(f).
3 and 14. The method of claim 1, wherein the predetermined rule comprises identity, synonym, and/or antonym.
The step is recited at a high-level of generality such that the limitations amount to no more than mere instructions to “apply” the judicial exception on a computer. They can also be viewed as nothing more than an attempt to generally link the use of the judicial exception to the technological environment of computers, see MPEP § 2106.05(f).
4 and 15. The method of claim 1, wherein the dataset comprises a plurality of classes.
The step is recited at a high-level of generality such that the limitations amount to no more than mere instructions to “apply” the judicial exception on a computer. They can also be viewed as nothing more than an attempt to generally link the use of the judicial exception to the technological environment of computers, see MPEP § 2106.05(f).
5 and 16. The method of claim 4, further comprising: training a multi-class classification model, using the filtered synthetic data.
The step is recited at a high-level of generality such that the limitations amount to no more than mere instructions to “apply” the judicial exception on a computer. They can also be viewed as nothing more than an attempt to generally link the use of the judicial exception to the technological environment of computers, see MPEP § 2106.05(f).
6 and 17. The method of claim 1, wherein the large language model is a generative pre-trained transformer model.
The step is recited at a high-level of generality such that the limitations amount to no more than mere instructions to “apply” the judicial exception on a computer. They can also be viewed as nothing more than an attempt to generally link the use of the judicial exception to the technological environment of computers, see MPEP § 2106.05(f).
7 and 18. The method of claim 1, wherein the large language model is a masked language model.
The step is recited at a high-level of generality such that the limitations amount to no more than mere instructions to “apply” the judicial exception on a computer. They can also be viewed as nothing more than an attempt to generally link the use of the judicial exception to the technological environment of computers, see MPEP § 2106.05(f).
8 and 19. The method of claim 1, wherein said determining of the one or more candidate immutable tokens comprise one or more of collocation, co-occurrence, repetitions, and/or part of speech analysis.
This is a mental process akin to a human evaluation/judgment/observation.
9. The method of claim 1, wherein determining the one or more candidate immutable tokens comprises identifying one or more tokens that maintain a meaning across one or more contexts in the dataset.
This is a mental process akin to a human evaluation/judgment/observation.
10. The method of claim 1, wherein determining the one or more candidate immutable tokens comprises linguistic analysis.
This is a mental process akin to a human evaluation/judgment/observation.
11. The method of claim 1, wherein the one or more candidate immutable tokens include at least one full word and/or phrase.
The step is recited at a high-level of generality such that the limitations amount to no more than mere instructions to “apply” the judicial exception on a computer. They can also be viewed as nothing more than an attempt to generally link the use of the judicial exception to the technological environment of computers, see MPEP § 2106.05(f).
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
The following is the reference relied upon in the rejections below:
Shani, Chen, Jilles Vreeken, and Dafna Shahaf. "Towards concept-aware large language models." Findings of the Association for Computational Linguistics: EMNLP 2023. 2023.
Claims 1-5, 7, 9-16, 18 and 20 are rejected under 35 U.S.C. 102a(1) as being anticipated by Shani.
Regarding Claim 1, Shani teaches:
A method for generating synthetic data, comprising ((P. 13162, Sec. 4, ¶4-5) “we iterate over all possible subset combinations of C (or perhaps sample from them, if the set is too big) and interleave them with the text using a special token, resulting in augmented sentences”):
receiving a dataset comprising a plurality of text entities ((P. 13159, Sec. 2, ¶2) “To probe LLM abilities, we use the 100 everyday things (ETs) dataset (Gu et al., 2022), containing natural and man-made items everyone should be familiar with (egg, dog, elevator, table, etc.).”
(P. 13162, Sec. 4, ¶4) “Given an input sentence, we sample a span of text T that can be linked to a Wikipedia article (similar to Wu et al. (2019)), such as “ice cream”, and mask it. We denote by S the masked sentence. We compile a subset of Wikipedia categories that the missing text T belongs to and their parent categories”);
determining one or more candidate immutable tokens from the dataset (The Examiner interprets “immutable tokens” according to its broadest reasonable interpretation (BRI) in view of the Applicant’s specification as encompassing unmasked text of a sentence. This interpretation is consistent with the illustrative descriptions in the Applicant’s specification at [0036], (see excerpt below).
Applicant’s written description at [0036]: “The proposed approach tackles the problem of synthetic data generation by identifying immutable tokens that must adhere to the rules specified for such tokens (e.g. exact word/phrase, synonyms, antonyms, etc.). This includes conducting a priori analysis to identify tokens that maintain desired meaning across contexts and labeling them as immutable or protected; … and performing post priori analysis of generated data to ensure the preservation of the meaning for immutable tokens.”
(P. 13162, Sec. 4, ¶4-5) “Given an input sentence, we sample a span of text T that can be linked to a Wikipedia article (similar to Wu et al. (2019)), such as “ice cream”, and mask it. We denote by S the masked sentence. We compile a subset of Wikipedia categories that the missing text T belongs to …, resulting in a set of candidate concepts C. Next, we iterate over all possible subset combinations of C (or perhaps sample from them, if the set is too big) and interleave them with the text using a special token, resulting in augmented sentences
{
S
i
'
}
such as “At the party we had cake and <Dairy food, Frozen desserts> ice cream”.”
Shani discloses Figure 4 on P. 13162 (reproduced below) depicting a method of using an LLM to create sentence augmentations. A span of text T (“ice cream”) is sampled from a given input sentence S and then masked. The remaining text of the input sentence, “At the party we had cake and”, is not masked. The masked text T is then replaced by candidate concepts, and the candidate concepts are then interleaved with input sentence S to generate augmented sentences. The augmented sentences each have the unmasked text “At the party we had cake and” remain unchanged (the unmasked text contains the exact words/phrase across contexts (candidate concepts)), therefore the input sentence’s original meaning is maintained (and therefore unmasked text of a sentence are immutable tokens).
PNG
media_image1.png
503
1823
media_image1.png
Greyscale
);
determining one or more immutable tokens from the one or more candidate immutable tokens, based on a predetermined rule or a subject matter expert analysis ((P. 13162, Sec. 4, ¶3-4) “We envision training a model to choose a subset from a closed set of concepts that will assist it the most in completing a missing sequence of text. Potential concepts could come from Wikipedia categories, or perhaps some knowledge graph. See our suggested approach in Figure 4. Given an input sentence, we sample a span of text T that can be linked to a Wikipedia article (similar to Wu et al. (2019)), such as “ice cream”, and mask it.”
To determine which span of text in an input sentence will be masked, a span of text T is sampled from the input sentence to link it to a Wikipedia article. If the sampled text can be linked to a Wikipedia article, then the sampled text is chosen to be masked. The remaining text that is not selected to be masked are immutable tokens that remain unchanged (the remaining text is not masked or replaced with Wikipedia concepts). When sampling text spans from the input sentence, the entire text of the input sentence are candidate immutable tokens (since any span of text can be masked or unmasked). Unmasked text (text that did not result in being linked with a Wikipedia article) are the determined ‘immutable tokens’ (and therefore determining immutable tokens from candidate immutable tokens (original sentence) based on a predetermined rule (whether spanned text links to a Wikipedia article)).);
generating synthetic data, using a large language model, wherein the large language model is instructed to maintain the one or more immutable tokens ((P. 13162, Figure 4 Caption) “a method to pretrain concept-aware LLMs. The model is trained to choose the categories that will best assist it in predicting the missing token(s)”
(P. 13162, Sec. 4, ¶4-5) “We compile a subset of Wikipedia categories that the missing text T belongs to … resulting in a set of candidate concepts C. Next, we iterate over all possible subset combinations of C (or perhaps sample from them, if the set is too big) and interleave them with the text using a special token, resulting in augmented sentences
{
S
i
'
}
such as “At the party we had cake and <Dairy food, Frozen desserts> ice cream”. We then mask T in all augmented sentences
{
S
i
'
}
and filter out sentences that do not reduce the prediction loss over T, compared to the non-augmented masked sentence S. That is, we only keep the original (non-masked) sentences augmented with subsets of concepts that help predict the missing text T.”
An LLM is trained (instructed) to generate augmented sentences that reduce prediction loss over mask T (how well the LLM can predict the missing text for mask T using candidate concepts). Augmented sentences
{
S
i
'
}
are unmasked text (immutable tokens) augmented with candidate concepts. The prediction loss measures how augmented sentences
{
S
i
'
}
compare to a non-augmented masked sentence S (original sentence S containing unmasked text (immutable tokens) and mask T), therefore the loss measures how close the meanings of the augmented sentences are to the meaning of the non-augmented masked sentence S. And therefore, augmented sentences
{
S
i
'
}
(‘synthetic data’) are generated using an LLM that is instructed to maintain the meaning of unmasked text (immutable tokens) by minimizing prediction loss.);
and filtering the generated synthetic data based on compliance with the one or more immutable tokens ((P. 13162, Sec. 4, ¶5) “We then mask T in all augmented sentences
{
S
i
'
}
and filter out sentences that do not reduce the prediction loss over T, compared to the non-augmented masked sentence S. That is, we only keep the original (non-masked) sentences augmented with subsets of concepts that help predict the missing text T.”
Shani discloses Steps 2 and 3 of Figure 4 on P. 13162 (reproduced below) depicting filtering augmented sentences by prediction loss. The losses of augmented sentences are compared to L(T|S), the loss of non-augmented masked sentence S (original sentence S with unmasked text and mask T). Therefore, augmented sentences with reduced prediction loss represent synthetic data that complies (shares meaning) with unmasked text (immutable tokens).
PNG
media_image2.png
646
1516
media_image2.png
Greyscale
).
Regarding Claims 2 and 13, Shani teaches:
The method of claim 1, wherein said filtering is performed using a large language model ((P. 13162, Sec. 4, ¶5-6) “we only keep the original (non-masked) sentences augmented with subsets of concepts that help predict the missing text T. We suggest to train an LLM over the augmented, non-masked sentences
{
S
i
'
}
corresponding to concepts that do reduce the loss, resulting in a model that teaches itself when and which concept(s) to predict, and how to best incorporate the predicted concept(s) into future token prediction.”).
Regarding Claims 3 and 14, Shani teaches:
The method of claim 1, wherein the predetermined rule comprises identity, synonym, and/or antonym ((P. 13162, Sec. 4, ¶4) “Given an input sentence, we sample a span of text T that can be linked to a Wikipedia article (similar to Wu et al. (2019)), such as “ice cream”, and mask it. … We compile a subset of Wikipedia categories that the missing text T belongs to and their parent categories, filtering out categories that are too narrow (“Independence Day (US) foods”) or too wide (“World cuisine”), resulting in a set of candidate concepts C.”
To determine which span of text will be masked, a span of text T is sampled from an input sentence to link it to a Wikipedia article based on the categories and parent categories the span of text belongs to. Categories that are too wide or too narrow are filtered, therefore the categories a span of text belongs to are the span’s identity.
If the span of text can be linked to a Wikipedia article (a category/identity has been found), then the sampled text is chosen to be masked. The remaining text that is not selected to be masked are immutable tokens that remain unchanged (the remaining text is not masked or replaced with Wikipedia concepts). Unmasked text (text that did not result in being linked with a Wikipedia article) are ‘immutable tokens’, therefore determining immutable tokens based on a predetermined rule of whether spanned text links to categories (‘identity’).).
Regarding Claims 4 and 15, Shani teaches:
The method of claim 1, wherein the dataset comprises a plurality of classes ((P. 13162, Sec. 4, ¶7) “We can treat the problem of concept prediction as a probabilistic multi-class classification (over a closed set of categories)”
(P. 13159, Sec. 2, ¶2) “we use the 100 everyday things (ETs) dataset (Gu et al., 2022), containing natural and man-made items everyone should be familiar with (egg, dog, elevator, table, etc.).”).
Regarding Claims 5 and 16, Shani teaches:
The method of claim 4, further comprising: training a multi-class classification model, using the filtered synthetic data ((P. 13162, Sec. 4, ¶5-6) “we only keep the original (non-masked) sentences augmented with subsets of concepts that help predict the missing text T. We suggest to train an LLM over the augmented, non-masked sentences
{
S
i
'
}
corresponding to concepts that do reduce the loss, resulting in a model that teaches itself when and which concept(s) to predict, and how to best incorporate the predicted concept(s) into future token prediction.”
(P. 13162, Sec. 4, ¶7) “We can treat the problem of concept prediction as a probabilistic multi-class classification (over a closed set of categories)”).
Regarding Claims 7 and 18, Shani teaches:
The method of claim 1, wherein the large language model is a masked language model ((P. 13162, Figure 4 Caption) “a method to pretrain concept-aware LLMs. The model is trained to choose the categories that will best assist it in predicting the missing token(s)”
(P. 13162, Sec. 4, ¶5) “we only keep the original (non-masked) sentences augmented with subsets of concepts that help predict the missing text T.”
An LLM is trained to predict missing tokens (concept/text) for a masked sentence’s mask T by using the unmasked text of the sentence, therefore the LLM is a masked language model.).
Regarding Claim 9, Shani teaches:
The method of claim 1, wherein determining the one or more candidate immutable tokens comprises identifying one or more tokens that maintain a meaning across one or more contexts in the dataset (Shani discloses Figure 4 on P. 13162 (reproduced above) depicting a method of using an LLM to create sentence augmentations. A span of text T (“ice cream”) is sampled from a given input sentence S and then masked (see P. 13162, Sec. 4, ¶4). The remaining text of the input sentence, “At the party we had cake and”, is not masked. The masked text T is then replaced by candidate concepts, and the candidate concepts are then interleaved with input sentence S to generate augmented sentences (see P. 13162, Sec. 4, ¶5). The augmented sentences each have the unmasked text, “At the party we had cake and”, remain unchanged (the unmasked text contains the exact words/phrase across contexts (candidate concepts)), therefore the input sentence’s original meaning is maintained (and therefore unmasked text of a sentence are immutable tokens that maintain a meaning across one or more contexts.).
Regarding Claim 10, Shani teaches:
The method of claim 1, wherein determining the one or more candidate immutable tokens comprises linguistic analysis ((P. 13162, Sec. 4, ¶4) “Given an input sentence, we sample a span of text T that can be linked to a Wikipedia article (similar to Wu et al. (2019)), such as “ice cream”, and mask it. … We compile a subset of Wikipedia categories that the missing text T belongs to and their parent categories, filtering out categories that are too narrow (“Independence Day (US) foods”) or too wide (“World cuisine”), resulting in a set of candidate concepts C.”
To determine which span of text in an input sentence will be masked, a span of text T is sampled from the input sentence to link it to a Wikipedia article (a category or parent category matching the span of text is found). By determining which Wikipedia categories the span of text belongs to, the span of text is being compared to categories semantically, therefore linguistic analysis is performed. If the span of text can be linked to a Wikipedia article, then the span of text is chosen to be masked (based on linguistic analysis), while the rest of the remaining text are ‘immutable tokens’ that remain unchanged (the remaining text is not masked or replaced with Wikipedia concepts), therefore determining immutable tokens based on linguistic analysis.).
Regarding Claim 11, Shani teaches:
The method of claim 1, wherein the one or more candidate immutable tokens include at least one full word and/or phrase (Shani discloses Figure 4 on P. 13162 (reproduced above) depicting a method of using an LLM to create sentence augmentations. A span of text T (“ice cream”) is sampled from a given input sentence S and then masked (see P. 13162, Sec. 4, ¶4). The remaining text of the input sentence, “At the party we had cake and”, is not masked. The unmasked text (‘immutable tokens’) is a phrase comprised of full words.).
Regarding Claim 12, the rejection of Claim 1 is incorporated. The difference in scope being:
A system comprising ((P. 13158, Abstract) “We sketch a method for pretraining LLMs using concepts”):
a datastore having stored therein a dataset ((P. 13159, Sec. 2, ¶2) “To probe LLM abilities, we use the 100 everyday things (ETs) dataset”
By using an ETs dataset to probe an LLM’s abilities, the dataset must be retrieved from storage, therefore a datastore having stored therein a dataset is implied.);
a computing node comprising a computer readable storage medium having program instructions embodied therewith (The Examiner interprets “computing node” according to its BRI in view of the Applicant’s specification as encompassing a computer operating with a processor. This interpretation is consistent with the illustrative descriptions in the Applicant’s specification at [0048].
(P. 13162, Figure 4 Caption) “The model is trained to choose the categories that will best assist it in predicting the missing token(s)”
A computer (computer system) is implied by training a model, which further implies a computer operating with a processor (microprocessor-based system), therefore a computer operating with a processor is a ‘computing node’. Using a computer to train a model further implies a computer readable storage medium having stored program instructions.),
the program instructions executable by a processor of the computing node to cause the processor to perform a method comprising (A computer operating with a processor (‘computing node’) is implied by training a model (see P. 13162, Figure 4 Caption), which further implies the processor executing program instructions.).
Regarding Claim 20, the rejection of Claim 1 is incorporated. The difference in scope being:
A computer program product ((P. 13158, Abstract) “We sketch a method for pretraining LLMs using concepts”),
the computer program product comprising a computer readable storage medium having program instructions embodied therewith (Pretraining LLMs implies using a computer, which further implies a computer readable storage medium having stored program instructions.),
the program instructions executable by a processor to cause the processor to perform a method comprising (A computer is implied by pretraining LLMs, which further implies a processor executing program instructions.):
receiving a dataset, the dataset comprising a plurality of classes ((P. 13162, Sec. 4, ¶7) “We can treat the problem of concept prediction as a probabilistic multi-class classification (over a closed set of categories)”
(P. 13159, Sec. 2, ¶2) “we use the 100 everyday things (ETs) dataset (Gu et al., 2022), containing natural and man-made items everyone should be familiar with (egg, dog, elevator, table, etc.).”);
generating synthetic data, using a generative large language model((P. 13162, Sec. 4, ¶5-6) “we only keep the original (non-masked) sentences augmented with subsets of concepts that help predict the missing text T. We suggest to train an LLM over the augmented, non-masked sentences
{
S
i
'
}
corresponding to concepts that do reduce the loss, resulting in a model that teaches itself when and which concept(s) to predict, and how to best incorporate the predicted concept(s) into future token prediction.”);
and filtering, using the generative large language model, the generated synthetic data based on one or more rules associated with the one or more immutable tokens ((P. 13162, Sec. 4, ¶6) “We suggest to train an LLM over the augmented, non-masked sentences
{
S
i
'
}
corresponding to concepts that do reduce the loss”
Shani discloses Steps 2 and 3 of Figure 4 on P. 13162 (reproduced above) depicting filtering augmented sentences (‘generated synthetic data’) by prediction loss. The losses of augmented sentences are compared to L(T|S), the loss of non-augmented masked sentence S (original sentence S with unmasked text and mask T). Therefore, augmented sentences with reduced prediction loss represent synthetic data that maintains a similar meaning with unmasked text (immutable tokens), and therefore synthetic data is filtered based on rules (prediction loss) associated with immutable tokens.).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 6 and 17 are rejected under 35 U.S.C. 103 as being unpatentable over Shani.
Regarding Claims 6 and 17, Shani teaches the method of claim 1. Furthermore, Shani also teaches the further limitations in a different embodiment:
wherein the large language model is a generative pre-trained transformer model ((P. 13163, Sec. 4.1, ¶2) “To augment the input sentence
S
0
we first retrieve the LLM top-k completions. We replace the “[MASK]” token with the first completion that is not a stopword or a sub-word and paraphrase using wordtune.”
(P. 13163, Figure 5 Caption) “We augment the input sentence by paraphrasing and predict the top k completions for each paraphrase”
(P. 13161, Sec. 3, ¶1) “We use the queries “Is <hyponym> a type of <hypernym>?” (for GPT), and “<hyponym> is a type of [MASK]” (BERT and T5; we then search for the hypernym along the top-k=50 completions).”
A GPT model is used to retrieve top-k completions by executing queries. The retrieved completions are then used to generate augmented sentences.).
Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to modify the method of Shani with the GPT model disclosed by Shani to use a GPT model to generate augmented sentences. By using a GPT model to generate augmented sentences, the GPT model can be queried to determine hypernyms and hyponyms that are most similar to a masked token of an original sentence, thereby generating augmented sentences with different words that preserve the original sentence’s context.
The following are the references relied upon in the rejections below:
Sanyal, Soumya, et al. "APOLLO: A simple approach for adaptive pretraining of language models for logical reasoning." Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2023.
Claims 8 and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Shani / Sanyal.
Regarding Claims 8 and 19, Shani teaches: The method of claim 1, however Shani does not teach determining candidate immutable tokens by performing parts of speech analysis, which is taught by Sanyal:
wherein said determining of the one or more candidate immutable tokens comprise one or more of collocation, co-occurrence, repetitions, and/or part of speech analysis (The Examiner interprets “immutable tokens” according to its BRI in view of the Applicant’s specification as encompassing unmasked text of a sentence obtained from masked language modeling. This interpretation is consistent with the illustrative descriptions in the Applicant’s specification at [0040], (see excerpt below).
Applicant’s written description at [0040]: “Synthetic data generation may be done by, for example, using Masked Language Modeling (MLM) to generate data. For example, an input text may read “Engine failed to start.” The corresponding MLM inputs may include: “<mask> failed to start.” … By using immutable tokens, a user can identify starting problems in the components, for example making “failed to start” an immutable token. Then, the MLM input becomes “<mask> failed to start.””
(P. 6309, Sec. 1, ¶4) “we modify the masked language modeling (MLM) loss (Devlin et al., 2019) to selectively mask specific words in the sentence, based on their parts-of-speech tags. … We hypothesize that masking words with parts-of-speech (POS) tags that are related to higher-order reasoning (such as adverbs, conjunctions, etc.) present more challenging masked positions for the PLM to predict. For instance, in Figure 1, we observe that the words marked in blue boxes are more related to reasoning compared to the non-highlighted words that mainly involve knowledge about specific nouns or English grammar.”
(P. 6308, Figure 1 Caption) “In random masking (Devlin et al., 2019), a word is masked at random. Predicting these words often require more of language understanding than higher-order reasoning (e.g., predicting “would” at the 2nd [MASK] place). In selective masking, a word is masked if its POS tag is from a specific set. These candidate words are marked in the blue box in the input sentence. Filling these words requires more reasoning (e.g., to predict “more” at the 2nd [MASK] place instead of “less”, which is also grammatically valid, the model needs a better understanding of the semantics of the sentence)”
Masked language modeling is modified to perform selective masking of words in an input sentence based on parts-of-speech (POS) tags. Candidate words are masked if a word matches a POS tag. The remaining text that is not masked are immutable tokens (see Figure 1 on P. 6308 depicting an input sentence with masked and unmasked text) since the text is not masked and remains the same (only masked words change since they are being predicted). Therefore, immutable tokens (unmasked text) are determined based on part-of-speech analysis.).
Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to modify the method of Shani with the selective masking technique disclosed by Sanyal to selectively mask words based on parts-of-speech. By selectively masking words based on parts-of-speech, a model predicting masked words would be required to have better reasoning and understanding of an input sentence’s semantics, thereby generating sentences that are most likely to conform to the original meaning of the input sentence.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Peng et al. (US 20240320559 A1) teaches creating an augmented training set for a multi-intent classification model by replacing words in a sentence with synonyms and excluding augmented sentences based on confidence levels.
Yohannes et al. (“Self-attention-based data augmentation method for text classification”) teaches generating augmented training data by using attention scores to select a word in a sentence to mask and then finding a synonym for the masked word.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to PEDRO J MORALES whose telephone number is (571)272-6106. The examiner can normally be reached 8:30 AM - 6:00 PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, MIRANDA M HUANG can be reached at (571)270-7092. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/PEDRO J MORALES/Examiner, Art Unit 2124
/Kevin W Figueroa/Primary Examiner, Art Unit 2124
Prosecution Insights