DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 2024/04/25. The submission is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1–20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Regarding independent Claims 1, 11, and 16
Step 1 — whether the claim falls within any statutory category. See MPEP 2106.03.
Claim 1 is drawn to a method (process) claim; Claim 11 is drawn to a computer-program product embodied in one or more non-transitory machine-readable storage media (manufacture) claim; and Claim 16 is drawn to a system (machine) claim. Therefore, each of these claims falls under one of the four categories of statutory subject matter (process/method, machine/product/apparatus, manufacture, or composition of matter). Step 1: YES.
Step 2A Prong One — whether the claim recites a judicial exception. See MPEP 2106.04, subsection II.
Regarding independent Claim 1, the claim is directed to a computer-implemented method comprising, in relevant part:
The limitations of "determining a first vector embedding for a first term having a highest similarity score, among the second plurality of terms, with a particular vector embedding of a particular term of the first plurality of terms"; "determining a second vector embedding for a second term having a highest similarity score, among the third plurality of terms, with the particular vector embedding of the particular term of the first plurality of terms"; "determining a third vector embedding for a third term having a highest similarity score, among the second plurality of terms, with another particular vector embedding of another particular term of the first plurality of terms"; "determining a fourth vector embedding for a fourth term having a highest similarity score, among the third plurality of terms, with the other particular vector embedding of the other particular term of the first plurality of terms"; "determining a first aggregate similarity score between the first chunk and the second chunk based at least in part on similarity scores between: the particular term and the first term, and the other particular term and the third term"; "determining a second aggregate similarity score between the first chunk and the third chunk based at least in part on similarity scores between: the particular term and the second term, and the other particular term and the fourth term"; and "based at least in part on determining that the first aggregate similarity score satisfies one or more conditions, storing an indication that the first chunk is similar to the second chunk; wherein the second aggregate similarity score does not satisfy the one or more conditions."
These limitations, under their broadest reasonable interpretation, are directed to the abstract idea of comparing terms, selecting the most similar term, computing aggregate similarity scores from the term-to-term scores, and judging whether a similarity score satisfies a condition—that is, evaluating and comparing information—which are concepts that can be performed in the human mind, including observation, evaluation, and judgment, and which fall within the "mental processes" grouping of abstract ideas (see MPEP § 2106.04(a)(2), subsection III). A human being can compare two lists of terms, judge which term is most similar to a given term, average the resulting similarity values, and decide whether the average exceeds a chosen threshold.
The determination of the "highest similarity score" and the "aggregate similarity score" limitations are further directed to the abstract idea of a mathematical concept, specifically the calculation of similarity values (e.g., cosine similarity) and their aggregation through mathematical operations (see MPEP § 2106.04(a)(2), subsection I). The recited computation of similarity scores between vector embeddings and their aggregation constitutes mathematical calculations and relationships.
Independent Claim 11 is a computer-program product claim reciting substantively the same operative limitations as Claim 1 and is directed to the same abstract idea for the same reasons.
Independent Claim 16 is a system claim reciting substantively the same operative limitations as Claim 1 and is directed to the same abstract idea for the same reasons.
Step 2A Prong One: YES, the claims recite a judicial exception.
Step 2A Prong Two — whether the claim as a whole integrates the recited judicial exception into a practical application. See MPEP 2106.04(d).
Regarding independent Claim 1, this claim recites the following additional elements beyond the judicial exception:
"using a masked language model to tune a machine learning model on a corpus of content different than another corpus of content on which the machine learning model was previously trained, wherein using the masked language model to tune the machine learning model causes additional terms to be added to a dictionary of the machine learning model; and wherein the corpus of content includes the additional terms"; "using the machine learning model as tuned to determine a plurality of vector embeddings for a plurality of terms in a plurality of chunks of content"; "tuning a text similarity model to identify similar texts by providing, to the text similarity model, the indication"; and "using the text similarity model to identify content in response to a query."
These additional elements, considered individually and in combination, do not integrate the abstract idea into a practical application. The recited masked language model, machine learning model, and text similarity model amount to generic tools recited at a high level of generality that merely apply or implement the abstract idea of determining similarity, and thus amount to no more than mere instructions to apply the exception using generic computer components (see MPEP § 2106.05(f)). The steps of "using a masked language model to tune a machine learning model" and "using the machine learning model as tuned to determine a plurality of vector embeddings" constitute the generation of data (embeddings) that is then subjected to the abstract similarity-comparison and scoring, and thus constitute insignificant pre-solution data-gathering and data-preparation activity that does not impose a meaningful limit on the exception (see MPEP § 2106.05(g)). The steps of "tuning a text similarity model … by providing … the indication" and "using the text similarity model to identify content in response to a query" likewise link the abstract idea to a particular technological environment (natural-language processing and search) and constitute insignificant post-solution application of the similarity determination (see MPEP § 2106.05(g), (h)).
The claim does not recite a specific improvement to the functioning of a computer or to another technology or technical field. The specification describes the alleged advance as identifying similar chunks of text "without relying on expert-provided labels" (Spec ¶¶ [0022]–[0028]), which is an improvement to the abstract process of determining text similarity itself rather than an improvement to computer functionality or to a technical field; the underlying models (BERT, RoBERTa, SBERT, ColBERT, FAISS) are described as pre-existing, conventional tools employed to carry out the abstract idea (Spec ¶¶ [0033], [0041], [0052], [0061]). Accordingly, the additional elements do not reflect a disclosed technological improvement embodied in the claim. See MPEP § 2106.04(d)(1) and § 2106.05(a).
Regarding independent Claim 11, this claim recites the additional element of "one or more non-transitory machine-readable storage media, including stored instructions configured to cause a computing system to perform a set of actions," in addition to the additional elements recited in common with Claim 1. The recited storage media and computing system amount to no more than generic computer components that merely act as a tool on which the abstract idea is implemented, and generally link the use of the judicial exception to a particular technological environment (see MPEP § 2106.05(f), (h)). Claim 11 therefore fails to integrate the exception into a practical application for the same reasons as Claim 1.
Regarding independent Claim 16, this claim recites the additional elements of "one or more processors" and "one or more non-transitory computer-readable media storing instructions," in addition to the additional elements recited in common with Claim 1. The recited processors and media amount to no more than generic computer components that merely act as a tool on which the abstract idea is implemented, and generally link the use of the judicial exception to a particular technological environment (see MPEP § 2106.05(f), (h)). Claim 16 therefore fails to integrate the exception into a practical application for the same reasons as Claim 1.
Step 2A Prong Two: NO, the additional elements do not integrate the exception into a practical application; the claims are directed to the abstract idea.
Step 2B — whether the claim as a whole amounts to significantly more than the recited judicial exception (an inventive concept). See MPEP 2106.05.
Regarding independent Claims 1, 11, and 16, the additional elements identified above under Step 2A Prong Two are re-evaluated here, individually and in combination, to determine whether they add a specific limitation beyond the judicial exception that is not well-understood, routine, and conventional in the field.
Considered individually, the recited masked language model, machine learning model, text similarity model, storage media, processors, and computing system are described in the specification as generic, pre-existing components. The specification discloses that the masked language model and token-based model are conventional BERT-based/RoBERTa models (Spec ¶¶ [0033], [0035]); that generating paragraph/sentence embeddings via major LLM providers and vector databases is conventional (Spec ¶ [0041]); that the ColBERT similarity technique is a known technique (Spec ¶ [0052]); that "FAISS is a library available on GitHub" for vector-to-vector similarity (Spec ¶ [0061]); and that the base text similarity model is a conventional off-the-shelf model such as "sentence-transformers_all-MiniLM-L12-v2" (Spec ¶ [0067]). These components therefore represent well-understood, routine, and conventional activity, and the steps of generating embeddings, computing similarity, and outputting content in response to a query recite functions that courts have recognized as well-understood, routine, and conventional, including receiving or transmitting data over a network and performing repetitive calculations (see MPEP § 2106.05(d)).
Considered in combination, the additional elements do not amount to significantly more than the abstract idea. The claim merely recites the abstract similarity-determination process performed by generic machine-learning tools in their ordinary capacity, followed by insignificant pre-solution data preparation (embedding generation) and post-solution application (tuning a similarity model and returning query results). The ordered combination adds nothing that is not already present when the elements are considered individually. See MPEP § 2106.05(I).
Step 2B: NO. Claims 1, 11, and 16 do not recite an inventive concept and are ineligible.
Regarding dependent Claims 2–10, 12–15, and 17–20
Step 1
Dependent Claims 2–10 fall within the process category; Claims 12–15 fall within the manufacture category; and Claims 17–20 fall within the machine category. Each falls within a statutory category. Step 1: YES.
Step 2A Prong One
The dependent claims recite the following limitations, which further narrow or elaborate the abstract idea identified above:
Claims 2 recites "masking terms in the other corpus of content, receiving predictions of the machine learning model for the masked terms, and providing feedback to the machine learning model on accuracies of the predictions," which further describes the data-preparation for, and is directed to a mathematical concept and mental process of evaluating prediction accuracy underlying, the abstract idea.
Claims 3, 12, and 17 recite that the similarity scores "are each determined based at least in part on cosine similarity," which is directed to the abstract idea of a mathematical concept (cosine similarity calculation). See MPEP § 2106.04(a)(2), subsection I.
Claims 6, 13, and 18 recite "averaging similarity scores between terms," which is directed to the abstract idea of a mathematical concept (averaging). See MPEP § 2106.04(a)(2), subsection I.
Claims 8, 15, and 20 recite that "the one or more conditions comprise a similarity threshold" and that a chunk is or is not marked similar/dissimilar based thereon, which is directed to the abstract idea of a mental process of evaluating and judging information against a threshold. See MPEP § 2106.04(a)(2), subsection III.
Claims 9, 15, and 20 recite "storing another indication that the first chunk is dissimilar to the third chunk" based on the second aggregate score satisfying other conditions, which is directed to the abstract idea of a mental process of evaluating and judging similarity/dissimilarity. See MPEP § 2106.04(a)(2), subsection III.
Claims 5 recites the other corpus consisting of publicly available text sources and the corpus comprising domain-specific text from an access-restricted private database, which further describes the nature of the data operated upon and does not remove the claim from the abstract idea.
Claims 7, 14, and 19 recite "accessing an index of similar chunks to determine that the second chunk is similar to a fourth chunk" and storing/providing a further indication, which is directed to the abstract idea of a mental process of comparing and evaluating information, together with insignificant data-retrieval activity.
Claims 10 recites receiving a natural-language query via a user interface, "ranking two or more candidate results … based on how similar text in the two or more candidate results are to the query," and causing display of a reference to at least one candidate result, which is directed to the abstract idea of evaluating, comparing, and ranking information (a mental process) together with insignificant extra-solution query-input and display activity.
The dependent claims therefore recite the same judicial exceptions as the independent claims from which they depend. Step 2A Prong One: YES.
Step 2A Prong Two
Claims 2–10, 12–15, and 17–20 recite the additional elements of the masked language model, machine learning model, text similarity model, index, user interface, display, processing device(s), storage media, and processor(s). For the reasons described above with respect to the independent claims, these additional elements amount to no more than generic computer components that merely act as a tool on which the abstract idea operates, mere instructions to apply the exception, insignificant pre- and post-solution activity, and generally linking the exception to a particular technological environment (see MPEP § 2106.05(f), (g), (h)). The dependent claims do not recite a specific technological improvement and merely narrow the previously cited abstract-idea limitations. Accordingly, these judicial exceptions are not meaningfully integrated into a practical application. Step 2A Prong Two: NO.
Step 2B
For the reasons described above with respect to the independent claims, the dependent claims do not provide anything more than the mathematical concepts and mental processes that are practically capable of being performed in the human mind, with the assistance of pen and paper, carried out by generic, well-understood, routine, and conventional computer components (Spec ¶¶ [0033], [0041], [0052], [0061], [0067]). The additional recited elements—including cosine-similarity computation, averaging, threshold comparison, index lookup, query input, ranking, and display—are recited at a high level of generality and represent well-understood, routine, and conventional activity that does not, individually or in ordered combination, amount to significantly more than the judicial exception (see MPEP § 2106.05(d)). Therefore, Claims 2–10, 12–15, and 17–20 also fail to recite an inventive concept. Step 2B: NO.
Accordingly, Claims 1–20 are rejected under 35 U.S.C. § 101 as being directed to an abstract idea without significantly more.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The text of those sections of Title 35, U.S. Code not included in this action can be found in a prior Office action.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1, 3, 5, 6, 10-13, and 16-18 are rejected under 35 U.S.C. 103 as being unpatentable over Malkiel et al. (Malkiel), US 11,580,764 B2, in view of Khattab et al. (Khattab), Non-Patent Literature, "ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT," arXiv:2004.12832v2 [cs.IR], published on 4 June 2020 and cited in the IDS filed on 4/25/2024, and further in view of Tai et al. (Tai), Non-Patent Literature, "exBERT: Extending Pre-trained Models with Domain-specific Vocabulary Under Constrained Training Resources.", published November 2020.
Regarding independent Claim 1, (Malkiel) teaches a computer-implemented method comprising:
"using a masked language model to tune a machine learning model on a corpus of content different than another corpus of content on which the machine learning model was previously trained" — (Malkiel) teaches a masked language model 402 whose feedback tunes a previously trained machine learning model on a different corpus, disclosing that "the masked language model 402 applies random masking 404 to portions of a document" and that "[f]eedback from the masked language model 402 fine-tunes contextual word embeddings generated by the language model 102 to specialize in a given domain 408 associated with a collection of documents 406 during a pre-training phase" (Malkiel, col. 7, ll. 59–65). The machine learning model was previously trained on another corpus of content because (Malkiel) "adopts the RoBERTa language model as a backbone and continues the training of the RoBERTa model on the collection of documents" (Malkiel, col. 8, ll. 58–61), and "[t]he first term is a masked language model (MLM) adopted from BERT. The MLM loss allows the model to specialize in the given collection of documents" (Malkiel, col. 9, ll. 3–6). The pre-trained RoBERTa/BERT backbone corresponds to the recited "machine learning model … previously trained" on "another corpus of content," and the domain-specific "collection of documents" on which training is continued corresponds to the recited "corpus of content."
"using the machine learning model as tuned to determine a plurality of vector embeddings for a plurality of terms in a plurality of chunks of content from a particular corpus of content that is different than the other corpus of content on which the machine learning model was previously trained; wherein the plurality of chunks of content comprises a first chunk, a second chunk, and a third chunk; wherein the first chunk comprises a first plurality of terms, the second chunk comprises a second plurality of terms, and the third chunk comprises a third plurality of terms" — (Malkiel) teaches using the tuned model to determine vector embeddings for the terms of plural chunks, disclosing that the system "computes a feature vector for each sentence by average pooling the token embeddings associated with each sentence separately," and that "[t]he token embeddings are the output of the last encoder layer of the model" (Malkiel, col. 9, ll. 8–11). As to the recited "vector embeddings," each such token embedding is a numerical vector produced by the model's last encoder layer to represent an individual token (term) of the text; this corresponds to the recited "vector embedding," which the present specification defines as "numerical or otherwise deterministically comparable values, such as values combined in a vector form, that describe content, such as a token in the case of a token or word embedding" (Spec ¶ [0034]). As to the recited first, second, and third chunks each comprising a respective plurality of terms, (Malkiel) operates on a source document and plural candidate documents each divided into paragraphs composed of plural sentences and tokens (Malkiel, col. 5, ll. 60–67), and the source document is drawn from the domain-specific corpus on which the model was tuned rather than from the corpus on which the backbone was previously trained (Malkiel, col. 8, ll. 58–61).
"determining a first aggregate similarity score between the first chunk and the second chunk based at least in part on similarity scores between: the particular term and the first term, and the other particular term and the third term" — (Malkiel) teaches determining an aggregate similarity score between two chunks from the constituent similarity scores of their matched units, disclosing that the system creates a "paragraph similarity matrix … by aggregating all the sentence similarity scores for a given document" (Malkiel, col. 6, ll. 1–6) and "aggregates all paragraph-similarity scores for each paragraph in the source document" (Malkiel, col. 10, ll. 25–27), wherein the aggregation is performed by combining the similarity scores of the most-similar matched pairs and normalizing them, such that "[e]ach row i in P_ij* is z-score normalized by a mean and standard deviation computed from the row i values" (Malkiel, col. 12, ll. 5–10; col. 12, ll. 25–40). The aggregate score so computed between the source chunk and a first candidate chunk corresponds to the recited "first aggregate similarity score."
"determining a second aggregate similarity score between the first chunk and the third chunk based at least in part on similarity scores between: the particular term and the second term, and the other particular term and the fourth term" — (Malkiel) teaches computing the same aggregate score for each additional candidate chunk, disclosing that "the system computes the above paragraph-similarity matrix for every candidate document c∈D" (Malkiel, col. 12, ll. 14–17) and aggregates the matched similarity scores for each such candidate (Malkiel, col. 12, ll. 25–40). The aggregate score computed between the source chunk and a second candidate chunk corresponds to the recited "second aggregate similarity score."
"based at least in part on determining that the first aggregate similarity score satisfies one or more conditions, storing an indication that the first chunk is similar to the second chunk; wherein the second aggregate similarity score does not satisfy the one or more conditions" — (Malkiel) teaches storing an indication of similarity for chunk pairs whose aggregate score satisfies a condition while withholding that indication for pairs that do not, disclosing that "similar documents are placed into a first tier 506 while less similar or unsimilar documents are placed into a second tier" (Malkiel, col. 8, ll. 18–24), wherein the condition is a threshold on the similarity metric defined by a "predefined positive margin 'm'" set such that dissimilar pairs are driven to "a cosine similarity that is less than or equal to 0" (Malkiel, col. 9, ll. 28–36). Placement of the satisfying pair in the first tier, and of the non-satisfying pair in the second tier, corresponds to storing an indication that the first chunk is similar to the second chunk where the second aggregate similarity score does not satisfy the condition.
"tuning a text similarity model to identify similar texts by providing, to the text similarity model, the indication" — (Malkiel) teaches providing the stored similarity indications to a downstream text similarity model to tune it, disclosing that "SBERT … is evaluated with (1) its original weights and (2) after fine-tuning with the pseudo labels" (Malkiel, col. 14, ll. 5–10), and distinguishing "vanilla SBERT (without continuing training on each dataset by utilizing our pseudo-labels)" (Malkiel, col. 15, ll. 7–10). SBERT corresponds to the recited "text similarity model," and the pseudo-labels generated by the self-supervised similarity determination correspond to the recited "indication."
"using the text similarity model to identify content in response to a query" — (Malkiel) teaches using the tuned text similarity model to identify content responsive to a query, disclosing "ranking a collection of documents according to their semantic similarity to a source (query) document" (Malkiel, col. 3, ll. 50–58; claim 8), wherein "[t]he source document 110 can also be referred to as a seed document or a query document" (Malkiel, col. 4, ll. 65–67).
(Malkiel) teaches something related to determining a vector embedding for a term having a highest similarity score with a term of another chunk, in that (Malkiel) selects a maximally similar unit between two chunks by computing "the cosine similarity between every pair of sentence embedding, forming a sentence-similarity matrix" (Malkiel, col. 10, ll. 1–5) and then "matching between the most similar paragraphs" (Malkiel, col. 12, ll. 25–40). (Malkiel) performs this highest-similarity selection between sentence-level and paragraph-level embeddings, each of which is formed by "average pooling the token embeddings" (Malkiel, col. 9, ll. 8–10), rather than between the embeddings of individual terms. However, (Malkiel) does not teach:
"determining a first vector embedding for a first term having a highest similarity score, among the second plurality of terms, with a particular vector embedding of a particular term of the first plurality of terms";
"determining a second vector embedding for a second term having a highest similarity score, among the third plurality of terms, with the particular vector embedding of the particular term of the first plurality of terms";
"determining a third vector embedding for a third term having a highest similarity score, among the second plurality of terms, with another particular vector embedding of another particular term of the first plurality of terms";
"determining a fourth vector embedding for a fourth term having a highest similarity score, among the third plurality of terms, with the other particular vector embedding of the other particular term of the first plurality of terms."
In the same field of endeavor, (Khattab) teaches these limitations.
By way of explanation of the recited terminology as it maps onto (Khattab): (Khattab) represents each text as a set of per-term vector embeddings rather than as a single pooled vector, disclosing that the relevance score of one text to another "is estimated via late interaction between their bags of contextualized embeddings" (Khattab, § 3.2, p. 4), denoted E_q for the terms of one text and E_d for the terms of the other. Each element E_{q_i} is the vector embedding of the i-th term of the first text, and each element E_{d_j} is the vector embedding of the j-th term of the second text; these per-term vectors correspond to the recited "vector embedding" of a "term." (Khattab) further teaches selecting, for a given term of the first text, the single term of the second text whose vector embedding yields the largest similarity value, disclosing that "every query embedding interacts with all document embeddings via a MaxSim operator, which computes maximum similarity (e.g., cosine similarity), and the scalar outputs of these operators are summed across query terms" (Khattab, § 3, p. 3), formally expressed as S_{q,d} := Σ_{i∈[|E_q|]} max_{j∈[|E_d|]} E_{q_i} · E_{d_j}^T (Khattab, § 3.2, Eq. (3), p. 4). The inner operator "max_{j∈[|E_d|]}" evaluates the similarity between the fixed term embedding E_{q_i} and every term embedding E_{d_j} of the other text, and returns the single largest such value; the term of the other text that produces that largest value is the term "having a highest similarity score" with the given term, and that term's embedding is the corresponding "vector embedding" that is determined. (Khattab) confirms that the similarity so maximized is a cosine similarity, disclosing that the computation "is conducted as a sum of maximum similarity computations, namely cosine similarity (implemented as dot-products due to the embedding normalization)" (Khattab, § 3.2, p. 4).
Applying the foregoing to each recited limitation:
As to "determining a first vector embedding for a first term having a highest similarity score, among the second plurality of terms, with a particular vector embedding of a particular term of the first plurality of terms," the particular term of the first chunk corresponds to a term of the first text having vector embedding E_{q_i}; the MaxSim operator of (Khattab) evaluates the cosine similarity of E_{q_i} against every term embedding of the second chunk and selects the maximum (Khattab, § 3.2, Eq. (3), p. 4); the term of the second chunk producing that maximum is the first term, and its embedding is the first vector embedding so determined.
As to "determining a second vector embedding for a second term having a highest similarity score, among the third plurality of terms, with the particular vector embedding of the particular term of the first plurality of terms," (Khattab) applies the same MaxSim operator between the query text and each separate document text (Khattab, § 3, p. 3; § 3.2, Eq. (3), p. 4); performed against the third chunk, the operator selects the term of the third chunk yielding the maximum cosine similarity with the same particular term embedding E_{q_i}, that term being the second term and its embedding the second vector embedding.
As to "determining a third vector embedding for a third term having a highest similarity score, among the second plurality of terms, with another particular vector embedding of another particular term of the first plurality of terms," (Khattab) performs the MaxSim operation for every term of the first text, as the summation in Eq. (3) runs over all i ∈ [|E_q|] (Khattab, § 3.2, Eq. (3), p. 4); for another particular term of the first chunk having embedding E_{q_i′}, the operator selects the term of the second chunk yielding the maximum cosine similarity, that term being the third term and its embedding the third vector embedding.
As to "determining a fourth vector embedding for a fourth term having a highest similarity score, among the third plurality of terms, with the other particular vector embedding of the other particular term of the first plurality of terms," the same per-term MaxSim operation of (Khattab) performed for the other particular term embedding E_{q_i′} against the third chunk selects the term of the third chunk yielding the maximum cosine similarity, that term being the fourth term and its embedding the fourth vector embedding (Khattab, § 3.2, Eq. (3), p. 4).
(Malkiel) and (Khattab) are analogous to the claimed invention as both are from the same field of endeavor of natural-language text-similarity determination using contextualized transformer (BERT-based) vector embeddings and cosine similarity. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to combine the self-supervised chunk-to-chunk similarity scoring of (Malkiel) with the per-term maximum-similarity (MaxSim) operator of (Khattab), by performing (Malkiel)'s highest-similarity matching upon the individual term embeddings as taught by (Khattab) rather than upon pooled sentence embeddings. The motivation to combine (Malkiel) and (Khattab) is provided by (Khattab), which teaches that "the fine-grained matching of interaction-based models and the precomputation of document representations of representation-based models can be combined by retaining yet judiciously delaying the query–document interaction" (Khattab, § 3, p. 3), thereby preserving the term-level matching signal that is otherwise lost when term embeddings are pooled into a single vector, and yielding more accurate similarity determination—a predictable improvement to (Malkiel), which itself already computes cosine similarity between embeddings (Malkiel, col. 11, ll. 23–30) and which recognizes that representing longer text by coarser pooled representations limits similarity accuracy (Malkiel, col. 1, ll. 15–25).
The combination of (Malkiel) and (Khattab), however, does not teach:
"wherein using the masked language model to tune the machine learning model causes additional terms to be added to a dictionary of the machine learning model; and wherein the corpus of content includes the additional terms."
In the same field of endeavor, (Tai) teaches this limitation. (Tai) teaches extending a pre-trained BERT model to a specialized domain "with a new additive vocabulary" (Tai, Abstract, p. 1), disclosing that "we derive an extension vocabulary from the target domain … corpus via WordPiece, while keeping the original general vocabulary used by BERT unchanged," that "[w]e then add a corresponding embedding layer for the extension vocabulary, which is randomly initialized at the beginning and can be optimized during pre-training," and that "[t]he overall vocabulary, containing 30,522 (original) and 17,748 (extension) tokens, is used for tokenizing input text" (Tai, § 3.1, p. 2). As to "causes additional terms to be added to a dictionary of the machine learning model," the derivation of the extension vocabulary and the addition of its embedding layer to the model's overall tokenizing vocabulary, optimized during the masked-language-model pre-training phase, corresponds to additional terms being added to the model's dictionary as a result of the masked-language-model tuning. As to "wherein the corpus of content includes the additional terms," (Tai) derives the extension vocabulary from the target-domain corpus itself (Tai, § 3.1, p. 2), such that the corpus on which the model is tuned contains the additional terms so added.
(Malkiel), (Khattab), and (Tai) are analogous to the claimed invention as all are from the same field of endeavor of domain adaptation of BERT-based language models for text representation and similarity. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to combine the masked-language-model domain specialization of (Malkiel) and the per-term embedding matching of (Khattab) with the additive domain vocabulary of (Tai), by adding domain-specific terms to the dictionary of the machine learning model during (Malkiel)'s masked-language-model tuning as taught by (Tai). The motivation to combine (Malkiel), (Khattab), and (Tai) is provided by (Tai), which teaches that "[a] specialized domain … has its own" vocabulary that the general pre-trained model does not cover (Tai, § 1, p. 1), such that adding a domain-specific additive vocabulary "achieves improved performance in several … downstream tasks" (Tai, § 2, p. 2). One of ordinary skill would therefore have been motivated to add (Tai)'s additive-vocabulary step to (Malkiel)'s masked-language-model tuning in order to represent domain terminology absent from the general vocabulary, with a reasonable expectation of the predictable result of improved domain-specific text representation and, in turn, more accurate similarity determination.
Regarding Claim 3, the computer-implemented method of Claim 1, from which Claim 3 depends, is rejected under the same rationale as set forth above with respect to Claim 1 over (Malkiel) in view of (Khattab) and further in view of (Tai).
Claim 3 additionally recites the following sub-limitations:
"a first similarity score between the first vector embedding and the particular vector embedding … is determined based at least in part on cosine similarity";
"a second similarity score between the second vector embedding and the particular vector embedding … is determined based at least in part on cosine similarity";
"a third similarity score between the third vector embedding and the other particular vector embedding … is determined based at least in part on cosine similarity";
"a fourth similarity score between the fourth vector embedding and the other particular vector embedding … is determined based at least in part on cosine similarity."
(Malkiel) teaches determining a similarity score between vector embeddings based at least in part on cosine similarity, disclosing that "SDR computes the cosine similarity between every pair of … embedding" (Malkiel, col. 10, ll. 1–5), and that "[t]he term C(f_p, f_q) measures the angular distance between f_p and f_q using the Cosine function," C(f_p, f_q) = f_p·f_q / (‖f_p‖‖f_q‖) (Malkiel, col. 11, ll. 23–30). (Malkiel) thus teaches determining a similarity score between two vector embeddings based at least in part on cosine similarity. However, (Malkiel) computes this cosine similarity between sentence-level embeddings rather than between the individual term vector embeddings recited in the four sub-limitations above, and therefore (Malkiel) does not teach:
"a first similarity score between the first vector embedding and the particular vector embedding … is determined based at least in part on cosine similarity";
"a second similarity score between the second vector embedding and the particular vector embedding … is determined based at least in part on cosine similarity";
"a third similarity score between the third vector embedding and the other particular vector embedding … is determined based at least in part on cosine similarity";
"a fourth similarity score between the fourth vector embedding and the other particular vector embedding … is determined based at least in part on cosine similarity."
In the same field of endeavor, (Khattab) teaches these sub-limitations. As set forth above with respect to Claim 1, the first, second, third, and fourth vector embeddings, and their determination as the terms of highest similarity with the particular and other particular vector embeddings of the first chunk, are taught by (Khattab)'s term-level maximum-similarity (MaxSim) operator. (Khattab) further teaches that each such similarity score is determined based at least in part on cosine similarity:
As to the first sub-limitation, (Khattab) teaches that "every query embedding interacts with all document embeddings via a MaxSim operator, which computes maximum similarity (e.g., cosine similarity)" (Khattab, § 3, p. 3); the maximum-cosine-similarity value between the particular term embedding of the first chunk and the first term embedding of the second chunk reads on the first similarity score determined based at least in part on cosine similarity.
As to the second sub-limitation, (Khattab) teaches computing "the maximum cosine similarity of each v ∈ E_q with vectors in E_d" (Khattab, § 3.2, p. 4); applied to the particular term embedding of the first chunk against the third chunk, the resulting maximum cosine similarity reads on the second similarity score determined based at least in part on cosine similarity.
As to the third sub-limitation, (Khattab) teaches that the relevance score "is conducted as a sum of maximum similarity computations, namely cosine similarity (implemented as dot-products due to the embedding normalization)" (Khattab, § 3.2, p. 4); applied to the other particular term embedding of the first chunk against the second chunk, the resulting maximum cosine similarity reads on the third similarity score determined based at least in part on cosine similarity.
As to the fourth sub-limitation, (Khattab) teaches the same cosine-based MaxSim computation (Khattab, § 3.2, Eq. (3), p. 4); applied to the other particular term embedding of the first chunk against the third chunk, the resulting maximum cosine similarity reads on the fourth similarity score determined based at least in part on cosine similarity.
(Malkiel) and (Khattab) are analogous to the claimed invention as both are from the same field of endeavor of natural-language text-similarity determination using contextualized transformer (BERT-based) embeddings and cosine similarity. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to combine the self-supervised document-similarity scoring of (Malkiel) with the term-level cosine-similarity (MaxSim) determination of (Khattab), by determining each of the first, second, third, and fourth similarity scores based at least in part on cosine similarity as taught by (Khattab). The motivation to combine (Malkiel) and (Khattab) is provided by (Khattab), which teaches that computing maximum cosine similarity at the term level provides "fine-grained matching" that "can be combined by retaining yet judiciously delaying the query–document interaction" (Khattab, § 3, p. 3), thereby yielding finer-grained and more accurate similarity scoring—a predictable improvement to (Malkiel), which itself already determines similarity scores using the cosine function (Malkiel, col. 11, ll. 23–30) and concedes that pooling text into coarser representations limits accuracy for longer text (Malkiel, col. 1, ll. 15–25).
The combination of (Malkiel) and (Khattab) teaches all of the sub-limitations added by Claim 3; (Tai) is relied upon with respect to Claim 3 for the reasons set forth in the rejection of Claim 1, from which Claim 3 depends, and the rationale and motivation for combining (Malkiel), (Khattab), and (Tai) are the same as those set forth above with respect to Claim 1.
Regarding Claim 5, the computer-implemented method of Claim 1, from which Claim 5 depends, is rejected under the same rationale as set forth above with respect to Claim 1 over (Malkiel) in view of (Khattab) and further in view of (Tai).
(Malkiel) teaches "the other corpus of content" on which the machine learning model was previously trained and the domain-specific "corpus of content" on which the model is tuned, disclosing that the model "adopts the RoBERTa language model as a backbone and continues the training of the RoBERTa model on the collection of documents" (Malkiel, col. 8, ll. 58–61), and that the masked language model fine-tunes the model "to specialize in a given domain 408 associated with a collection of documents 406" (Malkiel, col. 7, ll. 62–65). However, (Malkiel) does not expressly teach the following sub-limitations of Claim 5:
"wherein the other corpus of content consists of publicly available text sources"; and
"wherein the corpus of content comprises domain-specific text sources from an access-restricted private database."
In the same field of endeavor, (Tai) teaches these sub-limitations.
As to "wherein the other corpus of content consists of publicly available text sources," (Tai) teaches that the base BERT model on which exBERT builds was pre-trained on "a general language corpus (English Wikipedia and BooksCorpus)" (Tai, § 2, p. 2), English Wikipedia and BooksCorpus being publicly available text sources. The general language corpus of publicly available English Wikipedia and BooksCorpus reads on "the other corpus of content consists of publicly available text sources."
As to "wherein the corpus of content comprises domain-specific text sources from an access-restricted private database," (Tai) teaches pre-training the extended model on domain-specific biomedical sources, disclosing that "[w]e pre-train exBERT with biomedical articles from ClinicalKey and PubMed Central" (Tai, § 1, p. 1), wherein ClinicalKey is an access-restricted, subscription-based clinical database, and describing that "[a] specialized domain, such as the biomedical domain … has its own" domain-specific vocabulary and text (Tai, § 1, p. 1). The domain-specific biomedical text sources drawn from the access-restricted ClinicalKey database read on "the corpus of content comprises domain-specific text sources from an access-restricted private database."
(Malkiel) and (Tai) are analogous to the claimed invention as both are from the same field of endeavor of domain adaptation of BERT-based language models for text representation and similarity. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to combine the masked-language-model domain specialization of (Malkiel) with the publicly-available general corpus and access-restricted domain-specific corpus taught by (Tai), by pre-training the base model of (Malkiel) on publicly available text sources and tuning the model on domain-specific text sources from an access-restricted private database as taught by (Tai). The motivation to combine (Malkiel) and (Tai) is provided by (Tai), which teaches that a specialized domain "has its own" vocabulary and text not adequately covered by a model trained only on the general public corpus (Tai, § 1, p. 1), such that pre-training on the general public corpus and then specializing on the domain-specific corpus "achieves improved performance in several … downstream tasks" (Tai, § 2, p. 2)—the same problem the present specification identifies at ¶ [0042], which describes domain-specific content that "may be private, access-restricted, or non-published." One of ordinary skill would therefore have been motivated to draw the domain-specific corpus of (Malkiel) from an access-restricted private database as taught by (Tai), with a reasonable expectation of the predictable result of improved domain-specific text representation.
Regarding Claim 6, the computer-implemented method of Claim 1, from which Claim 6 depends, is rejected under the same rationale as set forth above with respect to Claim 1 over (Malkiel) in view of (Khattab) and further in view of (Tai).
(Malkiel) teaches determining an aggregate similarity score between two chunks by averaging the underlying similarity scores, disclosing that the model "aggregates all paragraph-similarity scores for each paragraph in the source document" (Malkiel, col. 10, ll. 25–27), wherein the aggregation is normalized by a mean such that "[e]ach row i in P_ij* is z-score normalized by a mean and standard deviation computed from the row i values" (Malkiel, col. 12, ll. 5–10), and that the feature vectors are formed "by average pooling the token embeddings associated with each sentence separately" (Malkiel, col. 9, ll. 8–10). (Malkiel) thus teaches averaging similarity scores to determine an aggregate similarity score between two chunks; however, (Malkiel) performs this averaging over sentence-level similarity scores rather than over similarity scores between individual terms, and therefore (Malkiel) does not teach the following sub-limitations:
"wherein determining the first aggregate similarity score between the first chunk and the second chunk comprises averaging similarity scores between terms in the first chunk and terms in the second chunk"; and
"wherein determining the second aggregate similarity score between the first chunk and the third chunk comprises averaging similarity scores between terms in the first chunk and terms in the third chunk."
In the same field of endeavor, (Khattab) teaches these sub-limitations. (Khattab) teaches determining an aggregate similarity score between two texts from the similarity scores between individual terms, disclosing that "every query embedding interacts with all document embeddings via a MaxSim operator, which computes maximum similarity (e.g., cosine similarity), and the scalar outputs of these operators are summed across query terms" (Khattab, § 3, p. 3), formally expressed as the aggregate relevance score S_{q,d} := Σ_i max_j (E_{q_i} · E_{d_j}) (Khattab, § 3.2, Eq. (3), p. 4). Applied to the first sub-limitation, the term-to-term similarity scores between the terms of the first chunk and the terms of the second chunk are aggregated across the terms of the first chunk to determine the first aggregate similarity score; and applied to the second sub-limitation, the term-to-term similarity scores between the terms of the first chunk and the terms of the third chunk are aggregated across the terms of the first chunk to determine the second aggregate similarity score. The averaging of the constituent term-to-term similarity scores taught by (Malkiel) (col. 9, ll. 8–10; col. 12, ll. 5–10) applied to the term-to-term similarity scores taught by (Khattab) reads on averaging the similarity scores between terms in the respective chunks.
(Malkiel) and (Khattab) are analogous to the claimed invention as both are from the same field of endeavor of natural-language text-similarity determination using contextualized transformer (BERT-based) embeddings and cosine similarity. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to combine the averaging aggregation of (Malkiel) with the term-to-term similarity scoring of (Khattab), by averaging the term-to-term similarity scores taught by (Khattab) to determine the first and second aggregate similarity scores as taught by (Malkiel). The motivation to combine (Malkiel) and (Khattab) is provided by (Khattab), which teaches that aggregating term-level maximum-similarity scores provides "fine-grained matching" that "can be combined by retaining yet judiciously delaying the query–document interaction" (Khattab, § 3, p. 3), thereby yielding finer-grained and more accurate aggregate similarity scores—a predictable improvement to (Malkiel), which itself already averages constituent similarity scores to determine an aggregate score (Malkiel, col. 12, ll. 5–10) and concedes that pooling text into coarser representations limits accuracy for longer text (Malkiel, col. 1, ll. 15–25).
The combination of (Malkiel) and (Khattab) teaches all of the sub-limitations added by Claim 6; (Tai) is relied upon with respect to Claim 6 for the reasons set forth in the rejection of Claim 1, from which Claim 6 depends, and the rationale and motivation for combining (Malkiel), (Khattab), and (Tai) are the same as those set forth above with respect to Claim 1.
Regarding Claim 10, the computer-implemented method of Claim 1, from which Claim 10 depends, is rejected under the same rationale as set forth above with respect to Claim 1 over (Malkiel) in view of (Khattab) and further in view of (Tai).
(Malkiel) teaches the following sub-limitations of Claim 10:
As to "wherein the query is a natural language query, the computer-implemented method further comprising: receiving, via a user interface, the query," (Malkiel) teaches receiving a natural-language query via a user interface, disclosing that the source document is a query selected by a user—"[t]he source document 110 can also be referred to as a seed document or a query document" (Malkiel, col. 4, ll. 65–67)—that "[i]n this example, the text is written in natural language" (Malkiel, col. 5, ll. 32–33), and that the system receives the source from and outputs to "a user associated with the source document via a user interface device" (Malkiel, col. 15, ll. 74–76; claim 15).
As to "wherein using the text similarity model to identify content in response to the query comprises: using the text similarity model, ranking two or more candidate results of a plurality of candidate results to the query based on how similar text in the two or more candidate results are to the query," (Malkiel) teaches ranking candidate results based on their similarity to the query document, disclosing that the model "rank[s] each document in the plurality of documents in accordance with the degree of semantic similarity to the source document based on the inferred similarity scores" (Malkiel, col. 6, ll. 60–67; col. 13, l. 113), such that "the candidate document 106 is given a higher rank and/or identified as being more relevant/similar to the source document" (Malkiel, col. 6, ll. 15–20).
As to "based at least in part on the ranking, causing display of a reference to at least one of the two or more candidate results of the plurality of candidate results to the query," (Malkiel) teaches causing display of a recommendation of at least one ranked candidate, disclosing "generating a recommendation including a subset of documents from the plurality of documents based on a rank assigned to each document" and "outputting the recommendation to a user associated with the source document via a user interface device" (Malkiel, col. 24, ll. 120–128; claim 15), wherein the recommendation is presented "visually in a graphical user interface (GUI)" using "a display device" (Malkiel, col. 17, ll. 106; col. 17, ll. 76).
Claim 11, an independent computer-program-product claim reciting limitations corresponding to those of Claim 1, is rejected under 35 U.S.C. 103 under the same rationale as set forth above with respect to Claim 1.
Claim 12 is rejected under 35 U.S.C. 103 under the same rationale as set forth above with respect to Claim 3.
Claim 13 is rejected under 35 U.S.C. 103 under the same rationale as set forth above with respect to Claim 6.
Claim 16, an independent system claim reciting limitations corresponding to those of Claims 1 and 11, is rejected under 35 U.S.C. 103 under the same rationale as set forth above with respect to Claim 1.
Claim 17 is rejected under 35 U.S.C. 103 under the same rationale as set forth above with respect to Claims 3 and 12.
Claim 18 is rejected under 35 U.S.C. 103 under the same rationale as set forth above with respect to Claims 6 and 13.
Claims 2 and 4 are rejected under 35 U.S.C. 103 as being unpatentable over Malkiel et al. (Malkiel), US 11,580,764 B2, in view of Khattab et al. (Khattab), Non-Patent Literature, "ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT," arXiv:2004.12832v2 [cs.IR], published on 4 June 2020 and cited in the IDS filed on 4/25/2024, in view of Tai et al. (Tai), Non-Patent Literature, "exBERT: Extending Pre-trained Models with Domain-specific Vocabulary Under Constrained Training Resources," published November 2020, and further in view of Devlin et al. (Devlin), Non-Patent Literature, "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding," arXiv:1810.04805v2 [cs.CL], published on 24 May 2019 and cited in the IDS filed on 4/25/2024.
Regarding Claim 2, the computer-implemented method of Claim 1, from which Claim 2 depends, is rejected under the same rationale as set forth above with respect to Claim 1 over (Malkiel) in view of (Khattab) and further in view of (Tai).
Claim 2 additionally recites the following limitation:
"wherein using the masked language model to tune the machine learning model comprises masking terms in the other corpus of content, receiving predictions of the machine learning model for the masked terms, and providing feedback to the machine learning model on accuracies of the predictions."
(Malkiel) teaches this limitation. As to "masking terms in the other corpus of content," (Malkiel) discloses that "the masked language model 402 applies random masking 404 to portions of a document" (Malkiel, col. 7, ll. 50-52), and that "the sentences in each pair are then tokenized, aggregated into batches, and randomly masked in a similar way to the RoBERTa pre-training paradigm" (Malkiel, col. 9, ll. 1–3). As to "receiving predictions of the machine learning model for the masked terms," (Malkiel) discloses that "the language model 102 attempts to reconstruct the masked portions" (Malkiel, col. 7, ll. 60–61). As to "providing feedback to the machine learning model on accuracies of the predictions," (Malkiel) discloses that "feedback from the masked language model 402 fine-tunes contextual word embeddings generated by the language model 102 to specialize in a given domain" (Malkiel, col. 7, ll. 61–64), wherein the masked-language-model loss (MLM loss) provides the feedback that adjusts the model based on the correctness of the reconstructed tokens (Malkiel, col. 9, ll. 3–6: "[t]he first term is a masked language model (MLM) adopted from BERT. The MLM loss allows the model to specialize in the given collection of documents").
(Malkiel) teaches masking, reconstruction of the masked portions, and feedback via the MLM loss as set forth above; however, to the extent applicant contends that (Malkiel) does not expressly teach "receiving predictions of the machine learning model for the masked terms, and providing feedback to the machine learning model on accuracies of the predictions" as a supervised mask-predict-and-score operation, this limitation is further taught by (Devlin). In the same field of endeavor, (Devlin) teaches masking terms, receiving predictions for the masked terms, and providing feedback on the accuracies of those predictions, disclosing that "we mask 15% of all WordPiece tokens in each sequence at random" and "only predict the masked words" (Devlin, § 3.1, p. 4), wherein "the final hidden vectors corresponding to the mask tokens are fed into an output softmax over the vocabulary" (Devlin, § 3.1, p. 4) to produce the predictions, and "T_i will be used to predict the original token with cross entropy loss" (Devlin, § 3.1, p. 4), the cross-entropy loss constituting the feedback provided to the model on the accuracy of the predicted masked terms relative to the original tokens.
(Malkiel) and (Devlin) are analogous to the claimed invention as both are from the same field of endeavor of masked-language-model pre-training of BERT-based language models for text representation. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to combine the masked-language-model tuning of (Malkiel) with the mask-predict-and-cross-entropy-feedback mechanism of (Devlin), by implementing (Malkiel)'s masked-language-model feedback as the token prediction and cross-entropy scoring taught by (Devlin). The motivation to combine (Malkiel) and (Devlin) is provided by (Devlin), which teaches that the masked-language-model objective of predicting masked tokens and updating the model by cross-entropy loss "allows us to obtain a … pre-trained model" that yields improved bidirectional text representations (Devlin, § 3.1, p. 4), and further because (Malkiel) itself expressly adopts its masked language model "from BERT" (Malkiel, col. 9, ll. 3–4), such that one of ordinary skill would have looked to (Devlin)'s foundational description of the BERT masked-language-model mask-predict-feedback procedure to implement (Malkiel)'s masked-language-model tuning, with a reasonable expectation of the predictable result of a domain-specialized language model.
Regarding Claim 4, the computer-implemented method of Claim 1, from which Claim 4 depends, is rejected under the same rationale as set forth above with respect to Claim 1 over (Malkiel) in view of (Khattab) and further in view of (Tai).
Claim 4 additionally recites the following limitation:
"wherein the machine learning model comprises a Bidirectional Encoder Representations from Transforms (BERT)-based uncased token-based model."
(Malkiel) teaches a machine learning model comprising a BERT-based token-based model, disclosing that "[t]he first term is a masked language model (MLM) adopted from BERT" (Malkiel, col. 9, ll. 3–4) and that the model "adopts the RoBERTa language model as a backbone and continues the training of the RoBERTa model on the collection of documents" (Malkiel, col. 8, ll. 58–61), RoBERTa being a BERT-based model that operates on tokens (Malkiel, col. 9, ll. 1–3, "[t]he sentences in each pair are then tokenized … and randomly masked in a similar way to the RoBERTa pre-training paradigm"). (Malkiel) teaches a BERT-based token-based model as set forth above; however, (Malkiel) does not expressly teach that the BERT-based token-based model is an "uncased" model.
In the same field of endeavor, (Devlin) teaches the added limitation. (Devlin) discloses the Bidirectional Encoder Representations from Transformers (BERT) model, which is "designed to pre-train deep bidirectional representations" and operates at the token level using "WordPiece embeddings" (Devlin, § 3, p. 4; § 3.1, p. 4), and which is provided in an uncased configuration in which the text is lowercased prior to WordPiece tokenization. (Devlin) describes the released BERT model in both cased and uncased configurations, the uncased BERT-based token-based model reading on the recited "Bidirectional Encoder Representations from Transforms (BERT)-based uncased token-based model." The present specification confirms that this limitation refers to the known BERT-based-uncased model, describing "a Bidirectional Encoder Representations from Transforms (BERT)-based-uncased or BERT-based-multilingual-uncased token-based model" (Spec ¶ [0033]).
(Malkiel) and (Devlin) are analogous to the claimed invention as both are from the same field of endeavor of BERT-based masked-language-model text representation. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to implement the BERT-based token-based model of (Malkiel) as the BERT-based uncased token-based model taught by (Devlin). The motivation to combine (Malkiel) and (Devlin) is provided by (Devlin), which teaches that the BERT-based token model yields improved bidirectional text representations that "can be fine-tuned … to create state-of-the-art models for a wide range of tasks" (Devlin, § 1, p. 1), and because (Malkiel) itself expressly adopts its masked language model "from BERT" (Malkiel, col. 9, ll. 3–4), such that one of ordinary skill would have selected a known BERT configuration, including the uncased configuration taught by (Devlin), to implement (Malkiel)'s BERT-based model, with a reasonable expectation of the predictable result of an operable domain-specialized text-representation model.
Claims 7, 14, and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Malkiel et al. (Malkiel), US 11,580,764 B2, in view of Khattab et al. (Khattab), Non-Patent Literature, "ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT," arXiv:2004.12832v2 [cs.IR], published on 4 June 2020 and cited in the IDS filed on 4/25/2024, in view of Tai et al. (Tai), Non-Patent Literature, "exBERT: Extending Pre-trained Models with Domain-specific Vocabulary Under Constrained Training Resources," published November 2020, and further in view of Johnson et al. (Johnson), Non-Patent Literature, "Billion-scale similarity search with GPUs," arXiv:1702.08734v1 [cs.CV], published on 28 February 2017.
Regarding Claim 7, the computer-implemented method of Claim 1, from which Claim 7 depends, is rejected under the same rationale as set forth above with respect to Claim 1 over (Malkiel) in view of (Khattab) and further in view of (Tai).
Claim 7 additionally recites the following sub-limitations:
"further comprising accessing an index of similar chunks to determine that the second chunk is similar to a fourth chunk, and, based at least in part on the index:";
"storing another indication that the first chunk is similar to the fourth chunk, and";
"tuning the text similarity model to identify similar texts by providing, to the text similarity model, the other indication."
(Malkiel) teaches the following sub-limitations of Claim 7:
"storing another indication that the first chunk is similar to the fourth chunk" (Malkiel teaches storing an indication that one chunk is similar to another chunk by placing "similar documents … into a first tier 506 while less similar or unsimilar documents are placed into a second tier," Malkiel, col. 8, ll. 18–24; applied to the fourth chunk, the storing of another indication that the first chunk is similar to the fourth chunk reads on this sub-limitation).
"tuning the text similarity model to identify similar texts by providing, to the text similarity model, the other indication" (Malkiel teaches providing stored similarity indications to tune the text similarity model, disclosing that "SBERT … is evaluated with (1) its original weights and (2) after fine-tuning with the pseudo labels," Malkiel, col. 14, ll. 5–10; applied to the other indication, the provision of that other indication to tune the text similarity model reads on this sub-limitation).
(Malkiel) teaches something related to accessing an index of similar chunks, disclosing that "[t]he system propagates each candidate document through the model to score and/or rank each document" (Malkiel, col. 8, ll. 12–14) and referring to "index information … which indicates whether the content of one document is likely to be related to another document" (Malkiel, col. 4, ll. 40–45). However, (Malkiel) does not teach "accessing an index of similar chunks to determine that the second chunk is similar to a fourth chunk."
In the same field of endeavor, (Khattab) teaches "accessing an index of similar chunks to determine that the second chunk is similar to a fourth chunk." (Khattab) teaches accessing a vector-similarity search index to retrieve chunks similar to a given chunk, disclosing that its late-interaction architecture "enables ColBERT to leverage vector-similarity search indexes … to retrieve" similar content (Khattab, § 3, p. 3), and that ColBERT retrieves from "a full collection using vector similarity indexes" (Khattab, §§ 3, 3.6, pp. 3–4). Accessing such a vector-similarity search index to determine that one chunk is similar to another chunk reads on this sub-limitation.
(Malkiel) and (Khattab) are analogous to the claimed invention as both are from the same field of endeavor of natural-language text-similarity determination using contextualized transformer (BERT-based) embeddings. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to combine the similarity indications and text-similarity-model tuning of (Malkiel) with the vector-similarity search index of (Khattab), by accessing an index of similar chunks to determine that the second chunk is similar to a fourth chunk as taught by (Khattab). The motivation to combine (Malkiel) and (Khattab) is provided by (Khattab), which teaches that leveraging vector-similarity search indexes enables efficient retrieval of similar content without exhaustively comparing every pair (Khattab, § 3, p. 3), thereby improving the efficiency of the similarity determination of (Malkiel)—a predictable improvement.
The combination of (Malkiel) and (Khattab), however, does not teach an index of chunks pre-identified as mutually similar such that a determination that the second chunk is similar to a fourth chunk may be obtained from the index and the similarity propagated to store the other indication.
In the same field of endeavor, (Johnson) teaches this feature. (Johnson) teaches constructing and accessing an index of similar items for similarity search, disclosing "a design for k-selection" applied to "similarity search" that constructs "a graph connecting 1 billion vectors" wherein "each edge connects a node to its k nearest neighbors" (Johnson, Abstract, p. 1; § 1, p. 1), using the "IVFADC indexing structure" for "approximate nearest-neighbor search" (Johnson, § 2, p. 2). Accessing such a nearest-neighbor index of similar vectors to determine that one indexed chunk is similar to another indexed chunk, and thereby to propagate the similarity so as to store the other indication, reads on "accessing an index of similar chunks to determine that the second chunk is similar to a fourth chunk" and the storing of "another indication that the first chunk is similar to the fourth chunk." The present specification confirms that this feature refers to such an index, describing that "FAISS is used as an index to find paragraph sets of vector embeddings that are similar to each other" and that "once a given paragraph has been determined to be similar to a particular other paragraph, the system may look up the particular other paragraph in an index to find other paragraphs that are already marked as similar" (Spec ¶¶ [0061]–[0062]).
(Malkiel), (Khattab), (Tai), and (Johnson) are analogous to the claimed invention as all are from the same field of endeavor of vector-embedding-based text similarity search and retrieval. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to combine the similarity indications and text-similarity-model tuning of (Malkiel), the vector-similarity search index of (Khattab), and the masked-language-model domain specialization of (Tai) with the nearest-neighbor similarity index of (Johnson), by accessing an index of similar chunks as taught by (Johnson) to determine that the second chunk is similar to a fourth chunk and to store and provide the other indication as taught by (Malkiel). The motivation to combine (Malkiel), (Khattab), (Tai), and (Johnson) is provided by (Johnson), which teaches that a nearest-neighbor similarity index enables similarity search at scale where "exhaustive search or exact indexing … is impractical" (Johnson, § 1, p. 1), such that one of ordinary skill would have used (Johnson)'s index to identify additional similar chunks efficiently and propagate the similarity determinations of (Malkiel) to further chunks, with a reasonable expectation of the predictable result of more efficient and scalable identification of similar chunks for tuning the text similarity model.
Claim 14 is rejected under 35 U.S.C. 103 under the same rationale as set forth above with respect to Claim 7.
Claim 19 is rejected under 35 U.S.C. 103 under the same rationale as set forth above with respect to Claims 7 and 14.
Claims 8 are rejected under 35 U.S.C. 103 as being unpatentable over Malkiel et al. (Malkiel), US 11,580,764 B2, in view of Khattab et al. (Khattab), Non-Patent Literature, "ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT," arXiv:2004.12832v2 [cs.IR], published on 4 June 2020 and cited in the IDS filed on 4/25/2024, in view of Tai et al. (Tai), Non-Patent Literature, "exBERT: Extending Pre-trained Models with Domain-specific Vocabulary Under Constrained Training Resources," published November 2020, and further in view of Mahapatra et al. (Mahapatra), US 12,159,109 B2.
Regarding Claim 8, the computer-implemented method of Claim 1, from which Claim 8 depends, is rejected under the same rationale as set forth above with respect to Claim 1 over (Malkiel) in view of (Khattab) and further in view of (Tai).
Claim 8 additionally recites the following sub-limitations:
"wherein the one or more conditions comprise a similarity threshold"; and
"wherein the text similarity model is not tuned with an indication that the first chunk is similar to the third chunk."
(Malkiel) teaches both sub-limitations.
As to "wherein the one or more conditions comprise a similarity threshold," (Malkiel) teaches that the condition for determining similarity comprises a threshold on the similarity metric, disclosing that the contrastive loss pushes dissimilar pairs apart "than a predefined positive margin 'm'" (Malkiel, col. 9, ll. 18–20), wherein "the margin 'm' is set to a value of one, which encourages inter-samples to have a cosine similarity that is less than or equal to 0" (Malkiel, col. 9, ll. 33–36). The predefined margin/threshold value against which the cosine similarity is compared to determine similarity or dissimilarity reads on the one or more conditions comprising a similarity threshold.
As to "wherein the text similarity model is not tuned with an indication that the first chunk is similar to the third chunk," (Malkiel) teaches that only chunk pairs satisfying the similarity condition are used, disclosing that "similar documents are placed into a first tier 506 while less similar or unsimilar documents are placed into a second tier" (Malkiel, col. 8, ll. 18–24), and that "recommendations are made from the first tier … including the subgroup of documents having the highest ranking" (Malkiel, col. 8, ll. 26–30). Because the first chunk and the third chunk correspond to a pair whose aggregate similarity score does not satisfy the similarity threshold (as set forth with respect to limitation 1[j] in the rejection of Claim 1), that pair is placed in the second tier and is not among the similar-pair indications provided to tune the text similarity model, which reads on the text similarity model not being tuned with an indication that the first chunk is similar to the third chunk.
(Mahapatra) additionally teaches that the one or more conditions comprise a similarity threshold, disclosing "selecting training sentences from the source domain corpus within a threshold of similarity to the target corpus embedding" (Mahapatra, claim 1; col. 17, ll. 137–156), wherein selecting within the threshold "comprises determining cosine similarity … within the threshold of similarity to the target corpus embedding" (Mahapatra, claim 2; col. 17, ll. 157–168). The selection of content for training based upon whether a cosine-similarity value falls within a similarity threshold reads on the one or more conditions comprising a similarity threshold.
(Malkiel), (Khattab), (Tai), and (Mahapatra) are analogous to the claimed invention as all are from the same field of endeavor of domain adaptation of language models and selection of text based on embedding similarity for training. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to combine the similarity condition and tiered similarity indications of (Malkiel), (Khattab), and (Tai) with the similarity-threshold selection of (Mahapatra), by implementing the one or more conditions of (Malkiel) as a similarity threshold as taught by (Mahapatra). The motivation to combine (Malkiel), (Khattab), (Tai), and (Mahapatra) is provided by (Mahapatra), which teaches that selecting content by a threshold of similarity provides training data of controlled quality for the downstream model (Mahapatra, claim 1; col. 17, ll. 137–156), such that one of ordinary skill would have applied a similarity threshold as the condition governing which similarity indications are used to tune the text similarity model of (Malkiel), with a reasonable expectation of the predictable result of tuning the text similarity model on appropriately similar chunk pairs while excluding pairs, such as the first and third chunks, that do not satisfy the threshold.
Claims 9, 15, and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Malkiel et al. (Malkiel), US 11,580,764 B2, in view of Khattab et al. (Khattab), Non-Patent Literature, "ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT," arXiv:2004.12832v2 [cs.IR], published on 4 June 2020 and cited in the IDS filed on 4/25/2024, in view of Tai et al. (Tai), Non-Patent Literature, "exBERT: Extending Pre-trained Models with Domain-specific Vocabulary Under Constrained Training Resources," published November 2020, and further in view of Thakur et al. (Thakur), Non-Patent Literature, "Augmented SBERT: Data Augmentation Method for Improving Bi-Encoders for Pairwise Sentence Scoring Tasks," arXiv:2010.08240v2 [cs.CL], published on 12 April 2021.
Regarding Claim 9, the computer-implemented method of Claim 1, from which Claim 9 depends, is rejected under the same rationale as set forth above with respect to Claim 1 over (Malkiel) in view of (Khattab) and further in view of (Tai).
(Malkiel) teaches the following sub-limitations of Claim 9:
As to "wherein the one or more conditions comprise a similarity threshold," (Malkiel) teaches that the condition comprises a threshold on the similarity metric, disclosing a "predefined positive margin 'm'" (Malkiel, col. 9, ll. 18–20) wherein "the margin 'm' is set to a value of one, which encourages inter-samples to have a cosine similarity that is less than or equal to 0" (Malkiel, col. 9, ll. 33–36; col. 11, ll. 44–46), the margin/threshold value reading on the one or more conditions comprising a similarity threshold.
As to "based at least in part on determining that the second aggregate similarity score satisfies one or more other conditions, storing another indication that the first chunk is dissimilar to the third chunk; wherein the first aggregate similarity score does not satisfy the one or more other conditions," (Malkiel) teaches determining, based on a separate dissimilarity condition, that two chunks are dissimilar and storing an indication accordingly, disclosing that inter-samples—sentence pairs from different documents—are treated as dissimilar and encouraged "to have a cosine similarity that is less than or equal to 0" (Malkiel, col. 11, ll. 44–46), and that "less similar or unsimilar documents are placed into a second tier" (Malkiel, col. 8, ll. 22–24). When the second aggregate similarity score (between the first chunk and the third chunk) satisfies the dissimilarity condition while the first aggregate similarity score (between the first chunk and the second chunk) does not satisfy that dissimilarity condition, (Malkiel) stores an indication that the first chunk is dissimilar to the third chunk by placing that pair into the second (dissimilar) tier.
(Malkiel) teaches something related to "tuning the text similarity model to identify dissimilar texts by providing, to the text similarity model, the other indication," disclosing that "SBERT … is evaluated with (1) its original weights and (2) after fine-tuning with the pseudo labels" (Malkiel, col. 14, ll. 5–10), wherein the pseudo-labels include negative sentence pairs produced by inter-document sampling (Malkiel, col. 9, ll. 20–24). However, (Malkiel) does not teach "tuning the text similarity model to identify dissimilar texts by providing, to the text similarity model, the other indication."
In the same field of endeavor, (Thakur) teaches "tuning the text similarity model to identify dissimilar texts by providing, to the text similarity model, the other indication." (Thakur) teaches tuning a text similarity model by providing labeled pairs, including dissimilar pairs, to the model, disclosing that "we use the cross-encoder to label a larger set of input pairs to augment the training data for the bi-encoder" (Thakur, Abstract, p. 1), producing a "silver dataset" of "weakly labeled examples" that "will be merged with the gold training dataset" (Thakur, § 3.1, p. 3) and used to train the bi-encoder, wherein the labeled pairs include negative (dissimilar) pairs (Thakur, § 3.1, pp. 3–4). The provision of labeled dissimilar pairs to train the bi-encoder reads on tuning the text similarity model to identify dissimilar texts by providing the other indication.
(Malkiel) and (Thakur) are analogous to the claimed invention as both are from the same field of endeavor of training text-similarity models on labeled similar and dissimilar text pairs using transformer-based embeddings. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to combine the dissimilarity labeling of (Malkiel) with the labeled-pair augmentation of (Thakur), by providing the stored dissimilarity indication of (Malkiel) to the text similarity model to identify dissimilar texts as taught by (Thakur). The motivation to combine (Malkiel) and (Thakur) is provided by (Thakur), which teaches that augmenting the training data of the bi-encoder with labeled pairs, including dissimilar pairs, improves the performance of the resulting similarity model (Thakur, Abstract, p. 1), such that one of ordinary skill would have provided the dissimilar-pair indications of (Malkiel) to tune the text similarity model to identify dissimilar texts, with a reasonable expectation of the predictable result of an improved text similarity model capable of distinguishing dissimilar texts.
Claim 15 is rejected under 35 U.S.C. 103 under the same rationale as set forth above with respect to Claim 9.
Claim 20 is rejected under 35 U.S.C. 103 under the same rationale as set forth above with respect to Claims 9 and 15.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to HUNG VAN LE whose telephone number is (571)270-0164. The examiner can normally be reached 8 a.m. - 5 p.m..
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Cesar Paula can be reached at (571) 272-4128. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/HUNG VAN LE/Examiner, Art Unit 2145
/CESAR B PAULA/Supervisory Patent Examiner, Art Unit 2145
Prosecution Insights