DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Drawings
Figures 4-5 should be designated by a legend such as --Prior Art-- because only that which is old is illustrated. See Applicant’s admission in the specification at Paragraphs 0069-0071 and 0075 and MPEP § 608.02(g). Corrected drawings in compliance with 37 CFR 1.121(d) are required in reply to the Office action to avoid abandonment of the application. The replacement sheet(s) should be labeled “Replacement Sheet” in the page header (as per 37 CFR 1.84(c)) so as not to obstruct any portion of the drawing figures. If the changes are not accepted by the examiner, the applicant will be notified and informed of any required corrective action in the next Office action. The objection to the drawings will not be held in abeyance.
Examiner Notes on Patent Subject Matter Eligibility under 35 U.S.C. 101
Independent Claims 1, 19, and 20 are directed towards a technical non-human approach for information retrieval of a relevant set of activity records using a similarity between embeddings generated by a particularly structured phrase-based attention mechanism in an encoder and thus do not represent a method of organizing human activity or a mental process under step 2A prong 1. Although the step of calculating a similarity metric can be classified as a mathematical concept under step 2A prong 1 of the 2019 Patent Subject Matter Eligibility Guidelines (2019 PEG), the recited process is not purely mathematical and again relies upon a particularly structured phrase-based attention mechanism in an encoder. Accordingly, independent claims 1 and 19-20 and their respective dependent claims are found to be directed towards patent eligible subject matter under step 2A prong 1 and/or prong 2.
Claim Objections
Claims 15-16 are objected to because of the following informalities:
In Claim 15, Lines 6-7, "except for selected keyword" appears to be grammatically incomplete and should be corrected to read -- except for the selected keyword--.
Claim 16 further limits and inherits the subject matter of objected claim 15, and thus, is also objected to due to minor informalities by virtue of its dependency.
Appropriate correction is required.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claim 5 is rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 5 depends upon claim 3 and references "the one or more scaled dot-product attention layers." This limitation lacks antecedent basis because this term was introduced in claim 4 rather than claim 3. For claim interpretation purposes in the interest of compact prosecution "method of claim 3" will be construed as --the method of claim 4--.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-3 and 17-20 are rejected under 35 U.S.C. 103 as being unpatentable over Carmody, et al. (U.S. PG Publication: 2021/0406685 A1) in view of Lenz, Jr. et al. (U.S. Patent: 10,902,009) and further in view of Wu, et al. ("Phrase-level Self-Attention Networks for Universal Sentence Encoding," 2018).
With respect to Claim 1, Carmody discloses:
A method comprising using at least one hardware processor to:
receive a user keyword array and a plurality of activity keyword arrays, wherein each of the user keyword array and the plurality of activity keyword arrays comprises a plurality of keywords, wherein each keyword comprise one or a plurality of tokens, and wherein each of the plurality of activity keyword arrays is associated with an activity record comprising a Uniform Resource Locator (URL) and an Internet Protocol (IP) address (receiving user specified keywords and plural “raw keyword data comprising a plurality of keyword activity records” that comprise a “uniform resource locator (URL)” and “IP address”, Paragraphs 0007, 0009, 0056, 0060, and 0062);
apply an encoder to the user keyword array to produce a user embedding vector (user input keyword search array undergoes vector encoding, Paragraphs 0055, 0066-0067, and 0082);
calculate a similarity metric between the user keyword array and the activity keyword array (search/user keywords semantically similar match determination with arrays of keywords from the activity records, Paragraphs 0007 and 0062-0064, and 0112-0113); and
and when the similarity metric indicates a match between the user keyword array and the activity record keyword array, add the activity record that is associated with the activity keyword array to a relevant set of activity records (adding "online resources in raw keyword data...that were matched to the user-specified keywords," in a relevant filtered list of activity items, Paragraphs 0062 and 0064); and
output the relevant set of activity records to one or more downstream functions (filtered list based upon the matching is output to be "used in one or more downstream functions, such as for reporting," Paragraphs 0069-0072 and 0114).
Although Carmody teaches vector embedding using a well-known algorithm/model such as Word2Vec (see Paragraph 0066), Carmody does not use the vectorization encoding in a similarity metric assessment to identify matching keyword records in a database corresponding to a user keyword array embedding because Carmody relies upon such similarity to suggest keywords to a user rather than in a search process. Lenz, however, discloses searching a database of audience activity (e.g., within a particular website) keywords using a semantic similarity metric between a user/search keyword embedding and a database embedding (Col. 7, Lines 3-31; Col. 10, Lines 26-54).
Carmody and Lenz are analogous art because they are from a similar field of endeavor in keyword/relevance searching of natural language data. Thus, it would have been obvious to one of ordinary skill in the art before the effective filing date to utilize the vector/embedding-based encoding and searching taught by Lenz in the filtered record search process taught by Carmody to provide a predictably provide a quantitative measure that allows for a measure of the similarity between the meaning and/or context of the keywords rather than a simple text match (Lenz, Col. 7, Lines 3-31).
While the process/functionality of Carmody and Lenz teach similar database and embedding-based similarity search process wherein both the user keyword search array and the database keyword record array are encoded into and have embeddings, the embedding mechanism of the combination differs from that of the claimed invention. In particular, Carmody in view of Lenz does not teach that the encoder comprises one or more phrase-localized attention layers, and wherein each of the one or more phrase-localized attention layers comprises one phrase-localized attention network for each of the plurality of keywords. Wu, however, teaches that the encoder comprises one or more phrase-localized attention layers, and wherein each of the one or more phrase-localized attention layers comprises one phrase-localized attention network for each of the plurality of keywords (encoder model with self-attention performed at the phrase level where the encoder features at least one phrase-localized attention layer and each layer includes a network for each keyword in the array for different phrasal divisions, Sections 2-2.2, Pages 3730-3731; Abstract; Fig. 1 showing each keyword at each layer with a corresponding split to apply the phrase level attention mechanism).
Carmody, Lenz, and Wu are analogous art because they are from a similar field of endeavor in natural language processing for similarity determination and downstream tasks. Thus, it would have been obvious to one of ordinary skill in the art before the effective filing date to use the phrase level encoding model taught by Wu in generating the embeddings used in semantic searching taught by Carmody in view of Lenz to provide a predictable result in the form of better generalizing unseen combinations of words for downstream NLP tasks with efficient memory consumption (Wu, Abstract; Section 1, Page 3729).
With respect to Claim 2, Wu further discloses:
The method of Claim 1, wherein the one or more phrase-localized attention layers are at least three phrase-localized attention layers (three phrase-localized attention layers based upon different natural language processing (NLP) splits, Fig. 1 and associated description at Section 2.2., Pages 3730-3731).
With respect to Claim 3, Wu further discloses:
The method of Claim 1, wherein the one or more phrase-localized attention layers consist of three phrase-localized attention layers (three phrase-localized attention layers based upon different natural language processing (NLP) splits, Fig. 1 and associated description at Section 2.2., Pages 3730-3731).
With respect to Claim 17, Lenz further discloses:
The method of Claim 1, wherein the similarity metric comprises a cosine similarity between the user embedding vector and the activity embedding vector (semantic similarity between user/search keyword embedding and audience record keyword embedding via cosine distance, Col. 7, Line 51- Col. 8, Line 15 and Col. 12, Line 58- Col. 13, Line 11).
With respect to Claim 18, Carmody further discloses:
The method of Claim 1, wherein the one or more downstream functions comprise a predictive model that predicts a buying intent of at least one company, associated with at least one IP address in the relevant set of activity records, based on the relevant set of activity records (Paragraph 0071- "predictive model 380 may predict the purchase intent of a company, indicating a likelihood that the company will purchase a product (e.g., good or service) that the user sells;" see also Paragraph 0114 and Fig. 3, Element 380).
Claim 19 is directed towards the steps of method claim 1 implemented as system functionality, and thus, is rejected under similar rationale. Moreover, Carmody teaches method implementation as a hardware processor running a program (Paragraph 0037).
Claim 20 is directed towards the steps of method claim 1 implemented as processor-executable instructions, and thus, is rejected under similar rationale. Moreover, Carmody teaches method implementation as a non-transitory computer-readable medium (Paragraph 0014).
Claims 4-13 are rejected under 35 U.S.C. 103 as being unpatentable over Carmody, et al. in view of Lenz, Jr. et al. in view of Wu, et al. and further in view of Nguyen, et al. (PhraseTransformer: Self-Attention using Local Context for Semantic Processing," 2021).
With respect to Claim 4, Carmody in view of Lenz and further in view of Wu teach the keyword array activity record searching utilizing vectors based upon a phrase-level attention encoding model, as applied to Claim 1. Carmody in view of Lenz and further in view of Wu do not teach that the encoder further comprises one or more scaled dot-product attention layers. Nguyen, however, discloses that a phrase-based encoder architecture includes at least one scaled dot product attention layer (see Fig. 2 phrase transformer; Section 3, Page 3).
Carmody, Lenz, Wu, and Nguyen are analogous art because they are from a similar field of endeavor in natural language processing for similarity determination. Thus, it would have been obvious to one of ordinary skill in the art before the effective filing date to include the scaled dot product attention layer taught by Nguyen in the encoder of Carmody in view of Lenz and further in view of Wu to provide a predictable result of better ensuring that the encoder model focuses on the most relevant parts of the token array.
With respect to Claim 5, Nguyen further discloses:
The method of Claim 3 Claim 4 (see preceding interpretation in the 35 U.S.C. 112(b) rejection), wherein the one or more scaled dot-product attention layers are subsequent to the one or more phrase-localized attention layers (see that the scaled dot-product attention layers are subsequent to phrase localization as depicted in Fig. 2, see also Section 3, Page 3).
With respect to Claim 6, Nguyen further discloses:
The method of Claim 4, wherein the one or more scaled dot-product attention layers are at least three scaled dot-product attention layers (see Fig. 2 (below) showing at least 3 scaled dot-product attention layers and the discussion at Section 3, Page 3).
PNG
media_image1.png
104
250
media_image1.png
Greyscale
With respect to Claim 7, Nguyen further discloses:
The method of Claim 4, wherein the one or more scaled dot-product attention layers consist of three scaled dot-product attention layers (see Fig. 2 (below) showing 3 scaled dot-product attention layers and the discussion at Section 3, Page 3).
PNG
media_image1.png
104
250
media_image1.png
Greyscale
With respect to Claim 8, Wu and Nguyen further disclose:
The method of Claim 1, wherein the encoder comprises at least three phrase-localized attention layers, followed by at least three scaled dot-product attention layers (Wu- three phrase-localized attention layers based upon different natural language processing (NLP) splits, Fig. 1 and associated description at Section 2.2., Pages 3730-3731 where Nguyen teaches that at least 3 scaled dot-product attention layers are subsequent to phrase localization as depicted in Fig. 2, see also Section 3, Page 3).
With respect to Claim 9, Wu and Nguyen further disclose:
The method of Claim 8, wherein each phrase-localized attention network and each of the at least three scaled dot-product attention layers utilize multi-head attention (Nguyen- see that the encoder at the phrase and scaled dot product attention layers utilize multi-head attention in Fig. 2 wherein Wu teaches phrase-localized attention layers based upon different natural language processing (NLP) splits, Fig. 1 and associated description at Section 2.2., Pages 3730-3731).
With respect to Claim 10, Nguyen further discloses:
The method of Claim 9, wherein the encoder utilizes keyword-level positional encoding to encode a position of each token within each of the plurality of keywords (see position encoding of an input sequence in Fig. 2).
With respect to Claim 11, Nguyen further discloses:
The method of Claim 1, wherein each phrase-localized attention network utilizes multi-head attention (phrase-based attention utilizing multi-head attention, Fig. 2 and Section 3, Pages 3-4; note that Wu teaches the phrase-level attention network as applied to claim 1).
With respect to Claim 12, Nguyen further discloses:
The method of Claim 1, wherein the encoder utilizes keyword-level positional encoding to encode a position of each token within each of the plurality of keywords (see position encoding of an input sequence in Fig. 2).
With respect to Claim 13, Wu and Nguyen further disclose:
The method of Claim 1, wherein the encoder consists of three phrase-localized attention layers, followed by three scaled dot-product attention layers (Wu- three phrase-localized attention layers based upon different natural language processing (NLP) splits, Fig. 1 and associated description at Section 2.2., Pages 3730-3731 where Nguyen teaches that at least 3 scaled dot-product attention layers are subsequent to phrase localization as depicted in Fig. 2, see also Section 3, Page 3).
Claims 14-16 are rejected under 35 U.S.C. 103 as being unpatentable over Carmody, et al. in view of Lenz, Jr. et al. in view of Wu, et al. and further in view of He, et al. (U.S. PG Publication: 2021/0232753 A1).
With respect to Claim 14, Carmody in view of Lenz and further in view of Wu teach the keyword array activity record searching utilizing vectors based upon a phrase-level attention encoding model, as applied to Claim 1. While this combination does teach the generation of encodings, Carmody in view of Lenz and further in view of Wu do not specifically disclose the well-known transformer model and its operations adding a decoder to the encoder commonly used in technologies such as large language model (LLM) prediction/inferencing.
He, however, discloses:
prior to applying the encoder, train a transformer network comprising the encoder and a decoder (transformer-based language model, Paragraph 0041, having a content embedder/encoder and LM decoder, Fig. 1, Element 104 and Fig. 3, Element 334), wherein the encoder receives a keyword array from a training dataset as an input and outputs an embedding vector (encoder generates embeddings from training input tokens, Paragraph 0021 and 0056), and wherein the decoder receives the embedding vector, output by the encoder, as an input and outputs a predicted keyword (decoder outputs a predicted keyword, Paragraphs 0055-0057).
Carmody, Lenz, Wu, and He are analogous art because they are from a similar field of endeavor in natural language processing for similarity determination. Thus, it would have been obvious to one of ordinary skill in the art before the effective filing date to incorporate the encoder-decoder architecture taught by He in the natural language processing framework taught by Carmody in view of Lenz in view of Wu to provide a predictable result of learning and then making inferences/generalizations on the weighted/numerical outputs of the encoder.
With respect to Claim 15, He further discloses:
prior to training the transformer network, generate the training dataset by: receiving a plurality of keyword arrays; and for each of the plurality of keyword arrays, for each of one or more iterations, selecting one keyword from the keyword array, generating an input consisting of all keywords in the keyword array except for selected keyword, labeling the input with a target consisting of the selected keyword, and adding the labeled input to the training dataset (receiving keyword token arrays and creating training arrays for the tokens by labeling a particular keyword/token with a mask in order to predict that masked token in a training process, 0019, 0021, 0041-0042, 0054-0055, and 0075).
With respect to Claim 16, He further discloses:
The method of Claim 15, wherein training the transformer network comprises, for each of at least a subset of the labeled inputs in the training dataset: applying the transformer network to the input in the labeled input to produce the predicted keyword for the input; computing a loss between the target, with which the input is labeled, and the predicted keyword; and updating the transformer network to minimize the computed loss (training process that involves prediction/reconstruction of the masked keyword target, Paragraphs 0041, 0055, 0057, and 0070, wherein training is based upon a loss function/cost as to how wrong the result of the prediction is compared to the expected result (i.e., is the masked reconstruction correct?), Paragraph 0075).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure:
Wu, Shaojuan, et al. ("Phrase-level attention network for few-shot inverse relation classification in knowledge graph," May 2023)- teaches a phrase-level attention network that function and keywords in an encoder (Section 1, Page 3004).
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JAMES S WOZNIAK whose telephone number is (571)272-7632. The examiner can normally be reached 7-3, off alternate Fridays.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant may use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Andrew Flanders can be reached at (571)272-7516. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
JAMES S. WOZNIAK
Primary Examiner
Art Unit 2655
/JAMES S WOZNIAK/Primary Examiner, Art Unit 2655
Prosecution Insights