Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
DETAILED ACTION
This Office Action is in response to the application filed on 02/20/2024.
Claims 1-20 are pending.
Information Disclosure Statement
The information disclosure statement (IDS) filed on 02/20/2024 has been considered (see form-1449, MPEP 609).
Drawings
The drawings filed on 02/20/2024 are accepted.
Examiner Notes
Examiner cites particular columns, paragraphs, figures and line numbers in the references as applied to the claims below for the convenience of the applicant. Although the specified citations are representative of the teachings in the art and are applied to the specific limitations within the individual claim, other passages and figures may apply as well. It is respectfully requested that, in preparing responses, the applicant fully consider the references in their entirety as potentially teaching all or part of the claimed invention, as well as the context of the passage as taught by the prior art or disclosed by the examiner.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The text of those sections of Title 35, U.S. Code not included in this action can be found in a prior Office action.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1-20 are rejected under 35 U.S.C. 103 as being unpatentable over Friede et al. (US PGPUB 2022/0300831, hereinafter Friede), in view of Moritz et al. (US PGPUB 2022/0310070, hereinafter Moritz).
As per as claim 1, Friede discloses:
A method comprising:
receiving an input by a self-attention machine learning model (Friede, e.g., [006], [0018-0019], “…a machine learning model … The input embeddings have: a query embedding set, the query embedding set comprising a subject embedding…”);
generating a set of queries using the input (Friede, e.g., [006], [0018-0019], “…a query embedding set, the query embedding set comprising a subject embedding, object embedding, and relation embedding, one of the subject embedding, the object embedding, and the relation embedding being the query embedding… generates the link prediction for the query embedding and an output associated with each of the input embeddings…”);
generating at least one of (i) two sets of keys using the input and (ii) two sets of values using the input (Friede, e.g., [abstract], [0024], [0030], [0122], “…Inputting training data (i.e., seeding) by providing KG1, KG2, seed alignments as triples of the form (e1, SameAs, e2)” and [0131-0132], “…a query in the form of (e1, SameAs, ?) or (?, SameAs, e2) may be provided to the input of the model...”); and
determining an output of the self-attention machine learning model using the two sets of keys, the two sets of values, or both (Friede, e.g., [0131-0135], “…produce results… outputs determined from a set of such queries constitute the “results”…” (the examiner asserts plurality of search results = sets and values).
To further clarify the language of “self-attention machine learning model using sets of keys and values” (although as stated above Friede functional disclose the features of “self-attention machine learning model using keys and values (Friede, e.g., [abstract], [0018-0019], [0024], [0030], [0122]).
However Moritz, in an analogous art, discloses “self-attention machine learning model using sets of keys and values” (Moritz, e.g., [0045-0049], “… input signal may include a sequence of input frames, which are transformed into a sequence of keys, values, and queries. Each query frame of the sequence of queries searches over the sequence of keys to compute the relationship of each key frame relative to the query frame. Each key frame is related to a value frame that encodes features about each input frame… self-attention module is trained to compute the sequence of outputs from the sequence of input frames based on learned transforms and a sequence of attention calculations. The sequence of attention calculations allows mapping the sequence of value frames to an output by using a current query frame and a sequence of key frames. In some example embodiments, the dilated self-attention module provides an attention mechanism to read information from the sequence of inputs by using the current query frame where the current query frame corresponds to a query vector…” and [0055-0058], “…self-attention module can use the principles of the attention mechanism to read information from a sequence of inputs based on a current query frame, such as query…in an attention mechanism of the dilated self-attention module, an input of a source sequence is initially transformed into key and value frames…”). Thus, it would have been obvious to one of ordinary skill in the art BEFORE the effective filling date of the claimed invention to combine the teaching of Moritz and Friede to performing attention calculations for each query frame with respect to a combination of a portion of the sequences of key and value frames restricted based on a location of the query frame and a dilation sequence of the key frames and a dilation sequence of value frames extracted by processing different frames of the sequences of key and value frames with a predetermined extraction function (Moritz, e.g., [abstract]).
As per as claim 2, the combination of Moritz and Friede disclose:
The method of Claim 1, wherein determining the output of the self-attention machine learning model includes:
identifying a query position for the set of queries (Moritz, e.g., [006], [0048] and [0070], “…self-attention mechanism may transform the frames of such an input sequence to a sequence of key frames, value frames, and query frames…a query frame, which corresponds to a frame position in the input sequence, may belong to a similar sound event as that of the query frame, where detailed information may be required to recognize their logical relation with one or more of the key frames, value frames, and query frames…”);
identifying a key position for the two sets of keys (Moritz, e.g., [006], [0048] and [0070]) (the examiner asserts multiple set of keys = metadata or parameter or dimensions) and (Priede, e.g., [0105-0108]);
determining whether the query position equals the key position (Friede, e.g., [0112], “…the profiles in the two knowledge graphs will match (e.g., companies have profiles on different social networks to increase visibility and people often have several profiles on different professional social networks to improve their chance of employment). Different professional social networks, however, may contain different information about the same entity types (e.g., one knowledge graph may contain information about one person's particular skills, while other may contain information about the person's endorsements and recommendations by other people). An embodiment of the present invention can be used for integrating such knowledge graphs by matching the equivalent entities…” and [0120], [0138], (matching with search results)); and
calculating an attention score using a first set of the two sets of keys when the query position is determined to be equal to the key position or using a second set of the two sets of keys when the query position is determined to be unequal to the key position (Friede, e.g., [0018-0020], [0024], [0030], “… first self-attention layer of the self-attention layers, the attention score for each of the knowledge graph embeddings and the current validation embedding set, generating, by the final layer of the neural network of the context transformer, the link prediction for the current validation embedding set and the output associated with each of the input embeddings, combining, by the decision head, the attention score and the output for each of the input embeddings to determine the significance score for each of the input embeddings; and outputting the link prediction and the significance score for the subset of the input embeddings; and determining an accuracy of model based on comparing the link…”).
As per as claim 3, the combination of Moritz and Friede disclose:
The method of Claim 2, wherein:
generating the two sets of keys includes combining the input with two parameter matrices (Friede, e.g., [0093-0095]);
each one of the two parameter matrices corresponds to one of the two sets of keys (Friede, e.g., [0093-0095]); and
the two parameter matrices are initialized to minimize off-diagonal components of the attention score with respect to on-diagonal components of the attention score (Friede, e.g., [0093-0095], “…self-attention, the model generates an attention score between each two inputs…attention the model puts on a specific input to answer the query…”).
As per as claim 4, the combination of Moritz and Friede disclose:
The method of Claim 2, wherein determining the output of the self-attention machine learning model further includes:
creating a normalized score using the attention score (Friede, e.g., [0017-0019],[0029-0030], “… created knowledge is highly accurate…generates an attention score for each of the input embeddings…”);
identifying a value position for the two sets of values (Moritz, e.g., [006], [0048] and [0070], “…self-attention mechanism may transform the frames of such an input sequence to a sequence of key frames, value frames, and query frames…a query frame, which corresponds to a frame position in the input sequence, may belong to a similar sound event as that of the query frame, where detailed information may be required to recognize their logical relation with one or more of the key frames, value frames, and query frames…”);
determining whether the query position equals the value position (Moritz, e.g., [006], [0048] and [0070]) and (Priede, e.g., [0105-0108]);
calculating an attention result using the normalized score and a first set of the two sets of values when the query position is determined to be equal to the value position or using the normalized score and a second set of the two sets of values when the query position is determined to be unequal to the value position (Friede, e.g., [0018-0020], [0024], [0030], “… first self-attention layer of the self-attention layers, the attention score for each of the knowledge graph embeddings and the current validation embedding set, generating, by the final layer of the neural network of the context transformer, the link prediction for the current validation embedding set and the output associated with each of the input embeddings, combining, by the decision head, the attention score and the output for each of the input embeddings to determine the significance score for each of the input embeddings; and outputting the link prediction and the significance score for the subset of the input embeddings; and determining an accuracy of model based on comparing the link…”); and
generating the output of the self-attention machine learning model using the attention result (Friede, e.g., [0018-0020], [0024], [0029-0030]).
As per as claim 5, the combination of Moritz and Friede disclose:
The method of Claim 1, wherein determining the output of the self-attention machine learning model includes:
creating a normalized score using an attention score (Friede, e.g., [0017-0019],[0029-0030], “… created knowledge is highly accurate…generates an attention score for each of the input embeddings…”);
identifying a query position for the set of queries (Moritz, e.g., [006], [0048] and [0070], “…self-attention mechanism may transform the frames of such an input sequence to a sequence of key frames, value frames, and query frames…a query frame, which corresponds to a frame position in the input sequence, may belong to a similar sound event as that of the query frame, where detailed information may be required to recognize their logical relation with one or more of the key frames, value frames, and query frames…”);
identifying a value position for the two sets of values (Moritz, e.g., [006], [0048] and [0070], “…self-attention mechanism may transform the frames of such an input sequence to a sequence of key frames, value frames, and query frames…a query frame, which corresponds to a frame position in the input sequence, may belong to a similar sound event as that of the query frame, where detailed information may be required to recognize their logical relation with one or more of the key frames, value frames, and query frames…”);
determining whether the query position equals the value position (Moritz, e.g., [006], [0048] and [0070]) and (Priede, e.g., [0105-0108]);
calculating an attention result using the normalized score and a first set of the two sets of values when the query position is determined to be equal to the value position or using the normalized score and a second set of the two sets of values when the query position is determined to be unequal to the value position (Friede, e.g., [0018-0020], [0024], [0030], “… first self-attention layer of the self-attention layers, the attention score for each of the knowledge graph embeddings and the current validation embedding set, generating, by the final layer of the neural network of the context transformer, the link prediction for the current validation embedding set and the output associated with each of the input embeddings, combining, by the decision head, the attention score and the output for each of the input embeddings to determine the significance score for each of the input embeddings; and outputting the link prediction and the significance score for the subset of the input embeddings; and determining an accuracy of model based on comparing the link…”); and
generating the output of the self-attention machine learning model using the attention result (Friede, e.g., [0018-0020], [0024], [0029-0030]).
As per as claim 6, the combination of Moritz and Friede disclose:
The method of Claim 5, wherein:
generating the two sets of values includes combining the input with two parameter matrices (Friede, e.g., [0093-0085]) and (Moritz, e.g., [0073-0074] and [0078], “…selection function accepts the location of the input as a parameter and returns the neighboring frames 610A and 610B in the sequence of inputs 602. The selection function may also accept a value for the look-ahead size and for the look-back size to form a window size around the current query frame for selecting the neighboring frames…”); and
each one of the two parameter matrices corresponds to one of the two sets of values Friede, e.g., [0093-0085]) and (Moritz, e.g., [0073-0074] and [0078]).
As per as claim 7, the combination of Moritz and Friede disclose:
The method of Claim 1, wherein the self-attention machine learning model forms a part of a large language machine learning model (Friede, e.g., [abstract], [006], [0013-0014] and [0018-0019]).
Claims 8-14 are essentially the same as claims 1-7 except that they set forth the claimed invention as an electronic device rather a method, respectively and correspondingly, therefore is rejected under the same reasons set forth in rejections of claims 1-7.
Claims 15-20 are essentially the same as claims 1-7 except that they set forth the claimed invention as a non transitory readable medium rather a method, respectively and correspondingly, therefore is rejected under the same reasons set forth in rejections of claims 1-7.
Additional Art Considered
The prior art made of record and not relied upon is considered pertinent to the Applicants’ disclosure.
The following patents and papers are cited to further show the state of the art at the time of Applicants’ invention with respect to split key and value self-attention machine learning which receiving an input by a self-attention machine learning model and generating a set of queries using the input and then generating at least one of two sets of keys using the input and two sets of values using the input. Further, identifying a query position for the set of queries, identifying a key position for the two sets of keys, and when the query position is determined to be equal to the key position, calculating an attention score using a first set of the two sets of keys, or, when the query position is determined to be unequal to the key position, calculating the attention score using a second set of the two sets of keys.
a. Gong et al. (US PGPUB 2022/0399026, hereafter Gong); “System and Method for self-attention based combining of multichannel Signals For Speech Processing” discloses “ Generating the weighted multichannel representation of the plurality of channels may include generating the weighted multichannel representation of the plurality of channels via a first self-attention machine learning model. Generating the weighted multichannel representation of the plurality of channels via the first self-attention machine learning model may include defining a plurality of attention weights. Generating the weighted multichannel representation of the plurality of channels may include defining each channel of the weighted multichannel representation of the plurality of channels as a linear combination of the plurality of channels”.
Gong further teaches first self-attention machine learning model and second self-attention machine learning model [0010].
Gong also teaches performing linear transformations to calculate the query and query and key may be calculated separately by performing linear transformations on the STFT input [0057-0058].
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to TUAN A PHAM whose telephone number is (571)270-3173. The examiner can normally be reached M-F 7:45 AM - 6:30 PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Tony Mahmoudi can be reached on 571-272-4078. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/TUAN A PHAM/Primary Examiner, Art Unit 2163
Prosecution Insights