Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 07.23.2024; 09.24.2024;
09.30.2024; 10.28.2024 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-21 are rejected under U.S.C 101 for containing an abstract idea without significantly more.
Regarding claim 1:
Step 1 – Is the claim to a process, machine, manufacture or composition of matter?
Yes, the claim is a process.
Step 2A – Prong 1 – Does the claim recite an abstract idea, law of nature, or natural phenomenon?
Yes, the claim recites an abstract idea.
detect that a browser hosted by the computer system is accessing a document; - This limitation is directed to the abstract idea of a mental process (including an observation, evaluation, judgement, opinion) which can be performed in the human mind, or by a human using pen and paper (see MPEP 2106.04(a)(2) Ill. C.)
identify a document space configured to receive third-party content; - This limitation is directed to the abstract idea of a mental process (including an observation, evaluation, judgement, opinion) which can be performed in the human mind, or by a human using pen and paper (see MPEP 2106.04(a)(2) Ill. C.)
examine the document to determine if the computer system is permitted to insert content provided by the enhanced, local large learning model; - This limitation is directed to the abstract idea of a mental process (including an observation, evaluation, judgement, opinion) which can be performed in the human mind, or by a human using pen and paper (see MPEP 2106.04(a)(2) Ill. C.)
Step 2A – Prong 2 – Does the claim recite additional elements that integrate the judicial exception into a practical application?
No, there are no additional elements that integrate the judicial exception into a practical application. The additional elements:
A computer system associated with a user, the computer system comprising:
This limitation is directed to a computer merely used as a tool to perform an existing process (see MPEP 2106.05(f) (2)).
a network interface; This limitation is directed to a computer merely used as a tool to perform an existing process (see MPEP 2106.05(f) (2)).
at least one processing device operable to - This limitation is directed to a computer merely used as a tool to perform an existing process (see MPEP 2106.05(f) (2)).
receive a pre-trained large learning model via the network interface; This limitation is directed to insignificant extra solution activity (mere data gathering, as per MPEP 2106.05(g))
access user data stored locally; This limitation is directed to insignificant extra solution activity (mere data gathering, as per MPEP 2106.05(g))
train the pre-trained large learning model using the locally stored user data to provide an enhanced local large learning model; Adding the words "apply it" (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea [see MPEP 2106.05(f)] and therefore fails to integrate the exception into a practical application.
at least partly in response to determining that the computer system is permitted to insert content provided by the enhanced, local large learning model, cause content generated or selected by the enhanced, local large learning model to be rendered at the document space configured to receive third-party content. Adding the words "apply it" (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea [see MPEP 2106.05(f)] and therefore fails to integrate the exception into a practical application.
Step 2B – Does the claim recite additional elements that amount to significantly more than the judicial exception?
No, there are no additional elements that amount to significantly more than the judicial exception. The additional elements are:
A computer system associated with a user, the computer system comprising:
This limitation is directed to a computer merely used as a tool to perform an existing process (see MPEP 2106.05(f) (2)).
a network interface; This limitation is directed to a computer merely used as a tool to perform an existing process (see MPEP 2106.05(f) (2)).
at least one processing device operable to - This limitation is directed to a computer merely used as a tool to perform an existing process (see MPEP 2106.05(f) (2)).
receive a pre-trained large learning model via the network interface; This limitation is directed to receiving or transmitting data over a network. The courts have recognized receiving or transmitting data over a network as well understood, routine, and conventional functions when they are claimed in a merely generic manner (e.g., at a high level of generality) or as insignificant extra-solution activity (see MPEP 2106.05(d) II.).
access user data stored locally; This limitation is directed to receiving or transmitting data over a network. The courts have recognized receiving or transmitting data over a network as well understood, routine, and conventional functions when they are claimed in a merely generic manner (e.g., at a high level of generality) or as insignificant extra-solution activity (see MPEP 2106.05(d) II.).
train the pre-trained large learning model using the locally stored user data to provide an enhanced local large learning model; Adding the words "apply it" (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea [see MPEP 2106.05(f)] and therefore fails to integrate the exception into a practical application.
at least partly in response to determining that the computer system is permitted to insert content provided by the enhanced, local large learning model, cause content generated or selected by the enhanced, local large learning model to be rendered at the document space configured to receive third-party content. Adding the words "apply it" (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea [see MPEP 2106.05(f)] and therefore fails to integrate the exception into a practical application.
Regarding claim 2,
Claim 2 is rejected under 35 U.S.C 101 because the claimed invention is directed to an abstract idea without significantly more. The claim is dependent on claim 1 which includes an abstract idea (see rejection for claim 1). The additional limitations:
wherein training the pre-trained large learning model using the locally stored user data to provide the enhanced local large learning model, further comprises performing transfer learning. Adding the words "apply it" (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea [see MPEP 2106.05(f)] and therefore fails to integrate the exception into a practical application.
Regarding claim 3,
Claim 3 is rejected under 35 U.S.C 101 because the claimed invention is directed to an abstract idea without significantly more. The claim is dependent on claim 2 which includes an abstract idea (see rejection for claim 2). The additional limitations:
an encoder configured to receive an input sequence and process it using multi-head self-attention, This limitation is directed to insignificant extra solution activity (mere data gathering, as per MPEP 2106.05(g)) under Step 2A Prong 2, and is directed to receiving or transmitting data over a network. The courts have recognized receiving or transmitting data over a network as well understood, routine, and conventional functions when they are claimed in a merely generic manner (e.g., at a high level of generality) or as insignificant extra-solution activity (see MPEP 2106.05(d) II.) under Step 2B.
where the input sequence is transformed into a set of query, key, and value vectors used to compute attention scores between respective given positions in the input; This claim is directed to mathematical calculation (see MPEP 2106.04(a)(2) l. C.)
a decoder configured to receive an output of the encoder and to generate an output sequence, This limitation is directed to insignificant extra solution activity (mere data gathering and data outputting, as per MPEP 2106.05(g)) under Step 2A Prong 2, and is directed to receiving or transmitting data over a network. The courts have recognized receiving or transmitting data over a network as well understood, routine, and conventional functions when they are claimed in a merely generic manner (e.g., at a high level of generality) or as insignificant extra-solution activity (see MPEP 2106.05(d) II.) under Step 2B.
the decoder configured to utilize multi-head attention and to generate an output sequence using relevant information from the input sequence. Adding the words "apply it" (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea [see MPEP 2106.05(f)] and therefore fails to integrate the exception into a practical application.
Regarding claim 4,
Claim 4 is rejected under 35 U.S.C 101 because the claimed invention is directed to an abstract idea without significantly more. The claim is dependent on claim 1 which includes an abstract idea (see rejection for claim 1). The additional limitations:
wherein the locally stored user data comprises user electronic communications. This claim merely recites a further limitation on the access user data stored locally from Claim 1 which was directed to adding the words "apply it" (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea [see MPEP 2106.05(f)] and therefore fails to integrate the exception into a practical application.
Regarding claim 5,
Claim 5 is rejected under 35 U.S.C 101 because the claimed invention is directed to an abstract idea without significantly more. The claim is dependent on claim 1 which includes an abstract idea (see rejection for claim 1). The additional limitations:
wherein the content generated or selected by the enhanced, local large learning model comprises text and image content This claim merely recites a further limitation on the examine the document to determine if the computer system is permitted to insert content provided by the enhanced, local large learning model from Claim 1 which was directed to the abstract idea of a mental process (including an observation, evaluation, judgement, opinion) which can be performed in the human mind, or by a human using pen and paper (see MPEP 2106.04(a)(2) Ill. C.)
Regarding claim 6,
Claim 6 is rejected under 35 U.S.C 101 because the claimed invention is directed to an abstract idea without significantly more. The claim is dependent on claim 1 which includes an abstract idea (see rejection for claim 1). The additional limitations:
wherein identifying the document space configured to receive third-party content, -This limitation is directed to the abstract idea of a mental process (including an observation, evaluation, judgement, opinion) which can be performed in the human mind, or by a human using pen and paper (see MPEP 2106.04(a)(2) Ill. C.)
examining the document to determine if the computer system is permitted to insert content provided by the enhanced, local large learning model, and This limitation is directed to the abstract idea of a mental process (including an observation, evaluation, judgement, opinion) which can be performed in the human mind, or by a human using pen and paper (see MPEP 2106.04(a)(2) Ill. C.)
causing content generated or selected by the enhanced, local large learning model to be rendered at the document space configured to receive third-party content, is performed in substantially real time. Adding the words "apply it" (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea [see MPEP 2106.05(f)] and therefore fails to integrate the exception into a practical application.
Regarding claim 7,
Claim 7 is rejected under 35 U.S.C 101 because the claimed invention is directed to an abstract idea without significantly more. The claim is dependent on claim 1 which includes an abstract idea (see rejection for claim 1). The additional limitations:
wherein examining the document to determine if the computer system is permitted to insert content provided by the enhanced, local large learning model further comprises This limitation is directed to the abstract idea of a mental process (including an observation, evaluation, judgement, opinion) which can be performed in the human mind, or by a human using pen and paper (see MPEP 2106.04(a)(2) Ill. C.)
determining a presence of a first tag, a first txt, or a first Disallow command inserted into HTML code of the document. This limitation is directed to the abstract idea of a mental process (including an observation, evaluation, judgement, opinion) which can be performed in the human mind, or by a human using pen and paper (see MPEP 2106.04(a)(2) Ill. C.)
Regarding claim 8 and 15, they are rejected under the same rationale with claim 1 because they are analogous claims.
Regarding claim 9 and 16, they are rejected under the same rationale with claim 2 because they are analogous claims.
Regarding claim 10 and 17, they are rejected under the same rationale with claim 3 because they are analogous claims.
Regarding claim 11 and 18, they are rejected under the same rationale with claim 4 because they are analogous claims.
Regarding claim 12 and 19, they are rejected under the same rationale with claim 5 because they are analogous claims.
Regarding claim 13 and 20, they are rejected under the same rationale with claim 6 because they are analogous claims.
Regarding claim 14 and 21, they are rejected under the same rationale with claim 7 because they are analogous claims.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 1, 5-8, 12-15, 19-21 are rejected under 35 U.S.C. 103 as being unpatentable over Bonawitz et al. (US 2020/0401946 A1) in view of Bashyam et al. (US 2009/0144159 A1)
Regarding claim 1, Bonawitz explicitly discloses:
A computer system associated with a user, the computer system comprising: a network interface; (Bonawitz, ¶[0052]: “As one example, the machine-learned model(s) 120 can be employed within the context of a text messaging application of the user computing device 102, an email application of the user computing device 102, a text composition interface of an application (e.g., a word processing application), or any other application. As other examples, the machine-learned model(s) 120 can be included as a browser plug-in, as an overlay or feature of a keyboard ( e.g., a virtual keyboard on the user computing device 102), or in other contexts”)
at least one processing device operable to: receive a pre-trained large learning model via the network interface; (Bonawitz, ¶[0027]: “In some implementations, the machine-learned model can be delivered to the user computing device partially or fully pre-trained. For example, the server computing device can partially or fully train the model on centrally collected data prior to transmitting the model to the user computing device.”)
access user data stored locally; (Bonawitz, ¶[0026]: “As one example, in some implementations, a new machine-learned model can be trained by the user computing device based on the locally logged data. In particular, an untrained model structure and a training algorithm can be provided to the user computing device (e.g., from a server computing device). The user computing device can be instructed to use the entries in the local database (or some subset thereof) as training data for the training algorithm.”)
train the pre-trained large learning model using the locally stored user data to provide an enhanced local large learning model; (Bonawitz, ¶[0054]: “In particular, the model trainer 122 can train or re-train one or more of the machine-learned models 120 using the locally logged data 119 as training data. The model trainer 122 can perform a number of generalization techniques ( e.g., weight decays, dropouts, etc.) to improve the generalization capability of the models being trained.”)
Bonawitz fails to disclose:
detect that a browser hosted by the computer system is accessing a document;
identify a document space configured to receive third-party content;
examine the document to determine if the computer system is permitted to insert content provided by the enhanced, local large learning model;
at least partly in response to determining that the computer system is permitted to insert content provided by the enhanced, local large learning model, cause content generated or selected by the enhanced, local large learning model to be rendered at the document space configured to receive third-party content
However, Bashyam explicitly discloses:
detect that a browser hosted by the computer system is accessing a document; (Bashyam, ¶[0016]: “For example, the exemplary system can include a browser requesting access to content from content servers through the Internet”, ¶[0017]: “User agent 102 is a client application used with a network protocol. For example, user agent 102 could be a web browser, a search engine crawler, a screen reader, or a Braille browser, and user agent 102 could be used to access the Internet. User agent 102 can be a software program that transmits request data (e.g., an HTTP/HTTPS/WAP/WAIS/ Gopher/RTSP request, etc.) to a web server and receives response data in response to the request data. For example, user agent 102 can send request data to content servers 118- 120 for a particular file or object data of a web page by its URL, and the content server of the web page can query the object data in a database and can send back the object data as part of the response data (e.g., HTTP/WAP response data) to the user agent.”)
identify a document space configured to receive third-party content; (Bashyam, ¶[0096]: “In yet another scenario, the original ads in the web content can be just an ad placeholder, which reserves a space for ads but no ads are placed yet. The ad insertion module can select a curtailed ad campaign and place it in the ad placeholder”)
examine the document to determine if the computer system is permitted to insert content provided by the enhanced, local large learning model; (Bashyam, ¶[0095]: “Thus, it is in the user's and the carrier's best interests to replace the ads with carefully selected ads, which are curtailed to meet the particular need of the user and/or to fit in the specification of a mobile device. In the best interests of the both the content provider and an advertiser, a prior permission from the web content provider/owner may be obtained before the replacement takes place. The permission can be a prior agreement, or a permission given/embedded in the response data.”, ¶[0097]: “Referring back to FIG. 10, assuming the permission for ads replacement is given, after receiving the response data, the proxy determines (1004) whether inline-replacement of ads in the web content is preferred. If inline-replacement is preferred, the proxy removes (1006) the ads in the web content, and replaces (1008) the ads in the web content with an ad campaign selected based on the user profiling data, as described above.”)
at least partly in response to determining that the computer system is permitted to insert content provided by the enhanced, local large learning model, cause content generated or selected by the enhanced, local large learning model to be rendered at the document space configured to receive third-party content. (Bashyam, ¶[0098]: “After replacing the ads of the web content, the proxy provides (1010) the web content with replaced ads to the mobile device.”)
The combination of Bonawitz and Bashyam are analogous art because they are in the same field of training time series data. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention, having the teachings of Bonawitz and Bashyam before them, to modify the teachings of Bonawitz to include the teachings of Bashyam’s selecting user relevant advertisements and inserting them into webpage advertising spaces subject to provider permission to provide a practical use for Bonawitz’s personalized predictions by presenting advertisements tailored to the user’s interests during web browsing while respecting the webpage provider’s insertion restrictions.
Regarding claim 5, the combination of Bonawitz and Bashyam discloses all the limitations of claim 1 (as shown in the rejections above).
Bonawitz in view of Bashyam further discloses:
wherein the content generated or selected by the enhanced, local large learning model comprises text and image content. (Bashyam, ¶[0079]: “For example, if the ad parameters of the mobile device and user profiling data indicate that the user is interested in buying a car and the device supports video streams, then the proxy would select a video ad campaign of cars. If the device does not support video stream, the proxy would select a still image ad campaign.”, ¶[0065]: “Subsequently, content server 118 provides (512) response data ( e.g., HTTP response), associated with the request data, to response monitor 240 of proxy 110. The response data can include, among other things, an HTML document, a Cascaded Style Sheet Files, images, and one or more JavaScript files, all of which constitute the requested webpage.”)
Regarding claim 6, the combination of Bonawitz and Bashyam discloses all the limitations of claim 1 (as shown in the rejections above).
Bonawitz in view of Bashyam further discloses:
wherein identifying the document space configured to receive third-party content, examining the document to determine if the computer system is permitted to insert content provided by the enhanced, local large learning model, and causing content generated or selected by the enhanced, local large learning model to be rendered at the document space configured to receive third-party content, is performed in substantially real time. (Bashyam, ¶[0093]: “After initial start step 1000, a proxy receives (1002) response data, which includes web content, from a content server. This web content can correspond to a web content requested by a user.”, ¶[0095]: “The permission can be a prior agreement, or a permission given/embedded in the response data”, ¶[0097]: “Referring back to FIG. 10, assuming the permission for ads replacement is given, after receiving the response data, the proxy determines (1004) whether inline-replacement of ads in the web content is preferred. If inline-replacement is preferred, the proxy removes (1006) the ads in the web content, and replaces (1008) the ads in the web content with an ad campaign selected based on the user profiling data, as described above.”)
Regarding claim 7, the combination of Bonawitz and Bashyam discloses all the limitations of claim 1 (as shown in the rejections above).
Bonawitz in view of Bashyam further discloses:
wherein examining the document to determine if the computer system is permitted to insert content provided by the enhanced, local large learning model further comprises determining a presence of a first tag, a first txt, or a first Disallow command inserted into HTML code of the document. (Bashyam, ¶[0096]: “The ad insertion module can remove or strip out the original ads, spaces and corresponding markup language tags (such as HTML tags or XML tags) from the web content, and place the curtailed ad campaign off the web content, as described above.”, ¶[0097]: “Referring back to FIG. 10, assuming the permission for ads replacement is given, after receiving the response data, the proxy determines (1004) whether inline-replacement of ads in the web content is preferred. If inline-replacement is preferred, the proxy removes (1006) the ads in the web content, and replaces (1008) the ads in the web content with an ad campaign selected based on the user profiling data, as described above.”)
Regarding claim 8 and claim 15, they are rejected under the same rationale with claim 1 because they are analogous claims.
Regarding claim 12 and claim 19, they are rejected under the same rationale with claim 5 because they are analogous claims.
Regarding claim 13 and claim 20, they are rejected under the same rationale with claim 6 because they are analogous claims.
Regarding claim 14 and claim 21, they are rejected under the same rationale with claim 7 because they are analogous claims.
Claim(s) 2, 4, 9, 11, 16 and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Bonawitz et al. (US 2020/0401946 A1) in view of Bashyam et al. (US 2009/0144159 A1) and further in view of Zhang et al. (“Towards Building the Federated GPT: Federated Instruction Tuning”)
Regarding claim 2, the combination of Bonawitz and Bashyam discloses all the limitations of claim 1 (as shown in the rejections above).
Bonawitz in view of Bashyam fails to disclose:
wherein training the pre-trained large learning model using the locally stored user data to provide the enhanced local large learning model, further comprises performing transfer learning.
However, Zhang explicitly discloses:
wherein training the pre-trained large learning model using the locally stored user data to provide the enhanced local large learning model, further comprises performing transfer learning. (Zhang, Pg. 4, Section 3.1, ¶[3]: “The framework assigns an LLM to each client and performs client selection to determine which clients will participate in local instruction tuning. During instruction tuning, clients use their local instruction dataset to update a small, trainable adapter that is added to the pre-trained model weights. This approach reduces the cost of fine-tuning and makes it compatible with the limited computational resources of local devices”)
The combination of Bonawitz, Bashyam and Zhang are analogous art because they are in the same field of training time series data. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention, having the teachings of Bonawitz, Bashyam and Zhang before them, to modify the teachings of Bonawitz and Bashyam to include the teachings of Zhang to enable the system to adapt language-based content selection to locally stored user instructions while reducing the computational and memory demands of fine-tuning, making a large language model more practical for resource-constrained user devices and retaining the training local for privacy.
Regarding claim 4, the combination of Bonawitz and Bashyam discloses all the limitations of claim 1 (as shown in the rejections above).
Bonawitz in view of Bashyam fails to disclose:
wherein the locally stored user data comprises user electronic communications
However, Zhang explicitly discloses:
wherein the locally stored user data comprises user electronic communications. (Zhang, Pg. 4, FIG. 1:
PNG
media_image1.png
499
911
media_image1.png
Greyscale
, Pg. 4, Section 3.1: “The overall framework, illustrated in Figure 1 and Algorithm 1, involves two primary components: local training operations on the client side and scheduling and aggregation operations on the server side, which work together to ensure efficient training”)
Regarding claim 9 and claim 16, they are rejected under the same rationale with claim 2 because they are analogous claims.
Regarding claim 11 and claim 18, they are rejected under the same rationale with claim 4 because they are analogous claims.
Claim(s) 3, 10 and 17 are rejected under 35 U.S.C. 103 as being unpatentable over Bonawitz et al. (US 2020/0401946 A1) in view of Bashyam et al. (US 2009/0144159 A1) and further in view of Vaswani et al. (“Attention Is All You Need”)
Regarding claim 3, the combination of Bonawitz and Bashyam discloses all the limitations of claim 1 (as shown in the rejections above).
Bonawitz in view of Bashyam fails to disclose:
wherein the large learning model comprises: an encoder configured to receive an input sequence and process it using multi-head self-attention,
where the input sequence is transformed into a set of query, key, and value vectors used to compute attention scores between respective given positions in the input;
a decoder configured to receive an output of the encoder and to generate an output sequence,
the decoder configured to utilize multi-head attention and to generate an output sequence using relevant information from the input sequence.
However, Vaswani explicitly discloses:
wherein the large learning model comprises: an encoder configured to receive an input sequence and process it using multi-head self-attention, (Vaswani, Pg. 2, Section 3: “Here, the encoder maps an input sequence of symbol representations (x1 ,…, xn) to a sequence of continuous representations z = (z1, …, zn).”, Pg. 3, FIG. 1:
PNG
media_image2.png
656
461
media_image2.png
Greyscale
, Pg. 3, Section 3.1: “Encoder: The encoder is composed of a stack of N = 6 identical layers. Each layer has two sub-layers. The first is a multi-head self-attention mechanism, and the second is a simple, position-wise fully connected feed-forward network”)
where the input sequence is transformed into a set of query, key, and value vectors used to compute attention scores between respective given positions in the input; (Vaswani, Pg. 3, Section 3.2: “An attention function can be described as mapping a query and a set of key-value pairs to an output, where the query, keys, values, and output are all vectors.”, Pg. 4, Section 3.2.1: “We call our particular attention "Scaled Dot-Product Attention" (Figure 2). The input consists of queries and keys of dimension dk, and values of dimension dv. We compute the dot products of the query with all keys, divide each by
d
k
, and apply a softmax function to obtain the weights on the values. In practice, we compute the attention function on a set of queries simultaneously, packed together into a matrix Q. The keys and values are also packed together into matrices K and V . We compute the matrix of outputs as:
PNG
media_image3.png
65
359
media_image3.png
Greyscale
”)
a decoder configured to receive an output of the encoder and to generate an output sequence, (Vaswani, Pg. 2, Section 3: “Given z, the decoder then generates an output sequence (y1,…, ym) of symbols one element at a time”)
the decoder configured to utilize multi-head attention and to generate an output sequence using relevant information from the input sequence. (Vaswani, Pg. 3, Section 3.1, ¶[2]: “The decoder is also composed of a stack of N = 6 identical layers. In addition to the two sub-layers in each encoder layer, the decoder inserts a third sub-layer, which performs multi-head attention over the output of the encoder stack. Similar to the encoder, we employ residual connections around each of the sub-layers, followed by layer normalization. We also modify the self-attention sub-layer in the decoder stack to prevent positions from attending to subsequent positions. This masking, combined with fact that the output embeddings are offset by one position, ensures that the predictions for position i can depend only on the known outputs at positions less than i.”)
The combination of Bonawitz, Bashyam and Vaswani are analogous art because they are in the same field of training time series data. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention, having the teachings of Bonawitz, Bashyam and Vaswani before them, to modify the teachings of Bonawitz and Bashyam to include the teachings of Vaswani to enable the model to capture relationships among different input positions and attend to different representation subspaces in parallel, improving its ability to represent contextual information while reducing reliance on sequential computation.
Regarding claim 10 and claim 17, they are rejected under the same rationale with claim 3 because they are analogous claims.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to AMY TRAN whose telephone number is (571)270-0693. The examiner can normally be reached Monday - Friday 7:30 am - 5:00 pm EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, David Yi can be reached at (571) 270-7519. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/AMY TRAN/Examiner, Art Unit 2126
/DAVID YI/Supervisory Patent Examiner, Art Unit 2126
Prosecution Insights