Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement submitted on 08/04/2026 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Response to Amendment
The Amendment filed 08/21/2026 has been entered. Claims 1-20 remain pending in the application.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 8, 10-11, 15-16 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Lepikhin et al (US 20230222318 A1 hereinafter Lepikhin) in view of Ruiz et al. (US 20230196211 A1 hereinafter Ruiz) and Bhat et al. (US 20240331367 A1) hereinafter Bhat.
As to independent claim 1, Lepikhin teaches a computer-implemented method for executing a transformer neural network, the method comprising: [transformer based neural network ¶6]
executing a first attention unit included in the transformer neural network to convert a first input token into a first query, a first key, and a first plurality of values, [transformer neural network (¶6), with QKV attention head (units) ¶42 " Each attention head generates a set of queries, a set of keys, and a set of values"]
computing a first plurality of outputs associated with the first input token based on the first query, the first key, and the first plurality of values; and [computes QKV used in generating outputs which are also combined ¶42 ". Each attention head generates a set of queries, a set of keys, and a set of values, and then applies any of a variety of variants of query-key-value (QKV) attention using the queries, keys, and values to generate an output. When there are multiple attention heads, the sub-layer 120 then combines the outputs of the multiple attention heads, e.g., by concatenating the outputs and, optionally, processing the concatenated outputs through a linear layer."]
performing a task associated with an input corresponding to the first input token based on the first input token and the first plurality of outputs. [performs task ¶16, based on input, tokens and outputs ¶6 " self-attention neural network, resulting in a significant increase in quality of outputs generated for tasks that require processing input sequences, generating output sequences, or both, without a significant increase in the computation cost. Moreover, by parallelizing the resulting self-attention neural network across multiple devices during training as described in this specification, the self-attention neural network can be effectively trained despite having significantly more parameters than existing self-attention networks. Additionally, by selecting which experts are used for any given input at any given position (also referred to as a “token”)"]
Lepikhin does not specifically teach wherein each value included in the first plurality of values represents a sub-task associated with the transformer neural network.
However, Ruiz teaches wherein each value included in the first plurality of values represents a sub-task associated with the transformer neural network; [uses a mixture of experts to perform subproblems (sub-tasks) in a network based on training data (values) with downstream tasks ¶30-31 "implementations train a diverse set of experts (e.g., by exploiting existing label structures in training data) and use cheap-to-compute performance proxies to select the relevant expert for each target task. For example, the mixture of experts can include multiple heterogeneous model variants (“experts”) that specialize to subproblems of a full upstream task. "]
Accordingly, it would have been obvious to a person of ordinary skill in the art before the effective filling date of the claimed invention to modify the attention model development disclosed by Lepikhin by incorporating the wherein each value included in the first plurality of values represents a sub-task associated with the transformer neural network disclosed by Ruiz because both techniques address the same field of machine learning and by incorporating Ruiz into Lepikhin improves performance especially on diverse sets of tasks with cheaper training [Ruiz ¶31]
Lepikhin and Ruiz do not specifically teach performing a task associated with an input corresponding to the first input token based on a subset of the first plurality of outputs that is selected based on the first input token and a task token associated with the task.
However, Bhat teaches performing a task associated with an input corresponding to the first input token based on a subset of the first plurality of outputs that is selected based on the first input token and a task token associated with the task. [task attention with a task token for performing tasks on feature subset (outputs become inputs) ¶45-50 "attend to features important for the current task"…"attention module can be equipped with a learnable task token δ.sub.i associated with each task"]
Accordingly, it would have been obvious to a person of ordinary skill in the art before the effective filling date of the claimed invention to modify the attention model disclosed by Lepikhin and Ruiz by incorporating the performing a task associated with an input corresponding to the first input token based on a subset of the first plurality of outputs that is selected based on the first input token and a task token associated with the task disclosed by Bhat because all techniques address the same field of machine learning and by incorporating Bhat into Lepikhin and Ruiz improves model ability to perform longer tasks alleviating capacity saturation [Bhat ¶6].
As to dependent claim 8, the rejection of claim 1 is incorporated, Lepikhin, Ruiz and Bhat further teach wherein performing the task comprises: generating a second input token based on the first plurality of outputs; [Lepikhin input/output sequences (tokens) ¶6]
executing a second attention unit included in the transformer neural network to convert the second input token into an output token; and [Lepikhin multiple attention layers in series Fig. 4, ¶54-55 "Attended layer inputs that are provided as input to a given sub-layer will also be referred to in this specification as “tokens.”"]
performing the task based on the output token. [Lepikhin prediction task ¶20]
As to dependent claim 10, the rejection of claim 1 is incorporated, Lepikhin, Ruiz and Bhat further teach wherein performing the task comprises generating a prediction of a class for a data sample associated with the first input token. [Lepikhin prediction task such as a diagnosis or presence of condition (class) ¶20]
As to independent claim 11, Lepikhin teaches one or more non-transitory computer-readable media storing instructions that, when executed by one or more processors, cause the one or more processors to perform the steps of: [medium, processor and instructions ¶122-123]
executing a first attention unit included in the transformer neural network to convert a first input token into a first query, a first key, and a first plurality of values, [transformer neural network (¶6), with QKV attention head (units) ¶42 " Each attention head generates a set of queries, a set of keys, and a set of values"]
computing a first plurality of outputs associated with the first input token based on the first query, the first key, and the first plurality of values; and [computes QKV used in generating outputs which are also combined ¶42 ". Each attention head generates a set of queries, a set of keys, and a set of values, and then applies any of a variety of variants of query-key-value (QKV) attention using the queries, keys, and values to generate an output. When there are multiple attention heads, the sub-layer 120 then combines the outputs of the multiple attention heads, e.g., by concatenating the outputs and, optionally, processing the concatenated outputs through a linear layer."]
performing a task associated with an input corresponding to the first input token based on the first input token and the first plurality of outputs. [performs task ¶16, based on input, tokens and outputs ¶6 " self-attention neural network, resulting in a significant increase in quality of outputs generated for tasks that require processing input sequences, generating output sequences, or both, without a significant increase in the computation cost. Moreover, by parallelizing the resulting self-attention neural network across multiple devices during training as described in this specification, the self-attention neural network can be effectively trained despite having significantly more parameters than existing self-attention networks. Additionally, by selecting which experts are used for any given input at any given position (also referred to as a “token”)"]
Lepikhin does not specifically teach wherein each value included in the first plurality of values represents a sub-task associated with the transformer neural network.
However, Ruiz teaches wherein each value included in the first plurality of values represents a sub-task associated with the transformer neural network; [uses a mixture of experts to perform subproblems (sub-tasks) in a network based on training data (values) with downstream tasks ¶30-31 "implementations train a diverse set of experts (e.g., by exploiting existing label structures in training data) and use cheap-to-compute performance proxies to select the relevant expert for each target task. For example, the mixture of experts can include multiple heterogeneous model variants (“experts”) that specialize to subproblems of a full upstream task. "]
Accordingly, it would have been obvious to a person of ordinary skill in the art before the effective filling date of the claimed invention to modify the attention model development disclosed by Lepikhin by incorporating the wherein each value included in the first plurality of values represents a sub-task associated with the transformer neural network disclosed by Ruiz because both techniques address the same field of machine learning and by incorporating Ruiz into Lepikhin improves performance especially on diverse sets of tasks with cheaper training [Ruiz ¶31].
Lepikhin and Ruiz do not specifically teach performing a task associated with an input corresponding to the first input token based on a subset of the first plurality of outputs that is selected based on the first input token and a task token associated with the task.
However, Bhat teaches performing a task associated with an input corresponding to the first input token based on a subset of the first plurality of outputs that is selected based on the first input token and a task token associated with the task. [task attention with a task token for performing tasks on feature subset (outputs become inputs) ¶45-50 "attend to features important for the current task"…"attention module can be equipped with a learnable task token δ.sub.i associated with each task"]
Accordingly, it would have been obvious to a person of ordinary skill in the art before the effective filling date of the claimed invention to modify the attention model disclosed by Lepikhin and Ruiz by incorporating the performing a task associated with an input corresponding to the first input token based on a subset of the first plurality of outputs that is selected based on the first input token and a task token associated with the task disclosed by Bhat because all techniques address the same field of machine learning and by incorporating Bhat into Lepikhin and Ruiz improves model ability to perform longer tasks alleviating capacity saturation [Bhat ¶6].
As to dependent claim 15, the rejection of claim 11 is incorporated, Lepikhin, Ruiz and Bhat further teach wherein performing the task comprises: generating a second input token based on the subset of the first plurality of outputs; [Lepikhin input/output sequences (tokens) ¶6]
executing a second attention unit included in the transformer neural network to convert the second input token into an output token; and [Lepikhin multiple attention layers in series Fig. 4, ¶54-55 "Attended layer inputs that are provided as input to a given sub-layer will also be referred to in this specification as “tokens.”"]
performing the task based on the output token. [Lepikhin prediction task ¶20]
As to dependent claim 16, the rejection of claim 15 is incorporated, Lepikhin, Ruiz and Bhat further teach wherein performing the task based on the output token comprises generating a prediction based on the output token. [Lepikhin prediction task such as a diagnosis or presence of condition (class) ¶20]
As to independent claim 20, Lepikhin teaches a system, comprising: [system ¶121]
one or more memories that store instructions, and one or more processors that are coupled to the one or more memories and, when executing the instructions, are configured to perform the steps of: [memory, processor and instructions ¶121-123]
executing a first attention unit included in the transformer neural network to convert a first input token into a first query, a first key, and a first plurality of values, [transformer neural network (¶6), with QKV attention head (units) ¶42 " Each attention head generates a set of queries, a set of keys, and a set of values"]
computing a first plurality of outputs associated with the first input token based on the first query, the first key, and the first plurality of values; and [computes QKV used in generating outputs which are also combined ¶42 ". Each attention head generates a set of queries, a set of keys, and a set of values, and then applies any of a variety of variants of query-key-value (QKV) attention using the queries, keys, and values to generate an output. When there are multiple attention heads, the sub-layer 120 then combines the outputs of the multiple attention heads, e.g., by concatenating the outputs and, optionally, processing the concatenated outputs through a linear layer."]
performing a task associated with an input corresponding to the first input token based on the first input token and the first plurality of outputs. [performs task ¶16, based on input, tokens and outputs ¶6 " self-attention neural network, resulting in a significant increase in quality of outputs generated for tasks that require processing input sequences, generating output sequences, or both, without a significant increase in the computation cost. Moreover, by parallelizing the resulting self-attention neural network across multiple devices during training as described in this specification, the self-attention neural network can be effectively trained despite having significantly more parameters than existing self-attention networks. Additionally, by selecting which experts are used for any given input at any given position (also referred to as a “token”)"]
Lepikhin does not specifically teach wherein each value included in the first plurality of values represents a sub-task associated with the transformer neural network.
However, Ruiz teaches wherein each value included in the first plurality of values represents a sub-task associated with the transformer neural network; [uses a mixture of experts to perform subproblems (sub-tasks) in a network based on training data (values) with downstream tasks ¶30-31 "implementations train a diverse set of experts (e.g., by exploiting existing label structures in training data) and use cheap-to-compute performance proxies to select the relevant expert for each target task. For example, the mixture of experts can include multiple heterogeneous model variants (“experts”) that specialize to subproblems of a full upstream task. "]
Accordingly, it would have been obvious to a person of ordinary skill in the art before the effective filling date of the claimed invention to modify the attention model development disclosed by Lepikhin by incorporating the wherein each value included in the first plurality of values represents a sub-task associated with the transformer neural network disclosed by Ruiz because both techniques address the same field of machine learning and by incorporating Ruiz into Lepikhin improves performance especially on diverse sets of tasks with cheaper training [Ruiz ¶31].
Lepikhin and Ruiz do not specifically teach performing a task associated with an input corresponding to the first input token based on a subset of the first plurality of outputs that is selected based on the first input token and a task token associated with the task.
However, Bhat teaches performing a task associated with an input corresponding to the first input token based on a subset of the first plurality of outputs that is selected based on the first input token and a task token associated with the task. [task attention with a task token for performing tasks on feature subset (outputs become inputs) ¶45-50 "attend to features important for the current task"…"attention module can be equipped with a learnable task token δ.sub.i associated with each task"]
Accordingly, it would have been obvious to a person of ordinary skill in the art before the effective filling date of the claimed invention to modify the attention model disclosed by Lepikhin and Ruiz by incorporating the performing a task associated with an input corresponding to the first input token based on a subset of the first plurality of outputs that is selected based on the first input token and a task token associated with the task disclosed by Bhat because all techniques address the same field of machine learning and by incorporating Bhat into Lepikhin and Ruiz improves model ability to perform longer tasks alleviating capacity saturation [Bhat ¶6].
Claims 2 and 12 are rejected under 35 U.S.C. 103 as being unpatentable over Lepikhin in view of Ruiz and Bhatr, as applied in claim 1 and 11 above, and further in view of Shazeer et al. (US 20200089755 A1 hereinafter Shazeer)
As to dependent claim 2, Lepikhin, Ruiz and Bhat teach the method of claim 1 above that is incorporated,
Lepikhin, Ruiz and Bhat do not specifically teach wherein performing the task comprises: determining a plurality of keys used to select the subset of the first plurality of outputs; and generating an output token based on the plurality of keys and the first plurality of outputs.
However, Shazeer teaches wherein performing the task comprises: determining a plurality of keys used to select the subset of the first plurality of outputs; and [generates keys from tensor output from layers ¶24 "convolutional layers to generate a mixed tensor; self-attend the mixed tensor to generate query keys; generate memory keys and memory values by passing the source tensor through one or more pointwise convolutional layers"]
generating an output token based on the plurality of keys and the first plurality of outputs. [Keys used for attention output ¶78, ¶24 "use the generated query keys, memory keys and memory values to apply an attention mechanism"]
Accordingly, it would have been obvious to a person of ordinary skill in the art before the effective filling date of the claimed invention to modify the attention model disclosed by Lepikhin, Ruiz and Bhat by incorporating the wherein performing the task comprises: determining a plurality of keys used to select the subset of the first plurality of outputs; and generating an output token based on the plurality of keys and the first plurality of outputs disclosed by Shazeer because all techniques address the same field of machine learning and by incorporating Shazeer into Lepikhin, Ruiz and Bhat enhances multi-task performance across tasks of different domain [Shazeer ¶29]
As to dependent claim 12, Lepikhin, Ruiz and Bhat teach the method of claim 11 above that is incorporated,
Lepikhin, Ruiz and Bhat do not specifically teach wherein performing the task comprises: determining a plurality of keys used to select the subset of the first plurality of outputs; and generating an output token based on the plurality of keys and the first plurality of outputs.
However, Shazeer teaches wherein performing the task comprises: determining a plurality of keys used to select the subset of the first plurality of outputs; and [generates keys from tensor output from layers ¶24 "convolutional layers to generate a mixed tensor; self-attend the mixed tensor to generate query keys; generate memory keys and memory values by passing the source tensor through one or more pointwise convolutional layers"]
generating an output token based on the plurality of keys and the first plurality of outputs. [Keys used for attention output ¶78, ¶24 "use the generated query keys, memory keys and memory values to apply an attention mechanism"]
Accordingly, it would have been obvious to a person of ordinary skill in the art before the effective filling date of the claimed invention to modify the attention model disclosed by Lepikhin, Ruiz and Bhat by incorporating the wherein performing the task comprises: determining a plurality of keys used to select the subset of the first plurality of outputs; and generating an output token based on the plurality of keys and the first plurality of outputs disclosed by Shazeer because all techniques address the same field of machine learning and by incorporating Shazeer into Lepikhin, Ruiz and Bhat enhances multi-task performance across tasks of different domain [Shazeer ¶29]
Claims 3-4 and 13-14 are rejected under 35 U.S.C. 103 as being unpatentable over Lepikhin in view of Ruiz, Bhat and Shazeer, as applied in claim 2 and 12 above, and further in view of ZHUO et al. (US 20250094775 A1 hereinafter Zhuo)
As to dependent claim 3, Lepikhin, Ruiz, Bhat and Shazeer teach the method of claim 2 above that is incorporated,
Lepikhin, Ruiz, Bhat and Shazeer do not specifically teach computing a plurality of scores based on the plurality of keys and the task token; and computing the output token based on the plurality of scores and the first plurality of outputs.
However, Zhuo teaches computing a plurality of scores based on the plurality of keys and the task token; and [computes scores based on keys, query and values related to tasks ¶21, ¶40 "The attention block 240 can process the query 215A, the key 225A, and the value 235A to generate attention scores 260A "]
computing the output token based on the plurality of scores and the first plurality of outputs. [generates output token using score ¶40 "generate attention scores 260A and eventually intermediate activation tensor 265A and output token 275A."]
Accordingly, it would have been obvious to a person of ordinary skill in the art before the effective filling date of the claimed invention to modify the attention model disclosed by Lepikhin, Ruiz, Bhat and Shazeer by incorporating the computing a plurality of scores based on the plurality of keys and the task token; and computing the output token based on the plurality of scores and the first plurality of outputs disclosed by Zhuo because all techniques address the same field of machine learning and by incorporating Zhuo into Lepikhin, Ruiz, Bhat and Shazeer reduces inefficiencies and duplicated computations [Zhuo ¶2]
As to dependent claim 4, the method of claim 3 above is incorporated, Lepikhin, Ruiz, Bhat, Shazeer and Zhuo further teach wherein computing the output token comprises scaling the first plurality of outputs based on the plurality of scores. [Zhuo scaling factor during attention using scores ¶27-28, ¶40 “. The attention scores 260A can include a product of the query 215A with the key 225A (e.g., or a concatenated variant thereof), scaled according to the scaling factor d.sub.k 245, masked according to mask 250B, normalized using softmax function σ 255, and/or multiplied with the values (or concatenated variants thereof)”]
As to dependent claim 13, Lepikhin, Ruiz, Bhat and Shazeer teach the method of claim 12 above that is incorporated,
Lepikhin, Ruiz, Bhat and Shazeer further teach executing a second attention unit included in the transformer neural network to convert the first input token and the task token into a second task token; [Lepikhin multiple attention layers in series changing inputs to outputs Fig. 4, ¶54-55 "Attended layer inputs that are provided as input to a given sub-layer will also be referred to in this specification as “tokens.”"]
Lepikhin, Ruiz, Bhat and Shazeer do not specifically teach computing a plurality of scores based on the plurality of keys and a task token associated with the first input token; and computing the output token based on the plurality of scores and the first plurality of outputs.
However, Zhuo teaches computing a plurality of scores based on the plurality of keys and a task token associated with the first input token; and [computes scores based on keys, query and values related to tasks ¶21, ¶40 "The attention block 240 can process the query 215A, the key 225A, and the value 235A to generate attention scores 260A "]
computing the output token based on the plurality of scores and the first plurality of outputs. [generates output token using score ¶40 "generate attention scores 260A and eventually intermediate activation tensor 265A and output token 275A."]
Accordingly, it would have been obvious to a person of ordinary skill in the art before the effective filling date of the claimed invention to modify the attention model disclosed by Lepikhin, Ruiz, Bhat and Shazeer by incorporating the computing a plurality of scores based on the plurality of keys and a task token associated with the first input token; and computing the output token based on the plurality of scores and the first plurality of outputs disclosed by Zhuo because all techniques address the same field of machine learning and by incorporating Zhuo into Lepikhin, Ruiz, Bhat and Shazeer reduces inefficiencies and duplicated computations [Zhuo ¶2]
As to dependent claim 14, the method of claim 13 above is incorporated, Lepikhin, Ruiz, Bhat, Shazeer and Zhuo further teach wherein the output token is computed by weighting the first plurality of outputs using the plurality of scores. [Zhuo scales during attention based on the score (weighting) ¶27-28, ¶40 “. The attention scores 260A can include a product of the query 215A with the key 225A (e.g., or a concatenated variant thereof), scaled according to the scaling factor d.sub.k 245, masked according to mask 250B, normalized using softmax function σ 255, and/or multiplied with the values (or concatenated variants thereof)”]
Claims 5 is rejected under 35 U.S.C. 103 as being unpatentable over Lepikhin in view of Ruiz and Bhat, as applied in claim 1 above, and further in view of Radford et al. (US 12079587 B1 hereinafter Radford)
As to dependent claim 5, Lepikhin, Ruiz, and Bhat teach the method of claim 1 above that is incorporated,
Lepikhin, Ruiz, and Bhat do not specifically teach wherein performing the task comprises: executing a second attention unit included in the transformer neural network to convert the first input token and a first task token into a second task token; and generating an output token based on the second task token and the first plurality of outputs.
However, Radford teaches wherein performing the task comprises: executing a second attention unit included in the transformer neural network to convert the first input token and a first task token into a second task token; and [Radford Fig. 1B illustrate multiple attention in series 135, 139 and task tokens Col. 15-16 ln. 33-21 "decoder can predict a language token, a task token"]
generating an output token based on the second task token and the first plurality of outputs. [Radford that generate tokens from tokens 131 Col. 10 ln. 14-20, Col. 16 ln. 46-67 "generate output tokens based on (e.g., using) previously generated output tokens"]
Accordingly, it would have been obvious to a person of ordinary skill in the art before the effective filling date of the claimed invention to modify the attention model disclosed by Lepikhin, Ruiz, and Bhat by incorporating the wherein performing the task comprises: executing a second attention unit included in the transformer neural network to convert the first input token and a first task token into a second task token; and generating an output token based on the second task token and the first plurality of outputs disclosed by Radford because all techniques address the same field of machine learning and by incorporating Radford into Lepikhin, Ruiz, and Bhat provides a more reliable solution that’s adaptable to a broad range of environments [Radford Col. 1 ln. 22-30].
Claims 6 is rejected under 35 U.S.C. 103 as being unpatentable over Lepikhin in view of Ruiz, Bhat and Radford, as applied in claim 5 and 15 above, and further in view of Zhuo.
As to dependent claim 6, Lepikhin, Ruiz, Bhat and Radford teach the method of claim 5 above that is incorporated,
Lepikhin, Ruiz, Bhat and Radford further teach converting the first input token and the first task token into a second query, a second key, and a value; and [Lepikhin each attention head receive tokens and make QKV ¶42, Fig 4]
Lepikhin, Ruiz, Bhat and Radford do not specifically teach computing the second task token based on the value and an attention score associated with the second query and the second key.
However, Zhuo teaches computing the second task token based on the value and an attention score associated with the second query and the second key. [Zhuo attention scores based on the tokens ¶40]
Accordingly, it would have been obvious to a person of ordinary skill in the art before the effective filling date of the claimed invention to modify the attention model disclosed by Lepikhin, Ruiz, Bhat and Radford by incorporating the computing the second task token based on the value and an attention score associated with the second query and the second key disclosed by Zhuo because all techniques address the same field of machine learning and by incorporating Zhuo into Lepikhin, Ruiz, Bhat and Radford reduces inefficiencies and duplicated computations [Zhuo ¶2]
Claims 7 and 18-19 are rejected under 35 U.S.C. 103 as being unpatentable over Lepikhin in view of Ruiz, Bhat, as applied in claim 1 and 11 above, and further in view of SHAIB et al. (US 20210183484 A1 hereinafter Shaib)
As to dependent claim 7, Lepikhin, Ruiz and Bhat teach the method of claim 1 above that is incorporated,
Lepikhin, Ruiz and Bhat do not specifically teach wherein performing the task comprises: generating a latent representation of a data sample associated with the first input token based on the first plurality of outputs; and performing the task based on the subset of the latent representation.
However, Shaib teaches wherein performing the task comprises: generating a latent representation of a data sample associated with the first input token based on the subset of the first plurality of outputs; and [generates latent scores of notes (data samples) input for outputs ¶53-54 "generating the latent attention scores. Accordingly, these can be used as explainable outputs for assessing a prediction of an adverse outcome"]
performing the task based on the latent representation. [generates predictions accordingly ¶53-54]
Accordingly, it would have been obvious to a person of ordinary skill in the art before the effective filling date of the claimed invention to modify the attention model disclosed by Lepikhin, Ruiz and Bhat by incorporating the wherein performing the task comprises: generating a latent representation of a data sample associated with the first input token based on the subset of the first plurality of outputs; and performing the task based on the latent representation disclosed by Shaib because all techniques address the same field of machine learning and by incorporating Shaib into Lepikhin, Ruiz and Bhat provides explainability for more interpretable results [Shaib ¶10]
As to dependent claim 18, Lepikhin, Ruiz and Bhat teach the method of claim 11 above that is incorporated,
Lepikhin, Ruiz and Bhat do not specifically teach wherein the transformer neural network is included in a representation learning network that generates, based on the subset of the first plurality of outputs, a latent representation of a data sample associated with the first input token.
However, Shaib teaches wherein the transformer neural network is included in a representation learning network that generates, based on the subset of the first plurality of outputs, a latent representation of a data sample associated with the first input token generates latent scores of notes (data samples) input for outputs ¶53-54 "generating the latent attention scores. Accordingly, these can be used as explainable outputs for assessing a prediction of an adverse outcome"]
Accordingly, it would have been obvious to a person of ordinary skill in the art before the effective filling date of the claimed invention to modify the attention model disclosed by Lepikhin, Ruiz and Bhat by incorporating the wherein the transformer neural network is included in a representation learning network that generates, based on the subset of the first plurality of outputs, a latent representation of a data sample associated with the first input token disclosed by Shaib because all techniques address the same field of machine learning and by incorporating Shaib into Lepikhin, Ruiz and Bhat provides explainability for more interpretable results [Shaib ¶10]
As to dependent claim 19, the method of claim 18 above is incorporated, Lepikhin, Lepikhin, Ruiz, Bhat, and Shaib further teach wherein performing the task comprises executing a prediction learning network to convert the latent representation of the data sample into a prediction of a class for the data sample. [generates predictions accordingly ¶53-54 predicted classification ¶18]
Claims 9 and 17 are rejected under 35 U.S.C. 103 as being unpatentable over Lepikhin in view of Ruiz and Bhat, as applied in claim 1 and 11 above, and further in view of Zhuo.
As to dependent claim 9, Lepikhin, Ruiz and Bhat teach the method of claim 1 above that is incorporated,
Lepikhin, Ruiz and Bhat do not specifically teach wherein computing the first plurality of outputs comprises: converting the first query and the first key into a plurality of attention scores; and computing the first plurality of outputs based on the plurality of attention scores and the first plurality of values.
However, Zhuo teaches wherein computing the first plurality of outputs comprises: converting the first query and the first key into a plurality of attention scores; and [computes scores based on keys, query and values related to tasks ¶21, ¶40 "The attention block 240 can process the query 215A, the key 225A, and the value 235A to generate attention scores 260A "]
computing the first plurality of outputs based on the plurality of attention scores and the first plurality of values. [generates output token using score ¶40 "generate attention scores 260A and eventually intermediate activation tensor 265A and output token 275A."]
Accordingly, it would have been obvious to a person of ordinary skill in the art before the effective filling date of the claimed invention to modify the attention model disclosed by Lepikhin, Ruiz and Bhat by incorporating the wherein computing the first plurality of outputs comprises: converting the first query and the first key into a plurality of attention scores; and computing the first plurality of outputs based on the plurality of attention scores and the first plurality of values disclosed by Zhuo ford because all techniques address the same field of machine learning and by incorporating Zhuo into Lepikhin, Ruiz and Bhat reduces inefficiencies and duplicated computations [Zhuo ¶2].
As to dependent claim 17, Lepikhin, Ruiz and Bhat teach the method of claim 11 above that is incorporated,
Lepikhin, Ruiz and Bhat do not specifically teach wherein computing the first plurality of outputs comprises: converting the first query and the first key into a plurality of attention scores; and computing the first plurality of outputs based on the plurality of attention scores and the first plurality of values.
However, Zhuo teaches wherein computing the first plurality of outputs comprises: converting the first query and the first key into a plurality of attention scores; and [computes scores based on keys, query and values related to tasks ¶21, ¶40 "The attention block 240 can process the query 215A, the key 225A, and the value 235A to generate attention scores 260A "]
computing the first plurality of outputs based on the plurality of attention scores and the first plurality of values. [generates output token using score ¶40 "generate attention scores 260A and eventually intermediate activation tensor 265A and output token 275A."]
Accordingly, it would have been obvious to a person of ordinary skill in the art before the effective filling date of the claimed invention to modify the attention model disclosed by Lepikhin, Ruiz and Bhat by incorporating the wherein computing the first plurality of outputs comprises: converting the first query and the first key into a plurality of attention scores; and computing the first plurality of outputs based on the plurality of attention scores and the first plurality of values disclosed by Zhuo ford because all techniques address the same field of machine learning and by incorporating Zhuo into Lepikhin, Ruiz and Bhat reduces inefficiencies and duplicated computations [Zhuo ¶2].
Response to Arguments
Applicant's arguments filed 08/21/2026. With respect to the 101 rejections these rejections have been withdrawn.
Applicant's arguments filed 08/21/2026. In the remark, applicant argues that:
(1) Lepikhin and Ruiz fail to teach new language including "performing a task associated with an input corresponding to the first input token based on a subset of the first plurality of outputs that is selected based on the first input token and a task token associated with the task." as recited in amended claim 1.
As to point (1), Applicant’s arguments with respect to claim 1 have been considered but are moot in view of a new ground of rejection as set forth above of Lepikhin in view of Ruiz and Bhat. New art is also on the new IDS filed 08/04/2026.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Applicant is required under 37 C.F.R. § 1.111(c) to consider these references fully when responding to this action.
Gu et al. (US 20230368003 A1) teaches sparce attention using a subset of attended tokens (see ¶4).
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to BEAU SPRATT whose telephone number is (571)272-9919. The examiner can normally be reached M-F 8:30-5 PST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Jennifer Welch can be reached on 5712127212. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/BEAU D SPRATT/ Primary Examiner, Art Unit 2143
