DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
This action is in response to the Application filed on 10/16/2023. Claims 1-20 are pending in the case. All claims are examined and rejected accordingly.
Information Disclosure Statement
As required by MPEP 609 (c), the Applicants’ submission of the Information Disclosure Statement(s) filed on 11/16/2024, 12/12/2026 and 02/13/2026 are acknowledged by the examiner and the cited references have been considered in the examination of the claims now pending.
Priority
The present application claims priority under 35 U.S.C. §119 to Chinese patent Application No. 202110415349.1 filed on April 18, 2011. Applicant’s claim for the benefit of a prior-filed application under 35 U.S.C. 119 and/or 35 U.S.C. 120 is acknowledged.
Claim Rejections - 35 USC § 101
5. 35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
6. Claims 1-19 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis:
According to the first part of the analysis, in the instant case, claims 1-20 fall within one of the four statutory categories (i.e., process, machine, manufacture, or composition of matter).
Step 2A Prong One Analysis: The limitations:
obtaining M first embedding vectors and a second embedding vector, wherein each first embedding vector of the M first embedding vectors indicates a known data unit of M known data units in target data and a first position of the known data unit in the target data, the second embedding vector indicates a second position, in the target data, of a first to-be-predicted data unit in the target data, and M is a positive integer (This step recites mathematical representation of data as embedding vectors and positional vectors which fall within the mental processes and mathematical concepts grouping of abstract ideas.);
processing the M first embedding vectors by using a target encoder, to obtain M first output vectors corresponding to the M known data units, wherein, for each known data unit of the M known data units, a first output vector of the M first output vectors corresponding to each a known data unit is generated based on the M first embedding vectors (This step recites mathematical calculation of vectors through an encoder model and fall within the mental processes and mathematical concepts grouping of abstract ideas.); and
processing the M first output vectors and the second embedding vector by using a target prediction network, to obtain the first to-be-predicted data unit vectors (This step recites mathematical processing of vectors through an encoder model and fall within the mental processes and mathematical concepts grouping of abstract ideas.);
determining, by a first submodel of a plurality of submodels, a first class associated with a first entity of the plurality of entities, the first-class indicative of calculation of a first impact score of the target action (This step involves performing a calculation to produce a class which is understood to be abstract mathematical concept.)
The above limitations in the context of this claim merely used mathematical model to transform one set of data (embedding vectors) to another set of data (predicted date unit) and does not recite a technological process other than mathematical computation. Thus, the claims are patent eligible because they do not recite a judicial exception.
Step 2A Prong Two Analysis: Does the claim recite additional elements? Do those additional elements, individually and in combination, integrate the judicial exception into a practical application?
The claim recites additional element:
A data processing apparatus, comprising: a processor; and a memory coupled to the processor to store instructions, which when executed by the processor, cause the processor to processor is configured to obtain the code and perform the operations … (claim 11)
A computer storage medium, wherein the computer storage medium stores one or more instructions, and when the instructions are executed by one or more computers, the one or more computers are enabled to … (claim 20)
Addental elements such as … embedding vectors, target encoders, target prediction network, predicting data unit mathematical model to transform one set of data (embedding vectors) to another set of data (predicted date unit) and does not recite a technological process other than mathematical computation and the additional limitations fail to integrate the abstract idea into a practical application. This judicial exception is not integrated into a practical application. Additional elements computer storage medium (in claims 20) and processor; and memory storing instructions (11-. These additional elements do not integrate the abstract idea into a practical application because they do not impose any meaningful limits on practicing the abstract idea.
Step 2B Analysis: Do the additional elements, considered individually and in combination, amount to significantly more than the judicial exception?
The claim recites additional element:
A data processing apparatus, comprising: a processor; and a memory coupled to the processor to store instructions, which when executed by the processor, cause the processor to processor is configured to obtain the code and perform the operations … (claim 11)
A computer storage medium, wherein the computer storage medium stores one or more instructions, and when the instructions are executed by one or more computers, the one or more computers are enabled to … (claim 20)
Addental elements such as … embedding vectors, target encoders, target prediction network, predicting data unit mathematical model to transform one set of data (embedding vectors) to another set of data (predicted date unit) and does not recite a technological process other than mathematical computation and the additional elements does not amount to significantly more than the abstract mathematical concept and limitations fail to integrate the abstract idea into a practical application. This judicial exception is not integrated into a practical application. Additional elements computer storage medium (in claims 20) and processor; and memory storing instructions (11) all amount to no more than adding insignificant extra-solution activity/specifications related to data gathering, data input, or data transmittal. These additional elements do not integrate the abstract idea into a practical application because they do not impose any meaningful limits on practicing the abstract idea.
The dependent claims respectively recite a judicial exception in limitations of: “wherein the first position indicates a relative position relationship between the known data unit and another known data unit and a relative position relationship between the known data unit and the first to-be-predicted data unit, and the second position indicates a relative position relationship between the first to-be-predicted data unit and each known data unit of M known data units in the target data.” (claims 2, 12); “wherein the target encoder is a first transformer layer, and the target prediction network is a second transformer layer.” (claims 3, 13 ), “wherein the first transformer layer comprises a plurality of serial transformer sub-layers, and the processing the M first embedding vectors by using the target encoder, to obtain the M first output vectors corresponding to the M known data units comprises: processing, by using the transformer sub-layer, data output by a previous transformer sub- layer adjacent to each transformer sub-layer, to obtain M intermediate vectors; and outputting the M intermediate vectors to a next transformer sub-layer adjacent to the transformer sub-layer, wherein if-in response to the transformer sub-layer is a transformer layer closest to an input side in the plurality of transformer sub-layers, input data of the transformer sub-layer is the M first embedding vectors; or if-in response to the transformer sub-layer is a transformer layer closest to an output side in the plurality of transformer sub-layers, output data of the transformer sub-layer is the M first output vectors.” (claims 4, 14 ), “wherein the target encoder comprises an attention head, and the processing the M first embedding vectors by using the target encoder comprises: obtaining attention information indicating that there is an attention association between any two of the M first embedding vectors when the attention head processes the M first embedding vectors; and processing the M first embedding vectors based on the attention information by using the target encoder.”(claims 5, 15 ), “performing embedding processing on the M known data units in the target data by using an embedding layer, to obtain M third embedding vectors; for each of the M known data units, obtaining a position vector of known data units, wherein the position vector indicates the first position; and integrating each of the M third embedding vectors and a corresponding position vector, to obtain the M first embedding vectors.” (claims 6,16), “wherein the target data further comprises a second to-be-predicted data unit, and a prediction order of the second to-be-predicted data unit and the first to-be-predicted data unit is randomly determined.”(claims 7, 17) “wherein in response to the second to-be-predicted data unit is predicted after the first to-be-predicted data unit, the method further comprising: obtaining a fourth embedding vector and a fifth embedding vector, wherein the fourth embedding vector indicates the first to-be-predicted data unit and the second position of the first to-be-predicted data unit in the target data, and the fifth embedding vector indicates a third position of the second to-be-predicted data unit in the target data; processing the M first embedding vectors and the fourth embedding vector by using the target encoder, to obtain the M known data units and M+1 second output vectors corresponding to the first to-be-predicted data unit; and processing the M+1 second output vectors and the fifth embedding vector by using the target prediction network, to obtain the second to-be-predicted data unit.” (claims 8, 18); “wherein, for each known data unit a second output vector corresponding to each known data unit is generated based on the M first embedding vectors, and the second output vectors corresponding to the first to-be-predicted data unit are generated based on the M first embedding vectors and the fourth embedding vector.” (claims 9, 19), “wherein the target data is text data, the known data unit is a known word in the text data, and the first to-be-predicted data unit is a to-be- predicted word in the text data; the target data is speech data, the known data unit is a known audio sequence in the speech data, and the first to-be-predicted data unit is a to-be-predicted audio sequence in the speech data; or the target data is image data, the known data unit is a known sample in the image data, and the first to-be-predicted data unit is a to-be-predicted sample in the image data.”(claim 10).
These additional limitations (in claims 2-10 and 12-19) also constitute concepts performed in the human mind which fall within the “Mathematical concepts” groupings of abstract ideas. The dependent claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to integration of the abstract idea into a practical application, the additional elements of non-transitory computer readable medium comprising: computer program code are again insignificant extra-solution activity steps that cannot provide an inventive concept. All of these additional elements as generically claimed are considered well-understood, routine, and conventional.
Therefore, these limitations, taken alone or in combination, do not integrate the abstract idea into a practical application or recite significantly more that the abstract idea. Thus, all of the dependent claims are also not patent eligible
7. Claim 20 is rejected under 35 U.S.C. 101 because the claimed is directed to non-statutory subject matter.
Specifically Claim 20 recites:
“A computer storage medium, wherein the computer storage medium stores one or more instructions, and when the instructions are executed by one or more computers, the one or more computers are enabled to: ...”
Claim 20 is rejected under 35 U.S.C. 101 because claim 20 is directed to a “computer readable storage medium” that is transitory medium since the specification in paragraph 308 states that “..the storage medium 2430 may be transitory storage…”. As such, in a broadest reasonable interpretation, the claimed medium can include signal per se which is non-statutory. Examiner recommends that the claims be amended to “non-transitory computer readable storage medium” in order to overcome these 101 rejections.
Appropriate correction is required.
Examiner Comments
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-20 are rejected under 35 U.S.C. 103 as being unpatentable over Shazeer (Pub. No. US 20210019624 B1, Pub. Date 2021-01-21) in view of Yang (NPL: Title: XLNet: Generalized Autoregressive Pretraining for Language Understanding, 2019).
Regarding independent Claim 1,
Shazeer teaches a data processing method (see Abstract: describing an encoder neural network configured to receive the input sequence and generate a respective subnetwork output), comprising:
obtaining M first embedding vectors and a second embedding vector (see Shazeer: Fig.1, [0030], “The embedding layer 120 is configured to, for each network input in the input sequence (M first embedding vectors), map the network input to a numeric representation of the network input (a second embedding vector) in an embedding space, e.g., into a vector in the embedding space. The embedding layer 120 then provides the numeric representations of the network inputs to the first subnetwork in the sequence of encoder subnetworks 130, i.e., to the first encoder subnetwork 130 of the N encoder subnetworks 130”), each first embedding vector of the M first embedding vectors indicates a known data unit of M known data units in target data and a first position of the known data unit in the target data (see Shazeer: Fig.1, [0031], “the embedding layer 120 is configured to map each network input to an embedded representation of the network input (known data unit of M known data units in target data) and then combine, e.g., sum or average, the embedded representation of the network input with a positional embedding of the input position of the network input in the input order (a first position of the known data unit) to generate a combined embedded representation of the network input. That is, each position in the input sequence has a corresponding embedding and for each network input the embedding layer 120 combines the embedded representation of the network input with the embedding of the network input's position in the input sequence. Such positional embeddings can enable the model to make full use of the order of the input sequence without relying on recurrence or convolutions.”), […], and M is a positive integer (see Shazeer: Fig.1, [0030], “The embedding layer 120 is configured to, for each network input in the input sequence, map the network input to a numeric representation of the network input in an embedding space.”)
processing the M first embedding vectors by using a target encoder, to obtain M first output vectors corresponding to the M known data units (see Shazeer: Fig.1, [0026], “The encoder neural network 110 is configured to receive the input sequence 102 and generate a respective encoded representation of each of the network inputs in the input sequence. Generally, an encoded representation is a vector or other ordered collection of numeric values.”), wherein, for each known data unit of the M known data units, a first output vector of the M first output vectors corresponding to a known data unit is generated based on the M first embedding vectors (see Shazeer: Fig.2, [0037], “Each encoder subnetwork 130 includes an encoder self-attention sub-layer 132. The encoder self-attention sub-layer 132 is configured to receive the subnetwork input for each of the plurality of input positions and, for each particular input position in the input order, apply an attention mechanism over the encoder subnetwork inputs at the input positions using one or more queries derived from the encoder subnetwork input at the particular input position to generate a respective output for the particular input position. In some cases, the attention mechanism is a multi-head attention mechanism. The attention mechanism and how the attention mechanism is applied by the encoder self-attention sub-layer 132.”);
Shazeer does not teach the method wherein:
the second embedding vector indicates a second position, in the target data, of a first to-be-predicted data unit in the target data;
processing the M first output vectors and the second embedding vector by using a target prediction network, to obtain the first to-be-predicted data unit.
However, Yang teaches the method wherein:
the second embedding vector indicates a second position, in the target data, of a first to-be-predicted data unit in the target data (see Yang: Fig.1, Pg.4, Section 2.3 explains that the target position aware representation for predicting a data unit at a selected position.)
PNG
media_image1.png
262
674
media_image1.png
Greyscale
processing the M first output vectors and the second embedding vector by using a target prediction network, to obtain the first to-be-predicted data unit data (see Yang: Fig.1, Pg.4, Section 2.3.
PNG
media_image2.png
232
636
media_image2.png
Greyscale
Because both Shazeer and Yang are in the same/similar field of endeavor of neural language modeling and sequence prediction, accordingly, it would have been obvious to a person of ordinary skill in the art, before the effective filing date of the invention, to modify the teaching of Shazeer to include the system that indicate a second position, in the target data, of a first to-be-predicted data unit in the target data and using a target prediction network, to obtain the first to-be-predicted data unit and use target prediction network, to obtain the first to-be-predicted data unit as taught by Yang. After modification of Shazeer, the transformer encoder-decoder architecture can also to incorporate target-position prediction technique as taught by Yang. One would have been motivated to make such a combination in order to improve sequence prediction process by providing efficient and accurate positional information of input data to process question answering, natural language inference, sentiment analysis, and document ranking. (see Yang; Abstract)
Regarding Claim 2,
As shown above, Shazeer and Yang and teaches all the limitations of claim 1. Shazeer further teaches the method wherein:
the first position indicates a relative position relationship between the known data unit and another known data unit and a relative position relationship between the known data unit and the first to-be-predicted data unit (see Shazeer: Fig.1, [0023], “The input sequence 102 has a respective network input at each of multiple input positions in an input order and the output sequence 152 has a respective network output at each of multiple output positions in an output order. That is, the input sequence 102 has multiple inputs arranged according to an input order and the output sequence 152 has multiple outputs arranged according to an output order.”), and the second position indicates a relative position relationship between the first to-be-predicted data unit and each known data unit of M known data units in the target data (see Shazeer: Fig.1, [0031], “each position in the input sequence has a corresponding embedding and for each network input the embedding layer 120 combines the embedded representation of the network input with the embedding of the network input's position in the input sequence. Such positional embeddings can enable the model to make full use of the order of the input sequence without relying on recurrence or convolutions.”)
Regarding Claim 3,
As shown above, Shazeer and Yang and teaches all the limitations of claim 2. Shazeer further teaches the method wherein:
the target encoder is a first transformer layer, and the target prediction network is a second transformer layer (see Shazeer: Fig.1, [0041], “Once the encoder neural network 110 has generated the encoded representations, the decoder neural network 150 is configured to generate the output sequence in an auto-regressive manner.”)
Regarding Claim 4,
As shown above, Shazeer and Yang and teaches all the limitations of claim 3. Shazeer further teaches the method wherein:
the first transformer layer comprises a plurality of serial transformer sub-layers, and the processing the M first embedding vectors by using the target encoder, to obtain the M first output vectors corresponding to the M known data units (see Shazeer: Fig.1, [0031], “the embedding layer 120 is configured to map each network input to an embedded representation of the network input and then combine, e.g., sum or average, the embedded representation of the network input with a positional embedding of the input position of the network input in the input order to generate a combined embedded representation of the network input. That is, each position in the input sequence has a corresponding embedding and for each network input the embedding layer 120 combines the embedded representation of the network input with the embedding of the network input's position in the input sequence. Such positional embeddings can enable the model to make full use of the order of the input sequence without relying on recurrence or convolutions.”), comprises:
processing, by using the transformer sub-layer, data output by a previous transformer sub- layer adjacent to each transformer sub-layer, by using the transformer sub-layer, to obtain M intermediate vectors (see Shazeer: Fig.1, [0042], “the decoder neural network 150 generates the output sequence, by at each of a plurality of generation time steps, generating a network output for a corresponding output position conditioned on (i) the encoded representations and (ii) network outputs at output positions preceding the output position in the output order.”; and
outputting the M intermediate vectors to a next transformer sub-layer adjacent to the transformer sub-layer, wherein if-in response to the transformer sub-layer is a transformer layer closest to an input side in the plurality of transformer sub-layers, input data of the transformer sub-layer is the M first embedding vectors (see Shazeer: Fig.1, [0046], “The embedding layer 160 is configured to, at each generation time step, for each network output at an output position that precedes the current output position in the output order, map the network output to a numeric representation of the network output in the embedding space. The embedding layer 160 then provides the numeric representations of the network outputs to the first subnetwork 170 in the sequence of decoder subnetworks, i.e., to the first decoder subnetwork 170 of the N decoder subnetworks.”); or
in response to the transformer sub-layer is a transformer layer closest to an output side in the plurality of transformer sub-layers, output data of the transformer sub-layer is the M first output vectors (see Shazeer: Fig.1, [0047], “the embedding layer 160 is configured to map each network output to an embedded representation of the network output and combine the embedded representation of the network output with a positional embedding of the output position of the network output in the output order to generate a combined embedded representation of the network output. The combined embedded representation is then used as the numeric representation of the network output. The embedding layer 160 generates the combined embedded representation in the same manner as described above with reference to the embedding layer 120”)
Regarding Claim 5,
As shown above, Shazeer and Yang and teaches all the limitations of claim 1. Shazeer further teaches the method wherein:
the target encoder comprises an attention head, and the processing the M first embedding vectors by using the target encoder (see Shazeer: Fig.1, [0059], “an attention mechanism maps a query and a set of key-value pairs to an output, where the query, keys, and values are all vectors. The output is computed as a weighted sum of the values, where the weight assigned to each value is computed by a compatibility function of the query with the corresponding key.”), comprises:
obtaining attention information indicating that there is an attention association between any two of the M first embedding vectors when the attention head processes the M first embedding vectors (see Shazeer: Fig.1, [0050], “Each decoder self-attention sub-layer 172 is configured to, at each generation time step, receive an input for each output position preceding the corresponding output position and, for each of the particular output positions, apply an attention mechanism over the inputs at the output positions preceding the corresponding position using one or more queries derived from the input at the particular output position to generate a updated representation for the particular output position.”); and
processing the M first embedding vectors based on the attention information by using the target encoder (see Shazeer: Fig.1, [0051], “the encoder-decoder attention sub-layer 174 applies attention over encoded representations while the encoder self-attention sub-layer 172 applies attention over inputs at output positions.”)
Regarding Claim 6,
As shown above, Shazeer and Yang and teaches all the limitations of claim 1. Shazeer further teaches the method wherein:
performing embedding processing on the M known data units in the target data by using an embedding layer, to obtain M third embedding vectors (see Shazeer: Fig.1, [0030], “The embedding layer 120 is configured to, for each network input in the input sequence, map the network input to a numeric representation of the network input in an embedding space, e.g., into a vector in the embedding space. The embedding layer 120 then provides the numeric representations of the network inputs to the first subnetwork in the sequence of encoder subnetworks 130, i.e., to the first encoder subnetwork 130 of the N encoder subnetworks 130.”;
for each of the M known data units, obtaining a position vector of each of the Ma known data units, wherein the position vector indicates the first position (see Shazeer: Fig.1, [0023], “The input sequence 102 has a respective network input at each of multiple input positions in an input order and the output sequence 152 has a respective network output at each of multiple output positions in an output order. That is, the input sequence 102 has multiple inputs arranged according to an input order and the output sequence 152 has multiple outputs arranged according to an output order.”; and
integrating each of the M third embedding vectors and a corresponding position vector, to obtain the M first embedding vectors (see Shazeer: Fig.1, [0042], “e decoder neural network 150 generates the output sequence, by at each of a plurality of generation time steps, generating a network output for a corresponding output position conditioned on (i) the encoded representations and (ii) network outputs at output positions preceding the output position in the output order.”)
Regarding Claim 7,
As shown above, Shazeer and Yang and teaches all the limitations of claim 1. Shazeer further teaches the method wherein:
the target data further comprises a second to-be-predicted data unit, and a prediction order of the second to-be-predicted data unit and the first to-be-predicted data unit is randomly determined (see Shazeer: Fig.1, [0045], “The decoder neural network 150 includes an embedding layer 160, a sequence of decoder subnetworks 170, a linear layer 180, and a soft-max layer 190. In particular, as shown in FIG. 1, the decoder neural network includes N decoder subnetworks 170. However, while the example of FIG. 1 shows the encoder 110 and the decoder 150 including the same number of subnetworks, in some cases the encoder 110 and the decoder 150 include different numbers of subnetworks. That is, the decoder 150 can include more or fewer subnetworks than the encoder 110.”)
Regarding Claim 8,
As shown above, Shazeer and Yang and teaches all the limitations of claim 7. Shazeer further teaches the method wherein:
in response to the second to-be-predicted data unit is predicted after the first to-be-predicted data unit (see Shazeer: Fig.1, [0047], “the embedding layer 160 is configured to map each network output to an embedded representation of the network output and combine the embedded representation of the network output with a positional embedding of the output position of the network output in the output order to generate a combined embedded representation of the network output. The combined embedded representation is then used as the numeric representation of the network output. The embedding layer 160 generates the combined embedded representation in the same manner as described above with reference to the embedding layer 120”)” the method further comprising:
obtaining a fourth embedding vector and a fifth embedding vector, wherein the fourth embedding vector indicates the first to-be-predicted data unit and the second position of the first to-be-predicted data unit in the target data (see Shazeer: Fig.1, [0030], “The embedding layer 120 is configured to, for each network input in the input sequence, map the network input to a numeric representation of the network input in an embedding space, e.g., into a vector in the embedding space. The embedding layer 120 then provides the numeric representations of the network inputs to the first subnetwork in the sequence of encoder subnetworks 130, i.e., to the first encoder subnetwork 130 of the N encoder subnetworks 130.”), and
the fifth embedding vector indicates a third position of the second to-be-predicted data unit in the target data (see Shazeer: Fig.1, [0032], “the positional embeddings are learned. As used in this specification, the term “learned” means that an operation or a value has been adjusted during the training of the sequence transduction neural network 108. Training the sequence transduction neural network 108 is described below with reference to FIG. 3. In some other cases, the positional embeddings are fixed and are different for each position.”);
processing the M first embedding vectors and the fourth embedding vector by using the target encoder, to obtain the M known data units and M+1 second output vectors corresponding to the first to-be-predicted data unit (see Shazeer: Fig.1, [0038], “each of the encoder subnetworks 130 also includes a residual connection layer that combines the outputs of the encoder self-attention sub-layer with the inputs to the encoder self-attention sub-layer to generate an encoder self-attention residual output and a layer normalization layer that applies layer normalization to the encoder self-attention residual output. These two layers are collectively referred to as an “Add & Norm” operation in FIG. 1.”); and
processing the M+1 second output vectors and the fifth embedding vector by using the target prediction network, to obtain the second to-be-predicted data unit (see Shazeer: Fig.1, [0042], “e decoder neural network 150 generates the output sequence, by at each of a plurality of generation time steps, generating a network output for a corresponding output position conditioned on (i) the encoded representations and (ii) network outputs at output positions preceding the output position in the output order.”)
Regarding Claim 9,
As shown above, Shazeer and Yang and teaches all the limitations of claim 7. Shazeer further teaches the method wherein:
for each known data unit a second output vector corresponding to each a known data unit is generated based on the M first embedding vectors (see Shazeer: Fig.1, [0037], “Each encoder subnetwork 130 includes an encoder self-attention sub-layer 132. The encoder self-attention sub-layer 132 is configured to receive the subnetwork input for each of the plurality of input positions and, for each particular input position in the input order, apply an attention mechanism over the encoder subnetwork inputs at the input positions using one or more queries derived from the encoder subnetwork input at the particular input position to generate a respective output for the particular input position.”), and
the second output vectors corresponding to the first to-be-predicted data unit are generated based on the M first embedding vectors and the fourth embedding vector (see Shazeer: Fig.1, [0041], “Once the encoder neural network 110 has generated the encoded representations, the decoder neural network 150 is configured to generate the output sequence in an auto-regressive manner.”)
Regarding Claim 10,
As shown above, Shazeer and Yang and teaches all the limitations of claim 3. Shazeer further teaches the method wherein:
the target data is text data, the known data unit is a known word in the text data, and the first to-be-predicted data unit is a to-be- predicted word in the text data (see Shazeer: Fig.1, [0019], “the system may be part of an image processing system. For example, the input sequence can be an image, i.e., a sequence of color values from the image, and the output can be a sequence of text that describes the image. As another example, the input sequence can be a sequence of text or a different context and the output sequence can be an image that describes the context.”);
the target data is speech data, the known data unit is a known audio sequence in the speech data (see Shazeer: Fig.1, [0016], “the system may be a speech recognition system. That is, if the input sequence is a sequence of audio data representing a spoken utterance, the output sequence may be a sequence of graphemes, characters, or words that represents the utterance, i.e., is a transcription of the input sequence.”), and
the first to-be-predicted data unit is a to-be-predicted audio sequence in the speech data (see Shazeer: Fig.1, [0019], “(see Shazeer: Fig.1, [0016], “the system may be a speech recognition system. That is, if the input sequence is a sequence of audio data representing a spoken utterance, the output sequence may be a sequence of graphemes, characters, or words that represents the utterance, i.e., is a transcription of the input sequence.”), or
the target data is image data, the known data unit is a known sample in the image data, and the first to-be-predicted data unit is a to-be-predicted sample in the image data (see Shazeer: Fig.1, [0019], “the system may be part of an image processing system. For example, the input sequence can be an image, i.e., a sequence of color values from the image, and the output can be a sequence of text that describes the image. As another example, the input sequence can be a sequence of text or a different context and the output sequence can be an image that describes the context.”)
Regarding independent claim 11 and 20,
Claim 11 is directed to an apparatus claim and Claim 20 is directed to a computer storage medium claim and have similar/same claim limitation as Claim 1 and are rejected under same rationale.
Regarding Claim 12-19,
Claims 12-19 are directed to an apparatus claim and have similar/same claim limitation as Claims 2-9 respectively and are rejected under same rationale.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
PGPUB
NUMBER:
INVENTOR-INFORMATION:
TITLE / DESCRIPTION
NPL: Title: BERT: Pre-training of Deep Bidirectional Transformers for
Language Understanding
Jacob Devlin
Title: BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Description: new language represent
model called BERT, which stands for
Bidirectional Encoder Representations from
Transformers.
US 20220092266 A1
CHOI; Inkyu
Title: METHOD AND DEVICE WITH NATURAL LANGUAGE PROCESSING
Description: The following description relates to a method and device with natural language processing.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ZELALEM W SHALU whose telephone number is (571)272-3003. The examiner can normally be reached M- F 0800am- 0500pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Cesar Paula can be reached at (571) 272-4128. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Zelalem Shalu/Examiner, Art Unit 2145
/CESAR B PAULA/Supervisory Patent Examiner, Art Unit 2145
Prosecution Insights