DETAILED ACTION
This communication is in response to the Application filed on 06/28/2024. Claim 1 is pending and has been examined.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Receipt is acknowledged that application claims priority to foreign application with application number VN 1-2023-08941 dated 12/14/2023. Copies of certified papers required by 37 CFR 1.55 have been received. Priority is acknowledged under 35 USC 119(e) and 37 CFR 1.78.
Claim Objections
Claim 1 is objected to because of the following informalities:
Claim 1, line 1-2, should be “a Vietnamese translation model using syntactic information comprising:”
Claim 1, line 4, should be “ Vietnamese language”
Claim 1, line 9, should be “ a sentence detector”
Claim 1, line 11-12, should be “by [[the]] an assumption that if the sentence ended with punctuation charactersa capitalized [[a]] following character and the punctuation character was not a bracket character,”
Claim 1, line 16 should be “[[the]] a next step involves determining which sentences,”
Claim 1, line 19 should be “for a given sentence in [[the]] an original text,”
Claim 1, line 20 should be “translated text with a closest similarity,”
Claim 1, line 22 should be “Step 4: Analysis of syntax of sentences,”
Claim 1, line 23 should be “
Claim 1, line 26-27 should be “are determined by a current word, a dependent word and type of relationship between the current word and the dependent word,”
Claim 1, line 30 should be “performed using [[the]] a LM-LSTM-CRF machine learning model,”
Claim 1, line 33 should be “Step 5: Building a machine translation model incorporating syntactic information,”
Claim 1, line 30 should be “performed using [[the]] a LM-LSTM-CRF machine learning model,”
Claim 1, line 35-38 should be “the encoder dependency parsing information about its dependent word and type of relationship with that dependent word,”
Claim 1, line 40-42 should be “model [[the]] sentence information, the matrices are updated in training process, the Decoder translated sentences, the syntactic information”
Claim 1, line 43 should be “representing [[the]] syntactic roles of words in the sentences”
Claim 1, line 45-46 should be “the machine translation model will learn comprehensive syntactic information from both the input sentences and the output translated sentences in [[the]] a training dataset.”
Appropriate correction is required.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. § 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
Claim 1 is rejected under 35 U.S.C. § 112(b) as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor regards as the invention. Claim 1 recites “a limitation of the sentence was determined by an assumption that if the sentence ended with punctuation characters such as ".", ";", "!", "?", "..." and a following character and was not a bracket character, this is a sign of recognizing an ending position of the sentence” It is unclear whether the following character or one of the exemplary punctuation characters cannot be a bracket character. For the purpose of prior art analysis, Examiner assumes the punctuation character cannot be a bracket character.
Claim 1 further recites “The sentences, before being fed into the model training, undergo syntactic structure analysis using dependency parsing,” It is unclear which of the processed sentences from Step 2 or the similar sentences from Step 3 is referred to by “The sentences.”. For the purpose of prior art analysis, Examiner assumes the processed sentences.
Claim 1 further recites “the Decoder phase learns syntactic information (in step 4) of the output sentences, syntactic information is represented through randomly generated syntactic vectors during model creation, representing the syntactic roles of words in the sentences, such as subjects, predicates, etc.” It is unclear which of the syntactic vectors or the syntactic information represented through the syntactic vectors represents the syntactic roles of words in sentences. For the purpose of prior art analysis, Examiner assumes the syntactic vectors represents syntactic roles.
Claim 1 further recites the limitation “the sentence information.” There is insufficient antecedent basis for this limitation in the claim. For the purpose of prior art analysis, Examiner assumes “the sentence information” is equivalent to the claim limitation of “syntactic information”.
Claim 1 further recites the limitation "the pairs of input and output translated sentences (in the step 4)" in line 34. There is insufficient antecedent basis for this limitation in the claim.
Regarding claim 1, the phrase "such as" renders the claim indefinite because it is unclear whether the limitations following the phrase are part of the claimed invention. See MPEP § 2173.05(d). Claim 1 recites unpatentable exemplary language once in lines 11-12, and again claim 1, lines 43-44.
Further, the claims are generally narrative and indefinite, failing to conform with current U.S. practice. They appear to be a literal translation into English from a foreign document and are replete with grammatical and idiomatic errors.
Proposed Claim Language
Below is an example of claim language that could correct the claim objections and the claim rejections under 35 USC § 112 as described above.
1. A method of constructing a Vietnamese machine translation model using syntactic information comprising:
collecting published articles from reputable news sources containing Vietnamese language and other languages;
segmenting sentences in text of the collected articles, wherein text of the collected articles undergoes a sentence segmentation phase that produces processed sentences from the text which then serve as training data for a machine translation model, wherein a sentence detector based on a linguistic feature identifies a boundary of sentences in a document, wherein a boundary of a sentence is determined by whether the sentence ends with punctuation characters and a capitalized following character, wherein the punctuation character is not a bracket character;
developing an algorithm for aligning sentences across texts after collecting texts that are translations of each other and performing sentence segmentation, wherein the developing comprises:
determining which sentences are translations of each other among pairs of sentences, wherein sentences in each text are passed through a large language model to generate embedding vectors; and
retrieving one or more translated sentences in translated text with a closest similarity using information retrieval tools for a given sentence in the original text, wherein similar sentence pairs are used as training data for the machine translation model;
analyzing a syntax of sentences to determine syntactic information, wherein each sentence, before being fed into the model training, undergoes syntactic structure analysis using dependency parsing, wherein dependency parsing of a sentence provides information about relationships between words in the sentence, the relationships being determined by a current word, a dependent word and type of relationship between the two words, wherein the relationships between two words are determined by first tokenizing sentences by identifying boundaries of words in the sentence, the tokenization performed using a LM-LSTM-CRF machine learning model, and second performing dependency parsing on the tokenized sentences based on a Deep Biaffine Attention model to establish relationships between words in a sentence; and
building a machine translation model incorporating the syntactic information, wherein the pairs of sentences are fed into the machine translation model comprising an architecture consisting of an Encoder and a Decoder, wherein each pair of sentences comprises an input sentence and an output translated sentence, wherein:
the Encoder learns the syntactic information including dependency parsing information using the input sentences, each word in the Encoder accompanied by information about its dependent word and type of relationship with the dependent word the dependent word and type of relationship with the dependent word being represented via word matrices and relationship type matrices that model syntactic information, wherein the matrices are updated in a training process; and
the Decoder learns syntactic information using the output translated sentences, the syntactic information represented through randomly generated syntactic vectors during model creation, the randomly generated syntactic vectors representing syntactic roles of words in the output translated sentences, the syntactic roles comprising subjects or predicates, wherein the syntactic vectors are updated during the training process, wherein the machine translation model learns comprehensive syntactic information from both the input sentences and the output translated sentences in a training dataset.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claim 1 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more. Claim 1 is a method claim, but under Step 2A, claim 1 recites abstract ideas and specifically mental processes. These mental processes are more particularly recited as:
[segmenting] sentences in text of the collected articles…
determining which sentences are translations of each other among text pairs…
[retrieving] one or more sentences in translated text with a closest similarity…
[analyzing] syntax of sentences … using dependency parsing…
[tokenizing sentences] using a LM-LSTM-CRF machine learning model…
dependency parsing through Deep Biaffine Attention model to establish relationships between words in a sentence…
[learning] the dependency parsing information (in the step 4) within the input sentences…
[updating] matrices… in a training process…
[learning] syntactic information (in step 4) of the output translated sentences…
[updating] syntactic vectors … during the training process…
[learning] comprehensive syntactic information from both the input sentences and output translated sentences in a training dataset.
Under Step 2A Prong One, claim 1 is directed to an abstract idea and specifically a mental process. As detailed above, the steps of segmenting, determining, retrieving, analyzing, tokenizing, dependency parsing, updating, learning, etc. may be practically performed in the human mind with the use of a physical aid such as a pen and paper. For example, a human could aggregate bilingual text from various news sources, segment the bilingual text into sentences for each language using sentence boundary detection, and align the bilingual sentences by first converting each sentence into embedding vectors and then computing a similarity value for every permutation of a pair of bilingual sentences. The human could then take an input sentence of a bilingual sentence pair, determine a word matrix that represent words within the input sentence and a dependency matrix that represents word dependencies within the input sentence, reorder words in the input sentence to match the dependency tree of an associated output translated sentence, thus updating the dependency matrix of the input sentence. Finally, the human could determine syntactic roles of words for both the re-ordered input sentences and output translated sentences, and then learn and use those syntactic roles to understand how to translate from the re-ordered input sentence to the output sentence.
Under Step 2A Prong Two, this judicial exception is not integrated into a practical application because claim 1 does not recite additional elements that integrate the exception into a practical application. In particular, claim 1 recites the additional elements of a machine translation model (pg. 1, Section “Background of the Invention”, Paragraph 1), a sentence detector (pg. 4, Paragraph 3), a large language model (pg. 4, Paragraph 5), a LM-LSTM-CRF machine learning model (pg. 5, Paragraph 1), a Deep Biaffine Attention model (pg. 5, Paragraph 1), an encoder (pg. 5, Paragraph 3), and a decoder (pg. 5, Paragraph 3). These additional elements are recited at a high level of generality and merely equate to “apply it” or otherwise merely uses a generic computer as a tool to perform an abstract which are not indicative of integration into a practical application as per MPEP 2106.05(f). Further, claims INDEPENDENTCLAIMS recite the additional elements of “collecting…” which amounts to insignificant extra-solution activities which are not indicative of integration into a practical application as per MPEP 2106.05(g). Accordingly, these additional elements do not integrate the abstract idea into a practical application because they do not impose any meaningful limits on practicing the abstract idea. The claim is directed to an abstract idea.
Under Step 2B, the claims do not recite additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract idea into a practical application, the additional elements of using a computer is noted as a general computer {machine translation model (pg. 1, Section “Background of the Invention”, Paragraph 1); sentence detector (pg. 4, Paragraph 3); large language model (pg. 4, Paragraph 5); LM-LSTM-CRF machine learning model (pg. 5, Paragraph 1); Deep Biaffine Attention model (pg. 5, Paragraph 1); encoder (pg. 5, Paragraph 3); decoder (pg. 5, Paragraph 3)}. Mere instructions to apply an exception using a generic computer component cannot provide an inventive concept. Further, the additional limitations in the claims noted above are directed towards insignificant extra-solution activities. The claims are not patent eligible.
For all of the above reasons, taken alone or in combination, claim 1 recites a non-statutory mental process.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim 1 is rejected under 35 U.S.C. 103 as obvious over “Dependency-to-Dependency Neural Machine Translation” (Wu et al.) in view of “KC4MT: A High-Quality Corpus for Multilingual Machine Translation” (Nguyen et al.) in view of “Sentence Boundary Detection in Adjudicatory Decisions in the United States” (Šavelka et al.) in view of “A Full End-to-End Semantic Role Labeler, Syntax-agnostic Over Syntax-aware” (Cai et al.) in view of “Effectiveness of Character Language Model for Vietnamese Named Entity Recognition” (Doan et al.).
For the sake of clarity, the following claim limitations are presented using the proposed claim language found above with respect to the section, “Proposed Claim Language”.
Claim 1
Regarding claim 1, Wu et al. disclose a method of constructing a Vietnamese machine translation model using syntactic information comprising:
analyzing a syntax of sentences to determine syntactic information, wherein each sentence, before being fed into the model training, undergoes syntactic structure analysis using dependency parsing (Wu et al. pg. 2, Section (II)(B), Paragraph 1, "we use a shift-reduce transition-based dependency parser algorithm to construct target dependency trees"), wherein dependency parsing of a sentence provides information about relationships between words in the sentence (Wu et al. pg. 2, Section (II)(B), Paragraph 2, "During parsing, a specific structure is used to record the dependency relationship between different words."), the relationships being determined by a current word, a dependent word and type of relationship between the two words (Wu et al. pg. 2, Section (II)(B), Paragraph 2, "we adopt the labeled arc-standard algorithm [16] to perform incremental parsing during translating. ... In this algorithm, a stack and a buffer are maintained to store the parsing state, based on which three kinds of transition actions are applied. Let
w
0
and
w
1
be the two topmost words in the stack, and
w
-
be the current new word in a sequence of input. Three transition actions are described as follows: • Shift(SH) : Push
w
-
to the stack. • Left-Reduce(LR(
d
)) : Link
w
0
and
w
1
with dependency label
d
as
w
0
→
d
w
1
, and reduce them to the head
w
0
. • Right-Reduce(RR(
d
)) : Link
w
0
and
w
1
with dependency label
d
as
w
0
←
d
w
1
, and reduce them to the head
w
1
."), wherein the relationships between two words are determined by first tokenizing sentences by identifying boundaries of words in the sentence (Wu et al. pg. 5, Section (VI)(A)(1), Paragraph 4, "For data pre-processing, we segment Chinese sentences with our in-house tool and segment English sentences with the scripts in Moses.7"), [the tokenization performed using a LM-LSTM-CRF machine learning model], and secondly, performing dependency parsing on the tokenized sentences [based on a Deep Biaffine Attention model] to establish relationships between words in a sentence (Wu et al. pg. 5, Section (VI)(A)(1), Paragraph 5, "For English and Chinese, we re-implement two arc-eager dependency parsers as in [24] to generate the parse trees. The parsers are trained on the Penn Treebank and Chinese Treebank corpus."); and
building a machine translation model incorporating the syntactic information (Wu et al. pg. 1, Section (I), Paragraph 8, "in this paper, we propose to use the dependency trees of both sides to improve NMT performance."), wherein the pairs of sentences are fed into the machine translation model comprising an architecture consisting of an Encoder and a Decoder (Wu et al. pg. 2, Section (II)(A), Paragraph 1, "Neural Machine Translation (NMT) is an end-to-end paradigm [5], [1] which directly models the conditional probability
P
(
Y
|
X
)
of target translation
Y
=
y
1
,
y
2
,
…
,
y
n
given source sentence
Y
=
x
1
,
x
2
,
…
,
x
n
. It usually consists of two parts: an encoder and a decoder."), wherein each pair of sentences comprises an input sentence and an output translated sentence (Wu et al. pg. 2, Section (II)(A), Paragraph 1, "Neural Machine Translation (NMT) is an end-to-end paradigm [5], [1] which directly models the conditional probability
P
(
Y
|
X
)
of target translation
Y
=
y
1
,
y
2
,
…
,
y
n
given source sentence
Y
=
x
1
,
x
2
,
…
,
x
n
."), wherein:
the Encoder learns the syntactic information including dependency parsing information using the input sentences (Wu et al. pg. 2, Section (III), Paragraph 1, "In this section, we propose a simple syntax-aware encoder for neural machine translation where source dependency trees are involved."), each word in the Encoder accompanied by information about its dependent word and type of relationship with the dependent word (Wu et al. pg. 2-3, Section (III), Paragraph 1, "Given the dependency tree of a source sentence
X
, two extra sequences are constructed named Child Enriched Sequence (CES) and Head Enriched Sequence (HES). ... Both the two sequences keep the dependency structures. In CES, for each tree node, its head node and its left sibling nodes are used as context. In HES, child nodes are used by their head nodes as context. Specially, the original source sentence
X
is the in-order traversal sequence." Sibling nodes are considered analogous to dependent words. The depedency tree of
X
is considered analogous to information about a source word's dependent word and type relationship), the dependent word and type of relationship with the dependent word being represented via word matrices and relationship type matrices that model the sentence information (Wu et al. pg. 3, Section (III), Paragraph 2, "two extra RNNs are used to encode the CES and the HES in addition to the bi-directional RNNs. Thus for each source word
x
i
, we have four hidden vectors."), wherein the matrices are updated in a training process (Wu et al. pg. 3, Section (III), Paragraph 2, "We denote the two hidden vectors from the bidirectional RNNs as
h
→
i
and
h
←
i
, and denote
h
→
i
l
as the hidden vector from the CES-RNN,
h
→
i
h
as the hidden vector from the HES-RNN. ... an MLP function is applied to the four recurrent states with a smaller hidden size"), and
the Decoder learns syntactic information using the output translated sentences (Wu et al. pg. 3, Section (IV), Paragraph 1-2, "To model
Y
’s dependency tree
T
, instead of directly modeling the tree itself, we predict a parsing action sequence
A
which can map
Y
to
T
. ... We use two recurrent neural networks in the decoder, called Word-RNN and Action-RNN, to jointly model the target word sequence and the transition action sequence."), the syntactic information represented through [randomly] generated syntactic vectors during model creation (Wu et al. pg. 4, Section (IV)(A), Paragraph 1, "We propose to leverage target syntactic context for translation generation. In our model, the syntactic context
K
j
at timestep
j
is defined as a vector which is computed by a feedforward network based on the current parsing configuration of the Action-RNN."), the [randomly] generated syntactic vectors representing syntactic roles of words in the output translated sentences (Wu et al. pg. 4, Section (IV)(A), Paragraph 1, "Denote that
w
0
and
w
1
are the two topmost words in the stack,
w
0
l
and
w
1
l
are their leftmost modifiers in the partial tree,
w
0
r
and
w
1
r
their rightmost modifiers. ... Based on these features, the syntactic context vector
K
j
is computed as [Equation 12]" See Equation 12, which illustrates computing syntactic context
K
j
based on
w
0
,
w
1
, and their respective modifiers. Determining modifiers using a partial dependency tree for a given word is considered analogous to representing syntactic roles of words) [comprising subjects or predicates], wherein the syntactic vectors are updated during the training process (Wu et al. pg. 4-5, Section (IV)(A), Paragraph 1-2, "Based on these features, the syntactic context vector
K
j
is computed as [Equation 12], where
W
k
,
U
k
,
W
b
,
U
b
are the weight matrices,
E
stands for the embedding matrix. ... We add
K
j
to the generation procedure in Eq. 2" See Equation 12, which illustrates computing syntactic context
K
j
based on the weight matrices cited above. Weight matrices are updated during training, thus
K
j
is implied to be updated during training), wherein the machine translation model learns comprehensive syntactic information from both the input sentences and output translated sentences in a training dataset (Wu et al. pg. 5, Section (V), Paragraph 1, "we propose a novel dependency-to-dependency NMT model by replacing the encoder of the SD-NMT model with the SE. Thus the new model is a dependency-to-dependency model where the dependency trees of both [source and target languages] are used.").
Wu et al. does not explicitly disclose collecting news articles or aligning sentences.
However, Nguyen et al. disclose collecting published articles from reputable news sources containing Vietnamese language and other languages (Nguyen et al. pg. 5495, Section 3.1, Paragraph 1, "We crawl bilingual websites in the news domain for Vietnamese-Chinese, Vietnamese-Laos, and Vietnamese-Khmer language pairs.");
segmenting sentences in text of the collected articles, wherein text of the collected articles undergoes a sentence segmentation phase (Nguyen et al. pg. 5495, Section 3, Paragraph 1, "Pages in html type are downloaded from bilingual websites using a crawler tool. These pages are preprocessed to get parallel text pairs. Then we do text alignment, paragraph alignment, and finally, sentence alignment by using tools we designed ourselves to get parallel sentence pairs.") that produces processed sentences from the text which then serve as training data for a machine translation model (Nguyen et al. pg. 5495, Section 3, Paragraph 1, "we review manually these pairs to get high quality parallel corpora."), [wherein a sentence detector based on a linguistic feature identifies a boundary of sentences in a document, a limitation of the sentence determined by whether the sentence ends with punctuation characters and a capitalized following character, wherein the punctuation character is not a bracket character]; and
developing an algorithm for aligning sentences across texts after collecting texts that are translations of each other and performing sentence segmentation (Nguyen et al. pg. 5496, Section 3.2, Paragraph 2, "In this work, we propose a sentence alignment method that overcomes the above-mentioned limitations"), wherein the developing comprises:
determining which sentences are translations of each other among pairs of sentences (Nguyen et al. pg. 5496, Section 3.2, Paragraph 3, "for each aligned paragraph pair, we segment them into sentences, then do sentence alignment based on the similarity of the two sentence embedding vectors and length ratio of the sentence pair." Finding the best alignment for sentence pairs is considered analogous to determining which sentences are translations of each other among sentence pairs), wherein sentences in each text are passed through a large language model to generate embedding vectors (Nguyen et al. pg. 5496, Section 3.2, Paragraph 4, "In this work, we use the publicly available LASER multilingual sentence embedding method (Artetxe and Schwenk, 2018) and model, which is pre-trained on 93 languages."); and
retrieving one or more translated sentences in translated text with a closest similarity using information retrieval tools for a given sentence in the original text (Nguyen et al. pg. 5496, Section 3.2, Paragraph 3, "for each aligned paragraph pair, we segment them into sentences, then do sentence alignment based on the similarity of the two sentence embedding vectors and length ratio of the sentence pair."), wherein similar sentence pairs are used as training data for the machine translation model (Nguyen et al. pg. 5500, Section 4.3, Paragraph 5, "We train systems on our separate bilingual data and the ALT Parallel Corpus for each language pair. ... We train Chinese-Vietnamese, Laos-Vietnamese, and Khmer-Vietnamese models for 20 epochs.").
It would have been obvious to a person having ordinary skill in the art before the time of the effective filing date of the claimed invention of the instant application to modify Wu et al. to incorporate Nguyen et al.’s corpus creation method.
The suggestion/motivation for doing so would have been that, “The multilingual parallel corpus is an important resource for many applications of natural language processing (NLP). For machine translation, the size and quality of the training corpus mainly affects the quality of the translation models.,” as noted by Nguyen et al. in the abstract.
Wu et al. in view of Nguyen et al. do not explicitly disclose all of sentence boundary detection.
However, Šavelka et al. disclose segmenting sentences in text of the collected articles, wherein text [of the collected articles] undergoes a sentence segmentation phase that produces processed sentences from the text (Šavelka et al. pg. 23, Section 2, Paragraph 1, "The goal in sentence boundary detection is to split a natural language text into individual sentences (i.e., identify each sentence’s boundaries).") [which then serve as training data for a machine translation model], wherein a sentence detector based on a linguistic feature (Šavelka et al. pg. 39, Section 6 "Custom SBD Systems", Paragraph 2, "we trained a number of conditional random fields models based on simple low-level textual features.") identifies a boundary of sentences in a document (Šavelka et al. pg. 35, Section 5, Paragraph 3, "We implement an SBD system consisting of a general condition random fields sequence labeling model (CRF; for details see Section 6)."), a limitation of the sentence determined by whether the sentence ends with punctuation characters and a capitalized following character (Šavelka et al. pg. 2, Section 1, Paragraph 1, "In English, the boundary character that starts a sentence is typically an initial capital letter in the first word of the sentence (i.e., the first character within the span of characters constituting a sentence is a capital letter). Punctuation at the end of the sentence is typically the end character of the sentence span." Identifying a sentence's beginning and end is considered analogous to determining whether a sentence ends with punctuation characters (i.e. the end of a sentence) and a following character (i.e. the beginning of a new sentence).), wherein the punctuation character is not a bracket character (Šavelka et al. pg. 29, Section 3, Paragraph 15, "Parentheticals within sentences occur frequently within adjudicatory decisions in the United States, especially within citations. We annotate the parenthetical as within the span of the overall sentence.").
It would have been obvious to a person having ordinary skill in the art before the time of the effective filing date of the claimed invention of the instant application to modify Wu et al. in view of Nguyen et al. to incorporate Šavelka et al.’s explicit sentence boundary detection.
The suggestion/motivation for doing so would have been that, “The most common source of errors is due to wrongly predicted sentence boundaries … This type of error is very serious because it causes broken sentences to be passed along for further processing within the pipeline,” as noted by the Šavelka et al. disclosure in pg. 42, Section 7, Paragraph 2.
Wu et al. in view of Nguyen et al. in view of Šavelka et al. do not explicitly disclose all of performing dependency parsing using a Deep Biaffine Attention model.
However, Cai et al. disclose performing dependency parsing on tokenized sentences based on a Deep Biaffine Attention model (Cai et al. pg. 2756, Section 3.2, Paragraph 1, "Typically, to predict and label arguments for a given predicate, a role classifier is employed on top of the BiLSTM encoder. ... Our model adopts the recently introduced biaffine attention (Dozat and Manning, 2017) to enhance our role scorer. Biaffine attention is a natural extension of bilinear attention (Luong et al., 2015) which is widely used in neural machine translation (NMT).") to establish relationships between words in a sentence (Cai et al. pg. 2754, Section 2, Paragraph 2, "In semantic dependency parsing, we can always identify two types of words, semantic head (predicate) and semantic dependent (argument). To build the needed predicate-argument structure, the model only needs to predict the role of any word pair from the given sentence. ... the predicate disambiguation and argument identification/labeling tasks can be naturally regarded as the labeling process over all the word pairs."); and
[representing] syntactic information … through randomly generated syntactic vectors during model creation, the randomly generated syntactic vectors representing syntactic roles of words in the output translated sentences comprising subjects or predicates (Cai et al. pg. 2756, Section 3.1, Paragraph 1, "The word representation of our model is the concatenation of several vectors: a randomly initialized word embedding
e
(
r
)
, a pre-trained word embedding
e
(
p
)
, a randomly initialized part-of-speech (POS) tag embedding
e
(
p
o
s
)
, a randomly initialized lemma embedding
e
(
l
)
. ... we employ an indicator embedding
e
(
i
)
to indicate whether a word is a predicate when predicting and labeling the arguments for each given predicate. The final word representation is given by
e
=
e
(
r
)
⊕
e
(
p
)
⊕
e
(
l
)
⊕
e
(
p
o
s
)
⊕
e
(
i
)
, where ⊕ is the concatenation operator." The final word representation
e
is a word embedding that is randomly generated during model creation (
e
(
r
)
,
e
(
p
o
s
)
,
e
(
l
)
) and represents syntactic roles comprising subject or predicates (
e
(
i
)
)).
It would have been obvious to a person having ordinary skill in the art before the time of the effective filing date of the claimed invention of the instant application to modify Wu et al. in view of Nguyen et al. in view of Šavelka et al. to incorporate Cai et al.’s biaffine-based dependency parsing.
The suggestion/motivation for doing so would have been that, “We experimentally show that though our biaffine attentional model remains simple and … achieves the best result on the benchmark for both Chinese and English,” as noted by the Cai et al. disclosure in pg. 2754, Section 1, Paragraph 5.
Wu et al. in view of Nguyen et al. in view of Šavelka et al. in view of Cai et al. do not explicitly disclose all of tokenization using an LM-LSTM-CRF model.
However, Doan et al. disclose wherein the relationships between two words are determined by first tokenizing sentences by identifying boundaries of words in the sentence (Doan et al. pg. 159, Section 2.4, Paragraph 2, "Given input
x
=
(
x
1
,
x
2
,
…
,
x
T
)
... Its character-level input is recorded as
c
=
(
c
0
,
_
,
c
1,1
,
c
1,2
…
,
c
1
,
_
,
c
2,1
,
.
.
.
,
c
n
,
_
)
, where
c
i
,
j
is the j-th character for word
w
i
and
c
i
,
_
, is the space character after
c
w
,
i
."), the tokenization performed using a LM-LSTM-CRF machine learning model (Doan et al. pg. 159, Section 2.4, Paragraph 1, "In particular, we integrated Character language model that (Liu et al., 2018) and (Akbik et al., 2018) proposed, with our system (LM LSTM-CRF).").
It would have been obvious to a person having ordinary skill in the art before the time of the effective filing date of the claimed invention of the instant application to modify Wu et al. in view of Nguyen et al. in view of Šavelka et al. in view of Cai et al. to incorporate Doan et al.’s LM-LSTM-CRF because such a modification is the result of simple substitution of one known element for another producing a predictable result. More specifically, Wu et al.’s Moses-based tokenizer and Doan et al.’s LM-LSTM-CRF-based tokenizer perform the same general and predictable function, the predictable function being tokenizing sentences into words. Since each individual element and its function are shown in the prior art, albeit shown in separate references, the difference between the claimed subject matter and the prior art rests not on any individual element or function but in the very combination itself - that is in the substitution of Wu et al.’s Moses-based tokenizer by replacing it with Doan et al.’s LM-LSTM-CRF-based tokenizer. Thus, the simple substitution of one known element for another producing a predictable result renders the claim obvious.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JACOB B VOGT whose telephone number is (571)272-7028. The examiner can normally be reached Monday - Friday 9:30am - 7pm EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Paras D Shah can be reached at (571)270-1650. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JACOB B VOGT/ Examiner, Art Unit 2653
/Paras D Shah/ Supervisory Patent Examiner, Art Unit 2653
03/16/2026