DETAILED ACTION
Introduction
This office action is in response to Applicant’s submission filed on 8/8/2023.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Objections
Claims 1 and claims that from which it depends from (claims 2-14) is objected to because of the following informalities: in line 4, there should be “embedding” in front of the word matrix.
Additionally,
For Claim 2, line 5, the word “embedding” should be placed in front of the word “matrix”.
For Claim 3, line 7, the word “embedding” should be placed in front of the word “matrix”.
For Claim 5, line 6, the word “embedding” should be placed in front of the word “matrix”.
For Claim 6, line 8, the word “embedding” should be placed in front of the word “matrix”.
For Claims 15 and claims that from which it depends from (claims 16-17) is objected to because of the following informalities: in line 5, the word “embedding” should be placed in front of the word “matrix”.
Appropriate correction is required.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
`Claims 1-20 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 1 recites the limitation “the feature” in line 5. There is insufficient antecedent basis for this limitation in the claim. Claims 2-14 depends on 1 and therefore is rejected for same rationale.
Claim 15 recites the limitation “the feature” in line 6. There is insufficient antecedent basis for this limitation in the claim. Claims 16-17 depends on 15 and therefore is rejected for same rationale.
Claim 18 recites the limitation “the feature” in line 4. There is insufficient antecedent basis for this limitation in the claim. Claims 19-20 depends on 18 and therefore is rejected for same rationale.
Statement Re: Compliance with 35 USC 101 (Abstract Idea)
The following is a statement describing how the present claims are in compliance with 35 USC 101. This discussion steps through the decision making set forth in MPEP 2106.
The present independent Claims 1, 15 and 18 recite method for named entity disambiguation using capsule networks.
With regard to Step 1 of the subject matter eligibility test of MPEP 2106, Claim 1, 15 and 18 are method or process claims, so the claims are within a statutory category.
With regard to Step 2A, both Claims 1, 15 and 18 are not considered an abstract idea, the claims are clearly beyond a mental process, although embedding matrix/vector and/or embeddings may involve or rely upon mathematical concepts, the limitation does not set forth or describe any mathematical relationships, calculations, formulas, or equations using words or mathematical symbols, and further, it is not considered to be a human gathering activity. Therefore, the claims are determined to be patent eligible and do not require further eligibility analysis. Accordingly, claims 1, 15 and 18 (and Claims 2-14, 16-17 and 19-20 dependents therefrom) are in compliance with 35 USC 101.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claims 18-20 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Jiang, L., Altenbek, G., Wu, D., Ma, Y., & Aierzhati, H. (2020). Chinese short text entity disambiguation based on the dual-channel hybrid network. IEEE Access, 8, 206164-206173.
Jiang discloses: 18. A computer-implemented method for named entity disambiguation, comprising: (pg.2, sect I. This paper proposes a DHCN model, which is a combination of a pooling CNN and capsule network with attention. The semantic information generated by the two models is fused in parallel to complete the disambiguation. To the best of our knowledge, this is the first time that capsules have been used in Chinese entity disambiguation tasks.)
receiving, into a neural capsule embedding network as input, an embedding vector, wherein the embedding vector contains embeddings representing words in a natural language text; (pg. 3, sect III, Given a sentence x = (x1, x2, x3,. . . , xn), it is vectorized and represented as the input of the model. … First, the BERT model is used to extract the features
of the pre-processed data, and the obtained semantic vector representation is used as the input of the CNN model and capsule network.) Also see fig. 1.
analyzing, by the neural capsule embedding network, the features of each word in context of the embedding vector considering tokens to the left and right of the word using at least one layer, each layer consisting of at least one set of filters; (pg. 5, sect III.C The core idea is that not all words in a sentence have the same contribution to NLP tasks. In this paper, the attention mechanism is used to further calculate the extracted semantic features, pg. 2, sect I e. Then, the text of the mention and the candidate entity set are spliced one by one to expand the text length so that it has sufficient context information. Pg. 5, sect III.B The CNN is generally composed of three parts, namely, the convolutional layer, pool layer, and fully connected layer. The function of the convolutional layer is to extract features. Local word order information from the input long text sequence is used to complete the extracted features. In this paper, windows with convolution kernel sizes of 3, 4, and 5 are used for convolution to obtain local features.) Also see fig. 1. [In the context of CNN convolution layers, kernels act as feature detectors and are essentially the same as filters, attention mechanism also means all the tokens to the left and right of the word and their associated weights have been accounted for contextually.]
through dynamic routing of capsules, by the neural capsule embedding network, converging to a final capsule layer mapping to each word in the input vector; (pg. 5, sect III.C, The model is composed of a convolutional layer and a dynamic routing layer. The core of
the dynamic routing layer is the dynamic routing algorithm, which uses a vector representation of capsule nodes instead of the scalar value representation of neuron nodes used in traditional neural networks to replace the pooling algorithm, thus extracting more abundant location space information instead of the scalar generated by the pooling operation. To capture the semantic information of Chinese short texts more fully, this paper completes the learning of semantic dependency via a two-layer capsule network.)
generating, by the neural capsule embedding network, an output vector, (pg. 5, sect III.C, Formula (11) calculates the output vector of the capsule network obtained after linear transformation according to the coupling coefficients and capsule output of the previous layer, formula (12) is a squashing operation that changes sr into a vector of length 1 without changing its direction nr, and Formula (13) is used to update the coupling coefficient. The final target vector is obtained through the above four formulas.) Also see fig. 1.
wherein each output vector value:
a) identifies if a word in the input is a named entity or not a named entity; (pg. 2, sect I, A capsule network is used to capture the semantic features generated by BERT, and an attention mechanism is used to extract the important information of capsule network features. The abovementioned learned semantic knowledge is combined, and the classification is completed by the fully connected layer. Pg. 5, sect III.D Finally, the fused semantic features are input into the fully connected layer with a sigmoid activation function for classification, as shown in formulas)
b) if the word is a named entity, identifies a unique ID number of the entity. (pg. 2, sect I, First, the candidate entity set is obtained from the external knowledge base. Then, the text of the mention and the candidate entity set are spliced one by one to expand the text length so that it has sufficient context information. The unique identification of the entity is used to make the classification label at the same time. Last, the text is input into the BERT model for feature extraction, and the trained features are taken as the input of the CNN model with max-pooling and mean-pooling to capture semantic features. A capsule network is used to capture the semantic features generated by BERT, and an attention mechanism is used to extract the important information of capsule network features. The abovementioned learned semantic knowledge is combined, and the classification is completed by the fully connected layer.)
Regarding Claim 19, Jiang discloses: 19. The method of Claim 18 further comprising: before receiving, into a neural capsule embedding network, an embedding vector as input: a) receiving, as input, a natural language text; (pg. 3, sect III, Given a sentence x = (x1, x2, x3,. . . , xn), it is vectorized and represented as the input of the model.)
b) converting words in the natural language text into embeddings and inserting embeddings into an embedding vector. (pg. 3, sect III, Given a sentence x = (x1, x2, x3,. . . , xn), it is vectorized and represented as the input of the model. Pg.3, sect III.A this paper splices the entity text sequence to be disambiguated with the corresponding entity text sequence in the external knowledge base to create the input of the BERT model. The sequence is as follows:
s = (x1, x2, . . . , xn, c1, c2, . . . , cm). (1)
where x1 xn is the entity text to be disambiguated, c1 cm is the candidate entity text, and s is the spliced text.) Also see fig. 1.
Regarding Claim 20, Jiang discloses: 20. The method of Claim 18 further comprising: before receiving, into a neural capsule embedding network, an embedding vector as input: a) receiving as input a natural language text; (pg. 3, sect III, Given a sentence x = (x1, x2, x3,. . . , xn), it is vectorized and represented as the input of the model.)
b) pre-processing the natural language text to identify features of the natural language text; (pg. 3, sect III, First, the BERT model is used to extract the features
of the pre-processed data, and the obtained semantic vector representation is used as the input of the CNN model and capsule network. The features extracted from the model are fused, and the fused information is classified by the fully connected layer.)
c) converting words in the natural language text into embeddings to include in an embedding vector. (pg. 3, sect III, Given a sentence x = (x1, x2, x3,. . . , xn), it is vectorized and represented as the input of the model. Pg.3, sect III.A this paper splices the entity text sequence to be disambiguated with the corresponding entity text sequence in the external knowledge base to create the input of the BERT model. The sequence is as follows:
s = (x1, x2, . . . , xn, c1, c2, . . . , cm). (1)
where x1 xn is the entity text to be disambiguated, c1 cm is the candidate entity text, and s is the spliced text.) Also see fig. 1.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or non-obviousness.
Claims 1-6, 10, and 13-17 are rejected under 35 U.S.C. 103 as being unpatentable over Jiang in view of Lin, Z., Feng, M., Santos, C. N. D., Yu, M., Xiang, B., Zhou, B., & Bengio, Y. (2017). A structured self-attentive sentence embedding. arXiv preprint arXiv:1703.03130.
Regarding Claim 1, Jiang discloses: 1. A computer-implemented method for named entity disambiguation, comprising: (pg.2, sect I. This paper proposes a DHCN model, which is a combination of a pooling CNN and capsule network with attention. The semantic information generated by the two models is fused in parallel to complete the disambiguation. To the best of our knowledge, this is the first time that capsules have been used in Chinese entity disambiguation tasks.)
receiving, into a neural capsule embedding network as input, (pg. 3, sect III, Given a sentence x = (x1, x2, x3,. . . , xn), it is vectorized and represented as the input of the model. … First, the BERT model is used to extract the features of the pre-processed data, and the obtained semantic vector representation is used as the input of the CNN model and capsule network.) Also see fig. 1.
analyzing, by the neural capsule embedding network, the features of each word in context (pg. 5, sect III.C The core idea is that not all words in a sentence have the same contribution to NLP tasks. In this paper, the attention mechanism is used to further calculate the extracted semantic features, pg. 2, sect I e. Then, the text of the mention and the candidate entity set are spliced one by one to expand the text length so that it has sufficient context information. Pg. 5, sect III.B The CNN is generally composed of three parts, namely, the convolutional layer, pool layer, and fully connected layer. The function of the convolutional layer is to extract features. Local word order information from the input long text sequence is used to complete the extracted features. In this paper, windows with convolution kernel sizes of 3, 4, and 5 are used for convolution to obtain local features.) Also see fig. 1. [In the context of CNN convolution layers, kernels act as feature detectors and are essentially the same as filters, attention mechanism also means all the tokens to the left and right of the word and their associated weights have been accounted for contextually.]
through dynamic routing of capsules, by the neural capsule embedding network, converging to a final capsule layer mapping to each word in the input matrix; (pg. 5, sect III.C, The model is composed of a convolutional layer and a dynamic routing layer. The core of
the dynamic routing layer is the dynamic routing algorithm, which uses a vector representation of capsule nodes instead of the scalar value representation of neuron nodes used in traditional neural networks to replace the pooling algorithm, thus extracting more abundant location space information instead of the scalar generated by the pooling operation. To capture the semantic information of Chinese short texts more fully, this paper completes the learning of semantic dependency via a two-layer capsule network.)
generating, by the neural capsule embedding network, an output matrix, (pg. 5, sect III.C, Formula (11) calculates the output vector of the capsule network obtained after linear transformation according to the coupling coefficients and capsule output of the previous layer, formula (12) is a squashing operation that changes sr into a vector of length 1 without changing its direction nr, and Formula (13) is used to update the coupling coefficient. The final target vector is obtained through the above four formulas.) Also see fig. 1.
wherein each output matrix value:
a) identifies if a word in the input is a named entity or not a named entity; (pg. 2, sect I, A capsule network is used to capture the semantic features generated by BERT, and an attention mechanism is used to extract the important information of capsule network features. The abovementioned learned semantic knowledge is combined, and the classification is completed by the fully connected layer. Pg. 5, sect III.D Finally, the fused semantic features are input into the fully connected layer with a sigmoid activation function for classification, as shown in formulas)
b) if the word is a named entity, identifies a unique ID number of the entity. (pg. 2, sect I, First, the candidate entity set is obtained from the external knowledge base. Then, the text of the mention and the candidate entity set are spliced one by one to expand the text length so that it has sufficient context information. The unique identification of the entity is used to make the classification label at the same time. Last, the text is input into the BERT model for feature extraction, and the trained features are taken as the input of the CNN model with max-pooling and mean-pooling to capture semantic features. A capsule network is used to capture the semantic features generated by BERT, and an attention mechanism is used to extract the important information of capsule network features. The abovementioned learned semantic knowledge is combined, and the classification is completed by the fully connected layer.)
However, Jiang does not explicitly disclose: an embedding matrix, wherein the embedding matrix contains embeddings and each row in the matrix is an embedding sentence;
Lin (in a related field of natural language processing) discloses: an embedding matrix, wherein the embedding matrix contains embeddings and each row in the matrix is an embedding sentence; (pg. 3, sect 2.1, Suppose we have a sentence, which has n tokens, represented in a sequence of word embeddings.
S = (w1, w2, · · · wn) (1)
Here wi is a vector standing for a d dimentional word embedding for the i-th word in the sentence. S is thus a sequence represented as a 2-D matrix, which concatenates all the word embeddings together. S should have the shape n-by-d. Abstract, This paper proposes a new model for extracting an interpretable sentence embedding by introducing self-attention. Instead of using a vector, we use a 2-D matrix to represent the embedding, with each row of the matrix attending on a different part of the sentence.)
Jiang and Lin are considered analogous art. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Jiang to combine with the teaching of Lin to disclose the above feature, because the technique described by Lin improve the named entity disambiguation as having multiple rows in sentence embedding is expected to provide more abundant information about the encoded content (Lin, [Abstract and sect 2.1, sect 4.4.1]).
Regarding Claim 2, Jiang in view of Lin discloses: discloses all the limitation of Claim 1. (see detail element mapping above).
Jiang further discloses: a) receiving, as input, a natural language text; (pg. 3, sect III, Given a sentence x = (x1, x2, x3,. . . , xn), it is vectorized and represented as the input of the model.)
b) converting words in the natural language text into embeddings (pg. 3, sect III, Given a sentence x = (x1, x2, x3,. . . , xn), it is vectorized and represented as the input of the model. Pg.3, sect III.A this paper splices the entity text sequence to be disambiguated with the corresponding entity text sequence in the external knowledge base to create the input of the BERT model. The sequence is as follows:
s = (x1, x2, . . . , xn, c1, c2, . . . , cm). (1)
where x1 xn is the entity text to be disambiguated, c1 cm is the candidate entity text, and s is the spliced text.) Also see fig. 1.
Lin further discloses: before receiving, into a neural capsule embedding network as input, an embedding matrix: and inserting an embedding sentence into each row in the matrix. (pg. 3, sect 2.1, Suppose we have a sentence, which has n tokens, represented in a sequence of word embeddings.
S = (w1, w2, · · · wn) (1)
Here wi is a vector standing for a d dimentional word embedding for the i-th word in the sentence. S is thus a sequence represented as a 2-D matrix, which concatenates all the word embeddings together. S should have the shape n-by-d. Abstract, This paper proposes a new model for extracting an interpretable sentence embedding by introducing self-attention. Instead of using a vector, we use a 2-D matrix to represent the embedding, with each row of the matrix attending on a different part of the sentence.)
The rationale for the combination would be similar to the one provided in Claim 1 see above.
Regarding Claim 3, Jiang in view of Lin discloses: discloses all the limitation of Claim 1. (see detail element mapping above).
Jiang further discloses: before receiving, into a neural capsule embedding network as input, an embedding matrix: a) receiving, as input, a natural language text; (pg. 3, sect III, Given a sentence x = (x1, x2, x3,. . . , xn), it is vectorized and represented as the input of the model.)
b) converting words in the natural language text into embeddings and inserting embeddings into an embedding vector; (pg. 3, sect III, Given a sentence x = (x1, x2, x3,. . . , xn), it is vectorized and represented as the input of the model. Pg.3, sect III.A this paper splices the entity text sequence to be disambiguated with the corresponding entity text sequence in the external knowledge base to create the input of the BERT model. The sequence is as follows:
s = (x1, x2, . . . , xn, c1, c2, . . . , cm). (1)
where x1 xn is the entity text to be disambiguated, c1 cm is the candidate entity text, and s is the spliced text.) Also see fig. 1.
Lin further discloses: c) converting the embedding vector to an embedding matrix by inserting an embedding sentence into each row in the matrix. (pg. 3, sect 2.1, Suppose we have a sentence, which has n tokens, represented in a sequence of word embeddings.
S = (w1, w2, · · · wn) (1)
Here wi is a vector standing for a d dimentional word embedding for the i-th word in the sentence. S is thus a sequence represented as a 2-D matrix, which concatenates all the word embeddings together. S should have the shape n-by-d. Abstract, This paper proposes a new model for extracting an interpretable sentence embedding by introducing self-attention. Instead of using a vector, we use a 2-D matrix to represent the embedding, with each row of the matrix attending on a different part of the sentence.)
The rationale for the combination would be similar to the one provided in Claim 1 see above.
Regarding Claim 4, Jiang in view of Lin discloses: discloses all the limitation of Claim 1. (see detail element mapping above).
Jiang further discloses: after receiving, into a neural capsule embedding network as input, (pg. 3, sect III, First, the BERT model is used to extract the features of the pre-processed data, and the obtained semantic vector representation is used as the input of the CNN model and capsule network. The features extracted from the model are fused, and the fused information is classified by the fully connected layer.)
Lin further discloses: after receiving, into a neural capsule embedding network as input, an embedding matrix, (pg. 1, Abstract, This paper proposes a new model for extracting an interpretable sentence embedding by introducing self-attention. Instead of using a vector, we use a 2-D matrix to represent the embedding, with each row of the matrix attending on a different part of the sentence.)
The rationale for the combination would be similar to the one provided in Claim 1 see above.
Regarding Claim 5, Jiang in view of Lin discloses: discloses all the limitation of Claim 1. (see detail element mapping above).
Jiang further discloses: before receiving, into a neural capsule embedding network as input, an embedding matrix: a) receiving, as input, a natural language text; (pg. 3, sect III, Given a sentence x = (x1, x2, x3,. . . , xn), it is vectorized and represented as the input of the model.)
b) pre-processing the natural language text to identify features of the natural language text; (pg. 3, sect III, First, the BERT model is used to extract the features
of the pre-processed data, and the obtained semantic vector representation is used as the input of the CNN model and capsule network. The features extracted from the model are fused, and the fused information is classified by the fully connected layer.)
Lin further discloses: c) converting words in the natural language text into embeddings and inserting an embedding sentence into each row in the matrix. (pg. 3, sect 2.1, Suppose we have a sentence, which has n tokens, represented in a sequence of word embeddings.
S = (w1, w2, · · · wn) (1)
Here wi is a vector standing for a d dimentional word embedding for the i-th word in the sentence. S is thus a sequence represented as a 2-D matrix, which concatenates all the word embeddings together. S should have the shape n-by-d. Abstract, This paper proposes a new model for extracting an interpretable sentence embedding by introducing self-attention. Instead of using a vector, we use a 2-D matrix to represent the embedding, with each row of the matrix attending on a different part of the sentence.)
The rationale for the combination would be similar to the one provided in Claim 1 see above.
Regarding Claim 6, Jiang in view of Lin discloses: discloses all the limitation of Claim 1. (see detail element mapping above).
Jiang further discloses: before receiving, into a neural capsule embedding network as input, (pg. 3, sect III, Given a sentence x = (x1, x2, x3,. . . , xn), it is vectorized and represented as the input of the model.)
b) pre-processing the natural language text to identify features of the natural language text; (pg. 3, sect III, First, the BERT model is used to extract the features
of the pre-processed data, and the obtained semantic vector representation is used as the input of the CNN model and capsule network. The features extracted from the model are fused, and the fused information is classified by the fully connected layer.)
c) converting words in the natural language text into embeddings and inserting embeddings into an embedding vector; (pg. 3, sect III, Given a sentence x = (x1, x2, x3,. . . , xn), it is vectorized and represented as the input of the model. Pg.3, sect III.A this paper splices the entity text sequence to be disambiguated with the corresponding entity text sequence in the external knowledge base to create the input of the BERT model. The sequence is as follows:
s = (x1, x2, . . . , xn, c1, c2, . . . , cm). (1)
where x1 xn is the entity text to be disambiguated, c1 cm is the candidate entity text, and s is the spliced text.)
Lin further discloses: before receiving, into a neural capsule embedding network as input, an embedding matrix: d) converting the embedding vector to an embedding matrix by inserting an embedding sentence into each row in the matrix. (pg. 3, sect 2.1, Suppose we have a sentence, which has n tokens, represented in a sequence of word embeddings.
S = (w1, w2, · · · wn) (1)
Here wi is a vector standing for a d dimentional word embedding for the i-th word in the sentence. S is thus a sequence represented as a 2-D matrix, which concatenates all the word embeddings together. S should have the shape n-by-d. Abstract, This paper proposes a new model for extracting an interpretable sentence embedding by introducing self-attention. Instead of using a vector, we use a 2-D matrix to represent the embedding, with each row of the matrix attending on a different part of the sentence.)
The rationale for the combination would be similar to the one provided in Claim 1 see above.
Regarding Claim 10, Jiang in view of Lin discloses: discloses all the limitation of Claim 1. (see detail element mapping above).
Jiang further discloses: wherein unique ID numbers are a predefined set of named entity IDs. (pg. 2, sect I, The unique identification of the entity is used to make the classification label at the same time. This paper counts the mentions in the training data that cannot link to the knowledge base, places the mentions with the same entity name in a list, finds the mentions with the same entity name in the external knowledge base, and uses the unique identification of the mentions to splice the text of the mention with the text of the knowledge base to construct the text that can be trained by the model.)
Regarding Claim 13, Jiang in view of Lin discloses: discloses all the limitation of Claim 1. (see detail element mapping above).
Jiang further discloses: wherein through dynamic routing of capsules, capsules agree on the features of words used to disambiguate a named entity. (pg. 5, sect III.C, The model is composed of a convolutional layer and a dynamic routing layer. The core of the dynamic routing layer is the dynamic routing algorithm, which uses a vector representation of capsule nodes instead of the scalar value representation of neuron nodes used in traditional neural networks to replace the pooling algorithm, thus extracting more abundant location space information instead of the scalar generated by the pooling operation. To capture the semantic information of Chinese short texts more fully, this paper completes the learning of semantic dependency via a two-layer capsule network. The formula for dynamic routing is as follows. … Abstract, In the end, the semantic features obtained by the above models are combined through a fully connected layer to complete the task of entity disambiguation.)
Regarding Claim 14, Jiang in view of Lin discloses: discloses all the limitation of Claim 1. (see detail element mapping above).
Jiang further discloses: wherein through dynamic routing of capsules, capsules agree on the features of words used to identify, classify, and disambiguate a named entity. (pg. 5, sect III.C, The model is composed of a convolutional layer and a dynamic routing layer. The core of the dynamic routing layer is the dynamic routing algorithm, which uses a vector representation of capsule nodes instead of the scalar value representation of neuron nodes used in traditional neural networks to replace the pooling algorithm, thus extracting more abundant location space information instead of the scalar generated by the pooling operation. To capture the semantic information of Chinese short texts more fully, this paper completes the learning of semantic dependency via a two-layer capsule network. The formula for dynamic routing is as follows. …Conclusion, The dual-channel hybrid network proposed in this paper is a parallel model that combines a hybrid pooled CNN model with a capsule network model with a self-attention mechanism to fully learn semantic information. After merging the learned knowledge of the two models, the proposed approach uses unique identifiers to construct a text classification method to perform entity disambiguation tasks. Abstract, In the end, the semantic features obtained by the above models are combined through a fully connected layer to complete the task of entity disambiguation.)
Regarding Claim 15, Jiang discloses: 15. A computer-implemented method for named entity disambiguation, comprising: (pg.2, sect I. This paper proposes a DHCN model, which is a combination of a pooling CNN and capsule network with attention. The semantic information generated by the two models is fused in parallel to complete the disambiguation. To the best of our knowledge, this is the first time that capsules have been used in Chinese entity disambiguation tasks.)
receiving, into a neural capsule embedding network as input, an embedding vector,(pg. 3, sect III, Given a sentence x = (x1, x2, x3,. . . , xn), it is vectorized and represented as the input of the model. … First, the BERT model is used to extract the features of the pre-processed data, and the obtained semantic vector representation is used as the input of the CNN model and capsule network.) Also see fig. 1.
analyzing, by the neural capsule embedding network, the features of each word in context of the embedding matrix considering tokens to the left and right of the word using at least one layer, each layer consisting of at least one set of filters; (pg. 5, sect III.C The core idea is that not all words in a sentence have the same contribution to NLP tasks. In this paper, the attention mechanism is used to further calculate the extracted semantic features, pg. 2, sect I e. Then, the text of the mention and the candidate entity set are spliced one by one to expand the text length so that it has sufficient context information. Pg. 5, sect III.B The CNN is generally composed of three parts, namely, the convolutional layer, pool layer, and fully connected layer. The function of the convolutional layer is to extract features. Local word order information from the input long text sequence is used to complete the extracted features. In this paper, windows with convolution kernel sizes of 3, 4, and 5 are used for convolution to obtain local features.) Also see fig. 1. [In the context of CNN convolution layers, kernels act as feature detectors and are essentially the same as filters, attention mechanism also means all the tokens to the left and right of the word and their associated weights have been accounted for contextually.]
through dynamic routing of capsules, by the neural capsule embedding network, converging to a final capsule layer mapping to each word in the input vector; (pg. 5, sect III.C, The model is composed of a convolutional layer and a dynamic routing layer. The core of
the dynamic routing layer is the dynamic routing algorithm, which uses a vector representation of capsule nodes instead of the scalar value representation of neuron nodes used in traditional neural networks to replace the pooling algorithm, thus extracting more abundant location space information instead of the scalar generated by the pooling operation. To capture the semantic information of Chinese short texts more fully, this paper completes the learning of semantic dependency via a two-layer capsule network.)
generating, by the neural capsule embedding network, an output matrix, (pg. 5, sect III.C, Formula (11) calculates the output vector of the capsule network obtained after linear transformation according to the coupling coefficients and capsule output of the previous layer, formula (12) is a squashing operation that changes sr into a vector of length 1 without changing its direction nr, and Formula (13) is used to update the coupling coefficient. The final target vector is obtained through the above four formulas.) Also see fig. 1.
wherein each output matrix value:
a) identifies if a word in the input is a named entity or not a named entity; (pg. 2, sect I, A capsule network is used to capture the semantic features generated by BERT, and an attention mechanism is used to extract the important information of capsule network features. The abovementioned learned semantic knowledge is combined, and the classification is completed by the fully connected layer. Pg. 5, sect III.D Finally, the fused semantic features are input into the fully connected layer with a sigmoid activation function for classification, as shown in formulas)
b) if the word is a named entity, identifies a unique ID number of the entity. (pg. 2, sect I, First, the candidate entity set is obtained from the external knowledge base. Then, the text of the mention and the candidate entity set are spliced one by one to expand the text length so that it has sufficient context information. The unique identification of the entity is used to make the classification label at the same time. Last, the text is input into the BERT model for feature extraction, and the trained features are taken as the input of the CNN model with max-pooling and mean-pooling to capture semantic features. A capsule network is used to capture the semantic features generated by BERT, and an attention mechanism is used to extract the important information of capsule network features. The abovementioned learned semantic knowledge is combined, and the classification is completed by the fully connected layer.)
However, Jiang does not explicitly disclose: an embedding matrix, wherein the embedding matrix contains embeddings and each row in the matrix is an embedding sentence;
Lin (in a related field of natural language processing) discloses: converting, by the neural capsule network, the embedding vector to an embedding matrix, by inserting an embedding sentence into each row in the matrix; (pg. 3, sect 2.1, Suppose we have a sentence, which has n tokens, represented in a sequence of word embeddings.
S = (w1, w2, · · · wn) (1)
Here wi is a vector standing for a d dimentional word embedding for the i-th word in the sentence. S is thus a sequence represented as a 2-D matrix, which concatenates all the word embeddings together. S should have the shape n-by-d. Abstract, This paper proposes a new model for extracting an interpretable sentence embedding by introducing self-attention. Instead of using a vector, we use a 2-D matrix to represent the embedding, with each row of the matrix attending on a different part of the sentence.)
Jiang and Lin are considered analogous art. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Jiang to combine with the teaching of Lin to disclose the above feature, because the technique described by Lin improve the named entity disambiguation as having multiple rows in sentence embedding is expected to provide more abundant information about the encoded content (Lin, [Abstract and sect 2.1, sect 4.4.1]).
Regarding Claim 16, Jiang in view of Lin discloses: discloses all the limitation of Claim 15. (see detail element mapping above).
Jiang further discloses: before receiving, into a neural capsule embedding network, an embedding vector as input: a) receiving, as input, a natural language text; (pg. 3, sect III, Given a sentence x = (x1, x2, x3,. . . , xn), it is vectorized and represented as the input of the model.)
b) converting words in the natural language text into embeddings and inserting embeddings into an embedding vector. (pg. 3, sect III, Given a sentence x = (x1, x2, x3,. . . , xn), it is vectorized and represented as the input of the model. Pg.3, sect III.A this paper splices the entity text sequence to be disambiguated with the corresponding entity text sequence in the external knowledge base to create the input of the BERT model. The sequence is as follows:
s = (x1, x2, . . . , xn, c1, c2, . . . , cm). (1)
where x1 xn is the entity text to be disambiguated, c1 cm is the candidate entity text, and s is the spliced text.) Also see fig. 1.
Regarding Claim 17, Jiang in view of Lin discloses: discloses all the limitation of Claim 15. (see detail element mapping above).
Jiang further discloses: before receiving, into a neural capsule embedding network, an embedding vector as input: a) receiving as input a natural language text; (pg. 3, sect III, Given a sentence x = (x1, x2, x3,. . . , xn), it is vectorized and represented as the input of the model.)
b) pre-processing the natural language text to identify features of the natural language text; (pg. 3, sect III, First, the BERT model is used to extract the features
of the pre-processed data, and the obtained semantic vector representation is used as the input of the CNN model and capsule network. The features extracted from the model are fused, and the fused information is classified by the fully connected layer.)
c) converting words in the natural language text into embeddings to include in an embedding vector. (pg. 3, sect III, Given a sentence x = (x1, x2, x3,. . . , xn), it is vectorized and represented as the input of the model. Pg.3, sect III.A this paper splices the entity text sequence to be disambiguated with the corresponding entity text sequence in the external knowledge base to create the input of the BERT model. The sequence is as follows:
s = (x1, x2, . . . , xn, c1, c2, . . . , cm). (1)
where x1 xn is the entity text to be disambiguated, c1 cm is the candidate entity text, and s is the spliced text.) Also see fig. 1.
Claims 7-9, and 12 are rejected under 35 U.S.C. 103 as being unpatentable over Jiang in view of Lin, and further in view of Deng, J., Cheng, L., & Wang, Z. (2020). Self-attention-based BiGRU and capsule network for named entity recognition. arXiv preprint arXiv:2002.00735.
Regarding Claim 7, Jiang in view of Lin discloses: discloses all the limitation of Claim 1. (see detail element mapping above).
Jiang in view of Lin does not explicitly disclose: wherein, the output matrix columns correspond to the locations of the words in the input string, and the output matrix rows correspond to named entity classes.
Deng (in the related art of Named Entity Recognition using self attention BiGRU and capsule network) discloses: wherein, the output matrix columns correspond to the locations of the words in the input string, and the output matrix rows correspond to named entity classes. (pg. 2, sect II.D, The CapsNet outputs capsule vector that represents each category. Pg. 6, sect III.C, After the entity is recognized by the CapsNet, the sentence sequence is labelled with entity labels. Because the labeling of each word in the sentence has a strong dependency, the labeling of one word affects the labeling information of the next word, so this paper introduces a markov transition matrix to represent the effects between entity labels. Pg. 3, sect III The input feature layer is mainly represented by BERT pre-trained model training character vector. The BiGRU network mainly captures the text context features, and the self-attention mechanism further acquires the deep features of the text. CapsNet predicts the entity label corresponding to the character, and then adds a Markov transfer function to constrain the prediction result to achieve entity recognition. Pg. 3, sect III.A, The character embedding vector is defined as ec = (ec [CLS], ecc1, ecc2, ..., eccn, ec[SEP ]), the sentence embedding vector is defined as es = (esA, esA, esA, ..., esA, esA),and the character position embedding vector is defined as ep = (ep0, ep1, ep2, ..., epn, epn+1). [The model first processes the input string to generate a rich sequence of feature vectors. The sequence has a vector for each word (or character), meaning the columns of the final matrix align with the input string's sequence. The CapsNet layer then takes these context-aware feature vectors and produces a capsule vector for each potential named entity class. In this context, the capsule output vectors are essentially the rows of the final matrix, with each row representing a different entity class. The Markov transition matrix refines the CapsNet's predictions. It ensures that the final entity labels adhere to valid sequence patterns, improving the overall accuracy. This provides a final constraint on the model's output, but the basic row/column structure is established by the CapsNet acting on the input sequence. Sequence-to-sequence labeling process, which is how such a matrix would be constructed. For a sequence of words, a named entity recognition (NER) model, such as the one described, assigns a label from a set of classes to each word.]
Jiang, Lin and Deng are considered analogous art. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Jiang and Lin to combine with the teaching of Deng to disclose the above feature, because the technique described by Deng has the benefit of superior performance without relying on external dictionary information, meaning it can identify named entities effectively using only the provided text data, rather than requiring a separate, pre-defined list of entities (Deng, [Abstract]).
Regarding Claim 8, Jiang in view of Lin and Deng discloses: discloses all the limitation of Claim 7. (see detail element mapping above).
Deng further discloses: wherein the named entity classes are a predefined set of named entity classes. (pg. 1, sect I, this paper proposes a new NER model called B-SABCN, which aims to enhance the model’s ability to capture important information. Firstly, the BERT pre-trained model is used to train the character embedded representation, which can improve the expression ability of character vector information. Secondly, the BiGRU network is used to capture the context information. For the problem of information redundancy, a self-attention mechanism is proposed to give different focus on features captured by the hidden layer of BiGRU. Finally, it is proposed to use CapsNet to classify entity categories in order to obtain
richer information to generate entity label sequence.)
The rationale for the combination would be similar to the one provided in Claim 7 see above.
Regarding Claim 9, Jiang in view of Lin and Deng discloses: discloses all the limitation of Claim 7. (see detail element mapping above).
Deng further discloses: wherein the named entity classes are clusters determined by the neural capsule embedding network. (pg. 1 sect I, it is proposed to use CapsNet to classify entity categories in order to obtain richer information to generate entity label sequence.) Also see fig. 5 from below
PNG
media_image1.png
578
600
media_image1.png
Greyscale
The rationale for the combination would be similar to the one provided in Claim 7 see above.
Regarding Claim 12, Jiang in view of Lin discloses: discloses all the limitation of Claim 1. (see detail element mapping above).
Jiang in view of Lin does not explicitly disclose: further comprising where each output matrix value: if the word is a named entity, identifies what class the named entity belongs to.
Deng further discloses: further comprising where each output matrix value: if the word is a named entity, identifies what class the named entity belongs to. (pg. 5, sect III.C, CapsNet is introduced to recognize entities, in which capsule represents the entity label, the modulus length of capsule vector represents the entity label prediction probability, and the direction of capsule vector represents the attribute of the entity. Since capsule uses vector representation instead of scalar representation, it has stronger ability to express entity information.) Also see figs. 1 and 5.
Jiang, Lin and Deng are considered analogous art. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Jiang and Lin to combine with the teaching of Deng to disclose the above feature, because the technique described by Deng has the benefit of superior performance without relying on external dictionary information, meaning it can identify named entities effectively using only the provided text data, rather than requiring a separate, pre-defined list of entities (Deng, [Abstract]).
Claim 11 is rejected under 35 U.S.C. 103 as being unpatentable over Jiang in view of Lin, and further in view of Srivastava (US 20210090302).
Regarding Claim 11, Jiang in view of Lin discloses: discloses all the limitation of Claim 1. (see detail element mapping above).
Jiang in view of Lin does not explicitly disclose: wherein unique ID numbers are determined by the neural capsule embedding network.
Srivastava (in a related field of encoding three-dimensional data for processing by capsule neural networks.) further discloses: wherein unique ID numbers are determined by the neural capsule embedding network. ([0084] The representations of the scene from various time steps can be processed by a capsule neural network to align the representations (e.g., when the mobile robot is moving) and to identify and track moving objects that are present in the scene.) [Capsule networks produce robust, identifiable embeddings. Capsule networ