Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Remarks
This Office Action is responsive to Applicants' Amendment filed on August 3, 2026, in which claims 1, 8, and 15 are currently amended. Claims 1-2, 4-9, 11-16, and 18-20 are currently pending.
Response to Arguments
The rejections to claims 1, 2, and 4-7 under 35 U.S.C. § 112(b) are hereby withdrawn, as necessitated by applicant's amendments and remarks made to the rejections.
Applicant’s arguments with respect to rejection of claims 1-2,4-9,11-16 and 18-20 under 35 U.S.C. 103 based on amendment have been considered and are persuasive. The argument is moot in view of a new ground of rejection set forth below.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1, 2, 7, 8, 9, 14, 15, and 16 are rejected under U.S.C. §103 as being unpatentable over the combination of Zhang (“Join-Chain Network: A Logical Reasoning View of the Multi-head Attention in Transformer”, 2022) and He (“RealFormer: Transformer Likes Residual Attention”, 2021).
PNG
media_image1.png
226
704
media_image1.png
Greyscale
FIG. 3 of instant
PNG
media_image2.png
430
1564
media_image2.png
Greyscale
FIG. 1 of Zhang
Regarding claim 1, Zhang teaches A method ([p. 3] "Theorem 3.2: Under the assumption 3.1, a join-chain network with the ¯H-head and ¯ L-layer self-attention block can express all the predicates in FOET")
receiving input comprising natural language texts; ([p. 1] "a new perspective on the mechanism of the pretrained models such as BERT for natural language understanding" [p. 2] " our framework has very wide applications in NLP tasks" [p. 3] "our work provides a novel explanation why the multi-head attention achieves great success in recent development of NLP from a new perspective")
the FOLNet neural network model comprising of a plurality of layers; ([p. 3] "Theorem 3.2: Under the assumption 3.1, a join-chain network with the ¯H-head and ¯ L-layer self-attention block can express all the predicates in FOET")
obtaining unary representations generated based on a plurality of tokens, and binary representations based on one or more pairs of the plurality of tokens ([p. 2] "Each Pmm is a unary predicate and each Wmm is a binary predicate" [p. 3] "Generally speaking, the self-attention matrix A learns all the values of the binary predicate W(x,y), and the value tensor V learns all the values of the unary predicate P(y)" we denote the domain of all the predicates, including all the binary predicates W(x,y) and unary predicates P(y), as {x1,x1,x3,...,xS}, which means x,y ∈ {x1,x2,x3,...,xS}" the tokens are the elements in the set {x1,…,xs})
encoding a logical inductive bias by at least: using, within each layer of the plurality of layers, a first interacting branch of the FOLNet neural network model to update unary representations based on applying a plurality of first neural logic operators on the unary representations and the binary representations, ([p. 2] "join operators will conduct the join operations on the inputs to each layer." [p. 3] "we find that the widely adopted multi-head attention mechanism can be understood as a join operator" [p. 3] "we denote the domain of all the predicates, including all the binary predicates W(x,y) and unary predicates P(y), as {x1,x1,x3,...,xS}, which means x,y ∈ {x1,x2,x3,...,xS}. Then in the multi-head attention mechanism, the core part is the product between the self-attention matrix A and the value tensor V, Z=AV" See FIG. 1 where branches are parallel and appear analogous to FIG. 3 and 4 of the instant Drawings. Zhang teaches that each attention head/join module performs the join operation between binary predicate W(x,y) and unary predicate P(y). In Transformer terms, A is the binary predicate matrix, V is the unary predicate tensor, and Z=AV is the updated join result. Thus, the first interacting branch can be read as the value/unary-output path updated by neural join/self-attention operators using both A and V)
and a second interacting branch of the FOLNet neural network to update binary representations based on applying a plurality of second neural logic operator on the unary representations and the binary representations, ([p. 3] "the self-attention matrix A learns all the values of the binary predicate W(x,y), and the value tensor V learns all the values of the unary predicate P(y). For each head of attention mechanism in each layer, the self-attention matrix A learns a binary predicate W(x,y)" Zhang teaches that each attention head in each layer learns a binary predicate through the self attention matrix A. Zhang explicitly treats A as learned binary predicate values and Z=AV as the join output.)
wherein the first interacting branch and the second interacting branch are parallel to each other ([p. 1] "Multiple paralleled attention heads" [p. 2] "join operation: P(x) = ∃yW(x,y)P(y) […] there are multiple join operations to conduct in each step, we need to include several join operation modules in each layer […] The aggregation after the join operators will aggregate the inputs from the skip-connection and the results of join operators. This reflects the process that we need to conduct ∧ operations after the join operations in each step" See FIG. 1 where branches appear parallel in view of FIG. 4 of the instant)
wherein the plurality of first neural logic operators comprises at least one operator that updates a unary representation from one or more of the binary representations, ([p. 3] "the self-attention matrix A learns all the values of the binary predicate W(x,y), and the value tensor V learns all the values of the unary predicate P(y). For each head of attention mechanism in each layer, the self-attention matrix A learns a binary predicate W(x,y)" Zhang teaches an attention matrix A to binary predicate W(x,y), value tensor V to unary predicate P(y), and Z=AV to the join producing a unary predicate output.).
However, Zhang does not explicitly teach executed by at least one processor, the method comprising:
pre-training a First-Order Logic Network (FOLNet) neural network model on unlabeled texts included in the natural language texts,
outputting one or more tensors based on the logical inductive bias;
predicting an outcome using the one or more tensors
and the plurality of second neural logic operators comprises at least one operator that updates a binary representation from one or more of the binary representations and at least one operator that updates a binary representation from one or more of the unary representations.
He, in the same field of endeavor, teaches executed by at least one processor, the method comprising: ([p. 936] "Our experiments are performed on NVIDIA A100 GPUs, based on the official ADMIN repository12. We follow Liu et al. (2020) to configure the amount of GPUs to use for different setups")
pre-training a First-Order Logic Network (FOLNet) neural network model on unlabeled texts included in the natural language texts, ([p. 932] "We follow the standard pre-training setup (dataset: Wikipedia + BookCorpus, vocab: uncased 30K, max sequence length: 5126, dropout: 10%, learning rate: 1e-4, learning rate schedule: warm up and then linearly decay to 0, weight decay: 0.01, optimizer: AdamW, objective: Masked Language Modeling + Next Sentence Prediction, etc.) to compare three Trans former models: Post-LN, Pre-LN, and RealFormer." [p. 937] "Our experiments are based on the official ETC repository13. We take the ETC-Large model (24 layers, 1024 hidden size, 16 heads), add residual attention edges (i.e., using running sum), and follow all the pre-training and fine-tuning recipes")
outputting one or more tensors based on the logical inductive bias; ([p. 931] "The first sub-layer contains a Multi-Head Attention module that computes output embeddings of a set of queries (Q) […] WO is a matrix that linearly transforms the concatenation of the outputs of all heads" [p. 932] "ResidualAttention(Q,K,V ,Prev) = Softmax(QKT √ dk +Prev)V [...] Finally, new attention scores QKT √ dk +Prev are passed over to the next layer.")
predicting an outcome using the one or more tensors ([p. 932] "Next Sentence Prediction")
and the plurality of second neural logic operators comprises at least one operator that updates a binary representation from one or more of the binary representations and at least one operator that updates a binary representation from one or more of the unary representations. ([p. 932] "formally, it adds Prev, the pre-softmax attention scores from the previous layer with shape (#heads,from seq len,to seq len),4 as one additional input to the Multi-Head Attention module in the current layer […] headi = ResidualAttention(QWQ i , KWK i ,VWV i ,Previ) and Previ is the slice of Prev with shape (from seq len, to seq len) corresponding to headi. ResidualAttention adds “residual scores” on top of Previ [...] Finally, new attention scores QKT √ dk +Prev are passed over to the next layer" (#heads,from seq len, to seq len) is a token pair representation (binary representation). He then modifies the current attention operation so that the previous binary score matrix is added to the newly generated pairwise score matrix and further states that these new attention scores are passed to the new layer. Therefore, He teaches a persistent binary state: previous binary scores -> updated binary scores, which is interpreted as updating a binary representation from one or more of the binary representations and at least one operator that updates a binary representation from one or more of the unary representations (Q' and K' which form Q,K->QK^T relationship)).
Zhang as well as He are directed towards transformers for multi-head attention. Therefore, Zhang as well as He are analogous art in the same field of endeavor. Zhang teaches that the computations performed by multi-head self-attention transformers may be interpreted as neural logical operations over unary and binary predicate representations. He teaches a transformer architecture in which the pairwise attention-score representation at a current layer is generated from current token-wise query and key representations and is further updated using the pairwise attention-score representation from a preceding layer. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to apply Zhang's disclosed logical interpretation of transformer attention to He's Realformer and would have understood He's current-layer attention computation as an operator updating a binary representation from unary representations, and He's residual attention addition as an operator updating a binary representation from a prior binary representation. He provides as additional motivation for combination ([p. 937] "We propose RealFormer, a simple, generic, and cheap technique based on the novel idea of residual attention to improve Transformer-based networks. Quantitatively, we show that RealFormer can improve a diverse set of state-of-the-art Transformer based models considerably for tasks like Masked Language Modeling, Neural Machine Translation, and long document modeling. Qualitatively, we show that RealFormer tends to have comparatively sparser attention, both within heads and across heads in adjacent layers"). This motivation for combination also applies to the remaining claims which depend on this combination.
Regarding claim 2, the combination of Zhang, and He teaches The method according to claim 1, wherein outputting the one or more tensors comprises preprocessing the input into one or more tokens to be converted into the one or more tensors. (He [p. 932] "We follow the standard pre-training setup (dataset: Wikipedia + BookCorpus, vocab: uncased 30K, max sequence length: 5126" [p. 930] "path to propagate token embeddings; (c) Our RealFormer layer that creates a “direct” path to propagate attention scores (by adding a simple skip edge on top of (a))" [p. 933] "All methods achieved similar (and great) results on Next Sentence Prediction" He discloses preprocessing natural-language input into tokens represented by token embeddings and corresponding tensor representations. In particular, He applies RealFormer to BERT using a 30K vocabulary and sequences of up to 512 tokens and describes the resulting token embeddings as being processed through matrix-valued Q, K, and V representations to produce output embeddings).
Regarding claim 7, the combination of Zhang, and He teaches The method according to claim 1, wherein the pre-training the FOLNet neural network model on unlabeled texts comprises training the FOLNet neural network model to solve one or more downstream tasks. (He [p. 932] "We follow the standard pre-training setup (dataset: Wikipedia + BookCorpus, vocab: uncased 30K, max sequence length: 5126, dropout: 10%, learning rate: 1e-4, learning rate schedule: warm up and then linearly decay to 0, weight decay: 0.01, optimizer: AdamW, objective: Masked Language Modeling + Next Sentence Prediction, etc.) to compare three Trans former models: Post-LN, Pre-LN, and RealFormer." [p. 937] "Our experiments are based on the official ETC repository13. We take the ETC-Large model (24 layers, 1024 hidden size, 16 heads), add residual attention edges (i.e., using running sum), and follow all the pre-training and fine-tuning recipes").
Regarding claim 8, Zhang teaches An apparatus comprising: ([p. 3] "Theorem 3.2: Under the assumption 3.1, a join-chain network with the ¯H-head and ¯ L-layer self-attention block can express all the predicates in FOET")
receiving code configured to cause the at least one processor to receive input comprising natural language texts; ([p. 1] "a new perspective on the mechanism of the pretrained models such as BERT for natural language understanding" [p. 2] "our framework has very wide applications in NLP tasks" [p. 3] "our work provides a novel explanation why the multi-head attention achieves great success in recent development of NLP from a new perspective")
the FOLNet neural network model comprising of a plurality of layers; ([p. 3] "Theorem 3.2: Under the assumption 3.1, a join-chain network with the ¯H-head and ¯ L-layer self-attention block can express all the predicates in FOET")
encoding code configured to cause the at least one processor to encode a logical inductive bias using the FOLNet neural network model; ([Abstract] "logical reasoning has become increasingly important for a wide range of applications (e.g., natural language processing). Towards this grand objective, we propose a symbolic reasoning architecture that chains many join operators together to model output logical expressions. In particular, we demonstrate that such an ensemble of join chains can express a broad subset of “tree-structured” first-order logical expressions, named FOET [...] we find that the widely used multi-head self attention module in transformer can be understood as a special neural operator that implements the union bound of the join operator in probabilistic predicate space" Zhang's join-chain network is designed specifically around recursive calculation of first-order logic predicates using join operations. Its self-attention blocks are explained as neural approximations of symbolic join operations.)
outputting code configured to cause the at least one processor to output one or more tensors based on the logical inductive bias; ([p. 3] "the tensor Z will the join operation between W(x,y) and P(y) […] the self-attention matrix A learns all the values of the binary predicate W(x,y), and the value tensor V learns all the values of the unary predicate P(y)" [Abstract] "we propose a symbolic reasoning architecture that chains many join operators together to model output logical expressions")
and predicting code configured to cause the at least one processor to predict an outcome using the one or more tensors, ([p. 1] "derive the value of P(x) for a given x" [p. 2] "we find a key operation for the calculation of P(x) and name it as the join operation")
and wherein the encoding code is further configured to cause the at least one processor to: obtain unary representations generated based on a plurality of tokens, and binary representations based on one or more pairs of the plurality of tokens([p. 2] "Each Pmm is a unary predicate and each Wmm is a binary predicate" [p. 3] "Generally speaking, the self-attention matrix A learns all the values of the binary predicate W(x,y), and the value tensor V learns all the values of the unary predicate P(y)" we denote the domain of all the predicates, including all the binary predicates W(x,y) and unary predicates P(y), as {x1,x1,x3,...,xS}, which means x,y ∈ {x1,x2,x3,...,xS}" the tokens are the elements in the set {x1,…,xs})
use, within each layer of the plurality of layers, a first interacting branch of the FOLNet neural network model to update unary representations based on applying a plurality of first neural logic operators on the unary representations and the binary representations, ([p. 2] "join operators will conduct the join operations on the inputs to each layer." [p. 3] "we find that the widely adopted multi-head attention mechanism can be understood as a join operator" [p. 3] "we denote the domain of all the predicates, including all the binary predicates W(x,y) and unary predicates P(y), as {x1,x1,x3,...,xS}, which means x,y ∈ {x1,x2,x3,...,xS}. Then in the multi-head attention mechanism, the core part is the product between the self-attention matrix A and the value tensor V, Z=AV" See FIG. 1 where branches are parallel and appear analogous to FIG. 3 and 4 of the instant Drawings. Zhang teaches that each attention head/join module performs the join operation between binary predicate W(x,y) and unary predicate P(y). In Transformer terms, A is the binary predicate matrix, V is the unary predicate tensor, and Z=AV is the updated join result. Thus, the first interacting branch can be read as the value/unary-output path updated by neural join/self-attention operators using both A and V)
and a second interacting branch to update binary representations based on applying a plurality of second neural logic operators on the unary representations and the binary representations,([p. 3] "the self-attention matrix A learns all the values of the binary predicate W(x,y), and the value tensor V learns all the values of the unary predicate P(y). For each head of attention mechanism in each layer, the self-attention matrix A learns a binary predicate W(x,y)" Zhang teaches that each attention head in each layer learns a binary predicate through the self attention matrix A. Zhang explicitly treats A as learned binary predicate values and Z=AV as the join output.)
wherein the first interacting branch and the second interacting branch are parallel to each other. ([p. 1] "Multiple paralleled attention heads" [p. 2] "join operation: P(x) = ∃yW(x,y)P(y) […] there are multiple join operations to conduct in each step, we need to include several join operation modules in each layer […] The aggregation after the join operators will aggregate the inputs from the skip-connection and the results of join operators. This reflects the process that we need to conduct ∧ operations after the join operations in each step" See FIG. 1 where branches appear parallel in view of FIG. 4 of the instant)
and wherein the plurality of first neural logic operators comprises at least one operator that updates a unary representation from one or more of the binary representations, ([p. 3] "the self-attention matrix A learns all the values of the binary predicate W(x,y), and the value tensor V learns all the values of the unary predicate P(y). For each head of attention mechanism in each layer, the self-attention matrix A learns a binary predicate W(x,y)" Zhang teaches an attention matrix A to binary predicate W(x,y), value tensor V to unary predicate P(y), and Z=AV to the join producing a unary predicate output.).
However, Zhang does not explicitly teach at least one memory configured to store program code; and at least one processor configured to read the program code and operate as instructed by the program code, the program code comprising:
pre-training code configured to cause the at least one processor to pre-train a First-Order Logic Network (FOLNet) neural network model on unlabeled texts included in the natural language texts,
and the plurality of second neural logic operators comprises at least one operator that updates a binary representation from one or more of the binary representations and at least one operator that updates a binary representation from one or more of the unary representations.
He, in the same field of endeavor, teaches at least one memory configured to store program code; and at least one processor configured to read the program code and operate as instructed by the program code, the program code comprising: ([p. 936] "Our experiments are performed on NVIDIA A100 GPUs, based on the official ADMIN repository. We follow Liu et al. (2020) to configure the amount of GPUs to use for different setups")
pre-training code configured to cause the at least one processor to pre-train a First-Order Logic Network (FOLNet) neural network model on unlabeled texts included in the natural language texts, ([p. 932] "We follow the standard pre-training setup (dataset: Wikipedia + BookCorpus, vocab: uncased 30K, max sequence length: 5126, dropout: 10%, learning rate: 1e-4, learning rate schedule: warm up and then linearly decay to 0, weight decay: 0.01, optimizer: AdamW, objective: Masked Language Modeling + Next Sentence Prediction, etc.) to compare three Trans former models: Post-LN, Pre-LN, and RealFormer." [p. 937] "Our experiments are based on the official ETC repository13. We take the ETC-Large model (24 layers, 1024 hidden size, 16 heads), add residual attention edges (i.e., using running sum), and follow all the pre-training and fine-tuning recipes")
and the plurality of second neural logic operators comprises at least one operator that updates a binary representation from one or more of the binary representations and at least one operator that updates a binary representation from one or more of the unary representations. ([p. 932] "formally, it adds Prev, the pre-softmax attention scores from the previous layer with shape (#heads,from seq len, to seq len),4 as one additional input to the Multi-Head Attention module in the current layer […] headi = ResidualAttention(QWQ i , KWK i ,VWV i ,Previ) and Previ is the slice of Prev with shape (from seq len, to seq len) corresponding to headi. ResidualAttention adds “residual scores” on top of Previ [...] Finally, new attention scores QKT √ dk +Prev are passed over to the next layer" (#heads,from seq len, to seq len) is a token pair representation (binary representation). He then modifies the current attention operation so that the previous binary score matrix is added to the newly generated pairwise score matrix and further states that these new attention scores are passed to the new layer. Therefore, He teaches a persistent binary state: previous binary scores -> updated binary scores, which is interpreted as updating a binary representation from one or more of the binary representations and at least one operator that updates a binary representation from one or more of the unary representations (Q' and K' which form Q,K->QK^T relationship)).
Zhang as well as He are directed towards transformers for multi-head attention. Therefore, Zhang as well as He are analogous art in the same field of endeavor. Zhang teaches that the computations performed by multi-head self-attention transformers may be interpreted as neural logical operations over unary and binary predicate representations. He teaches a transformer architecture in which the pairwise attention-score representation at a current layer is generated from current token-wise query and key representations and is further updated using the pairwise attention-score representation from a preceding layer. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to apply Zhang's disclosed logical interpretation of transformer attention to He's Realformer and would have understood He's current-layer attention computation as an operator updating a binary representation from unary representations, and He's residual attention addition as an operator updating a binary representation from a prior binary representation. He provides as additional motivation for combination ([p. 937] "We propose RealFormer, a simple, generic, and cheap technique based on the novel idea of residual attention to improve Transformer-based networks. Quantitatively, we show that RealFormer can improve a diverse set of state-of-the-art Transformer based models considerably for tasks like Masked Language Modeling, Neural Machine Translation, and long document modeling. Qualitatively, we show that RealFormer tends to have comparatively sparser attention, both within heads and across heads in adjacent layers"). This motivation for combination also applies to the remaining claims which depend on this combination.
Regarding claim 9, the combination of Zhang, and He teaches The apparatus according to claim 8, wherein the outputting code is further configured to cause the at least one processor to preprocess the input into one or more tokens to be converted into the one or more tensors.(He [p. 931] "The first sub-layer contains a Multi-Head Attention module that computes output embeddings of a set of queries (Q) […] WO is a matrix that linearly transforms the concatenation of the outputs of all heads" [p. 932] "ResidualAttention(Q,K,V ,Prev) = Softmax(QKT √ dk +Prev)V [...] Finally, new attention scores QKT √ dk +Prev are passed over to the next layer.").
Regarding claim 14, the combination of Zhang, and He teaches The apparatus according to claim 8, wherein the pre- training code is further configured to cause the at least one processor to train the FOLNet neural network model to solve one or more downstream tasks.(He [p. 932] "We follow the standard pre-training setup (dataset: Wikipedia + BookCorpus, vocab: uncased 30K, max sequence length: 5126, dropout: 10%, learning rate: 1e-4, learning rate schedule: warm up and then linearly decay to 0, weight decay: 0.01, optimizer: AdamW, objective: Masked Language Modeling + Next Sentence Prediction, etc.) to compare three Trans former models: Post-LN, Pre-LN, and RealFormer." [p. 937] "Our experiments are based on the official ETC repository13. We take the ETC-Large model (24 layers, 1024 hidden size, 16 heads), add residual attention edges (i.e., using running sum), and follow all the pre-training and fine-tuning recipes").
Regarding claim 15, claim 15 is directed towards a system for performing the method of claim 1. Therefore, the rejection applied to claim 1 also applies to claim 15. Claim 15 also recites additional elements A non-transitory computer-readable storage medium, storing instructions, which, when executed by at least one processor, cause the at least one processor to: (He [p. 936] "Our experiments are performed on NVIDIA A100 GPUs, based on the official ADMIN repository12. We follow Liu et al. (2020) to configure the amount of GPUs to use for different setups").
Similarly, regarding claim 16, claim 16 is directed towards a system for performing the method of claim 2. Therefore, the rejection applied to claim 2 also applies to claim 16.
Claims 4-6, 11-13, and 18-20 are rejected under U.S.C. §103 as being unpatentable over the combination of Zhang and He and Dong (“NEURAL LOGIC MACHINES”, 2019).
Regarding claim 4, the combination of Zhang, and He teaches The method according to claim 1.
However, the combination of Zhang, and He doesn't explicitly teach wherein the plurality of first neural logic operators and the plurality of second neural logic operators include learnable Horn clauses.
Dong, in the same field of endeavor, teaches the plurality of first neural logic operators and the plurality of second neural logic operators include learnable Horn clauses ([p. 5] "It can be verified that NLMs can realize the forward chaining of a partial set of Horn clauses").
The combination of Zhang and He as well as Dong are directed towards machine learning systems for representation learning and reasoning using neural networks. Therefore, the combination of Zhang and He as well as Dong are analogous art in the same field of endeavor. It would have been obvious before the effective filing date of the claimed invention to combine the teachings of the combination of Zhang and He with the teachings of Dong by using NLM with a Transformer-XL model. Dong provides as additional motivation for combination ([p. 18] ” the input properties or relations can be derived from other neural architectures (e.g., CNNs) […] A convolutional neural network (CNN) is applied to the input extracting multiple features for future reasoning. CNN and NLM can be optimized jointly. This enables reasoning over noisy input”). This motivation for combination also applies to the remaining claims which depend on this combination.
Regarding claim 5, the combination of Zhang, He, and Dong teaches The method according to claim 4, wherein the plurality of first neural logic operators and the plurality of second neural logic operators are forward-chained into the FOLNet neural network model. (He [p. 931] "The second sub-layer contains a fully-connected Feed-Forward Network (FFN) module with one hidden layer").
Regarding claim 6, the combination of Zhang, and He teaches The method according to claim 1.
However, the combination of Zhang, and He doesn't explicitly teach wherein the FOLNet neural network model is a fully differentiable neural architecture.
Dong, in the same field of endeavor, teaches wherein the FOLNet neural network model is a fully differentiable neural architecture. ([p. 10] "being fully differentiable, NLMs can be plugged into existing convolutional or recurrent neural architectures for logic reasoning").
The combination of Zhang and He as well as Dong are directed towards machine learning systems for representation learning and reasoning using neural networks. Therefore, the combination of Zhang and He as well as Dong are analogous art in the same field of endeavor. It would have been obvious before the effective filing date of the claimed invention to combine the teachings of the combination of Zhang and He with the teachings of Dong by using NLM with a Transformer-XL model. Dong provides as additional motivation for combination ([p. 18] ” the input properties or relations can be derived from other neural architectures (e.g., CNNs) […] A convolutional neural network (CNN) is applied to the input extracting multiple features for future reasoning. CNN and NLM can be optimized jointly. This enables reasoning over noisy input”). This motivation for combination also applies to the remaining claims which depend on this combination.
Regarding claim 11, the combination of Zhang, and He teaches The apparatus according to claim 8.
However, the combination of Zhang, and He doesn't explicitly teach wherein the plurality of first neural logic operators and the plurality of second neural logic operators include learnable Horn clauses.
Dong, in the same field of endeavor, teaches the plurality of first neural logic operators and the plurality of second neural logic operators include learnable Horn clauses ([p. 5] "It can be verified that NLMs can realize the forward chaining of a partial set of Horn clauses").
The combination of Zhang and He as well as Dong are directed towards machine learning systems for representation learning and reasoning using neural networks. Therefore, the combination of Zhang and He as well as Dong are analogous art in the same field of endeavor. It would have been obvious before the effective filing date of the claimed invention to combine the teachings of the combination of Zhang and He with the teachings of Dong by using NLM with a Transformer-XL model. Dong provides as additional motivation for combination ([p. 18] ” the input properties or relations can be derived from other neural architectures (e.g., CNNs) […] A convolutional neural network (CNN) is applied to the input extracting multiple features for future reasoning. CNN and NLM can be optimized jointly. This enables reasoning over noisy input”). This motivation for combination also applies to the remaining claims which depend on this combination.
Regarding claim 12, the combination of Zhang, He, and Dong teaches, wherein the plurality of first neural logic operators and the plurality of second neural logic operators are forward-chained into the FOLNet neural network model(He [p. 931] "The second sub-layer contains a fully-connected Feed-Forward Network (FFN) module with one hidden layer").
Regarding claim 13, the combination of Zhang, and He teaches The apparatus according to claim 11.
However, the combination of Zhang, and He doesn't explicitly teach the FOLNet neural network model is a fully differentiable neural architecture.
Dong, in the same field of endeavor, teaches the FOLNet neural network model is a fully differentiable neural architecture ([p. 10] "being fully differentiable, NLMs can be plugged into existing convolutional or recurrent neural architectures for logic reasoning").
The combination of Zhang and He as well as Dong are directed towards machine learning systems for representation learning and reasoning using neural networks. Therefore, the combination of Zhang and He as well as Dong are analogous art in the same field of endeavor. It would have been obvious before the effective filing date of the claimed invention to combine the teachings of the combination of Zhang and He with the teachings of Dong by using NLM with a Transformer-XL model. Dong provides as additional motivation for combination ([p. 18] ” the input properties or relations can be derived from other neural architectures (e.g., CNNs) […] A convolutional neural network (CNN) is applied to the input extracting multiple features for future reasoning. CNN and NLM can be optimized jointly. This enables reasoning over noisy input”). This motivation for combination also applies to the remaining claims which depend on this combination.
Regarding claims 18-20, claims 18-20 are directed towards a system for performing the method of claims 4-6. Therefore, the rejections applied to claims 4-6 also apply to claims 18-20.
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SIDNEY VINCENT BOSTWICK whose telephone number is (571)272-4720. The examiner can normally be reached M-F 7:30am-5:00pm EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Miranda Huang can be reached on (571)270-7092. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/SIDNEY VINCENT BOSTWICK/Examiner, Art Unit 2124
Prosecution Insights