Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA . A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 3/18/2026 has been entered.
Remarks
This Office Action is responsive to Applicants' Amendment filed on March 18, 2026, in which claims 1, 4-6, 8, 11-13, 15, and 18-20 are currently amended. Claims 1, 2, 4-9, 11-16, and 18-20 are currently pending.
Response to Arguments
Applicant’s arguments with respect to rejection of claims 1, 2, 4-9, 11-16, and 18-20 under 35 U.S.C. 101 based on amendment have been considered and are persuasive. The rejection under 35 U.S.C. 101 is hereby withdrawn, as necessitated by applicant's amendments and remarks made to the rejections.
Applicant’s arguments with respect to rejection of claims 1, 2, 4-9, 11-16, and 18-20 under 35 U.S.C. 102/103 based on amendment have been considered.
Applicant’s arguments are moot in view of a new ground of rejection set forth below.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1, 2, and 4-7 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Regarding claim 1, "a second interacting branch of the FOLNet neural network that is parallel to the first interacting branch to update binary representations" is grammatically indefinite. It's unclear what is parallel to the first interacting branch: the FOLNet neural network, the second interacting branch, both, or neither. Similarly, it is grammatically unclear what is updating binary representations. If the second interacting branch is what is intended to be described in parallel with the first interacting branch, then the later recitation "wherein the first interacting branch and the second interacting branch are parallel to each other" seems redundant, adding further ambiguity. In the interest of further examination the claim is interpreted as "a second interacting branch of the FOLNet neural network, the second interacting branch to update binary representations [...] wherein the first interacting branch and the second interacting branch are parallel to each other.
Claims 2 and 4-7 are rejected with respect to their dependence on rejected claim 1.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1, 2, 7-9, and 14-16 are rejected under U.S.C. §103 as being unpatentable over the combination of Zhang (“Join-Chain Network: A Logical Reasoning View of the Multi-head Attention in Transformer”, 2022) and Dai (“Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context”, 2019).
PNG
media_image1.png
226
704
media_image1.png
Greyscale
FIG. 3 of instant
PNG
media_image2.png
430
1564
media_image2.png
Greyscale
FIG. 1 of Zhang
Regarding claim 1, Zhang teaches A method ([p. 3] "Theorem 3.2: Under the assumption 3.1, a join-chain network with the ¯H-head and ¯ L-layer self-attention block can express all the predicates in FOET")
receiving input comprising natural language texts;([p. 1] "a new perspective on the mechanism of the pretrained models such as BERT for natural language understanding" [p. 2] " our framework has very wide applications in NLP tasks" [p. 3] "our work provides a novel explanation why the multi-head attention achieves great success in recent development of NLP from a new perspective")
the FOLNet neural network model comprising of a plurality of layers; ([p. 3] "Theorem 3.2: Under the assumption 3.1, a join-chain network with the ¯H-head and ¯ L-layer self-attention block can express all the predicates in FOET")
obtaining unary representations generated based on a plurality of tokens, and binary representations based on one or more pairs of the plurality of tokens([p. 2] "Each Pmm is a unary predicate and each Wmm is a binary predicate" [p. 3] "Generally speaking, the self-attention matrix A learns all the values of the binary predicate W(x,y), and the value tensor V learns all the values of the unary predicate P(y)" we denote the domain of all the predicates, including all the binary predicates W(x,y) and unary predicates P(y), as {x1,x1,x3,...,xS}, which means x,y ∈ {x1,x2,x3,...,xS}" the tokens are the elements in the set {x1,…,xs})
encoding a logical inductive bias by at least: using, within each layer of the plurality of layers, a first interacting branch of the FOLNet neural network model to update unary representations based on applying a plurality of first neural logic operators on the unary representations and the binary representations, ([p. 2] "join operators will conduct the join operations on the inputs to each layer." [p. 3] "we find that the widely adopted multi-head attention mechanism can be understood as a join operator" [p. 3] "we denote the domain of all the predicates, including all the binary predicates W(x,y) and unary predicates P(y), as {x1,x1,x3,...,xS}, which means x,y ∈ {x1,x2,x3,...,xS}. Then in the multi-head attention mechanism, the core part is the product between the self-attention matrix A and the value tensor V, Z=AV" See FIG. 1 where branches are parallel and appear analogous to FIG. 3 and 4 of the instant Drawings. Zhang teaches that each attention head/join module performs the join operation between binary predicate W(x,y) and unary predicate P(y). In Transformer terms, A is the binary predicate matrix, V is the unary predicate tensor, and Z=AV is the updated join result. Thus, the first interacting branch can be read as the value/unary-output path updated by neural join/self-attention operators using both A and V)
and a second interacting branch of the FOLNet neural network that is parallel to the first interacting branch to update binary representations based on applying a plurality of second neural logic operator on the unary representations and the binary representations, ([p. 3] "the self-attention matrix A learns all the values of the binary predicate W(x,y), and the value tensor V learns all the values of the unary predicate P(y). For each head of attention mechanism in each layer, the self-attention matrix A learns a binary predicate W(x,y)" Zhang teaches that each attention head in each layer learns a binary predicate through the self-attention matrix A. Zhang explicitly treats A as learned binary predicate values and Z=AV as the join output.)
wherein the first interacting branch and the second interacting branch are parallel to each other([p. 1] "Multiple paralleled attention heads" [p. 2] "join operation: P(x) = ∃yW(x,y)P(y) […] there are multiple join operations to conduct in each step, we need to include several join operation modules in each layer […] The aggregation after the join operators will aggregate the inputs from the skip-connection and the results of join operators. This reflects the process that we need to conduct ∧ operations after the join operations in each step" See FIG. 1 where branches appear parallel in view of FIG. 4 of the instant).
However, Zhang does not explicitly teach executed by at least one processor, the method comprising:
pre-training a First-Order Logic Network (FOLNet) neural network model on unlabeled texts included in the natural language texts,
outputting one or more tensors based on the logical inductive bias;
predicting an outcome using the one or more tensors.
Dai, in the same field of endeavor, teaches executed by at least one processor, the method comprising: ([p. 4] "we can cache as many previous segments as the GPU memory allows, and reuse all of them as the extra context when processing the current segment. Thus, we can cache a predefined length-M old hidden states spanning (possibly) multiple segments, and refer to them as the memory" [p. 7] "we also compare Transformer-XL with baselines under the same GPU memory constraints")
pre-training a First-Order Logic Network (FOLNet) neural network model on unlabeled texts included in the natural language texts, ([p. 5] "As a benefit of the inductive bias, a model trained on a memory of some certain length can automatically generalize to a memory several times longer during evaluation" [p. 6] "WikiText-103 is the largest available word-level language modeling benchmark with long-term de pendency. It contains 103M training tokens from 28K articles, with an average length of 3.6K to kens per article, which allows testing the ability of long-term dependency modeling. We set the attention length to 384 during training and 1600 during evaluation [...] we train 18-layer and 24-layer Transformer-XLs with increased model sizes. With the attention length 784 during training and 3,800 during evaluation" Training prior to evaluation interpreted as synonymous with pre-training. WikiText-103 is an unlabeled text database that the model is pre-trained on.)
outputting one or more tensors based on the logical inductive bias;([p. 5] Dai explicitly outputs multiple tensors including the pairwise attention score tensor An_T,i,j, the token-wise tensors an_T, on_T, hn_T and the final output representation tensors)
predicting an outcome using the one or more tensors([p. 2] "yielding a categorical probability distribution over the next token").
Zhang as well as Dai are directed towards Transformer-based natural language modeling using self-attention. Therefore, Zhang as well as Dai are analogous art in the same field of endeavor. It would have been obvious before the effective filing date of the claimed invention to combine the teachings of Zhang with the teachings of Dai. Zhang explains what Transformer multi-head attention is doing logically and Dai provides a concrete Transformer language-model apparatus and pretraining context in which that attention mechanism is used. A person of ordinary skill in the art would have understood, in view of Zhang, that Dai's Transformer-XL attention mechanism already implements a neural logical join over unary value tensors and binary attention predicates, thereby encoding a logical inductive bias within Dai's pretrained natural-language Transformer model. Dai provides as additional motivation for combination for using Transformer-XL specifically ([p. 1] “we improve the state-of the-art results of bpc/perplexity to 0.99 on en wiki8, 1.08 on text8, 18.3 on WikiText-103, 21.8 on One Billion Word, and 54.5 on Penn Treebank (without finetuning)”). This motivation for combination also applies to the remaining claims which depend on this combination.
Regarding claim 2, the combination of Zhang and Dai teaches The method according to claim 1, wherein outputting the one or more tensors comprises preprocessing the input into one or more tokens to be converted into the one or more tensors.(Dai [p. 1] "we improve the state-of the-art results of bpc/perplexity to 0.99 on en wiki8, 1.08 on text8, 18.3 on WikiText-103, 21.8 on One Billion Word, and 54.5 on Penn Treebank (without finetuning). When trained only on WikiText-103, Transformer-XL manages to generate reasonably coherent, novel text articles with thousands of tokens" [p. 2] "Given a corpus of tokens x =(x1,...,xT), the task of language modeling is to estimate the joint probability P(x)" Dai receives input tokens, segments them into S_T, S_T+1, etc., maps tokens to embeddings and then feeds those embeddings into the transformer).
Regarding claim 7, the combination of Zhang and Dai teaches The method according to claim 1, wherein the pre-training the FOLNet neural network model on unlabeled texts comprises training the FOLNet neural network model to solve one or more downstream tasks.(Dai [p. 8 §4.4] "Trained only on WikiText-103 which is medium-sized, Transformer-XL is already able to generate relatively coherent articles with thousands of tokens without manual cherry picking" Text/article generation interpreted as downstream task).
Regarding claim 8, Zhang teaches An apparatus comprising([p. 3] "Theorem 3.2: Under the assumption 3.1, a join-chain network with the ¯H-head and ¯ L-layer self-attention block can express all the predicates in FOET")
receiving code configured to cause the at least one processor to receive input comprising natural language texts; ([p. 1] "a new perspective on the mechanism of the pretrained models such as BERT for natural language understanding" [p. 2] " our framework has very wide applications in NLP tasks" [p. 3] "our work provides a novel explanation why the multi-head attention achieves great success in recent development of NLP from a new perspective")
the FOLNet neural network model comprising of a plurality of layers;([p. 3] "Theorem 3.2: Under the assumption 3.1, a join-chain network with the ¯H-head and ¯ L-layer self-attention block can express all the predicates in FOET")
encoding code configured to cause the at least one processor to encode a logical inductive bias using the FOLNet neural network model; ([Abstract] "logical reasoning has become increasingly important for a wide range of applications (e.g., natural language processing). Towards this grand objective, we propose a symbolic reasoning architecture that chains many join operators together to model output logical expressions. In particular, we demonstrate that such an ensemble of join chains can express a broad subset of “tree-structured” first-order logical expressions, named FOET [...] we find that the widely used multi-head self attention module in transformer can be understood as a special neural operator that implements the union bound of the join operator in probabilistic predicate space" Zhang's join-chain network is designed specifically around recursive calculation of first-order logic predicates using join operations. Its self-attention blocks are explained as neural approximations of symbolic join operations.)
outputting code configured to cause the at least one processor to output one or more tensors based on the logical inductive bias; ([p. 3] "the tensor Z will the join operation between W(x,y) and P(y) […] the self-attention matrix A learns all the values of the binary predicate W(x,y), and the value tensor V learns all the values of the unary predicate P(y)" [Abstract] "we propose a symbolic reasoning architecture that chains many join operators together to model output logical expressions")
and predicting code configured to cause the at least one processor to predict an outcome using the one or more tensors, ([p. 1] "derive the value of P(x) for a given x" [p. 2] "we find a key operation for the calculation of P(x) and name it as the join operation")
and wherein the encoding code is further configured to cause the at least one processor to: obtain unary representations generated based on a plurality of tokens, and binary representations based on one or more pairs of the plurality of tokens([p. 2] "Each Pmm is a unary predicate and each Wmm is a binary predicate" [p. 3] "Generally speaking, the self-attention matrix A learns all the values of the binary predicate W(x,y), and the value tensor V learns all the values of the unary predicate P(y)" we denote the domain of all the predicates, including all the binary predicates W(x,y) and unary predicates P(y), as {x1,x1,x3,...,xS}, which means x,y ∈ {x1,x2,x3,...,xS}" the tokens are the elements in the set {x1,…,xs})
use, within each layer of the plurality of layers, a first interacting branch of the FOLNet neural network model to update unary representations based on applying a plurality of first neural logic operators on the unary representations and the binary representations, ([p. 2] "join operators will conduct the join operations on the inputs to each layer." [p. 3] "we find that the widely adopted multi-head attention mechanism can be understood as a join operator" [p. 3] "we denote the domain of all the predicates, including all the binary predicates W(x,y) and unary predicates P(y), as {x1,x1,x3,...,xS}, which means x,y ∈ {x1,x2,x3,...,xS}. Then in the multi-head attention mechanism, the core part is the product between the self-attention matrix A and the value tensor V, Z=AV" See FIG. 1 where branches are parallel and appear analogous to FIG. 3 and 4 of the instant Drawings. Zhang teaches that each attention head/join module performs the join operation between binary predicate W(x,y) and unary predicate P(y). In Transformer terms, A is the binary predicate matrix, V is the unary predicate tensor, and Z=AV is the updated join result. Thus, the first interacting branch can be read as the value/unary-output path updated by neural join/self-attention operators using both A and V)
and a second interacting branch to update binary representations based on applying a plurality of second neural logic operators on the unary representations and the binary representations,([p. 3] "the self-attention matrix A learns all the values of the binary predicate W(x,y), and the value tensor V learns all the values of the unary predicate P(y). For each head of attention mechanism in each layer, the self-attention matrix A learns a binary predicate W(x,y)" Zhang teaches that each attention head in each layer learns a binary predicate through the self attention matrix A. Zhang explicitly treats A as learned binary predicate values and Z=AV as the join output.)
wherein the first interacting branch and the second interacting branch are parallel to each other.([p. 1] "Multiple paralleled attention heads" [p. 2] "join operation: P(x) = ∃yW(x,y)P(y) […] there are multiple join operations to conduct in each step, we need to include several join operation modules in each layer […] The aggregation after the join operators will aggregate the inputs from the skip-connection and the results of join operators. This reflects the process that we need to conduct ∧ operations after the join operations in each step" See FIG. 1 where branches appear parallel in view of FIG. 4 of the instant).
However, Zhang does not explicitly teach : at least one memory configured to store program code; and at least one processor configured to read the program code and operate as instructed by the program code, the program code comprising:
pre-training code configured to cause the at least one processor to pre-train a First-Order Logic Network (FOLNet) neural network model on unlabeled texts included in the natural language texts, .
Dai, in the same field of endeavor, teaches : at least one memory configured to store program code; and at least one processor configured to read the program code and operate as instructed by the program code, the program code comprising: ([p. 4] "we can cache as many previous segments as the GPU memory allows, and reuse all of them as the extra context when processing the current segment. Thus, we can cache a predefined length-M old hidden states spanning (possibly) multiple segments, and refer to them as the memory" [p. 7] "we also compare Transformer-XL with baselines under the same GPU memory constraints")
pre-training code configured to cause the at least one processor to pre-train a First-Order Logic Network (FOLNet) neural network model on unlabeled texts included in the natural language texts, ([p. 5] "As a benefit of the inductive bias, a model trained on a memory of some certain length can automatically generalize to a memory several times longer during evaluation" [p. 6] "WikiText-103 is the largest available word-level language modeling benchmark with long-term de pendency. It contains 103M training tokens from 28K articles, with an average length of 3.6K to kens per article, which allows testing the ability of long-term dependency modeling. We set the attention length to 384 during training and 1600 during evaluation [...] we train 18-layer and 24-layer Transformer-XLs with increased model sizes. With the attention length 784 during training and 3,800 during evaluation" Training prior to evaluation interpreted as synonymous with pre-training. WikiText-103 is an unlabeled text database that the model is pre-trained on.).
Zhang as well as Dai are directed towards Transformer-based natural language modeling using self-attention. Therefore, Zhang as well as Dai are analogous art in the same field of endeavor. It would have been obvious before the effective filing date of the claimed invention to combine the teachings of Zhang with the teachings of Dai. Zhang explains what Transformer multi-head attention is doing logically and Dai provides a concrete Transformer language-model apparatus and pretraining context in which that attention mechanism is used. A person of ordinary skill in the art would have understood, in view of Zhang, that Dai's Transformer-XL attention mechanism already implements a neural logical join over unary value tensors and binary attention predicates, thereby encoding a logical inductive bias within Dai's pretrained natural-language Transformer model. Dai provides as additional motivation for combination for using Transformer-XL specifically ([p. 1] “we improve the state-of the-art results of bpc/perplexity to 0.99 on en wiki8, 1.08 on text8, 18.3 on WikiText-103, 21.8 on One Billion Word, and 54.5 on Penn Treebank (without finetuning)”). This motivation for combination also applies to the remaining claims which depend on this combination.
Regarding claim 9, the combination of Zhang and Dai teaches The apparatus according to claim 8, wherein the outputting code is further configured to cause the at least one processor to preprocess the input into one or more tokens to be converted into the one or more tensors.(Dai [p. 1] "we improve the state-of the-art results of bpc/perplexity to 0.99 on en wiki8, 1.08 on text8, 18.3 on WikiText-103, 21.8 on One Billion Word, and 54.5 on Penn Treebank (without finetuning). When trained only on WikiText-103, Transformer-XL manages to generate reasonably coherent, novel text articles with thousands of tokens" [p. 2] "Given a corpus of tokens x =(x1,...,xT), the task of language modeling is to estimate the joint probability P(x)" Dai receives input tokens, segments them into S_T, S_T+1, etc., maps tokens to embeddings and then feeds those embeddings into the transformer).
Regarding claim 14, the combination of Zhang and Dai teaches The apparatus according to claim 8, wherein the pre- training code is further configured to cause the at least one processor to train the FOLNet neural network model to solve one or more downstream tasks.(Dai [p. 8 §4.4] "Trained only on WikiText-103 which is medium-sized, Transformer-XL is already able to generate relatively coherent articles with thousands of tokens without manual cherry picking" Text/article generation interpreted as downstream task).
Regarding claim 15, claim 15 is directed towards a system for performing the method of claim 1. Therefore, the rejection applied to claim 1 also applies to claim 15. Claim 15 also recites additional elements A non-transitory computer-readable storage medium, storing instructions, which, when executed by at least one processor, cause the at least one processor to: (Dai [p. 4] "we can cache as many previous segments as the GPU memory allows, and reuse all of them as the extra context when processing the current segment. Thus, we can cache a predefined length-M old hidden states spanning (possibly) multiple segments, and refer to them as the memory" [p. 7] "we also compare Transformer-XL with baselines under the same GPU memory constraints").
Similarly, regarding claim 16, claim 16 is directed towards a system for performing the method of claim 2. Therefore, the rejection applied to claim 2 also applies to claim 16.
Claims 4-6, 11-13, and 18-20 are rejected under U.S.C. §103 as being unpatentable over the combination of Zhang and Dai and in further view of Dong (“NEURAL LOGIC MACHINES”, 2019).
Regarding claim 4, the combination of Zhang and Dai teaches The method according to claim 1.
However, the combination of Zhang and Dai doesn't explicitly teach, wherein the plurality of first neural logic operators and the plurality of second neural logic operators include learnable Horn clauses.
Dong, in the same field of endeavor, teaches the plurality of first neural logic operators and the plurality of second neural logic operators include learnable Horn clauses([p. 5] "It can be verified that NLMs can realize the forward chaining of a partial set of Horn clauses").
The combination of Zhang and Dai as well as Dong are directed towards machine learning systems for representation learning and reasoning using neural networks. Therefore, the combination of Zhang and Dai as well as Dong are analogous art in the same field of endeavor. It would have been obvious before the effective filing date of the claimed invention to combine the teachings of the combination of Zhang and Dai with the teachings of Dong by using NLM with a Transformer-XL model. Dong provides as additional motivation for combination ([p. 18] ” the input properties or relations can be derived from other neural architectures (e.g., CNNs) […] A convolutional neural network (CNN) is applied to the input extracting multiple features for future reasoning. CNN and NLM can be optimized jointly. This enables reasoning over noisy input”). This motivation for combination also applies to the remaining claims which depend on this combination.
Regarding claim 5, the combination of Zhang, Dai, and Dong teaches The method according to claim 4, wherein the plurality of first neural logic operators and the plurality of second neural logic operators are forward-chained into the FOLNet neural network model.(Dai [p. 5] "Positionwise-Feed-Forward(on_T)").
Regarding claim 6, the combination of Zhang and Dai teaches The method according to claim 1.
However, the combination of Zhang and Dai doesn't explicitly teach, wherein the FOLNet neural network model is a fully differentiable neural architecture..
Dong, in the same field of endeavor, teaches the FOLNet neural network model is a fully differentiable neural architecture.([p. 10] "being fully differentiable, NLMs can be plugged into existing convolutional or recurrent neural architectures for logic reasoning").
The combination of Zhang and Dai as well as Dong are directed towards machine learning systems for representation learning and reasoning using neural networks. Therefore, the combination of Zhang and Dai as well as Dong are analogous art in the same field of endeavor. It would have been obvious before the effective filing date of the claimed invention to combine the teachings of the combination of Zhang and Dai with the teachings of Dong by using NLM with a Transformer-XL model. Dong provides as additional motivation for combination ([p. 18] ” the input properties or relations can be derived from other neural architectures (e.g., CNNs) […] A convolutional neural network (CNN) is applied to the input extracting multiple features for future reasoning. CNN and NLM can be optimized jointly. This enables reasoning over noisy input”). This motivation for combination also applies to the remaining claims which depend on this combination.
Regarding claim 11, the combination of Zhang and Dai teaches The apparatus according to claim 8.
However, the combination of Zhang and Dai doesn't explicitly teach, wherein the plurality of first neural logic operators and the plurality of second neural logic operators include learnable Horn clauses.
Dong, in the same field of endeavor, teaches the plurality of first neural logic operators and the plurality of second neural logic operators include learnable Horn clauses([p. 5] "It can be verified that NLMs can realize the forward chaining of a partial set of Horn clauses").
The combination of Zhang and Dai as well as Dong are directed towards machine learning systems for representation learning and reasoning using neural networks. Therefore, the combination of Zhang and Dai as well as Dong are analogous art in the same field of endeavor. It would have been obvious before the effective filing date of the claimed invention to combine the teachings of the combination of Zhang and Dai with the teachings of Dong by using NLM with a Transformer-XL model. Dong provides as additional motivation for combination ([p. 18] ” the input properties or relations can be derived from other neural architectures (e.g., CNNs) […] A convolutional neural network (CNN) is applied to the input extracting multiple features for future reasoning. CNN and NLM can be optimized jointly. This enables reasoning over noisy input”). This motivation for combination also applies to the remaining claims which depend on this combination.
Regarding claim 12, the combination of Zhang, Dai, and Dong teaches The apparatus according to claim 11, wherein the plurality of first neural logic operators and the plurality of second neural logic operators are forward-chained into the FOLNet neural network model(Dai [p. 5] "Positionwise-Feed-Forward(on_T)").
Regarding claim 13, the combination of Zhang and Dai teaches The apparatus according to claim 8.
However, the combination of Zhang and Dai doesn't explicitly teach, wherein the FOLNet neural network model is a fully differentiable neural architecture.
Dong, in the same field of endeavor, teaches the FOLNet neural network model is a fully differentiable neural architecture ([p. 10] "being fully differentiable, NLMs can be plugged into existing convolutional or recurrent neural architectures for logic reasoning").
The combination of Zhang and Dai as well as Dong are directed towards machine learning systems for representation learning and reasoning using neural networks. Therefore, the combination of Zhang and Dai as well as Dong are analogous art in the same field of endeavor. It would have been obvious before the effective filing date of the claimed invention to combine the teachings of the combination of Zhang and Dai with the teachings of Dong by using NLM with a Transformer-XL model. Dong provides as additional motivation for combination ([p. 18] ” the input properties or relations can be derived from other neural architectures (e.g., CNNs) […] A convolutional neural network (CNN) is applied to the input extracting multiple features for future reasoning. CNN and NLM can be optimized jointly. This enables reasoning over noisy input”). This motivation for combination also applies to the remaining claims which depend on this combination.
Regarding claims 18-20, claims 18-20 are directed towards a system for performing the method of claims 4-6. Therefore, the rejections applied to claims 4-6 also apply to claims 18-20.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Riegel (US 20210365817 A1) is directed towards a first order neural network for performing unary and binary logic on natural language text.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SIDNEY VINCENT BOSTWICK whose telephone number is (571)272-4720. The examiner can normally be reached M-F 7:30am-5:00pm EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Miranda Huang can be reached on (571)270-7092. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/SIDNEY VINCENT BOSTWICK/Examiner, Art Unit 2124
Prosecution Insights