Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
DETAILED ACTION
This office action is in response to the claims filed on 03/18/2026.
Claims 1-20 are presented for examination.
Response to Arguments
In reference to applicant’s argument regrading rejections under 35 U.S.C. § 103:
The applicant’s Argument on page 3 in the remark filed on 03/18/2026:
Thus, the first encoder RNN operates on the words of an input source language text segment, and the second decoder RNN operates on a sequence of hidden states generated by the first encoder RNN. The Examiner's 'note' states that "each step of the recurrent neural network is generating the different input sequence" in relation to Bertoldi. The relevance of this statement is unclear. Claim 1 does not require each step of the RNN to generate a different input sequence. Claim 1 requires each step of the RNN to operate on a different input of the same sequence ("the" sequence of inputs). Thus, the statement of the Office action indicates that the claim language may not have been properly read or interpreted; to the extent that Bertoldi generates a different input sequence in each step of a recurrent neural network, Bertoldi clearly does not correspond to what is recited in claim 1.
Applicant also notes that "each step of the recurrent neural network" in claim 1 is each step of the same recurrent neural network ("the" recurrent neural network) - not steps of two different RNNs.
Examiner’s Response:
Examiner respectfully disagrees to applicant’s argument since the applicant’s argument is not persuasive, as Bertoldi clearly teaches that each step of the recurrent neural network being for operation on a different input of the sequence, as it can be seen at Par.0101-0103], Par.0147], “first, a recurrent neural network encodes the input source language text segment word by word into a sequence of hidden states; then, another recurrent neural network decodes the source hidden sequence into the estimated translation. Both the encoder and decoder networks are implemented with gated recurrent units. In particular, the decoder network operates like a language model: it predicts the next target word from the last target word, the last hidden state of the decoder, and a convex combination of the encoder hidden states.” Examiner’s note, the input is generated from one state of the recurrent neural network to other state of the recurrent neural network, such as the neural network encoder encode the input language text statement into the hidden state and the neural network decoder to decode the input from the hidden state to the estimated translation. Therefore, the input sequence is inputted from one state and become the output to be inputted to another state of the neural network, that is corresponding to each step of the recurrent neural network being for operation on a different input of the sequence. Therefore, the applicant’s argument is not persuasive, the rejection is till maintained.
Applicant’s argument on page 5 in the remark filed on 03/18/2026:
Bertoldi does not disclose passing the state output of one iteration of said larger neural network, having operated on a predetermined plurality of inputs of the sequence, to a subsequent iteration of the same neural network as the state input, said subsequent iteration for operation on a next predetermined plurality of inputs of the sequence. Thus, Bertoldi does not disclose "iteratively applying the derivative neural network to the sequence of inputs by: implementing a sequence of instances of the derivative neural network in the hardware" and "providing the one or more state outputs from each instance of the derivative neural network at the hardware as the one or more state inputs to a subsequent instance of the derivative neural network at the hardware".
In relation to the claim feature "wherein the derivative neural network is a feed- forward neural network", the Office action relies on paragraph [0147] of Bertoldi, which states that the weights of the decoder network are computed through a feed-forward network called an attention model. However, the mere fact that the weights of the decoder network are computed using a feed-forward network does not make the decoder network itself a feed-forward neural network. On the contrary, the second decoder network is specifically a "recurrent neural network", not a feed-forward neural network (see [00147]). As stated in [00147] of Bertoldi, both the first encoder network and the second encoder network are implemented with gated recurrent units. Thus, Bertoldi does not disclose the claim feature "wherein the derivative neural network is a feed-forward neural network".
Examiner’s Response:
Examiner’s reposefully disagrees to applicant’s argument since Bertoldi clearly discloses that "iteratively applying the derivative neural network to the sequence of inputs by: implementing a sequence of instances of the derivative neural network in the hardware" and "providing the one or more state outputs from each instance of the derivative neural network at the hardware as the one or more state inputs to a subsequent instance of the derivative neural network at the hardware, as it can be seen at Bertoldi, [Abstract, “System and method for providing a computer-assisted translation from a source language to a target language, using a generic NMT model and a translation memory. An input text segment is received, and input context information is received. The input context information is indicative of circumstances in which the input text segment is used, the input text segment being in the source language.”, [Par.0101-0103], “The system comprises a server 102, connected to user devices 104, 106, 108, 110, 112 by a networked internet connection 114. Each user device 104, 106, 108, 110, 112 has a CAT tool plugin. The server 102 is a general-purpose computer system connected to the internet…” And [par.0147], “The NMT model uses a two-step process to calculate an estimated translation into the target language from an input source language text segment: first, a recurrent neural network encodes the input source language text segment word by word into a sequence of hidden states; then, another recurrent neural network decodes the source hidden sequence into the estimated translation. Both the encoder and decoder networks are implemented with gated recurrent units. In particular, the decoder network operates like a language model: it predicts the next target word from the last target word, the last hidden state of the decoder, and a convex combination of the encoder hidden states. The weights of this convex combination are dynamically computed through a simple feed-forward network, called attention model.” Examiner’s note, the method for generating the input language text into the estimated translation by generating on the neural network is implemented on the computer system (hardware). The recurrent neural network is generated on the generic computer, and each of the recurrent neural network (derivative neural network) iteratively generates the sequence of input, for example, the first recurrent neural network encodes the input into word by word sequence at the hidden state and the second recurrent neural network decodes the sequence hidden state (output from the first recurrent neural network becomes the input is inputted into the decoder) to output the estimated translation.) ,
wherein the derivative neural network is a feed-forward neural network (Bertoldi , [0147] ... Both the encoder and decoder networks are implemented with gated recurrent units. In particular, the decoder network operates like a language model: it predicts the next target word from the last target word, the last hidden state of the decoder, and a convex combination of the encoder hidden states. The weights of this convex combination are dynamically computed through a simple feed-forward network, called attention model.” Examiner’s note, the derivative neural network include encoder and decoder are dynamically computed through a simple feed-forward network).
Therefore, the applicant’s argument is not persuasive, the rejection is still maintained.
Applicant’s Argument on pages 6-7 in the remark filed on 03/18/2026:
Bertoldi and Sherstinsky
As previously explained, Sherstinsky simply discusses the mathematical theory behind RNNs and LSTMs. The Office action relies on Sherstinsky section 3 and Fig. 2 as allegedly disclosing the acknowledged missing features of Bertoldi relating to "receiving a representation of the RNN; transforming the representation of the RNN into a derivative neural network for operation over a predetermined plurality of inputs of the sequence of inputs, the derivative neural network having one or more state inputs and one or more state outputs and being equivalent to the RNN over a predetermined plurality of steps of the RNN', and "wherein each instance of the derivative neural network operates on a respective predetermined plurality of inputs of the sequence of inputs". This is incorrect….
Lee
The Office action relies on paragraph [0124] of Lee as allegedly disclosing "providing the one or more state outputs from each instance of the derivative neural network at the hardware as the one or more state inputs to a subsequent instance of the derivative neural network at the hardware so as to operate the RNN over a sequence of inputs longer than the predetermined plurality of inputs," which is acknowledged to be missing from both Bertoldi and Sherstinsky. Applicant respectfully disagrees.
Examiner’s Response:
Examiner respectfully reminds the applicant that Sherstinsky is only brought to cure the specific deficiencies of Bertoldi regarding their respective limitation of the claim recites, such as Bertoldi teaches implementing in hardware a recurrent neural network (RNN) for operation on a sequence of inputs, each step of the recurrent neural network being for operation on a different input of the sequence, the hardware being adapted to perform feed-forward neural networks, And iteratively applying the derivative neural network to the sequence of inputs by: implementing a sequence of instances of the derivative neural network in the hardware, and providing the one or more state outputs from each instance of the derivative neural network at the hardware as the one or more state inputs to a subsequent instance of the derivative neural network at the hardware so as to operate the RNN over a sequence of inputs, wherein the derivative neural network is a feed-forward neural network. Furthermore, Sherstinsky teaches receiving a representation of the RNN, transforming the representation of the RNN into a derivative neural network for operation over a predetermined plurality of inputs of the sequence of inputs, the derivative neural network having one or more state inputs and one or more state outputs and being equivalent to the RNN over a predetermined plurality of steps of the RNN, wherein each instance of the derivative neural network operates on a respective predetermined plurality of inputs of the sequence of inputs,
Additionally, Lee is also brought in to cure the specific deficiencies of Bertoldi and Sherstinsky regarding their respective claim limitations, as Lee teaches “operate the RNN over a sequence of inputs longer than the predetermined plurality of inputs”.
Examiner still understands that the combination of the teaching of Bertoldi , Sherstinsky and Lee are still teach each of the limitations of the claim 1, since Bertoldi, Sherstinsky and Lee are analogous in arts because they have the same field of endeavor of training data by using the neural network.
Therefore, the applicant’s argument is not persuasive, the rejection is still maintained.
In reference to applicant’s argument regrading rejection of the Obviousness-type Double Patenting:
Claims 1, 5, 10 and 19 continue to be provisionally rejected under the obviousness- type double patenting doctrine as being unpatentable over claims 6, 7, 10, 13 and 19 of copending Application Serial No. 17/368,302. Applicant respectfully maintains that the rejection is in error as the Office action has not properly analyzed the claims, and fails to consider the claims of the '302 application as currently pending. For the record Applicant maintains all arguments previously presented. Notwithstanding this, solely for the purpose of removing this improper ground of rejection so that prosecution may be concluded, a terminal disclaimer is being filed herewith, with traverse and without acquiescence. Withdrawal of this ground of rejection is requested.
Examiner’s Response:
Examiner respectfully disagrees to applicant’s argument because the applicant’s argument is not persuasive, the applicant’s argument does not provide the detail of why the current double patenting rejection is in error. Additionally, examiner clearly states that “The claims 1-7, 10-12, 14-20 are provisionally rejected on the ground of non-statutory double patenting as being unpatentable over claim 1, 7, 8, 10, 13, 14, 15, 17, 18, 19, 20, 21of co-pending Application No. 17368,302 in view of Sherstinsky (Fundamentals of Recurrent Neural Networks (RNN) and Long Short-Term Memory (LSTM) network), hereinafter referred to as Sherstinsky), as follows”. Therefore, the applicant’s argument is not persuasive, the rejection is still maintained.
Double Patenting
The non-statutory double patenting rejection is based on a judicially created doctrine grounded in public policy (a policy reflected in the statute) so as to prevent the unjustified or improper timewise extension of the “right to exclude” granted by a patent and to prevent possible harassment by multiple assignees. A non-statutory double patenting rejection is appropriate where the conflicting claims are not identical, but at least one examined application claim is not patentably distinct from the reference claim(s) because the examined application claim is either anticipated by, or would have been obvious over, the reference claim(s). See, e.g., In re Berg, 140 F.3d 1428, 46 USPQ2d 1226 (Fed. Cir. 1998); In re Goodman, 11 F.3d 1046, 29 USPQ2d 2010 (Fed. Cir. 1993); In re Longi, 759 F.2d 887, 225 USPQ 645 (Fed. Cir. 1985); In re Van Ornum, 686 F.2d 937, 214 USPQ 761 (CCPA 1982); In re Vogel, 422 F.2d 438, 164 USPQ 619 (CCPA 1970); In re Thorington, 418 F.2d 528, 163 USPQ 644 (CCPA 1969).
A timely filed terminal disclaimer in compliance with 37 CFR 1.321(c) or 1.321(d) may be used to overcome an actual or provisional rejection based on nonstatutory double patenting provided the reference application or patent either is shown to be commonly owned with the examined application, or claims an invention made as a result of activities undertaken within the scope of a joint research agreement. See MPEP § 717.02 for applications subject to examination under the first inventor to file provisions of the AIA as explained in MPEP § 2159. See MPEP § 2146 et seq. for applications not subject to examination under the first inventor to file provisions of the AIA . A terminal disclaimer must be signed in compliance with 37 CFR 1.321(b).
The USPTO Internet website contains terminal disclaimer forms which may be used. Please visit www.uspto.gov/patent/patents-forms. The filing date of the application in which the form is filed determines what form (e.g., PTO/SB/25, PTO/SB/26, PTO/AIA /25, or PTO/AIA /26) should be used. A web-based eTerminal Disclaimer may be filled out completely online using web-screens. An eTerminal Disclaimer that meets all requirements is auto-processed and approved immediately upon submission. For more information about eTerminal Disclaimers, refer to www.uspto.gov/patents/process/file/efs/guidance/eTD-info-I.jsp.
The claims 1-7, 10-12, 14-20 are provisionally rejected on the ground of non-statutory double patenting as being unpatentable over claim 1, 7, 8, 10, 13, 14, 15, 17, 18, 19, 20, 21of co-pending Application No. 17368,302 in view of Sherstinsky (Fundamentals of Recurrent Neural Networks (RNN) and Long Short-Term Memory (LSTM) network), hereinafter referred to as Sherstinsky), as follows.
This is a provisional non-statutory double patenting rejection.
Instant application 17368384
Co-pending application 17368302
Claims 1. 20
A method of implementing in hardware a recurrent neural network (RNN) for operation on a sequence of inputs, each step of the recurrent neural network being for operation on a different input of the sequence, the hardware being adapted to perform feed-forward neural networks, the method comprising:
receiving a representation of the RNN;
transforming the representation of the RNN into a derivative neural network for operation over a predetermined plurality of inputs of the sequence of inputs,
the derivative neural network having one or more state inputs and one or more state outputs and being equivalent to the RNN over a predetermined plurality of steps of the RNN;
and iteratively applying the derivative neural network to the sequence of inputs by:
implementing a sequence of instances of the derivative neural network in the hardware,
wherein each instance of the derivative neural network operates on a respective predetermined plurality of inputs of the sequence of inputs;
and providing the one or more state outputs from each instance of the derivative neural network at the hardware as the one or more state inputs to a subsequent instance of the derivative neural network at the hardware so as to operate the RNN over a sequence of inputs longer than the predetermined plurality of inputs;
wherein the derivative neural network is a feed-forward neural network.
Claims (1+10 + 21)
The claim 1:
A computer-implemented method of configuring hardware adapted to perform non-recurrent neural networks to implement a recurrent neural network (RNN), the method comprising:
receiving a representation of the RNN;
implementing the representation of the RNN as a test neural network for operation on a sequence of test inputs,
wherein implementing the representation of the RNN as a test neural network
comprises transforming the representation of the RNN into a test neural network for operation over a first predetermined plurality of steps by unrolling the RNN over the first predetermined plurality of steps so as to form the test neural network, the test neural network being equivalent to the RNN over the first predetermined plurality of steps;
operating the test neural network on the sequence of test inputs for the first predetermined plurality of steps, each step of the test neural network comprising an instance of the two or more values of the RNN, and collecting statistics for provision to a number format selection algorithm;
applying a number format selection algorithm to the statistics so as to derive a common number format for the first predetermined plurality of instances of the two or more values of the RNN;
and configuring the hardware adapted to perform non-recurrent neural networks to implement the RNN over a sequence of inputs as a derivative neural network using the common number format as the number format for each instance of the respective two or more values in the derivative neural network, wherein the derivative neural network represents the RNN unrolled over a second predetermined plurality of steps, the second predetermined plurality of steps being different to the first predetermined plurality of steps.
CLAIM 10:
The method of claim 9, wherein the implementation of the RNN on the hardware is formed by:transforming the representation of the RNN into a derivative neural network for operation over a second predetermined plurality of inputs of the sequence of inputs, the derivative neural network having one or more state inputs and one or more state outputs and being equivalent to the RNN over the second predetermined plurality of steps of the RNN; anditeratively applying the derivative neural network to the sequence of inputs by:implementing a sequence of instances of the derivative neural network in hardware; and providing the one or more state outputs from each instance of the derivative neural network at the hardware as the one or more state inputs to a subsequent instance of the derivative neural network at the hardware so as to operate the RNN over a sequence of inputs longer than the second predetermined plurality of inputs..
The claim 21:
A non-transitory computer readable storage medium having stored thereon computer readable instructions that, when executed at a computer system, cause the computer system to perform a computer-implemented method of configuring hardware adapted to perform non-recurrent neural networks to implement a recurrent neural network (RNN), the method comprising:receiving a representation of the RNN;implementing the representation of the RNN as a test neural network for operationon a sequence of test inputs, wherein implementing the representation of the RNN as a test neural network comprises transforming the representation of the RNN into a test neural network for operation over a first predetermined plurality of steps by unrolling the RNN over the first predetermined plurality of steps so as to form the test neural network, the test neural network being equivalent to the RNN over the first predetermined plurality of steps;operating the test neural network on the sequence of test inputs for the first predetermined plurality of steps, each step of the test neural network comprising an instance of the two or more values of the RNN, and collecting statistics for provision to a number format selection algorithm;applying a number format selection algorithm to the statistics so as to derive a common number format for the first predetermined plurality of instances of the two or more values of the RNN; andconfiguring the hardware adapted to perform non-recurrent neural networks to implement the RNN over a sequence of inputs as a derivative neural network using the common number format as the number format for each instance of the respective two or more values in the derivative neural network, wherein the derivative neural network represents the RNN unrolled over a second predetermined plurality of steps, the second predetermined plurality of steps being different to the first predetermined plurality of steps.
The claim 2:
The method of claim 1, wherein the predetermined plurality of steps is equal in number to the predetermined plurality of inputs.
The claim 8:
The method of claim 1, wherein the test neural network is configured to operate on a predefined plurality of test inputs, the predefined plurality of test inputs being equal in number to the first predetermined plurality of steps.
The claim 3:
wherein the one or more state outputs from each instance of the derivative neural network are provided as the one or more state inputs to the subsequent instance of the derivative neural network in the sequence of instances of the derivative neural network.
The claim 10:
The method of claim 9, wherein the implementation of the RNN on the hardware is formed by:transforming the representation of the RNN into a derivative neural network for operation over a second predetermined plurality of inputs of the sequence of inputs, the derivative neural network having one or more state inputs and one or more state outputs and being equivalent to the RNN over the second predetermined plurality of steps of the RNN; anditeratively applying the derivative neural network to the sequence of inputs by:implementing a sequence of instances of the derivative neural network in hardware; andproviding the one or more state outputs from each instance of the derivative neural network at the hardware as the one or more state inputs to a subsequent instance of the derivative neural network at the hardware so as to operate the RNN over a sequence of inputs longer than the second predetermined plurality of input.
The claim 4:
The method of claim 1, wherein the implementing a sequence of instances of the derivative neural network comprises implementing an instance of the derivative neural network and, on completion of that instance, causing the next instance of the derivative neural network in the sequence to be implemented in hardware.
The claim 10:
The method of claim 9, wherein the implementation of the RNN on the hardware is formed by:transforming the representation of the RNN into a derivative neural network for operation over a second predetermined plurality of inputs of the sequence of inputs, the derivative neural network having one or more state inputs and one or more state outputs and being equivalent to the RNN over the second predetermined plurality of steps of the RNN; anditeratively applying the derivative neural network to the sequence of inputs by:implementing a sequence of instances of the derivative neural network in hardware; andproviding the one or more state outputs from each instance of the derivative neural network at the hardware as the one or more state inputs to a subsequent instance of the derivative neural network at the hardware so as to operate the RNN over a sequence of inputs longer than the second predetermined plurality of
The claim 5:
The method of claim 1, wherein the transforming comprises unrolling the recurrent neural network over the predetermined plurality of steps so as to form the derivative neural network for operation over the predetermined plurality of inputs of the sequence of inputs.
The claim 7 :
The method of claim 6, wherein the transforming comprises unrolling the RNN over the first predetermined plurality of steps so as to form the test neural network.
The claim 7:
The method of claim 1, wherein the hardware and its control logic are incapable of executing the received representation of the RNN.
The claim 18:
The data processing system of claim 17, further comprising the hardware accelerator, wherein the control logic is further configured to cause the representation of the RNN to be implemented at the hardware accelerator as the derivative neural network using the common number format for the two or more values of the RNN.
The claim 8:
The method of claim 1, wherein the hardware and its control logic are incapable of executing dynamic neural networks.
However, the co-pending Application No. 17368,302 does not teach the claim 3 of the instant application.
On the other hand, Sherstinsky teaches the hardware and its control logic are incapable of executing dynamic neural networks, (Section 7, Pg. 22, Column 2, Lines 46-49,).
It would have been obvious to a person of ordinary skill in the arts at the time of the applicant's invention to modify the teachings of the co-pending Application No. 17368,302 by incorporating the hardware and its control logic are incapable of executing dynamic neural networks, as taught by Sherstinsky for the purpose of outputting the node of the warping function, (Section 7, Pg. 22, Column 2, Lines 46-49,).
The claim 10:
The method of claim 1, wherein the RNN comprises one or more cells, each cell arranged to receive a cell state input generated at a preceding step, and the transforming the representation of the RNN further comprises, at each cell:
identifying non-causal operations which are for performance without dependence on the cell state input; and
in the derivative neural network, grouping together at least some of the non-causal operations at a plurality of instances of the cell over at least some of the predetermined plurality of steps
for processing in parallel at the hardware.
The claim 13:
The method of claim 10, wherein the RNN comprises one or more cells, each cell arranged to receive a cell state input generated at a preceding step, and the transforming the RNN into the test neural network further comprises, at each cell:
identifying non-causal operations which are for performance without dependence on the state input generated at a preceding step; and
in the derivative neural network, grouping together at least some of the non-causal operations at a plurality of instances of the cell over at least some of the predetermined plurality of steps for processing in parallel at the hardware.
Claim 11:
The method of claim 10, wherein the cell comprises causal operations which are for performance in dependence on the cell state input.
The claim 14:
wherein the cell comprises causal operations which are for performance in dependence on the cell state input and the transforming the RNN further comprises configuring the test neural network such that the result of the non-causal operations performed at the cell in respect of an input from the sequence of test inputs is combined with the causal operations performed at the cell in respect of that same test input.
Claim 12 :
The method of claim 10, wherein at least part of the cell state input is generated at the preceding instance of the cell at the preceding step.
The claim 13:
wherein the RNN comprises one or more cells, each cell arranged to receive a cell state input generated at a preceding step, and the transforming the RNN into the test neural network further comprises, at each cell:identifying non-causal operations which are for performance without dependence on the state input generated at a preceding step; andin the derivative neural network, grouping together at least some of the non-causaloperations at a plurality of instances of the cell over at least some of the predetermined plurality of steps for processing in parallel at the hardware.
Claim 13:
wherein the grouping together comprises combining the at least some non-causal operations for performance as a single convolution operation for the plurality of instances of the cell in the derivative neural network.
However, the co-pending Application No. 17368,302 does not teach the claim 13 of the instant application.
On the other hand, Sherstinsky teaches the wherein the grouping together comprises combining the at least some non-causal operations for performance as a single convolution operation for the plurality of instances of the cell in the derivative neural network, ((Section 7.1, Pg. 23, Column 1, Lines 11-18).
It would have been obvious to a person of ordinary skill in the arts at the time of the applicant's invention to modify the teachings of the co-pending Application No. 17368,302 by incorporating the the grouping together comprises combining the at least some non-causal operations for performance as a single convolution operation for the plurality of instances of the cell in the derivative neural network., as taught by Sherstinsky for the purpose of a single input sample,
x
→
n
,
will be replaced by the convolutions of the context, ((Section 7.1, Pg. 23, Column 1, Lines 11-).
Claim 14:
The method of claim 10, wherein the transforming the representation of the RNN further comprises splitting the at least some of the non-causal operations from the causal operations.
The claim 15:
The method of claim 13, wherein the two or more values are used in the non- causal operations and the RNN comprises two or more other values which are used in the causal operations, and the applying the number format selection algorithm to the statistics is performed so as to independently derive the common number format for the two or more values of the RNN and a second common number format for the two or more other values of the RNN.
Claim 15:
The method of claim 11, wherein the implementing a sequence of instances of the derivative neural network in hardware comprises, for each instance, causing the hardware to process one or more of the groups of non-causal operations in parallel.
Claim 13:
wherein the RNN comprises one or more cells, each cell arranged to receive a cell state input generated at a preceding step, and the transforming the RNN into the test neural network further comprises, at each cell:identifying non-causal operations which are for performance without dependence on the state input generated at a preceding step; andin the derivative neural network, grouping together at least some of the non-causal operations at a plurality of instances of the cell over at least some of the predetermined plurality of steps for processing in parallel at the hardware.
Claim 16:
wherein the hardware comprises an accelerator having a plurality of processing elements for executing a neural network
and each group of non-causal operations is processed in parallel over the at least some of the plurality of processing elements.
Claims (13 + 17):
Claim 17:
A data processing system for configuring a hardware accelerator adapted to perform non-recurrent neural networks to implement a recurrent neural network (RNN), the data processing system comprising:a processor;a transformation unit configured to receive a representation of the RNN and transform the representation of the RNN into a test neural network for operation over a first predetermined plurality of steps, wherein the transformation unit is configured to unroll the RNN over the first predetermined plurality of steps so as to form the test neural network, the test neural network being equivalent to the RNN over the first predetermined plurality of steps;control logic configured to implement the representation of the RNN at the processoras the test neural network for operation on a sequence of test inputs, each step of the test neural network comprising an instance of the two or more values of the RNN; anda format selection unit configured to cause the processor to operate the test neural network on the sequence of test inputs for the first predetermined plurality of steps and collect statistics for provision to a number format selection algorithm,wherein the format selection unit is configured to apply the number format selection algorithm to the statistics so as to derive a common number format for the first predetermined plurality of instances of the two or more values of the RNN;wherein the control logic is configured to configure the hardware accelerator adapted to perform non-recurrent neural networks to implement the RNN over a sequence of inputs as a derivative neural network using the common number format as the number format for each instance of the respective two or more values in the derivative neural network, wherein the derivative neural network represents the RNN unrolled over a second predetermined plurality of steps, the second predetermined plurality of steps being different to the first predetermined plurality of steps.
Claim 13:
wherein the RNN comprises one or more cells, each cell arranged to receive a cell state input generated at a preceding step, and the transforming the RNN into the test neural network further comprises, at each cell:identifying non-causal operations which are for performance without dependence on the state input generated at a preceding step; andin the derivative neural network, grouping together at least some of the non-causal operations at a plurality of instances of the cell over at least some of the predetermined plurality of steps for processing in parallel at the hardware.
Claim 13 :
And in the derivative neural network, grouping together at least some of the non-causal operations at a plurality of instances of the cell over at least some of the predetermined plurality of steps for processing in parallel at the hardware.
Claim 17:
, wherein the transforming the RNN further comprises configuring the derivative neural network such that the result of the non-causal operations performed at an instance of the cell is combined with the causal operation performed in respect of that same instance.
The claim 14:
The method of claim 13, wherein the cell comprises causal operations which are for performance in dependence on the cell state input and the transforming the RNN further comprises configuring the test neural network such that the result of the non-causal operations performed at the cell in respect of an input from the sequence of test inputs is combined with the causal operations performed at the cell in respect of that same test input.
Claim 18:
wherein the recurrent neural network comprises a plurality of cells.
Claim 13 :
The method of claim 10, wherein the RNN comprises one or more cells, each cell arranged to receive a cell state input generated at a preceding step, and the transforming the RNN into the test neural network further comprises,
Claim 19.:
A data processing system for implementing a recurrent neural network (RNN) for operation on a sequence of inputs, the system comprising:
a transformation unit configured to receive a representation of the RNN and transform
the representation of the RNN into a derivative neural network for operation over a predetermined plurality of inputs of the sequence of inputs, the derivative neural network having one or more state inputs and one or more state outputs and being equivalent to the RNN over a predetermined plurality of steps of the RNN;
a hardware accelerator for processing neural networks; and
iteration logic configured to iteratively apply the derivative neural network to the
sequence of inputs by:
causing a sequence of instances of the derivative neural network to be
implemented at the hardware accelerator; and
providing the one or more state outputs from each representation of the
derivative neural network at the hardware accelerator as the one or more state inputs
to a subsequent representation of the derivative neural network at the hardware
accelerator so as to cause the hardware accelerator to operate the RNN over a
sequence of inputs longer than the predetermined plurality of inputs.
Claim 19:
The data processing system of claim 18, further comprising:
a transformation unit configured to transform the representation of the RNN derivative neural network for operation over a predetermined plurality of inputs of a sequence of inputs, the derivative neural network having one or more state inputs and one or more state outputs and being equivalent to the RNN over a predetermined plurality of steps of the RNN;
iteration logic configured to, subsequent to the operation of the test neural network at the processor, iteratively apply the derivative neural network to the sequence of inputs by:
causing a sequence of instances of the derivative neural network to be implemented at the hardware accelerator; and
providing the one or more state outputs from each representation of the derivative neural network at the hardware accelerator as the one or more state inputs to a subsequent representation of the derivative neural network at the hardware accelerator so as to cause the hardware accelerator to operate the RNN over a sequence of inputs longer than the predetermined plurality of inputs.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1-15, 17, 18, 20 are rejected under 35 U.S.C. 103 as being unpatentable over Bertoldi et al. (PUB. No. US.20200073947-hereinafter Bertoldi) in view of Sherstinsky (Fundamentals of Recurrent Neural Networks (RNN) and Long Short-Term Memory (LSTM) network), hereinafter referred to as Sherstinsky, and further in view of Lee, (PUB. No: US. 20190172466-hereinafter, Lee).
Regarding claim 1, Bertoldi teaches a method of implementing in hardware a recurrent neural network (RNN) for operation on a sequence of inputs, each step of the recurrent neural network being for operation on a different input of the sequence, the hardware being adapted to perform feed-forward neural networks (Bertoldi, [Par.0101-0103], “The system comprises a server 102, connected to user devices 104, 106, 108, 110, 112 by a networked internet connection 114. Each user device 104, 106, 108, 110, 112 has a CAT tool plugin. The server 102 is a general-purpose computer system connected to the internet… The input source language text segments and corresponding input context information metadata are then sent to server 102 over internet connection 114. They may be sent in sequential order (i.e. starting with the first input source language text segment to appear in the input source language text document, and the corresponding input context information metadata), if the user selects to perform translation in a sequential order.” An [Par.0147], “The NMT model uses a two-step process to calculate an estimated translation into the target language from an input source language text segment: first, a recurrent neural network encodes the input source language text segment word by word into a sequence of hidden states; then, another recurrent neural network decodes the source hidden sequence into the estimated translation. Both the encoder and decoder networks are implemented with gated recurrent units. In particular, the decoder network operates like a language model: it predicts the next target word from the last target word, the last hidden state of the decoder, and a convex combination of the encoder hidden states. The weights of this convex combination are dynamically computed through a simple feed-forward network, called attention model.” Examiner’s note, the server/generic computer system translates the input text segments by encoding the input text segment into word by word using the recurrent neural network, such as first, a recurrent neural network encodes the input source language text segment word by word into a sequence of hidden states; then, another recurrent neural network decodes the source hidden sequence into the estimated translation. The input is generated from one state of the recurrent neural network to other state of the recurrent neural network, such as the neural network encoder encode the input language text statement into the hidden state and the neural network decoder to decode the input from the hidden state to the estimated translation. Therefore, the input sequence is inputted from one state and become the output to be inputted to another state of the neural network, that is corresponding to each step of the recurrent neural network being for operation on a different input of the sequence.),
And iteratively applying the derivative neural network to the sequence of inputs by: implementing a sequence of instances of the derivative neural network in the hardware (Bertoldi, [Abstract], “System and method for providing a computer-assisted translation from a source language to a target language, using a generic NMT model and a translation memory. An input text segment is received, and input context information is received. The input context information is indicative of circumstances in which the input text segment is used, the input text segment being in the source language.” And [Par.0101-0103], “The system comprises a server 102, connected to user devices 104, 106, 108, 110, 112 by a networked internet connection 114. Each user device 104, 106, 108, 110, 112 has a CAT tool plugin. The server 102 is a general-purpose computer system connected to the internet… The input source language text segments and corresponding input context information metadata are then sent to server 102 over internet connection 114. They may be sent in sequential order (i.e. starting with the first input source language text segment to appear in the input source language text document, and the corresponding input context information metadata), if the user selects to perform translation in a sequential order.” [par.0147], “The NMT model uses a two-step process to calculate an estimated translation into the target language from an input source language text segment: first, a recurrent neural network encodes the input source language text segment word by word into a sequence of hidden states; then, another recurrent neural network decodes the source hidden sequence into the estimated translation. Both the encoder and decoder networks are implemented with gated recurrent units. In particular, the decoder network operates like a language model: it predicts the next target word from the last target word, the last hidden state of the decoder, and a convex combination of the encoder hidden states. The weights of this convex combination are dynamically computed through a simple feed-forward network, called attention model.” Examiner’s note, the method for generating the input language text into the estimated translation by generating on the neural network is implemented on the computer system (hardware). The recurrent neural network is generated on the generic computer, and each of the recurrent neural network (derivative neural network) iteratively generates the sequence of input, for example, the first recurrent neural network encodes the input into word by word sequence at the hidden state and the second recurrent neural network decodes the sequence hidden state (output from the first recurrent neural network become the input is inputted into the decoder) to output the estimated translation.) ,
And providing the one or more state outputs from each instance of the derivative neural network at the hardware as the one or more state inputs to a subsequent instance of the derivative neural network at the hardware so as to operate the RNN over a sequence of inputs (Bertoldi, [Par.0101-0103], “The system comprises a server 102, connected to user devices 104, 106, 108, 110, 112 by a networked internet connection 114. Each user device 104, 106, 108, 110, 112 has a CAT tool plugin. The server 102 is a general-purpose computer system connected to the internet… The input source language text segments and corresponding input context information metadata are then sent to server 102 over internet connection 114. They may be sent in sequential order (i.e. starting with the first input source language text segment to appear in the input source language text document, and the corresponding input context information metadata), if the user selects to perform translation in a sequential order.” [0147] The NMT model uses a two-step process to calculate an estimated translation into the target language from an input source language text segment: first, a recurrent neural network encodes the input source language text segment word by word into a sequence of hidden states; then, another recurrent neural network decodes the source hidden sequence into the estimated translation. Both the encoder and decoder networks are implemented with gated recurrent units. In particular, the decoder network operates like a language model: it predicts the next target word from the last target word, the last hidden state of the decoder, and a convex combination of the encoder hidden states. The weights of this convex combination are dynamically computed through a simple feed-forward network, called attention model.” Examiner’s note, the recurrent neural network is generated on the generic computer, and each of the recurrent neural network (derivative neural network) iteratively generates the sequence of input, for example, the first recurrent neural network encodes the input into word by word sequence at the hidden state and the second recurrent neural network decodes the sequence hidden state (output from the first recurrent neural network to be the input is inputted into the decoder) into the estimated translation.).
wherein the derivative neural network is afeed-forward neural network (Bertoldi , [0147] The NMT model uses a two-step process to calculate an estimated translation into the target language from an input source language text segment: first, a recurrent neural network encodes the input source language text segment word by word into a sequence of hidden states; then, another recurrent neural network decodes the source hidden sequence into the estimated translation. Both the encoder and decoder networks are implemented with gated recurrent units. In particular, the decoder network operates like a language model: it predicts the next target word from the last target word, the last hidden state of the decoder, and a convex combination of the encoder hidden states. The weights of this convex combination are dynamically computed through a simple feed-forward network, called attention model.” Examiner’s note, Therefore, the derivative neural network include encoder and decoder are dynamically computed through a simple feed-forward network.).
However, Bertoldi does not teach the method comprising: receiving a representation of the RNN, transforming the representation of the RNN into a derivative neural network for operation over a predetermined plurality of inputs of the sequence of inputs, the derivative neural network having one or more state inputs and one or more state outputs and being equivalent to the RNN over a predetermined plurality of steps of the RNN, wherein each instance of the derivative neural network operates on a respective predetermined plurality of inputs of the sequence of inputs, operate the RNN over a sequence of inputs longer than the predetermined plurality of inputs.
On the other hand, Sherstinsky teaches the method comprising: receiving a representation of the RNN (Sherstinsky , [Sec.2, ], “In this section, we will derive the Recurrent Neural Network (RNN) from differential equations [9,10]. Let ⃗s(t) be the value of the d-dimensional state signal vector and consider the general nonlinear first-order non-homogeneous ordinary differential
equation, which describes the evolution of the state signal as a function of time, t:…”),
transforming the representation of the RNN into a derivative neural network for operation over a predetermined plurality of inputs of the sequence of inputs, the derivative neural network having one or more state inputs and one or more state outputs and being equivalent to the RNN over a predetermined plurality of steps of the RNN (Sherstinsky , [Sec.3, Fig.2], “
PNG
media_image1.png
541
417
media_image1.png
Greyscale
…
PNG
media_image2.png
656
402
media_image2.png
Greyscale
PNG
media_image3.png
297
937
media_image3.png
Greyscale
” Examiner’s note, training sample is subdivided into the M segments (plurality of the inputs), wherein, each RNN cell is considered as the derivative neural network operation on the predetermined input segment sequence x(0), x(1), x(2).);
wherein each instance of the derivative neural network operates on a respective predetermined plurality of inputs of the sequence of inputs (Sherstinsky , [Sec.3, Fig.2], “
PNG
media_image1.png
541
417
media_image1.png
Greyscale
…
PNG
media_image2.png
656
402
media_image2.png
Greyscale
PNG
media_image3.png
297
937
media_image3.png
Greyscale
” Examiner’s note, training sample is subdivided into the M segments (plurality of the input sequences are generated by the RNN cell);
Bertoldi and Sherstinsky are analogous in arts because they have the same field of endeavor of training data in neural network.
Accordingly, it would have been obvious to one of the ordinary skills in the art before the effective filing date of the claimed invention to modify the recurrent neural network being for operation on a different input of the sequence, the hardware being adapted to perform feed-forward neural networks, as taught by Bertoldi, to include the receiving a representation of the RNN, transforming the representation of the RNN into a derivative neural network for operation over a predetermined plurality of inputs of the sequence of inputs, the derivative neural network having one or more state inputs and one or more state outputs and being equivalent to the RNN over a predetermined plurality of steps of the RNN, wherein each instance of the derivative neural network operates on a respective predetermined plurality of inputs of the sequence of inputs, as taught by Sherstinsky . The modification would have been obvious because one of the ordinary skills in art would be motivated to get closer to the desired output sample, (Sherstinsky , [page10, right column, first paragraph], “
PNG
media_image4.png
367
431
media_image4.png
Greyscale
“).
However, neither Bertoldi nor Sherstinsky teaches operate the RNN over a sequence of inputs longer than the predetermined plurality of inputs.
On the other hand, Lee teaches operate the RNN over a sequence of inputs longer than the predetermined plurality of inputs (Lee, [Par.0124], “As illustrated, a start token <Start> is input to a first hidden node, and “n” is generated. For example, the language processing apparatus 100 may verify that a degree of difficulty of generating a token sequence to be generated subsequently to “n” is less than or equal to the threshold value, before the generated token “n” is input to the main decoder 503 again. Instead of inputting <Start> and “n” to a second hidden node of the main decoder 503, the language processing apparatus 100 performs decoding using the sub-decoder 505. Herein, the sub-decoder 505 may generate a target text relatively faster than the main decoder 503. For example, less than all previous generated decoded tokens may be input to the sub-decoder 505, and thus sub-decoder 505 may decode context information faster than if the main decoder 503 performed the same decoding,” Examiner’s note, the main decoder perform the decoding longer than the sub decoder.)
Bertoldi, Sherstinsky and Lee are analogous in arts because they have the same field of endeavor of training data in neural network.
Accordingly, it would have been obvious to one of the ordinary skills in the art before the effective filing date of the claimed invention to modify the providing the one or more state outputs from each instance of the derivative neural network at the hardware as the one or more state inputs to a subsequent instance of the derivative neural network at the hardware so as to operate the RNN over a sequence of inputs, as taught by Bertoldi, to include the r operate the RNN over a sequence of inputs longer than the predetermined plurality of inputs, as taught by Lee . The modification would have been obvious because one of the ordinary skills in art would be motivated to generate the text faster, (Lee, [Par.0124], “Herein, the sub-decoder 505 may generate a target text relatively faster than the main decoder 503. For example, less than all previous generated decoded tokens may be input to the sub-decoder 505, and thus sub-decoder 505 may decode context information faster than if the main decoder 503 performed the same decoding. As another example, the sub-decoder 505 (and/or sub-decoder 505′) may include less hidden layers, nodes, and/or connections weights than the main decoder 503, and thus be smaller than the main decoder 503. The sub-decoder(s) or various selectable sub-decoders, for example, may also be respectively trained by being optimized for a certain type of prefix, and thus perform decoding faster and more accurately than the main decoder 504.”).
Regarding claim 2, Bertoldi as modified in view of Sherstinsky teaches the method of claim 1, wherein the predetermined plurality of steps is equal in number to the predetermined plurality of inputs. (Sherstinsky, [Sec.3, Fig.2], “
PNG
media_image1.png
541
417
media_image1.png
Greyscale
…
PNG
media_image2.png
656
402
media_image2.png
Greyscale
PNG
media_image3.png
297
937
media_image3.png
Greyscale
” Examiner’s note, training sample is subdivided into the M segments (plurality of the input sequences are generated by the RNN cell);
Bertoldi and Sherstinsky are analogous in arts because they have the same field of endeavor of training data in neural network.
Accordingly, it would have been obvious to one of the ordinary skills in the art before the effective filing date of the claimed invention to modify the recurrent neural network, as taught by Bertoldi, to include the predetermined plurality of steps is equal in number to the predetermined plurality of inputs, as taught by Sherstinsky . The modification would have been obvious because one of the ordinary skills in art would be motivated to get closer to the desired output sample, (Sherstinsky , [page10, right column, first paragraph], “
PNG
media_image4.png
367
431
media_image4.png
Greyscale
“).
Regarding claim 3, Bertoldi teaches The method of claim 1, wherein the one or more state outputs from each instance of the derivative neural network are provided as the one or more state inputs to the subsequent instance of the derivative neural network in the sequence of instances of the derivative neural network (Bertoldi , [0147] The NMT model uses a two-step process to calculate an estimated translation into the target language from an input source language text segment: first, a recurrent neural network encodes the input source language text segment word by word into a sequence of hidden states; then, another recurrent neural network decodes the source hidden sequence into the estimated translation. Both the encoder and decoder networks are implemented with gated recurrent units. In particular, the decoder network operates like a language model: it predicts the next target word from the last target word, the last hidden state of the decoder, and a convex combination of the encoder hidden states. The weights of this convex combination are dynamically computed through a simple feed-forward network, called attention model.” Examiner’s note, each of the recurrent neural network (derivative neural network) iteratively generates the sequence of input, for example, the first recurrent neural network encodes the input into word by word sequence at the hidden state and the second recurrent neural network decodes the sequence hidden state (output from the first recurrent neural network to be the input is inputted into the decoder) into the estimated translation.).
Regarding claim 4, Bertoldi teaches the method of claim 1, wherein the implementing a sequence of instances of the derivative neural network comprises implementing an instance of the derivative neural network and, on completion of that instance, causing the next instance of the derivative neural network in the sequence to be implemented in hardware (Bertoldi, [Par.0101-0103], “The system comprises a server 102, connected to user devices 104, 106, 108, 110, 112 by a networked internet connection 114. Each user device 104, 106, 108, 110, 112 has a CAT tool plugin. The server 102 is a general-purpose computer system connected to the internet… The input source language text segments and corresponding input context information metadata are then sent to server 102 over internet connection 114. They may be sent in sequential order (i.e. starting with the first input source language text segment to appear in the input source language text document, and the corresponding input context information metadata), if the user selects to perform translation in a sequential order.” [0147] The NMT model uses a two-step process to calculate an estimated translation into the target language from an input source language text segment: first, a recurrent neural network encodes the input source language text segment word by word into a sequence of hidden states; then, another recurrent neural network decodes the source hidden sequence into the estimated translation. Both the encoder and decoder networks are implemented with gated recurrent units. In particular, the decoder network operates like a language model: it predicts the next target word from the last target word, the last hidden state of the decoder, and a convex combination of the encoder hidden states. The weights of this convex combination are dynamically computed through a simple feed-forward network, called attention model.” Examiner’s note, the recurrent neural network is generated on the generic computer, and each of the recurrent neural network (derivative neural network) iteratively generates the sequence of input, for example, the first recurrent neural network encodes the input into word by word sequence at the hidden state, that is considered as the completion of that instance, and then the second recurrent neural network decodes the sequence hidden state (output from the first recurrent neural network to be the input is inputted into the decoder) into the estimated translation.).
Regarding claim 5, Bertoldi as modified in view of Sherstinsky teaches the method of claim 1, wherein the transforming comprises unrolling the recurrent neural network over the predetermined plurality of steps so as to form the derivative neural network for operation over the predetermined plurality of inputs of the sequence of inputs. (Section 3, Fig.2, “Unrolling (or unfolding) for a finite number of steps is a standard, straightforward technique for approximating RNNs by FIR sequences. However, due to the truncation inherent in limiting the number of steps, the resulting unfolded RNN model introduces artificial discontinuities in the approximated version of the target output sequence. In general, the more steps are included in the unrolled RNN subsequence, the closer it can get to the desired output samples, but the less efficient the system becomes, due to the increased number of computations”).
Bertoldi and Sherstinsky are analogous in arts because they have the same field of endeavor of training data in neural network.
Accordingly, it would have been obvious to one of the ordinary skills in the art before the effective filing date of the claimed invention to modify the recurrent neural network, as taught by Bertoldi, to include the the transforming comprises unrolling the recurrent neural network over the predetermined plurality of steps so as to form the derivative neural network for operation over the predetermined plurality of inputs of the sequence of inputs, as taught by Sherstinsky. The modification would have been obvious because one of the ordinary skills in art would be motivated to get closer to the desired output sample, (Sherstinsky , [page10, right column, first paragraph], “
PNG
media_image4.png
367
431
media_image4.png
Greyscale
“).
Regarding claim 7, Bertoldi as modified in view of Sherstinsky the method of claim 1, wherein the hardware and its control logic are incapable of executing the received representation of the RNN. (Section 3, Pg. 6, Column 2, Lines 6-13, “It is convenient to use the term ‘‘cell’’ when referring to Eqs. (30) and (32) in the uninitialized state. In other words, the sequence has been defined by these equations, but its terms not yet computed. Then the cell can be said to be ‘‘unfolded’’ or ‘‘unrolled’’ by specifying the initial conditions on the state signal,
s
→
[
n
]
, and numerically evaluating Eq. (30) or Eq. (32) for a finite range of discrete steps, indexed by
n
. This process is illustrated in Fig. 2.)
Bertoldi and Sherstinsky are analogous in arts because they have the same field of endeavor of training data in neural network.
Accordingly, it would have been obvious to one of the ordinary skills in the art before the effective filing date of the claimed invention to modify the recurrent neural network, as taught by Bertoldi, to include the hardware and its control logic are incapable of executing the received representation of the RNN, as taught by Sherstinsky. The modification would have been obvious because one of the ordinary skills in art would be motivated to get closer to the desired output sample, (Sherstinsky , [page10, right column, first paragraph], “
PNG
media_image4.png
367
431
media_image4.png
Greyscale
“).
Regarding claim 8, Bertoldi as modified in view of Sherstinsky teaches the method of claim 1, wherein the hardware and its control logic are incapable of executing dynamic neural networks. (Section 7, Pg. 22, Column 2 , “Note that all data signals in Eq. (178) (
u
→
,
s
→
, and
r
→
) have the dynamic range of 2, because they are at the output node of the warping function,
G
d
(
z
→
), which is the hyperbolic tangent. Hence, for the respective term to have the norm of <1, the associated parameter matrices must have the norm of <
1
2
”).
Bertoldi and Sherstinsky are analogous in arts because they have the same field of endeavor of training data in neural network.
Accordingly, it would have been obvious to one of the ordinary skills in the art before the effective filing date of the claimed invention to modify the recurrent neural network, as taught by Bertoldi, to include the hardware and its control logic are incapable of executing the received representation of the RNN, as taught by Sherstinsky. The modification would have been obvious because one of the ordinary skills in art would be motivated to get closer to outputting the node of the warping function, (Section 7, Pg. 22, Column 2, Lines 46-49], (Section 7, Pg. 22, Column 2, Lines 46-49, “Note that all data signals in Eq. (178) (
u
→
,
s
→
, and
r
→
) have the dynamic range of 2, because they are at the output node of the warping function,
G
d
(
z
→
), which is the hyperbolic tangent. Hence, for the respective term to have the norm of <1, the associated parameter matrices must have the norm of <
1
2
”).
Regarding claim 10, Bertoldi as modified in view of Sherstinsky teaches the method of claim 1, wherein the RNN comprises one or more cells, each cell arranged to receive a cell state input generated at a preceding step, and the transforming the representation of the RNN further comprises, at each cell: (Fig. 2., Pg. 7)
PNG
media_image5.png
200
400
media_image5.png
Greyscale
identifying non-causal operations which are for performance without dependence on the cell state input; and (Section 8, Pg. 27, Column 1, Lines 13-22, “For these applications, it will be important to evaluate the performance impact, attributed to the non-causal input context windows, as compared to the different
baselines, such the Vanilla LSTM network, the bi-directional LSTM network, the Transformer, and other state-of-the-art models. Also of particular relevance to this use case will be to measure the effectiveness of the external input gate in helping to eliminate the non-essential content from the input sequences. Finally, adopting the Augmented LSTM network to other practical domains and publishing the results is respectfully encouraged”).
in the derivative neural network, grouping together at least some of the non-causal operations at a plurality of instances of the cell over at least some of the predetermined plurality of steps for processing in parallel at the hardware (Sec.3, page 10, “
PNG
media_image6.png
527
358
media_image6.png
Greyscale
”
).
Bertoldi and Sherstinsky are analogous in arts because they have the same field of endeavor of training data in neural network.
Accordingly, it would have been obvious to one of the ordinary skills in the art before the effective filing date of the claimed invention to modify the recurrent neural network, as taught by Bertoldi, to include the RNN comprises one or more cells, each cell arranged to receive a cell state input generated at a preceding step, and the transforming the representation of the RNN further comprises, at each cell:identifying non-causal operations which are for performance without dependence on the cell state input; andin the derivative neural network, grouping together at least some of the non-causal operations at a plurality of instances of the cell over at least some of the predetermined plurality of steps for processing in parallel at the hardware., as taught by Sherstinsky. The modification would have been obvious because one of the ordinary skills in art would be motivated to morphing the canonical RNN system into the more robust LSTM network through a series of extensions and embellishments, (Section8,], “We subsequently addressed the shortcomings of the standard RNN by morphing the canonical RNN system into the more robust LSTM network through a series of extensions and embellishments. In addition to the logical construction of the Vanilla LSTM network from the canonical RNN, we included a self-contained overview of the Vanilla LSTM network, complete with the specifications of all principal entities as well as clear, descriptive, yet concise, presentations of the forward pass and, importantly, the backward pass, without skipping any steps.”)
Regarding claim 11, Bertoldi as modified in view of Sherstinsky teaches the method of claim 10, wherein the cell comprises causal operations which are for performance in dependence on the cell state input. (Section 1, Pg. 3, Column 2, Lines 50-58, “The time delay terms on the right-hand side of Eq. (9) comprise the ‘‘memory’’ aspects of the system. They enable the quantity holding the instantaneous time rate of change of the state signal,
d
s
→
(
t
)
d
t
, to incorporate contributions from the state, the readout, and the input signal values, measured at different points in time, relative to the current time,
t
. Qualitatively, these temporal elements enrich the expressive power of the model by capturing causal
and/or contextual information”).
Bertoldi and Sherstinsky are analogous in arts because they have the same field of endeavor of training data in neural network.
Accordingly, it would have been obvious to one of the ordinary skills in the art before the effective filing date of the claimed invention to modify the recurrent neural network, as taught by Bertoldi, to include the the cell comprises causal operations which are for performance in dependence on the cell state input, as taught by Sherstinsky. The modification would have been obvious because one of the ordinary skills in art would be motivated to measure the input signal at different points in time, (Section 1, Pg. 3, Column 2, Lines 50-58, “The time delay terms on the right-hand side of Eq. (9) comprise the ‘‘memory’’ aspects of the system. They enable the quantity holding the instantaneous time rate of change of the state signal,
d
s
→
(
t
)
d
t
, to incorporate contributions from the state, the readout, and the input signal values, measured at different points in time, relative to the current time,
t
. Qualitatively, these temporal elements enrich the expressive power of the model by capturing causal
and/or contextual information”).
Regarding claim 12, Bertoldi as modified in view of Sherstinsky teaches the method of claim 10, wherein at least part of the cell state input is generated at the preceding instance of the cell at the preceding step. (Section 6, Pg. 17, Column 1, Lines 16-20, “
a
→
∈
R
d
s
— an accumulation node of the cell (linearly combines the signals from the preceding step and the present. step as net input to a warping function at the present step; each cell contains several purpose-specific control and data accumulation nodes)”).
Bertoldi and Sherstinsky are analogous in arts because they have the same field of endeavor of training data in neural network.
Accordingly, it would have been obvious to one of the ordinary skills in the art before the effective filing date of the claimed invention to modify the recurrent neural network, as taught by Bertoldi, to include at least part of the cell state input is generated at the preceding instance of the cell at the preceding step, as taught by Sherstinsky. The modification would have been obvious because one of the ordinary skills in art would be motivated to get closer to outputting the node of the warping function, (Section 7, Pg. 22, Column 2, Lines 46-49], (Section 7, Pg. 22, Column 2, Lines 46-49, “Note that all data signals in Eq. (178) (
u
→
,
s
→
, and
r
→
) have the dynamic range of 2, because they are at the output node of the warping function,
G
d
(
z
→
), which is the hyperbolic tangent. Hence, for the respective term to have the norm of <1, the associated parameter matrices must have the norm of <
1
2
”).
Regarding claim 13, Bertoldi in in view of Sherstinsky teaches the method of claim 10, wherein the grouping together comprises combining the at least some non-causal operations for performance as a single convolution operation for the plurality of instances of the cell in the derivative neural network. (Section 7.1, Pg. 23, Column 1, Lines 11-18, “In Eqs. (126)-(129), the matrix-vector products,
W
x
c
u
x
→
n
,
W
x
c
s
x
→
n
,
W
x
a
x
→
n
,
and
W
x
c
u
x
→
n
,
respectively, which involve a single input sample,
x
→
n
,
will be replaced by the convolutions of the context window filters,
W
x
c
u
n
,
W
x
c
s
n
,
W
x
c
r
n
,
and
W
x
d
u
n
,
respectively, with the input signal,
x
→
n
,
thereby involving all input samples within the context window in the computation of the respective accumulation signal”).
Bertoldi and Sherstinsky are analogous in arts because they have the same field of endeavor of training data in neural network.
Accordingly, it would have been obvious to one of the ordinary skills in the art before the effective filing date of the claimed invention to modify the recurrent neural network, as taught by Bertoldi, to include the grouping together comprises combining the at least some non-causal operations for performance as a single convolution operation for the plurality of instances of the cell in the derivative neural network., as taught by Sherstinsky. The modification would have been obvious because one of the ordinary skills in art would be motivated to replacing the single input sample,
x
→
n
,
will be replaced by the convolutions of the context, (Section 7.1, Pg. 23, Column 1, Lines 11-18, “In Eqs. (126)-(129), the matrix-vector products,
W
x
c
u
x
→
n
,
W
x
c
s
x
→
n
,
W
x
a
x
→
n
,
and
W
x
c
u
x
→
n
,
respectively, which involve a single input sample,
x
→
n
,
will be replaced by the convolutions of the context window filters,
W
x
c
u
n
,
W
x
c
s
n
,
W
x
c
r
n
,
and
W
x
d
u
n
,
respectively, with the input signal,
x
→
n
,
thereby involving all input samples within the context window in the computation of the respective accumulation signal”).
Regarding claim 14, Bertoldi in in view of Sherstinsky teaches the method of claim 10, wherein the transforming the representation of the RNN further comprises splitting the at least some of the non-causal operations from the causal operations. (Section 1, Pg. 3, Column 2, Lines 50-58, “The time delay terms on the right-hand side of Eq. (9) comprise the ‘‘memory’’ aspects of the system. They enable the quantity holding the instantaneous time rate of change of the state signal,
d
s
→
(
t
)
d
t
, to incorporate contributions from the state, the readout, and the input signal values, measured at different points in time, relative to the current time,
t
. Qualitatively, these temporal elements enrich the expressive power of the model by capturing causal and/or contextual information”).
Bertoldi and Sherstinsky are analogous in arts because they have the same field of endeavor of training data in neural network.
Accordingly, it would have been obvious to one of the ordinary skills in the art before the effective filing date of the claimed invention to modify the recurrent neural network, as taught by Bertoldi, to include the ransforming the representation of the RNN further comprises splitting the at least some of the non-causal operations from the causal operations, as taught by Sherstinsky. T The modification would have been obvious because one of the ordinary skills in art would be motivated to measure the input signal at different points in time, (Section 1, Pg. 3, Column 2, Lines 50-58, “The time delay terms on the right-hand side of Eq. (9) comprise the ‘‘memory’’ aspects of the system. They enable the quantity holding the instantaneous time rate of change of the state signal,
d
s
→
(
t
)
d
t
, to incorporate contributions from the state, the readout, and the input signal values, measured at different points in time, relative to the current time,
t
. Qualitatively, these temporal elements enrich the expressive power of the model by capturing causal
and/or contextual information”).
Regarding claim 15, Bertoldi in in view of Sherstinsky teaches the method of claim 11, wherein the implementing a sequence of instances of the derivative neural network in hardware comprises, for each instance, causing the hardware to process one or more of the groups of non-causal operations in parallel. (Section 7.5, Pg. 26, Column 2, Lines 6-15, “Arranged to parallel the structure of Θ, defined in Eq. (215), the total partial derivative of the objective function,
E
, with respect to the model parameters, Θ, at the step with the index,
n
, is:
PNG
media_image7.png
200
400
media_image7.png
Greyscale
Finally,
d
E
d
Θ
, the total derivative of the objective function,
E
, with respect to the model parameters, Θ, for the entire unrolled sequence is computed by Eq. (172). Aggregated over a batch of segments
d
E
d
Θ
is plugged in to the Gradient Descent training algorithm for learning the model parameters, Θ”).
Bertoldi and Sherstinsky are analogous in arts because they have the same field of endeavor of training data in neural network.
Accordingly, it would have been obvious to one of the ordinary skills in the art before the effective filing date of the claimed invention to modify the recurrent neural network, as taught by Bertoldi, to include the implementing a sequence of instances of the derivative neural network in hardware comprises, for each instance, causing the hardware to process one or more of the groups of non-causal operations in parallel, as taught by Sherstinsky. The modification would have been obvious because one of the ordinary skills in art would be motivated to Arranged to parallel the structure of Θ, defined in Eq. (215), (Section 7.5, Pg. 26, Column 2, Lines 6-15, “Arranged to parallel the structure of Θ, defined in Eq. (215), the total partial derivative of the objective function,
E
, with respect to the model parameters, Θ, at the step with the index,
n
, is:
PNG
media_image7.png
200
400
media_image7.png
Greyscale
Finally,
d
E
d
Θ
, the total derivative of the objective function,
E
, with respect to the model parameters, Θ, for the entire unrolled sequence is computed by Eq. (172). Aggregated over a batch of segments
d
E
d
Θ
is plugged in to the Gradient Descent training algorithm for learning the model parameters, Θ”).
Regarding claim 16, Bertoldi as modified in view of Chang teaches the method of claim 11, wherein the hardware comprises an accelerator having a plurality of processing elements for executing a neural network and each group of non-causal operations is processed in parallel over the at least some of the plurality of processing elements. (Section III, Pg. 2, Column 2, Lines 10-24, “An accelerator called
n
n
-X for deep neural networks is described in [21]–[24].
n
n
-X is a high-performance co-processor implemented on FPGA. The design is based on computational elements called collections that are capable of performing convolution, non-linear functions and pooling. The accelerator efficiently pipelines the collections achieving up to 240 G-op/s. RNNs are different from CNNs in the context that they require a different arrangement of computation modules.
This allows different hardware optimization strategies that should be exploited. A LSTM learning algorithm using Simultaneous Perturbation Stochastic Approximation (SPSA)
for hardware friendly implementation was described in [25]. The paper focuses on transformation of the learning phase of LSTM for FPGA”).
Bertoldi, Sherstinsky and Chang are analogous in arts because they have the same field of endeavor of training data in neural network.
Accordingly, it would have been obvious to one of the ordinary skills in the art before the effective filing date of the claimed invention to modify the recurrent neural network, as taught by Bertoldi, to include the hardware comprises an accelerator having a plurality of processing elements for executing a neural network and each group of non-causal operations is processed in parallel over the at least some of the plurality of processing elements, as taught by Chang. The modification would have been obvious because one of the ordinary skills in art would be motivated to speed up the performance, (Chang, [Sec. VI. C], “Figure 7 shows the timing results. One can observe that the implemented hardware LSTM was significantly faster than other platforms, even running at lower clock frequency of 142 MHz (Zynq ZC7020 CPU uses 667 MHz). Scaling the implemented design by replicating the number of LSTM modules running in parallel will provide faster speed up. Using 2 LSTM cells in parallel can be 16× faster than Exynos5422 on quad-core ARM Cortex-A7”).
Regarding claim 17, Bertoldi in in view of Sherstinsky teaches the method of claim 11, wherein the transforming the RNN further comprises configuring the derivative neural network such that the result of the non-causal operations performed at an instance of the cell is combined with the causal operation performed in respect of that same instance. (Section 6.9, Pg. 20, Column 1, Lines 33-46, “We also define the total partial derivatives of the objective function, E, with respect to three intermediate (i.e., away from the border) variables and another border variable of the Vanilla LSTM cell:
PNG
media_image8.png
200
400
media_image8.png
Greyscale
The border quantity in Eq. (143)
ψ
→
n
,
is of special significance as it is the total partial derivative of the objective function,
E
, with respect to the state signal,
s
→
n
,
at index,
n
,
of the Vanilla LSTM cell. As in the standard RNN, all parameter updates in the Vanilla LSTM network depend on
ψ
→
n
,
making it the most important error gradient sequence of the system.
Bertoldi and Sherstinsky are analogous in arts because they have the same field of endeavor of training data in neural network.
Accordingly, it would have been obvious to one of the ordinary skills in the art before the effective filing date of the claimed invention to modify the recurrent neural network, as taught by Bertoldi, to include the transforming the RNN further comprises configuring the derivative neural network such that the result of the non-causal operations performed at an instance of the cell is combined with the causal operation performed in respect of that same instance., as taught by Sherstinsky. The modification would have been obvious because one of the ordinary skills in art would be motivated to Arranged to parallel the structure of Θ, defined in Eq. (215), (Section 7.5, Pg. 26, Column 2, Lines 6-15, “Arranged to parallel the structure of Θ, defined in Eq. (215), the total partial derivative of the objective function,
E
, with respect to the model parameters, Θ, at the step with the index,
n
, is:
PNG
media_image7.png
200
400
media_image7.png
Greyscale
Finally,
d
E
d
Θ
, the total derivative of the objective function,
E
, with respect to the model parameters, Θ, for the entire unrolled sequence is computed by Eq. (172). Aggregated over a batch of segments
d
E
d
Θ
is plugged in to the Gradient Descent training algorithm for learning the model parameters, Θ”).
Regarding claim 18, Bertoldi in in view of Sherstinsky teaches the method of claim 1, wherein the recurrent neural network comprises a plurality of cells. (Section 2, Pg. 5, Column 2, Lines 8-13, “The RNN formulation in Eq. (30), diagrammed in Fig. 1, will be later logically evolved into the LSTM system. Before that, it is beneficial to introduce the process of ‘‘unrolling’’ and the notion of a ‘‘cell’’ of an RNN. These concepts will be simpler todescribe using the standard RNN definition, which is derived next from Eq. (30) based on stability arguments”).
Bertoldi and Sherstinsky are analogous in arts because they have the same field of endeavor of training data in neural network.
Accordingly, it would have been obvious to one of the ordinary skills in the art before the effective filing date of the claimed invention to modify the recurrent neural network, as taught by Bertoldi, to include the the recurrent neural network comprises a plurality of cells., as taught by Sherstinsky. The modification would have been obvious because one of the ordinary skills in art would be motivated to introduce the process of ‘‘unrolling’’ and the notion of a ‘‘cell’’ of an RNN, (Section 2, Pg. 5, Column 2, Lines 8-13, “The RNN formulation in Eq. (30), diagrammed in Fig. 1, will be later logically evolved into the LSTM system. Before that, it is beneficial to introduce the process of ‘‘unrolling’’ and the notion of a ‘‘cell’’ of an RNN. These concepts will be simpler todescribe using the standard RNN definition, which is derived next from Eq. (30) based on stability arguments”).
Regarding claim 19, Bertoldi teaches a data processing system for implementing a recurrent neural network (RNN) for operation on a sequence of inputs, (Bertoldi, [Par.0101-0103], “The system comprises a server 102, connected to user devices 104, 106, 108, 110, 112 by a networked internet connection 114. Each user device 104, 106, 108, 110, 112 has a CAT tool plugin. The server 102 is a general-purpose computer system connected to the internet… The input source language text segments and corresponding input context information metadata are then sent to server 102 over internet connection 114. They may be sent in sequential order (i.e. starting with the first input source language text segment to appear in the input source language text document, and the corresponding input context information metadata), if the user selects to perform translation in a sequential order.” An d[Par.0147], “The NMT model uses a two-step process to calculate an estimated translation into the target language from an input source language text segment: first, a recurrent neural network encodes the input source language text segment word by word into a sequence of hidden states; then, another recurrent neural network decodes the source hidden sequence into the estimated translation. Both the encoder and decoder networks are implemented with gated recurrent units. In particular, the decoder network operates like a language model: it predicts the next target word from the last target word, the last hidden state of the decoder, and a convex combination of the encoder hidden states. The weights of this convex combination are dynamically computed through a simple feed-forward network, called attention model.” Examiner’s note, the server/generic computer system translates the input text segments by encoding the input text segment into word by word using the recurrent neural network, such as first, a recurrent neural network encodes the input source language text segment word by word into a sequence of hidden states; then, another recurrent neural network decodes the source hidden sequence into the estimated translation. The input is generated from one state of the recurrent neural network to other state of the recurrent neural network, such as the neural network encoder encode the input language text statement into the hidden state and the neural network decoder to decode the input from the hidden state to the estimated translation. Therefore, the input sequence is inputted from one state and become the output to be inputted to another state of the neural network, that is corresponding to each step of the recurrent neural network being for operation on a different input of the sequence..),
and iteration logic configured to iteratively apply the derivative neural network to the sequence of inputs by causing a sequence of instances of the derivative neural network to be implemented at the hardware (Bertoldi, [Par.0101-0103], “The system comprises a server 102, connected to user devices 104, 106, 108, 110, 112 by a networked internet connection 114. Each user device 104, 106, 108, 110, 112 has a CAT tool plugin. The server 102 is a general-purpose computer system connected to the internet… The input source language text segments and corresponding input context information metadata are then sent to server 102 over internet connection 114. They may be sent in sequential order (i.e. starting with the first input source language text segment to appear in the input source language text document, and the corresponding input context information metadata), if the user selects to perform translation in a sequential order.” [par.0147], “The NMT model uses a two-step process to calculate an estimated translation into the target language from an input source language text segment: first, a recurrent neural network encodes the input source language text segment word by word into a sequence of hidden states; then, another recurrent neural network decodes the source hidden sequence into the estimated translation. Both the encoder and decoder networks are implemented with gated recurrent units. In particular, the decoder network operates like a language model: it predicts the next target word from the last target word, the last hidden state of the decoder, and a convex combination of the encoder hidden states. The weights of this convex combination are dynamically computed through a simple feed-forward network, called attention model.” Examiner’s note, the method for generating the input language text into the estimated translation by generating on the neural network is implemented on the computer system (hardware). The recurrent neural network is generated on the generic computer, and each of the recurrent neural network (derivative neural network) iteratively generates the sequence of input, for example, the first recurrent neural network encodes the input into word by word sequence at the hidden state and the second recurrent neural network decodes the sequence hidden state (output from the first recurrent neural network become the input is inputted into the decoder) to output the estimated translation.) ,
and providing the one or more state outputs from each representation of the derivative neural network at the hardware accelerator as the one or more state inputs to a subsequent representation of the derivative neural network (Bertoldi, [Abstarct], [Par.0101-0103], “The system comprises a server 102, connected to user devices 104, 106, 108, 110, 112 by a networked internet connection 114. Each user device 104, 106, 108, 110, 112 has a CAT tool plugin. The server 102 is a general-purpose computer system connected to the internet… The input source language text segments and corresponding input context information metadata are then sent to server 102 over internet connection 114. They may be sent in sequential order (i.e. starting with the first input source language text segment to appear in the input source language text document, and the corresponding input context information metadata), if the user selects to perform translation in a sequential order.” [0147] The NMT model uses a two-step process to calculate an estimated translation into the target language from an input source language text segment: first, a recurrent neural network encodes the input source language text segment word by word into a sequence of hidden states; then, another recurrent neural network decodes the source hidden sequence into the estimated translation. Both the encoder and decoder networks are implemented with gated recurrent units. In particular, the decoder network operates like a language model: it predicts the next target word from the last target word, the last hidden state of the decoder, and a convex combination of the encoder hidden states. The weights of this convex combination are dynamically computed through a simple feed-forward network, called attention model.” Examiner’s note, the recurrent neural network is generated on the generic computer, and each of the recurrent neural network (derivative neural network) iteratively generates the sequence of input, for example, the first recurrent neural network encodes the input into word by word sequence at the hidden state and the second recurrent neural network decodes the sequence hidden state (output from the first recurrent neural network to be the input is inputted into the decoder) into the estimated translation.).
to operate the RNN over a sequence of inputs longer than the predetermined plurality of inputs (Bertoldi, [Par.0101-0103], “The system comprises a server 102, connected to user devices 104, 106, 108, 110, 112 by a networked internet connection 114. Each user device 104, 106, 108, 110, 112 has a CAT tool plugin. The server 102 is a general-purpose computer system connected to the internet… The input source language text segments and corresponding input context information metadata are then sent to server 102 over internet connection 114. They may be sent in sequential order (i.e. starting with the first input source language text segment to appear in the input source language text document, and the corresponding input context information metadata), if the user selects to perform translation in a sequential order.” [0147] The NMT model uses a two-step process to calculate an estimated translation into the target language from an input source language text segment: first, a recurrent neural network encodes the input source language text segment word by word into a sequence of hidden states; then, another recurrent neural network decodes the source hidden sequence into the estimated translation. Both the encoder and decoder networks are implemented with gated recurrent units. In particular, the decoder network operates like a language model: it predicts the next target word from the last target word, the last hidden state of the decoder, and a convex combination of the encoder hidden states. The weights of this convex combination are dynamically computed through a simple feed-forward network, called attention model.” Examiner’s note, the recurrent neural network is generated on the generic computer, and each of the recurrent neural network (derivative neural network) iteratively generates the sequence of input, for example, the first recurrent neural network encodes the input into word by word sequence at the hidden state and the second recurrent neural network decodes the sequence hidden state (output from the first recurrent neural network to be the input is inputted into the decoder) into the estimated translation.);
wherein the derivative neural network is a feed-forward neural network (Bertoldi, [0147] The NMT model uses a two-step process to calculate an estimated translation into the target language from an input source language text segment: first, a recurrent neural network encodes the input source language text segment word by word into a sequence of hidden states; then, another recurrent neural network decodes the source hidden sequence into the estimated translation. Both the encoder and decoder networks are implemented with gated recurrent units. In particular, the decoder network operates like a language model: it predicts the next target word from the last target word, the last hidden state of the decoder, and a convex combination of the encoder hidden states. The weights of this convex combination are dynamically computed through a simple feed-forward network, called attention model.).
However, Bertoldi does not teach the system comprising:a transformation unit configured to receive a representation of the RNN and transform the representation of the RNN into a derivative neural network for operation over a predetermined plurality of inputs of the sequence of inputs, the derivative neural network having one or more state inputs and one or more state outputs and being equivalent to the RNN over a predetermined plurality of steps of the RNN, a hardware accelerator for processing neural networks, the hardware accelerator being adapted to perform feed-forward neural networks; causing a sequence of instances of the derivative neural network to be implemented at the hardware accelerator, wherein each instance of the derivative neural network operates on a respective predetermined plurality of inputs of the sequence of inputs , the derivative neural network at the hardware accelerator so as to cause the hardware accelerator.
On the other hand, Sherstinsky teaches the system comprising:a transformation unit configured to receive a representation of the RNN and transform the representation of the RNN into a derivative neural network for operation over a predetermined plurality of inputs of the sequence of inputs, the derivative neural network having one or more state inputs and one or more state outputs and being equivalent to the RNN over a predetermined plurality of steps of the RNN (Sherstinsky , [Sec.3, Fig.2], “
PNG
media_image1.png
541
417
media_image1.png
Greyscale
…
PNG
media_image2.png
656
402
media_image2.png
Greyscale
PNG
media_image3.png
297
937
media_image3.png
Greyscale
” Examiner’s note, training sample is subdivided into the M segments (plurality of the inputs), wherein, each RNN cell is considered as the derivative neural network operation on the predetermined input segment sequence x(0), x(1), x(2).);
wherein each instance of the derivative neural network operates on a respective predetermined plurality of inputs of the sequence of inputs (Sherstinsky , [Sec.3, Fig.2], “
PNG
media_image1.png
541
417
media_image1.png
Greyscale
…
PNG
media_image2.png
656
402
media_image2.png
Greyscale
PNG
media_image3.png
297
937
media_image3.png
Greyscale
” Examiner’s note, training sample is subdivided into the M segments (plurality of the input sequences are generated by the RNN cell);
Bertoldi and Sherstinsky are analogous in arts because they have the same field of endeavor of training data in neural network.
Accordingly, it would have been obvious to one of the ordinary skills in the art before the effective filing date of the claimed invention to modify the recurrent neural network being for operation on a different input of the sequence, the hardware being adapted to perform feed-forward neural networks, as taught by Bertoldi, to include the system comprising:a transformation unit configured to receive a representation of the RNN and transform the representation of the RNN into a derivative neural network for operation over a predetermined plurality of inputs of the sequence of inputs, the derivative neural network having one or more state inputs and one or more state outputs and being equivalent to the RNN over a predetermined plurality of steps of the RNN, wherein each instance of the derivative neural network operates on a respective predetermined plurality of inputs of the sequence of inputs, as taught by Sherstinsky . The modification would have been obvious because one of the ordinary skills in art would be motivated to get closer to the desired output sample, (Sherstinsky , [page10, right column, first paragraph], “
PNG
media_image4.png
367
431
media_image4.png
Greyscale
“).
However, neither Bertoldi nor Sherstinsky teaches a hardware accelerator for processing neural networks, the hardware accelerator being adapted to perform feed-forward neural networks; causing a sequence of instances of the derivative neural network to be implemented at the hardware accelerator, the derivative neural network at the hardware accelerator so as to cause the hardware accelerator to operate the RNN over a sequence of inputs longer than the predetermined plurality of inputs.
On the other hand, Lee teaches operate the RNN over a sequence of inputs longer than the predetermined plurality of inputs (Lee, [Par.0124], “As illustrated, a start token <Start> is input to a first hidden node, and “n” is generated. For example, the language processing apparatus 100 may verify that a degree of difficulty of generating a token sequence to be generated subsequently to “n” is less than or equal to the threshold value, before the generated token “n” is input to the main decoder 503 again. Instead of inputting <Start> and “n” to a second hidden node of the main decoder 503, the language processing apparatus 100 performs decoding using the sub-decoder 505. Herein, the sub-decoder 505 may generate a target text relatively faster than the main decoder 503. For example, less than all previous generated decoded tokens may be input to the sub-decoder 505, and thus sub-decoder 505 may decode context information faster than if the main decoder 503 performed the same decoding,” Examiner’s note, the main decoder perform the decoding longer than the sub decoder.)
Bertoldi, Sherstinsky and Lee are analogous in arts because they have the same field of endeavor of training data in neural network.
Accordingly, it would have been obvious to one of the ordinary skills in the art before the effective filing date of the claimed invention to modify the providing the one or more state outputs from each instance of the derivative neural network at the hardware as the one or more state inputs to a subsequent instance of the derivative neural network at the hardware so as to operate the RNN over a sequence of inputs, as taught by Bertoldi, to include the r operate the RNN over a sequence of inputs longer than the predetermined plurality of inputs, as taught by Lee . The modification would have been obvious because one of the ordinary skills in art would be motivated to generate the text faster, (Lee, [Par.0124], “Herein, the sub-decoder 505 may generate a target text relatively faster than the main decoder 503. For example, less than all previous generated decoded tokens may be input to the sub-decoder 505, and thus sub-decoder 505 may decode context information faster than if the main decoder 503 performed the same decoding. As another example, the sub-decoder 505 (and/or sub-decoder 505′) may include less hidden layers, nodes, and/or connections weights than the main decoder 503, and thus be smaller than the main decoder 503. The sub-decoder(s) or various selectable sub-decoders, for example, may also be respectively trained by being optimized for a certain type of prefix, and thus perform decoding faster and more accurately than the main decoder 504.”).
On the other hand, Chang teaches a hardware accelerator for processing neural networks (Section III, Pg. 2, Column 2, Lines 25-33, “Another FPGA implementation that focus on standard RNN is described by [26]. Their approach was to unfold the RNN model into a fixed number of timesteps B and compute them in parallel. The hardware architecture is composed of a hidden layer module and duplicated output layer modules. First, the hidden layer serially processes the input x for B timesteps. Then, with the results of the hidden layer, the duplicated logic computes output h for B timesteps in parallel”).);
the hardware accelerator being adapted to perform feed-forward neural networks (Chang, (Sec.4a, page3 , “The main operations to be implemented in hardware are matrix-vector multiplications and non-linear functions (hyperbolic tangent and logistic sigmoid). Both are modifications of the modules presented in [24]. For this design, the number format of choice is Q8.8 fixed point. The matrixvector multiplication is computed by a Multiply ACcumulate (MAC) unit, which takes two streams: vector stream and weight matrix row stream. The same vector stream is multiplied and accumulated with each weight matrix row to produce an output vector with same size of the weight’s height. The MAC is reset after computing each output element to avoid accumulating previous matrix rows computations. The bias b can be added in the multiply accumulate by adding the bias vector to the last column of the weight matrix and adding an extra vector element set to unity. This way there is no need to add extra input ports for the bias nor add extra pre-configuration step to the MAC unit. The results from the MAC units are added together. The adder’s output goes to an element wise non-linear function, which is implemented with linear mapping”);),
causing a sequence of instances of the derivative neural network to be
implemented at the hardware accelerator (Chang, (Abstract, Pg. 1, Lines 10-16, “We implemented a RNN with 2 layers and 128 hidden units in hardware and it has been tested using a character level language model. The implementation is more than 21
×
faster than the ARM Cortex-A9 CPU embedded on the Zynq 7020 FPGA. This work can potentially evolve to a RNN co-processor for future mobile devices”).);
the derivative neural network at the hardware accelerator so as to cause the hardware accelerator. (Chang, Abstract, Pg. 1, Lines 10-16, “We implemented a RNN with 2 layers and 128 hidden units in hardware and it has been tested using a character level language model. The implementation is more than 21
×
faster than the ARM Cortex-A9 CPU embedded on the Zynq 7020 FPGA. This work can potentially evolve to a RNN co-processor for future mobile devices”).
Bertoldi, Sherstinsky and Chang are analogous in arts because they have the same field of endeavor of training data in neural network.
Accordingly, it would have been obvious to one of the ordinary skills in the art before the effective filing date of the claimed invention to modify the and iteration logic configured to iteratively apply the derivative neural network to the sequence of inputs by:causing a sequence of instances of the derivative neural network to be implemented at the hardware, as taught by Bertoldi, to include the hardware accelerator for processing neural networks, the hardware accelerator being adapted to perform feed-forward neural networks; causing a sequence of instances of the derivative neural network to be implemented at the hardware accelerator, the derivative neural network at the hardware accelerator so as to cause the hardware accelerator, as taught by Chang. The modification would have been obvious because one of the ordinary skills in art would be motivated to speed up the performance, (Chang, [Sec. VI. C], “Figure 7 shows the timing results. One can observe that the implemented hardware LSTM was significantly faster than other platforms, even running at lower clock frequency of 142 MHz (Zynq ZC7020 CPU uses 667 MHz). Scaling the implemented design by replicating the number of LSTM modules running in parallel will provide faster speed up. Using 2 LSTM cells in parallel can be 16× faster than Exynos5422 on quad-core ARM Cortex-A7”).
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to EM N TRIEU whose telephone number is (571)272-5747. The examiner can normally be reached on Mon-Fri from 9:00-5:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Omar Fernandez Rivas can be reached on (571) 272-2589. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/E.T./Examiner, Art Unit 2128
/OMAR F FERNANDEZ RIVAS/Supervisory Patent Examiner, Art Unit 2128
Prosecution Insights