DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Status of the Claims
Claims 1-20 are pending for examination.
Claims 1, 13 and 19 are amended.
Claims 1-20 are rejected under 35 U.S.C. §103.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1, 3-4, 7-9, 12-13, 15-17 and 19-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Taylor et al. (U.S. 2012/0089497 hereinafter Taylor) and Gupta et al. (U.S. 2022/0217645 hereinafter Gupta) in view of Iyer et al. (U.S. 2022/0075605 hereinafter Iyer) in further view of Uehara (U.S. 2022/0237898 hereinafter Uehara).
As Claim 1, Taylor teaches a method of training a neural network, the method comprising:
provisioning, as part of a distributed computer system, a memory buffer (Taylor (¶0019 line 1-3), order book feeds) that is concurrently readable (Taylor (¶0206 line 2-6), “a parallel set of tree traversal engines can operate in parallel and interleave their accesses to memory. Furthermore, the SVU module may optionally cache recently accessed tree nodes in on-chip memory in order to further reduce memory read latency”) and writable by a plurality of separate worker instances (Taylor (¶0148 line 1-14), “parallel engines update and maintain the order and price aggregation data structures in parallel. In one embodiment, the data structures are maintained in the same physical memory. In this case, the one or more order engines (worker instances) and one or more price engines (worker instants) interleave their accesses to memory, masking the memory access latency of the memory technology and maximizing throughput of the system.”);
in a data preparation phase:
executing, on the distributed computer system, the plurality of separate worker processes that concurrently perform generation of training data for each one of a plurality of identifiers, where each one of the plurality of separate worker processes perform, for a respective one of the plurality of identifiers (Taylor (¶0148 line 1-14), “parallel engines update and maintain the order and price aggregation data structures in parallel. In one embodiment, the data structures are maintained in the same physical memory. In this case, the one or more order engines (worker instances) and one or more price engines (worker instants) interleave their accesses to memory, masking the memory access latency of the memory technology and maximizing throughput of the system.”), at least:
(a1) obtaining, for the respective one of the plurality of identifiers, a plurality of tuples that each include: a current state, an action, a calculated reward, and a next state, wherein the current state and the next state are represented as n-dimensional vectors that are each based on state data from a data transaction processing system, where n is a number of features used in training the neural network (Taylor (¶0019 line 10-14, ¶0017 line 16-21, fig. 2(a)), “The Options Price Reporting Authority (OPRA) feed is the most significant source of derivatives market data, and it belongs to the class of feeds known as "level 1" feeds. Level 1 feeds report quotes (current state), trades (an action), trade cancels (next state) and corrections (a calculated reward), and a variety of summary events”. Fig. 2(a) shows an example of order book feeds. In figure 2(a), each record (Order #) corresponds to a tuple (a row the database). Each record contain information corresponding to report quotes (current state), trades (an action), trade cancels (next state) and corrections (a calculated reward), and a variety of summary events. There are n-number of records which is the “n-dimension vectors” (each column in figure 2(a) is a “n-dimensional vectors”. “n-dimensional vector” is so broad that 1-dimenional or a column on the database meet the limitation.), and
(a2) writing, to the memory buffer, the plurality of tuples that have been obtained, wherein obtained tuples from multiple ones of the plurality of separate worker processes are written to the memory buffer concurrently (Taylor (¶0148 line 1-14), “parallel engines update and maintain the order and price aggregation data structures in parallel. In one embodiment, the data structures are maintained in the same physical memory. In this case, the one or more order engines (worker instances) and one or more price engines (worker instants) interleave their accesses to memory, masking the memory access latency of the memory technology and maximizing throughput of the system.”);
Taylor may not explicitly disclose:
in a training phase:
performing a plurality of iterations that each include at least:
(b1) sampling, from across the memory buffer that includes tuples loaded via the data preparation phase, a batch of tuples,
(b2) for each corresponding tuple in the sampled batch of tuples, generating a target Q-value based on the current state, the action, the calculated reward and the next state included in the corresponding tuple,
(b3) calculating, based at least on the target Q-values for each tuple in the sampled batch of tuples, a loss value for the sampled batch of tuples, and
wherein the data preparation phase and the training phase are performed on state data of the data transaction processing system that is associated with a plurality of different operational periods.
Gupta teaches:
in a training phase:
performing a plurality of iterations (Gupta (¶0054 line 14-18), “deep Q-learning is a special case of Q-learning where the optimal or target Q value and the current predicted Q value are estimated and converged separately using two different iterative processes”) that each include at least:
(b1) sampling, from across the memory buffer that includes tuples loaded via the data preparation phase, a batch of tuples (Gupta (¶0054 line 1-6), “The objective of the Q-learning is typically formulated as a minimization problem between a target or optimal Q value (maximum possible value from the next state) and a current predicted Q value. Here, the Q values are given by a so-called Q value function (equally called action-value function) Q(s, a).” (Gupta (¶0058), “The state space, the action space and reward of the initialized deep Q-learning network in block 202 may be defined as follows”),
(b2) for each corresponding tuple in the sampled batch of tuples, generating a target Q-value based on the current state, the action, the calculated reward and the next state included in the corresponding tuple (Gupta (¶0065 line 1-6, formula 3), “the Q value function for the deep Q-learning network in block 202 may be defined as follows. As described above, the Q value function (Q: SxA -> R) in deep Q-learning gives the sum of the immediate reward of choosing action a in state s∈S and the optimum (future) reward from the next state onwards”. Formula 3 shows a relationship between Q-value, current state, action and next state),
(b3) calculating, based at least on the target Q-values for each tuple in the sampled batch of tuples, a loss value for the sampled batch of tuples (Gupta (¶0075), “The approximate Q value function Q(sn akSn) given as an output of the deep neural network in the first optimization loop and the target Q value function Q(sn akSn) calculated in the second optimization loop using the deep neural network are, then, compared by the computing device. Specifically, the computing device evaluates, in block 305 (and block 322), a mean squared error between the approximate Q value function and the target Q value function. The computing device updates, in block 306 (and block 323), weights of the deep neural network to minimize the mean squared error.”), and
wherein the data preparation phase and the training phase are performed on state data of the data transaction processing system that is associated with a plurality of different operational periods (Gupta (¶0085 line 1-4), “the traffic conditions in the plurality of cells may change over time. Therefore, there is a need for re-optimizing the setting of the Po and α parameters for the plurality of cells dynamically over time.”).
Taylor teaches a system to update order books based on market depth data such as current state, action and target state. Gupta discloses a system and method to generate Q-value using current state, action and target state. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify system of Taylor instead be a predictions system taught by Gupta, with a reasonable expectation of success. The motivation would be to allow the system to “to learn which actions to take in any given circumstances so as to maximize the accumulated reward over time” (Gupta (¶0053, middle portion)).
Taylor in view of Gupta may not explicitly disclose:
storing a first neural network and a second neural network;
(b4) updating weights of the first neural network by performing gradient descent using the calculated loss value; and
wherein the weight of the second neural network are updated less frequently than weights of the first neural network,
Iyer teaches:
storing a first neural network and a second neural network (Iyer (¶0033), “the global ML model may be updated less frequently and may take longer to train. The local ML model, on the other hand, uses the workflow data from a given developer. Thus, in some embodiments, the global model may be updated every few weeks, whereas the local model may be updated every few days.”);
(b4) updating weights of the first neural network (Iyer (¶0042 line 1-4), “Backpropagation may be used to "pop the hood" on the hidden layers of the neural network to see how much of the loss every node is responsible for, and subsequently updating the weights in such a way that minimizes the loss by giving the nodes with higher error rates lower weights, and vice versa”) by performing gradient descent using the calculated loss value (Iyer (¶0092 line 4-7), “A cost function, such as mean square error (MSE) or gradient descent, may be used to punish predictions that are slightly wrong much less than predictions that are very wrong”); and
wherein the weight of the second neural network are updated less frequently than weights of the first neural network (Iyer (¶0033), “the global ML model may be updated less frequently and may take longer to train. The local ML model, on the other hand, uses the workflow data from a given developer. Thus, in some embodiments, the global model may be updated every few weeks, whereas the local model may be updated every few days.”),
Taylor in view of Gupta discloses a machine learning system and method to generate Q-value using current state, action and target state. Iyer discloses a machine learning system that includes global and local models. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify system of Taylor in view of Gupta instead be a predictions system taught by Iyer, with a reasonable expectation of success. The motivation would be to allow the system to “predict a larger number of sequences and/or to more accurately make predictions. Auto-completion may occur in real-time in some embodiments to save time and effort by the user” (Iyer (abstract)).
Taylor in view of Gupta in further view of Iyer may not explicitly disclose:
after performing the plurality of iterations, updating weights of the second neural network based on a combination of the weights from the first neural network and weights of a prior instance of the second neural network,
Uehara teaches:
after performing the plurality of iterations, updating weights of the second neural network based on a combination of the weights from the first neural network and weights of a prior instance of the second neural network (Uehara (¶0102 last 5 lines), “The data of the weight parameter after learning that is transmitted from the client 20 to the integration server 30 may be a difference from the weight parameter of the latest version of the master model synchronized with the integration server 30.”),
Taylor in view of Gupta in further view of Iyer discloses a machine learning system includes global and local models. Uehara discloses a system and method for updating the global model based on the difference between local model and global model. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify system of Taylor in view of Gupta in further view of Iyer instead be a predictions system taught by Uehara, with a reasonable expectation of success. The motivation would be to allow the system to “efficiently perform the learning and to improve the inference accuracy of the model” (Uehara (¶0108, middle portion)).
As Claim 3, besides claim 1, Taylor and Gupta in view of Iyer in further view of Uehara teaches wherein the memory buffer includes a first memory buffer and a second memory buffer (Uehara (¶0102 last 5 lines), “The data of the weight parameter after learning that is transmitted from the client 20 to the integration server 30 may be a difference from the weight parameter of the latest version of the master model synchronized with the integration server 30.”).
As Claim 4, besides Claim 3, Taylor and Gupta in view of Iyer in further view of Uehara teaches wherein the first memory buffer is associated with a first component of a reward function and the second memory buffer is associated with a second component of the reward function (Uehara (¶0102 last 5 lines), “The data of the weight parameter after learning that is transmitted from the client 20 to the integration server 30 may be a difference from the weight parameter of the latest version of the master model synchronized with the integration server 30.”).
As Claim 7, besides Claim 1, Taylor and Gupta in view of Iyer in further view of Uehara teaches wherein the data transaction processing system is a simulation system, the method further comprising:
executing a simulated matching process by the simulation system that simulates how data transaction requests are processed based on a dynamic timer value (Gupta (¶0085 line 1-4), “the traffic conditions in the plurality of cells may change over time. Therefore, there is a need for re-optimizing the setting of the Po and α parameters for the plurality of cells dynamically over time.”).
As Claim 8, besides Claim 7, Taylor and Gupta in view of Iyer in further view of Uehara teaches further comprising:
selecting, as part of the data preparation phase, a timer value to provide to the simulation system as the dynamic timer value (Gupta (¶0085 line 1-4), “the traffic conditions in the plurality of cells may change over time. Therefore, there is a need for re-optimizing the setting of the Po and α parameters for the plurality of cells dynamically over time.”).
As Claim 9, besides Claim 8, Taylor and Gupta in view of Iyer in further view of Uehara teaches wherein the timer value is selected based on a most recent version of the first neural network or the second neural network produced from a prior iteration of the training phase (Gupta (¶0085 line 1-4), “the traffic conditions in the plurality of cells may change over time. Therefore, there is a need for re-optimizing the setting of the Po and α parameters for the plurality of cells dynamically over time.”).
As Claim 12, besides Claim 1, Taylor and Gupta in view of Iyer in further view of Uehara teaches wherein each of the plurality of different operational periods corresponds to a different operational day that the data transaction processing system has operated (Iyer (¶0033), “the global ML model may be updated less frequently and may take longer to train. The local ML model, on the other hand, uses the workflow data from a given developer. Thus, in some embodiments, the global model may be updated every few weeks, whereas the local model may be updated every few days.”).
As Claim 13, Taylor teaches a computer system comprising:
a processing system comprising instructions that are configured to, when executed by at least one hardware processor included with the processing system, cause the at least one hardware processor (Taylor (¶0075 line 1-2), processor 812 and RAM) to perform operations comprising:
The rest of the limitation(s) are rejected for the same reasons as Claim 1.
As Claim 15, the Claim is rejected for the same reasons as Claim 3.
As Claim 16, the Claim is rejected for the same reasons as Claim 7.
As Claim 17, the Claim is rejected for the same reasons as Claim 12.
As Claim 19, the Claim is rejected for the same reasons as Claim 1.
As Claim 20, the Claim is rejected for the same reasons as Claim 4.
Claim(s) 2 and 14 is/are rejected under 35 U.S.C. 103 as being unpatentable over Taylor, Gupta and Iyer in view of Uehara in further view of Gadanho et al. (U.S. 2010/0070436 hereinafter Gadanho).
As Claim 2, besides Claim 1, Taylor and Gupta in view of Iyer in further view of Uehara may not explicitly disclose wherein, as part of (a2), only tuples with a calculated reward that is non-zero are written to the memory buffer.
Gadanho teaches:
wherein, as part of (a2), only tuples with a calculated reward that is non-zero are written to the memory buffer (Gadanho (¶0077 last 2 lines), “All examples having confidence values of zero or below are then removed from the training set”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify dataset of Taylor and Gupta in view of Iyer in further view of Uehara instead be a training set taught by Gadanho, with a reasonable expectation of success. The motivation would be so that “the combined preference value is generated in response to the confidence values of the duplicates. For example, the model processor 115 may discard all preferences that have a value below a given threshold (and e.g. average the rest) or may simply select the preference value corresponding to the highest confidence value” (Gadanho (¶0070)).
As Claim 14, the Claim is rejected for the same reasons as Claim 2.
Claim(s) 5-6 and 18 is/are rejected under 35 U.S.C. 103 as being unpatentable over Taylor, Gupta and Iyer in view of Uehara in further view of Keskar et al. (U.S. 2019/0251431 hereinafter Keskar).
As Claim 5, besides Claim 1, Taylor and Gupta in view of Iyer in further view of Uehara may not explicitly disclose wherein the plurality of iterations is no more than a total number of the plurality of identifiers.
Keskar teaches:
wherein the plurality of iterations is no more than a total number of the plurality of identifiers (Keskar (¶0065 line 13-16), “the location and placement of the joint training strategy intervals may be based on a number of training iterations ( e.g., a number of
training samples presented to the system) for each task type”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify iteration module of Taylor and Gupta in view of Iyer in further view of Uehara instead be a iteration module taught by Keskar, with a reasonable expectation of success. The motivation would be to allow “the performance metrics rapidly improve to values that are better than the performance metrics of the joint training strategy only approach of FIG.9B and more closely reach the performance metrics of the separately trained versions of system 300 in FIG. 9A” (Keskar (¶0071 last 5 lines)).
As Claim 6, besides Claim 5, Taylor, Gupta and Iyer in view of Uehara in further view of Keskar teaches wherein the plurality of iterations is equal to the total number of the plurality of identifiers (Keskar (¶0065 line 13-16), “the location and placement of the joint training strategy intervals may be based on a number of training iterations ( e.g., a number of training samples presented to the system) for each task type”).
As Claim 18, the Claim is rejected for the same reasons as Claim 6.
Claim(s) 10 is/are rejected under 35 U.S.C. 103 as being unpatentable over Taylor, Gupta and Iyer in view of Uehara in further view of Karnagel et al. (U.S. 2020/0327357 hereinafter Karnagel).
As Claim 10, besides Claim 1, Taylor and Gupta in view of Iyer in further view of Uehara may not explicitly disclose wherein the number of features is at least 100.
Karnagel teaches:
wherein the number of features is at least 100 (Karnagel (¶0034 line 1-3), system accommodates hundreds or thousands of features).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify features of Taylor and Gupta in view of Iyer in further view of Uehara instead be features taught by Karnagel, with a reasonable expectation of success. The motivation would be to allow system to accommodate rich sample data (Karnagel (¶0034 line 1-3)).
Claim(s) 11 is/are rejected under 35 U.S.C. 103 as being unpatentable over Taylor, Gupta, Iyer and Uehara in view of Karnagel in further view of Yang et al. (U.S. 2019/0311246 hereinafter Yang).
As Claim 11, besides claim 10, Taylor, Gupta and Iyer in view of Uehara in further view of Karnagel may not explicitly disclose wherein a number of weights within the first neural network and the second neural network is at least 30,000.
Karnagel teaches wherein a number of weights within the first neural network and the second neural network is at least 30,000 (Yang (¶0018 line 11-14), “Training a CNN model may require significant amount of computing power, even with a physical AI chip because a CNN model may include tens of thousands of weights.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify weights of Taylor, Gupta and Iyer in view of Uehara in further view of Karnagel in further view of Karnagel instead be weights taught by Yang, with a reasonable expectation of success. The motivation would be to allow “the weights in the CNN can easily vary and be loaded into the virtual AI chip without the cost associated with a physical AI chip” (Yang (¶0018 line 7-9)).
Response to Arguments
Claim Rejections – 35 U.S.C. §103:
As per Taylor, Applicant argues that Taylor’s level 1 feed data is not presented as a “tuple” in the manner required by the claim much less “a current state, an action, a calculated reward, and a next state” (first paragraph of page 9 in the remarks).
Applicants’ arguments are not persuasive. Taylor (¶0019 line 10-14, ¶0017 line 16-21, fig. 2(a)) teaches “The Options Price Reporting Authority (OPRA) feed is the most significant source of derivatives market data, and it belongs to the class of feeds known as "level 1" feeds. Level 1 feeds report quotes (current state), trades (an action), trade cancels (next state) and corrections (a calculated reward), and a variety of summary events”. Fig. 2(a) shows an example of order book feeds. In figure 2(a), each record (Order #) corresponds to a tuple (a row the database). Each record contain information corresponding to report quotes (current state), trades (an action), trade cancels (next state) and corrections (a calculated reward), and a variety of summary events.
As per Taylor, Applicant argues that Taylor does not disclose “wherein the current state and the next state are represented as a n-dimensional vectors … where n number is the number of features used to train the neural network” (second and third paragraph of page 9 in the remarks).
Applicants’ arguments are not persuasive. Taylor (¶0019 line 10-14, ¶0017 line 16-21, fig. 2(a)) teaches “The Options Price Reporting Authority (OPRA) feed is the most significant source of derivatives market data, and it belongs to the class of feeds known as "level 1" feeds. Level 1 feeds report quotes (current state), trades (an action), trade cancels (next state) and corrections (a calculated reward), and a variety of summary events”. Fig. 2(a) shows an example of order book feeds. In figure 2(a), each record (Order #) corresponds to a tuple (a row the database). Each record contain information corresponding to report quotes (current state), trades (an action), trade cancels (next state) and corrections (a calculated reward), and a variety of summary events. There are n-number of records which is the “n-dimension vectors” (each column in figure 2(a) is a “n-dimensional vectors”. “n-dimensional vector” is so broad that 1-dimenional or a column on the database meet the limitation). Applicants also mention notifications of correction in paragraph 0005 of Taylor. However, the argument is unrelated to correction column in the figure 2(a).
As per Arik, Applicant argues that Arik does not disclose steps b2 and b3 (second paragraph of page 10 in the remarks).
Applicants’ arguments are moot because Arik is no longer used in the current rejection. The limitations are taught by Gupta, Iyer and Uehara.
As per Arik, Applicant argues that Arik does not disclose steps b4 (first paragraph of page 11 in the remarks).
Applicants’ arguments are moot because Arik is no longer used in the current rejection. The limitations are taught by Gupta, Iyer and Uehara.
As per Taylor, Applicant argues that Taylor does not disclose “provisioning … a memory buffer that is concurrently readable and writable … (a2) writing, to the memory buffer, the plurality of tuples that have been obtained from multiple ones of the plurality of separate worker instances are written to the memory buffer concurrently” (first paragraph of page 11 in the remarks).
Applicant’s arguments are not persuasive. Taylor (¶0148 line 1-14) teaches that “parallel engines update and maintain the order and price aggregation data structures in parallel. In one embodiment, the data structures are maintained in the same physical memory. In this case, the one or more order engines (worker instances) and one or more price engines (worker instants) interleave their accesses to memory, masking the memory access latency of the memory technology and maximizing throughput of the system.” Parallel engines update and maintain the order and price aggregation data structures in parallel. Each order is construed as “order #” in figure 2(a). Like responses to arguments above, each row in figure 2(s) is a tuple including report quotes (current state), trades (an action), trade cancels (next state) and corrections (a calculated reward), and a variety of summary events.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Kim et al. (U.S. 2024/0126971) teaches a system and method to optimize Q-value.
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to NHAT HUY T NGUYEN whose telephone number is (571)270-7333. The examiner can normally be reached M-F: 12:00-8:00 EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Viker Lamardo can be reached at 571-270-5871. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/NHAT HUY T NGUYEN/Primary Examiner, Art Unit 2147
Prosecution Insights