Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
The office action is in response to the application filed on January 24, 2024.
Claims 1-20 are pending and have been examined. Claims 1-20 are rejected.
Claim Objections
Claim 19 is objected to because of the following informalities:
"plurality or RL agents" should read "plurality of RL agents".
Appropriate correction is required.
Claim 20 is objected to because of the following informalities:
" The non-transitory computer-readable storage medium" should read " The non-transitory computer-readable medium".
Appropriate correction is required.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless – (a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1, 2, 10, 11, 19, and 20 are rejected under 35 U.S.C. 102(a)(2) as being anticipated by Saba et. Al, (U.S PATENT No. US12,141,512Bl, filed on September 30, 2021, hereinafter "SabaUVM"), whose date is before the effective filing date of the instant application, i.e., January 24, 2024. Therefore, SabaUVM constitutes prior art under 35 U.S.C. 102(a)(2).
With respect to Claims 1, 10, and 19:
SabaUVM teaches:
“receiving a current state of a testbench environment for an integrated circuit (IC) design;” ([col. 12 lines 5-15] discloses receiving a current state of a testbench environment, “The machine learning model receives status information as an input (e.g., in the form of a vector) from the machine learning state manager 713 which can take a snapshot of the state of the ML environment. The machine learning state manager 713 monitors the protocol agents and captures information representing their state (e.g., configuration parameters, state machine status, memory, and/or register contents)).” [col. 1 lines 45-55] further discloses that this receiving of a current state is in relation to IC design, “…a number of verifications steps implemented during the design of an integrated circuit. Verification can be performed to verify whether a design meets a functional description, is compliant with one or more protocols, or does not violate any layout rules. For example, verification can be implemented based on a hard ware description language (e.g., VHDL or Verilog), at the circuit schematic level (e.g., RTL-register transistor level), or at the layout level.” [col. 11 lines 15-17] further teaches that this IC design process can occur in a testbench environment, “A UVM test 700a defines a test scenario for a testbench. The test itself is defined by at least a test class that contains the environment, configuration properties, and possibly class overrides.”)
“determining, via an RL model, policy, or function of the RL agent, an action to be applied to the testbench environment based on the current state;” ([col. 4 lines 5-35] discloses utilizing a machine learning model to determine and select an action to be applied to the testbench environment based on the current state, “FIGS. 10A-10B illustrate example flows corresponding to machine learning model processing of the current state according to some embodiments...As illustrated in FIG. l0A, at 1002, an input vector representing at least a subset of the current state information is input into the machine learning model (see e.g., 711)…in FIG. 10B, at 1005, a determination can be made before selecting an action at 1010 as to whether each action corresponding to the output from the machine learning model can be run. For example, an action might be subject to one or more rules that specify that said action can only be run when one or more conditions are met, or an action cannot be run when any one or more of a set of conditions are met.” [col. 6 lines 60-65, col. 7 lines 1-5] further teaches determining an action to be applied to the testbench environment, “The machine learning exploration unit generally comprises a machine learning manager that can instantiate a machine learning model that selects one or more actions or action sequences to be executed on a design under test. The machine learning manager can also analyze those actions or action sequences and corresponding state information to generate outputs representing different states identified and the actions or action sequences that were taken to achieve those states.”)
“transmitting the action to the testbench environment;” ([col. 7 lines 19-25] discloses transmitting the action to the testbench environment, “When, the machine learning model selects one or more actions or a sequence of actions they are transmitted (either directly or indirectly---e.g., by reference) to the simulator 370 (see action(s) 394) to be executed at the design under test 371.”) Examiner’s Note: The simulator module [Fig. 3A] encompasses design under tests (DUT) which further embodies the specified UVM testbench environment [Fig. 7A-7B].
“in response to transmitting the action, receiving from the testbench environment a reward value and a new state of the testbench environment;” ([col. 12 lines 32-46] discloses that as a result of transmitting the action, a reward value and new state of the testbench environment is received, “The output of the machine learning model is mapped by the machine learning sequencer to the corresponding action and transmitted to the correct protocol agent. Each action selected may cause a corresponding state change. This is monitored using the associated UVM monitor (e.g., 723a) and tracked for purposes of at least applying a reward to the machine learning model which may be based on whether the action achieved a desired target state as discussed herein. The reward is implemented using at least the machine learning rewarder 712 to apply one or more rules to the collected state information in order to determine an applicable reward, where the reward is intended to encourage the machine learning model to achieve its goals by rewarding progress and success, and possibly by penalizing failure to achieve a positive change.”)
“and training the RL model, policy, or function based on the reward value, the current state, and the action, the training causing the RL model, policy, or function to learn mappings between states of the testbench environment and actions to be applied to the testbench environment that maximize the reward value over time.” ([col. 12 lines 61-67] teaches training the reinforcement learning model based on a reward value, current state, and the associated action, “After an action or sequence of actions has been selected based on the result of the machine learning model, a machine learning rewarder 712 can process the current and previous states using one or more rules to select a reward to be applied to the machine learning model. In this way the machine learning model is trained using online inference training.” [col. 12 lines 48-54] further teaches that the training of the reinforcement learning model enables additional learning/mapping that once applied to the testbench environment, will maximize the reward value, “…machine learning sequence library 736m uses the machine learning model to select the next action or sequence of actions to be performed. For example, the output of machine learning model is mapped to different actions or sequences of action where the magnitude of each output corresponds to the machine learning models prediction of the probability that the corresponding action is correct.” [col. 4 lines 47-50] further teaches an instance of maximizing the reward value over time, “For instance, a reward of 0.001 could be used to encourage discovery of new states, or a reward of 1 could be used when the desired state is achieved.”)
Therefore, Claims 1, 10, and 19 are rejected.
With respect to Claims 2 and 11:
SabaUVM teaches:
“wherein the action comprises a set of values that correspond to input stimuli to be provided as input to the IC design.” ([col. 11 lines 21-24] teaches action-related input stimuli being utilized by the IC design environment (UVM test), “The UVM test 700a generally operates by apply sequences of stimuli to one or more components in a design under test via sequence calls.” [col. 4 lines 61-66] further teaches the notion of a set of values associated with the input stimuli, “The state space library manager 112 collects information garnered from the machine learning exploration unit for corresponding designs. For example, the state space library manager maintains data representing each state (as corresponding to a set of variables and their associated unique combination of values).” [col. 12 lines 22-31] further teaches a set of values that correspond to input stimuli (input computations) “…and process accordingly for input into the machine learning mode 711. When a stimulus is applied to a machine learning model a series of computations occur to determine an output. For example, multiple inputs are applied to a machine learning model have multiple layers. At each layer one or more computations take place. Those computations eventually flow to an output stage where the results are ready for interpretation. For instance, each output might comprise a probability that a selection corresponding to that probability is the correct selection.”) Examiner’s Note: “Output” is directly correlated with the action.
Therefore, Claims 2 and 11 are rejected.
With respect to Claim 20:
SabaUVM teaches:
“wherein the RL agent is associated with an interface of the IC design and wherein the action determined by the RL agent corresponds to input stimuli to be input to the IC design via the interface.” ([col. 11 lines 40-53] discloses the RL agent’s (protocol agent) association with an interface of the IC design (protocol interface) in which the determined action corresponds to input stimuli that will be implemented by said interface, “Each protocol agent (see protocol agents 720a-n) includes a UVM sequencer, a UVM driver, and a UVM monitor. For example, UVM sequencer 721a can operate independently to select sequences, or can operate at the direction of the virtual sequencer 730a. The UVM driver 722a implements the selected action from 721a on the protocol interface 725a (see also 72Sa-n). The UVM monitor 723a monitors the protocol interface (725a) to receive any results of the selected action. Each protocol agent can operate independently (generally by randomly selecting actions) or may operate in a coordinated manner as controlled by the virtual sequencer 730a. Additionally, each protocol agent is coupled to a corresponding protocol interface that enables the implementation of the selected action at the design under test.”)
Therefore, Claim 20 is rejected.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or non-obviousness.
Claims 3 and 12 are rejected under 35 U.S.C. 103 as being unpatentable over Saba et. Al, (U.S PATENT No. US12,141,512Bl, filed on September 30, 2021, hereinafter "SabaUVM"), in view of Nagaraja, (U.S PGPUB No. US 20180260498 A1, filed on January 01, 2018). Both disclosure dates are before the effective filing date of the instant application, i.e., January 24, 2024. Therefore, SabaUVM and Nagaraja constitute prior art under 35 U.S.C. 103.
With respect to Claims 3 and 12:
SabaUVM alone does not appear to explicitly disclose the extent of the claim limitations:
However, Nagaraja teaches:
“wherein the input stimuli pertain to a particular interface of the IC design that is associated with the RL agent, and wherein each RL agent in the plurality of RL agents is associated with a different interface of the IC design.” (Paragraph [0098] teaches that the input stimuli pertain to a particular interface (design-related libraries/databases) of the IC design (chip design configuration) which is associated with the RL agent (multi-objective AI agent) of a plurality of RL agents that are further associated with different interfaces (plurality of domains), “The SoC specification input is received (210). An abstract graph database libraries are created to store details regarding a chip design technology and standards involved and required for a chip designing process…After initialization, AI are setup for various domains in the form of MDPs, SMDPs, HAMs and MAX-Q to further optimize and generate desired SoC configuration. The AI setup represents a Multi objective AI agent. The AI setup for a domain includes multi-objective AI agents with environment, and a plurality of tasks. The AI setup for a domain is executed and repeated for a plurality of domains such as first domain, second domain, and a third domain (213, 214, 215). Then a multi objective AI environment, agents and tasks are set up by compiling the data received from the AI set up for the plurality of domains (216).”)
SabaUVM-Nagaraja are analogous art and in the same field of invention because both references pertain to automating and streamlining highly complex phases of the chip design and engineering verification lifecycle. While SabaUVM teaches automatically generating the perfect sequence of stimuli (tests) to stress-test a Design Under Test (DUT) and find bugs faster, Nagaraja teaches autonomously discovering the best structural chip design, balancing performance by evaluating domains, subdomains, and application specific instructions. It would have been obvious to a person having ordinary skill in the art (PHOSITA) before the effective filing date of the claimed invention to implement the base reference of SabaUVM (generating action sequences for verification) with the teachings of Nagaraja (optimal System on Chip (SoC) architecture construction) in order to replace manual heuristics with self-learning RL agents to drastically reduce time-to-market, minimize human error, and discover highly robust chip layouts and testing parameters that human engineers might overlook. One of ordinary skill in the art would be motivated to do so because by integrating Nagaraja's framework into the methods of SabaUVM one would be able to note that a system as such would be beneficial, "for optimizing various power components of the SoC such as leakage and dynamic power and decreasing effects of ageing in hardware due to electron migration on hardware performance using a SoC Power Agent 504, {[0108] of Nagaraja}."
Therefore, Claims 3 and 12 are rejected.
Claims 4 and 13 are rejected under 35 U.S.C. 103 as being unpatentable over SabaUVM, in view of Walsh et. Al (U.S PGPUB No. US20230237370Al, filed on February 08, 2022, hereinafter “Walsh”), in further view of Green (U.S PGPUB No. 20210264085A1, filed on May 21, 2020). All disclosure dates are before the effective filing date of the instant application, i.e., January 24, 2024. Therefore, SabaUVM, Walsh, and Green constitute prior art under 35 U.S.C. 103.
With respect to Claims 4 and 13:
SabaUVM alone does not appear to explicitly disclose the extent of the claim limitations:
However, Walsh teaches:
“wherein the reward value indicates whether application of the action to the testbench environment resulted in an improvement -----------.” (Paragraph [0003] introduces the notion of applying an action to the testbench environment to gain an improvement in coverage metrics for the IC design through the presence of a reward value, “The agent 100 then takes an action 106 which transitions it to a new state and produces a reward term. Various existing RL algorithms and models 108 provide routines for eventually finding an optimal policy 110, a mapping from states to actions, that will maximize some function (such as the expected sum) of the reward terms.” Paragraph [0011] further teaches an application of the action indicating progress measured by a reward value, “comprising learning a policy that selects an action based on observations by the agent and based on a value function that estimates a future reward for each possible action; mapping core actions of the agent to either a changing velocity dimension and a steering dimension…” Paragraph [0072] further teaches applying an action to improve the IC design, “the progress reward alone may not be enough to incentivize the agent to win the race. If the opponent was sufficiently fast, the agent would learn to follow it and accumulate large rewards without risking potentially catastrophic collisions. Adding rewards specifically for passing can help the agent learn to overtake other cars. A passing reward can be used that is proportional to the distance by which the agent improved its position relative to each opponent within the local region. The reward can be symmetric, so if an opponent gained ground on the agent, the agent would see a proportional negative reward.”)
SabaUVM-Walsh are analogous art and in the same field of invention because both references pertain to utilizing Reinforcement Learning to train artificial intelligence agents through targeted scenarios, progressive training phases, and reward structures. While SabaUVM teaches finetuning the testing of microchips or software by training models to hit specific hardware target states, Walsh teaches creating a highly skilled racing AI agent capable of winning complex tactical races by training it through a dynamic curriculum. It would have been obvious to a person having ordinary skill in the art (PHOSITA) before the effective filing date of the claimed invention to implement the base reference of SabaUVM (Design Under Test Environment) with the teachings of Walsh (autonomous driving modular RL architecture) in order to solve the sample inefficiency and sparse reward problems that typically plague standard reinforcement learning. One of ordinary skill in the art would be motivated to do so because by integrating Walsh's framework into the methods of SabaUVM one would be able to note that, "These results gave the agent an overall score of 104-52 over the humans. The results clearly show that the agent outraced the best human drivers in the world. The agent combined impressive speed with real racing skills, success fully passing top drivers on straights and in curves, {[0090] of Walsh}."
The combination of SabaUVM-Walsh does not appear to explicitly disclose:
“-------improvement in one or more coverage metrics for the IC design.”
However, Green teaches:
“-------improvement in one or more coverage metrics for the IC design.” (Paragraph [0005] discloses improving coverage metrics (functional coverage goals) for the IC design, “The functional verification process includes a process to achieve functional coverage closure. Functional coverage closure is achieved when the designer can prove that all of the intended verification goals (i.e., functional coverage goals) have been achieved by one or more test scenarios developed by the designer or some other engineer who understands the functionality of the electronic system.” Paragraph [0050] further discloses improving/meeting coverage metrics for the IC design, “The learning test generator may be configured to train itself based on the inputs generated by the formal analysis engine, the outputs generated as a result of those inputs, and whether or not the coverage goals are met. After the training goal is achieved, a plurality of tests are generated and then integrated into the Design Under Test (DUT) simulation to achieve coverage of all goals. Herein, reference to any combination of the aforementioned coverage goals can be referred to as verification goals, or functional coverage goals, coverage goals, or simply test goals.”) Examiner’s Note: IC design in this embodiment pertains to chip design.
SabaUVM-Walsh-Green are analogous art and in the same field of invention because all three references pertain to bridging the gap between abstract RL theory and practical engineering, ensuring the agent can integrate directly into traditional, industry-standard workflows (like UVM testbenches). While SabaUVM teaches addressing sub-goals by generating complex sequences of stimulus actions to ensure the agent is rewarded frequently for hitting intermediate milestones, Walsh teaches enhancing the training environment to dynamically scale up processes in difficulty (variable opponents/tactical scenarios) based on the agent’s performance. Similarly, Green teaches developing the design rules and RL framework needed to maximize coverage. It would have been obvious to a person having ordinary skill in the art (PHOSITA) before the effective filing date of the claimed invention to implement the base reference of SabaUVM (Multi-Model Ensembles & Reward Shaping) with the teachings of Walsh (Multi-Model Ensembles & Reward Shaping) and the teachings of Green (End-to-End Autonomous Agents) in order to accelerate agent convergence and improve generalization in complex environments. One of ordinary skill in the art would be motivated to do so because by integrating Walsh and Green’s frameworks into the methods of SabaUVM one would be able to recognize that a system as such could, "achieve greater accuracy on large data sets than if only a single computing device were employed to implement the learning process. Distributed learning enables the learning process to train itself without being limited by the computer speed, stability, or storage capability of a single computing device, {[0098] of Green}…more accurately predict goals (i.e., from the learning test generator) and discriminate (i.e., learner) between predicted goals and goals actually achieved, {[0069] of Green}."
Therefore, Claims 4 and 13 are rejected.
Claims 5, 6, 14, and 15 are rejected under 35 U.S.C. 103 as being unpatentable over SabaUVM, in view of Green (U.S PGPUB No. 20210264085A1, filed on May 21, 2020). Both disclosure dates are before the effective filing date of the instant application, i.e., January 24, 2024. Therefore, SabaUVM, and Green constitute prior art under 35 U.S.C. 103.
With respect to Claims 5 and 14:
SabaUVM teaches:
“wherein the reward value is a positive value in a scenario where application of the action to the testbench environment resulted in an improvement --------” ([col. 7 lines 25-34] discloses the presence of positive reward value-based improvement results for the IC design, “The training of the machine learning model is further implemented using a rewards module. As suggested by the name, the rewards module is used to encourage a movement made by the machine learning model. For example, a reward might be given when the machine learning model causes the discovery of a new state (or merely a change in state) or when a target state is reached. The reward for reaching a new state might be smaller than a reward that is given when a target state is reached (e.g., 0.001 for a new state, and 1.0 for reaching the target state).” [col. 5 lines 58-65] further discloses that these improvement results are associated with the IC design, “The verification environment 114 includes a testing environment where some or all test sequences are executed against a design. For example, a set of test sequences are executed, and verification is performed to determine whether the expected behavior occurred, e.g., state changes, read, write, output, or other operations. In some embodiment, the verification environment can process a set of test sequences to determine the coverage of those test sequences…[col. 6 lines 1-2], “can be analyzed to determine the cover for different functional behavior (e.g., protocols).”)
SabaUVM does not appear to explicitly disclose:
“--------- in one or more coverage metrics for the IC design.”
However, Green teaches:
“-------- in one or more coverage metrics for the IC design.” (Paragraph [0005] discloses improving coverage metrics (functional coverage goals) for the IC design, “The functional verification process includes a process to achieve functional coverage closure. Functional coverage closure is achieved when the designer can prove that all of the intended verification goals (i.e., functional coverage goals) have been achieved by one or more test scenarios developed by the designer or some other engineer who understands the functionality of the electronic system.” Paragraph [0050] further discloses improving/meeting coverage metrics for the IC design, “The learning test generator may be configured to train itself based on the inputs generated by the formal analysis engine, the outputs generated as a result of those inputs, and whether or not the coverage goals are met. After the training goal is achieved, a plurality of tests are generated and then integrated into the Design Under Test (DUT) simulation to achieve coverage of all goals. Herein, reference to any combination of the aforementioned coverage goals can be referred to as verification goals, or functional coverage goals, coverage goals, or simply test goals.”) Examiner’s Note: IC design in this embodiment pertains to chip design.
SabaUVM-Green are analogous art and in the same field of invention because both references pertain to applying Reinforcement Learning to automate and optimize constrained-random verification (CRV). While SabaUVM teaches using intelligent, autonomous RL agents to explore hardware designs more efficiently, Green teaches focusing on a factory-style pipeline that uses a learning configurator and a pre-trained learning test generator to output a single, ready-to-integrate agent that automatically hits a broad set of functional coverage goals. It would have been obvious to a person having ordinary skill in the art (PHOSITA) before the effective filing date of the claimed invention to implement the base reference of SabaUVM (Multi-Model Modular Approach) with the teachings of Green (End-to-End Configurator Approach) in order to replace or augment manual, random test generation with intelligent, automated test generation to achieve functional coverage faster. One of ordinary skill in the art would be motivated to do so because by integrating Green's framework into the methods of SabaUVM one would be able to note that a system as such aims, "to achieve more accurate predictions based on the learning processes encompassed in the GAIL step, {[0081] of Green}."
Therefore, Claims 5 and 14 are rejected.
With respect to Claims 6 and 15:
Saba teaches:
“wherein the reward value is a zero or negative value in a scenario where application of the action to the testbench environment resulted in no improvement ----------” ([col. 7 lines 34-38] discloses the presence of negative reward value-based results for the IC design, “Additionally, a negative reward might be provided when a reset state is applied (e.g., -1.0) to avoid the machine learning model from selecting actions that might loop or otherwise result in repeatedly achieving the same state.” [col. 5 lines 58-65] further discloses that these improvement results or (lack thereof) are associated with the IC design, “The verification environment 114 includes a testing environment where some or all test sequences are executed against a design. For example, a set of test sequences are executed, and verification is performed to determine whether the expected behavior occurred, e.g., state changes, read, write, output, or other operations. In some embodiment, the verification environment can process a set of test sequences to determine the coverage of those test sequences…[col. 6 lines 1-2], “can be analyzed to determine the cover for different functional behavior (e.g., protocols).”)
SabaUVM does not appear to explicitly disclose:
“---------in one or more coverage metrics for the IC design.”
However, Green teaches:
“--------in one or more coverage metrics for the IC design.” (Paragraph [0005] discloses improving/not improving coverage metrics (functional coverage goals) for the IC design, “The functional verification process includes a process to achieve functional coverage closure. Functional coverage closure is achieved when the designer can prove that all of the intended verification goals (i.e., functional coverage goals) have been achieved by one or more test scenarios developed by the designer or some other engineer who understands the functionality of the electronic system.” Paragraph [0050] further discloses improving/meeting/not-meeting coverage metrics for the IC design, “The learning test generator may be configured to train itself based on the inputs generated by the formal analysis engine, the outputs generated as a result of those inputs, and whether or not the coverage goals are met. After the training goal is achieved, a plurality of tests are generated and then integrated into the Design Under Test (DUT) simulation to achieve coverage of all goals. Herein, reference to any combination of the aforementioned coverage goals can be referred to as verification goals, or functional coverage goals, coverage goals, or simply test goals.” Paragraph [0080] further details circumstances related to not meeting coverage metrics for the IC design, “Whether the predicted goals achieved by the learning test generator satisfy all of the predetermined goals that must be achieved to end the training process? If the answer to either question is “no”, the parameters that control the prediction mechanism of the learning test generator may be updated based on the GAIL algorithm.”) Examiner’s Note: IC design in this embodiment pertains to chip design.
Therefore, Claims 6 and 15 are rejected.
Claims 7 and 16 are rejected under 35 U.S.C. 103 as being unpatentable over SabaUVM, in view of Saba et. Al (U.S PATENT No. US12,038,477Bl, filed on September 30, 2021, hereinafter “SabaPro”). Both disclosure dates are before the effective filing date of the instant application, i.e., January 24, 2024. Therefore, SabaUVM and SabaPro constitute prior art under 35 U.S.C. 103.
With respect to Claims 7 and 16:
SabaUVM alone does not appear to explicitly disclose the extent of the claim limitations:
However, SabaPro teaches:
“setting the new state as the current state; and repeating the determining and transmitting of the action, the receiving of the reward value and the new state, and the training of the RL model, policy, or function until all coverage goals for the IC design are met.” ([col. 11 lines 63-67, col. 12 lines 1-9] details the process in which a current target/state becomes a new target/state, “If at 725 it is determined that no new targets were discovered at 725 then the process will continue to 728 where the currently selected target is marked as processed. However, if the determination is that one or more targets were discovered, those targets can be added to the management data structure discussed in regard to 704 before proceeding to 728 where the selected target is marked as being processed. Once the currently selected target has been processed, the flow returns to 705, where based on then current conditions a determination is made as to whether there are targets that have not yet been processed. If there are unprocessed targets the flow will continue to loop until all targets have been processed subject to the limitations discussed in regard to 709.” [col. 7 lines 1-5] discloses determining and transmitting the action, “When, the machine learning model selects one or more actions or a sequence of actions they are transmitted (either directly or indirectly---e.g., by reference) to the simulator 370 (see action(s) 394) to be executed at the design under test 371.” [col. 7 lines 10-13] further teaches receiving a reward value and a new state, “For example, a reward might be given when the machine learning model causes the discovery of a new state (or merely a change in state) or when a target state is reached.” [col. 14 lines 20-26] further teaches training the machine learning model until all coverage goals are met, “At 811, it is determined whether a stop criteria has been met. If the stop criteria has been met the process ends at 812. However, if the stop criteria has not been met then the process returns to 806 where another target is selected for sequence generation. The stop criteria might comprise any of whether all target states (excluding the reset state) have been processed…” [col. 5 lines 40-52] further ensures that IC design related coverage goals are expected to be met and achieved by the machine learning model, “The verification environment 114 includes a testing environment where some or all test sequences are executed against a design…In some embodiment, the verification environment can process a set of test sequences to determine the coverage of those test sequences…sets of test sequences can be analyzed to determine the cover for different functional behavior (e.g., protocols).”) Examiner’s Note: A set of test sequences comprise of a minimum number of combinable actions [col. 6].
SabaUVM-SabaPro are analogous art and in the same field of invention because both references pertain to using multi-model architectures and shifting from random to guided exploration to break down complex verification goals into smaller, state-specific sub-goals. While SabaUVM teaches using intelligent, autonomous RL agents to explore hardware designs more efficiently and drastically reducing the simulation time wasted on redundant cycles, SabaPro teaches generating precise stimulus sequences to hit any desired verification target. It would have been obvious to a person having ordinary skill in the art (PHOSITA) before the effective filing date of the claimed invention to implement the base reference of SabaUVM (Multi-Model Modular Approach) with the teachings of SabaPro (Map Building/Curriculum & Novelty Exploration) in order to maximize design state coverage and automate test generation. One of ordinary skill in the art would be motivated to do so because by integrating SabaPro's framework into the methods of SabaUVM one would be able to note that a system as such, "improves the reproducibility, reliability, and results using machine learning models, {[col. 17 lines 28-30] of SabaPro}."
Therefore, Claims 7 and 16 are rejected.
Claims 8 and 17 are rejected under 35 U.S.C. 103 as being unpatentable over SabaUVM, in view of Walsh et. Al (U.S PGPUB No. US20230237370Al, filed on February 08, 2022, hereinafter “Walsh”), in further view of Caicedo et. Al (Active Object Localization with Deep Reinforcement Learning, published November 18, 2015, hereinafter “Caicedo”). All disclosure dates are before the effective filing date of the instant application, i.e., January 24, 2024. Therefore, SabaUVM, Walsh, and Caicedo constitute prior art under 35 U.S.C. 103.
With respect to Claims 8 and 17:
SabaUVM alone does not appear to explicitly disclose:
“wherein the current state includes a concatenated sequence of one or more prior actions determined by the RL agent.”
However, Walsh teaches:
“wherein the current state includes a concatenated sequence-----------” ([0084] discloses a concatenated sequence of inputs determined by the RL agent and relative to the current state, “The state features input to the neural networks are either directly available from the game state or processed into more convenient forms and concatenated before being input to the models for training. Features include but are not limited to the following. A set of time trial features captured the car's 3D velocity, 3D angular velocity, 3D g-force, load on each tire, and the tire slip angles, sine and cosine components, the local course surface inclination, the car's orientation with respect to the course center line, and a set of course points outlining upcoming sections of track. The agent also received indicators if it contacted a fixed barrier, was considered off-course by the game, and received real valued values for the game's view of the car's most recent steering angle, throttle intensity, and brake intensity.”) Examiner’s Note: The term “state” can encompass the agent’s current and or past observations/actions [0047]. The actions further being an extension of outstanding information in pertinence to a probability distribution of the agent’s actions [0046].
SabaUVM-Walsh are analogous art and in the same field of invention because both references pertain to utilizing Reinforcement Learning to train artificial intelligence agents through targeted scenarios, progressive training phases, and reward structures. While SabaUVM teaches leveraging multiple trained models to generate diverse action sequences, ensuring thorough exploration to "verify functionality" of a design, Walsh teaches varying the number, configuration, and behavior of opponents to prevent the agent from overfitting to a single driving style, forcing it to learn a robust, generalizable policy. It would have been obvious to a person having ordinary skill in the art (PHOSITA) before the effective filing date of the claimed invention to implement the base reference of SabaUVM (Design Under Test Environment) with the teachings of Walsh (autonomous driving modular RL architecture) in order to implement divide-and-conquer skill decomposition (or curriculum/multi-model learning). One of ordinary skill in the art would be motivated to do so because by integrating Walsh's framework into the methods of SabaUVM one would be able to note that, "these results gave the agent an overall score of 104-52 over the humans. The results clearly show that the agent outraced the best human drivers in the world. The agent combined impressive speed with real racing skills, success fully passing top drivers on straights and in curves, {[0090] of Walsh}."
The combination of SabaUVM-Walsh does not appear to explicitly disclose:
“------------of one or more prior actions determined by the RL agent.”
However, Caicedo teaches:
“------------of one or more prior actions determined by the RL agent.” ([pg. 3, sec. 3.2] discloses that the current state encompasses a sequence/history/log of prior actions, “The state representation is a tuple (o,h), where o is a feature vector of the observed region, and h is a vector with the history of taken actions.” [pg. 4, Fig. 3] further discloses that this sequence of prior actions is concatenated, “The output of the CNN is concatenated with the action history vector to complete the state representation. It is processed by the Q-network which predicts the value of the 9 actions.” [pg. 7, sec. 5.3] further teaches the determination of these prior action sequences by the RL agent, “Figure 6 shows sequences of attended regions as seen by the agent, as well as the actions selected in each step. Notice that the actions chosen attempt to keep the object in the center of the box, and also that the final object appears to have normalized scale and aspect ratio.”)
SabaUVM-Walsh-Caicedo are analogous art and in the same field of invention because all three references pertain to improving sequential decision-making tasks historically reliant on human heuristics or exhaustive brute-force search. While SabaUVM teaches solving the "state-space explosion" problem in hardware testing by using RL to intelligently find bugs and coverage gaps instead of using random test generation, Walsh teaches utilizing an automated training schedule that adjusts the difficulty and mix of scenarios based on the agent's real-time performance metrics. Similarly, Caicedo teaches bridging reinforcement learning with computer vision by providing a dynamic, attention-based alternative to static, computationally heavy sliding-window or region-proposal object detection methods. It would have been obvious to a person having ordinary skill in the art (PHOSITA) before the effective filing date of the claimed invention to implement the base reference of SabaUVM (Multi-Model Ensembles & Reward Shaping) with the teachings of Walsh (Multi-Model Ensembles & Reward Shaping) and the teachings of Caicedo (Object Localization) in order to utilize action-reward feedback loops to train agents to reach specific target states through a series of goal-directed progressive steps. One of ordinary skill in the art would be motivated to do so because by integrating Walsh and Caicedo’s frameworks into the methods of SabaUVM one would be able to recognize that a system as such could, "improving average precision by approximately 3 percent points. The history vector also works better than appending a few more frames to the state representation with the additional benefit of increasing dimensionality by a negligible factor, {[pg. 3, sec. 3.2] of Caicedo}."
Therefore, Claims 8 and 17 are rejected.
Claims 9 and 18 are rejected under 35 U.S.C. 103 as being unpatentable over SabaUVM, in view of Walsh et. Al (U.S PGPUB No. US20230237370Al, filed on February 08, 2022, hereinafter “Walsh”). Both disclosure dates are before the effective filing date of the instant application, i.e., January 24, 2024. Therefore, SabaUVM and Walsh constitute prior art under 35 U.S.C. 103.
With respect to Claims 9 and 18:
SabaUVM alone does not explicitly disclose the extent of the claim limitations:
However, Walsh teaches:
“wherein the RL model, policy, or function of the RL agent was previously trained using a replay buffer, the replay buffer comprising information pertaining to a suite of test cases that was previously executed against the testbench environment.” (Paragraph [0052] discloses the replay buffer encompassing information relevant to previously executed training processes in the testing environment, “Data from those workers can be streamed back to an experience replay buffer on a centralized trainer which can perform computations to update the learning agent's policy and other models, as is standard in Deep Reinforcement Learning architectures. When workers complete rollout tasks, a new scenario can be drawn from the set of candidate scenarios and then its parameters, such as launch conditions and opponent types, can be further sampled.” Paragraph [0056] further showcases that the RL model was previously trained using a replay buffer, “The trainer 302 can refine the models 306 and policy 308 used to determine actions in the environment. Various representations of these models 306 and policies 308 are possible, including deep neural networks. Policy refinement can be performed by sampling a "batch" of data 310 from an experience replay buffer 312 that has been populated with data from the various scenarios that have been run in the past.”)
SabaUVM-Walsh are analogous art and in the same field of invention because both references pertain to utilizing Reinforcement Learning to train artificial intelligence agents through targeted scenarios, progressive training phases, and reward structures. While SabaUVM teaches Reward Shaping, explicitly rewarding models when an action sequence causes beneficial movement toward a target state, Walsh teaches creating a highly skilled racing AI agent capable of winning complex tactical races by training it through a dynamic curriculum. It would have been obvious to a person having ordinary skill in the art (PHOSITA) before the effective filing date of the claimed invention to implement the base reference of SabaUVM (Design Under Test Environment) with the teachings of Walsh (autonomous driving modular RL architecture) in order to accelerate learning and improve agent policy robustness for complex environments. One of ordinary skill in the art would be motivated to do so because by integrating Walsh's framework into the methods of SabaUVM one would be able to note that, "these results gave the agent an overall score of 104-52 over the humans. The results clearly show that the agent outraced the best human drivers in the world. The agent combined impressive speed with real racing skills, success fully passing top drivers on straights and in curves, {[0090] of Walsh}."
Therefore, Claims 9 and 18 are rejected.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure: Liu et. Al (A Multi-Agent Generative AI Framework for IC Module-Level Verification Automation), Yan et. Al (WO2022183143A2), and Lee et. Al (US20200364314A1).
Any inquiry concerning this communication or earlier communications from the examiner should be directed to NOOR F CHEEMA whose telephone number is (571)272-9642. The examiner can normally be reached Monday-Friday 7:30am-5:00pm alternative Fridays off.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Mariela Reyes can be reached at (571) 270-1006. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/N.F.C./ Examiner, Art Unit 2142 /Mariela Reyes/Supervisory Patent Examiner, Art Unit 2142
Prosecution Insights