DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claims 1-20 are pending for examination.
This Office Action is Non-Final.
Priority
Acknowledgment is made of applicant's claim for foreign priority based on an application filed in IN on 12/21/2023 and 10/04/2024. It is noted, however, that applicant has not filed a certified copy of the IN202321087767 and IN202421075245 applications as required by 37 CFR 1.55.
Specification
The disclosure is objected to because of the following informalities:
[0024] line 6: change to "along with random [[testcase]] test cases"
Appropriate correction is required.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claims 1-4, 9-14, and 19-20 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Zhang et al. (NPL: "LLM4DV: Using Large Language Models for Hardware Test Stimuli Generation," hereinafter “Zhang”).
Regarding Claim 1, Zhang discloses a method (Page 2, Fig. 1b; Page 3: LLM4DV framework and method), comprising:
generating, by a processor of an apparatus (Page 5, Section 4: "With Verilator and cocotb, we could efficiently simulate and test the modules in Python." One of ordinary skill in the art would understand that running LLM4DV and various programs requires a processor and a computing apparatus), one or more random test cases with an aid of a generative artificial intelligence (GAI)-assisted random verification program (GRVP) framework (Page 2: "We establish a benchmarking framework named LLM4DV (Large Language Model for Design verification… we propose and develop a framework that uses prompted LLMs to generate test stimuli for hardware designs." Page 2, Fig. 1b, Page 3, Fig. 2 and Pages 5-6, Sections 4.1-4.3: generates test stimuli using pure random, coverpoint type-based, and mixed coverpoint type-based sampling to expose uncovered bins. Page 9, B1: all three missed-bin sampling methods either are or eventually use random sampling. Thus, the generated test stimuli by LLM4DV are random test stimuli to expose the randomly-sampled uncovered bins); and
performing, by the processor (see above details on LLM4DV + required processor to run various programming languages and libraries), validation testing on a design of a processor using the one or more random test cases (Page 2, first bullet: "We design and construct three DUT modules: a Primitive Data Prefetcher Core, an Ibex Instruction Decoder, and an Ibex CPU." The DUT modules are designs of components of a processor (i.e., Primitive Data Prefetcher, Instruction Decoder) or a processor design itself (i.e., CPU). Page 2, third bullet and Page 8, algorithm 1: “We show that our framework with optimal improvement measures achieves 98.94%, 86.19%, and 5.61% coverage rates on these [DUT] modules." Coverage rates are achieved by performing design verification using the aforementioned (random-based) test stimuli on the DUT modules).
Regarding Claim 2, Zhang discloses the method of Claim 1, as referenced above, wherein the generating of the one or more random test cases comprises utilizing a retrieval augmented generator (RAG) in the GRVP framework (Page 5, Section 4; Page 6, Section 4.2; Page 11, Section D.2: LLM4DV framework uses OpenAI’s GPT 3.5 pre-trained with RISC-V knowledge) to perform operations comprising:
receiving a template-based user input along with one or more random feature specifications (Page 4, Section 3.2: receives coverage-feedback template, including iterative user query messages containing “a list of uncovered bins.” Page 4-5, Section 3.3; Page 9, Section B.1: the uncovered bins will later be randomly sampled to improve test stimuli generation);
receiving one or more application programming interfaces (APIs) (Page 11, Section D.2: GPT 3.5 and Llama 2 are pre-trained with RISC-V knowledge, including ALU operation bins ADD, ADDI, XOR, etc. Pretrained encompasses receiving knowledge on ALU operations/API);
learning from documents and examples (Page 11, Section D.2: GPT 3.5 pretrained with RISC-V knowledge. RISC-V is a known instruction set architecture (ISA), defined in documents/source code. Page 4-5, Section 3.2-3.3: also learns from previous responses (examples)); and
prompting with an aid of relevant chunks shared to a large language model (LLM) in the GRVP framework (Page 2; Fig. 1b; Page 4, Section 3.1: "In each generation cycle, the prompt generator generates a prompt based on a template and the current coverage feedback from the coverage monitor. The LLM takes in the prompt and generates a natural language response." Template and current coverage feedback are relevant chunks shared to the LLM in the LLM4DV framework to create new responses).
Regarding Claim 3, Zhang discloses the method of Claim 2, as referenced above, wherein the generating of the one or more random test cases further comprises:
utilizing the LLM to generate one or more test cases along with documentation based on an input received from the RAG (Page 2, Fig. 1b; Page 8, algorithm 1: LLM generates test stimulus input (“test case”), which is then inserted into the testbench to generate a coverage rate (documentation) based on the current coverage input received from the previous LLM response); and
generating the one or more random test cases in a predefined language based on the one or more test cases generated by the LLM (Page 3, Fig. 2; Page 5, first paragraph; Page 5, Section 4: generates test stimuli using random sampling of previous responses (previous test stimuli) from one of the three sampling methods to reach uncovered bins. The test stimuli are performed in Python. Thus, random-bin test stimuli are generated in Python based on previous test stimuli generated by the LLM).
Regarding Claim 4, Zhang discloses the method of Claim 2, as referenced above, wherein the utilizing of the RAG further comprises utilizing the RAG to perform operations comprising:
utilizing the LLM to generate one or more test cases along with documentation based on an input received from the RAG (Page 2, Fig. 1b; Page 8, algorithm 1: LLM generates test stimulus input (“test case”), which is then inserted into the testbench to generate a coverage rate (documentation) based on the current coverage input received from a previous LLM response);
integrating reference libraries (Page 5, Section 4: uses Verilator and cocotb); and
learning examples with the APIs to enhance generation of random test programs (Page 2, Fig. 1b; Pages 4-5, Section 3.3: learns from previous responses and randomly sampled uncovered bins to improve test stimuli generation with respect to exposing the uncovered bins. Page 11, Section D.2: when testing Ibex Instruction Decoder, the previous responses may be test inputs for ALU operation bins. The next generation cycle generates test stimuli using the “prompting improvements” (see Page 2, Fig. 1b). Therefore, the LLM learns from previous responses (examples) containing ALU operations (ALU API) to enhance generation of randomly-sampled test bin programs for higher coverage).
Regarding Claim 9, Zhang discloses the method of Claim 2, as referenced above, wherein the generating of the one or more random test cases further comprises utilizing a Python-based random verification program (PyRVP) generator to generate the one or more APIs (Page 5, Section 4: “With Verilator and cocotb, we could efficiently and test the modules in Python.” Page 8, Algorithm 1; Page 11, Section D.2: Ibex Instruction Decoder verification involves generating test stimuli to cover ALU operation bins, wherein the generated stimuli is inputted into a testbench to calculate the resulting coverage. Pages 4-5, Section 3.3; Page 6, Section 4.2: Ibex Instruction Decoder verification uses Mixed Coverpoint Type-based and Pure Random Sampling to improve stimuli generation. Therefore, the LLM generates test stimuli (verification program generator) in Python, where the test stimuli are generated ALU operations/ALU API later inputted into a testbench. The LLM is also a random verification program generator as missed-bin sampling randomly samples uncovered bins to allow the LLM generate test stimuli and prioritize said random bins).
Regarding Claim 10, Zhang discloses the method of Claim 9, as referenced above, wherein the generating of the one or more random test cases further comprises utilizing the PyRVP generator to generate one or more random verification programs in Assembly that are used in validating the design of the processor (Page 6, Section 4.2; Page 11, Section D.2: generates test stimuli for ALU operation bin coverage when verifying Ibex Instruction Decoder of RISC-V processor design. ALU operations are Assembly operations).
Regarding Claim 11, the method of Claim 1 performs the same steps as the apparatus of Claim 11, and Claim 11 is rejected using the same art and rationale set forth above in the rejection of Claim 1 by the teachings of Zhang.
Zhang further discloses an apparatus (Page 5, Section 4: LLM4DV uses Verilator, cocotb, Python, and GPT 3.5. One of ordinary skill in the art would understand that a computer is used to run LLM4DV), comprising:
a memory configured to store information, data and instructions (see above – one of ordinary skill in the art would understand that memory containing data and instructions are required to execute LLM4DV and algorithm 1 (page 8));
and a processor coupled to the memory and configured to perform operations (see above – understood that a processor executes the data and instructions).
Regarding Claim 12, Zhang teaches the apparatus of Claim 11 above. The method of Claim 2 performs the same steps as the apparatus of Claim 12, and Claim 12 is rejected using the same art and rationale set forth above in the rejection of Claim 2 by the teachings of Zhang.
Regarding Claim 13, Zhang teaches the apparatus of Claim 12 above. The method of Claim 3 performs the same steps as the apparatus of Claim 13, and Claim 13 is rejected using the same art and rationale set forth above in the rejection of Claim 3 by the teachings of Zhang.
Regarding Claim 14, Zhang teaches the apparatus of Claim 12 above. The method of Claim 4 performs the same steps as the apparatus of Claim 14, and Claim 14 is rejected using the same art and rationale set forth above in the rejection of Claim 4 by the teachings of Zhang.
Regarding Claim 19, Zhang teaches the apparatus of Claim 11 above. The method of Claim 9 performs the same steps as the apparatus of Claim 19, and Claim 19 is rejected using the same art and rationale set forth above in the rejection of Claim 9 by the teachings of Zhang.
Regarding Claim 20, Zhang teaches the apparatus of Claim 19 above. The method of Claim 10 performs the same steps as the apparatus of Claim 20, and Claim 20 is rejected using the same art and rationale set forth above in the rejection of Claim 10 by the teachings of Zhang.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 5-7 and 15-17 are rejected under 35 U.S.C. 103 as being unpatentable over Zhang in view of Mishra (NPL: “System-Level Register and Sequence Verification with UVM and Embedded C/C++”).
Regarding Claim 5, Zhang discloses the method of Claim 2, as referenced above.
Zhang does not disclose:
wherein the generating of the one or more random test cases further comprises utilizing a C or C++-based random verification program (CRVP) generator to generate the one or more APIs.
However, Mishra teaches:
wherein the generating of the one or more random test cases further comprises utilizing a C or C++-based random verification program (CRVP) generator to generate the one or more APIs (Page 3: IDS-NG generates C/C++ random value tests. The tests are built using an API library for register access. Thus, IDS-NG generates C/C++ random value test APIs, which are built using the API library).
Therefore, it would have been obvious before the effective filing date of the claimed invention to one of ordinary skill in the art to which said subject matter pertains to combine Zhang and Mishra performing a simple substitution of one known element (Zhang: Page 2, Fig. 1b; Page 3-4, Section 3.1; Page 5, Section 4; Page 11, Section D.2: generate test stimuli in Python targeting random-sampled uncovered bins for Ibex Instruction Decoder and register verification) for another (Mishra: Page 3: C/C++ random value tests for register verification) to obtain predictable results (generating random tests for register coverage in a specific language).
Regarding Claim 6, Zhang in view of Mishra teaches the method of Claim 5, as referenced above, wherein the generating of the one or more random test cases further comprises utilizing the CRVP generator to generate one or more random verification programs in C that are used in validating the design of the processor (Mishra: Page 3: IDS-NG generates C/C++ random value tests, where users will run these tests on their own model of the chosen CPU for their SoC [design] (see Page 2). Validating SoC design containing a CPU model encompasses testing the CPU model along with the remaining SoC design).
Regarding Claim 7, Zhang in view of Mishra teaches the method of Claim 6, as referenced above, wherein the one or more C-random verification programs are instruction set architecture (ISA)-independent (Misha: Page 3: IDS-NG generates random C/C++ tests. One of ordinary skill in the art would understand that C source code is ISA-independent).
Regarding Claim 15, Zhang teaches the apparatus of Claim 12 above. The method of Claim 5 performs the same steps as the apparatus of Claim 15, and Claim 15 is rejected using the same art and rationale set forth above in the rejection of Claim 5 by the teachings of Zhang in view of Mishra.
Regarding Claim 16, Zhang in view of Mishra teaches the apparatus of Claim 15 above. The method of Claim 6 performs the same steps as the apparatus of Claim 16, and Claim 16 is rejected using the same art and rationale set forth above in the rejection of Claim 6 by the teachings of Zhang in view of Mishra.
Regarding Claim 17, Zhang in view of Mishra teaches the apparatus of Claim 16 above. The method of Claim 7 performs the same steps as the apparatus of Claim 17, and Claim 17 is rejected using the same art and rationale set forth above in the rejection of Claim 7 by the teachings of Zhang in view of Mishra.
Claims 8 and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Zhang in view of Mishra, in further view of Tseng (US 20090164861 A1), in further view of 123calculus (NPL: “Random Polynomial Generator”).
Regarding Claim 8, Zhang in view of Mishra teaches the method of Claim 6, as referenced above, wherein the utilizing of the CRVP generator to generate one or more C-random verification programs comprises utilizing the CRVP generator to perform operations (Mishra: Page 3: IDS-NG performs the operations cited below) comprising:
running application and operating system (OS) fragments on simulators (Page 3: “we also ship with Zephyr, an open-source real-time operating system (RTOS), so that users can run applications in the simulation environment”);
collecting snapshots of input and output registers… (Mishra: Page 5: coverage results gathered during test simulation are reported, including results for reading and writing values from registers. Coverage results encompass collecting snapshots (Page 4, IDS-NG Sequence View: read register value at a point in time) of read/input and write/output registers to calculate said coverage results
PNG
media_image1.png
206
416
media_image1.png
Greyscale
);
creating random expressions… (Mishra: Page 3: create random value test); and calculating outputs using known values as inputs (Mishra: Pages 4-5: calculating coverage results using known/defined arguments, constants variables, etc. as inputs for a test sequence).
Zhang in view of Mishra does not teach:
collecting snapshots of… control and status registers (CSRs);
creating random expressions of an order of 3 or 4;
However, Tseng teaches:
collecting snapshots of… control and status registers (CSRs) ([0021]: polls internal registers by repeatedly reading a status bit in one of the design’s internal registers until the bit is set to determine when the test completes);
Additionally, 123calculus teaches:
creating random expressions of an order of 3 or 4 (Page 1: random polynomial generator of degree 3);
Therefore, it would have been obvious before the effective filing date of the claimed invention to one of ordinary skill in the art to which said subject matter pertains to combine Zhang, Mishra, and Tseng by implementing the internal register polling taught by Tseng. One of ordinary skill in the art would be motivated to make this modification in order to determine when the test completes (Tseng: [0021]).
Furthermore, it would have been obvious before the effective filing date of the claimed invention to one of ordinary skill in the art to which said subject matter pertains to combine Zhang, Mishra, Tseng, and 123calculus by implementing the random polynomial generator taught by 123calculus. One of ordinary skill in the art would be motivated to make this modification in order to create a random expression that mathematical operators to test ALU operations (123calculus: Page 1).
Regarding Claim 18, Zhang in view of Mishra teaches the apparatus of Claim 16 above. The method of Claim 8 performs the same steps as the apparatus of Claim 18, and Claim 18 is rejected using the same art and rationale set forth above in the rejection of Claim 8 by the teachings of Zhang in view of Mishra, in further view of Tseng, in further view of 123calculus.
Prior Art of Record
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure:
Wang et al. (NPL: "ChatCPU: An Agile CPU Design and Verification Platform with LLM") – generative LLM that generates CPU design based on user-provided spec template and tests the design using a testbench; LLM trained on existing open-source hardware design (RAG)
Baumgartner et al. (US 20060190867 A1)
Oshima et al. (US 20130014091 A1)
Datta et al. (US 20040098575 A1) – [0040]: “ISA independent higher level language like C”
Almog et al. (US 20110099431 A1) – tests interrupts for processor verification; generates test code blocks in other languages such as C
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to CATHERINE MARIE NGUYEN whose telephone number is (571)272-6160. The examiner can normally be reached M-F 7am-4pm ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, ASHISH THOMAS can be reached at (571) 272-0631. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/C.M.N./Examiner, Art Unit 2114
/ASHISH THOMAS/Supervisory Patent Examiner, Art Unit 2114