DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
This Office Action is in response to Applicant’s amendment filed 07/16/2026 which has
been entered and made of record. Claims 1, 3-4, 7-8, 10, 12-14, and 16-18 have been amended. No claim has been cancelled or newly added. Claims 1-20 are pending in the application. Applicant’s amendments to the Claims have overcome each and every objection
previously set forth in the Non-Final Office Action mailed April 17th, 2026.
Response to Arguments
Applicant’s arguments with respect to claim(s) 1-20 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument (due to applicant’s arguments directed to newly amend limitation(s) which is addressed by new prior art presented in this Office Action).
Claim Rejections - 35 USC § 112
Previous 35 U.S.C. 112(b) rejection for claims 4-8 and 13-17 have been withdrawn.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-2, 9-11, and 18-19 is/are rejected under 35 U.S.C. 103 as being unpatentable over QIU et al. (U.S. Patent Application Publication No. 2025/0104394), hereinafter referenced as Qiu in view of Buehler (Cephalo: Multi-Modal Vision-Language Models for Bio-Inspired Materials Analysis and Design), hereinafter referenced as Buehler, Kunz (U.S. Patent Application Publication No. 20250156635), hereinafter referenced as Kunz, and Pillai (U.S. Patent Application Publication No. 2025/0321870), hereinafter referenced as Pillai.
Regarding claim 1, Qiu teaches a system, comprising: one or more processors to execute one or more operations comprising: (fig. 2 shows system 200 with processor 206 which is implemented executing method/operations from abstract); obtain visual data and a first prompt corresponding to a context of the visual data (paragraph 27 teaches "Vision language machine learning models for computer vision systems and methods are trained using subsets of data samples from a dataset. In some examples, models are trained using a data of labeled images." and paragraph 3 teaches "context prompt vectors define context information associated with an image classification task to be performed by the VLM"); images show visual data, the context prompt shows first prompt associated with context information of the image and since this is performed by the VLM, it is obtained by such; determine a second prompt based at least in part on the at least one description (paragraph 3 teaches "generating prompts for each of the plurality of candidate classes by appending respective class text tokens to the context prompt vectors for each of the plurality of candidate classes"); prompts (plural) means second prompt must be generated/determined and it is based on the description since is for the plurality of candidate classes by appending class text tokens which are based on text description thus this second prompt is also based on such;
However, Qiu fails to explicitly teach apply at least the visual data and the first prompt to a vision language model (VLM) to generate at least one description of the visual data, the first prompt describing one or more attributes of the visual data to test;
However, Buehler teaches apply at least the visual data and the first prompt to a vision language model (VLM) to generate at least one description of the visual data, (Buehler, abstract teaches “Cephalo is presented as a series of multimodal vision large language models (V-LLMs)…Cephalo is trained on integrated image and text data from thousands of scientific papers and science-focused Wikipedia data demonstrates it can interpret complex visual scenes, generate precise language descriptions, and answer queries about images effectively.” And page 2 paragraph 1 teaches “superior ability of in-context learning, where models adapt their responses based on the con-text provided in the prompt.”); this shows applying visual data (integrated image and interpreting of complex visual scenes) and first prompt to VLM/Cephalo to generate language descriptions of the visual data; the first prompt describing one or more attributes of the visual data to test (Buehler, page 1, right col., second paragraph teaches “Herein, generative AI is a particularly useful tool that can aid not only discovery but also finding engineering solutions for materials. This includes applications that include diverse properties that range from enhanced toughness to improved biological properties”); the properties/attributes mentioned here shows first prompt would describe such of the visual data and is to test because Buehler, page 36, left col., second to last paragraph mentions “This approach allowed for the integration of complex logic in expanding image descriptions, thus enriching the training dataset. Once the image extraction algorithm was finalized, the same method would be applied to process figure-caption pairs, preparing the dataset for comprehensive testing and further model development”. Buehler is considered to be analogous art because it is reasonably pertinent to the problem faced by the inventor of application of visual data and prompt to VLM to generate description. Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Qiu's invention with the visual data, description and VLM techniques of Buehler to train a vision-LLM in a computationally efficient and sustainable manner (Buehler, page 34, right col., third paragraph). This would lead to an improved user experience due to a more efficient system.
However, the combination of Qiu and Buehler fails to teach the second prompt describing a file format; convert the at least one description to the file format defined by the second prompt by applying at least the at least one description of the visual data and the second prompt to a large language model (LLM);
However, Kunz teaches the second prompt describing a file format (Kunz, abstract teaches "a system message included with the input prompt may instruct the LLM to generate text in the intermediate file format"); this shows prompt (second prompt when viewed in combination) describes file format since it is instructing to generate text in file format; convert the at least one description to the file format defined by the second prompt by applying at least the at least one description of the visual data and the second prompt to a large language model (LLM) (Kunz, paragraph 12 teaches "input prompt can be created in or converted to this intermediate file format", paragraph 53 teaches “generating a prompt using the natural language text, the contextual information, and a system message,”, and abstract teaches “utilizing an intermediate file format for either an input prompt to or output from a large language model (LLM)”); this shows converting description to the file format defined by second prompt by applying the prompt to LLM (applying prompt to LLM since “input” in abstract describes such and also prompt is associated with contextual information and text [considered description of visual data] thus that’s also input to LLM); Kunz is considered to be analogous art because it is reasonably pertinent to the problem faced by the inventor of file formatting and LLM being used in a conjunctive way. Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify the combination of Qiu and Buehler with the file formatting and LLM techniques of Kunz to ensure a deeper sense of language context and flow (Kunz, paragraph 22). This would be due to the conversion into specific file formats leading to better analysis.
However, the combination of Qiu, Buehler and Kunz fails to teach and use at least the file format to execute at least one automated test.
However, Pillai teaches and use at least the file format to execute at least one automated test. (Pillai, paragraph 9 teaches "test database including one or more natural language-based test cases and an app description file including a natural language description of a particular test application run on one or more test devices. The app description file is updated to reflect one or more changes in the test application...the orchestrator automatically tests the test application by iteratively executing each of the one or more actions via the corresponding test application interface call."); since description file (the file format from above when viewed in combination) includes description of test application and description file is updated to reflect changes in test application, the automatic tests of test application shows using the description in the file format for the automated test. Pillai is considered to be analogous art because it is reasonably pertinent to the problem faced by the inventor of having files, automated testing and LLM in conjunction with one another.
Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify the combination of Qiu, Buehler and Kunz with the LLM and testing techniques of Pillai to reduce the efforts required to update and maintain test scripts manually, thus providing improved scalability that is particularly beneficial for large and complex applications with frequent updates (Pillai, paragraph 104). This would be done by having the description converted to file format and using that specific file for the automated testing.
Regarding claim 2, the combination of Qiu, Buehler, Kunz and Pillai teaches wherein the visual data comprises at least one of (i) a generated image or video, (ii) a captured image or video, (iii) a composite image from a plurality of image sources, or (iv) a sequence of visual frames (Qiu, paragraph 27 teaches "Vision language machine learning models for computer vision systems and methods are trained using subsets of data samples from a dataset. In some examples, models are trained using a data of labeled images."); this shows the visual data as images of which one of ordinary skill in the art would understand that the images must be captured or generated.
Regarding claim 9, the combination of Qiu, Buehler, Kunz and Pillai teaches wherein the one or more processors are comprised in at least one of:
a control system for an autonomous or semi-autonomous machine (Qiu, paragraph 113 teaches "FIG. 7 depicts a schematic diagram of control system" (which has vehicle shown in fig. 3B as reference 308);
a perception system for an autonomous or semi-autonomous machine;
a system implemented using a robot (Qiu, paragraph 110 teaches "In some embodiments where vehicle 600 is an at least partially autonomous robot");
an aerial system;
a medical system;
a boating system;
a smart area monitoring system (Qiu, paragraph 122 teaches "Monitoring system 1000 may also be a surveillance system.");
a system for performing deep learning operations (Qiu, paragraph 51 teaches "machine-learning model 210 may be a deep-learning or neural network algorithm");
a system for performing simulation operations;
a system for generating or presenting virtual reality (VR) content, augmented reality (AR) content, or mixed reality (MR) content;
a system for performing digital twin operations;
a system implemented using an edge device;
a system incorporating one or more virtual machines (VMs);
a system for generating synthetic data;
a system implemented at least partially in a data center;
a system for performing conversational artificial intelligence (AI) operations;
a system for performing generative AI operations;
a system implementing language models(Qiu, paragraph 79 teaches "prompt generator 418 is configured to align the LLM 416 and the VLM ");
a system implementing vision language models (VLMs) (Qiu, paragraph 79 teaches "prompt generator 418 is configured to align the LLM 416 and the VLM ");
a system implementing large language models (LLMs) (Qiu, paragraph 79 teaches "prompt generator 418 is configured to align the LLM 416 and the VLM ");
a system implementing small language models (SLMs);
a system implementing multi-modal language models(Qiu, paragraph 72 teaches "multi-model prompt learning systems");
a system for hosting one or more real-time streaming applications;
a system for performing light transport simulation;
a system for performing collaborative content creation for 3D assets; or
a system implemented at least partially using cloud computing resources.
Regarding claim 10, the processor(s)/product claim 10 recites similar limitations as system claim 1, and thus is rejected under similar rationale. Note: the models (both VLM and LLM) in claim 1 are interpreted to be the neural network models in claim 10.
Regarding claim 11, the processor(s)/product claim 11 recites similar limitations as system claim 2, and thus is rejected under similar rationale.
Regarding claim 18, the method claim 18 recites similar limitations as system claim 1, and thus is rejected under similar rationale. Note: the models (both VLM and LLM) in claim 1 are interpreted to be the neural network models in claim 18.
Regarding claim 19, the method claim 19 recites similar limitations as system claim 2, and thus is rejected under similar rationale.
Claim(s) 3-8, 12-17 and 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over the combination of Qiu, Buehler, Kunz and Pillai as applied to claims 1, 10, and 18 above, and further in view of Vodeniktov (U.S. Patent Application Publication No. 2025/0077913), hereinafter referenced as Vodeniktov.
Regarding claim 3, the combination of Qiu, Buehler, Kunz and Pillai teaches wherein the one or more operations to determine the second prompt is based at least in part on the at least one description, (Qiu, paragraph 3 teaches "generating prompts for each of the plurality of candidate classes by appending respective class text tokens to the context prompt vectors for each of the plurality of candidate classes"); prompts (plural) means second prompt must be generated/determined and it is based on the description since is for the plurality of candidate classes by appending class text tokens which are based on text description thus this second prompt is also based on such.
However, the combination of Qiu, Buehler, Kunz and Pillai fails to teach second prompt is based at least in part on...and a skeletal JSON
However, Vodeniktov teaches second prompt is based at least in part on...and a skeletal JSON (Vodeniktov, paragraph 46 teaches "user can input the text for a prompt or prompt template to be transmitted to an LLM. A prompt template that is input to the parameters section 240 may refer to fields of an input data structure (e.g., a JSON file) to generate a prompt to transmit to the LLM"); prompt template used to refer fields of input data structure shows input data structure like JSON file used in prompt generation thus second prompt would be based on such, and template alongside the reference to field in the data structure shows skeletal JSON. Vodeniktov is considered to be analogous art because it is reasonably pertinent to the problem faced by the inventor of prompt based on JSON file and used in conjunction with an LLM. Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify the combination of Qiu, Buehler, Kunz and Pillai invention with the JSON techniques of Vodeniktov to provide enhanced computing capability (Vodeniktov, paragraph 26). This would be done by the sorted/neat format of the JSON file so data can be retrieved faster.
Regarding claim 4, the combination of Qiu, Buehler, Kunz, Pillai and Vodeniktov teaches wherein the file format is JSON format, (Vodeniktov, paragraph 46 teaches "data structure (e.g., a JSON file)"); this shows file format to be JSON; and wherein the at least one description comprises at least one of the one or more attributes of the visual data, (Qiu, paragraph 36 teaches "providing the VLM or other model with additional information (e.g., text descriptions, labels, etc.) about the image prior to performing computer vision tasks, including classification, object detection, and semantic segmentation"); this shows text description comprises additional information about the image/visual data including attributes like classification and object detection; the one or more attributes comprising at least one of an object identification, a spatial orientation, or an environmental parameter (Qiu, paragraph 56 teaches "configured to perform video object detection to identify one or more objects in corresponding images of the video stream data"); this shows the object identification from above is to identify objects thus attributes comprise object identification. The same motivations used in claim 3 apply here in claim 4.
Regarding claim 5, the combination of Qiu, Buehler, Kunz, Pillai and Vodeniktov teaches wherein the one or more operations further comprise at least one operation to: receive, by the VLM, a subsequent prompt to validate the one or more attributes of the visual data (Qiu, paragraph 37 teaches "Accordingly, one model learned/generated in accordance with the principles of the present disclosure (e.g., the prompt generator) can be used to solve different classification problems"); solving different classification problems by prompt generator shows subsequent prompt to validate the attribute of classification (described above) within the image/visual data, and this would be received by VLM since the previous paragraph 36 mentions the VLM performing the tasks; and generate, by the VLM, a corresponding validation output (Qiu, paragraph 53 teaches "the machine-learning model 210 can compare output results (e.g., annotations) with those included in the training dataset 212. Since the training dataset 212 includes the expected results, the machine-learning model 210 can determine when performance is acceptable. After the machine-learning model 210 achieves a predetermined performance level (e.g., 100% agreement with the outcomes associated with the training dataset"); the machine-learning (ML) model comparing output results in its own predictions against expected outcome determines whether some criteria has been met thus this acts as a validation output and the VLM here from above is a type of ML model.
Regarding claim 6, the combination of Qiu, Buehler, Kunz, Pillai and Vodeniktov teaches wherein the corresponding validation output comprises at least one of a pass-fail result, a confidence score, or an interpretation of the corresponding validation output, (Qiu, abstract teaches "and outputting a class prediction probability for a sample image based on the plurality" and Pillai, paragraph 59 teaches "subsequently flags these anomalies in the test report for review along with the typical test results such as pass and fail"); probability shows the confidence score as validation output because its giving a likelihood of an occurrence; and wherein the subsequent prompt is structured to query the one or more attributes of the visual data (Pillai, paragraph 38 teaches "execution engine (102) may intermittently issue prompts with queries for gathering more comprehensive information regarding one or more physical, functional and contextual aspects"); this shows subsequent prompts to query attributes which would be of the visual data from Qiu when viewed in combination. The same motivations used in claim 1 apply here in claim 6.
Regarding claim 7, the combination of Qiu, Buehler, Kunz, Pillai and Vodeniktov teaches wherein the one or more operations further comprise: at least one operation to: extract, by the VLM, the one or more attributes from the visual data using multimodal feature extraction corresponding with visual and textual elements based at least on the first prompt (Pillai, paragraph 41 teaches "employs one or more LLMs including text and vision input models, visual language models (VLMs)," and paragraph 59 teaches "anomalies, for example, may include incompletely rendered images or color or orientation issues. The orchestrator (206) may identify these anomalies, for example, via an AI agent that uses one or more of a text, vision, or multimodal LLM to detect unexpected artefacts during testing without needing a list of hardcoded or explicitly defined anomalies."); since LLM includes VLM, this detect/extraction of anomalies such as color (which is attribute since to know anomalies of color you would need original color) is done by VLM and multimodal LLM and is corresponding to visual/vision and textual/text elements mentioned, also this would be based on the first prompt from Qiu because it comes in a step after the first prompt since the test is using description file which is from the second prompt. The same motivations used in claim 1 apply here in claim 7.
Regarding claim 8, the combination of Qiu, Buehler, Kunz, Pillai and Vodeniktov teaches wherein the one or more operations further comprise: at least one operation to: map one or more attributes of the at least one description into a structured data format compatible with a testing system (Pillai, paragraph 36 teaches "app description file (112) stored in the test database (114) includes comprehensive natural language-based description of all the GUI screens within the test application (106). This description, for example, includes all elements"); description file stored in database ensures a mapping of attributes of the description file in structured data format since including all elements ensures the attributes are also included, and test database ensures structured data format compatible with test system. The same motivations used in claim 1 apply here in claim 8.
Regarding claim 12, the processor(s)/product claim 12 recites similar limitations as system claim 3, and thus is rejected under similar rationale.
Regarding claim 13, the processor(s)/product claim 13 recites similar limitations as system claim 4, and thus is rejected under similar rationale.
Regarding claim 14, the processor(s)/product claim 14 recites similar limitations as system claim 5, and thus is rejected under similar rationale.
Regarding claim 15, the processor(s)/product claim 15 recites similar limitations as system claim 6, and thus is rejected under similar rationale.
Regarding claim 16, the processor(s)/product claim 16 recites similar limitations as system claim 7, and thus is rejected under similar rationale.
Regarding claim 17, the processor(s)/product claim 17 recites similar limitations as system claim 8, and thus is rejected under similar rationale.
Regarding claim 20, the method claim 20 recites similar limitations as system claim 3, and thus is rejected under similar rationale. Note: the skeletal JSON mapped and aforementioned in claim 3 above acts as the JSON template.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Rezaeian (U.S. Patent Application Publication No. 2025/0022595) paragraph 29 teaches “LLM 136 receives a prompt and a summary note from prompt engine 134 and converts the unstructured text (e.g., a string of characters) of the summary note into a structured data format, such as a JSON file.”; this shows prompt of unstructured text converted to structured data format like JSON file.
Levine (U.S. Patent Application Publication No. 2025/0336381) paragraph 353 teaches “AI server 130 is primarily responsible for using a large language model (LLM) to parse the user-input text prompt and to then format it into a file type that the track generation server can use to select loops from the loop storage server 140 to use for the desired track”; this shows the use of LLM to parse user text prompt to format it into a file type.
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to NAUMAN U AHMAD whose telephone number is (703)756-5306. The examiner can normally be reached Monday - Friday 9:00am - 5:00pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kee Tung can be reached at (571) 272-7794. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/KEE M TUNG/Supervisory Patent Examiner, Art Unit 2611
/N.U.A./ Examiner, Art Unit 2611
