Notice of Pre-AIA or AIA Status
2. The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
2. In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
3. Claims 1-4, 7-8, 10-13 and 16--20 are rejected under 35 U.S.C. 103 as being unpatentable over Mabote (US 2022/0058306) in view of Stump (US 2023/0152790) and further in view of Wang “Screen2Words: Automatic Mobile UI Summarization with Multimodal Learning”.
Regarding Claim 1:
Mabote discloses a system, comprising:
a memory that stores executable components (Mabote: ¶45 discloses memory storing software instructions corresponding to the components of HMI development system) and
a processor, operatively coupled to the memory, that executes the executable components, the executable components comprising (Mabote: ¶45 discloses a processor coupled to the memory and executing the software stored in the memory):
a user interface component configured to render a human-machine interface (HMI) development interface and (Mabote: ¶46 discloses a user interface component rendering interactive interface displays that serve as an industrial HMI development platform; ¶62 discloses HMI project file defining HMI displays or screens containing graphical objects and links between the graphical objects and controller data tags, ¶64 discloses arranging the graphical object relative to one another on the HMI interface displays and ¶65 discloses linking animation properties of the graphical objects to controller data tags representing values generated by the industrial automation system); and
configured to render (Mabote: ¶46 discloses user interface component 204 rendering output through the HMI development interface and ¶77 discloses presenting and editing the HMI project through that development platform).
Mabote does not explicitly disclose:
to receive, via interaction with the HMI development interface, a submission of an HMI project
a generative artificial intelligence (AI) component configured to formulate a natural language summary of functionality of the HMI project based on analysis of the HMI project and content of one or more custom models trained with training data
the natural language summary
However, Stump discloses:
to receive, via interaction with the HMI development interface, a submission of (Stump: ¶46 discloses a user interface component 204 receiving user input including legacy projects through an IDE client or interface screens, ¶52 discloses that the IDE supports HMI development and ¶42 discloses project data submitted by developers that includes visualizations, and ¶57 discloses developers submitting HMI screen configuration input through interact with user interface component).
Mabote and Stump are combinable because they are in the same field of endeavor, both disclose systems and methods for developing HMI and other industrial automation projects through computerized development interfaces. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to disclose receiving an existing HMI containing project through its development interface. The motivation for doing so is taught by Stump, which explains that receiving submitted legacy projects allows existing projects to predictably be migrated to the platform supported by IDE system in ¶95.
The proposed combination of Mabote in view of Stump does not explicitly disclose:
a generative artificial intelligence (AI) component configured to formulate a natural language summary of functionality of the HMI project based on analysis of the HMI project and content of one or more custom models trained with training data
However, Wang discloses:
a generative artificial intelligence (AI) component configured to formulate a natural language summary of functionality of the HMI project based on analysis of the HMI project and content of one or more custom models trained with training data (Wang: Abstract discloses automatically generating a natural language summary of an entire UI screen that identifies its important contents and functionalities rather than merely describing its appearance. Section 1 discloses a transformer encoder decoder model that analyzes the UI screenshot, screen text, graphical elements and spatial layout and view hierarchy structure and decodes that information into a natural a language summary)
the natural language summary (Wang Section 4.2 discloses the generated output as a varying length natural language screen summary).
Mabote and Stump in view of Wang are combinable because both disclose systems and methods for processing and analysis of graphical user interface screens and their underlying structural representations. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to apply Wang’s trained summarization model to the submitted HMI project and render the resulting summary through Mabote’s existing development interface output path. Wang explains that a screen allows users to “easily grasp the purpose and the state of a screen” in Section 1. Therefore, allowing the HMI developer to understand the purpose and content of the submitted project without individually inspecting each screen element.
Regarding Claim 2:
The proposed combination of Mabote, Stump and Wang further discloses the system of claim 1, wherein the natural language summary describes at least one of an overall purpose of the HMI project, a type of machine or industrial process that the HMI project is designed to visualized, a type of information provided by the display screens, or an amount of memory or processing power required to execute the HMI project (Wang: Section 1, discloses that the generated screen summary conveys the important contents and functionalities of the UI screen and allows a user to grasp the purpose and state of the screen).
Mabote and Stump in view of Wang are in the same field of endeavor, both disclose systems and methods for presenting information through a graphical user interface. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to disclose Wang’s summarization model to describe the purpose or importance conveyed by Mabote’s HMI screens because Wang’s abstract teaches that a summary should convey the screen’s “important and functionalities of the screen” and permit the user to grasp its purpose and state. Applying that teaching to Mabote’s HMI project would predictably produce a summary identifying the project’s overall monitoring purpose or the operational information displayed, such as temperatures, flows, pressures or fill levels.
Regarding Claim 3:
The proposed combination of Mabote, Stump and Wang further discloses the system of claim 1, wherein the user interface is further configured to receive a question regarding the HMI project (Stump: ¶46 discloses user interface component 204 receiving user input through IDE interface screens. Wang Section 8 discloses receiving free form open ended natural language question regarding a UI screen as an input to a visual question answer system, when incorporated into the HMI supporting IDE of Mabote and Stump the UI screen and associated project information teach the HMI project that the question received asks about), and the generative Al component is further configured to formulate a natural language answer to the question based on analysis of the question, the HMI project, and the content of one or more custom models (Wang: Section 8 teaches visual question answering that receives a UI screen and a free form natural language question and produces a natural language answer including answers to questions concerning what is displayed on the screen, actions available through the screen or the content of a particular screen element. Section 4.1 discloses encoding a mobile UI screen and analyzing screen’s graphical user interface elements etc.).
Mabote and Stump in view of Wang are in the same field of endeavor, both disclose systems and methods for presenting information through a graphical user interface. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to disclose extending Wang’s summarization model to receive questions because Wang identifies visual question answer as a natural extension of the system that receives a UI screen and an open ended natural language question and produces a natural language answer as disclosed in Section 8 of Wang. This modification would allow an HMI developer to request targeted information about a submitted HMI project rather than receiving only a general summary and obtain an answer based on the project’s interface.
Regarding Claim 4:
The proposed combination of Mabote, Stump and Wang further discloses the system of claim 3, wherein the question is at least one of a question regarding a type of industrial application for which the HMI project is designed, a question regarding animation or data link properties of a graphical object contained in the HMI project, a question regarding computational resources required to execute the HMI project, a question regarding a type of information that one of the display screens is designed to convey, or a question regarding functionality of a control graphic or graphical object included on one of the display screens (Wang: Section 8 discloses receiving questions concerning the information presented by a UI screen, explicitly identifying what’s on the screen as a question addressed by screen2words and identifying further questions concerning the actions available through a screen or the content of a particular screen element. Mabote ¶64 discloses HMI display screens designed to convey operational and status information for an industrial automation system).
Mabote and Stump in view of Wang are in the same field of endeavor, both disclose systems and methods for presenting information through a graphical user interface. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to disclose Wang’s summarization model to describe the purpose or importance conveyed by Mabote’s HMI screens because Wang teaches that a summary should convey the screen’s “important and functionalities of the screen” (Wang, Abstract) and permit the user to grasp its purpose and state. Applying that teaching to Mabote’s HMI project would predictably produce a summary identifying the project’s overall monitoring purpose or the operational information displayed, such as temperatures, flows, pressures or fill levels.
Regarding Claim 7:
The proposed combination of Mabote, Stump and Wang further discloses the system of claim 1, wherein the training data comprises at least one of information defining industrial standards, technical specifics for respective types of industrial control applications, knowledge of different industrial verticals, information describing industrial best practices, technical specifications for different types of industrial devices or machines, control design rules, sample HMI display layouts for respective types of control applications, or customer-specific training data describing in-house HMI design preferences (Stump: ¶89 discloses aggregating diverse industrial system projects and supplying them to the training component as training data, ¶91 discloses learning control design patterns classified by industrial vertical, industrial application and machine type and ¶92-93 disclose training from HMI screens, animation objects and other visualization associated with respective machines, processes and automation applications therefore teaching sample display layouts for respective control application).
Mabote and Stump are combinable because they are in the same field of endeavor, both disclose systems and methods for developing HMI and other industrial automation projects through computerized development interfaces. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to disclose receiving an existing HMI containing project through its development interface. The motivation for doing so is taught by Stump, which explains that receiving submitted legacy projects allows existing projects to predictably be migrated to the platform supported by IDE system in ¶95.
Regarding Claim 8:
The proposed combination of Mabote, Stump and Wang further disclose the system of claim 1, wherein the HMI project is configured to, in response to execution on an HMI terminal device, read data from the sources of data and visualize the data via the graphical objects defined by the HMI project (Mabote: ¶48 discloses executing the resulting HMI application on an HMI terminal and populating or animating its graphical displays with real-time industrial process data, ¶37 discloses that the HMI reads data from industrial controllers and presents the data through graphical display screens, ¶62 discloses that the HMI project defines graphic objects and links those objects to industrial control data sources and ¶65 discloses using the linked data values to control the runtime animation and visual state of the graphical objects).
Regarding Claim 10:
Claim 10 has been analyzed with regards to claim 1 (see rejection above) and is rejected for the same reasons of obviousness set forth above.
Regarding Claim 11:
Claim 11 has been analyzed with regards to claim 2 (see rejection above) and is rejected for the same reasons of obviousness set forth above.
Regarding Claim 12:
Claim 12 has been analyzed with regards to claim 3 (see rejection above) and is rejected for the same reasons of obviousness set forth above.
Regarding Claim 13:
Claim 13 has been analyzed with regards to claim 4 (see rejection above) and is rejected for the same reasons of obviousness set forth above.
Regarding Claim 16:
Claim 16 has been analyzed with regards to claim 7 (see rejection above) and is rejected for the same reasons of obviousness set forth above.
Regarding Claim 17:
Claim 17 has been analyzed with regards to claim 8 (see rejection above) and is rejected for the same reasons of obviousness set forth above.
Regarding Claim 18:
Claim 18 has been analyzed with regards to claim 1 (see rejection above) and is rejected for the same reasons of obviousness set forth above.
Regarding Claim 19:
Claim 19 has been analyzed with regards to claim 2 (see rejection above) and is rejected for the same reasons of obviousness set forth above.
Regarding Claim 20:
Claim 20 has been analyzed with regards to claim 3 (see rejection above) and is rejected for the same reasons of obviousness set forth above.
It is noted that Mabote discloses non transitory computer readable media at least at ¶94.
4. Claims 5 and 14 are rejected under 35 U.S.C. 103 as being unpatentable over Mabote in view of Stump, further in view of Wang and further in view of Qin (US 2024/0346256).
Regarding Claim 5:
The proposed combination of Mabote, Stump and Wang further discloses the system of claim 3, except wherein the generative AI component is further configured to, in response to receipt of the question, formulate a prompt, directed to a generative AI model, designed to obtain a response from the generative Al model comprising information used by the generative Al component to determine the natural language answer to the question, wherein the prompt is generated based on analysis of the HMI project and the content of one or more custom models.
However, Qin discloses:
wherein the generative AI component is further configured to, in response to receipt of the question, formulate a prompt, directed to a generative AI model, designed to obtain a response from the generative Al model comprising information used by the generative Al component to determine the natural language answer to the question, wherein the prompt is generated based on analysis of the HMI project and the content of one or more custom models (Qin: ¶45-46 discloses retrieving information relevant to the question and generating an augmented prompt containing the original question, contextual information and retrieved domain specific content, applied to the preceding combination, Mabote’s HMI project supplies Qin’s prompt contextual information and the domain specific information encoded by Stump and Wang’s trained models supplies Qin’s augmentation content used to formulate the prompt and determine the natural language answer).
The combined system of Mabote, Stump, Wang and Qin are combinable because they are from the same field of endeavor, i.e., both disclose systems for generating contextual natural language answers to user questions. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to disclose using Qin’s augmented prompt technique in order to produce useful answers with Wang’s generative transformer using HMI projects present in Mabote. The motivation for doing so is that augmented prompts generate “more focused and accurate” answers as disclosed in ¶19 of Qin.
Regarding Claim 14
Claim 14 has been analyzed with regards to claim 5 (see rejection above) and is rejected for the same reasons of obviousness set forth above.
5. Claims 6 and 15 are rejected under 35 U.S.C. 103 as being unpatentable over Mabote in view of Stump, further in view of Wang and further in view of Mirkovic (US 2006/0271364).
Regarding Claim 6:
The proposed combination of Mabote, Stump and Wang further discloses the system of claim 3, except wherein the generative AI component is further configured to, in response to determining, based on analysis of the question, that additional information will allow the generative Al component to determine an answer to the question having a probability of addressing the question that exceeds a threshold probability, generate a natural language response that prompts for the additional information, render the natural language response via the user interface component, and formulate the natural language answer further based on analysis of the additional information.
However, Mirkovic discloses:
wherein the generative AI component is further configured to, in response to determining, based on analysis of the question, that additional information will allow the generative Al component to determine an answer to the question having a probability of addressing the question that exceeds a threshold probability, generate a natural language response that prompts for the additional information, render the natural language response via the user interface component, and formulate the natural language answer further based on analysis of the additional information (Mirkovic: ¶116 discloses determining a probability that a natural language input is correctly understood, ¶140-143 discloses comparing the confidence to defined thresholds and when confidence is not good enough, generating a clarification question rather than acting on the bad output, ¶129 and ¶142 disclose processing the user’s answer to the clarification question through the system).
The combined system of Mabote, Stump, Wang and Mirkovic are combinable because they are from the same field of endeavor, i.e., both disclose interactive systems that process natural language user input and generate corresponding responses. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to disclose confidence based clarification steps as done in Mirkovic. The motivation for doing so is “the user is asked for clarification, 1414, and the system then processes the user's answer to the clarification question” in Mirkovic ¶142. The modification would predictably improve the likelihood that the resulting answer accurately addresses the developer’s question.
Regarding Claim 15:
Claim 15 has been analyzed with regards to claim 6 (see rejection above) and is rejected for the same reasons of obviousness set forth above.
6. Claim 9 is rejected under 35 U.S.C. 103 as being unpatentable over Mabote in view of Stump, further in view of Wang and further in view of Chen (US 2024/0020116).
Regarding Claim 9:
The proposed combination of Mabote, Stump and Wang further discloses the system of claim 1, wherein the natural language summary is a first natural language summary (Wang: Section 1 and Section 4.2 discloses generating a natural language summary describing the contents and functionality of a UI screen), the user interface component is further configured to receive, via interaction with the HMI development interface, control code that, in response to execution on an industrial controller, programs the industrial controller to monitor and control an industrial automation system (Stump: ¶46 discloses receiving programming code through interaction with the IDE interface, ¶52 discloses that the IDE supports both HMI development and industrial controller programming and ¶54 discloses deploying the control code for execution on an industrial controller to implement the automation project),
The proposed combination of Mabote, Stump and Wang does not disclose and the generative AI component is further configured to formulate a second natural language summary of functionality of the control code based on analysis of the control code and the content of the one or more custom models.
However, Chen discloses and the generative AI component is further configured to formulate a second natural language summary of functionality of the control code based on analysis of the control code and the content of the one or more custom models (Chen: ¶46 discloses a machine learning model that analyzes programming code input and generates natural language describing the code’s purpose and functionality, ¶111 discloses using a docstring generation model to generate natural language text based on received computer code samples and identify the intent of the code).
The combined system of Mabote, Stump, Wang and Chen are combinable because they are from the same field of endeavor, i.e., both disclose interactive systems that assist in development and understanding of a program. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to disclose Chen’s code summarization model into the combined industrial IDE so that in addition to Wang’s HMI summary, the system would generate a separate natural language summary explaining the functionality of the received industrial control code. Chen explicitly teaches “generating natural language summaries of code, which can be useful for quickly understanding the purpose and functionality of a codebase” in ¶114.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to IAN SCOTT MCLEAN whose telephone number is (703)756-4599. The examiner can normally be reached "Monday - Friday 8:00-5:00 EST, off Every 2nd Friday".
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Hai Phan can be reached at (571) 272-6338. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/IAN SCOTT MCLEAN/Examiner, Art Unit 2654
/HAI PHAN/Supervisory Patent Examiner, Art Unit 2654
