Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 05/23/2024 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claim(s) 1, 8, 11, 14, and 17-18 is/are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Pan, X., Huan, Z., Li, Y., & Cao, Y. (2024, January 30). Enhancement of GUI display error detection using improved faster R-CNN and multi-scale attention mechanism. MDPI. https://www.mdpi.com/2076-3417/14/3/1144, hereinafter “Pan”.
Claim 1:
Pan teaches an electronic device comprising: at least one electronic processor (i.e. pg. 11, Section 5. Experiments, The experimental hardware configuration for this study comprises an Intel Xeon Cas cade Lake 8255C(2.5 GHz)processor) configured to: make a first call via an accessibility application programming interface (“API”) (i.e. pg. 5, Section 4. Research Method, “The algorithm systematically traverses the JSON file, extracting relevant component information, such as TextView and ImageView.”, wherein the BRI for and accessibility API encompasses the algorithm to extract certain GUI component information from associated GUI screenshots) to request first accessibility data associated with a first user interface (“UI”) displayed by a first device (i.e. pg. 5, Section 4. Research Method, “Our approach relies on the Rico dataset [34], comprising design data from 93,000 Android applications and offering over 66,000 UI screen captures, each paired with its corresponding JSON file”, wherein the BRI for first accessibility data encompasses the retrieved UI display data of at least one device that may be an android device); receive the first accessibility data via the accessibility API, the first accessibility data comprising at least one of (a) information identifying one or more UI elements in the first UI or (b) information identifying one or more UI events in the first UI (i.e. pg. 5, Section 4. Research Method, “Our approach relies on the Rico dataset [34], comprising design data from 93,000 Android applications and offering over 66,000 UI screen captures, each paired with its corresponding JSON file”, wherein it is noted that a first device may be an android device and have an associated screen capture with information identifying one or more UI elements, such as through a corresponding JSON file); train (i.e. pg. 11-12, Section 5.1. Experiment settings, “pre-training Appl. Sci. 2024, 14, 1144 12 of 16 is executed on the COCO2017 dataset, followed by training the enhanced Faster R-CNN algorithm using the pre-trained parameters. The objective is to identify four types of GUI display issues: image missing, component occlusion, text overlap, and empty values”, wherein an R-CNN model is trained on errors that have been added to and thus associated with at least a first device), based on at least the first accessibility data and an error state associated with the first device, an artificial intelligence (“AI”) model (i.e. pg. 5, Section 4. Research Method, “the algorithm systematically traverses the JSON file, extracting relevant component information, such as TextView and ImageView. Utilizing this component information, Algorithm 1 duplicates and adjusts positions and sizes according to predefined rules, thereby generating UI screen captures with the associated issues”, wherein it is noted that the BRI for an error state associated with the first device encompasses how an error may be added to a UI associated with the first device, which is subsequently used as training data); make a second call via the accessibility API to request second accessibility data associated with a second UI displayed by a second device (i.e. pg. 14. Section 5.2. Experiment Result Analysis - Section 6. Conclusions, “the methodology can be transferred and extended to detect GUI display problems on other platforms. For instance, we identified similar display issues on the Windows platform, affirming the universality of these problems… this research approach offers valuable insights for software automation testing with broad applications, not only in the industrial sector but also for various terminal devices employing GUI displays”, wherein it is noted that a first device may be an android device and that a second device may be a windows platform device); receive the second accessibility data via the accessibility API, the second accessibility data comprising
(a) information identifying one or more UI elements in the second UI or (b) information identifying one or more UI events in the second UI (i.e. pg. 14, Section 5.2. Experiment Result Analysis “screenshots are employed to detect GUI display issues. Due to the minimal differences observed in screenshots from various platforms, the methodology can be transferred and extended… we identified similar display issues on the Windows platform”, wherein a second device may have second accessibility data identified relating to a windows GUI display and windows GUI elements); and detect, based on at least the second accessibility data and the AI model, the error state on the second device (i.e. pg. 14, “the methodology can be transferred and extended to detect GUI display problems on other platforms. For instance, we identified similar display issues on the Windows platform, affirming the universality of these problems”, wherein it is noted that the model produces error identification of similar errors found in android GUI interfaces as well as in windows GUI interfaces).
Claim 8:
Pan teaches the electronic device of claim 1, wherein the information identifying the one or more UI elements in the first UI includes at least one selected from the group consisting of: (a) a respective element type of the one or more UI elements in the first UI, (b) a respective identifier of the one or more UI elements in the first UI, or (c) a respective state or condition of the one or more UI elements in the first UI (i.e. pg. 1-2, “This involves leveraging deep learning for the automatic recognition of GUI elements in screen captures. The recognition of GUI elements within images, based on pixels, can be viewed as a specialized object detection task within the domain of GUI testing”, wherein pixels are used to identify certain UI elements types, locations, and a state of occlusion).
Claim 11:
Pan teaches the electronic device of claim 1,
wherein the information identifying the one or more UI events in the first UI includes one or more notifications of changes in states or conditions of the UI elements in the first UI (i.e. pg. 5, 4. Research method, The algorithm systematically traverses the JSON file, extracting relevant component information, such as TextView and ImageView. Utilizing this component information, Algorithm 1 duplicates and adjusts positions and sizes according to predefined rules, thereby generating UI screen captures with the associated issues… In the event of a component occlusion error, the text or image becomes obscured by other components).
Claim 14:
Pan teaches the electronic device of claim 1, wherein the first accessibility data is generated based on metadata associated with the one or more UI elements in the first UI, the metadata being exposed via the accessibility API (i.e. pg. 12, Section 5.1. Experiment Settings, “This experiment is dedicated to investigating four prevalent GUI display issues: component occlusion, image missing, text overlap, and empty values.”, wherein the BRI for metadata encompasses data about the type of error related to an identified UI element).
Claim 17:
Pan teaches the electronic device of claim 1, wherein the first accessibility data is generated by a platform executing on the first device during rendering of the first UI (i.e. pg. 12, 5.1. Experiment Settings, “Leveraging the Rico dataset, an algorithm is employed to autonomously produce images representative of these aforementioned GUI issues. As shown in Table 1, the resulting training dataset comprises a total of 40,000 images, evenly distributed with 10,000 images for each issue type, accompanied by corresponding annotation files”, wherein the initial GUI image data is retrieved from an android device running an application and augmented to contain a GUI issue).
Claim 18:
Claim 18 is the method claim reciting similar limitations to Claim 1 and is rejected for similar reasons.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 2-5 and 19 is/are rejected under 35 U.S.C. 103 as being unpatentable over Pan, X., Huan, Z., Li, Y., & Cao, Y. (2024, January 30). Enhancement of GUI display error detection using improved faster R-CNN and multi-scale attention mechanism. MDPI. https://www.mdpi.com/2076-3417/14/3/1144, hereinafter “Pan” and further light of U.S. Patent Application Publication NO. 20220114045 “Koehler”.
Claim 2:
Pan teaches the electronic device of claim 1.
While Pan teaches identifying the error state of the first device (i.e. pg. 6, Section 4. Research Method, When confronted with a text overlap error, characterized by the overlapping of text content, thereby impairing readability, Algorithm 1 initially acquires the position and size Appl. Sci. 2024, 14, 1144 7 of 16 of the TextView),
Pan may not explicitly teach
wherein the error state of the first device is reported through a helpdesk service.
However, Koehler teaches
wherein the error state of the first device is reported through a helpdesk service (i.e. para. [0097], FIG. 8 is a view of table 72 which lists various remediation action(s) that the remediation action module 42 may be configured to take when a particular sensor is activated Examples of corrective actions include… the transmitting of a notification to a predetermined destination such as a help desk or a human via various means (reference numeral 86).
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to add wherein the error state of the first device is reported through a helpdesk service, to the error state detection of Pan, with an error state may be sent to a help desk, as taught by Coleman. One would have been motivated to combine Pan with the help desk of Coleman and would have had a reasonable expectation of success as the combination provides assistance to develop a historic record and then using the historic record to determine when current behavior or configuration varies from the historic and alert of such variance and/or take remedial action.
Claim 3:
Pan teaches the electronic device of claim 1.
Pan may not explicitly teach
wherein the at least one electronic processor is further configured to determine the error state of the first device based on at least first endpoint management data, the first endpoint management data including state information of the first device.
However, Koehler teaches
wherein the at least one electronic processor is further configured to determine the error state of the first device based on at least first endpoint management data, the first endpoint management data including state information of the first device (i.e. para. [0041], “embodiments may be configured to track the behavior of the managed computer system, and to compare current behavior with the learned proper behavior (operation) that is determined over time (i.e., historic record). When a problem, failure, and/or an anomaly is detected, such embodiments may be configured to take remedial action”, wherein the BRI for first endpoint management data encompasses the behavior data indicates failure or an anomaly state information of a first managed device).
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to add wherein the at least one electronic processor is further configured to determine the error state of the first device based on at least first endpoint management data, the first endpoint management data including state information of the first device, to the error state detection of Pan, with wherein the at least one electronic processor is further configured to determine the error state of the first device based on at least first endpoint management data, the first endpoint management data including state information of the first device, as taught by Koehler. One would have been motivated to combine Pan with the help desk of Koehler and would have had a reasonable expectation of success as the combination provides assistance to develop a historic record and then using the historic record to determine when current behavior or configuration varies from the historic and alert of such variance and/or take remedial action.
Claim 4:
Pan teaches the electronic device of claim 1.
Pan may not explicitly teach
wherein the AI model is further trained based on at least a solution to the error state, and wherein the at least one electronic processor is further configured to: output, based on at least the AI model, a command to an agent executing on the second device to implement the solution on the second device.
However, Koehler teaches
wherein the AI model (i.e. para. [0074], FIG. 4 is a table 48 showing a plurality of sensors as included in the alert detection system 26. In an embodiment, each sensor has an associated algorithm… A particular sensor algorithm may therefore be configured to allow for the detection of complex conditions using an array of conditions, logic, artificial intelligence (AI) techniques, and the like) is further trained based on at least a solution to the error state (i.e. para. [0097], “ FIG. 8 is a view of table 72 which lists various remediation action(s) that the remediation action module 42 may be configured to take when a particular sensor is activated. It should be appreciated that it sometimes possible to automatically remediate problems, failures, and/or anomalies by the invocation of an action”, wherein a solution may be presented in response to a sensed error or faulting state), and wherein the at least one electronic processor is further configured to: output, based on at least the AI model, a command to an agent executing on the second device to implement the solution on the second device (i.e. para. [0097], “Examples of corrective actions … the transmitting of a notification to a predetermined destination such as a help desk or a human via various means (reference numeral 86), and, generally, execution of a script or a program”, wherein a help desk person may have the relevant script or program solution transferred to their device)
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to add the at least one electronic processor is further configured to: output, based on at least the AI model, a command to an agent executing on the second device to implement the solution on the second device, to the error state detection of Pan, with the at least one electronic processor is further configured to: output, based on at least the AI model, a command to an agent executing on the second device to implement the solution on the second device, as taught by Koehler. One would have been motivated to combine Pan with the help desk of Koehler and would have had a reasonable expectation of success as the combination provides assistance to develop a historic record and then using the historic record to determine when current behavior or configuration varies from the historic and alert of such variance and/or take remedial action.
Claim 5:
Pan and Koehler teach the electronic device of claim 4.
Koehler further teaches
wherein the solution includes at least one selected from a group consisting of restarting at least one application executing on the second device, modifying an application configuration of at least one application of the second device, and modifying a network configuration of the second device (i.e. para. [0097], examples of corrective actions include the restarting of a faulting application (reference numeral 74), the rebooting of the operating system (reference numeral 76), the restarting of the entire device, such as the managed computer system (reference numeral 78), the resetting of a problematic peripheral such as a network adapter (reference numeral 80), the resetting of a device attached to the managed computer system (reference numeral 82), the restarting of a software service on the managed computer system (reference numeral 84)).
Claim 19:
Claim 19 is the method claim reciting similar limitations to Claim 4 and is rejected for similar reasons.
Claim(s) 6-7 and 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Pan, X., Huan, Z., Li, Y., & Cao, Y. (2024, January 30). Enhancement of GUI display error detection using improved faster R-CNN and multi-scale attention mechanism. MDPI. https://www.mdpi.com/2076-3417/14/3/1144, hereinafter “Pan” and further light of U.S. Patent NO. 12056006 “Bishop”.
Claim 6:
Pan teaches the electronic device of claim 1.
Pan may not explicitly teach
wherein the AI model is further trained based on first endpoint management data and a solution to the error state, the first endpoint management data including state information of the first device, wherein the at least one electronic processor is further configured to: predict, based on at least the AI model and second endpoint management data including state information of a third device, an occurrence of the error state on the third device; and output, based on at least the AI model, a command to an agent executing on the third device to preemptively implement the solution on the third device.
However, Bishop teaches
wherein the AI model (i.e. Fig. 3, a proactive intelligence model 303) is further trained based on first endpoint management data and a solution to the error state, the first endpoint management data including state information of the first device (i.e. Col, 6, lines 53-65, “a given asset signature is generated for at least a given one of the plurality of IT assets. The given asset signature is generated based at least in part on the obtained monitoring data. The given asset signature represents a status of the given IT asset. The obtained monitoring data may comprise, for the given IT asset: one or more support tickets associated with the given IT asset; … crowdsourced issue indicators and at least one of diagnosis and remediation data for the crowdsourced issue indicators;”, wherein the remediation of a problem in a first client device may be stored in section for known remediations), wherein the at least one electronic processor is further configured to: predict, based on at least the AI model and second endpoint management data including state information of a third device, an occurrence of the error state on the third device (i.e. Col. 11, Lines 58-64 Fig. 4, The asset and issue signatures allow IT asset-to-IT asset comparison as well as issue pattern-to-IT asset comparison in steps 411 and 413, making it possible to detect affected IT assets based on incoming service requests 410-3, and for issues not yet known by the model); and output, based on at least the AI model, a command to an agent executing on the third device to preemptively implement the solution on the third device (i.e. Col. 13, lines 51-61 and Col 14, lines 1-5, Fig. 4, “when the signature of an IT asset or a group of IT assets matches the signature or pattern of the given issue, the known remediation actions may be applied (e.g., corresponding to a rule-based analysis in steps 401-403 of the process flow 400, a positive determination in step 419 in the process flow 400… The technical solutions described herein, however, provide another option where the proactive intelligence model 303 is able to automatically generate remediation actions (e.g., scripts)”, wherein a known remediation may be derived from a first client).
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to add wherein the AI model is further trained based on first endpoint management data and a solution to the error state, the first endpoint management data including state information of the first device, wherein the at least one electronic processor is further configured to: predict, based on at least the AI model and second endpoint management data including state information of a third device, an occurrence of the error state on the third device; and output, based on at least the AI model, a command to an agent executing on the third device to preemptively implement the solution on the third device, to the GUI error state detection of Pan, with how a proactive model may look at telemetry data and preempt a fix on a separate device using a solution learned from a first device, as taught by Bishop. One would have been motivated to combine GUI data diagnosis Pan with the automated prediction and solution execution of Bishop and would have had a reasonable expectation of success as the combination provides a novel approach for successfully determining algorithm inputs and functions for the IT asset or IT asset group (e.g., a PC fleet) applicability of a curated remediation recommendation.
Claim 7:
Pan and Bishop teach the electronic device of claim 6.
Bishop further teaches wherein the at least one electronic processor is further configured to: receive third endpoint management data after outputting the command, the third endpoint management device including state information of the third device; and train the AI model based on at least the third endpoint management data (i.e. Col, 10 lines 45-56, In step 425, a remediation outcome is stored for each of the managed IT assets 306 on which the remediation script was deployed in step 423. In step 427, a determination is made as to whether the remediation was effective for each of the managed IT assets 306 on which the remediation script was deployed in step 423. If the result of the step 427 determination is no (e.g., remediation was not effective), rollback of the remediation script is performed in step 429. Following step 429, or if the result of the step 427 determination is yes (e.g., remediation was effective), remediation feedback is provided to the proactive intelligence model 303 in step 431).
Claim 20:
Pan teaches an electronic device comprising: at least one electronic processor (i.e. pg. 11, Section 5. Experiments, The experimental hardware configuration for this study comprises an Intel Xeon Cas cade Lake 8255C(2.5 GHz)processor) configured to: make a first call via an accessibility application programming interface (“API”) (i.e. pg. 5, Section 4. Research Method, “The algorithm systematically traverses the JSON file, extracting relevant component information, such as TextView and ImageView.”, wherein the BRI for and accessibility API encompasses the algorithm to extract certain GUI component information from associated GUI screenshots) to request first accessibility data associated with a first user interface (“UI”) displayed by a first device (i.e. pg. 5, Section 4. Research Method, “Our approach relies on the Rico dataset [34], comprising design data from 93,000 Android applications and offering over 66,000 UI screen captures, each paired with its corresponding JSON file”, wherein the BRI for first accessibility data encompasses the retrieved UI display data of at least one device that may be an android device); receive the first accessibility data via the accessibility API, the first accessibility data comprising at least one of (a) information identifying one or more UI elements in the first UI or (b) information identifying one or more UI events in the first UI (i.e. pg. 5, Section 4. Research Method, “Our approach relies on the Rico dataset [34], comprising design data from 93,000 Android applications and offering over 66,000 UI screen captures, each paired with its corresponding JSON file”, wherein it is noted that a first device may be an android device and have an associated screen capture with information identifying one or more UI elements, such as through a corresponding JSON file); train an artificial intelligence (“AI”) model (i.e. pg. 11-12, Section 5.1. Experiment settings, “pre-training Appl. Sci. 2024, 14, 1144 12 of 16 is executed on the COCO2017 dataset, followed by training the enhanced Faster R-CNN algorithm using the pre-trained parameters. The objective is to identify four types of GUI display issues: image missing, component occlusion, text overlap, and empty values”, wherein an R-CNN model is trained on errors that have been added to and thus associated with at least a first device), based on at least the first accessibility data, an error state associated with the first device, and endpoint management data associated with the first device, the endpoint management data including state information of the first device (i.e. pg. 5, Section 4. Research Method, “the algorithm systematically traverses the JSON file, extracting relevant component information, such as TextView and ImageView. Utilizing this component information, Algorithm 1 duplicates and adjusts positions and sizes according to predefined rules, thereby generating UI screen captures with the associated issues”, wherein it is noted that the BRI for an error state associated with the first device encompasses how an error may be added to a UI associated with the first device, which is subsequently used as training data. Wherein the BRI for endpoint management data encompasses metadata indicating a GUI display issue with a screenshot from a first device); and (i.e. pg. 14, “the methodology can be transferred and extended to detect GUI display problems on other platforms. For instance, we identified similar display issues on the Windows platform, affirming the universality of these problems”, wherein it is noted that the model produces error identification of similar errors found in android GUI interfaces as well as in windows GUI interfaces).
While Pan teaches learning a problem from a first type of device and applying the knowledge to a second type of device to identify a similar issue, Pan may not explicitly teach to
predict, based on at least the AI model and second endpoint management data including state information of a second device, an occurrence of the error state on the second device
However, Bishop teaches to
predict, based on at least the AI model and second endpoint management data including state information of a second device, an occurrence of the error state on the second device (i.e. Col. 7, Lines 18-24, Fig. 4, “Responsive to determining that the given asset signature exhibits at least a threshold level of asset-to-issue similarity with at least a given one of the one or more issue signatures associated with at least a given one of the one or more issues, one or more proactive remedial actions are selected in step 206 for remedying the given issue. The selected one or more proactive remedial actions are applied to the given IT asset prior to detecting that the given IT asset has encountered the given issue”, wherein the BRI to predict an occurrence of an error state encompasses how a proactive action is taken when a second device shows known signs of a given issue that may have been learned from a first IT asset)
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to add predict, based on at least the AI model and second endpoint management data including state information of a second device, an occurrence of the error state on the second device, to the GUI error state detection of Pan, with how a proactive model may look at telemetry data and preempt a fix on a separate device using a solution learned from a first device, as taught by Bishop. One would have been motivated to combine GUI data diagnosis Pan with the automated prediction and solution execution of Bishop and would have had a reasonable expectation of success as the combination provides a novel approach for successfully determining algorithm inputs and functions for the IT asset or IT asset group (e.g., a PC fleet) applicability of a curated remediation recommendation.
Claim(s) 9-10 and 16 is/are rejected under 35 U.S.C. 103 as being unpatentable over Pan, X., Huan, Z., Li, Y., & Cao, Y. (2024, January 30). Enhancement of GUI display error detection using improved faster R-CNN and multi-scale attention mechanism. MDPI. https://www.mdpi.com/2076-3417/14/3/1144, hereinafter “Pan” and further light of U.S. Patent Application Publication NO. 20230351257 “Mourya”.
Claim 9:
Pan teaches the electronic device of claim 1.
Pan may not explicitly teach
wherein the information identifying the one or more UI elements in the first UI is organized as a hierarchical tree.
However, Mourya teaches
wherein the information identifying the one or more UI elements in the first UI is organized as a hierarchical tree (i.e. para. [0231], Tree Structures: Tree structures, such as hierarchical data structures, may be used to represent the organization and relationships between different components of the GUI).
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to add wherein the information identifying the one or more UI elements in the first UI is organized as a hierarchical tree, to the GUI error state detection of Pan, with wherein the information identifying the one or more UI elements in the first UI is organized as a hierarchical tree, as taught by Mourya. One would have been motivated to combine Pan with the Mourya and would have had a reasonable expectation of success as the combination provides significant reduction in manual effort for identifying new intents and training phrases, integration into existing processing pipelines, and enhancing overall system efficiency.
Claim 10:
Pan and Mourya teach the electronic device of claim 9.
Mourya further teaches wherein the one or more UI elements in the first UI comprises a first UI element that includes a second UI element; the hierarchical tree comprises a first hierarchical level that is above a second hierarchical level (i.e. para. [0231], hierarchical data structures, may be used to represent the organization and relationships between different components of the GUI); the first hierarchical level includes a first node representing the first UI element; and the second hierarchical level includes a second node representing the second UI element (i.e. para. [0231], if the GUI 900 has nested menus or sub-options, a tree structure can be employed to represent these relationships).
Claim 16:
Pan teaches the electronic device of claim 14.
Pan may not explicitly teach
wherein the metadata specifies hierarchical relationships between the one or more UI elements in the first UI.
However, Mourya teaches
wherein the metadata specifies hierarchical relationships between the one or more UI elements in the first UI (i.e. para. [0231], “Tree Structures: Tree structures, such as hierarchical data structures, may be used to represent the organization and relationships between different components of the GUI”, wherein the metadata includes data about the hierarchical tree for the different nested menus and sub-options within a GUI).
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to add wherein the metadata specifies hierarchical relationships between the one or more UI elements in the first UI, to the GUI error state detection of Pan, with wherein the metadata specifies hierarchical relationships between the one or more UI elements in the first UI, as taught by Mourya. One would have been motivated to combine Pan with the Mourya and would have had a reasonable expectation of success as the combination provides significant reduction in manual effort for identifying new intents and training phrases, integration into existing processing pipelines, and enhancing overall system efficiency.
Claim(s) 12-13 is/are rejected under 35 U.S.C. 103 as being unpatentable over Pan, X., Huan, Z., Li, Y., & Cao, Y. (2024, January 30). Enhancement of GUI display error detection using improved faster R-CNN and multi-scale attention mechanism. MDPI. https://www.mdpi.com/2076-3417/14/3/1144, hereinafter “Pan” and further light of U.S. Patent Application Publication NO. 20210342709 “Klug”.
Claim 12:
Pan teaches the electronic device of claim 1,
While Pan teaches a first call via an accessibility API to identify elements within a GUI (i.e. pg. 5, Section 4. Research Method, “Our approach relies on the Rico dataset [34], comprising design data from 93,000 Android applications and offering over 66,000 UI screen captures, each paired with its corresponding JSON file”, wherein the object detection algorithm retrieves GUI data from screenshots of android platform applications that have rendered the GUI), Pan may not explicitly teach
wherein the first call made via the accessibility API is made to a platform rendering the first UI.
However, Klug teaches
wherein the first call made via the accessibility API is made to a platform rendering the first UI (i.e. para. [0082], “a user may navigate to a sub-topic using a graphical user interface and a graph display that appears in a visual listing format (e.g., a tree format). Alternatively, section groups may be sent in a data format in response to API calls from services, servers, or requesting devices”, wherein in an alternative embodiment a tree format for a GUI may be called via an API call to a device from a requesting server).
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to wherein the first call made via the accessibility API is made to a platform rendering the first UI, to the GUI error state detection of Pan, with an API call is made to a device in order to retrieve it’s accessibility data, as taught by Klug. One would have been motivated to combine Pan with the Klug and would have had a reasonable expectation of success as the combination would use API (application programming interface) calls in order to enable a user to better solve a problem or address a complex issue. .
Claim 13:
Pan and Klug teach the electronic device of claim 12.
Klug further teaches wherein the platform includes one of an operating system executing on the first device or a browser application executing on the first device (i.e. para. [0091], The virtual machines may be run as guests by a hypervisor, as described further below. The PaaS model delivers a computing system that may include an operating system, programming language execution environment, database, and web server).
Claim(s) 15 is/are rejected under 35 U.S.C. 103 as being unpatentable over Pan, X., Huan, Z., Li, Y., & Cao, Y. (2024, January 30). Enhancement of GUI display error detection using improved faster R-CNN and multi-scale attention mechanism. MDPI. https://www.mdpi.com/2076-3417/14/3/1144, hereinafter “Pan” and further light of U.S. Patent Application Publication NO. 20230057720 “Aradhya”.
Claim 15:
Pan teaches the electronic device of claim 14.
Pan may not explicitly teach
wherein the metadata is specified in one of (a) application code of application for which the first UI is being rendered or (b) content code of web content for which the first UI is being rendered.
However, Aradhya teaches
wherein the metadata is specified in one of (a) application code of application for which the first UI is being rendered or (b) content code of web content for which the first UI is being rendered (i.e. para. [0045], “ the logs reflect a replication error event from an RPO deviation indicated on the UI, which itself was caused by an RPO Deviation. The built application error graph 200 includes RPO Deviation (node 202) at its root, followed by RPO Deviation on UI (node 204) and Replication Error Event (node 206).”, wherein the BRI for metadata specified in an application code encompasses how an application error graph tracks the deviation of a UI element).
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to add wherein the metadata is specified in one of (a) application code of application for which the first UI is being rendered or (b) content code of web content for which the first UI is being rendered, to the GUI error state detection of Pan, with wherein the metadata is specified in one of (a) application code of application for which the first UI is being rendered or (b) content code of web content for which the first UI is being rendered, as taught by Aradhya. One would have been motivated to combine Pan with the Aradhya and would have had a reasonable expectation of success as the combination provides the identification of hedge code for each feature as described herein facilitates an immediate focus on the likely root cause of the issue, which can be extremely advantageous to a developer who is not familiar with the feature.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
U.S. Patent Application Publication NO. 20230216913 “Goldberg”, para. [0015], teaches that in other aspects the response indicates accessibility of the website data the second software uses has not changed based the first RPA program modifies how it issues alerts based on the comparison to reduce a likelihood that the alert associated with the response would have been issued.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DAVID H TAN whose telephone number is (571)272-7433. The examiner can normally be reached M-F 7:30-4:30.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Cesar Paula can be reached at (571) 272-4128. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/D.T./Examiner, Art Unit 2145
/CESAR B PAULA/Supervisory Patent Examiner, Art Unit 2145
Prosecution Insights