DETAILED ACTION
Election/Restrictions
Applicant’s election without traverse of Species I, claims 1-17 and 20, in the reply filed on 12/17/2025 is acknowledged.
Claims 18-20 are withdrawn from further consideration pursuant to 37 CFR 1.142(b) as being drawn to a nonelected SPECIES, there being no allowable generic or linking claim. Election was made without traverse in the reply filed on 12/17/2025.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claim 16 recites the limitation "one or more machine (ML) models". There is insufficient antecedent basis for this limitation in the claim because it is unclear if this one or more machine (ML) models is the same as the one or more machine (ML) models in claim 15, or a newly introduced one or more machine (ML) models. Claim 16 then recites “the one or more ML models”, and this is unclear if this refers to the one or more machine (ML) models in claim 15 or 16.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claim(s) 1, 4-6, 15-16 is/are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Li et al (NPL: Screen2Vec: Semantic Embedding of GUI Screens and GUI Components).
Regarding claim 1, Li discloses a system comprising:
at least one processor (pg. 2 Figure 1 illustrates the architecture of Screen2Vec. Overall, the pipeline of Screen2Vec consists of two levels: the GUI component level (shown in the gray shade) and the GUI screen level.; The GUI component level model encodes the textual content; the GUI component would need a processor to operate (e.g. such as to encode)); and
memory storing instructions that, when executed by the at least one processor, cause the system to perform a set of operations (CRM is inherently present for the architecture of Screen2Vec to process, where CRM would be implemented by a processor that requires a CRM, e.g., a RAM, to function), the set of operations comprising:
capturing a current screenshot of a window associated with performing a first activity (pg. 3 2.1 Dataset: The Rico dataset contains interaction traces on 66,261 unique GUI screens from 9,384 free Android apps collected using a hybrid crowdsourcing plus automated discovery approach; the Rico dataset includes a screenshot image) ;
processing image information associated with the current screenshot into a set of current screenshot embeddings (pg. 4 Layout Embeddings: We then use an image autoencoder to encode each image into a 64-dimensional embedding vector; GUI Embedding Combining Layer: To combine the embedding vectors of multiple GUI components on a screen into a single fixedlength embedding vector, we use an Recurrent Neural Network (RNN);
receiving a plurality of trigger embeddings for a plurality of trigger screenshots defined for a plurality of plugins (pg. 3 After training, the description embedding vector is concatenated on, resulting in the 1536-dimensional GUI screen embedding vector (if included in the training, the description dominates the entire embedding, overshadowing information specific to that screen within the app).;
determining semantic similarities between the set of current screenshot embeddings and the plurality of trigger embeddings (pg. 7 3.1 Nearest Neighbors: We conducted a Mechanical Turk study to compare the similarity between the nearest neighbor results generated by the different models. We selected 50 screens from apps and app domains that most users are familiar with; Table 4 shows the mean screen similarity rated by the Mechanical Turk workers for the top-5 nearest neighbor results of the sample source screens generated by the 3 models);
determining a top subset of trigger screenshots of the plurality of trigger screenshots having greater semantic similarity to the current screenshot (pg. 7 3.1 Nearest Neighbors: for a given screen, what are the top-N most similar screens in the dataset?); and
detecting an activity trigger for each plugin of a top subset of plugins corresponding to the top subset of trigger screenshots, wherein each plugin is associated with performing a second activity related to the first activity (pg. 1 abstract: Through several sample downstream tasks, we demonstrate Screen2Vec’s key useful properties: representing between-screen similarity through nearest neighbors, composability, and capability to represent user tasks; pg. 10 3.3 Screen Embedding Sequences for Representing Mobile Tasks: GUI screens are not only useful data sources individually on their own, but also as building blocks to represent a user’s task. A task in an app, or across multiple apps, can be represented as a sequence of GUI screens that makes up the user interaction trace for performing this task using app GUIs).
Regarding claim 4, Li disclose the system of claim 1, wherein the image information of the current screenshot is processed using one or more machine learning (ML) models (Li pg. 2 1 INTRODUCTION: We present a new self-supervised technique (i.e., the type of machine learning approach that trains a model without human-labeled data by withholding some part of the data, and tasking the network with predicting it) Screen2Vec for generating more comprehensive semantic representations of GUI screens and components)
Regarding claim 5, Li disclose the system of claim 4, wherein the one or more ML models includes at least one of a screen region detection function or a semantic embedding model (Li pg. 2 1 INTRODUCTION: Screen2Vec: a new self-supervised technique for generating more comprehensive semantic embeddings of GUI screens and components using their textual content, visual design and layout patterns, and app meta-data.).
Regarding claim 6, the combination of Li and Andre disclose the system of claim 1, wherein each trigger screenshot of the plurality of trigger screenshots is processed using one or more ML models, and wherein the one or more ML models includes at least one of a screen region detection function or a semantic embedding model (Li pg. 2 1 INTRODUCTION: Screen2Vec: a new self-supervised technique for generating more comprehensive semantic embeddings of GUI screens and components using their textual content, visual design and layout patterns, and app meta-data).
Regarding claim 15, Li teaches a method of detecting an activity trigger for one or more plugins, comprising:
capturing a current screenshot of a window associated with performing a first activity (pg. 3 2.1 Dataset: the Rico dataset includes a screenshot image) ;
using one or more machine learning (ML) models to process image information associated with the current screenshot into a set of current screenshot embeddings (pg. 4 Layout Embeddings: We then use an image autoencoder to encode each image into a 64-dimensional embedding vector; GUI Embedding Combining Layer: To combine the embedding vectors of multiple GUI components on a screen into a single fixedlength embedding vector, we use an Recurrent Neural Network (RNN);
receiving a plurality of trigger embeddings for a plurality of trigger screenshots defined for a plurality of plugins (pg. 3 After training, the description embedding vector is concatenated on, resulting in the 1536-dimensional GUI screen embedding vector (if included in the training, the description dominates the entire embedding, overshadowing information specific to that screen within the app).;;
determining semantic similarities between the set of current screenshot embeddings and the plurality of trigger embeddings (pg. 7 3.1 Nearest Neighbors: We conducted a Mechanical Turk study to compare the similarity between the nearest neighbor results generated by the different models. We selected 50 screens from apps and app domains that most users are familiar with; Table 4 shows the mean screen similarity rated by the Mechanical Turk workers for the top-5 nearest neighbor results of the sample source screens generated by the 3 models);
determining a top subset of trigger screenshots of the plurality of trigger screenshots having greater semantic similarity to the current screenshot (pg. 7 3.1 Nearest Neighbors: for a given screen, what are the top-N most similar screens in the dataset?); and
detecting an activity trigger for each plugin of a top subset of plugins corresponding to the top subset of trigger screenshots, wherein each plugin is associated with performing a second activity related to the first activity (pg. 1 abstract: Through several sample downstream tasks, we demonstrate Screen2Vec’s key useful properties: representing between-screen similarity through nearest neighbors, composability, and capability to represent user tasks; pg. 10 3.3 Screen Embedding Sequences for Representing Mobile Tasks: GUI screens are not only useful data sources individually on their own, but also as building blocks to represent a user’s task. A task in an app, or across multiple apps, can be represented as a sequence of GUI screens that makes up the user interaction trace for performing this task using app GUIs)..
Regarding claim 16, Li teaches the method of claim 15, wherein the plurality of trigger screenshots is processed using one or more machine learning (ML) models (Li pg. 2 1 INTRODUCTION: We present a new self-supervised technique (i.e., the type of machine learning approach that trains a model without human-labeled data by withholding some part of the data, and tasking the network with predicting it) Screen2Vec for generating more comprehensive semantic representations of GUI screens and components), and wherein the one or more ML models includes at least one of a screen region detection function or a semantic embedding model (Li pg. 2 1 INTRODUCTION: Screen2Vec: a new self-supervised technique for generating more comprehensive semantic embeddings of GUI screens and components using their textual content, visual design and layout patterns, and app meta-data.).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 2 is/are rejected under 35 U.S.C. 103 as being unpatentable over Li as applied to claim 1 above, and further in view of Andre et al (US 20230359789).
Regarding claim 2, Li disclose the system of claim 1, but fails to teach where Andre teaches wherein the first activity is performed using one of an application (Andre ¶17 The simulation of an action set can be performed in a simulated environment that reflects a corresponding state of the domain. For example, the initial simulated environment can reflect the CAD application in its current state), a website, or a plugin displayed in the window.
Therefore, it would have been obvious to one with ordinary skill in the art before the effective filing date of the invention to have implemented the teaching of wherein the first activity is performed using one of an application a website, or a plugin displayed in the window from Andre into the system as disclosed by Li. The motivation for doing this is to improve selecting context-relevant actions.
Claim(s) 3 and 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Li as applied to claim 1 and 15 above, and further in view of Li (US 20240338234, hereinafter Li-Google).
Regarding claim 3, Li disclose the system of claim 1, but fails to teach where Li-Google teaches the set of operations further comprising: causing display of a notification regarding the activity trigger for each plugin of the top subset of plugins, wherein each plugin is selectable from the notification (¶72 For any given screenshot, all the UI elements are scored, and the one with the highest score is selected for possible action; ¶76-79 the image sequence of FIG. 6 shows the four steps of the focus_and_type action; ¶91 when screen representation comes from a DOM tree or accessibility tree, an action always succeeds. Besides, after a reference to an element is obtained, an action can be applied no matter where the element moves)
Therefore, it would have been obvious to one with ordinary skill in the art before the effective filing date of the invention to have implemented the teaching of the set of operations further comprising: causing display of a notification regarding the activity trigger for each plugin of the top subset of plugins, wherein each plugin is selectable from the notification from Li-Google into the system as disclosed by Li. The motivation for doing this is to reliably build agents capable of user interface (UI) navigation.
Regarding claim 20, Li disclose the method of claim 15, but fails to teach where Li-Google teaches causing display of a notification regarding the activity trigger for each plugin of the top subset of plugins, wherein each plugin is selectable from the notification (¶72 For any given screenshot, all the UI elements are scored, and the one with the highest score is selected for possible action; ¶76-79 the image sequence of FIG. 6 shows the four steps of the focus_and_type action; ¶91 when screen representation comes from a DOM tree or accessibility tree, an action always succeeds. Besides, after a reference to an element is obtained, an action can be applied no matter where the element moves).
Therefore, it would have been obvious to one with ordinary skill in the art before the effective filing date of the invention to have implemented the teaching of causing display of a notification regarding the activity trigger for each plugin of the top subset of plugins, wherein each plugin is selectable from the notification from Li-Google into the method as disclosed by Li. The motivation for doing this is to reliably build agents capable of user interface (UI) navigation.
Claim(s) 10-11 is/are rejected under 35 U.S.C. 103 as being unpatentable over Li et al (NPL: Screen2Vec: Semantic Embedding of GUI Screens and GUI Components) in view of Li (US 20240338234, hereinafter Li-Google).
Regarding claim 10, Li discloses a method of detecting an activity trigger for one or more plugins, comprising:
capturing a current screenshot of a window associated with performing a first activity (pg. 3 2.1 Dataset: the Rico dataset includes a screenshot image);
processing image information associated with the current screenshot into a set of current screenshot embeddings (pg. 4 Layout Embeddings: We then use an image autoencoder to encode each image into a 64-dimensional embedding vector; GUI Embedding Combining Layer: To combine the embedding vectors of multiple GUI components on a screen into a single fixedlength embedding vector, we use an Recurrent Neural Network (RNN);
receiving a plurality of trigger embeddings for a plurality of trigger screenshots defined for a plurality of plugins (pg. 3 After training, the description embedding vector is concatenated on, resulting in the 1536-dimensional GUI screen embedding vector (if included in the training, the description dominates the entire embedding, overshadowing information specific to that screen within the app);
determining semantic similarities between the set of current screenshot embeddings and the plurality of trigger embeddings (pg. 7 3.1 Nearest Neighbors: We conducted a Mechanical Turk study to compare the similarity between the nearest neighbor results generated by the different models. We selected 50 screens from apps and app domains that most users are familiar with; Table 4 shows the mean screen similarity rated by the Mechanical Turk workers for the top-5 nearest neighbor results of the sample source screens generated by the 3 models);
determining a top subset of trigger screenshots of the plurality of trigger screenshots having greater semantic similarity to the current screenshot (pg. 7 3.1 Nearest Neighbors: for a given screen, what are the top-N most similar screens in the dataset?);
detecting an activity trigger for each plugin of a top subset of plugins corresponding to the top subset of trigger screenshots, wherein each plugin is associated with performing a second activity related to the first activity (pg. 1 abstract: Through several sample downstream tasks, we demonstrate Screen2Vec’s key useful properties: representing between-screen similarity through nearest neighbors, composability, and capability to represent user tasks; pg. 10 3.3 Screen Embedding Sequences for Representing Mobile Tasks: GUI screens are not only useful data sources individually on their own, but also as building blocks to represent a user’s task. A task in an app, or across multiple apps, can be represented as a sequence of GUI screens that makes up the user interaction trace for performing this task using app GUIs).; and
Li does not teach where Li-Google teaches causing display of a notification regarding the activity trigger for each plugin of the top subset of plugins, wherein each plugin is selectable from the notification (¶72 For any given screenshot, all the UI elements are scored, and the one with the highest score is selected for possible action; ¶76-79 the image sequence of FIG. 6 shows the four steps of the focus_and_type action; ¶91 when screen representation comes from a DOM tree or accessibility tree, an action always succeeds. Besides, after a reference to an element is obtained, an action can be applied no matter where the element moves)
Therefore, it would have been obvious to one with ordinary skill in the art before the effective filing date of the invention to have implemented the teaching of display of a notification regarding the activity trigger for each plugin of the top subset of plugins, wherein each plugin is selectable from the notification from Li-Google into the system as disclosed by Li. The motivation for doing this is to reliably build agents capable of user interface (UI) navigation.
Regarding claim 11, the combination of Li and Li-Google teaches discloses method of claim 10, wherein the image information of the current screenshot is processed using one or more machine learning (ML) models (Li pg. 2 1 INTRODUCTION: We present a new self-supervised technique (i.e., the type of machine learning approach that trains a model without human-labeled data by withholding some part of the data, and tasking the network with predicting it) Screen2Vec for generating more comprehensive semantic representations of GUI screens and components), and wherein the one or more ML models includes at least one of a screen region detection function or a semantic embedding model (Li pg. 2 1 INTRODUCTION: Screen2Vec: a new self-supervised technique for generating more comprehensive semantic embeddings of GUI screens and components using their textual content, visual design and layout patterns, and app meta-data.).
Allowable Subject Matter
Claims 7-9, 12-14, 17 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Regarding claim 7, and similarly regarding claims 12 and 17, the prior art of record, alone or in combination, fails to teach at least, the set of operations further comprising: receiving a selection of a plugin of the top subset of plugins; receiving at least one semantic cue defined for the selected plugin; receiving a plurality of previous screenshot embeddings based on processing a plurality of previous screenshots; determining semantic similarities between the plurality of previous screenshot embeddings and the at least one semantic cue; and determining a top subset of previous screenshots having greater semantic similarity to the at least one semantic cue.
At best, Li et al NPL: Screen2Vec: Semantic Embedding of GUI Screens and GUI Components) teaches in pg. 7 3.1 Nearest Neighbors: We conducted a Mechanical Turk study to compare the similarity between the nearest neighbor results generated by the different models. We selected 50 screens from apps and app domains that most users are familiar with; Table 4 shows the mean screen similarity rated by the Mechanical Turk workers for the top-5 nearest neighbor results of the sample source screens generated by the 3 models.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to KEVIN KY whose telephone number is (571)272-7648. The examiner can normally be reached Monday-Friday 9-5PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Vincent Rudolph can be reached at 571-272-8243. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/KEVIN KY/Primary Examiner, Art Unit 2671
Prosecution Insights