DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 2026/02/27, 2026/02/04, 2025/06/12, 2024/04/01, 2023/12/19 & 2023/09/12. The submission is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Double Patenting
The nonstatutory double patenting rejection is based on a judicially created doctrine grounded in public policy (a policy reflected in the statute) so as to prevent the unjustified or improper timewise extension of the "right to exclude" granted by a patent and to prevent possible harassment by multiple assignees. A nonstatutory double patenting rejection is appropriate where the conflicting claims are not identical, but at least one examined application claim is not patentably distinct from the reference claim(s) because the examined application claim is either anticipated by, or would have been obvious over, the reference claim(s). See, e.g., In re Berg, 140 F.3d 1428, 46 USPQ2d 1226 (Fed. Cir. 1998); In re Goodman, 11 F.3d 1046, 29 USPQ2d 2010 (Fed. Cir. 1993); In re Longi, 759 F.2d 887, 225 USPQ 645 (Fed. Cir. 1985); In re Van Ornum, 686 F.2d 937, 214 USPQ 761 (CCPA 1982); In re Vogel, 422 F.2d 438, 164 USPQ 619 (CCPA 1970); In re Thorington, 418 F.2d 528, 163 USPQ 644 (CCPA 1969).
A timely filed terminal disclaimer in compliance with 37 CFR 1.321(c) or 1.321(d) may be used to overcome an actual or provisional rejection based on nonstatutory double patenting provided the reference application or patent either is shown to be commonly owned with the examined application, or claims an invention made as a result of activities undertaken within the scope of a joint research agreement. See MPEP § 717.02 for applications subject to examination under the first inventor to file provisions of the AIA as explained in MPEP § 2159. See MPEP § 2146 et seq. for applications not subject to examination under the first inventor to file provisions of the AIA . A terminal disclaimer must be signed in compliance with 37 CFR 1.321(b).
The filing of a terminal disclaimer by itself is not a complete reply to a nonstatutory double patenting (NSDP) rejection. A complete reply requires that the terminal disclaimer be accompanied by a reply requesting reconsideration of the prior Office action. Even where the NSDP rejection is provisional the reply must be complete. See MPEP § 804, subsection I.B.1. For a reply to a non-final Office action, see 37 CFR 1.111(a). For a reply to final Office action, see 37 CFR 1.113(c). A request for reconsideration while not provided for in 37 CFR 1.113(c) may be filed after final for consideration. See MPEP §§ 706.07(e) and 714.13.
The USPTO Internet website contains terminal disclaimer forms which may be used. Please visit www.uspto.gov/patent/patents-forms. The actual filing date of the application in which the form is filed determines what form (e.g., PTO/SB/25, PTO/SB/26, PTO/AIA /25, or PTO/AIA /26) should be used. A web-based eTerminal Disclaimer may be filled out completely online using web-screens. An eTerminal Disclaimer that meets all requirements is auto-processed and approved immediately upon submission. For more information about eTerminal Disclaimers, refer to www.uspto.gov/patents/apply/applying-online/eterminal-disclaimer.
A. Claim 1 is rejected on the ground of nonstatutory double patenting as being unpatentable over claim 1 of U.S. Patent No. 11,789,753 B2 in view of Fu et al. (Fu), Non-Patent Literature, "Understanding Mobile GUI: from Pixel-Words to Screen-Sentences," arXiv:2105.11941v1, published May 25, 2021. Although the claims at issue are not identical, they are not patentably distinct from each other for the reasons below. In the following comparison, the differences between the claims are shown in bold.
Application 18/550,203
U.S. Patent No. 11,789,753 B2
Claim 1. A computer-implemented method for training and utilization of machine-learned models for user interface prediction, comprising:
Claim 1. A computer-implemented method for training and utilization of machine-learned models for user interface interaction understanding, comprising:
obtaining, by a computing system comprising one or more computing devices, interface data descriptive of a single user interface comprising a plurality of interface elements, wherein the interface data comprises one or more interface images depicting the single user interface;
obtaining, by a computing system comprising one or more computing devices, interface data that comprises a sequence of two or more user interfaces obtained through performance of one or more user interactions which result in generation of the sequence of two or more user interfaces, wherein, for each user interface in the sequence of two or more user interfaces, the interface data comprises one or more interface images depicting the user interface;
determining, by the computing system, a plurality of intermediate embeddings based at least in part on one or more of the one or more interface images or textual content depicted in the one or more interface images;
determining, by the computing system, a plurality of intermediate embeddings based at least in part on the interface data;
processing, by the computing system, the plurality of intermediate embeddings with a machine-learned interface prediction model to obtain one or more user interface embeddings; and
processing, by the computing system, the plurality of intermediate embeddings with a machine-learned interface prediction model to obtain one or more user interface embeddings; and
performing, by the computing system, a pre-training task based at least in part on the one or more user interface embeddings to obtain a pre-training output.
performing, by the computing system, a pre-training task based at least in part on the one or more user interface embeddings to obtain a pre-training output.
Claim 1 of U.S. Patent No. 11,789,753 B2 (col. 21, line 60 – col. 22, line 15) recites every limitation of instant claim 1 except the following. It recites interface data comprising a sequence of two or more user interfaces, rather than interface data descriptive of a single user interface comprising a plurality of interface elements. It also recites determining the intermediate embeddings based on "the interface data," rather than on the interface images or the textual content depicted in them. The differing preambles ("user interface prediction" versus "user interface interaction understanding") state the intended use of the same training method and do not distinguish the claims.
Fu teaches pre-training a transformer model on interface data descriptive of a single user interface comprising a plurality of interface elements. The interface elements are the Pixel-Words of a single screenshot (Fu, page 3, Figure 2 caption: "Given a screenshot, we first generate Pixel-Words"; page 3, right column, paragraph 1: "our Pixel-Words include text components (e.g. text or blocks of texts) and graphic components (e.g. icons and images)"; page 7, right column, paragraph 2: "For each screenshot, 15% of Pixel-Words are randomly masked"). Fu further teaches determining the intermediate embeddings from the textual content depicted in the screenshot (Fu, page 3, right column, § 3.2, paragraph 3 through page 4, left column, paragraph 1: "we use an off-the-shelf OCR tool to extract the location and text content, then feed the text into Sentence-BERT[24] to get the representation of tokens").
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method recited in claim 1 of U.S. Patent No. 11,789,753 B2 to obtain interface data descriptive of a single user interface comprising a plurality of interface elements, and to determine the intermediate embeddings from the textual content depicted in the interface images, as taught by Fu. The motivation is that this allows the interface prediction model to be pre-trained on individual screenshots that are not organized as user-interaction sequences (Fu, page 7, right column, paragraph 2: "Before we train our Screen Transformer in downstream tasks, we first pretrain it on the RICO-PW dataset"). Such pre-training "is effective to learn a better representation by modeling the relation between Pixel-Words" (Fu, page 7, right column, paragraph 3).
B. Claim 1 is rejected on the ground of nonstatutory double patenting as being unpatentable over claim 1 of U.S. Patent No. 12,197,930 B2 in view of Fu. Although the claims at issue are not identical, they are not patentably distinct from each other for the reasons below. In the following comparison, the differences between the claims are shown in bold.
Application 18/550,203
U.S. Patent No. 12,197,930 B2
Claim 1. A computer-implemented method for training and utilization of machine-learned models for user interface prediction, comprising:
Claim 1. A computer-implemented method for training and utilization of machine-learned models for user interface interaction understanding, comprising:
obtaining, by a computing system comprising one or more computing devices, interface data descriptive of a single user interface comprising a plurality of interface elements, wherein the interface data comprises one or more interface images depicting the single user interface;
obtaining, by a computing system comprising one or more computing devices, interface data that comprises a sequence of two or more user interfaces obtained through performance of one or more user interactions which result in generation of the sequence of two or more user interfaces, wherein, for each user interface in the sequence of two or more user interfaces, the interface data comprises one or more interface images depicting the user interface;
determining, by the computing system, a plurality of intermediate embeddings based at least in part on one or more of the one or more interface images or textual content depicted in the one or more interface images;
determining, by the computing system, a plurality of intermediate embeddings based at least in part on the interface data;
processing, by the computing system, the plurality of intermediate embeddings with a machine-learned interface prediction model to obtain one or more user interface embeddings; and
processing, by the computing system, the plurality of intermediate embeddings with a machine-learned interface prediction model to obtain one or more user interface embeddings; and
performing, by the computing system, a pre-training task based at least in part on the one or more user interface embeddings to obtain a pre-training output.
performing, by the computing system, fine-tuning on the machine-learned interface prediction model based on the one or more user interface embeddings.
Claim 1 of U.S. Patent No. 12,197,930 B2 (col. 21, lines 42–64) recites every limitation of instant claim 1 except the following:
the interface data of a single user interface comprising a plurality of interface elements;
intermediate embeddings determined from the interface images or the textual content depicted in them;
a pre-training task to obtain a pre-training output, in place of fine-tuning the model on the user interface embeddings.
The first two differences are taught by Fu, as set forth in the rejection over claim 1 of U.S. Patent No. 11,789,753 B2. As to the third, Fu teaches performing a pre-training task, masked Pixel-Words prediction, on the output embeddings of the Screen Transformer to obtain a pre-training output (the predicted Pixel-Words), before fine-tuning the model on downstream tasks (Fu, page 4, right column, § 3.3, paragraph 2: "The Screen Transformer is pretrained with a self-supervised task: masked Pixel-Words prediction"; page 7, right column, "PW2SS Evaluation," paragraph 1: "We first pretrain the Screen Transformer with the masked Pixel-Words prediction task").
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method recited in claim 1 of U.S. Patent No. 12,197,930 B2 in the following ways, as taught by Fu:
to obtain interface data of a single user interface comprising a plurality of interface elements;
to determine the intermediate embeddings from the textual content depicted in the interface images;
to perform a pre-training task on the user interface embeddings to obtain a pre-training output.
The motivation is that pre-training the model before it is fine-tuned "is effective to learn a better representation by modeling the relation between Pixel-Words" and further improves downstream accuracy (Fu, page 7, right column, paragraph 3).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The text of those sections of Title 35, U.S. Code not included in this action can be found in a prior Office action.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1, 2, 10, 11, 14, 15, 18, and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Fu et al. (Fu), Non-Patent Literature, "Understanding Mobile GUI: from Pixel-Words to Screen-Sentences," arXiv:2105.11941v1, published May 25, 2021.
As to independent Claim 1, Fu teaches a computer-implemented method for training and utilization of machine-learned models for user interface prediction (Fu, page 3, right column, paragraph 2: "the pretrained Screen Transformer can be finetuned to accomplish various downstream tasks like relation prediction, clickability prediction, screen retrieval and app type classification"), comprising:
"obtaining, by a computing system comprising one or more computing devices, interface data descriptive of a single user interface comprising a plurality of interface elements," Fu teaches obtaining a single screenshot of a mobile GUI screen and extracting from it Pixel-Words, which are the text and graphic components of that screen (Fu, page 3, Figure 2 caption: "Given a screenshot, we first generate Pixel-Words with an OCR module to extract Text Pixel-Words"; page 3, right column, paragraph 1: "our Pixel-Words include text components (e.g. text or blocks of texts) and graphic components (e.g. icons and images)"). The screenshot corresponds to the single user interface, and the Pixel-Words correspond to the plurality of interface elements.
"wherein the interface data comprises one or more interface images depicting the single user interface;" Fu teaches that the input is the screenshot image of the screen (Fu, page 3, right column, § 3.2, paragraph 3: "We extract the Pixel-Words from a screenshot"). The screenshot corresponds to the interface image depicting the single user interface.
"determining, by the computing system, a plurality of intermediate embeddings based at least in part on one or more of the one or more interface images or textual content depicted in the one or more interface images;" Fu teaches applying OCR to the screenshot and encoding the depicted text into embeddings, and detecting and classifying graphic components of the screenshot and encoding them into embeddings (Fu, page 3, right column, § 3.2, paragraph 3 through page 4, left column, paragraph 1: "we use an off-the-shelf OCR tool to extract the location and text content, then feed the text into Sentence-BERT[24] to get the representation of tokens" and "use the BERT embeddings of semantic labels as the tokens"). The Text and Graphic Pixel-Word embeddings correspond to the plurality of intermediate embeddings.
"processing, by the computing system, the plurality of intermediate embeddings with a machine-learned interface prediction model to obtain one or more user interface embeddings; and" Fu teaches feeding the Pixel-Word embeddings into a transformer model that outputs contextual embeddings (Fu, page 4, right column, § 3.3, paragraph 1: "we take the Pixel-Words from the previous stage as tokens and feed them into the Screen Transformer to model their relation and form a Screen-Sentence. We design our Screen Transformer as a 6-layer transformer architecture referring to BERT"; page 5, left column, paragraph 2: "the embedding output by the Screen Transformer"). The Screen Transformer corresponds to the machine-learned interface prediction model, and its output embeddings correspond to the one or more user interface embeddings.
"performing, by the computing system, a pre-training task based at least in part on the one or more user interface embeddings to obtain a pre-training output." Fu teaches pre-training the Screen Transformer with a masked Pixel-Words prediction task whose output is the predicted Pixel-Words (Fu, page 4, right column, § 3.3, paragraph 2: "The Screen Transformer is pretrained with a self-supervised task: masked Pixel-Words prediction"; page 4, right column, paragraph 3 through page 5, left column, paragraph 1: "we randomly mask 15% of the total Pixel-Words fed into the Screen Transformer, and take the objective fucntion [sic] as ℓ2 norm between the masked Pixel-Words and predicted Pixel-Words"). The masked Pixel-Words prediction corresponds to the pre-training task, and the predicted Pixel-Words correspond to the pre-training output.
Fu teaches that its method is directed to screenshots of mobile phone user interfaces and to neural-network computing methodologies (Fu, page 1, left column, Abstract: "The ubiquity of mobile phones makes mobile GUI understanding an important task"; page 1, left column, CCS Concepts: "Mobile phones; • Computing methodologies → Image representations; Neural networks"). However, Fu does not explicitly teach "computer-implemented," "by a computing system comprising one or more computing devices," and "by the computing system."
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to perform the steps of Fu by a computing system comprising one or more computing devices. The motivation is that the steps Fu describes cannot be carried out without such a system. These steps are:
capturing a mobile-phone screenshot;
extracting its text with an OCR tool;
computing Sentence-BERT embeddings;
storing the screenshot, embeddings, and model parameters;
training and executing a 6-layer transformer.
Implementing Fu's method on a conventional computing system is therefore the ordinary and predictable way of practicing Fu's own teaching (KSR, 550 U.S. at 416–417).
As to Claim 2, Fu teaches the computer-implemented method of claim 1. Claim 2 is rejected under the same rationale as claim 1 as to the limitations of claim 1. Fu further teaches wherein the method further comprises:
"evaluating, by the computing system, a loss function that evaluates a difference between ground truth data and the pre-training output; and" Fu teaches an objective function that computes the ℓ2 norm between the masked Pixel-Words and the Pixel-Words predicted by the Screen Transformer (Fu, page 4, right column, § 3.3, paragraph 3 through page 5, left column, paragraph 1: "we randomly mask 15% of the total Pixel-Words fed into the Screen Transformer, and take the objective fucntion [sic] as ℓ2 norm between the masked Pixel-Words and predicted Pixel-Words"). The ℓ2 objective function corresponds to the loss function. The original masked Pixel-Words correspond to the ground truth data, and the predicted Pixel-Words correspond to the pre-training output.
"adjusting, by the computing system, one or more parameters of the machine-learned interface prediction model based at least in part on the loss function." Fu teaches pre-training the Screen Transformer on the masked Pixel-Words prediction task over multiple epochs using the AdamW optimizer (Fu, page 7, right column, "PW2SS Evaluation," paragraph 2: "We pretrain the model for 50 epochs, and using 5 epoch for warming up. We use the AdamW optimizer with learning rate 𝑟 = 10−4"). Optimizing the Screen Transformer with the AdamW optimizer on the ℓ2 objective function corresponds to adjusting one or more parameters of the machine-learned interface prediction model based at least in part on the loss function.
The steps are performed by the computing system for the reasons set forth in the rejection of claim 1.
As to Claim 10, Fu teaches the computer-implemented method of claim 1. Claim 10 is rejected under the same rationale as claim 1 as to the limitations of claim 1. Fu further teaches wherein the method further comprises:
"performing, by the computing system, one or more prediction tasks with the machine-learned interface prediction model based at least in part on the one or more user interface embeddings to obtain one or more respective interface prediction outputs." Fu teaches fine-tuning the pre-trained Screen Transformer to perform downstream tasks, each of which produces a prediction from the embeddings output by the Screen Transformer:
Fu, page 3, right column, paragraph 2: "the pretrained Screen Transformer can be finetuned to accomplish various downstream tasks like relation prediction, clickability prediction, screen retrieval and app type classification."
Fu, page 5, left column, paragraph 2: "We use a classifier (a 3-layer MLP) to determine the clickability of the Pixel-Word from the embedding output by the Screen Transformer."
Fu, page 5, right column, paragraph 2: "We also use a maxpooling operation to aggregate all the output embedding from Screen Transformer to get the representation of the given screen. Then we feed the representation into a classifier (a 3-layer MLP) to predict the app type of the screen."
The downstream tasks correspond to the one or more prediction tasks performed with the machine-learned interface prediction model based at least in part on the one or more user interface embeddings. The predicted clickability, relation, retrieved screens, and app type correspond to the one or more respective interface prediction outputs.
The step is performed by the computing system for the reasons set forth in the rejection of claim 1.
As to Claim 11, Fu teaches the computer-implemented method of claim 10. Claim 11 is rejected under the same rationale as claim 10 as to the limitations of claim 10. Fu further teaches:
"wherein the respective one or more interface prediction outputs comprise at least one of:" Claim 11 recites the interface prediction outputs in the alternative. Under the broadest reasonable interpretation, a reference that teaches any one of the recited alternatives teaches the limitation. Fu teaches the following alternative.
"a classification output indicative of an application category for an application associated with the single user interface; or" Fu teaches an app type classification task that classifies each screenshot into one of 26 app categories using the output embeddings of the Screen Transformer:
Fu, page 5, right column, paragraph 2: "The app type classification task is to recognize the app type for each screenshot. We also use a maxpooling operation to aggregate all the output embedding from Screen Transformer to get the representation of the given screen. Then we feed the representation into a classifier (a 3-layer MLP) to predict the app type of the screen."
Fu, page 8, left column, "App Type Classification," paragraph 1: "All the screenshots in P2S-UI dataset are divided into the 26 categories according to the app type (e.g. SHOP, SOCIAL)."
The predicted app type of the screen corresponds to a classification output indicative of an application category for an application associated with the single user interface.
As to independent Claim 15, Fu teaches a computing system, comprising (Fu, page 1, left column, Abstract; page 1, left column, CCS Concepts: "Mobile phones; • Computing methodologies → Image representations; Neural networks"):
"configured to generate learned representations for user interfaces," Fu teaches that the Screen Transformer outputs an embedding for each Pixel-Word and aggregates the output embeddings into a representation of the screen (Fu, page 5, left column, paragraph 2: "the embedding output by the Screen Transformer"; page 5, right column, paragraph 2: "We also use a maxpooling operation to aggregate all the output embedding from Screen Transformer to get the representation of the given screen"). The Screen Transformer corresponds to a machine-learned interface prediction model configured to generate learned representations for user interfaces.
"the machine-learned interface prediction model having been trained by performance of operations, the operations comprising:" Fu teaches that the Screen Transformer is trained by pre-training on screenshots before being fine-tuned (Fu, page 4, right column, § 3.3, paragraph 2: "The Screen Transformer is pretrained with a self-supervised task: masked Pixel-Words prediction"; page 7, right column, "PW2SS Evaluation," paragraph 2: "We pretrain the model for 50 epochs"). The recited operations of obtaining, determining, processing, and performing a pre-training task correspond to the limitations of claim 1. They are taught by Fu for the reasons set forth in the rejection of claim 1.
Fu teaches that its method is directed to screenshots of mobile phone user interfaces, and that the trained Screen Transformer is applied to downstream tasks after pre-training (Fu, page 1, left column, Abstract; page 5, left column, paragraph 2). However, Fu does not explicitly teach "one or more processors;" and "one or more tangible, non-transitory computer readable media storing computer-readable instructions that store a machine-learned interface prediction model."
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to implement Fu's system with one or more processors and to store Fu's trained Screen Transformer on one or more tangible, non-transitory computer readable media. The motivation is that Fu's pre-trained Screen Transformer must be stored after training and executed to perform its downstream tasks on mobile-phone screenshots (Fu, page 5, left column, paragraph 2; page 5, right column, paragraph 2). Storing a trained neural network in non-transitory memory and executing it with a processor is the ordinary and predictable way of practicing Fu's own teaching (KSR, 550 U.S. at 416–417).
As to Claim 14, Fu teaches the computer-implemented method of claim 1. Claim 14 is rejected under the same rationale as claim 1 as to the limitations of claim 1. Fu further teaches wherein:
"determining the plurality of intermediate embeddings comprises processing, by the computing system, the one or more interface images or textual content depicted in the one or more interface images" Fu teaches processing the text depicted in the screenshot, extracted by OCR, to produce the Pixel-Word embeddings (Fu, page 3, right column, § 3.2, paragraph 3 through page 4, left column, paragraph 1: "we use an off-the-shelf OCR tool to extract the location and text content, then feed the text into Sentence-BERT[24] to get the representation of tokens"). The OCR-extracted text corresponds to the textual content depicted in the one or more interface images.
"with an embedding portion of the machine-learned interface prediction model to obtain the plurality of intermediate embeddings; and" Fu teaches that the Pixel-Word representations are processed by a linear layer and combined with 2-D position embeddings to form the input of the Screen Transformer (Fu, page 4, right column, § 3.3, paragraph 1: "The embeddings of Text Pixel-Words and Graphic Pixel-Words are processed by a linear layer to ensure they have the same size. Position embeddings are added to each corresponding Pixel-Words to provide spatial position"). The linear layer and the position embedding layers correspond to the embedding portion of the machine-learned interface prediction model. The resulting input embeddings correspond to the plurality of intermediate embeddings.
"processing the plurality of intermediate embeddings with the machine-learned interface prediction model comprises processing, by the computing system, the plurality of intermediate embeddings with a transformer portion of the machine-learned interface prediction model to obtain the one or more user interface embeddings." Fu teaches feeding the input embeddings into a 6-layer transformer that outputs an embedding for each Pixel-Word (Fu, page 4, right column, § 3.3, paragraph 1: "We design our Screen Transformer as a 6-layer transformer architecture referring to BERT [10]. As Figure 2 shows, the input of the transformer consists of three parts: Pixel-Words embeddings, corresponding 2-D position embeddings and a layout embedding"; page 5, left column, paragraph 2: "the embedding output by the Screen Transformer"). The 6-layer transformer corresponds to the transformer portion of the machine-learned interface prediction model, and its output embeddings correspond to the one or more user interface embeddings.
The steps are performed by the computing system for the reasons set forth in the rejection of claim 1.
As to Claim 18, Fu teaches the computing system of claim 15. Claim 18 is rejected under the same rationale as claim 15 as to the limitations of claim 15. Fu further teaches:
"wherein the operations further comprise performing one or more prediction tasks with the machine-learned interface prediction model based at least in part on the one or more user interface embeddings to obtain one or more respective interface prediction outputs; and" Fu teaches fine-tuning the pre-trained Screen Transformer to perform downstream tasks, each of which produces a prediction from the embeddings output by the Screen Transformer (Fu, page 3, right column, paragraph 2; page 5, left column, paragraph 2; page 5, right column, paragraph 2). The downstream tasks correspond to the one or more prediction tasks performed with the machine-learned interface prediction model based at least in part on the one or more user interface embeddings. The predicted clickability, relation, retrieved screens, and app type correspond to the one or more respective interface prediction outputs.
The remaining limitations of claim 18 correspond to the limitations of claim 11 and are rejected under the same rationale as claim 11.
As to Claim 19, Fu teaches the computing system of claim 15. Claim 19 is rejected under the same rationale as claim 15 as to the limitations of claim 15. The remaining limitations of claim 19 correspond to the limitations of claim 14 and are rejected under the same rationale as claim 14.
Claims 3, 5, and 17 are rejected under 35 U.S.C. 103 as being unpatentable over Fu in view of Wu et al. (Wu), Non-Patent Literature, "LAMPreT: Layout-Aware Multimodal PreTraining for Document Understanding," arXiv:2104.08405v1, published April 16, 2021.
As to Claim 3, Fu teaches the computer-implemented method of claim 1. Claim 3 is rejected under the same rationale as claim 1 as to the limitations of claim 1. Fu further teaches wherein:
"performing the one or more pre-training tasks comprises processing, by the computing system, the one or more user interface embeddings with the machine-learned interface prediction model or a separate pre-training prediction head to obtain the pre-training output," Fu teaches that the masked Pixel-Words prediction is performed by the Screen Transformer, which outputs predicted Pixel-Words from the contextual embeddings it computes (Fu, page 4, right column, § 3.3, paragraph 2: "The Screen Transformer is pretrained with a self-supervised task: masked Pixel-Words prediction"; page 4, right column, § 3.3, paragraph 3 through page 5, left column, paragraph 1: "we design the masked Pixel-Words prediction task to regress the embedding of the masked Pixel-Words"). The Screen Transformer corresponds to the machine-learned interface prediction model, and the predicted Pixel-Words correspond to the pre-training output. The step is performed by the computing system for the reasons set forth in the rejection of claim 1.
Fu teaches modifying the input Pixel-Words of a single screenshot before they are processed by the Screen Transformer, by masking 15% of the Pixel-Words (Fu, page 7, right column, "PW2SS Evaluation," paragraph 2). However, Fu does not teach the following limitations.
In the same field of endeavor, Wu teaches a layout-aware multimodal pre-training framework for web-page documents parsed from HTML into content blocks (Wu, page 1, right column, paragraph 3: "we utilize an in-house document tokenizer to parse HTML formatted pages into several content blocks"). Wu teaches:
"prior to determining the plurality of intermediate embeddings, the method comprises replacing, by the computing system, one or more of the plurality of interface elements" Wu teaches an image-text matching (ITM) pre-training objective in which images of an input document are swapped out before the document is input to, and embedded by, the transformer (Wu, page 5, left column, paragraph 2: "for a given document d containing one or more images, we sample a few candidate images from other documents {d′} within the same mini-batch during training, and swap them with some images in d with certain probabilities"; page 5, Figure 3: "(LObj2) Image Swapping"). Swapping images of document d before the input is embedded corresponds to replacing one or more of the plurality of interface elements prior to determining the plurality of intermediate embeddings.
"with one or more respective second interface elements of a second user interface different than the single user interface; and" Wu teaches that the replacement images are drawn from documents other than the input document (Wu, page 5, left column, paragraph 2: "candidate images from other documents {d′}"; page 5, Figure 3: "Images from other documents"). The images of the other documents d′ correspond to the respective second interface elements of a second user interface different than the single user interface.
"wherein the pre-training output is configured to indicate whether the single user interface is an unmodified user interface." Wu teaches a binary prediction that distinguishes swapped inputs from unswapped inputs (Wu, page 5, left column, paragraph 2: "The model is then required to predict whether the textual contents match the resulting image sequences as binary classification"; page 5, left column, footnote 3: "With probability δ the images in d are swapped, and 1 − δ the images in d are remained the same"). The binary ITM prediction corresponds to a pre-training output configured to indicate whether the single user interface is an unmodified user interface. Wu further teaches that this prediction is output by a dedicated MLP (Wu, page 12, Table 6: "MLP for ITM prediction," output dimension "2, binary"), which corresponds to the separate pre-training prediction head.
Fu and Wu are analogous to the claimed invention as both are from the same field of endeavor of self-supervised pre-training of transformer-based machine-learned models to generate learned representations of multimodal content for downstream prediction tasks. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to combine the Pixel-Word Screen Transformer pre-training of Fu with the image-swapping ITM pre-training objective of Wu. Specifically, one would replace Pixel-Words of a screenshot with Pixel-Words sampled from other screenshots and train a prediction head to indicate whether the screenshot is unmodified. The motivation to combine is as recited by Wu (page 5, left column, paragraph 2: the ITM objective is adopted "[t]o further sharpen the model capability of handling multimodal inputs"). Fu's Pixel-Words are multimodal, combining Text Pixel-Words and Graphic Pixel-Words (Fu, page 3, right column, paragraph 1), so adding Wu's ITM objective to Fu's pre-training predictably improves the Screen Transformer's cross-modal representation of the single screenshot.
As to Claim 5, Fu teaches the computer-implemented method of claim 1. Claim 5 is rejected under the same rationale as claim 1 as to the limitations of claim 1. Fu further teaches wherein:
"performing the one or more pre-training tasks comprises processing, by the computing system, the one or more user interface embeddings with the machine-learned interface prediction model or a separate pre-training prediction head to obtain the pre-training output," Fu teaches that the masked Pixel-Words prediction is performed by the Screen Transformer, which outputs predicted Pixel-Words from the contextual embeddings it computes (Fu, page 4, right column, § 3.3, paragraph 2: "The Screen Transformer is pretrained with a self-supervised task: masked Pixel-Words prediction"). The Screen Transformer corresponds to the machine-learned interface prediction model, and the predicted Pixel-Words correspond to the pre-training output. The step is performed by the computing system for the reasons set forth in the rejection of claim 1.
"wherein the pre-training output comprises a predicted completion for the one or more portions of the one or more interface images that have been masked," Fu teaches predicting the masked Pixel-Words, which include Graphic Pixel-Words, i.e., icons and images depicted in the screenshot (Fu, page 3, right column, paragraph 1: "our Pixel-Words include text components (e.g. text or blocks of texts) and graphic components (e.g. icons and images)"; page 4, right column, § 3.3, paragraph 3 through page 5, left column, paragraph 1: "we design the masked Pixel-Words prediction task to regress the embedding of the masked Pixel-Words" and "we randomly mask 15% of the total Pixel-Words fed into the Screen Transformer"). The predicted masked Graphic Pixel-Words correspond to a predicted completion for the one or more portions of the one or more interface images that have been masked.
Fu teaches masking Pixel-Words at the input to the Screen Transformer and regressing their embeddings (Fu, page 5, left column, paragraph 1). However, Fu does not teach the following limitations.
In the same field of endeavor, Wu teaches a layout-aware multimodal pre-training framework for web-page documents parsed from HTML into content blocks (Wu, page 1, right column, paragraph 3). Wu teaches:
"prior to determining the plurality of intermediate embeddings, the method comprises masking, by the computing system, one or more portions of the one or more interface images; and" Wu teaches an image fitting (IMG-FIT) pre-training objective in which images of the input document are masked by replacing the images themselves with a mask image before they are embedded (Wu, page 5, right column, paragraph 3: "One or more images are masked out by replacing them with a mask-image-token, which is a white image in our implementation"; page 5, Figure 3: "Masked(White)-Image"). Replacing an image of the document with a white mask image before visual embedding corresponds to masking one or more portions of the one or more interface images prior to determining the plurality of intermediate embeddings.
"the prediction completion being selected from a pool of candidate images." Wu teaches that the model selects, for each masked image, the most suitable image from a set of candidate images using a classification head (Wu, page 5, right column, paragraph 3 through page 6, left column, paragraph 1: "This objective requires the model to select the most suitable images from a set of candidate images for the masked-out images" and "the candidate set is constructed by collecting the images from all the documents within a mini-batch during training, and a classification MLP layer is applied to predict the most suitable ones"). The set of candidate images collected from the mini-batch corresponds to the pool of candidate images. The image selected by the classification MLP corresponds to the prediction completion selected from the pool.
Fu and Wu are analogous to the claimed invention as both are from the same field of endeavor of self-supervised pre-training of transformer-based machine-learned models to generate learned representations of multimodal content for downstream prediction tasks. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to combine the masked Pixel-Words prediction of Fu with the image masking and candidate-image selection of Wu. Specifically, one would mask graphic regions of the screenshot before embedding and train a classification head to select the completion of each masked region from a pool of candidate images. The motivation to combine is as recited by Wu (page 2, left column, paragraph 1: Wu proposes "block-level pretraining objectives aiming to exploit the structure of a document," including "image fitting predictions requires the model to select the most suitable image for a missing image block"). Wu shows that removing the image fitting objective degrades downstream performance (Wu, page 8, Table 4: "w/o Image Fitting Objective"). Adding Wu's objective to Fu's pre-training therefore predictably improves the Screen Transformer's representation of the graphic content of the screenshot.
As to Claim 17, Fu teaches the computing system of claim 15. Claim 17 is rejected under the same rationale as claim 15 as to the limitations of claim 15. Fu further teaches wherein:
"performing the one or more pre-training tasks comprises processing the one or more user interface embeddings with the machine-learned interface prediction model or a separate pre-training prediction head to obtain the pre-training output," Fu teaches that the masked Pixel-Words prediction is performed by the Screen Transformer, which outputs predicted Pixel-Words from the contextual embeddings it computes (Fu, page 4, right column, § 3.3, paragraph 2: "The Screen Transformer is pretrained with a self-supervised task: masked Pixel-Words prediction"). The Screen Transformer corresponds to the machine-learned interface prediction model, and the predicted Pixel-Words correspond to the pre-training output.
"wherein the pre-training output comprises a prediction for the one or more portions of the one or more interface images." Fu teaches predicting the masked Pixel-Words, which include Graphic Pixel-Words, i.e., icons and images depicted in the screenshot (Fu, page 3, right column, paragraph 1: "our Pixel-Words include text components (e.g. text or blocks of texts) and graphic components (e.g. icons and images)"; page 4, right column, § 3.3, paragraph 3 through page 5, left column, paragraph 1: "we design the masked Pixel-Words prediction task to regress the embedding of the masked Pixel-Words"). The predicted masked Graphic Pixel-Words correspond to a prediction for the one or more portions of the one or more interface images.
Fu teaches masking Pixel-Words at the input to the Screen Transformer (Fu, page 5, left column, paragraph 1). However, Fu does not teach the following limitation.
In the same field of endeavor, Wu teaches a layout-aware multimodal pre-training framework for web-page documents parsed from HTML into content blocks (Wu, page 1, right column, paragraph 3). Wu teaches:
"prior to determining the plurality of intermediate embeddings, the operations further comprise masking one or more portions of the one or more interface images; and" Wu teaches an image fitting (IMG-FIT) pre-training objective in which images of the input document are masked by replacing the images themselves with a mask image before they are embedded (Wu, page 5, right column, paragraph 3: "One or more images are masked out by replacing them with a mask-image-token, which is a white image in our implementation"; page 5, Figure 3: "Masked(White)-Image"). Replacing an image of the document with a white mask image before visual embedding corresponds to masking one or more portions of the one or more interface images prior to determining the plurality of intermediate embeddings.
Fu and Wu are analogous to the claimed invention as both are from the same field of endeavor of self-supervised pre-training of transformer-based machine-learned models to generate learned representations of multimodal content for downstream prediction tasks. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to combine the masked Pixel-Words prediction of Fu with the image masking of Wu, such that graphic regions of the screenshot are masked before the intermediate embeddings are determined and then predicted by the Screen Transformer. The motivation to combine is as recited by Wu (page 2, left column, paragraph 1: Wu proposes "block-level pretraining objectives aiming to exploit the structure of a document," including image fitting predictions for "a missing image block"). Wu shows that removing its image fitting objective degrades downstream performance (Wu, page 8, Table 4: "w/o Image Fitting Objective"). Masking the image itself before embedding therefore predictably prevents the model from reading the masked content from its embedding and improves the Screen Transformer's representation of the graphic content of the screenshot.
Claims 4 and 16 are rejected under 35 U.S.C. 103 as being unpatentable over Fu in view of Wu, and further in view of Luong et al. (Luong), US 2021/0089724 A1, published March 25, 2021, cited in the IDS filed June 12, 2025
As to Claim 4, Fu in view of Wu teaches the computer-implemented method of claim 3. Claim 4 is rejected under the same rationale as claim 3 as to the limitations of claim 3.
Fu teaches producing a prediction for each individual masked Pixel-Word of the screenshot (Fu, page 4, right column, § 3.3, paragraph 3 through page 5, left column, paragraph 1: "the objective fucntion [sic] as ℓ2 norm between the masked Pixel-Words and predicted Pixel-Words"). However, Fu does not teach the following limitation.
In the same field of endeavor, Luong teaches:
"wherein the pre-training output is further configured to indicate whether each of the plurality of interface elements is an unmodified interface element." Luong teaches a pre-training task in which some input tokens are replaced and the encoder outputs, for every input token, a prediction of whether that token is original or a replacement:
Luong, paragraph [0029]: the method "replaces the masked tokens with samples from a 'generator' ... and then trains the encoder to predict whether each token comes from the original data or is a replacement produced by the generator."
Luong, paragraph [0038]: "the prediction 24 produced by the machine learned language encoder model for each updated input token 22 predicts whether such updated input token is one of the original input tokens 18 or one of the replacement input tokens 23a and 23b."
The per-token predictions 24 correspond to a pre-training output further configured to indicate whether each of the plurality of interface elements is an unmodified interface element. Luong's original input tokens correspond to unmodified interface elements.
Fu and Luong are analogous to the claimed invention as both are from the same field of endeavor of self-supervised pre-training of transformer-based machine-learned models to generate learned representations for downstream prediction tasks. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to combine the Screen Transformer pre-training of Fu with the per-token original-or-replacement prediction of Luong, such that the pre-training output indicates, for each Pixel-Word, whether it is original or replaced. The motivation to combine is as recited by Luong (paragraph [0029]: "this task is more sample efficient than masked language modeling because the loss comes from all input tokens instead of only the subset that was masked out"), which predictably provides a training signal from every Pixel-Word of the screenshot.
As to Claim 16, Fu teaches the computing system of claim 15. Claim 16 is rejected under the same rationale as claim 15 as to the limitations of claim 15. The first two limitations of claim 16 correspond to the limitations of claim 3 and are rejected under the same rationale as claim 3.
Fu teaches producing a prediction for each individual masked Pixel-Word of the screenshot (Fu, page 4, right column, § 3.3, paragraph 3 through page 5, left column, paragraph 1: "the objective fucntion [sic] as ℓ2 norm between the masked Pixel-Words and predicted Pixel-Words"). However, Fu does not teach the following limitation.
In the same field of endeavor, Luong teaches:
"the pre-training output is further configured to indicate whether each of the plurality of interface elements is an unmodified interface element." Luong teaches a pre-training task in which some input tokens are replaced and the encoder outputs, for every input token, a prediction of whether that token is original or a replacement:
Luong, paragraph [0029]: the method "replaces the masked tokens with samples from a 'generator' ... and then trains the encoder to predict whether each token comes from the original data or is a replacement produced by the generator."
Luong, paragraph [0038]: "the prediction 24 produced by the machine learned language encoder model for each updated input token 22 predicts whether such updated input token is one of the original input tokens 18 or one of the replacement input tokens 23a and 23b."
The per-token predictions 24 correspond to a pre-training output further configured to indicate whether each of the plurality of interface elements is an unmodified interface element. Luong's original input tokens correspond to unmodified interface elements.
Fu and Luong are analogous to the claimed invention as both are from the same field of endeavor of self-supervised pre-training of transformer-based machine-learned models to generate learned representations for downstream prediction tasks. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to combine the Screen Transformer pre-training of Fu with the per-token original-or-replacement prediction of Luong, such that the pre-training output indicates, for each Pixel-Word, whether it is original or replaced. The motivation to combine is as recited by Luong (paragraph [0029]: "this task is more sample efficient than masked language modeling because the loss comes from all input tokens instead of only the subset that was masked out"), which predictably provides a training signal from every Pixel-Word of the screenshot.
Claim 6 is rejected under 35 U.S.C. 103 as being unpatentable over Fu in view of Xu et al. (Xu), Non-Patent Literature, "LayoutLM: Pre-training of Text and Layout for Document Image Understanding," arXiv:1912.13318v5, published June 16, 2020.
As to Claim 6, Fu teaches the computer-implemented method of claim 1. Claim 6 is rejected under the same rationale as claim 1 as to the limitations of claim 1. Fu further teaches wherein:
"performing the pre-training task comprises processing, by the computing system, the one or more user interface embeddings with the machine-learned interface prediction model or a separate pre-training prediction head to obtain the pre-training output," Fu teaches that the masked Pixel-Words prediction is performed by the Screen Transformer, which outputs predicted Pixel-Words from the contextual embeddings it computes (Fu, page 4, right column, § 3.3, paragraph 2: "The Screen Transformer is pretrained with a self-supervised task: masked Pixel-Words prediction"). The masked Pixel-Words prediction corresponds to the pre-training task, the Screen Transformer corresponds to the machine-learned interface prediction model, and the predicted Pixel-Words correspond to the pre-training output. The step is performed by the computing system for the reasons set forth in the rejection of claim 1.
Fu teaches masking Pixel-Words, including Text Pixel-Words extracted from the screenshot by OCR, and regressing the Sentence-BERT embeddings of the masked Pixel-Words (Fu, page 3, right column, § 3.2, paragraph 3; page 5, left column, paragraph 1). However, Fu does not teach the following limitations.
In the same field of endeavor, Xu teaches a pre-training method for a transformer model that processes text extracted by OCR from scanned document images (Xu, page 2, left column, paragraph 2: "our focus is the document pre-training based on scanned document images, while digital-born documents are less challenging because they can be considered as a special case where OCR is not required"; page 4, left column, paragraph 2: "with the bounding box of each word from OCR results"). Xu teaches:
"prior to determining the plurality of intermediate embeddings, the method comprises masking, by the computing system, one or more portions of the textual content depicted in the one or more interface images; and" Xu teaches a Masked Visual-Language Model (MVLM) pre-training task in which input text tokens obtained by OCR from the document image are masked before being embedded (Xu, page 4, left column, § 2.4, paragraph 1: "During the pre-training, we randomly mask some of the input tokens but keep the corresponding 2-D position embeddings"). Masking the OCR-derived input tokens before they are embedded corresponds to masking one or more portions of the textual content depicted in the one or more interface images prior to determining the plurality of intermediate embeddings.
"wherein the pre-training output comprises a predicted textual completion for the one or more portions of the textual content depicted in the one or more interface images and that have been masked." Xu teaches that the model predicts the masked text tokens themselves (Xu, page 4, left column, § 2.4, paragraph 1: "the model is trained to predict the masked tokens given the contexts"). The predicted masked tokens correspond to a predicted textual completion for the one or more portions of the textual content depicted in the one or more interface images that have been masked.
Fu and Xu are analogous to the claimed invention as both are from the same field of endeavor of self-supervised pre-training of transformer-based machine-learned models to generate learned representations of multimodal content for downstream prediction tasks. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to combine the masked Pixel-Words prediction of Fu with the MVLM task of Xu. Specifically, one would mask OCR-derived text of the Text Pixel-Words before embedding and train the Screen Transformer to predict the masked text. The motivation to combine is as recited by Xu (page 4, left column, § 2.4, paragraph 1: through MVLM "the LayoutLM model not only understands the language contexts but also utilizes the corresponding 2-D position information, thereby bridging the gap between the visual and language modalities"). Fu expressly follows Xu's layout setting (Fu, page 4, right column, § 3.3, paragraph 1: "Following the same setting as LayoutLM[30]"), so applying Xu's MVLM task to Fu's Text Pixel-Words is a predictable use of a known technique.
Claims 7, 12, 13, and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Fu in view of He et al. (He), Non-Patent Literature, "ActionBert: Leveraging User Actions for Semantic Understanding of User Interfaces," arXiv:2012.12350v1, published December 22, 2020.
As to Claim 7, Fu teaches the computer-implemented method of claim 1. Claim 7 is rejected under the same rationale as claim 1 as to the limitations of claim 1.
Fu teaches masking Pixel-Words of the screenshot during pre-training (Fu, page 5, left column, paragraph 1). Fu also teaches using the view hierarchy (VH) of the screenshot, which describes the UI components, to generate annotations for training (Fu, page 4, left column, paragraph 3). However, Fu does not teach the following limitations.
In the same field of endeavor, He teaches a transformer-based model pre-trained with self-supervised tasks to learn embeddings of user interfaces and their components (He, page 2, left column, paragraph 1: "a transformer-style multi-modal model to capture the context and semantic meaning of UI elements by introducing new self-supervised pre-training tasks based on user actions and UI-specific features"). He teaches:
"wherein, prior to determining the plurality of intermediate embeddings, the method comprises masking, by the computing system, one or more portions of structural data" He teaches a masked VH text prediction pre-training task in which text components of the UI's view hierarchy are masked in the training example before it is input to, and embedded by, the model (He, page 5, left column, paragraph 2: "Pre-training task #3: Masked VH text prediction. ... We randomly mask 15% text components from the UI view hierarchy" and "x_mask is the training example with masked text"). The view hierarchy corresponds to the structural data. Masking its text components in the training example x_mask before the model computes embeddings from x_mask corresponds to masking one or more portions of structural data prior to determining the plurality of intermediate embeddings.
"indicative of one or more positions of one or more respective interface elements of the plurality of interface elements." He teaches that the leaf nodes of the view hierarchy specify the bounding-box position of each UI component (He, page 2, left column, § 2, paragraph 1 and Figure 2: "A View hierarchy is a tree-based representation of a user interface," with each leaf node including a "bounds" attribute, e.g., "bounds": [0, 84, 196, 280]; page 3, right column, paragraph 2: "each vision token is cropped from the UI using the bounding box of a VH leaf node"). The "bounds" of each VH leaf node correspond to the structural data being indicative of one or more positions of one or more respective interface elements of the plurality of interface elements.
Fu and He are analogous to the claimed invention as both are from the same field of endeavor of self-supervised pre-training of transformer-based machine-learned models to generate learned representations of user interfaces for downstream prediction tasks. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to combine the masked Pixel-Words pre-training of Fu with the masked VH prediction of He. Specifically, one would mask portions of the screenshot's view hierarchy, which indicates the positions of the UI components, before the intermediate embeddings are determined. The motivation to combine is as recited by He (page 2, left column, Figure 2 caption: "The view hierarchy provides useful semantic information (marked red) for machines to understand the UI"). Fu already relies on the view hierarchy of the same screenshots to generate its training annotations (Fu, page 4, left column, paragraph 3), so adding He's masked VH task to Fu's pre-training predictably lets the Screen Transformer learn from this semantic information.
As to Claim 12, Fu teaches the computer-implemented method of claim 1. Claim 12 is rejected under the same rationale as claim 1 as to the limitations of claim 1. Fu further teaches:
"one or more textual embeddings," Fu teaches encoding the text of each Text Pixel-Word, extracted from the screenshot by OCR, into an embedding with Sentence-BERT (Fu, page 3, right column, § 3.2, paragraph 3 through page 4, left column, paragraph 1: "we use an off-the-shelf OCR tool to extract the location and text content, then feed the text into Sentence-BERT[24] to get the representation of tokens"). The Sentence-BERT representations of the Text Pixel-Words correspond to the one or more textual embeddings.
"and one or more positional embeddings." Fu teaches adding a 2-D position embedding, derived from the bounding box of each Pixel-Word, to the input of the Screen Transformer (Fu, page 4, right column, § 3.3, paragraph 1: "the input of the transformer consists of three parts: Pixel-Words embeddings, corresponding 2-D position embeddings and a layout embedding" and "Position embeddings are added to each corresponding Pixel-Words to provide spatial position"). The 2-D position embeddings correspond to the one or more positional embeddings.
Fu teaches representing each Graphic Pixel-Word detected in the screenshot by the BERT embedding of its predicted semantic label (Fu, page 4, left column, paragraph 1). However, Fu does not teach the following limitation.
In the same field of endeavor, He teaches:
"wherein the plurality of intermediate embeddings comprises one or more image embeddings," He teaches generating an input vision embedding for each UI component by encoding an image crop of the component from the screenshot with a convolutional vision encoder (He, page 3, right column, paragraph 2: "each vision token is cropped from the UI using the bounding box of a VH leaf node" and "a vision encoder, e.g., ResNet-50, is used to generate the input vision embedding from the cropped images"). He further teaches that these vision embeddings are input to the transformer together with text and positional embeddings (He, page 3, right column, paragraph 5: "The four types of input features are processed through a linear layer followed by a normalization layer ... Then they are summed up and passed as input to ActionBert"). The input vision embeddings correspond to the one or more image embeddings included in the plurality of intermediate embeddings.
Fu and He are analogous to the claimed invention as both are from the same field of endeavor of self-supervised pre-training of transformer-based machine-learned models to generate learned representations of user interfaces for downstream prediction tasks. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to combine the Pixel-Word textual and positional embeddings of Fu with the image-crop vision embeddings of He, so that the intermediate embeddings input to the Screen Transformer include image embeddings of the UI components. The motivation to combine is as recited by He (page 3, right column, paragraph 2: "Similar to the text tokens, the vision tokens are also specific to the nature of UIs"). Encoding the pixels of each UI component directly, rather than only its predicted label as in Fu, predictably preserves visual information about each component that a label cannot convey.
As to Claim 13, Fu in view of He teaches the computer-implemented method of claim 12. Claim 13 is rejected under the same rationale as claim 12 as to the limitations of claim 12. Fu further teaches:
"determining, by the computing system based at least in part on the interface data, the one or more textual embeddings from the textual content depicted in the one or more interface images." Fu teaches extracting the text depicted in the screenshot with OCR and encoding it into embeddings with Sentence-BERT (Fu, page 3, right column, § 3.2, paragraph 3 through page 4, left column, paragraph 1: "We extract the Pixel-Words from a screenshot" and "we use an off-the-shelf OCR tool to extract the location and text content, then feed the text into Sentence-BERT[24] to get the representation of tokens"). The screenshot corresponds to the interface data. The text extracted by OCR corresponds to the textual content depicted in the one or more interface images, and the Sentence-BERT representations correspond to the one or more textual embeddings. The step is performed by the computing system for the reasons set forth in the rejection of claim 1.
Fu teaches detecting graphic UI components in the screenshot and representing each by the BERT embedding of its predicted semantic label (Fu, page 4, left column, paragraph 1). However, Fu does not teach the following limitations.
In the same field of endeavor, He teaches:
"wherein determining the plurality of intermediate embeddings comprises determining, by the computing system, the one or more image embeddings from the one or more interface images," He teaches generating the input vision embeddings by cropping images from the UI screenshot and encoding the crops with a vision encoder (He, page 3, right column, paragraph 2: "If the VH is not available, we fine-tune a Faster-RCNN (Ren et al. 2015) to detect UI components in a screenshot and crop components from the detected bounding boxes. Furthermore, a vision encoder, e.g., ResNet-50, is used to generate the input vision embedding from the cropped images"). The input vision embeddings generated from crops of the screenshot correspond to the one or more image embeddings determined from the one or more interface images.
"wherein the one or more image embeddings are respectively associated with at least one interface element of the plurality of interface elements; and" He teaches that each vision token is the crop of a single UI component (He, page 3, right column, paragraph 2: "each vision token is cropped from the UI using the bounding box of a VH leaf node"). Each vision embedding, generated from the crop of one UI component, corresponds to an image embedding respectively associated with at least one interface element of the plurality of interface elements.
Fu and He are analogous to the claimed invention as both are from the same field of endeavor of self-supervised pre-training of transformer-based machine-learned models to generate learned representations of user interfaces for downstream prediction tasks. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to combine the OCR-based textual embeddings of Fu with the per-component image-crop embeddings of He, so that each UI component in the screenshot is represented by both a textual embedding and an image embedding of its own pixels. The motivation to combine is as recited by He (page 3, right column, paragraph 2: "Similar to the text tokens, the vision tokens are also specific to the nature of UIs"). Associating an image embedding with each UI component predictably gives the Screen Transformer the visual appearance of each component in addition to its text.
As to independent Claim 20, the claim recites one or more tangible, non-transitory computer readable media storing computer-readable instructions that when executed by one or more processors cause the one or more processors to perform operations. Fu does not explicitly teach the computer readable media and processors, which would have been obvious for the reasons set forth in the rejection of claim 15. Except for the limitations addressed below, the recited operations correspond to the limitations of claims 1 and 15, and claim 20 is rejected under the same rationale as claims 1 and 15. Fu further teaches:
"determining a plurality of intermediate embeddings based at least in part on one or more of the structural data, the one or more interface images, or textual content depicted in the one or more interface images;" This limitation recites the basis for the intermediate embeddings in the alternative. Under the broadest reasonable interpretation, a reference that teaches any one of the recited alternatives teaches the limitation. Fu teaches determining the Pixel-Word embeddings from the text depicted in the screenshot, extracted by OCR (Fu, page 3, right column, § 3.2, paragraph 3 through page 4, left column, paragraph 1: "we use an off-the-shelf OCR tool to extract the location and text content, then feed the text into Sentence-BERT[24] to get the representation of tokens"). The OCR-extracted text corresponds to the textual content depicted in the one or more interface images.
Fu teaches representing the position and size of each Pixel-Word in the screenshot as a bounding box (x_min, y_min, x_max, y_max, w, h) (Fu, page 4, right column, § 3.3, paragraph 1). However, Fu does not teach the following limitations.
In the same field of endeavor, He teaches:
"wherein the interface data comprises structural data and an interface image depicting the single user interface," He teaches that the input data of a user interface comprises both the UI screenshot and the view hierarchy of that UI (He, page 2, left column, § 2, paragraph 1: "A View hierarchy is a tree-based representation of a user interface. It has various attributes related to the appearance and functionality of UI components"; page 3, right column, paragraph 2: "each vision token is cropped from the UI using the bounding box of a VH leaf node"). The view hierarchy corresponds to the structural data, and the UI screenshot from which the vision tokens are cropped corresponds to the interface image depicting the single user interface.
"wherein the structural data is indicative of one or more positions of one or more respective interface elements of the plurality of interface elements;" He teaches that each leaf node of the view hierarchy specifies the bounding-box position of its UI component (He, page 2, left column, Figure 2: leaf nodes each including a "bounds" attribute, e.g., "bounds": [0, 84, 196, 280]; page 3, right column, paragraph 2: "the bounding box of a VH leaf node"). The "bounds" attribute of each view hierarchy leaf node corresponds to the structural data being indicative of one or more positions of one or more respective interface elements of the plurality of interface elements.
Fu and He are analogous to the claimed invention as both are from the same field of endeavor of self-supervised pre-training of transformer-based machine-learned models to generate learned representations of user interfaces for downstream prediction tasks. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to combine the Screen Transformer of Fu with the view hierarchy input of He, such that the interface data obtained for the screenshot comprises both the screenshot and view hierarchy structural data indicating the positions of the user interface components. The motivation to combine is as recited by He (page 2, left column, Figure 2 caption: "The view hierarchy provides useful semantic information (marked red) for machines to understand the UI"). Fu already uses the view hierarchy of the same screenshots to obtain component positions for training (Fu, page 4, left column, paragraph 3: "we carefully design a pipeline to extract annotation from VH"), so supplying the view hierarchy together with the screenshot as input interface data predictably provides the component positions without requiring them to be inferred from pixels.
Claim 8 is rejected under 35 U.S.C. 103 as being unpatentable over Fu in view of He, and further in view of Xu.
As to Claim 8, Fu in view of He teaches the computer-implemented method of claim 7. Claim 8 is rejected under the same rationale as claim 7 as to the limitations of claim 7. Fu further teaches wherein:
"performing the one or more pre-training tasks comprises processing, by the computing system, the one or more user interface embeddings with the machine-learned interface prediction model or a separate pre-training prediction head to obtain the pre-training output," Fu teaches that the masked Pixel-Words prediction is performed by the Screen Transformer, which outputs predicted Pixel-Words from the contextual embeddings it computes (Fu, page 4, right column, § 3.3, paragraph 2: "The Screen Transformer is pretrained with a self-supervised task: masked Pixel-Words prediction"). The Screen Transformer corresponds to the machine-learned interface prediction model, and the predicted Pixel-Words correspond to the pre-training output. The step is performed by the computing system for the reasons set forth in the rejection of claim 1.
Fu teaches a Graphic Classifier that recognizes the semantic category of each graphic UI component (Fu, page 4, left column, paragraph 5). However, Fu does not teach the following limitation.
In the same field of endeavor, He teaches:
"the one or more portions of the structural data are further descriptive of one or more class labels for one or more respective interface elements of the plurality of interface elements; and" He teaches that the masked view hierarchy text components include the component class field of each UI component (He, page 2, left column, § 2, paragraph 1: "we leverage the content description, resource id, component class, component text fields in the leaf nodes of view hierarchy" and "The resource id and component class indicate the type of the component, e.g., button, checkbox or image"; page 5, left column, paragraph 2: "each text token in ActionBert is a concatenation of multiple fields (content description, source id, component type and text) from the view hierarchy" and "We randomly mask 15% text components from the UI view hierarchy"; page 2, left column, Figure 2: "class":"android.widget.ImageButton"). The component class and component type fields contained in the masked view hierarchy text components correspond to the portions of the structural data being further descriptive of one or more class labels for one or more respective interface elements.
Fu and He are analogous to the claimed invention as both are from the same field of endeavor of self-supervised pre-training of transformer-based machine-learned models to generate learned representations of user interfaces for downstream prediction tasks. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to combine the Screen Transformer pre-training of Fu with the masking of view hierarchy components, including their component class fields, taught by He. The motivation to combine is as recited by He (page 2, left column, § 2, paragraph 1: the component class indicates "the type of the component, e.g., button, checkbox or image"), which gives the Screen Transformer a self-supervised signal for the type of each UI component.
He predicts the embedding of the masked view hierarchy text by regression (He, page 5, left column, paragraph 2). The combination of Fu and He, however, does not teach the following limitation.
In the same field of endeavor, Xu teaches:
"wherein the pre-training output comprises one or more predicted class labels for the one or more respective interface elements." Xu teaches a pre-training task in which masked input tokens are directly predicted as discrete tokens (Xu, page 4, left column, § 2.4, paragraph 1: "we randomly mask some of the input tokens but keep the corresponding 2-D position embeddings, and then the model is trained to predict the masked tokens given the contexts"). Applied to the masked component class field of He, the directly predicted masked tokens correspond to a pre-training output comprising one or more predicted class labels for the one or more respective interface elements.
Fu, He, and Xu are analogous to the claimed invention as all are from the same field of endeavor of self-supervised pre-training of transformer-based machine-learned models to generate learned representations for downstream prediction tasks. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to combine the Screen Transformer pre-training of Fu, modified to mask view hierarchy component classes as taught by He, with the direct masked-token prediction of Xu. The masked component class of each UI component would then be predicted as a class label. The motivation to combine is that He itself identifies direct token prediction as the known alternative to its embedding regression (He, page 5, left column, paragraph 2: "compared to BERT where each word token is directly predicted, ActionBert predicts the high-dimensional text embedding"). Xu teaches that direct prediction of masked tokens enables the model to understand "the language contexts" and utilize "the corresponding 2-D position information" (Xu, page 4, left column, § 2.4, paragraph 1). Substituting one known prediction target for another yields the predictable result of predicting the class label of each masked component (KSR, 550 U.S. at 416).
Claim 9 is rejected under 35 U.S.C. 103 as being unpatentable over Fu in view of He, and further in view of Li et al. (Li), Non-Patent Literature, "Widget Captioning: Generating Natural Language Description for Mobile User Interface Elements," arXiv:2010.04295v1, published October 8, 2020, cited in the IDS filed April 1, 2024.
As to Claim 9, Fu in view of He teaches the computer-implemented method of claim 7. Claim 9 is rejected under the same rationale as claim 7 as to the limitations of claim 7. Fu further teaches wherein:
"performing the one or more pre-training tasks comprises processing, by the computing system, the one or more user interface embeddings with the machine-learned interface prediction model or a separate pre-training prediction head to obtain the pre-training output," Fu teaches that the masked Pixel-Words prediction is performed by the Screen Transformer, which outputs predicted Pixel-Words from the contextual embeddings it computes (Fu, page 4, right column, § 3.3, paragraph 2: "The Screen Transformer is pretrained with a self-supervised task: masked Pixel-Words prediction"). The Screen Transformer corresponds to the machine-learned interface prediction model, and the predicted Pixel-Words correspond to the pre-training output. The step is performed by the computing system for the reasons set forth in the rejection of claim 1.
Fu teaches using the view hierarchy of the screenshot to generate annotations for training (Fu, page 4, left column, paragraph 3). However, Fu does not teach the following limitation.
In the same field of endeavor, He teaches:
"the one or more portions of the structural data further comprise one or more content descriptors for one or more respective interface elements of the plurality of interface elements; and" He teaches that the masked view hierarchy text components include the content description field of each UI component (He, page 2, left column, § 2, paragraph 1: "we leverage the content description, resource id, component class, component text fields in the leaf nodes of view hierarchy" and "The content description is a brief description of the functionality of this UI component provided by the developer"; page 5, left column, paragraph 2: "each text token in ActionBert is a concatenation of multiple fields (content description, source id, component type and text) from the view hierarchy" and "We randomly mask 15% text components from the UI view hierarchy"). The content description field contained in the masked view hierarchy text components corresponds to the portions of the structural data further comprising one or more content descriptors for one or more respective interface elements.
Fu and He are analogous to the claimed invention as both are from the same field of endeavor of self-supervised pre-training of transformer-based machine-learned models to generate learned representations of user interfaces for downstream prediction tasks. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to combine the Screen Transformer pre-training of Fu with the masking of view hierarchy components, including their content description fields, taught by He. The motivation to combine is as recited by He (page 2, left column, § 2, paragraph 2: content description "is invisible to the user but can be used by applications like Screen Readers to understand the UI"), which gives the Screen Transformer a self-supervised signal for the functionality of each UI component.
He predicts the embedding of the masked view hierarchy text by regression (He, page 5, left column, paragraph 2). The combination of Fu and He, however, does not teach the following limitation.
In the same field of endeavor, Li teaches a transformer-based model that generates content descriptions for UI elements from the screenshot and the view hierarchy of a mobile UI (Li, page 1, left column, Abstract: "We propose widget captioning, a novel task for automatically generating language descriptions for UI elements from multimodal input including both the image and the structural representations of user interfaces"). Li teaches:
"wherein the pre-training output comprises one or more predicted content descriptors for the one or more respective interface elements." Li teaches that the widget captions are the contentDescription properties of the view hierarchy, and that the model decodes a caption for each element (Li, page 3, left column, § 3.1.1, paragraph 2: "we consider an element missing captions when both its contentDescription and text properties in the view hierarchy are missing"; page 5, right column, § 4, paragraph 2: "the encoder takes both the structural information and the pixel appearance of the UI element, and the decoder outputs the caption based on the encodings"; page 6, left column, § 4.1, paragraph 1: "We use a Transformer model (Vaswani et al., 2017) to encode the set of elements on a screen"). The captions output by the decoder for the UI elements correspond to one or more predicted content descriptors for the one or more respective interface elements.
Fu, He, and Li are analogous to the claimed invention as all are from the same field of endeavor of machine-learned models that generate learned representations of user interfaces for downstream prediction tasks. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to combine the Screen Transformer pre-training of Fu, modified to mask view hierarchy content descriptions as taught by He, with the caption decoding of Li. The masked content description of each UI component would then be predicted as a content descriptor. The motivation to combine is as recited by Li (page 1, right column, Figure 1 caption: "These captions are crucial for accessibility and language-based interaction in general"; page 1, right column, paragraph 1: "a significant portion of mobile apps today lack widget captions in their user interfaces"). Predicting the masked content description directly, rather than its embedding, predictably trains the Screen Transformer to produce the content descriptors that accessibility services require.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to HUNG VAN LE whose telephone number is (571)270-0164. The examiner can normally be reached 8 a.m. - 5 p.m..
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Cesar Paula can be reached at (571) 272-4128. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/HUNG VAN LE/Examiner, Art Unit 2145
/CESAR B PAULA/Supervisory Patent Examiner, Art Unit 2145
Prosecution Insights