Prosecution Insights
Last updated: November 20, 2026
Application No. 18/401,186

AI-GENERATED DATASETS FOR AI MODEL TRAINING AND VALIDATION

Final Rejection §103
Filed
Dec 29, 2023
Examiner
JACKSON, JAKIEDA R
Art Unit
2657
Tech Center
2600 — Communications
Assignee
Microsoft Technology Licensing, LLC
OA Round
4 (Final)
74%
Grant Probability
Favorable
5-6
OA Rounds
1m
Est. Remaining
90%
With Interview

Examiner Intelligence

Grants 74% — above average
74%
Career Allowance Rate
686 granted / 925 resolved
+12.2% vs TC avg
Strong +16% interview lift
Without
With
+15.9%
Interview Lift
resolved cases with interview
Typical timeline
3y 0m
Avg Prosecution
40 currently pending
Career history
977
Total Applications
across all art units

Statute-Specific Performance

§101
27.2%
-12.8% vs TC avg
§103
42.2%
+2.2% vs TC avg
§102
20.8%
-19.2% vs TC avg
§112
2.7%
-37.3% vs TC avg
Black line = Tech Center average estimate • Based on career data from 925 resolved cases

Office Action

§103
DETAILED ACTION Notice of Pre-AIA or AIA Status The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA . Response to Amendment In response to the Office Action mailed June 16, 2026, Applicant submitted an amendment filed on September 14, 2026, in which the applicant amended and requested reconsideration. Response to Arguments Applicants argue that the prior art cited fails to teach the claims as amended. Applicants’ arguments are persuasive, but are moot in view of new grounds of rejection. Claim Rejections - 35 USC § 103 The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action: A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made. Claim(s) 1-2, 4-7, 15-16 and 19 is/are rejected under 35 U.S.C. 103 as being unpatenable over Folkens et al. (PGPUB 2016/0283595), hereinafter referenced as Folkens in view of Jelveh (PGPUB 2019/0087691). Regarding claim 1, Folkens discloses a method, comprising: obtaining a screenshot of an active portion of application window rendered by an application (capture screen; fig. 2 with p. 0128-0129); identifying an image within the screenshot (image of interest; fig. 2 element 230 with p. 0128-0129); generating, with a multi-modal model, a caption of the image (tag; fig. 2, element 240 with p. 0128-0129); and generating a label of the screenshot based on the caption of the image and the application window metadata (tag; fig. 2, element 240 with p. 0128-0129, 0097, 0112-0115), but does not specifically teach wherein the label describes a user activity associated with a state of the application depicted in the screenshot and training a machine learning model to predict user activities from screenshots using the screenshot and the label as a labeled training example, wherein the screenshot is an input to the machine learning model with the label as a corresponding target output, such that the trained machine learning model is configured to receive an individual screenshot of an individual application and output a user activity prediction for use in personalizing a user experience provided by the individual application. Jelveh discloses a method wherein the label describes a user activity associated with a state of the application depicted in the screenshot and training a machine learning model to predict user activities from screenshots using the screenshot and the label as a labeled training example, wherein the screenshot is an input to the machine learning model with the label as a corresponding target output, such that the trained machine learning model is configured to receive an individual screenshot of an individual application and output a user activity prediction for use in personalizing a user experience provided by the individual application (p. 0004, 0019-0020, 0031 with fig. 1, elements 106, 108, 110 and 112); model-generation component (108) that generates a model (110) trained using image data captured from a computing screen and implicit and explicit user provided training data. Regarding p. 0078-0079 and Fig. 4, element 402 presents a time series chart of actions (e.g., opened Chrome, checked stock market prices, zoomed in on an image, handled email, engaged in a Skype call, and so on) taken by a user in connection with computing interaction. A gaming component interface 404 represents status of training of the model 110 and associated functionalities achieved as well as path to achieving a comprehensively trained model (e.g., level of evolving). Element 406 provides an interface through which users can explicitly specify intentions or actions which are labeled. Fig. 8 describes receiving screen captures reflecting user interaction, element 802, training a model using the received data, element 804 and receiving user input concerning intent or confirmation of intent for further training, element 806), to improve the accuracy and reliability of activity prediction. Therefore, it would have been obvious to one of ordinary skill of the art, before the effective filing date of the claimed invention, to identify not only visual content depicted in an image but also user activities associated with application states depicted in the screenshots. Regarding claim 2, Folkens discloses a method further comprising: identifying a region of text within the screenshot (fig. 2, element 230); and extracting text from the region of text (OCR; p. 0113). In addition, Jelveh discloses generating the label of the screenshot at least in part based on the extracted text (OCR; p. 0019-0024). Regarding claim 4, Folkens discloses a method further comprising: automatically navigating the application to a website and causing an automated agent to interact with the website in accordance with a usage history of the website (generating results automatically; fig. 3 with p. 0130). In addition, Klein discloses a method wherein the screenshot is captured after the automated agent interacts with the website (p. 0021, 0033-0034, 0054). Regarding claim 5, it is interpreted and rejected for similar reasons as set forth above. In addition, Folkens discloses a method further comprising: generating an activity set by grouping screenshots taken while causing an automated agent to navigate to frequently visited locations of the application, validating a feature of an individual application that uses an individual machine learning model trained with the activity set (measure of confidence assigned to correctly characterized content; p. 0021, 0037, 0067, 0086-0089, 0099). Regarding claim 6, Folkens discloses a method further comprising: generating an activity set by grouping screenshots taken while causing an automated agent to navigate through a stream of locations within the application window (classification; p. 0094-0099, 0107-0108, 0032, 0068-0069); and training a machine learning model with the activity set (retag/retrain for improved processing; p. 0115-0123, 0127, 0166). Regarding claim 7, it is interpreted and rejected for similar reasons as set forth above. In addition, Folkens discloses a method further comprising: providing the individual screenshot to a feature of the individua; application that uses the trained machine learning model; (fig. 3); and validating the feature by comparing the user activity prediction output by the feature with a label of the individual screenshot (comparing data; p. 0212). Regarding claim 15, it is interpreted and rejected for similar reasons as set forth in claim 1. Regarding claim 16, Folkens discloses a medium wherein the instructions further cause the system to obtain metadata comprising a tree of properties of application windows of a desktop that includes the application (list/rank/order/relevance; p. 0097, 0241-0242, 0258-0259). Regarding claim 19, Folkens discloses a medium wherein the label is generated by a large language model based on a prompt that instructs the large language model to label the annotated screenshot with a particular use (large library; p. 0213). Claim(s) 3 and 18 is/are rejected under 35 U.S.C. 103 as being unpatentable over Folkens in view of Jelveh and in further view of Guest (PGPUB 2018/0129657). Regarding claim 3, it is interpreted and rejected for similar reasons as set forth above, however, Folkens in view of Jelveh fails to disclose obtaining application window metadata comprising descriptions of user interface elements depicted in the screenshot of the application, wherein the label of the screenshot is generated in part based on the application window metadata. Guest discloses a medium comprising obtaining application window metadata comprising descriptions of user interface elements depicted in the screenshot of the application, wherein the label of the screenshot is generated in part based on the application window metadata (p. 0056-0076 with fig. 5, element 505 wherein it obtains application specific metadata associated with the application contributing image data to a screenshot, element 510 wherein the system captures a screenshot of the displayed application, element 515 wherein the system stores application specific metadata in association with the captured screenshot and element 520 wherein the metadata contains an image URL, image alternative text and information identifying objects displayed in the screenshot), to facilitate more accurate identification of user activities associated with the state. Therefore, it would have been obvious to one of ordinary skill of the art, before the effective filing date of the claimed invention, to modify the method as described above, to provide additional semantic information regarding images displayed within an application. Regarding claim 18, it is interpreted and rejected for similar reasons as set forth above. In addition, Guest discloses a medium wherein the metadata includes a description of an image displayed in the application, wherein the description of the image is obtained from the application and wherein the label of the screenshot is generated in part based on the description of the image (p. 0056-0076 with fig. 5, element 505 wherein it obtains application specific metadata associated with the application contributing image data to a screenshot, element 510 wherein the system captures a screenshot of the displayed application, element 515 wherein the system stores application specific metadata in association with the captured screenshot and element 520 wherein the metadata contains an image URL, image alternative text and information identifying objects displayed in the screenshot). Claim(s) 9-14 is/are rejected under 35 U.S.C. 103 as being unpatenable over Folkens in view of Jelveh and in further view of Spivack et al. (PGPUB 20100268720), hereinafter referenced as Spivack. Regarding claim 9, it is interpreted and rejected for similar reasons as set forth in claim 1. In addition, Folkens discloses a system comprising: a processing unit (p. 0094-0096); and a memory storing having computer-executable instructions, which, when executed by the processing unit (p. 0094-0096), cause the processing unit to: receive a source text derived from a screenshot of an application window of an application (OCR; p. 0112-0115); receive a label that annotates the screenshot (tag; fig. 2, element 240 with p. 0128-0129); determine a level of correctness of the label in relation to the source text (measure of confidence assigned to correctly characterized content; p. 0021, 0037, 0067, 0086-0089); determine that the label satisfies a quality criteria (measure of confidence assigned to correctly characterized content; p. 0021, 0037, 0067, 0086-0089); determine a label grade of the label based on the level of correctness and the determination that the label satisfies the quality criteria (measure of confidence assigned to correctly characterized content; p. 0021, 0037, 0067, 0086-0089); and determine that the label grade exceeds a defined threshold and validates a feature of an application that uses a machine learning model trained on labeled screenshots of application windows, wherein validating (measure of confidence assigned to correctly characterized content; p. 0021, 0037, 0067, 0086-0089, 0099). It is noted that Jelveh teaches determining recognizing users behaviors (p. 0019, 0031, 0036-0038, 0066, 0092), but does not specifically teach using a label quality classifier, that the label satisfies a quality criteria, wherein the quality criteria indicates whether the label is useful for understanding user behavior with the application that led to the screenshot. Spivack discloses a system comprising determining using a label quality classifier, that the label satisfies a quality criteria (tag), wherein the quality criteria (rating) indicates whether the label is useful for understanding user behavior with the application that led to the screenshot (tracking user behavior; p. 0088, 0096, 0099, 0156-0159, 0170-0178), to assist with customizing, enhance and optimize the experience. Therefore, it would have been obvious to one of ordinary skill of the art, before the effective filing date of the claimed invention, to modify the method as described above, to associate properties and attributes defined in one or more ontologies from behaviors and patterns. Regarding claim 10, Folkens discloses a system wherein the computer-executable instructions further cause the processing unit to: identify a portion of the source text that satisfies a usefulness criteria, wherein the level of correctness of the label is determined in relation to the identified portion of the source text (measure of confidence assigned to correctly characterized content; p. 0021, 0037, 0067, 0086-0089). Regarding claim 11, Folkens discloses a system wherein the label is one of a plurality of labels, and wherein the computer-executable instructions further cause the processing unit to: compute a diversity score of the plurality of labels, wherein the label grade is additionally based on the diversity score (weightings/uniqueness/numeric value; p. 0099-0101, 0122, 0127). Regarding claim 12, Folkens discloses a system wherein the diversity score is computed based on distances between embedding scores computed for each of the plurality of labels, and wherein the label grade (weightings) is proportional to the diversity score (uniqueness; p; 0099-0101, 0122, 0127). Regarding claim 13, Folkens discloses a system, wherein the source text is processed by a short text clustering engine before the portion of the source text is determined to satisfy the usefulness criteria (association/classification; p. 0107-0108, 0032, 0068-0069, 0094-0099). Regarding claim 14, Folkens discloses a system wherein the computer-executable instructions further cause the processing unit to: identify, across a plurality of labels applied to a plurality of screenshots, clusters of related explanations of label incorrectness or low label quality (lower weight; p. 0094-0099, 0107-0108, 0032, 0068-0069). Claim(s) 17 and 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Folkens in view of Jelveh and in further view of Winn et al. (PGPUB 2020/0150832), hereinafter referenced as Winn. Regarding claim 17, Folkens in view of Jelveh disclose a medium as described above, but fails to teach wherein the screenshot is of a desktop, and wherein the screenshot is cropped to the application based on a location and a size of the application window obtained from the metadata. Winn discloses a medium wherein the screenshot is of a desktop, and wherein the screenshot is cropped to the application based on a location and a size of the application window obtained from the metadata (automatically enhance, crop, reduce file size; p. 0079, 0135), to provide a high-quality output. Therefore, it would have been obvious to one of ordinary skill of the art, before the effective filing date of the claimed invention, to modify the method as described above, to refresh the user interface. Regarding claim 20, it is interpreted and rejected for similar reasons as set forth above. In addition, Folkens discloses a medium wherein the screenshot is generated by a computing device configured with a language (p. 0113). In addition, Winn teaches a medium wherein a screen resolution and a user interface theme selected to create screenshots under a diversity of computing environments (image resolution; p. 0054, 0066-0067, 0079, 0135, 0022-0023). Conclusion The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. This information has been detailed in the PTO 892 attached (Notice of References Cited). Roblek et al. discloses segmenting content displayed on a computing device into regions based on pixels of a screenshot image that captures the content. Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a). A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action. Any inquiry concerning this communication or earlier communications from the examiner should be directed to JAKIEDA R JACKSON whose telephone number is (571)272-7619. The examiner can normally be reached Mon - Fri 6:30a-2:30p. Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice. If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Daniel Washburn can be reached at 571.272.5551. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300. Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000. /JAKIEDA R JACKSON/Primary Examiner, Art Unit 2657
Read full office action

Prosecution Timeline

Show 4 earlier events
Feb 23, 2026
Final Rejection mailed — §103
May 26, 2026
Request for Continued Examination
May 28, 2026
Response after Non-Final Action
Jun 16, 2026
Non-Final Rejection mailed — §103
Aug 07, 2026
Interview Requested
Aug 26, 2026
Examiner Interview Summary
Sep 14, 2026
Response Filed
Nov 04, 2026
Final Rejection mailed — §103 (current)

Precedent Cases

Applications granted by this same examiner with similar technology

Patent 12802681
ARTIFICIAL INTELLIGENCE-BASED TEXT-TO-SPEECH SYSTEM AND METHOD
2y 2m to grant Granted Nov 17, 2026
Patent 12802695
Stereo Signal Encoding Method and Apparatus, and Stereo Signal Decoding Method and Apparatus
2y 1m to grant Granted Nov 17, 2026
Patent 12801861
DISTANCE-BASED LOGIT VALUES FOR NATURAL LANGUAGE PROCESSING
1y 11m to grant Granted Nov 17, 2026
Patent 12784764
SYSTEMS AND METHODS RELATING TO SYNCHRONIZATION AND ANALYSIS OF AUDIO COMMUNICATIONS DATA AND TEXT DATA
2y 8m to grant Granted Nov 03, 2026
Patent 12784757
SYSTEM AND METHODS FOR GENERATING REALISTIC WAVEFORMS
1y 11m to grant Granted Nov 03, 2026
Study what changed to get past this examiner. Based on 5 most recent grants.

Strategy Recommendation AI-generated — please review before filing

Get a prosecution strategy drawn from examiner precedents, rejection analysis, and claim mapping.
Typically takes 5-10 seconds — AI-generated, attorney review required before filing

Prosecution Projections

5-6
Expected OA Rounds
74%
Grant Probability
90%
With Interview (+15.9%)
3y 0m (~1m remaining)
Median Time to Grant
High
PTA Risk
Based on 925 resolved cases by this examiner. Grant probability derived from career allowance rate.

Sign in with your work email

Enter your email to receive a magic link. No password needed.

Personal email addresses (Gmail, Yahoo, etc.) are not accepted.

Free tier: 3 strategy analyses per month