Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
DETAILED ACTION
Response to Arguments
Applicant's arguments filed 7/2/2026 have been fully considered but they are not persuasive.
Applicant argues that the amendments to independent claims 1, 15, and 16 (and the corresponding amendment to claim 5) embed the identification of the predetermined point “in a particular practical application” and therefore do not attempt to monopolize all techniques for such identification. Applicant further asserts that the claims are now more clearly directed to statutory subject matter.
The Examiner respectfully disagrees. The amended limitations continue to recite an abstract idea belonging to the mental-processes grouping without integrating that idea into a practical application or adding significantly more.
The core of the amended independent claims remains the mental process of observing a visual display (an image generated by a program), locating a UI element and a reference point on that element, evaluating the information conveyed relative to the reference point, and reporting the result. The newly added language “wherein identifying the predetermined point includes performing image analysis within an expanded region to locate an image pattern corresponding to a first part of the respective user interface element and deriving the predetermined point from the located image pattern” does not alter this idea. These steps describe the same cognitive activities a human software tester or game analyst would perform when examining a screenshot: expand the visual search area around an expected location, scan for a distinctive visual pattern or landmark that forms part of the UI element, and mark a reference point (edge, center, origin, etc.) based on the location of that pattern. Such observation, evaluation, and judgment of visual data remain mental processes even when described with such image-analysis terminology. See MPEP § 2106.04(a)(2)(III) and the 2025 Memorandum reminding examiners that image-analysis steps involving observation, evaluation, and judgment of visual data qualify as mental processes when they mirror human cognitive activities.
Under Step 2A, Prong Two, the additional elements (generic “image,” “image analysis,” “image processor,” computer-readable medium, and the “expanded region” search) are recited at a high level of generality and amount to no more than instructions to implement the abstract idea on a generic computer or to perform insignificant extra-solution activity. There is still no recitation of a specific improvement to computer functionality or to image-analysis technology itself, no particular machine, no transformation, and no meaningful limitation beyond generally linking the abstract idea to the field of program-state analysis from screenshots. The “expanded region / locate pattern / derive point” sequence is itself part of the abstract idea. It does not supply the required practical application.
Under Step 2B, the ordered combination remains nothing more than the abstract idea performed with well-understood, routine, conventional computer-vision and data-output steps. Consequently, the 101 rejection is maintained and has been updated to address the amended and newly added claims.
Furthermore, applicant argues that White fails to anticipate the amended independent claims because White at most describes “padding around the expected location of the content to be detected” and does not teach the specific two-step sequence of (1) performing image analysis within an expanded region to locate an image pattern corresponding to a first part of the respective user interface element and (2) deriving the predetermined point from that located image pattern. Applicant further contends that selecting a region for analysis is different from locating a pattern and deriving a point from it, and that White’s motion/optical-flow discussion concerns gameplay events rather than derivation of a predetermined point on a UI element.
The Examiner respectfully disagrees. White expressly teaches:
Analysis performed on video frames generated by a game program (the claimed “given image generated by a program”).
Regions of the display that are defined (via configuration file) and that deliberately include padding “at least some amount of padding around the expected location of the content to be detected” expressly to accommodate movement or variation. This padding creates an expanded region that encompasses variable positions of the UI content.
Detectors that perform image analysis (pattern matching, icon recognition, etc.) inside those padded/expanded regions specifically to locate the visual content/pattern corresponding to HUD/UI elements (the “image pattern corresponding to a first part of the respective user interface element”).
Once the pattern/content is located within the expanded region, its position becomes the operative reference location used for subsequent information extraction (bar fill relative to an origin, icon identity within a defined region relative to the detected location, OCR within a region relative to the detected location, etc.). In other words, the predetermined point is derived from the located image pattern.
The claimed two-step sequence is therefore present: image analysis is performed inside an expanded (padded) region; that analysis locates the relevant image pattern corresponding to a first part of the UI element; and the location of the located pattern is used as (i.e., the predetermined point is derived from) the reference for identifying the conveyed information. White’s disclosure is not limited to merely “selecting a region”; the detectors actively locate the pattern inside the padded region, and the system then operates with reference to that located content. The fact that White also discusses motion detection for gameplay events does not negate the separate, clear teaching of UI/HUD pattern detection inside padded regions for the purpose of extracting state information.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-19 are rejected under 35 U.S.C. 101 because the claimed invention is directed to a judicial exception (i.e., a law of nature, a natural phenomenon, or an abstract idea) without significantly more. The claims are directed to the abstract idea of mental processes and/ or certain methods of organizing human activity. The claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception as discussed below.
Step 1 of the 2019 Revised Patent Subject Matter
More specifically, regarding Step 1, of the 2019 Revised Patent Subject Matter Eligibility Guidance, the claims are directed to a machine, process, and/or an article of manufacturer, which are statutory categories of invention.
Step 2a – Prong 1 of the 2019 Revised Patent Subject Matter Eligibility Guidance
Next, the claims are analyzed to determine whether it is directed to a judicial exception.
Claims 1-19 recite an abstract idea that falls within the “mental processes” grouping of abstract ideas. MPEP § 2106.04(a)(2)(III). The claims recite, in relevant part (quoting representative claim 1 as amended):
“identifying at least a first location within the image occupied by a respective user interface element”;
“identifying in the image a predetermined point on the respective user interface element, wherein identifying the predetermined point includes performing image analysis within an expanded region to locate an image pattern corresponding to a first part of the respective user interface element and deriving the predetermined point from the located image pattern”;
“identifying the information conveyed by the respective user interface element, with reference to the predetermined point”; and
“outputting the information for subsequent analysis.”
These limitations cover concepts that can be practically performed in the human mind (or with pen and paper). A human (e.g., a software tester or game analyst) can visually scan a screenshot or displayed image of a program (including an expanded area around an expected UI location), locate a visual pattern or landmark corresponding to a first part of a UI element (e.g., the left edge or distinctive portion of a health bar, icon, or text field), derive a reference point from that located pattern (e.g., the left edge or center of the matched pattern), judge the information conveyed relative to that point (e.g., bar fill level, icon identity, numerical value), and report the extracted information. The “expanded region,” “image pattern,” and “deriving” steps remain observational and evaluative mental acts of the same character as those previously rejected; they merely describe how a human would search a slightly broader visual field and mark a landmark. Dependent claims 2-14 and 17-19 further limit the mental process with conventional details (configuration files, static/moving points, bar-type/image-type/character elements, OCR, frequency rules, probability prioritization, motion-vector prediction, event identification) that do not change the abstract character. Claims 15 and 16 recite the same abstract idea implemented via a medium or processor.
Step 2a – Prong 2 of the 2019 Revised Patent Subject Matter Eligibility Guidance
The second prong of step 2a is the consideration if the claim limitations are directed to a practical application.
Limitations that are indicative of integration into a practical application:
-Improvements to the functioning of a computer, or to any other technology or technical field - see MPEP 2106.05(a)
-Applying or using a judicial exception to effect a particular treatment or prophylaxis for a disease or medical condition - see Vanda Memo
-Applying the judicial exception with, or by use of, a particular machine - see MPEP 2106.05(b)
-Effecting a transformation or reduction of a particular article to a different state or thing – see MPEP 2106.05(c)
-Applying or using the judicial exception in some other meaningful way beyond generally linking the use of the judicial exception to a particular technological environment, such that the claim as a whole is more than a drafting effort designed to monopolize the exception - see MPEP 2106.05(e) and Vanda Memo
Limitations that are not indicative of integration into a practical application:
-Adding the words "apply it" (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea- see MPEP 2106.05(f)
-Adding insignificant extra-solution activity to the judicial exception - see MPEP 2106.05(g)
-Generally linking the use of the judicial exception to a particular technological environment or field of use - see MPEP 2106.05(h)
The claims do not integrate the abstract idea into a practical application. The additional elements are recited at a high level of generality and amount to mere instructions to apply the abstract idea on generic computing components.
The additional elements include: a “given image generated by a program,” “performing image analysis within an expanded region,” “image processor,” “non-transitory computer readable storage medium,” and “computer executable instructions.” These are generic computer functions and conventional data-gathering steps. There is no recitation of a specific improvement to the functioning of a computer or to any other technology or technical field (contrast Enfish, LLC v. Microsoft Corp., 822 F.3d 1327 (Fed. Cir. 2016)); no particular machine; no transformation of an article; and no meaningful application beyond generally linking the abstract idea to the technological environment of image analysis or software testing. The “expanded region,” “locating an image pattern,” and “deriving the predetermined point” steps are themselves part of the abstract idea (or insignificant extra-solution activity / field-of-use limitations). The “outputting” step is insignificant extra-solution activity. Even when considered as an ordered combination, the claims simply automate the mental process of analyzing a program-generated image on a generic computer and do not improve image-analysis technology itself (no novel detector architecture, no training methodology, no hardware integration).
Step 2b of the 2019 Revised Patent Subject Matter Eligibility Guidance
Next, the claims as a whole are analyzed to determine whether any element, or combination of elements, is sufficient to ensure that the claim amounts to significantly more than the exception.
The claims do not include additional elements, individually or as an ordered combination, that amount to significantly more than the judicial exception. All recited steps are well-understood, routine, and conventional activities in the fields of image analysis, computer vision, and software/game testing: receiving or capturing program-generated images/screenshots; searching expanded/padded regions for visual patterns corresponding to UI elements; deriving reference locations from matched patterns; extracting information (bar fill, icon identity, OCR text); and outputting results. See Electric Power Group, LLC v. Alstom S.A., 830 F.3d 1350 (Fed. Cir. 2016); Content Extraction & Transmission LLC v. Wells Fargo Bank, N.A., 776 F.3d 1343 (Fed. Cir. 2014). The ordered combination adds nothing inventive; it is simply the abstract idea performed on a computer.
Consequently, claims 1-19 are not patent-eligible under 35 U.S.C. § 101.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale or otherwise available to the public before the effective filing date of the claimed invention.
Claim(s) 1-16 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by White (US 2022/0040570 A1).
1. White discloses an image-based program state analysis method, comprising the steps of:
for a given image generated by a program (video frames generated by the game program), [0031];
identifying at least a first location within the image occupied by a respective user interface element (regions corresponding to HUD/UI content), [0024]-[0026], [0031], [0035]-[0037];
identifying in the image a predetermined point on the respective user interface element, wherein identifying the predetermined point includes performing image analysis within an expanded region to locate an image pattern corresponding to a first part of the respective user interface element and deriving the predetermined point from the located image pattern (sections or regions of the display considered for detection include at least some amount of padding around the expected location of the content to be detected (e.g. an expanded region); detectors perform image analysis, including pattern/icon recognition, within those regions to locate the relevant image content/pattern corresponding to a UI element/first part; the location of the detected content serves as the operative point for subsequent analysis, i.e., the predetermined point is derived from the located pattern), [0024]-[0026], [0028]-[0029], [0031], [0035]-[0037], [0046], Figs. 1, 6-7;
identifying the information conveyed by the respective user interface element, with reference to the predetermined point; and outputting the information for subsequent analysis, [0024]-[0026], [0028]-[0029], [0035]-[0037], [0046], (Figs. 6-7).
2. White discloses the method of claim 1 in which the first location or a region comprising the first location is provided by a configuration file for the program [0031], [0033], [0035].
3. White discloses the method of claim 1, in which the predetermined point on the respective user interface element is static within images generated by the program (HUD regions are typically static in layout for a short period).
4. White discloses the method of claim 1, in which the predetermined point on the respective user interface element moves within images generated by the program, [0024]-[0026], [0031], [0036].
5. White discloses the method of claim 4, wherein the expanded region encompasses variable positions of the respective user interface element within images generated by the program (padding allows for movement/variations wherein content may also move over time), [0024]-[0026], [0031], [0036].
6. White discloses the method of claim 5, in which the given image comprises a plurality of further user interface elements that move in the same way as a first respective user interface element, and the step of identifying a predetermined point on respective further user interface elements comprises: identifying in the image a predetermined point on one or more of the further respective user interface elements as respective predetermined relative offsets to the identified location of the predetermined point on the first respective user interface element, [0024]-[0026], [0031], [0036].
7. White discloses the method of claim 1 in which a respective user interface element is a bar-type element, and the step of identifying the information comprises: performing image analysis to detect how much of a maximum bar originating at the predetermined point is currently depicted within the image; and estimates a value for the property based on by how much is currently depicted, (Health/shield/ammo meters), [0024], (Fig. 1).
8. White discloses the method of claim 1 in which a respective user interface element is an image-type element, and the step of identifying the information comprises: performing image analysis to identify an image element within a defined region relative to the predetermined point, by: comparing a representation of the image type element within the defined region to a corresponding representation of a plurality of reference image elements; and obtaining an item definition associated with the best matching reference item (Skull icons, ammo icons, weapon icons: compare representation in the defined region (relative to predetermined point) to reference images; “obtaining an item definition associated with the best matching reference item” (e.g., kill count from skull match)), [0032], [0036], (Fig. 1).
9. White discloses the method of claim 1 in which a respective user interface element is one or more numerical character elements, and the step of identifying the information comprises: performing image analysis to identify one or more numerical character elements within a defined region relative to the predetermined point using optical character recognition, [0034], [0038].
10. White discloses the method of claim 1 in which a respective user interface element is one or more alphanumeric character elements, and the step of identifying the information comprises: performing image analysis to identify one or more alphanumeric character elements within a defined region relative to the predetermined point using optical character recognition, [0034], [0038].
11. White discloses the method of claim 10, in which the step of performing image analysis comprises adopting a search strategy for the alphanumeric character elements responsive to the expected text justification within the defined region, [0034], [0038].
12. White discloses the method of claim 1 in which the step of identifying the information does not comprise recognizing text (non-text detectors operate independently of OCR), [0024], [0036]-[0037].
13. White discloses the method of claim 1, in which the step of identifying the information conveyed by a respective user element is performed at a frequency responsive to the type of information (sampling every 100 ms or every nth frame, with per-detector stability thresholds), [0031], [0040].
14. White discloses the method of claim 1, in which the subsequent analysis, based upon a given image or a sequence of such images, comprises one or more selected from the list consisting of: i. identifying an event based upon a change in the identified information; ii. identifying an event based upon a threshold value associated with the identified information; iii. identifying a trend based upon the identified information; and iv. identifying a status based upon the identified information, [0024]-[0025], [0036].
15. White discloses a non-transitory, computer readable storage medium containing a computer program comprising computer executable instructions that when executed by a computer system, cause the computer system to perform an image-based program state analysis method, comprising the steps of: for a given image generated by a program, identifying at least a first location within the image occupied by a respective user interface element; identifying in the image a predetermined point on the respective user interface elements wherein identifying the predetermined point includes performing image analysis within an expanded region to locate an image pattern corresponding to a first part of the respective user interface element and deriving the predetermined point from the located image pattern; identifying the information conveyed by the respective user interface element, with reference to the predetermined point; and outputting the information for subsequent analysis as similarly discussed above.
16. White discloses an image-based program state analysis system, comprising: an image processor configured, for a given image generated by a program, to: identify at least a first location within the image occupied by a respective user interface element; identify in the image a predetermined point on the respective user interface element, wherein identifying the predetermined point includes performing image analysis within an expanded region to locate an image pattern corresponding to a first part of the respective user interface element and deriving the predetermined point from the located image pattern; identify the information conveyed by the respective user interface element, with reference to the predetermined point; and output the information for subsequent analysis as similarly discussed above.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 17 is/are rejected under 35 U.S.C. 103 as being unpatentable over White (US 2022/0040570 A1) as applied above and further in view of Sudakov (US 2021/0295084 A1).
17. White discloses the image-based program state analysis method of claim 1, and further discloses that the expanded region is defined by a configuration file for the program, [0031], [0033], [0035]. White does not expressly disclose that the configuration file includes an indication of relative probability of where a respective user interface element is found, and that the image analysis prioritizes locations within the expanded region according to the relative probability. Sudakov teaches generating a UI mask that includes probability information indicating the likelihood that areas of an image correspond to a user interface element, and identifying/prioritizing UI areas for further analysis according to those probability values, [0052]-[0060]. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify White with Sudakov and would have been motivated to do so to improve detection efficiency and reduce computational cost when searching for dynamic or variably positioned UI elements, by focusing analysis first on the most probable locations within the expanded region.
Claim(s) 18 is/are rejected under 35 U.S.C. 103 as being unpatentable over White (US 2022/0040570 A1) as applied above and further in view of LIN (US 2022/0398747 A1).
18. White discloses the method of claim 1 and further discloses motion detection and optical flow for analyzing gameplay data over time, [0029], [0035]. White does not expressly disclose after the image pattern is located in a preceding image, identifying the predetermined point in a subsequent image includes predicting a position of the image pattern from the preceding image and a motion vector and centering the image analysis on the predicted position. LIN discloses determining a position of a feature in a first/preceding frame, using an optical-flow/motion-vector prediction to generate a position estimate of a search area in a subsequent frame, and centering (or positioning) the search area on the predicted location before performing the image analysis/search within that predicted region [0039]-[0040], [0048], [0065]. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify White with LIN and would have been motivated to do so to improve tracking efficiency and accuracy for moving UI elements across successive video frames.
Allowable Subject Matter
Claim 19 is objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Filing of New or Amended Claims
The examiner has the initial burden of presenting evidence or reasoning to explain why persons skilled in the art would not recognize in the original disclosure a description of the invention defined by the claims. See Wertheim, 541 F.2d at 263, 191 USPQ at 97 (“[T]he PTO has the initial burden of presenting evidence or reasons why persons skilled in the art would not recognize in the disclosure a description of the invention defined by the claims.”). However, when filing an amendment an applicant should show support in the original disclosure for new or amended claims. See MPEP § 714.02 and § 2163.06 (“Applicant should specifically point out the support for any amendments made to the disclosure.”). Please see MPEP 2163 (II) 3. (b)
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Correspondence
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SENG H LIM whose telephone number is (571)270-3301. The examiner can normally be reached Monday-Friday (9-5).
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Xuan Thai can be reached at (571) 272-7147. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Seng H Lim/Primary Examiner, Art Unit 3715
Prosecution Insights