DETAILED ACTION
This Office Action is sent in response to Applicant's Response filed 06/03/2026 for 18502223. Claims 1-4, 6-8, 10-14, 16-18, 20-21, and 23-26 are presented.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
Response to Arguments
Applicant's summary of the telephonic interview conducted 05/07/2026 has been acknowledged and accepted.
In view of Applicant's amendments, the objection to the drawings has been withdrawn.
In view of Applicant's amendments, the 112 rejection of claims 21 and 25 has been withdrawn.
Applicant's arguments with respect to the 103 rejection of claim 1 has been fully considered but are not persuasive.
Applicant’s arguments with respect to the 103 rejection of claim 1 have been fully considered but are not persuasive in view of the new and/or updated citations used in the current rejection of record under Manske in view of Naderi in response to the newly amended limitations.
Applicant argues that Manske, alone or in combination, does not teach "performing a schema extraction anchoring of the one or more target UI elements by linking the DOM to corresponding types of the one or more relevant UI elements" [pgs. 9:3-10:2]. Examiner respectfully disagrees.
Applicant's specification does not disclose a special definition of "schema extraction anchoring", merely an example of how one is performed, "To determine the relevant DOM text, the interface engine embeds information by encoding and incorporating translational and scale invariant features for the ML model (note that the features can be based on information already extracted by the interface engine). The translational and scale invariant features are then fed through various layers of the ML model, for example appropriate attention, convolutions, fully connected, etc. A predictor of the ML model computes logits responsible to assign the appropriate relations between input fields and labels" [Specification, para 0147]. Thus Applicant appears to argue "schema extraction anchoring" is one which involves a machine learning model embedding encoding and invariant features to assign relationships using computed logits. This definition of "anchoring" is not recited in the claim. The broadest reasonable interpretation of "anchoring" includes any link between elements as consistent with Applicant's specification [para 0146] and does not preclude probability-based matching.
In view of Applicant's assertion that the schema extraction links the DOM text to corresponding types of the relevant UI elements by connecting or creating associations between fields and labels [pg. 10:2], the Office Action discusses "linking the DOM to the corresponding types of the one or more relevant UI elements to generate associations between fields and labels of the page" as recited in currently rejected dependent claim 24 when Manske discloses identifying a mapping, interpreted as "linking", between application attributes present in a DOM tree and attributes rendered an application user interface [para 0032, 0045-0046] and using the mapping to fill application entry fields with identified field labels [para 0046-0047, 0063, 0065-0071].
Applicant also argues that Manske and Naderi, alone or in combination, do not teach "extrapolating, by the interface engine, one or more target UI elements of an invisible region of the page from the one or more relevant UI elements to identify and understand what type of off screen elements exist" [pg. 10:3-11:1]. Examiner respectfully disagrees.
In response to Applicant’s arguments against the references individually (that at Naderi "does not involve extrapolating to identify and understand what type of off-screen elements exist" [pg. 10:4]), one cannot show nonobviousness by attacking references individually where the rejections are based on combinations of references. See In re Keller, 642 F.2d 413, 208 USPQ 871 (CCPA 1981); In re Merck & Co., 800 F.2d 1091, 231 USPQ 375 (Fed. Cir. 1986). In this case, the Office Action cites the combination of Manske using application attributes present in a DOM tree to identify reusable data objects rendered, interpreted as "existing", in an application user interface [para 0046] with Naderi recognizing form fields not currently displayed, interpreted as "off screen elements", on an application screen [para 0033, 0051] to teach the newly amended limitations of claim 1.
In response to Applicant's arguments that the references fail to show certain features of Applicant’s invention, it is noted that the features upon which Applicant relies (i.e., where "a ML model of the interface engine determines an HTML structure for the identified input fields of the visible region and uses that HTML structure to find a lookalike HTML structure in the invisible region" [pgs. 10:5-11:1]) are not recited in the rejected instant claim 1. Although the claims are interpreted in light of the specification, limitations from the specification are not read into the claims. See In re Van Geuns, 988 F.2d 1181, 26 USPQ2d 1057 (Fed. Cir. 1993). In this case, the Office Action discusses "searching for similar hypertext markup language (HTML) structures from the DOM" as recited in currently rejected dependent claim 3 when Manske discloses identifying attributes rendered in an application user interface by searching for attributes in the application DOM structure of an HTML form [para 0032].
Therefore, the combination of Manske and Naderi teach the limitations of claim 1. Claim 1 remains rejected.
Claim 11 recites similar limitations to those recited in claim 1 and remains rejected upon a similar basis as claim 1 as stated above.
Dependent claims 2-4, 6-8, 10, 12-14, 16-18, 20-21, and 23-26 remain rejected at least based on their dependence from independent claims 1 and 11.
Claim Objections
Claims 1, 11, and 26 are objected to because of the following informalities.
Claims 1 and 11 recite "the one or more target UI elements" which lacks antecedent basis and has been interpreted as "[[the]] one or more target UI elements".
Claim 26 recites thevone" which appears to include a typo and has been interpreted as "[[thevone]] --the one--".
Appropriate correction is required.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries set forth in Graham v. John Deere Co., 383 U.S. 1, 148 USPQ 459 (1966), that are applied for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1-4, 6-8, 10-14, 16-18, 20-21, and 23-26 is/are rejected under 35 U.S.C. 103 as being unpatentable over Manske et al. (US 20180095651 A1) in view of Naderi (US 20120054593 A1).
As to claim 1, Manske discloses a method for detecting and automatically filling one or more user interface (UI) elements of a page … on a screen, the method being executed by an interface engine implemented as a computer program within a computing environment [Figs. 1, 8, 10, para 0023, 0030, 0034-0036, 0045, fill target data objects (read: elements) in target application part (read: user interface) in web application with reuse platform (read: interface engine) executing as displayed system application], the method comprising:
analyzing, by the interface engine, document object model (DOM) of the page to extract DOM text of one or more relevant UI elements of the one or more UI elements [para 0030-0032, 0043-0046, reuse application searches document object model of target application part in web application for semantic attributes (read: DOM text) of reusable data object (read: relevant UI element) of target application data object (read: UI element)];
performing, by the interface engine, computer vision (CV) analysis to determine one or more types of the one or more relevant UI elements [para 0032, 0048-0049, machine learning algorithm inspects properties (read: types) of reusable data objects in DOM structure, where algorithm identifying property labels is consistent with the broadest reasonable interpretation of computer vision analyzing label data using machine learning per Applicant's specification (para 0086)];
performing a schema extraction anchoring of the one or more target UI elements by linking the DOM to corresponding types of the one or more relevant UI elements [para 0036, 0046, 0065-0068, connect (read: anchor) element identified in DOM structure (read: schema extraction) to target application data object]; and
extrapolating, by the interface engine, one or more target UI elements … of the page from the one or more relevant UI elements to identify and understand what type of [] elements exist [para 0032, 0044-0048, reuse application identifies reusable data object of target application part in web application] and
extracting data from a source to paste into at least the one or more target UI elements… [Figs. 5-7, para 0022, 0027, 0033-0035, 0037, reuse application copies and pastes source attribute value to target application data object].
However, Manske does not specifically disclose one or more user interface (UI) elements of a page that are not visible on a screen; and one or more target UI elements of an invisible region of the page, wherein "[] elements" are "off screen elements", and the invisible region being outside of a web browser window or an interface frame.
Naderi discloses:
one or more user interface (UI) elements of a page that are not visible on a screen [para 0033, 0051, webpage element not currently displayed on system display (read: screen)];
one or more target UI elements of an invisible region of the page [para 0033, 0051, 0054-0055, additional form field (read: target UI element) in form not currently displayed (read: invisible region) on webpage],
off screen elements [para 0033, 0051, webpage element not currently displayed], and
the invisible region being outside of a web browser window or an interface frame [para 0033, 0051, 0055, 0078, form not currently displayed in visible graphical user interface (read: interface frame) on system display].
Manske and Naderi are analogous art to the claimed invention being from a similar field of endeavor of user interface systems. Thus, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify the user interface elements, target user interface elements, screen elements, and page as disclosed by Manske with user interface elements of a page not visible on a screen, target user interface elements on an invisible region of the page outside of a web browser window or an interface frame, and off screen elements as disclosed by Naderi with a reasonable expectation of success.
One of ordinary skill in the art would be motivated to modify Manske as described above to provide a simplified and ergonomic way of filling forms [Naderi, para 0034].
As to claim 2, Manske discloses the method of claim 1, wherein the method includes linking the DOM text to the one or more target UI elements based on the one or more types and the one or more relevant UI elements as determined by the CV analysis [para 0032, 0063-0071, match DOM semantic attributes to reusable target application data object using properties of reusable objects inspected by algorithm].
As to claim 3, Manske discloses the method of claim 1, wherein the extrapolating of the one or more target UI elements by the interface engine comprises searching for similar hypertext markup language (HTML) structures from the DOM [para 0032, 0068-0070, reuse application searches HTML attributes in DOM structure].
As to claim 4, Manske discloses the method of claim 1, wherein the one or more UI elements comprises one or more input fields of the page [Fig. 3, para 0034, 0046-0048, target application data object includes entry field of target application part in web application].
As to claim 6, Manske discloses the method of claim 1, wherein the one or more types of the one or more relevant UI elements comprise [para 0032, 0068-0071, reuse application searches for reusable data object label in HTML DOM structure, note strikethrough indicates non-selected alternatives].
As to claim 7, Manske discloses the method of claim 1, wherein the performing of the CV analysis by the interface engine comprises capturing a visible region of the page to identify the one or more types of the one or more relevant UI elements in the visible region [Figs. 2-3, 10, para 0032, 0045-0046, algorithm identifies object label of reusable data object from DOM tree from user interface querying target user interface area displayed in application user interface (read: visible region)].
As to claim 8, Manske discloses the method of claim 1, wherein the interface engine inputs data into the one or more target UI elements … [Figs. 5-7, para 0022, 0027, 0033-0035, 0037, reuse application copies and pastes source attribute value to target application data object].
However, Manske does not specifically disclose wherein the invisible region of the page comprises a destination form of a subsequent.
Naderi discloses inputting data into the one or more target UI elements without scrolling the page [para 0034, 0038, 0055, fill data in form fields without needing to scroll webpage form].
Manske and Naderi are analogous art to the claimed invention being from a similar field of endeavor of user interface systems. Thus, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify inputting data into target UI elements as disclosed by Manske with inputting data into target UI elements without scrolling a page as disclosed by Naderi with a reasonable expectation of success.
One of ordinary skill in the art would be motivated to modify Manske as described above to provide a simplified and ergonomic way of filling forms [Naderi, para 0034].
As to claim 10, Manske discloses the method of claim 1.
However, Manske does not specifically disclose wherein the invisible region of the page comprises a destination form of a subsequent or destination page.
Naderi discloses wherein the invisible region of the page comprises a destination form of a subsequent or destination page [para 0033-0034, 0078, form with section not currently visible in webpage graphical user interface on system display, note form section requiring scrolling to be visible in graphical user interface falls under broadest reasonable interpretation of destination form including a different interface frame than a current page as consistent with Applicant's specification (para 0143)].
Manske and Naderi are analogous art to the claimed invention being from a similar field of endeavor of user interface systems. Thus, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify the page as disclosed by Manske with an invisible region of a page comprising a destination form of a subsequent or destination page as disclosed by Naderi with a reasonable expectation of success.
One of ordinary skill in the art would be motivated to modify Manske as described above to provide a simplified and ergonomic way of filling forms [Naderi, para 0034].
As to claim 11, Manske and Naderi, combined at least for the reasons above, disclose a system comprising: a memory storing code; and at least one processor configured to execute the code to cause within the system [para 0038-0039, 0045-0046, platform includes storage device storing program performed by processor to fill target data objects]: limitations substantially similar to those recited in claim 1 and is rejected under similar rationale.
As to claims 12-14, 16-18, and 20, Manske and Naderi, combined at least for the reasons above, disclose the system of claim 11 comprising limitations substantially similar to those recited in claims 2-4, 6-8, and 10, respectively, and are rejected under similar rationale.
As to claim 21, Manske discloses the method of claim 1, wherein extrapolating of the one or more target UI elements … comprising using a cache to avoid scrolling [para 0032, 0043-0048, reuse application identifies reusable data object of target application part in web application as stored in platform (read: cache), note the limitation "to avoid conventional scrolling" is not being given patentable weight as the term "to" suggests or makes optional and does not require the step to be performed as the limitation is an intended result of the "cache" as recited in the claim (see MPEP 2111.04)].
However, Manske does not specifically disclose the one or more target UI elements of the invisible region.
Naderi discloses the one or more target UI elements of the invisible region [para 0033, 0051, 0054-0055, additional form field (read: target UI element) in form not currently displayed (read: invisible region) on webpage].
Manske and Naderi are analogous art to the claimed invention being from a similar field of endeavor of user interface systems. Thus, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify the target UI elements as disclosed by Manske with target UI elements of an invisible region as disclosed by Naderi with a reasonable expectation of success.
One of ordinary skill in the art would be motivated to modify Manske as described above to provide a simplified and ergonomic way of filling forms [Naderi, para 0034].
As to claim 23, Manske discloses the method of claim 1, wherein the schema extraction comprises one or more schema elements [para 0036, 0046, 0065-0071, identify DOM attribute (read: schema element) including entry field].
However, Manske does not specifically disclose where one or more schema elements causes a new feature to appear within the page.
Naderi discloses where one or more schema elements causes a new feature to appear within the page [Fig. 10, para 0059-0060, navigating drop down field fills value (read: new feature) in electronic form].
Manske and Naderi are analogous art to the claimed invention being from a similar field of endeavor of user interface systems. Thus, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify schema elements as disclosed by Manske with schema elements causing a new feature to appear as disclosed by Naderi with a reasonable expectation of success.
One of ordinary skill in the art would be motivated to modify Manske as described above to provide a simplified and ergonomic way of filling forms [Naderi, para 0034].
As to claim 24, Manske discloses the method of claim 1, wherein the interface engine performs the schema extraction linking the DOM to the corresponding types of the one or more relevant UI elements to generate associations between fields and labels of the page [para 0036, 0046, 0065-0071, connect (read: link) identified DOM structure element to target application data object, note the limitation "to generate associations between fields and labels of the page" is not being given patentable weight as the term "to" suggests or makes optional and does not require the step to be performed as the limitation is an intended result of the "linking" as recited in the claim (see MPEP 2111.04); nevertheless, note algorithm connects identified DOM field attribute to target application including entry field and field label].
As to claim 25, Manske discloses the method of claim 1, wherein the interface engine inserts the data into the one or more target UI elements … [Figs. 5-7, para 0022, 0027, 0033-0035, 0037, reuse application pastes source attribute value to target application data object].
However, Manske does not specifically disclose wherein the interface engine inserts the data into the one or more target UI elements of the invisible region without scrolling.
Naderi discloses wherein the interface engine inserts the data into the one or more target UI elements of the invisible region without scrolling [para 0033-0034, 0038, 0054-0055, fill data in form fields (read: target UI elements) without needing to scroll form to section not currently visible requiring scrolling].
Manske and Naderi are analogous art to the claimed invention being from a similar field of endeavor of user interface systems. Thus, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify inserting data into target UI elements as disclosed by Manske with inserting data into target UI elements of an invisible region as disclosed by Naderi with a reasonable expectation of success.
One of ordinary skill in the art would be motivated to modify Manske as described above to provide a simplified and ergonomic way of filling forms [Naderi, para 0034].
As to claim 26, Manske discloses the method of claim 1, wherein the interface engine works … to identify one or more off screen elements that are of a same type as thevone or more types of the one or more relevant UI elements in view based on tag and attributes for the one or more off screen elements [para 0023, reuse platform executes].
However, Manske does not specifically disclose wherein the interface engine works off screen.
Naderi discloses wherein the interface engine works off screen [para 0033, 0051, 0055, form wizard accesses form fields not currently displayed].
While Manske does not teach "to identify one or more off screen elements that are of a same type as thevone or more types of the one or more relevant UI elements in view based on tag and attributes for the one or more off screen elements", one of ordinary skill in the art would recognize that the limitation "identify one or more off screen elements that are of a same type as thevone or more types of the one or more relevant UI elements in view based on tag and attributes for the one or more off screen elements" is an intended result of the "interface engine" and is not relevant to patentability [see MPEP 2111.04].
Nevertheless, in an effort to advance compact prosecution, Manske teaches wherein the interface engine works to identify one or more [] screen elements that are of a same type as thevone or more types of the one or more relevant UI elements in view based on tag and attributes for the one or more [] screen elements [para 0031, 0045-0047, 0049, 0065-0071, platform matches target application data object (read: screen element) to identical reusable data object by inspecting attributes and attribute values of compared application data objects] and Naderi teaches one or more off screen elements [para 0033, 0051, 0055, form wizard accesses form fields not currently displayed].
Manske and Naderi are analogous art to the claimed invention being from a similar field of endeavor of user interface systems. Thus, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify the interface engine identifying screen elements based on tag and attributes as disclosed by Manske with an interface engine working off screen to identify off screen elements as disclosed by Naderi with a reasonable expectation of success.
One of ordinary skill in the art would be motivated to modify Manske as described above to provide a simplified and ergonomic way of filling forms [Naderi, para 0034].
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Jain et al. (US 20250131185 A1) generally discloses robotic process automation using computer vision to automatically fill application forms.
Gligan et al. (US 20210097274 A1) generally discloses a document processing framework for robotic process automation.
Voicu et al. (US 11200073 B1) generally discloses schema extraction anchoring of hidden UI elements to a document object model.
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to LINDA HUYNH whose telephone number is (571)272-5240 and email is linda.huynh@uspto.gov. The examiner can normally be reached M-F between 9am-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Adam Queler can be reached at (571) 272-4140. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/LINDA HUYNH/Primary Examiner, Art Unit 2172
Prosecution Insights