Detailed Action
Notice of Pre-AIA or AIA status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Election/Restrictions
Applicant’s election without traverse of Group I in the reply filed on January 30, 2026 is acknowledged.
Information Disclosure Statement
The information disclosure statements filed on December 27, 2022, June 26, 2023, March 5, 2024, March 3, 2025, and December 31, 2025 comply with the provisions of 37 C.F.R. § 1.97, 1.98, and MPEP § 609, and therefore have been placed in the application file. The information referred to therein has been considered as to the merits.
Specification
The Office objects to the specification for having the following informalities.
The term “fully connected edge” in paragraphs 107–110, 119, 122–123, 224, 245, appears to be a mistranslation or typographical error, where the correct term would be a fully connected graph, rather than edge. For example, as shown throughout the figures, the so-called “fully connected edge 541” points to an entire graph that is fully connected (every node has an edge to every other node, including to itself).
Appropriate correction is required.
Claim Objections
The Office objects to claims 3 and 4 for having the following informalities:
(1) The term “fully connected edge” in claim 3 appears to be a mistranslation or typographical error, where the correct term is a fully connected graph, rather than edge. See FIG. 5A.
(2) Claims 3 and 4 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Appropriate correction is required
Claim Rejections – 35 U.S.C. § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. § 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1, 2, 5, 6, 14–16, and 20 are rejected under 35 U.S.C. § 102(a)(2) as being anticipated by U.S. Patent Application Publication No. 2023/0074584 A1 (“Chiou”).
Claim 1
Chiou discloses:
A method of analyzing an application screen, the method comprising:
“To model keyboard navigation flow of a web page, a Keyboard Navigation Flow Graph (KNFG) is defined.” Chiou ¶ 61.
generating a plurality of links for a plurality of user interface (UI) elements included in the application screen;
“A keyboard navigation flow of a page under test is represented by a set of KNFGs,” Chiou ¶ 61, where “[t]he node set of a KNFG, comprises a node for each HTML element in the page under test,” Chiou ¶ 62, and where “[e]ach of the edges represents a keyboard entry such as Tab that may allow movement between nodes.” Chiou ¶ 66.
generating a UI map for each of at least one primitive action, which is a user input for navigating the application screen, based on the plurality of links; and
As shown in FIG. 2A, a keyboard navigation flow graph 200 is generated by “iteratively exploring the page using only keyboard based actions (i.e.,
Φ
K
) until no new navigation information is found (i.e., the graph has reached a fixed point).” Chiou ¶ 69. Specifically, the set of keyboard actions
Φ
K
may include “all standard keyboard commands used to navigate a web application's user interface as defined by W3C and web accessibility testing communities.” Chiou ¶ 69.
identifying positions of a focus indicating UI elements with which a user is to interact among the plurality of UI elements, wherein the UI map comprises a route via which the positions of the focus move between the plurality of UI elements by the at least one primitive action.
“After each action, the process analyzes the page to determine the focus transition that occurred.” Chiou ¶ 69. “After triggering an action
ϕ
∈
Φ
K
on a node
v
i
, the process detects the focus change from
v
i
to
v
i
+
1
and creates an edge in the graph
v
i
,
v
i
+
1
,
ϕ
,
δ
,
V
s
, indicating that the browser focus could shift from a source node
v
i
to a target node
v
i
+
1
by pressing keystroke ϕ while
v
i
is in focus.” Chiou ¶ 70.
Claim 2
Chiou discloses the method of claim 1, wherein the generating of the plurality of links comprises:
identifying the plurality of UI elements, based on the application screen;
“The example process identifies the nodes by rendering the page under test in a browser and then analyzing the DOM of the page under test to identify each unique HTML element.” Chiou ¶ 62.
generating a plurality of nodes corresponding to the plurality of UI elements; and
“Each node is uniquely identified by its XPath in the DOM.” Chiou ¶ 62.
generating the plurality of links, based on a node feature of the plurality of nodes, and wherein the node feature comprises at least one of features regarding sizes, positions, content, images, names, or hierarchy of the plurality of UI elements.
As understood by the Examiner, the act of generating the links “based on” the node features does not require the links themselves to represent node features, because, in this application, a “link” refers to a navigational pathway for the input focus to move between two nodes in the UI map. (Spec. ¶ 68). Instead, the above claim language merely requires one or more of the node features listed in the claim to influence the method’s assessment of whether there is a navigational “link” between two nodes.
For its part, the process that Chiou uses to generate edges between nodes is influenced by at least the nodes’ sizes, contents, names, and hierarchy.
With respect to sizes, Chiou discloses that “[t]he first iteration of this process begins by interacting with each node
v
∈
V
s
.” Chiou ¶ 69. Vs is defined as a subset of all nodes in the page under test with special predefined characteristics, one of which is elements that are not rendered due to having “a height or width of zero pixels.” Chiou ¶ 68.
With respect to content, Chiou further discloses that Vs may further include strictly “non-disabled elements that do not exhibit a final computed DOM layout style of type=‘hidden’, visibility:hidden, [or] display:none.” Chiou ¶ 68.
With respect to names, Chiou further discloses that “[s]yntactically linked nodes such as a <label> and its bounded form element and elements wrapped within other inline control elements are grouped, since these nodes are intended to represent a single functionality.” Chiou ¶ 62. In other words, irrespective of membership in Vs, Chiou’s method ensures that nodes having certain names like “label” are not eligible to receive edges, and groups them together with other nodes such that edges can only be directed to/from the whole group, rather than the individual node.
Finally, with respect to hierarchy, Chiou discloses taking into consideration whether a node “inherit[ed] their ancestor’s rendered hidden properties” for inclusion in the V-s subset.
Claim 5
Chiou discloses the method of claim 2,
wherein the generating of the UI map comprises generating the UI map by using an edge labeler, and wherein the edge labeler is a model trained to receive an edge for a plurality of nodes including the plurality of links to output the UI map.
“After triggering an action
ϕ
∈
Φ
K
on a node υi, the process detects the focus change from
v
i
to
υ
i
+
1
and creates an edge in the graph
v
i
,
v
i
+
1
,
ϕ
,
δ
,
V
S
, indicating that the browser focus could shift from a source node
υ
i
to a target node
υ
i
+
1
by pressing keystroke ϕ while
υ
i
is in focus.” Chiou ¶ 70. To be clear, in this rejection, the edge labeler corresponds to the “process” described above, which, given an edge in the graph, labels that edge with
ϕ
for the primitive that caused the transition.
Claim 6
Chiou discloses the method of claim 1, wherein the generating of the UI map comprises:
mapping the at least one primitive action for each of the plurality of links; and
“For a given
v
, the process first sets the browser's focus on and then executes every action in ΦK on
υ
.” Chiou ¶ 69. “After triggering an action
ϕ
∈
Φ
K
on a node υi, the process detects the focus change from
v
i
to
υ
i
+
1
and creates an edge in the graph
v
i
,
v
i
+
1
,
ϕ
,
δ
,
V
S
, indicating that the browser focus could shift from a source node
υ
i
to a target node
υ
i
+
1
by pressing keystroke ϕ while
υ
i
is in focus.” Chiou ¶ 70.
generating the UI map, based on what primitive action has been mapped to each of the plurality of links.
“[T]he intra-state edges 262 in the example keyboard navigation flow graph 200 in FIG. 2A are identified by iteratively exploring the page using only keyboard based actions (i.e., ΦK) until no new navigation information is found (i.e., the graph has reached a fixed point).” Chiou ¶ 69.
Claims 14–16
Claims 14–16 recite an electronic device with general purpose computer hardware for performing exactly the same method as set forth in claims 1, 2, and 6. Therefore, the findings from the rejection of claims 1, 2, and 6 are hereby reincorporated by reference, as applied to claims 14–16, taken in conjunction with Chiou’s additional disclosure of the general purpose computer system. See Chiou ¶¶ 114–116.
Claim 20
Claim 20 recites a computer readable medium programmed with the same instructions as the memory of claim 14. Therefore, the findings set forth in the rejection of claim 14 for Chiou’s disclosure of the memory also anticipate the computer readable medium of claim 20 for the same reasons.
Claim Rejections – 35 U.S.C. § 103
The following is a quotation of 35 U.S.C. § 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned at the time any inventions covered therein were effectively filed absent any evidence to the contrary. Applicant is advised of the obligation under 37 C.F.R. § 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned at the time a later invention was effectively filed in order for the examiner to consider the applicability of 35 U.S.C. § 102(b)(2)(C) for any potential 35 U.S.C. § 102(a)(2) prior art against the later invention.
Claims 18 and 19 are rejected under 35 U.S.C. § 103 as being unpatentable over Chiou as applied to claim 14 above, and further in view of U.S. Patent Application Publication No. 2017/0242713 A1 (“Darie”).
Claim 18
Chiou teaches the electronic device of claim 14,
further comprising a communication interface, wherein the at least one processor, when executing the at least one instruction, is further configured to: receive the application screen from an external electronic device through the communication interface,
Throughout Chiou’s disclosure, Chiou explains that its process is performed on a web application. See Chiou Title, Abstract, and ¶¶ 60–67; see Chiou ¶¶ 124–125 (teaching the claimed communication interface). As such, Chiou at least teaches that the application being modeled is one received from an external electronic device (i.e., a web server).
That said, Chiou’s system does not transmit at least one of the UI map or the positions of the focus back to the external electronic device through the communication interface.
Darie, however, teaches a system comprising:
a communication interface,
“All components of the system communicate with each other over the air though an internet connection 110, which may be wired, wireless or combination thereof, as appropriate.” Darie ¶ 27.
wherein the at least one processor, when executing the at least one instruction, is further configured to: receive the application screen from an external electronic device through the communication interface,
“Described herein is a method for creating and utilizing device navigational maps for connected devices. An application systematically scans through the entire connected device and builds a detailed data model of the connected device, which is used as a navigational map (NM).” Darie ¶ 26. The detailed process for building the model of the connected device is shown in FIG. 5, but at a high level, it involves getting the current screen of the device, and activating each of the elements on the screen to crawl all of the screens of the application, creating a vertex for each screen, and an edge to represent transitions therebetween. Darie ¶¶ 41–45.
and transmit at least one of the UI map or the positions of the focus to the external electronic device through the communication interface.
“Given a start screen and a destination screen, the navigation engine 152 computes the path and navigates through the path directly on the connected device. There are two modes of navigating: automatic mode and manual mode. In automatic mode, the map building tool analyzes the current screen and automatically tries to expand the navigational map by invoking the screen elements using the remote command executor. The remote command executor 154 listens to commands received from the navigation map builder 105, for example, and executes it on the connected device.” Darie ¶ 31.
Claim 19
Chiou and Darie teach the electronic device of claim 18, further comprising:
a display, wherein the application screen is displayed on the display as an execution screen of a third-party application not provided with an application program interface.
“In some implementations, a server transmits data (e.g., an HTML page) to a client device (e.g., for purposes of displaying data to and receiving user input from a user interacting with the client device).” Chiou ¶ 124. “A web page 310 that rendered in a browser on a display portion of a computing system is input for analysis.” Chiou ¶ 76.
Other Pertinent Art
The prior art made of record and not relied upon is considered pertinent to Applicants’ disclosure. All four references describe approaches similar to the approach in claim 3, where a dense graph is pruned or sparsified where relevant (though none quite render claim 3 obvious).
Muhan Zhang and Yixin Chen, Link Prediction Based on Graph Neural Networks, Advances in Neural Information Processing Systems 31 (2018), available at https://proceedings.neurips.cc/paper_files/paper/2018/file/53f0d7c537d99b3824f0f99d62ea2428-Paper.pdf
Li Wang et al., Pruning graph neural networks by evaluating edge properties, 256 Knowledge-Based Systems (Nov. 28, 2022) available at https://doi.org/10.1016/j.knosys.2022.109847
Jacinto Reis and Alexandre Mota, Aiding exploratory testing with pruned GUI models, 133 Information Processing Letters 49–55 (2018), available at https://doi.org/10.1016/j.ipl.2018.01.008
L. Mauch and B. Yang, A novel layerwise pruning method for model reduction of fully connected deep neural networks, 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) 2382–86, (2017), available at https://doi.org/10.1109/ICASSP.2017.7952583
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Justin R. Blaufeld whose telephone number is (571)272-4372. The examiner can normally be reached M-F 9:00am - 4:00pm ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, James K Trujillo can be reached at (571) 272-3677. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
Justin R. Blaufeld
Primary Examiner
Art Unit 2151
/Justin R. Blaufeld/Primary Examiner, Art Unit 2151
Prosecution Insights