DETAILED ACTION
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
This Office Action is responsive to the Amendment filed on 9/3/2026. Claims 1-16 are pending in the case.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-2, 8-9, and 15-16 are rejected under 35 U.S.C. 103 as being unpatentable over Hailpern et al. (US 20180329726 A1, hereinafter Hailpern) in view of Milov et al. (US 20090103769 A1, hereinafter Milov), and Song (US 20130227014 A1).
As to independent claim 1, Hailpern teaches a method for recording and playback, comprising:
recording an operation of a user in a first device based on an operation recording instruction to obtain a user operation record (“The user-activatable element programming engine 102 records (at 302) user actions in multiple different environments, including environments provided by multiple applications, for example. The recording can be initiated in response to a user input, such as user selection of the record element 110 (FIG 1).”);
acquiring environment data of the first device (“Initiation of the recording starts a record mode, in which user actions of made with respect to different applications can be monitored and recorded. During use of the applications, different ones of the applications can be in focus at different times.” Paragraph 0033-0034); and
performing user operations of the user operation record sequentially in a second device based on an operation playback instruction (“At system 2, a user action replay engine 502 (similar to the user action replay engine 104 discussed above) can use the template 500 to cause the customized user-activatable element 112 to be displayed in a display 504 in system 2 as 506, so that a user at system 2 can activate the customized user-activatable element 506 to replay the associated recorded user actions at system 2.” Paragraph 0043).
Hailpern does not appear to expressly teach wherein the performing the user operations of the user operation record comprises, for one of the user operations:
determining a corresponding target control in the second device based on the environment data of the first device and environment data of the second device, and
performing a respective user operation based on the target control.
Milov teaches wherein the performing the user operations of the user operation record comprises, for one of the user operations:
determining a corresponding target control in the second device based on the environment data of the first device and environment data of the second device (The cognitive control framework (CCF) system of embodiments of the present invention performs image analysis and object detection processing on display data from the GUI of the application program. The CCF system includes comparing an image captured during a recording phase (called IR) to the corresponding image captured during the playback phase (called IP). One task of the system is to detect an object in the IR to which the user applied an action, find the corresponding object in the IP, and continue progress on the execution path of the execution scenario by applying the action to the detected object. These steps may be repeated for multiple objects within an image, and may be repeated across multiple pairs of IRs and IPs over time.” Paragraph 0021,0018, “playback might be executed in different operating environments such as different screen resolutions, different visual schemes, different window layouts, and so on.” Paragraph 0022); and
performing a respective user operation based on the target control (“An object that the user has applied an action to may be called an "object of action.", paragraph 0021).
Accordingly, it would have been obvious to a person of skill in the art before the effective filing date of the claimed invention to modify the method of Hailpern to comprise wherein the performing the user operations of the user operation record comprises, for one of the user operations: determining a corresponding target control in the second device based on the environment data of the first device and environment data of the second device, and performing a respective user operation based on the target control. One would have been motivated to make such a combination to “introduce only minimal disturbances and delay of application execution during playback.” Milov [0022] last sentence.
Hailpern and Milo do not appear to expressly teach wherein the environment data of the first device includes device-related data including a device system platform of the first device, and the environment data of the second device includes device-related data including a device system platform of the second device, and wherein the device system platform of the first device is different from the device system platform of the second device.
Song teaches wherein the environment data of the first device includes device-related data including a device system platform of the first device (“a transmission device operating in an Android.RTM. environment” paragraph 0154), and the environment data of the second device includes device-related data including a device system platform of the second device, and wherein the device system platform of the first device is different from the device system platform of the second device (“Further, according to exemplary embodiments of the present disclosure, if a transmission device operating in an Android.RTM. environment shares specific data operating in the Android.RTM. environment with a reception device operating in an iOS.RTM. environment, the transmission device may share the data with the reception device by sharing a user interface of an application used to process the data to be shared with the reception device.” Paragraph 0154-0155).
Accordingly, it would have been obvious to a person of skill in the art before the effective filing date of the claimed invention to modify the method of Hailpern to comprise wherein the environment data of the first device includes device-related data including a device system platform of the first device, and the environment data of the second device includes device-related data including a device system platform of the second device, and wherein the device system platform of the first device is different from the device system platform of the second device. One would have been motivated to make such a combination to improve computing device autonomous GUI operation.
As to dependent claim 2, Hailpern teaches the method according to claim 1, Hailpern further teaches wherein the environment data includes:
device-related data, application-related data, user-related data, or user operation-related data (“Initiation of the recording starts a record mode, in which user actions of made with respect to different applications can be monitored and recorded. During use of the applications, different ones of the applications can be in focus at different times.” Paragraph 0033-0034).
Hailpern does not appear to expressly teach wherein the user operation-related data includes:
a page screenshot, an XML layout file of a page, or a control screenshot of an operation object control.
Milov teaches wherein the user operation-related data includes:
a page screenshot, an XML layout file of a page, or a control screenshot of an operation object control (“This input data is captured and stored by the CCF system. Next, at block 108, the display data may be captured (e.g. screen shots are taken). In one embodiment, the display data may captured only when user input has been received by the application program. The display data is also saved. At block 110, the data captured during blocks 106 and 108 may be analyzed and saved. These processes may be repeated a plurality of times. The result of the processing of the recording component comprises an execution scenario 112 for the application program being processed by the system. In one embodiment, the execution scenario comprises a script containing Extended Markup Language (XML) tags.” Paragraph 0018).
Accordingly, it would have been obvious to a person of skill in the art before the effective filing date of the claimed invention to modify the method of Hailpern to comprise wherein the user operation-related data includes: a page screenshot, an XML layout file of a page, or a control screenshot of an operation object control. One would have been motivated to make such a combination to “introduce only minimal disturbances and delay of application execution during playback.” Milov [0022] last sentence.
As to dependent claim 16, Hailpern teaches the method according to claim 1, Hailpern does not appear to expressly teach wherein the environment data of the first device includes user operation-related data including an XML layout file having control attributes, and
wherein the determining the corresponding target control comprises comparing the control attributes from the XML layout file of the first device with control attributes from an XML layout file of the second device to position the target control in the second device.
Song teaches wherein the environment data of the first device includes user operation-related data including an XML layout file having control attributes (“a layout file used to share a user interface of an application between devices operating in different OS environments may be implemented by an XML file.” Paragraph 0055), and
wherein the determining the corresponding target control comprises comparing the control attributes from the XML layout file of the first device with control attributes from an XML layout file of the second device to position the target control in the second device (“As described above, the transmission device 10, which may acquire the layout file name matching the resource ID of the main screen, may extract a layout file using the acquired layout filename and may transmit the layout file to the reception device 20. The subsequent steps thereof may be similar or the same as the steps of sharing the application operating in the foreground mode and thus a description thereof will be omitted. As described above, according to an exemplary embodiment of the present disclosure, if a user interface is shared by sharing a layout file, then at least one of data, such as text, images, audio and video may be shared between devices operating in different OS environments.” Paragraph 0152-0153).
Accordingly, it would have been obvious to a person of skill in the art before the effective filing date of the claimed invention to modify the method of Hailpern to comprise wherein the environment data of the first device includes user operation-related data including an XML layout file having control attributes, and wherein the determining the corresponding target control comprises comparing the control attributes from the XML layout file of the first device with control attributes from an XML layout file of the second device to position the target control in the second device. One would have been motivated to make such a combination to improve computing device autonomous GUI operation.
Claims 8-9 and 15 are substantially the same as claims 1-2 and are therefore rejected under similar rationale as above.
Allowable Subject Matter
Claims 3-7 and 10-14 are objected to as being dependent upon a rejected base claim but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Response to Arguments
Applicants’ arguments with respect to claim 1 have been considered but are moot in view of the new grounds of rejection presented above.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure:
Kochura US 20170060368 A1 Computer-vision based execution of GUI application actions.
Stan US 11294793 Robotic process automation (RPA) debugging systems and methods.
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MAHELET SHIBEROU whose telephone number is (571)270-7493. The examiner can normally be reached Monday-Friday 9:00 AM-5:00 PM Eastern Time.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kieu Vu can be reached at 571-272-4057. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MAHELET SHIBEROU/Primary Examiner, Art Unit 2171
