DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
Receipt is acknowledged of claim amendments with associated arguments/remarks, received December 04, 2025. Claims 1, 7-9, 14-19 are pending in which claims 1, 14, 17 were amended. Claims 2-6, 10-13, 20 are cancelled.
Response to Arguments
Applicant’s arguments, see Remarks, pg 7, filed 12/04/2025, with respect to the rejections of claim 1, 7-9, 14-19 under 35 USC § 112( has been fully considered and, in light of the associated amendment, is persuasive. Therefore, the rejection has been withdrawn.
Applicant’s arguments, see Remarks, pg 7, filed 12/04/2025, with respect to the rejections of claim 1, 7-9, 14-16 under 35 USC § 112(a), enablement has been fully considered and, in light of the associated amendment, is persuasive. Therefore, the rejection has been withdrawn.
Applicant’s arguments, see Remarks, pg 7, filed 12/04/2025, with respect to the rejections of claim 1, 7-9, 14-19 under 35 USC § 112(b) has been fully considered and, in light of the associated amendment, is persuasive. Therefore, the rejection has been withdrawn.
Applicant’s arguments, see Remarks, pg 7, filed 12/04/2025, with respect to the rejections of claim 1, 7-9, 14-19 under 35 USC has been fully considered and, in light of the associated amendment, is persuasive. Therefore, the rejection has been withdrawn. The amendments changed the scope of each independent claim and, upon further consideration, a new grounds of rejection is made for claims 14-16 under Bataller et al (US 2017/0001308) and for claims 1, 7-9, 17-19 under Bataller et al (US 2017/0001308) in view of Viet et al (US 2020/0059441). The examiner also notes no argument was presented by the applicant to distinguish the applicant’s claimed invention from any of the previous cited prior art (Remarks – 12/04/2025, pg 7-8). Respectfully, the argument is not persuasive.
All arguments were addressed.
Claim Objections
Claim 17 is objected to because of the following informalities: Claim 17 is objected to for the amended claim limitation “wherein the computer system is configured to perform the steps of” that appears to be missing the colon after “of” before listing the associated claimed steps.. Appropriate correction is required.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claims 14-16 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Bataller et al (US 2017/0001308, previously cited in Non-Final Rejection - 06/04/2025).
Regarding Claim 14, Bataller et al teach a method for guiding a user on a graphical interface with computer vision (computer system 100 with computer vision for automating a manual process 300 to create an automated system 200 executed by a robot 240; Fig 1-4 and ¶ [0017], [0038], [0054], [0062]) comprising:
providing a step-by-step guide for a mobile workflow (“mobile workflow” described broadly as “any human computer task to be automated with computer vision and robotic touch arm” specification ¶ [0002]-[0003], [0023]) (computer vision techniques are applied to identify activities of the user on a mobile device (¶ [0071]-[0073]) for a user manual process to be automated, S310-340, to create a process definition (a step-by-step guide for mobile workflow), S350, using process definition generator 140 (based on pipeline 110, 120, 130); Fig 1, 3 and ¶ [0017], [0031]-[0033], [0054]-[0059]);
observing a mobile device with a graphical user interface with a computer vision system (computer vision techniques are applied to the identified activities manually performed by a user interacting with a computer (mobile device ¶ [0071]), S310-S330, with images (observing) obtained by the image capturer 110; Fig 1, 3 and ¶ [0022]-[0023], [0055]-[0057]);
providing an ability to use computer vision to wherein the computer vision system is configured to observe an area of user touch interest at a create time for the mobile workflow (the image capturer 110 (computer vision system) obtains images of a computer display that contains a touchscreen user interface, S320 and the images are analyzed by the activity identifier 120, S330; Fig 1, 3 and ¶ [0022]-[0023], [0056]-[0057]);
running a computer vision algorithm in communication with the computer vision system (images of user activities are captured by the image capturer 110, S320, and identified by the activity identifier 120, S330, using computer vision techniques ; Fig 1, 3 and ¶ [0021]-[0023], [0056]-[0057]);
finding, with the computer vision system, the area of user touch interest at play back time (the activity information generator 130 identifies the image data representing the user screen touch activity, such as the location of the physical screen touch that user touched to indicate a response, found in the image capturer 110 images, S340; Fig 1, 3 and ¶ [0025]-[0026], [0058]), wherein the computer vision algorithm is implemented on one rectangle in a grid of interest (the touchscreen of the user interface is represented with a rectangle (“Yes” and “No” buttons represented as rectangle via pixel regions ¶ [0053]) based on pixel coordinates and the activity information generator 130 executes identification of the activity (touch) was performed in the region of interest, S330; Fig 1, 3 and ¶ [0032]-[0033], [0057]);
selecting a matching algorithm to use for a given step of the method (the activity execution engine 230 executes activities based on process definition (matching algorithm) identified from the activity trigger engine 220; Fig 2, 3 and ¶ [0044], [0051], [0060]), wherein the matching algorithm is selected from one of a template matching algorithm, a feature matching algorithm, and a color matching algorithm (the process definition may be a process for generating a template, including a template matching algorithm (¶ [0055], [0059]), a feature matching algorithm (a mouse cursor above a particular screen (grid) region to click; ¶ [0056]-[0057]) and a color matching algorithm (a different color in response to a click, ¶ [0056]);
linking each user input click with the step-by-step guide via the graphical interface (the activities for a given process definition are analyzed for multiple users to determine a particular similar procedure; ¶ [0034]-[0035], [0061]) while graphical user interface is being observed by computer vision system (the activities associated with the process (user input via graphic interface) are identified using computer vision techniques; ¶ [0033]-[0035], [0056]-[0057]),
wherein the linking involves the selected matching algorithm (the activity execution engine 230 executes activities based on process definition (matching algorithm) identified from the activity trigger engine 220; Fig 2, 3 and ¶ [0044], [0051], [0060]); and
automating the mobile workflow with the computer vision system and a robotic touch arm (the process definition for all of the activities are automated with movement instructions by the process definition generator 140, S350, so the given activity may be executed by the robot arm 240 via the process definition; Fig 1-3 and ¶ [0030]-[0031], [0051], [0059]-[0060]), wherein the computer vision system observes the robotic touch arm interacting with the graphical user interface (the activity execution engine 230 may execute instructions for the robot 240 to perform the activity (execute the graphic interface activity); Fig 2, 3 and ¶ [0046]-[0047], [0051], [0059]-[0060]).
Regarding Claim 15, Bataller et al teach method of claim 14 (as described above), wherein the matching algorithm determines one or more algorithms to use for any step of the step-by-step guide (the activity execution engine 230 executes activities based on process definition (matching algorithm) identified from the activity trigger engine 220; Fig 2, 3 and ¶ [0044], [0051], [0060]).
Regarding Claim 16, Bataller et al teach method of claim 15 (as described above), wherein the one or more algorithms include a template matching algorithm, a feature matching algorithm, and a color matching algorithm (the process definition (matching algorithm) may be a process for generating a template, including a template matching algorithm (¶ [0055], [0059]), a feature matching algorithm (a mouse cursor above a particular screen (grid) region to click; ¶ [0056]-[0057]) and a color matching algorithm (a different color in response to a click, ¶ [0056]).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 7-9, 17-19 are rejected under 35 U.S.C. 103 as being unpatentable over Bataller et al (US 2017/0001308, previously cited in Non-Final Rejection - 06/04/2025) in view of Viet et al (US 2020/0059441, previously cited in Non-Final Rejection - 06/04/2025).
Regarding Claim 1, Bataller et al teach a computerized method (method 300 for automating a manual process using system 100; Fig 1, 3 and ¶ [0017], [0054]) comprising:
providing a step-by-step guide for a mobile workflow (“mobile workflow” described broadly as “any human computer task to be automated with computer vision and robotic touch arm” specification ¶ [0002]-[0003], [0023]) (computer vision techniques are applied to identify activities of the user on a mobile device (¶ [0071]-[0073]) for a user manual process to be automated, S310-340, to create a process definition (a step-by-step guide for mobile workflow), S350, using process definition generator 140 (based on pipeline 110, 120, 130); Fig 1, 3 and ¶ [0017], [0031]-[0033], [0054]-[0059]);
observing a mobile device with a graphical user interface with a computer vision system (computer vision techniques are applied to the identified activities manually performed by a user interacting with a computer (mobile device ¶ [0071]), S310-S330, with images (observing) obtained by the image capturer 110; Fig 1, 3 and ¶ [0022]-[0023], [0055]-[0057]), wherein the computer vision system is configured to observe an area of user touch interest at a create time for the mobile workflow performed using the graphical user interface of the mobile device (the image capturer 110 (computer vision system) obtains images of a computer display that contains a touchscreen user interface, S320 and the images are analyzed by the activity identifier 120, S330; Fig 1, 3 and ¶ [0022]-[0023], [0056]-[0057]), wherein the mobile workflow comprises one or more tasks performed by a human user (tasks to be automated are based on a user manual process; Fig 1, 3 and ¶ [0021]-[0023], [0056]);
running a computer vision algorithm in communication with the computer vision system (images of user activities are captured by the image capturer 110, S320, and identified by the activity identifier 120, S330, using computer vision techniques ; Fig 1, 3 and ¶ [0021]-[0023], [0056]-[0057]);
locating the area of user touch interest at a play back time when automated mobile workflow is running (the activity information generator 130 identifies the image data representing the user screen touch activity, such as the location of the physical screen touch that user touched to indicate a response, found in the image capturer 110 images, S340; Fig 1, 3 and ¶ [0025]-[0026], [0058]);
optimizing a workflow guide (the process definitions (workflow guide) are generated by the process definition generator 140 from activity identified by the activity information generator 130, S350, and may be inspected and optimized for a particular process; Fig 1, 3 and ¶ [0030], [0061]);
linking each user input click with the step-by-step guide via the graphical interface (the activities for a given process definition are analyzed for multiple users to determine a particular similar procedure; ¶ [0034]-[0035], [0061]) while graphical user interface is being observed by computer vision system (the activities associated with the process (user input via graphic interface) are identified using computer vision techniques; ¶ [0033]-[0035], [0056]-[0057]);
determining a grid of interest relative to the screen (the touchscreen of the user interface is represented with a region of interest (“Yes” and “No” buttons representing a “grid” region based on pixel coordinates and the activity information generator 130 executes identification of the activity (touch) was performed in the region of interest, S330; Fig 1, 3 and ¶ [0032]-[0033], [0057]), wherein the grid comprises normalized pixels; and
automating the mobile workflow with a computer vision functionality and a robotic touch arm using the step-by-step guide (the process definition for all of the activities are automated with movement instructions by the process definition generator 140, S350, so the given activity may be executed by the robot arm 240 via the process definition; Fig 1-3 and ¶ [0030]-[0031], [0051], [0059]-[0060]);
guiding a user's interaction with the graphical user interface using the computer vision system (a user may open an application that automatically performs processes according to process definitions, automated by the activity execution engine 230, with a user interface to record user response with activity trigger engine 220; Fig 2, 3 and ¶ [0041]-[0043], [0059]);
wherein the computer vision algorithm is implemented on one rectangle in the grid of interest (the touchscreen of the user interface is represented with a rectangle (“Yes” and “No” buttons represented as rectangle via pixel regions ¶ [0053]) to determine if a trigger event (touch) was performed by user; Fig 2, 3 and ¶ [0042]-[0043], [0059]),
selecting a matching algorithm to use for a given step of the mobile workflow (the activity execution engine 230 executes activities based on process definition (matching algorithm) identified from the activity trigger engine 220; Fig 2, 3 and ¶ [0044], [0051], [0060]), wherein the matching algorithm is one of a template matching algorithm, a feature matching algorithm, and a color matching algorithm (the process definition may be a process for generating a template, including a template matching algorithm (¶ [0055], [0059]), a feature matching algorithm (a mouse cursor above a particular screen (grid) region to click; ¶ [0056]-[0057]) and a color matching algorithm (a different color in response to a click, ¶ [0056]).
Bataller et al does not teach implementing a pixel normalization algorithm for pixels of a screen of the mobile device observed using the computer vision algorithm; and the grid comprises normalized pixels.
Viet et al is analogous art pertinent to the technological problem addressed in this application and teaches implementing a pixel normalization algorithm for pixels of a screen of the mobile device observed using the computer vision algorithm (the robotic training system 102 includes a color analysis device 108 to normalize the color, array size or both, at the pixel level (row 130 and column 132), which can occur when the user uses the user interface 118 to select actions and the device 116 provides data to the robotic training system 102 (computer vision ¶ [0009]); Fig 1 and ¶ [0025]-[0027], [0047]-[0049]); and the grid (grid is not defined in regard to a size or number of pixel with respect to the image , specification ¶ [0024], and can therefore have either interpretation and could be the entire computer screen) (an image can be identified based on pixel dimension (a “grid”) entry arrays in the x and y axis direction; ¶ [0046], [0049]) comprises normalized pixels (an image can be identified based on pixel dimension (a “grid”) entry arrays in the x and y axis direction and the color analysis 108 of training system 108 may normalize the product; Fig 1 and ¶ [0046]-[0049]).
It would have been obvious to one of ordinary skill in the art, before the effective filing date of this application, to combine the teachings of Bataller et al a with Viet et al including implementing a pixel normalization algorithm for pixels of a screen of the mobile device observed using the computer vision algorithm; and the grid comprises normalized pixels. By normalizing pixels between images, an appropriate color and size analysis can be performed, such as evaluating values to a threshold amount and thereby more accurately detecting pixel clusters to identify shapes between images for robotic automation of processing based on computer vision techniques, as recognized by Viet et al (¶ [0009]-[0010]).
Regarding Claim 7, Bataller et al in view of Viet et al teach the computerized method of claim 1 (as described above), wherein the computer vision algorithm is implemented for one rectangle in a nine by nine (9X9) grid of interest (Viet et al, a region can be identified in the image, such that the array size can be cropped to a particular size (such as 9x9, with the example size given of 200 pixels by 200 pixels); ¶ [0047], [0049]).
Regarding Claim 8, Bataller et al in view of Viet et al teach the computerized method of claim 1 (as described above), wherein the computer vision algorithm is implemented for one rectangle in a single two hundred and fifty-five by two hundred and fifty-five (255x255) grid of interest (Viet et al, a region can be identified in the image, such that the array size can be cropped to a particular size (such as 255x255, with the example size given of 200 pixels by 200 pixels); ¶ [0047], [0049]).
Regarding Claim 9, Bataller et al in view of Viet et al teach the computerized method of claim 1 (as described above), wherein the grid of interest scales (scale is not defined and is interpreted as a selected pixel region within an image) from 9x9 to 255x255 (Viet et al, a region can be identified in the image, such that the array size can be cropped to a particular size (that includes a size between 9x9 and 255x255, with the example size given of 200 pixels by 200 pixels); ¶ [0047], [0049]).
Regarding Claim 17, Bataller et al teach a computer system for guiding a user on a graphical interface with computer vision (computer system 100 for automating a manual process 300 to create an automated system 200 executed by a robot 240; Fig 1-4 and ¶ [0017], [0038], [0054], [0062]) comprising:
a step-by-step guide for a mobile workflow (“mobile workflow” described broadly as “any human computer task to be automated with computer vision and robotic touch arm” specification ¶ [0002]-[0003], [0023]) (computer vision techniques are applied to identify activities of the user on a mobile device (¶ [0071]-[0073]) for a user manual process to be automated, S310-340, to create a process definition (a step-by-step guide for mobile workflow), S350, using process definition generator 140 (based on pipeline 110, 120, 130); Fig 1, 3 and ¶ [0017], [0031]-[0033], [0054]-[0059]);
a computer vision system running a computer vision algorithm (computer vision techniques are applied by system 100 to the identified activities manually performed by a user interacting with a computer (mobile device ¶ [0071]), and in particular, images of user activities are captured by the image capturer 110, S320, and identified by the activity identifier 120, S330, using computer vision techniques; Fig 1, 3 and ¶ [0021]-[0023], [0056]-[0057]);
a display, a processor, a memory, a storage, and one or more I/O devices wherein the computer system (computer system 400 (system 100) includes processor 410, memory 420, storage device 430 and I/O device 440 with display unit; Fig 4 and ¶ [0062]-[0066]) is configured to perform the steps of
producing a graphical interface on the display (the I/O display unit displays information for a user interface; ¶ [0035], [0063]);
observing with the computer vision algorithm, an area of touch interest for the mobile workflow (computer vision techniques are applied to the identified activities manually performed by a user interacting with a computer (mobile device ¶ [0071]), S310-S330, with images (observing) obtained by the image capturer 110; Fig 1, 3 and ¶ [0022]-[0023], [0055]-[0057]), wherein the mobile workflow includes one or more user input actions performed via the one or more I/O devices (the image capturer 110 (computer vision system) obtains images of a computer display that contains a touchscreen user interface, S320, and the images are analyzed by the activity identifier 120, S330; Fig 1, 3 and ¶ [0022]-[0023], [0056]-[0057]), wherein the one or more user input actions correspond to actions in the step-by-step guide (tasks to be automated are based on a user manual process; Fig 1, 3 and ¶ [0021]-[0023], [0056]);
determining a grid of interest (the touchscreen of the user interface is represented with a region of interest (“Yes” and “No” buttons representing a “grid” region based on pixel coordinates and the activity information generator 130 executes identification of the activity (touch) was performed in the region of interest, S330; Fig 1, 3 and ¶ [0032]-[0033], [0057]), wherein the grid comprises normalized pixels;
linking the mobile workflow on the graphical interface to a step-by-step guide (the activities for a given process definition are analyzed for multiple users to determine a particular similar procedure; ¶ [0034]-[0035], [0061]), wherein the mobile workflow comprises one or more tasks performed by a human user(tasks to be automated are based on a user manual process; Fig 1, 3 and ¶ [0021]-[0023], [0056]); and
automating the mobile workflow with the computer vision system and a robotic touch arm (the process definition for all of the activities are automated with movement instructions by the process definition generator 140, S350, so the given activity may be executed by the robot arm 240 via the process definition; Fig 1-3 and ¶ [0030]-[0031], [0051], [0059]-[0060]);
wherein the computer vision algorithm is implemented on one rectangle in the grid of interest (the touchscreen of the user interface is represented with a rectangle (“Yes” and “No” buttons represented as rectangle via pixel regions ¶ [0053]) to determine if a trigger event (touch) was performed by user; Fig 2, 3 and ¶ [0042]-[0043], [0059]),
wherein the step of linking includes a matching algorithm (the activity execution engine 230 executes activities based on process definition (matching algorithm) identified from the activity trigger engine 220; Fig 2, 3 and ¶ [0044], [0051], [0060]) that determines which of a template matching algorithm, a feature matching algorithm, and a color matching algorithm are used for any step of the step-by-step guide (the process definition may be a process for generating a template, including a template matching algorithm (¶ [0055], [0059]), a feature matching algorithm (a mouse cursor above a particular screen (grid) region to click; ¶ [0056]-[0057]) and a color matching algorithm (a different color in response to a click, ¶ [0056]), wherein the step of linking further includes linking each user input click with the step-by-step guide via the graphical interface (the activities for a given process definition are analyzed for multiple users to determine a particular similar procedure; ¶ [0034]-[0035], [0061]) while graphical user interface is being observed by computer vision system (the activities associated with the process (user input via graphic interface) are identified using computer vision techniques; ¶ [0033]-[0035], [0056]-[0057]).
Bataller et al does not teach the grid comprises normalized pixels.
Viet et al is analogous art pertinent to the technological problem addressed in this application and teaches the grid (grid is not defined in regard to a size or number of pixel with respect to the image, specification ¶ [0024], and can therefore have either interpretation and could be the entire computer screen) (an image can be identified based on pixel dimension (a “grid”) entry arrays in the x and y axis direction; ¶ [0046], [0049]) comprises normalized pixels (an image can be identified based on pixel dimension (a “grid”) entry arrays in the x and y axis direction and the color analysis 108 of training system 108 may normalize the product; Fig 1 and ¶ [0046]-[0049]).
It would have been obvious to one of ordinary skill in the art, before the effective filing date of this application, to combine the teachings of Bataller et al a with Viet et al including the grid comprises normalized pixels. By normalizing pixels between images, an appropriate color and size analysis can be performed, such as evaluating values to a threshold amount and thereby more accurately detecting pixel clusters to identify shapes between images for robotic automation of processing based on computer vision techniques, as recognized by Viet et al (¶ [0009]-[0010]).
Regarding Claim 18, Bataller et al in view of Viet et al teach the system of claim 17, wherein the grid of interest is a nine by nine (9x9) grid of interest (Viet et al, a region can be identified in the image, such that the array size can be cropped to a particular size (such as 9x9, with the example size given of 200 pixels by 200 pixels); ¶ [0047], [0049]).
Regarding Claim 19, Bataller et al in view of Viet et al teach the system of claim 17 (as described above), wherein the grid of interest is a (255x255) grid of interest (Viet et al, a region can be identified in the image, such that the array size can be cropped to a particular size (that includes a size between 9x9 and 255x255, with the example size given of 200 pixels by 200 pixels); ¶ [0047], [0049]).
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to KATHLEEN M BROUGHTON whose telephone number is (571)270-7380. The examiner can normally be reached Monday-Friday 8:00-5:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, John Villecco can be reached at (571) 272-7319. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/KATHLEEN M BROUGHTON/Primary Examiner, Art Unit 2661