DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on July 13, 2026 has been entered.
Response to Amendment
Receipt is acknowledged of claim amendments with associated arguments/remarks, received July 13, 2026. Claims 1, 7-9, 14-19 are pending of which claim 14 was amended. Claims 2-6, 10-13, 20 were previously cancelled.
Response to Arguments
Applicant’s arguments, see Remarks, pg 6, filed 07/13/2026, with respect to the objection of claim 14 for a minor objection has been considered and based on the amendment is persuasive. Therefore, the objection is withdrawn.
Applicant’s arguments, see Remarks, pg 6-19, filed 07/13/2026, with respect to the rejections of claim 1, 7-9, 14-19 under 35 U.S.C. §§ 102, 103 has been fully considered. Respectfully, the arguments are not persuasive. Applicant's arguments fail to comply with 37 CFR 1.111(b) because they amount to a general allegation that the claims define a patentable invention without specifically pointing out how the specification supports the claim language and patentably distinguish the claimed invention from the references.
Regarding Claim 1, 14, 17, Applicant argues the prior art does not disclose “selecting a matching algorithm to use for a given step of the mobile workflow, wherein the matching algorithm is one of a template matching algorithm, a feature matching algorithm, and a color matching algorithm”
The full description for the matching algorithm provided in specification is:
[0024] In step 106, process 100 can implement logic for matching algorithms. A master matching algorithm can be implemented to determine which algorithm to use for a given step (e.g. template matching, feature matching, color matching, etc.).
Applicant argues the prior art “does not support disclosure of any named matching algorithm” (Remarks – 07/13/2026 pg 7-10). In the remarks, the applicant argues no algorithm is taught and provides detailed explanation as to what the applicant believes is the disclosure provided as compared to the citation of the prior art. Respectfully, the examiner is not persuaded.
The specification does not disclose details as the applicant argues are taught. The applicant argues the interpretation of the disclosure. According to the MPEP § 608.01(h) regarding mode of operation of an invention, “If the best mode contemplated by the inventor at the time of filing the application is not disclosed, such defect cannot be cured by submitting an amendment seeking to put into the specification something required to be there when the application was originally filed. In re Hay, 534 F.2d 917, 189 USPQ 790 (CCPA 1976). Any proposed amendment of this type should be treated as new matter.”
The applicant argues details as to the intent of the interpretation of the claim language but does so with hindsight insight (Remarks – 07/13/2026, pg 7-10). At the time of the disclosure, the applicant submitted paragraph [0024] as disclosure. The written description has been previously called into question by the examiner for lack of detail to demonstrate the inventor had possession of the invention at the time of filing (most recently Non-Final Rejection – 06/04/2025). The applicant provides details as to how the applicant intends for the claim limitations to be interpreted but did not provide any such detail in the original filing. It is unclear as to what point in time the invention has determined the intent of the interpretation of the claim limitations as argued in the Remarks – 07/13/2026 but no such disclosure was originally filed – 06/30/2021. Applicant’s remarks as to how to interpret the algorithms are longer than the original disclosure. The applicant provides no citations to serve as evidence to support the applicant’s arguments that the applicant’s invention is distinct from the prior art.
Pursuant to MPEP § 2106(II) patent subject matter eligibility, claims are given the broadest reasonable interpretation (BRI) for examining a claim for eligibility and claim interpretation is determined in light of the specification. In this case, the applicant provides no detail in the original disclosure. The drawings provided no additional guidance (see, for example, original disclosure drawings – 06/30/2021 pg 6 which displays a box around a face to detail “what is computer vision” and pg 7 with generic imagery of binary code for “how it really works?”). In this case, the examiner cited prior art that teaches on the claim limitations based on the original disclosure using the BRI. After further consideration during careful reexamination of the application, it has been determined the application lacks the necessary written disclosure to demonstrate possession of the invention at the time of filing under 35 U.S.C. § 112(a).
Regarding the prior art Bataller et al (US 2017/0001308), the invention is directed to automate a manual human process using computer vision techniques so the robot may automatically perform interactions associated with the determined activities (¶ [0004]). For the robot to automatically perform the given task, the activities and activity information performed by a user are obtained using computer vision techniques (¶ [0005], [0009], [0022]-[0029]).
Although applicant argues a template matching algorithm is not taught by the prior art, no details as to the interpretation of this limitation are disclosed by applicant at the time of filing the application. Applicant argues a “stored image template is compared against a target image to locate the template’s pattern” (pg 8, ln 6-7) and argues a document template is not the correct interpretation. The prior art disclosure describes the same concept as the process definition is to identify activity information associated with an activity with using image snapshot portions of a display screen of a user before a particular activity and after a given activity to generate a procedure for the robot to automate the process of the user (see citations pertaining to the activity information generator 130, cited in at least ¶ [0025]-[0026], [0030]-[0033], [0055]-[0057]). This appears to be the same conceptual idea as applicant claims is the meaning of the template matching. Bataller et al additionally teaches “a process for generating a template e-mail” (¶ [0059]), which easily reads upon the broad language of the specification of “template matching” (applicant specification ¶ [0024]). Furthermore, in generating a template e-mail the same automation process with the snapshot processing of the user to generate the activity information and generating a process definition using computer vision techniques is taught by the prior art. The specification provided no substantive detail regarding the interpretation and the prior art teaches the limitation. Applicant is not persuasive.
Applicant further argues a feature matching algorithm is not taught by the prior art. Again, no detail is provided in the specification to support the applicant’s interpretation is not equivalent to the teachings of Bataller et al and the applicant claims limitations are in the claim language which are not. Specifically, the applicant argues “no keypoints are extracted, no feature descriptors are computed, and no cross-image feature correspondence is established” (Remarks, pg 9 ln 1-2). However, the applicant does not claim these limitations and the specification does not provide evidence to support these features. Furthermore, the limitations were not included in the original claim language. The claim is generic with “feature matching” and the specification provides no guidance. The prior art describes using the images to identify activities, such as a mouse cursor above a particular region (button on a computer screen), with the button the same “feature” (¶ [0057]). Applicant is not persuasive.
Applicant argues a color matching algorithm is not taught by the prior art and applicant again relies upon features not disclosed in the specification or claimed in the claim, specifically the use of a color histogram, a spatial correspondence and a classification process (Remarks pg 10 ln 18-20). Color matching taught by the prior art is equivalent as the processing to automate the user activities includes data to demonstrate “a different color” in response to the user activity before and after the activity (a left mouse click; ¶ [0056]). Applicant is not persuasive.
Applicant further argues the prior art does not teach “the Robotic Touch Arm Interacting with the Graphical User Interface” (Remarks pg 10 – 11) and appears to be arguing a real-world robot is performing a function. The applicant does not provide sufficient support to the claim interpretation. The extent of the applicant’s disclosure is:
[0023]… In step 104, process 100 can provide the ability to automate mobile workflows with computer vision and a robotic touch arm.
There is no disclosure provided by the applicant in the original disclosure, including the specification, drawings and original claims to support the applicant’s interpretation this is distinct in the applicant’s disclosure from the prior art. The prior art teaches the movement instructions for the robot 240 are generated for the robot to execute the given touch activity (example ¶ [0051]) and if the process definition executed by the robot does not match the intended execution of the given activities, a notification is provided to an administrator or new instructions (¶ [0051]). This is equivalent to the applicant’s argued intent of the limitation to “observes the robotic touch arm interacting with the graphical user interface”. Applicant is not persuasive.
Applicant further argues the “grid of interest” is not taught by the prior art as the applicant argues a single pixel coordinate is taught by the prior art pixel region whereas the applicant claims a grid as “a structured spatial decomposition of the screen into a plurality of rectangular cells” (Remarks pg 12 ln 16-18). Applicant’s disclosure is:
[0025] In step 108, process 100 can implement algorithms for pixel normalization and grid of interest. For example, process can implement computer vision for one rectangle in a 9X9 grid or one rectangle in a single 255 by 255 grid, etc. In one example, the grid can scale from 9 by 9 to 255 by 255.
The prior art refers to a rectangular region of a button represented in a pixel region (¶ [0053]). The applicant’s interpretation of “coordinates” is misplaced as the prior art is describing a region with coordinates and not a single “point”.
The prior art, Viet et al (US 2020/0059441 Robotic Process Automation) pertains to “a robotic training system [that] can use computer vision and user interface component analysis to identify a user interface control, e.g., icon, selected during a process that the robotic training system is automating.” (¶ [0003]). In enabling the robotic training system 102 to more easily identify image data based on the automation script from the user interface data (Fig 1 and ¶ [0024]), a number of modules (104, 106, 108, 110, 112) are implemented to analyze the image data as it pertains to an event location (a user interaction) associated with the user interface 118 (¶ [0020]-[0024], [0032]).
Processing of the user interface 118 data for automation includes determining a difference image 126, based on an image difference generator 106, which “generator 106 may use a color analysis device 108 during creation of the difference image 126” and “the color analysis device 108 can determine a quantity of pixels depicted in some of the images 124a-d” (¶ [0040]). The “color analysis device 108 can determine a row 130 and a column 132 in the difference image 126 that intersect with the event location 128. The row 130, the column 132, or both, can have a dimension of one pixel across the corresponding length or a dimension of multiple pixels across the corresponding length.” (¶ [0041]). The “color analysis device 108…to determine pixels in the difference image 126…enable the robotic training system 102 to more easily identify shapes depicted in the difference image 126” (¶ [0044], Fig 1). The color analysis device 108 is further stated “to use any appropriate color scale, array sizes, or both” in determining the image location based on the event location from the mouse click (¶ [0047]).
The size of the difference image 126 is further analyzed by a binary image generator 110 that may “crop a size of the difference image” with an explicit example of “a cropped version of the difference image 126, may have a size of 200 pixels by 200 pixels”, and “in some examples, another component of the robotic training system 102 may crop the difference image 126 or create a smaller version of the difference image 126” (¶ [0049]).
Both the color analysis device 108 and the binary image generator 110 are processing image data based on the user interface data for a robotic training system towards generating an automation script (broadly described by applicant as a “computer vision algorithm”) with prior art teaching both the color analysis device 108 and the binary image generator 110 analyzing the user interface data on the image in an appropriate array (“grid of interest”) size.
Applicant is not persuasive.
Applicant further argues the prior art does not teach “the matching algorithm determines one or more algorithms to use for any step of the step-by-step guide” (Remarks pg 12-13), with the disclosure of specification ¶ [0024] (see above for full applicant disclosure). The prior art describes the methodology of generating the process definition, which includes multiple algorithms for a robot to perform the given activity with the instructions including what activity to execute (“process definition may indicate the one or more identified activities and, for each identified activity, indicate the activity information associated with the activity. . . . The process definition may indicate the sequence for the one or more identified activities. For example, the process definition may indicate that a screen touch is first performed, then another screen touch, and then a key press.” ¶ [0032]). The art teaches the process definition indicates the sequence for the activities, which would include the algorithm to perform each screen touch activity. The applicant appears to be arguing the applicant’s claim language and disclosure is based on dynamic alternative algorithms using metadata. However, the applicant’s claim language does not claim such detail and the specification does not support the applicant’s argument. Applicant is not persuasive.
Applicant argues “a template matching algorithm, a feature matching algorithm, and a color matching algorithm” are not taught in dependent claim 16 based on disjunctive (claim 14) versus conjunctive language (claim 16). Although the applicant appears mistaken regarding the claim language of claim 14 which recites “and” the algorithm “is selected from one of” with each of the algorithms taught by the prior art (as discussed above). Applicant is not persuasive. Furthermore, after careful review of the claim 16, it has been determined claim 16 is indefinite for lack of clarity as to whether or not the algorithms are the same algorithms as claim 14 or different algorithms and is rejected under 35 U.S.C. § 112(b).
Applicant argues the second prior art reference Viet et al (US 2020/0059441) does not teach to implement a pixel normalizing algorithm and does not teach that the grid comprises normalized pixels (Remarks – 07/13/2026 pg 14-19).
Applicant’s disclosure for the teaching is specification ¶ [0024]-[0025] (recited above). The examiner notes the applicant’s interpretation of Viet is longer than the actual disclosure provided by the applicant regarding the limitation and the applicant provides no citation to the applicant’s own disclosure to support the claim in how the applicant believes the claim should be interpreted. The applicant did not provide any notice as to how the applicant’s claim is distinct from the prior art and supported by the applicant’s specification. Rather, the applicant states that “these are fundamentally different operations performed in fundamentally different contexts” (Remarks – 07/13/2026, pg 17 ln 16-17). The examiner interpreted the claim limitation using a broadest reasonable interpretation. As noted in the rejections (see Final Rejection – 02/13/2026 pg 10) the grid is not defined in regard to a size or a pixel number and could have either interpretation and, if interpreted as a relative term such as pixel dimension, can span an entire computer screen (understood to be with lower resolution). The cited prior art Viet et al is directed to robotic process automation (“to perform during a robotic automated process and includes, for one of the operations, a reference to the cropped image to enable the other system to identify the control using computer vision” ¶ [0009]). Viet further states that that the computer vision processing includes the training system with the color analysis device 108 to performed image processing analysis on a region of interest in the image with 256 locations on the array based on an 8-bit color scale and “may use any appropriated color scale, array sizes or both” with the region of interest including a normalization process to the pixel color (¶ [0047]). The claim language broadly recites “the grip comprises normalized pixels” and there is no context as to what is normalized. The specification provides no guidance as to what algorithm is implemented towards what is normalized within the pixel data. Applicant is not persuasive.
Examiner has carefully reviewed applicant’s disclosure and associated claims. Examiner maintains patentable subject matter is not identified in the application. As discussed above, a careful review of the applicant’s disclosure and claim language required reexamination of the application and updated rejections are provided below. Rejections are updated below based on applicant’s arguments and further review of the disclosure in light of the prior art. Respectfully, the examiner is not persuaded by the applicant’s arguments and the examiner did not identify patentable subject matter in the disclosure.
All arguments were addressed.
Claim Rejections - 35 USC § 112(a) – written description
The following is a quotation of the first paragraph of 35 U.S.C. 112(a):
(a) IN GENERAL.—The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor or joint inventor of carrying out the invention.
The following is a quotation of the first paragraph of pre-AIA 35 U.S.C. 112:
The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor of carrying out his invention.
Claims 1, 7-9, 14-19 are rejected under 35 U.S.C. 112(a) or 35 U.S.C. 112 (pre-AIA ), first paragraph, as failing to comply with the written description requirement. Claims 1, 14, and 17 recite at least the following limitations lacking written description: “optimizing a workflow guide”; “linking each user input click”; “implementing a pixel normalization algorithm”; “guiding a user’s interaction”; “selecting a matching algorithm”; “template matching algorithm”; “feature matching algorithm” and “color matching algorithm”. Each of these limitations fail to comply with the written description requirement. Claims 7-9 are rejected as dependent on claim 1. Claims 15-16 are rejected as dependent on claim 14. Claims 18-19 are rejected as dependent on claim 17. The claims each contain subject matter which was not described in the specification in such a way as to reasonably convey to one skilled in the relevant art that the inventor or a joint inventor, or for pre-AIA the inventor(s), at the time the application was filed, had possession of the claimed invention.
The written description standard under 35 U.S.C. § 112(a) requires that the “specification shall contain a written description of the invention”. To satisfy the written description requirement, a patent specification must describe the claimed invention in sufficient detail that one skilled in the art can reasonably conclude that the inventor had possession of the claimed invention. See, e.g., Moba, B.V. v. Diamond Automation, Inc., 325 F.3d 1306, 1319, 66 USPQ2d 1429, 1438 (Fed. Cir. 2003); Vas-Cath, Inc. v. Mahurkar, 935 F.2d at 1563, 19 USPQ2d at 1116. An applicant shows possession of the claimed invention by describing the claimed invention with all of its limitations using such descriptive means as words, structures, figures, diagrams, and formulas that fully set forth the claimed invention. Lockwood v. Amer. Airlines, Inc., 107 F.3d 1565, 1572, 41 USPQ2d 1961, 1966 (Fed. Cir. 1997). Possession may be shown in a variety of ways including description of an actual reduction to practice, or by showing that the invention was “ready for patenting” such as by the disclosure of drawings or structural chemical formulas that show that the invention was complete, or by describing distinguishing identifying characteristics sufficient to show that the applicant was in possession of the claimed invention. See, e.g., Pfaff v. Wells Elecs., Inc., 525 U.S. 55, 68, 119 S.Ct. 304, 312, 48 USPQ2d 1641, 1647 (1998); Eli Lilly, 119 F.3d at 1568, 43 USPQ2d at 1406; Amgen, Inc. v. Chugai Pharm., 927 F.2d 1200, 1206, 18 USPQ2d 1016, 1021 (Fed. Cir. 1991).
There is a presumption that an adequate written description of the claimed invention is present when the application is filed. In re Wertheim, 541 F.2d 257, 263, 191 USPQ 90, 97 (CCPA 1976) (“we are of the opinion that the PTO has the initial burden of presenting evidence or reasons why persons skilled in the art would not recognize in the disclosure a description of the invention defined by the claims”). However, as discussed in subsection I., supra, the issue of a lack of adequate written description may arise even for an original claim when an aspect of the claimed invention has not been described with sufficient particularity such that one skilled in the art would recognize that the applicant had possession of the claimed invention. The claimed invention as a whole may not be adequately described if the claims require an essential or critical feature which is not adequately described in the specification and which is not conventional in the art or known to one of ordinary skill in the art (emphasis). While it is not necessary for the examiner to present factual evidence, to make a prima facie case it is necessary to point out the claim limitations that are not adequately supported and explain any other reasons that the claim is not fully supported by the disclosure to show that the inventor had possession of the invention. See for example, Hyatt v. Dudas, 492 F.3d 1365, 1371, 83 USPQ2d 1373, 1376-1377 (Fed. Cir. 2007).
The courts have described the essential question to be addressed in a description requirement issue in a variety of ways. An objective standard for determining compliance with the written description requirement is, “does the description clearly allow persons of ordinary skill in the art to recognize that he or she invented what is claimed.” In re Gosteli, 872 F.2d 1008, 1012, 10 USPQ2d 1614, 1618 (Fed. Cir. 1989). Under Vas-Cath, Inc.v. Mahurkar, 935 F.2d 1555, 1563-64, 19 USPQ2d 1111, 1117 (Fed. Cir. 1991), to satisfy the written description requirement, an applicant must convey with reasonable clarity to those skilled in the art that, as of the filing date sought, he or she was in possession of the invention, and that the invention, in that context, is whatever is now claimed. The test for sufficiency of support in a parent application is whether the disclosure of the application relied upon “reasonably conveys to the artisan that the inventor had possession at that time of the later claimed subject matter.” Ralston Purina Co.v.Far-Mar-Co., Inc., 772 F.2d 1570, 1575, 227 USPQ 177, 179 (Fed. Cir. 1985) (quoting In re Kaslow, 707 F.2d 1366, 1375, 217 USPQ 1089, 1096 (Fed. Cir. 1983)). See MPEP§ 2163.
Regarding Claims 1, 14, 17, with dependent claims 7-9, 15-16, and 18-19, respectively, the applicant claims multiple limitations that, upon careful review of applicant’s R.C.E. and associated arguments, the examiner has thoughtfully determined the applicant’s specification does not meet the minimum threshold to satisfy the written description for, at least, the following limitations:
“optimizing a workflow guide” – no steps or details were provided in the specification as to what a workflow guide is, how it is applicable to the process, what steps are included in the guide, if there is a different guide for the “step-by-step guide” from “a mobile workflow” or from an “automated mobile workflow” and therefore there is no clear description for how a “workflow guide” may be optimized. The terminology is broad and there is no common knowledge of such optimization known in the art of image processing. No drawings were provided to support a description for what the workflow guide is and there are no meaningful drawings provided to sufficiently describe how such guide would be optimized.
“linking each user input click” – no steps or details were provided in the specification as to how a user input click would be linked with a “step-by-step guide” and as discussed, there are no details as to what is the step-by-step guide is, and it is not disclosed as to what steps are included in the guide, if there is a different guide for the “step-by-step guide” from “a mobile workflow” or from an “automated mobile workflow” and therefore there is no clear description for how linking the user input click to the guide is accomplished. The terminology is broad and there is no common knowledge of such optimization known in the art of image processing. No drawings were provided to support a description for linking the guide is and there are no meaningful drawings provided to sufficiently describe how such linking would be performed.
“implementing a pixel normalization algorithm” – no steps or details were provided in the specification as to what the pixel normalization algorithm is and no details are provided as to how such algorithm would be implemented. Concerns regarding the disclosure regarding the claimed algorithms were discussed at length in response to the arguments above. There are no details as to how an analysis is performed – is it based on statistical analysis or on computational speed or on number of pixels. Is data output based on a probability or a binary match. How many matches can be made? It is not clear what the normalization process is or how it is implemented. There is no standardization to this process in the field of image processing and therefore it cannot be derived without clear details from the application, which was not provided in the original disclosure.
“guiding a user’s interaction” – no steps or details were provided in the specification regarding the step-by-step guide or the workflow guide and there are no details regarding how a user would by guided in the process, what steps are included in the guide, if there is a different guide for the “step-by-step guide” from “a mobile workflow” or from an “automated mobile workflow” and therefore there is no clear description for how “guiding a user’s interaction” would be performed. The terminology is broad and there is no common knowledge of how guiding a user’s interaction is performed in the art of image processing. No drawings were provided to support a description for guiding a user’s interaction and there are no meaningful drawings provided to sufficiently describe how such guide is performed.
“selecting a matching algorithm” – no steps or details were provided in the specification as to what the matching algorithm is and no details are provided as to how such algorithm would be implemented. Concerns regarding the disclosure regarding the claimed algorithms were discussed at length in response to the arguments above and were discussed in previous office actions (see for example Non-Final Rejection – 03/07/2024, 06/04/2025, Final Rejection – 11/27/2024). There are no details as to what the matching algorithm is or how the algorithm is performed – is it based on statistical analysis or on computational speed or on number of pixels. Is data output based on a probability or a binary match. How many matches can be made? It is not clear what the matching process is or how it is implemented. There is no standardization to this process in the field of image processing and therefore it cannot be derived without clear details from the application, which was not provided in the original disclosure.
“template matching algorithm” – no steps or details were provided in the specification as to what the template matching algorithm is and no details are provided as to how such algorithm would be implemented. Concerns regarding the disclosure regarding the claimed algorithms were discussed at length in response to the arguments above and were discussed in previous office actions (see for example Non-Final Rejection – 03/07/2024, 06/04/2025, Final Rejection – 11/27/2024). There are no details as to what the matching algorithm is or how the algorithm is performed – is it based on statistical analysis or on computational speed or on number of pixels. Is data output based on a probability or a binary match. How many matches can be made? It is not clear what the matching process is or how it is implemented. There is no standardization to this process in the field of image processing and therefore it cannot be derived without clear details from the application, which was not provided in the original disclosure.
“feature matching algorithm” – no steps or details were provided in the specification as to what the feature matching algorithm is and no details are provided as to how such algorithm would be implemented. Concerns regarding the disclosure regarding the claimed algorithms were discussed at length in response to the arguments above and were discussed in previous office actions (see for example Non-Final Rejection – 03/07/2024, 06/04/2025, Final Rejection – 11/27/2024). There are no details as to what the matching algorithm is or how the algorithm is performed – is it based on statistical analysis or on computational speed or on number of pixels. Is data output based on a probability or a binary match. How many matches can be made? It is not clear what the matching process is or how it is implemented. There is no standardization to this process in the field of image processing and therefore it cannot be derived without clear details from the application, which was not provided in the original disclosure.
“color matching algorithm” – no steps or details were provided in the specification as to what the color matching algorithm is and no details are provided as to how such algorithm would be implemented. Concerns regarding the disclosure regarding the claimed algorithms were discussed at length in response to the arguments above and were discussed in previous office actions (see for example Non-Final Rejection – 03/07/2024, 06/04/2025, Final Rejection – 11/27/2024). There are no details as to what the matching algorithm is or how the algorithm is performed – is it based on statistical analysis or on computational speed or on number of pixels. Is data output based on a probability or a binary match. How many matches can be made? It is not clear what the matching process is or how it is implemented. There is no standardization to this process in the field of image processing and therefore it cannot be derived without clear details from the application, which was not provided in the original disclosure.
Claim Rejections - 35 USC § 112, enablement
The following is a quotation of the first paragraph of 35 U.S.C. 112(a):
(a) IN GENERAL.—The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor or joint inventor of carrying out the invention.
The following is a quotation of the first paragraph of pre-AIA 35 U.S.C. 112:
The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor of carrying out his invention.
Claims 1, 7-9, 14-19 are rejected under 35 U.S.C. 112(a) or 35 U.S.C. 112 (pre-AIA ), first paragraph, as failing to comply with the enablement requirement. The claims contain subject matter which was not described in the specification in such a way as to enable one skilled in the art to which it pertains, or with which it is most nearly connected, to make and/or use the invention.
Regarding Claims 1, 14, 17 with dependent claims 7-9, 15-16, 18-19 respectively, the applicant claims a “matching algorithm” that is described in the specification such that “a master matching algorithm can be implemented to determine which algorithm to use for a given step (e.g. template matching, feature matching, color matching, etc.)” (specification ¶ [0024]). After careful consideration, It is the Examiner’s position that the subject matter described above is not described in the specification in such a way as to enable one skilled in the art to which it pertains, to make and use the invention, without undue experimentation.
In accordance with MPEP § 2164, the examiner has the initial burden of establishing a prima facie case of lack of enablement. The question posed when making a lack of enablement rejection is: Is the experimentation needed to practice the invention undue or unreasonable? See Mineral Separation v. Hyde, 242 U.S. 261, 270 (1916). The test for lack of enablement was established in In re Wands, 858 F.2d 731, 737, 8 USPQ2d 1400, 1404 (Fed. Cir. 1988) and set forth several factors which must be considered by the examiner when making a determination of lack of enablement. These factors can be found in MPEP § 2164.01(a). Furthermore, the examiner need not discuss every factor. The examiner need only to focus on those factors, reasons, and evidence that lead the examiner to conclude that the specification fails to teach how to make and use the claimed invention without undue experimentation.
In Re Wands Factors
A) The breadth of the claims
Claims 1, 14, 17 recite a “matching algorithm” in the limitation “selecting a matching algorithm to use for a given step of the mobile workflow” of claim 1 or equivalents thereof in claim 14, 17. In the examiner’s opinion, the breadth of each claim is based on broadly claimed element including “a matching algorithm” that is only referred to once in the specification (“A master matching algorithm can be implemented to determine which algorithm to use for a given step” ¶ [0024]), without further explanation (as discussed above in the written description rejection).
B) The nature of the invention
The invention of the aforementioned claims 1 and 14 are both directed towards “a method for guiding a user on a graphical interface with computer vision” wherein most limitations require the examiner to present an interpretation due to the lack of written description.
C) The state of the prior art
The examiner carefully reviewed the claimed invention and interpreted the invention as a process associated with “robotic process automation” as described by cited prior art, including Ma et al (US 2020/0206920). Processes associated with robotic process automation are recognized as a term of art. However the specific “matching algorithm” as claimed by applicant is not a term known in the art. The applicant did not disclose a written description to adequately provide details to explain what a “matching algorithm” is or how it is performed for the examiner to sufficiently determine the state of the prior art or to clearly convey adequate notice to the public the limitations of the claimed invention.
D) The level of one of ordinary skill in the art
The examiner is of the opinion that it is well known to one of ordinary skill in the art that robotic process automation is known in the art; it is also the examiner’s opinion that “computer vision” is a known term in the art. However, the Examiner's position on claims 1, 7-9, and 14-17 is that it is not well known to one of ordinary skill in the art to know what a “matching algorithm” is, how it is performed, what the criteria is to produce results and how the results are integrated in performing additional steps within the independent claim. Applicant has not described the invention is sufficient detail for one of ordinary skill in the art to ascertain how applicant’s invention is carried out, which is discussed above in the response to the arguments and the written description rejections above; one of ordinary skill in the art would have trouble understanding applicant’s invention based on applicant’s disclosure and all interpretations are considered speculative.
F) The amount of direction provided by the inventor
As discussed above in the rejection under 35 U.S.C. 112(a), written description, applicant has not provided details on the aforementioned claim limitations. The specification merely describes a “master matching algorithm” with a result such that “a master matching algorithm can be implemented to determine which algorithm to use for a given step (e.g. template matching, feature matching, color matching, etc.)” (specification ¶ [0024]). No detail is provided in what the “matching algorithm” is, how it is derived or how it is implemented.
No support is disclosed in the specification as to how the algorithm is used in the context of the claim language. More specifically, applicant does not disclose in any particular detail how the “the master matching algorithm determines one or more algorithms to use for any step of the step-by-step guide” as claimed by applicant. In the Examiner’s opinion, there appears to be a lack of understanding as to what a “master matching algorithm” is and how it is applied to generate the results as claimed.
G) The existence of working examples
There is neither mention of a working example, nor any example in any of the prior art. Applicant discusses the applicant’s interpretation in recent Remarks (07/13/2026), while the applicant desires to simultaneously claim a priority date from a provisional application of 07/14/2020.
H) The quantity of experimentation needed based on the disclosure
Since the invention as claimed is not described in detail in the specification, the amount of experimentation would be great in order to make/use the invention. As mentioned previously, applicant has not provided any direction on the above mentioned claim limitations. Applicant merely discloses these limitations in a single sentence representing a paragraph of the specification and does not provide any additional detail. Thus, one of ordinary skill in the art would have to engage in undue experimentation in order to figure out how to create the claimed invention. Therefore, since the specification provides no detail on how these claim limitations are made and used, the disclosure is non-enabling. See MPEP § 2164.06.
When considering all of the pertinent In re Wands factors, 858 F.2d 731, 737, 8 USPQ2d 1400, 1404 (Fed. Cir. 1988), the Examiner has reached the conclusion that one of ordinary skill in the art would not be enabled to make and/or use the claimed invention without undue experimentation, particularly since the amount of direction provided by the applicant is minimal. Furthermore, because the terminology “matching algorithm” is not recognized terminology to one of ordinary skill in the art and is not sufficiently described in the specification, the claim language is arguably not interpretable to properly conduct a search and identify potential prior art under 35 USC § 102 or 35 USC § 103. The applicant is required to specifically point out in the specification how the interpretation of the claim language is intended should it be different than how the examiner interpreted the claims.
Claim Rejections - 35 USC § 112(b)
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 7-9, 16 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claims 7-9, 18, 19 each recite limitations associated with a grid of interest associated with a dimension (in example, “nine by nine (9x9)”). However, the claim limitation does not define the units in the measurement of the grid dimension and the specification does not provide adequate guidance as to the interpretation. The dimension could be interpreted as either a length (micron, inch etc.) or a relative size (pixel). The specification does not provide guidance as to the intended interpretation (see Specification ¶ [0025]). As evidence that it would be unclear to one of ordinary skill in the art to definitely know which interpretation is meant by the applicant, examiner has included as evidence the reference Kim et al (Adaptive planar vision marker composed of LED arrays for sensing under low visibility), which discloses the use of LED arrays for computer vision techniques in robotics to visually detect a given marker and measure dimensions of the markers and the distance from the marker. The disclosure discusses both the use of size in terms of a pixel dimension and a metric dimension see for example, 2.1 Image processing algorithms, which discusses size by pixel dimension and 2.2 Experiments under the low visibility conditions, which discusses size by metric dimension. It is therefore unclear in any of claims 7-9, 18, 19 how to interpret the dimensional measurement for adequate notice of the metes and bounds of the claim limitation. Thus, Applicant has failed to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor regards as the invention.
Claim 16 is rejected for the limitation “a template matching algorithm, a feature matching algorithm, and a color matching algorithm” while the base claim in which it depends, claim 14 recites “a template matching algorithm, a feature matching algorithm, and a color matching algorithm.” It therefore is unclear if the algorithms of claim 16 are different algorithms or the same algorithms as claim 14. Thus, Applicant has failed to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor regards as the invention.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claims 14-16 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Bataller et al (US 2017/0001308, previously cited in Non-Final Rejection - 06/04/2025, Final Rejection - 02/13/2026).
Regarding Claim 14, Bataller et al teach a method for guiding a user on a graphical interface with computer vision (computer system 100 with computer vision for automating a manual process 300 to create an automated system 200 executed by a robot 240; Fig 1-4 and ¶ [0017], [0038], [0054], [0062]) comprising:
providing a step-by-step guide for a mobile workflow (“mobile workflow” described broadly as “any human computer task to be automated with computer vision and robotic touch arm” specification ¶ [0002]-[0003], [0023]) (computer vision techniques are applied to identify activities of the user on a mobile device (¶ [0071]-[0073]) for a user manual process to be automated, S310-340, to create a process definition (a step-by-step guide for mobile workflow), S350, using process definition generator 140 (based on pipeline 110, 120, 130); Fig 1, 3 and ¶ [0017], [0031]-[0033], [0054]-[0059]);
observing a mobile device with a graphical user interface with a computer vision system (computer vision techniques are applied to the identified activities manually performed by a user interacting with a computer (mobile device ¶ [0071]), S310-S330, with images (observing) obtained by the image capturer 110; Fig 1, 3 and ¶ [0022]-[0023], [0055]-[0057]),
wherein the computer vision system is configured to observe an area of user touch interest at a create time for the mobile workflow (the image capturer 110 (computer vision system) obtains images of a computer display that contains a touchscreen user interface, S320 and the images are analyzed by the activity identifier 120, S330; Fig 1, 3 and ¶ [0022]-[0023], [0056]-[0057]);
running a computer vision algorithm in communication with the computer vision system (images of user activities are captured by the image capturer 110, S320, and identified by the activity identifier 120, S330, using computer vision techniques ; Fig 1, 3 and ¶ [0021]-[0023], [0056]-[0057]);
finding, with the computer vision system, the area of user touch interest at play back time (the activity information generator 130 identifies the image data representing the user screen touch activity, such as the location of the physical screen touch that user touched to indicate a response, found in the image capturer 110 images, S340; Fig 1, 3 and ¶ [0025]-[0026], [0058]), wherein the computer vision algorithm is implemented on one rectangle in a grid of interest (the touchscreen of the user interface is represented with a rectangle (“Yes” and “No” buttons represented as rectangle via pixel regions ¶ [0053]) based on pixel coordinates and the activity information generator 130 executes identification of the activity (touch) was performed in the region of interest, S330; Fig 1, 3 and ¶ [0032]-[0033], [0057]);
selecting a matching algorithm to use for a given step of the method (the activity execution engine 230 executes activities based on process definition (matching algorithm) identified from the activity trigger engine 220; Fig 2, 3 and ¶ [0044], [0051], [0060]), wherein the matching algorithm is selected from one of a template matching algorithm, a feature matching algorithm, and a color matching algorithm (the process definition may be a process for generating a template, including a template matching algorithm (¶ [0055], [0059]), a feature matching algorithm (a mouse cursor above a particular screen (grid) region to click; ¶ [0056]-[0057]) and a color matching algorithm (a different color in response to a click, ¶ [0056]);
linking each user input click with the step-by-step guide via the graphical interface (the activities for a given process definition are analyzed for multiple users to determine a particular similar procedure; ¶ [0034]-[0035], [0061]) while graphical user interface is being observed by computer vision system (the activities associated with the process (user input via graphic interface) are identified using computer vision techniques; ¶ [0033]-[0035], [0056]-[0057]),
wherein the linking involves the selected matching algorithm (the activity execution engine 230 executes activities based on process definition (matching algorithm) identified from the activity trigger engine 220; Fig 2, 3 and ¶ [0044], [0051], [0060]); and
automating the mobile workflow with the computer vision system and a robotic touch arm (the process definition for all of the activities are automated with movement instructions by the process definition generator 140, S350, so the given activity may be executed by the robot arm 240 via the process definition; Fig 1-3 and ¶ [0030]-[0031], [0051], [0059]-[0060]), wherein the computer vision system observes the robotic touch arm interacting with the graphical user interface (the activity execution engine 230 may execute instructions for the robot 240 to perform the activity (execute the graphic interface activity); Fig 2, 3 and ¶ [0046]-[0047], [0051], [0059]-[0060]).
Regarding Claim 15, Bataller et al teach method of claim 14 (as described above), wherein the matching algorithm determines one or more algorithms to use for any step of the step-by-step guide (the activity execution engine 230 executes activities based on process definition (matching algorithm) identified from the activity trigger engine 220; Fig 2, 3 and ¶ [0044], [0051], [0060]).
Regarding Claim 16, Bataller et al teach method of claim 15 (as described above), wherein the one or more algorithms include a template matching algorithm, a feature matching algorithm, and a color matching algorithm (the process definition (matching algorithm) may be a process for generating a template, including a template matching algorithm (¶ [0055], [0059]), a feature matching algorithm (a mouse cursor above a particular screen (grid) region to click; ¶ [0056]-[0057]) and a color matching algorithm (a different color in response to a click, ¶ [0056]).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 7-9, 17-19 are rejected under 35 U.S.C. 103 as being unpatentable over Bataller et al (US 2017/0001308, previously cited in Non-Final Rejection - 06/04/2025 Final Rejection - 02/13/2026) in view of Viet et al (US 2020/0059441, previously cited in Non-Final Rejection - 06/04/2025, Final Rejection - 02/13/2026).
Regarding Claim 1, Bataller et al teach a computerized method (method 300 for automating a manual process using system 100; Fig 1, 3 and ¶ [0017], [0054]) comprising:
providing a step-by-step guide for a mobile workflow (“mobile workflow” described broadly as “any human computer task to be automated with computer vision and robotic touch arm” specification ¶ [0002]-[0003], [0023]) (computer vision techniques are applied to identify activities of the user on a mobile device (¶ [0071]-[0073]) for a user manual process to be automated, S310-340, to create a process definition (a step-by-step guide for mobile workflow), S350, using process definition generator 140 (based on pipeline 110, 120, 130); Fig 1, 3 and ¶ [0017], [0031]-[0033], [0054]-[0059]);
observing a mobile device with a graphical user interface with a computer vision system (computer vision techniques are applied to the identified activities manually performed by a user interacting with a computer (mobile device ¶ [0071]), S310-S330, with images (observing) obtained by the image capturer 110; Fig 1, 3 and ¶ [0022]-[0023], [0055]-[0057]), wherein the computer vision system is configured to observe an area of user touch interest at a create time for the mobile workflow performed using the graphical user interface of the mobile device (the image capturer 110 (computer vision system) obtains images of a computer display that contains a touchscreen user interface, S320 and the images are analyzed by the activity identifier 120, S330; Fig 1, 3 and ¶ [0022]-[0023], [0056]-[0057]), wherein the mobile workflow comprises one or more tasks performed by a human user (tasks to be automated are based on a user manual process; Fig 1, 3 and ¶ [0021]-[0023], [0056]);
running a computer vision algorithm in communication with the computer vision system (images of user activities are captured by the image capturer 110, S320, and identified by the activity identifier 120, S330, using computer vision techniques ; Fig 1, 3 and ¶ [0021]-[0023], [0056]-[0057]);
locating the area of user touch interest at a play back time when automated mobile workflow is running (the activity information generator 130 identifies the image data representing the user screen touch activity, such as the location of the physical screen touch that user touched to indicate a response, found in the image capturer 110 images, S340; Fig 1, 3 and ¶ [0025]-[0026], [0058]);
optimizing a workflow guide (the process definitions (workflow guide) are generated by the process definition generator 140 from activity identified by the activity information generator 130, S350, and may be inspected and optimized for a particular process; Fig 1, 3 and ¶ [0030], [0061]);
linking each user input click with the step-by-step guide via the graphical interface (the activities for a given process definition are analyzed for multiple users to determine a particular similar procedure; ¶ [0034]-[0035], [0061]) while graphical user interface is being observed by computer vision system (the activities associated with the process (user input via graphic interface) are identified using computer vision techniques; ¶ [0033]-[0035], [0056]-[0057]);
determining a grid of interest relative to the screen (the touchscreen of the user interface is represented with a region of interest (“Yes” and “No” buttons representing a “grid” region based on pixel coordinates and the activity information generator 130 executes identification of the activity (touch) was performed in the region of interest, S330; Fig 1, 3 and ¶ [0032]-[0033], [0057]), wherein the grid comprises normalized pixels; and
automating the mobile workflow with a computer vision functionality and a robotic touch arm using the step-by-step guide (the process definition for all of the activities are automated with movement instructions by the process definition generator 140, S350, so the given activity may be executed by the robot arm 240 via the process definition; Fig 1-3 and ¶ [0030]-[0031], [0051], [0059]-[0060]);
guiding a user's interaction with the graphical user interface using the computer vision system (a user may open an application that automatically performs processes according to process definitions, automated by the activity execution engine 230, with a user interface to record user response with activity trigger engine 220; Fig 2, 3 and ¶ [0041]-[0043], [0059]);
wherein the computer vision algorithm is implemented on one rectangle in the grid of interest (the touchscreen of the user interface is represented with a rectangle (“Yes” and “No” buttons represented as rectangle via pixel regions ¶ [0053]) to determine if a trigger event (touch) was performed by user; Fig 2, 3 and ¶ [0042]-[0043], [0059]),
selecting a matching algorithm to use for a given step of the mobile workflow (the activity execution engine 230 executes activities based on process definition (matching algorithm) identified from the activity trigger engine 220; Fig 2, 3 and ¶ [0044], [0051], [0060]), wherein the matching algorithm is one of a template matching algorithm, a feature matching algorithm, and a color matching algorithm (the process definition may be a process for generating a template, including a template matching algorithm (¶ [0055], [0059]), a feature matching algorithm (a mouse cursor above a particular screen (grid) region to click; ¶ [0056]-[0057]) and a color matching algorithm (a different color in response to a click, ¶ [0056]).
Bataller et al does not teach implementing a pixel normalization algorithm for pixels of a screen of the mobile device observed using the computer vision algorithm; and the grid comprises normalized pixels.
Viet et al is analogous art pertinent to the technological problem addressed in this application and teaches implementing a pixel normalization algorithm for pixels of a screen of the mobile device observed using the computer vision algorithm (the robotic training system 102 includes a color analysis device 108 to normalize the color, array size or both, at the pixel level (row 130 and column 132), which can occur when the user uses the user interface 118 to select actions and the device 116 provides data to the robotic training system 102 (computer vision ¶ [0009]); Fig 1 and ¶ [0025]-[0027], [0047]-[0049]); and the grid (grid is not defined in regard to a size or number of pixel with respect to the image , specification ¶ [0024], and can therefore have either interpretation and could be the entire computer screen) (an image can be identified based on pixel dimension (a “grid”) entry arrays in the x and y axis direction; ¶ [0046], [0049]) comprises normalized pixels (an image can be identified based on pixel dimension (a “grid”) entry arrays in the x and y axis direction and the color analysis 108 of training system 108 may normalize the product; Fig 1 and ¶ [0046]-[0049]).
It would have been obvious to one of ordinary skill in the art, before the effective filing date of this application, to combine the teachings of Bataller et al a with Viet et al including implementing a pixel normalization algorithm for pixels of a screen of the mobile device observed using the computer vision algorithm; and the grid comprises normalized pixels. By normalizing pixels between images, an appropriate color and size analysis can be performed, such as evaluating values to a threshold amount and thereby more accurately detecting pixel clusters to identify shapes between images for robotic automation of processing based on computer vision techniques, as recognized by Viet et al (¶ [0009]-[0010]).
Regarding Claim 7, Bataller et al in view of Viet et al teach the computerized method of claim 1 (as described above), wherein the computer vision algorithm is implemented for one rectangle in a nine by nine (9X9) grid of interest (Viet et al, a region can be identified in the image, such that the array size can be cropped to a particular size (such as 9x9, with the example size given of 200 pixels by 200 pixels); ¶ [0047], [0049]).
As discussed in the Response above, Viet teaches that the array size may be of any appropriate size (¶ [0047]). The applicant does not disclose the rationale of why a 9x9 grid would be a necessary size or provide a specific benefit. It would have been obvious to one of ordinary skill in the art before the effective filing date for Viet to apply a 9x9 grid size for the computer vision algorithm as the dimension allows for analysis of small shapes of interest, as recognized by Viet et al (¶ [0009]-[0010]).
Regarding Claim 8, Bataller et al in view of Viet et al teach the computerized method of claim 1 (as described above), wherein the computer vision algorithm is implemented for one rectangle in a single two hundred and fifty-five by two hundred and fifty-five (255x255) grid of interest (Viet et al, a region can be identified in the image, such that the array size can be cropped to a particular size (such as 255x255, with the example size given of 200 pixels by 200 pixels); ¶ [0047], [0049]).
As discussed in the Response above, Viet teaches that the array size may be of any appropriate size (¶ [0047]). The applicant does not disclose the rationale of why a 255x255 grid would be a necessary size or provide a specific benefit. It would have been obvious to one of ordinary skill in the art before the effective filing date for Viet to apply a 255x255 grid size for the computer vision algorithm as the dimension allows for analysis of large shapes of interest, as recognized by Viet et al (¶ [0009]-[0010]).
Regarding Claim 9, Bataller et al in view of Viet et al teach the computerized method of claim 1 (as described above), wherein the grid of interest scales (scale is not defined and is interpreted as a selected pixel region within a range; see MPEP 2131.03(I) that states a range is anticipated if the prior art provides a specific example within the range) from 9x9 to 255x255 (Viet et al, a region can be identified in the image, such that the array size can be cropped to a particular size (that includes a size between 9x9 and 255x255, with the example size given of 200 pixels by 200 pixels); ¶ [0047], [0049]).
As discussed in the Response above, Viet teaches that the array size may be of any appropriate size (¶ [0047]). Viet further states the region of interest may be cropped (“a cropped version of the difference image 126, may have a size of 200 pixels by 200 pixels”. ¶ [0049]). The applicant does not disclose the interpretation of “from 9x9 to 255x255” and it is interpreted as any size between the two array sizes. Again, the applicant did not discuss the intent of the ranges or a specific benefit. It would have been obvious to one of ordinary skill in the art before the effective filing date for Viet to apply an array within the range of 9x9 to 255x255 for the computer vision algorithm as the dimension allows for analysis of small shapes of interest, as recognized by Viet et al (¶ [0009]-[0010]).
Regarding Claim 17, Bataller et al teach a computer system for guiding a user on a graphical interface with computer vision (computer system 100 for automating a manual process 300 to create an automated system 200 executed by a robot 240; Fig 1-4 and ¶ [0017], [0038], [0054], [0062]) comprising:
a step-by-step guide for a mobile workflow (“mobile workflow” described broadly as “any human computer task to be automated with computer vision and robotic touch arm” specification ¶ [0002]-[0003], [0023]) (computer vision techniques are applied to identify activities of the user on a mobile device (¶ [0071]-[0073]) for a user manual process to be automated, S310-340, to create a process definition (a step-by-step guide for mobile workflow), S350, using process definition generator 140 (based on pipeline 110, 120, 130); Fig 1, 3 and ¶ [0017], [0031]-[0033], [0054]-[0059]);
a computer vision system running a computer vision algorithm (computer vision techniques are applied by system 100 to the identified activities manually performed by a user interacting with a computer (mobile device ¶ [0071]), and in particular, images of user activities are captured by the image capturer 110, S320, and identified by the activity identifier 120, S330, using computer vision techniques; Fig 1, 3 and ¶ [0021]-[0023], [0056]-[0057]);
a display, a processor, a memory, a storage, and one or more I/O devices wherein the computer system (computer system 400 (system 100) includes processor 410, memory 420, storage device 430 and I/O device 440 with display unit; Fig 4 and ¶ [0062]-[0066]) is configured to perform the steps of:
producing a graphical interface on the display (the I/O display unit displays information for a user interface; ¶ [0035], [0063]);
observing with the computer vision algorithm, an area of touch interest for the mobile workflow (computer vision techniques are applied to the identified activities manually performed by a user interacting with a computer (mobile device ¶ [0071]), S310-S330, with images (observing) obtained by the image capturer 110; Fig 1, 3 and ¶ [0022]-[0023], [0055]-[0057]), wherein the mobile workflow includes one or more user input actions performed via the one or more I/O devices (the image capturer 110 (computer vision system) obtains images of a computer display that contains a touchscreen user interface, S320, and the images are analyzed by the activity identifier 120, S330; Fig 1, 3 and ¶ [0022]-[0023], [0056]-[0057]), wherein the one or more user input actions correspond to actions in the step-by-step guide (tasks to be automated are based on a user manual process; Fig 1, 3 and ¶ [0021]-[0023], [0056]);
determining a grid of interest (the touchscreen of the user interface is represented with a region of interest (“Yes” and “No” buttons representing a “grid” region based on pixel coordinates and the activity information generator 130 executes identification of the activity (touch) was performed in the region of interest, S330; Fig 1, 3 and ¶ [0032]-[0033], [0057]), wherein the grid comprises normalized pixels;
linking the mobile workflow on the graphical interface to a step-by-step guide (the activities for a given process definition are analyzed for multiple users to determine a particular similar procedure; ¶ [0034]-[0035], [0061]), wherein the mobile workflow comprises one or more tasks performed by a human user(tasks to be automated are based on a user manual process; Fig 1, 3 and ¶ [0021]-[0023], [0056]); and
automating the mobile workflow with the computer vision system and a robotic touch arm (the process definition for all of the activities are automated with movement instructions by the process definition generator 140, S350, so the given activity may be executed by the robot arm 240 via the process definition; Fig 1-3 and ¶ [0030]-[0031], [0051], [0059]-[0060]);
wherein the computer vision algorithm is implemented on one rectangle in the grid of interest (the touchscreen of the user interface is represented with a rectangle (“Yes” and “No” buttons represented as rectangle via pixel regions ¶ [0053]) to determine if a trigger event (touch) was performed by user; Fig 2, 3 and ¶ [0042]-[0043], [0059]),
wherein the step of linking includes a matching algorithm (the activity execution engine 230 executes activities based on process definition (matching algorithm) identified from the activity trigger engine 220; Fig 2, 3 and ¶ [0044], [0051], [0060]) that determines which of a template matching algorithm, a feature matching algorithm, and a color matching algorithm are used for any step of the step-by-step guide (the process definition may be a process for generating a template, including a template matching algorithm (¶ [0055], [0059]), a feature matching algorithm (a mouse cursor above a particular screen (grid) region to click; ¶ [0056]-[0057]) and a color matching algorithm (a different color in response to a click, ¶ [0056]), wherein the step of linking further includes linking each user input click with the step-by-step guide via the graphical interface (the activities for a given process definition are analyzed for multiple users to determine a particular similar procedure; ¶ [0034]-[0035], [0061]) while graphical user interface is being observed by computer vision system (the activities associated with the process (user input via graphic interface) are identified using computer vision techniques; ¶ [0033]-[0035], [0056]-[0057]).
Bataller et al does not teach the grid comprises normalized pixels.
Viet et al is analogous art pertinent to the technological problem addressed in this application and teaches the grid (grid is not defined in regard to a size or number of pixel with respect to the image, specification ¶ [0024], and can therefore have either interpretation and could be the entire computer screen) (an image can be identified based on pixel dimension (a “grid”) entry arrays in the x and y axis direction; ¶ [0046], [0049]) comprises normalized pixels (an image can be identified based on pixel dimension (a “grid”) entry arrays in the x and y axis direction and the color analysis 108 of training system 108 may normalize the product; Fig 1 and ¶ [0046]-[0049]).
It would have been obvious to one of ordinary skill in the art, before the effective filing date of this application, to combine the teachings of Bataller et al a with Viet et al including the grid comprises normalized pixels. By normalizing pixels between images, an appropriate color and size analysis can be performed, such as evaluating values to a threshold amount and thereby more accurately detecting pixel clusters to identify shapes between images for robotic automation of processing based on computer vision techniques, as recognized by Viet et al (¶ [0009]-[0010]).
Regarding Claim 18, Bataller et al in view of Viet et al teach the system of claim 17, wherein the grid of interest is a nine by nine (9x9) grid of interest (Viet et al, a region can be identified in the image, such that the array size can be cropped to a particular size (such as 9x9, with the example size given of 200 pixels by 200 pixels); ¶ [0047], [0049]).
As discussed in the Response above, Viet teaches that the array size may be of any appropriate size (¶ [0047]). The applicant does not disclose the rationale of why a 255x255 grid would be a necessary size or provide a specific benefit. It would have been obvious to one of ordinary skill in the art before the effective filing date for Viet to apply a 255x255 grid size for the computer vision algorithm as the dimension allows for analysis of large shapes of interest, as recognized by Viet et al (¶ [0009]-[0010]).
Regarding Claim 19, Bataller et al in view of Viet et al teach the system of claim 17 (as described above), wherein the grid of interest is a (255x255) grid of interest (Viet et al, a region can be identified in the image, such that the array size can be cropped to a particular size (that includes a size between 9x9 and 255x255, with the example size given of 200 pixels by 200 pixels); ¶ [0047], [0049]).
As discussed in the Response above, Viet teaches that the array size may be of any appropriate size (¶ [0047]). The applicant does not disclose the rationale of why a 255x255 grid would be a necessary size or provide a specific benefit. It would have been obvious to one of ordinary skill in the art before the effective filing date for Viet to apply a 255x255 grid size for the computer vision algorithm as the dimension allows for analysis of large shapes of interest, as recognized by Viet et al (¶ [0009]-[0010]).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Kim et al (Adaptive planar vision marker composed of LED arrays for sensing under low visibility), which discloses the use of LED arrays for computer vision techniques in robotics to visually detect a given marker and measure dimensions of the markers and the distance from the marker.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to KATHLEEN M BROUGHTON whose telephone number is (571)270-7380. The examiner can normally be reached Monday-Friday 8:00-5:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, John Villecco can be reached at (571) 272-7319. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/KATHLEEN M BROUGHTON/ Primary Examiner, Art Unit 2661
