Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
The Amendment filed June 25 2026 has been entered and considered. Claim 1 has been amended. Claim 2 has been canceled. New claim 6 has been added. In light of the amendment the prior art rejection of Claim 1 is withdrawn as moot. The new grounds of rejection set forth in the present action were necessitated by Applicants’ claim amendments; accordingly, this action is made final.
101 Rejection
In view of the amendments to claim 1, the rejection under 35 USC 101 is withdrawn.
Response to Arguments
Applicant’s arguments with respect to claim 1 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1 and 3-6 are rejected under 35 U.S.C. 103 as being unpatentable over Nakasu et al. (US Patent Pub. No. 2017/0017303 A1), published 2017, in view of Groß et al. (US Patent Pub. No. 2022/0108561 A1), filed 2020, further in view of Watson et al. (US Patent No. 11,150,200 B1), filed 2020.
Regarding claim 1, Nakasu teaches an operation monitoring method for monitoring an operation performed by an operator (Para. 35, “The operation recognition device 100 in accordance with the first embodiment is configured to acquire a movement of a user who performs an operation for an operation target”), the method comprising; acquiring an operation procedure manual from an operation procedure storage server (Para. 112, “The correct action pattern may be included in the area information. It may be registered in advance, as the content of the procedure manual, in a database that is internal or external to the operation recognition device 100”): specifying a predetermined action to be performed by the operator based on a description in the operation procedure manual (Para. 38, “The user performs an operation in accordance with operation items for the operation target device having an identification ID 202 described in a procedure manual or the like”); determining whether or not the operator is performing the predetermined action on a workpiece by specifying an action of the operator based on the coordinate data of the operator and comparing the action of the operator with the predetermined action (Para. 115, “The action pattern recognizer 104 is configured to acquire the movement information from the acquirer 101 and recognize the action pattern of the user from the movement information.”; Para. 165, “For example, the position and the scale of the finger or the like of the user is recognized using geometrical information of the operation target, the operation area, the correct-answer area, etc. obtained from the acquirer 101 and using geometrical information of the corresponding portion within the image, and estimates the operation position.”); and in response to determining that the operator is performing the predetermined action performing a correct operation determination (Para. 122, “The operation recognizer 102 obtains, in the same or similar manner as in the previous embodiments, the movement information and the area information from the acquirer 101. Information on the individual operations is included in the area information as in the same or similar manner as in the previous embodiments, and in accordance with this embodiment, the correct action pattern is further included therein. Accordingly, the operation recognizer 102 is allowed to identify the correct action pattern for the current operation and perform the correct operation determination.”); and a workpiece camera attached to the operator (Para. 164, “The camera may be installed near the operation target device or may be attached to the user or mounted to the user's belongings.”).
Nakasu does not explicitly disclose generating, from an image of the operator captured by a process camera, coordinate data of a skeleton of the operator using a detector learned in advance to detect the skeleton of the operator from the image, specifying an action of the operator based on the coordinate data of the skeleton of the operator, or performing quality determination of the workpiece by inputting image data of the workpiece generated by a workpiece camera into a discriminator learned in advance to output a result of the quality determination of the workpiece. However, they do disclose using a camera that can be attached to the operator, determining position information of the operator using a camera, and the possibility of using a neural network to discriminate action patterns (Paras. 118, 120, 164).
Groß teaches generating, from an image of the operator captured by a process camera, coordinate data of a skeleton of the operator using a detector learned in advance to detect the skeleton of the operator from the image (Para. 99, “This environment information is next evaluated in step 720 (feature extraction) to create a skeleton model of the patient (step 725, skeleton model creation)… If frameworks such as OpenPose are used, a 2D camera such as an ordinary RGB camera can be used instead of a 3D camera.”); determining whether or not the operator is performing the predetermined action by specifying an action of the operator based on the coordinate data of the skeleton of the operator (Para. 107, “Once feature classification 765 has been performed, the classified features are evaluated contextually (referred to as movement classification in step 770)… The aim is to output the movement correction (step 450), i.e. output instructions to the patient to prompt him or her to adjust his or her movements in his or her own movement pattern so that these correspond or at least approximate the physiological movement pattern. Feedback is given to the patient by the service robot 17 (indication of errors, request for correction, affirmation of behavior/praise).”).
Groß does not explicitly disclose performing quality determination of the workpiece by inputting image data of the workpiece generated by a workpiece camera attached to the operator into a discriminator learned in advance to output a result of the quality determination of the workpiece. However, they do disclose the use of neural networks for classification.
Watson teaches performing quality determination of the workpiece by inputting image data of the workpiece generated by a workpiece camera into a discriminator learned in advance to output a result of the quality determination of the workpiece (Col. 8, Lines 49-52, “The defect detection portion 140dp processes image data corresponding to user labeled images of defects to train a classification model, which in various implementations may be an AI classification model.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Nakasu to incorporate the teachings of Groß and Watson to include generating, from an image of the operator captured by a process camera, coordinate data of a skeleton of the operator using a detector learned in advance to detect the skeleton of the operator from the image, specifying an action of the operator based on the coordinate data of the skeleton of the operator, and performing quality determination of the workpiece by inputting image data of the workpiece generated by a workpiece camera into a discriminator learned in advance to output a result of the quality determination of the workpiece. Nakasu discloses a method for monitoring an operator’s actions through the use of many different sensors and determining whether the performed action is in compliance with a procedure manual. They also teach using a camera attached to the operator as well as neural networks for action discrimination. However, Nakasu does not disclose generating coordinate data of a skeleton of the operator using a detector learned in advance to perform action determination. Groß teaches to generate coordinate data of a skeleton of the operator using a neural network to specify an action of the operator. One of ordinary skill in the art would have recognized that implementing the skeleton coordinate method of Groß into the camera-based position determination system disclosed by Nakasu would have predictably enhanced the accuracy of coordinate information and increased the precision of action determination. Nakasu also discloses that it is known to perform an inspection operation captured by a camera comparing images before and after the operation to automatically determine whether the correct operation has been performed (Para. 4). One of ordinary skill in the art would have understood that supplementing the proactive action-confirmation system of Nakasu with the robust machine learning based post-operation verification step of Watson provides critical redundancy as well as a superior classification technique compared to the disclosed image comparison.
Regarding claim 3, Nakasu as modified teaches all of the elements of claim 1, as stated above, as well as wherein the quality determination of the workpiece is performed based on a moving image obtained by capturing the workpiece (Para. 163, “The acquirer 101 acquires an image captured by an image capturing device. It is contemplated that the operation section by the user is contained in the image. In addition, a video may be used as the image.”).
Regarding claim 4, Nakasu as modified teaches all of the elements of claim 3, as stated above, as well as wherein a length of the moving image is changed in accordance with a content of an operation being performed by the operator or a type of the workpiece (Para. 127, “an action that the user performs prior to attempting to perform the operation is detected as the trigger, and the operation position estimation is started after the detection of the trigger, so that it is made possible to reduce the processing load of the estimation device 100.”, using a trigger and recording the operation necessarily means the length of the moving image is changed depending on the content of the operation).
Regarding claim 5, Nakasu as modified teaches all of the elements of claim 3, as stated above, as well as wherein the moving image is captured before and after a timing at which it is determined whether or not the operator is performing the predetermined action on the workpiece (Para. 128, “The operation recognizer 102 is configured to detect a trigger on the basis of the movement information. The operation recognizer 102 includes a trigger detector configured to detect the trigger. It is assumed here that the operation position estimation is not performed until the trigger is detected.”).
Regarding claim 6, Nakasu as modified teaches all of the elements of claim 1, as stated above, as well as wherein specifying the predetermined action comprises specifying the predetermined action based on an order of operations described in the operation procedure manual and a previous action of the operator (Para. 44, “In addition, for example, the user may have to perform a plurality of operations in a predetermined sequence, for example, the user may have to press a button first and then flip up a switch, and further turns a dial. In such a case, an operation area corresponding to the subsequent operation that the user should perform is referred to as a “correct-answer area.””).
Conclusion
Pertinent Prior Art: Kitazumi, US 12,361,531, “Operation determination apparatus and operation determination method”, filed 2021. Similar operation determination method that also detects workpieces using machine learning.
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DAVID A WAMBST whose telephone number is (703)756-1750. The examiner can normally be reached M-F 9-6:30 EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Gregory Morse can be reached at (571)272-3838. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/DAVID ALEXANDER WAMBST/Examiner, Art Unit 2663
/GREGORY A MORSE/Supervisory Patent Examiner, Art Unit 2698
Prosecution Insights