DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
The applicant’s claim to priority of PRO 63/450,741on 3/08/2023 is acknowledged.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1-3, 5-8, 11-13, 15, 19 and 20 are rejected under 35 U.S.C. 102(a)(2) as being anticipated by Shang et al. (US 20180009000 hereinafter Shang).
Regarding claim 1 (and similarly 19 and 20), Shang teaches a robotic system (See at least: Fig. 14), comprising:
a robotically-controlled equipment comprising a projector (See at least: Fig. 14 item 204 “dispensing system”, 200 “robotic arm” and 14-1 “projector”), the robotically-controlled equipment being configured to be positioned to direct a projection emitted by the projector onto a target object in a workspace(See at least: [0135] via “A projector 14-1 is mounted on the overhead frame 209 next to the overhead camera 202. The projector 14-1 can be used to highlight potential problem areas or locations to the operator (incorrect fluid dispensing, malformed fasteners, etc.) that have been detected during automated inspection of the part 208. These locations are highlighted by selectively projecting patterns of light 14-2 and 14-3 and indicating to the operator that further attention by human operators or the automated sealing system 10 is required in those areas.”); and
a processor coupled to the robotically-controlled equipment and configured to (See at least: Fig. 2 item 205 robot controller): receive an indication of an issue to be resolved by an intervening worker with respect to the target object; and
position the robotically-controlled equipment to direct the projection emitted by the projector onto the target object (See at least: Fig. 14 items 14-2 and 14-3 “patterns of light”; [0135] via “A projector 14-1 is mounted on the overhead frame 209 next to the overhead camera 202. The projector 14-1 can be used to highlight potential
problem areas or locations to the operator (incorrect fluid dispensing, malformed fasteners, etc.) that have been detected during automated inspection of the part 208. These locations are highlighted by selectively projecting patterns of light 14-2 and 14-3 and indicating to the operator that further attention by human operators or the automated sealing system 10 is required in those areas.”).
Regarding claim 2, Shang teaches wherein the robotically-controlled equipment comprises a robotic arm (See at least: Fig. 14 200 “robotic arm”). Regarding claim 3, Shang teaches wherein the projector is mounted on or otherwise integrated into an end effector attached to a distal end of the robotic arm (See at least: Fig. 10 items 10-1, 10-2 “laser line projectors”).
Regarding claim 5, Shang teaches wherein the projector comprises a laser projector and the projection comprises a laser projection projected onto the target object (See at least: Fig. 10 items 10-1, 10-2 “laser line projectors”; [0131] An alternative embodiment of the dispensing system is shown in FIG. 10 and includes two or more laser line projectors (10-1 and 10-2) and camera 302 (which may be either a stereo or a monocular camera) observing intersections of two laser planes with the part. The two laser line projectors 10-1 and 10-2 would be mounted so that the extracted line of best fit from each projector 10-1 and 10-2 would provide an instant surface normal via methods including and but not limited to the cross-product of the line of best fit unit vector. Ideally, the camera would be mounted in such a way so that a single camera could be used for 2D planar fastener detection and also view the two laser projections on the part.”).
Regarding claim 6, Shang teaches wherein the projection includes a color associated with the issue to be resolved with respect to the target object (See at least: Fig. 10 and 14. Note: The laser and/or light that is used to highlight an issue has a distinct color property).
Regarding claim 7, Shang teaches wherein the projection causes a shape or other graphic associated with the issue to be resolved with respect to the target object to be projected onto the target object (See at least: Fig. 14 items 14-2 and 14-3 “patterns of light”; [0135] via “A projector 14-1 is mounted on the overhead frame 209 next to the overhead camera 202. The projector 14-1 can be used to highlight potential
problem areas or locations to the operator (incorrect fluid dispensing, malformed fasteners, etc.) that have been detected during automated inspection of the part 208. These locations are highlighted by selectively projecting patterns of light 14-2 and 14-3 and indicating to the operator that further attention by human operators or the automated sealing system 10 is required in those areas.”).
Regarding claim 8, Shang teaches wherein the processor is further configured to direct the projection onto a specific part of the target object with respect to which the issue is to be resolved (See at least: Fig. 14 items 14-2 and 14-3 “patterns of light”; [0135] via “A projector 14-1 is mounted on the overhead frame 209 next to the overhead camera 202. The projector 14-1 can be used to highlight potential problem areas or locations to the operator (incorrect fluid dispensing, malformed fasteners, etc.) that have been detected during automated inspection of the part 208. These locations are highlighted by selectively projecting patterns of light 14-2 and 14-3 and indicating to the operator that further attention by human operators or the automated sealing system 10 is required in those areas.”).
Regarding claim 11, Shang teaches wherein the processor is configured to take into consideration, when positioning the robotically-controlled equipment, a line of sight of the intervening worker (See at least: Fig. 14; Note: The system is designed so an intervening worker can see the highlighted error. )
Regarding claim 12, Shang teaches a camera positioned and configured to generate image data of the workspace and wherein the processor is coupled to the camera and is further configured to use the image data to detect the issue the be resolved by the intervening worker (See at least: [0066] Both the vision processor 203 and the robotic arm controller 205 may be known microprocessors or computers which are configured to communicate with each other, the vision processor 203 being programmed with instructions or algorithms to analyze the real-time images from sensors 202 and 302, and any other additional sensors that may be included in various embodiments, and based on the analysis of these images, is further programmed with instructions or algorithms to instruct the robotic arm controller 205 to position the robotic arm 200 in pre-selected locations with the at least one feature to be sealed.”; [0131] An alternative embodiment of the dispensing system is shown in FIG. 10 and includes two or more laser line projectors (10-1 and 10-2) and camera 302 (which may be either a stereo or a monocular camera) observing intersections of two laser planes with the part. The two laser line projectors 10-1 and 10-2 would be mounted so that the extracted line of best fit from each projector 10-1 and 10-2 would provide an instant surface normal via methods including and but not limited to the cross-product of the line of best fit unit vector. Ideally, the camera would be mounted in such a way so that a single camera could be used for 2D planar fastener detection and also view the two laser projections on the part.”).
Regarding claim 13, Shang teaches wherein the processor is further configured to display to the intervening worker, via a displayed user interface, information associated with the issue to be resolved by the intervening worker (See at least: [0064] via “ A touch-screen monitor 206 with the user interface is interfaced with the vision processor 203 is used by an operator to observe the system status and to command actions.“).
Regarding claim 15, Shang teaches wherein the target object is moving through the workspace and the processor is configured to track the target object and to reposition the robotically-controlled equipment as needed to continue to direct the projection emitted by the projector onto the target object (See at least: Fig. 6; [0078] The calibration procedure can be automated by using the dispensing system camera 302 to align the nozzle 307 with the visual target. The depth is calibrated by a) maneuvering the tip 411 of the nozzle 307 to touch the target and record the depth value using the rangefinder 301; b) command the robotic arm 200 to a predefined height with respect to the vision target; c) visual servo X and Y position of nozzle 307 using camera 302 until the nozzle 307 aligns with a feature in the calibration target (605) within desired accuracy. Visual servoing is accomplished by taking the relative pose of the nozzle tip 411 with regard to the calibration target 605 and using a control loop in the robotic arm controller 205 to adjust the position of the nozzle tip 411 in a plane parallel to and above the vision target 605.; Note: Calibration of the system ensures the robotically-controlled equipment and the projector are positioned to project onto the target object when needed).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 4 and 14 are rejected under 35 U.S.C. 103 as being unpatentable over Shang in view of Link et al. (US 12254617 hereinafter Link).
Regarding claim 4, Shang fails to teach the following limitation, but Link teaches wherein the processor is configured to pause or postpone a task 470 may be in communication with the defect detector 460 in the system processing unit 170. The part rejection unit 470 may receive information from the defect detector 460, and based on a determination of whether the defects meet predetermined quality threshold requirements, issue a command to stop the inspection process, flag the inspected object as defected. The part rejection unit 470 with and the timer unit 480 will track the defective object and take action to remove the defective object from the transport 150 or alert the operator of a defect condition through, for example, the alert notification 130, or the network interface 445.”).
Therefore, it would have been obvious to one of ordinary skill in the art at the time of the invention to modify Shang in view of Link to teach wherein the processor is configured to pause or postpone a task the robotic arm was being used or was being prepared to be used to perform with respect to the target object in response to receiving said indication of the issue to be resolved by an intervening worker with respect to the target object so that a worker can address the issue before it causes any harm or propagates the issue further for the robotic task.
Regarding claim 14, Shang fails to teach the following limitation, but Link teaches wherein the processor is further configured to slow or interrupt a task being performed in the workspace with respect to the target object to enable the issue to be resolved (See at least: Col. 9 lines 32-44 via “A part rejection unit 470 may be in communication with the defect detector 460 in the system processing unit 170. The part rejection unit 470 may receive information from the defect detector 460, and based on a determination of whether the defects meet predetermined quality threshold requirements, issue a command to stop the inspection process, flag the inspected object as defected. The part rejection unit 470 with and the timer unit 480 will track the defective object and take action to remove the defective object from the transport 150 or alert the operator of a defect condition through, for example, the alert notification 130, or the network interface 445.”).
Therefore, it would have been obvious to one of ordinary skill in the art at the time of the invention to modify Shang in view of Link to teach wherein the processor is further configured to slow or interrupt a task being performed in the workspace with respect to the target object to enable the issue to be resolved so that a worker can address the issue before it causes any harm or propagates the issue further for the robotic task.
Claim 9 is rejected under 35 U.S.C. 103 as being unpatentable over Shang in view of Yamazaki (US 20250178206 hereinafter Yamazaki).
Regarding claim 9, Shang teaches the processor is further configured to direct the projection onto a specific part of the target object with respect to which the issue is to be resolved (Refer at least to claim 1 for reasoning and rationale.) but fails to teach wherein the projection causes a mesh or array to be projected onto the target object. However, Yamazaki teaches this limitation (See at least: [0017] As a specific example of the three-dimensional measuring instrument 10, it is possible to use a stereo camera that includes two two-dimensional cameras each capturing a two-dimensional image of a measurement target and a projector projecting an image including a grid of reference points onto the measurement target… ).
Therefore, it would have been obvious to one of ordinary skill in the art at the time of the invention to modify Shang in view of Yamazaki to teach wherein the projection causes a mesh or array to be projected onto the target object so that a worker can see the issue and address it before it causes any harm or propagates the issue further for the robotic task.
Claim 10 is rejected under 35 U.S.C. 103 as being unpatentable over Shang in view of Hsieh et al. (US 20230007960 hereinafter Hsieh).
Regarding claim 10, Shang fails to teach the following limitation, but Hsieh teaches wherein the processor is configured to position the robotically-controlled equipment at least in part by computing one or more Euler angles (See at least: [0045] As shown in FIG. 5 and FIG. 6, when the tray 20 is transparent, the position and contour of the objects can be accurately identified by using the object identification system 10 provided in the embodiment of the present disclosure. The object identification system 10 can identify the coordinate value and Euler angle of the objects according to the position and contour of the objects, so that the mechanical arm 100 can adjust a grasping angle of each one of the plurality of objects 60 positioned in a same layer according to the coordinate value and Euler angle of the objects and achieve accurate grasping of the objects.”).
Therefore, it would have been obvious to one of ordinary skill in the art at the time of the invention to modify Shang in view of Hsieh to teach wherein the processor is configured to position the robotically-controlled equipment at least in part by computing one or more Euler angles so that the orientation of the robotically controlled equipment can be clearly defined and clearly moved as desired and needed.
Claims 16-17 are rejected under 35 U.S.C. 103 as being unpatentable over Shang in view of Son et al. (US 20120198488 hereinafter Son).
Regarding claim 16, Shang fails to teach the following limitation, but Son teaches wherein the processor is further configured to receive an indication that the issue has been resolved (See at least: [0209] One way includes transmitting another command signal from terminal 1 170 (and/or an associated network server) to control at least one feature or setting of the image display apparatus 100 in an attempt to correct the error. When received, a processor of the image display apparatus may automatically control the at least one feature or setting indicated in the command signal. Subsequent steps may then include requesting and receiving a second captured image at terminal 1 170 to confirm whether the problem has been resolved.”).
Therefore, it would have been obvious to one of ordinary skill in the art at the time of the invention to modify Shang in view of Son to teach wherein the processor is further configured to receive an indication that the issue has been resolved so that the system and/or worker will know the issue is resolved and the task may continue.
Regarding claim 17, Shang fails to teach the following limitation, but Son teaches wherein the processor is configured to determine based on one or more images of the workspace that the issue has been resolved (See at least: [0209] One way includes transmitting another command signal from terminal 1 170 (and/or an associated network server) to control at least one feature or setting of the image display apparatus 100 in an attempt to correct the error. When received, a processor of the image display apparatus may automatically control the at least one feature or setting indicated in the command signal. Subsequent steps may then include requesting and receiving a second captured image at terminal 1 170 to confirm whether the problem has been resolved.”).
Therefore, it would have been obvious to one of ordinary skill in the art at the time of the invention to modify Shang in view of Son to teach wherein the processor is configured to determine based on one or more images of the workspace that the issue has been resolved so that the system and/or worker will know the issue is resolved and the task may continue.
Claim 18 is rejected under 35 U.S.C. 103 as being unpatentable over Shang in view of Noda et al. (US 20110288667 hereinafter Noda).
Regarding claim 18, Shang fails to teach the following limitation, but Noda teaches wherein the processor is further configured to cause the robotically-controlled equipment to resume a previously-interrupted task based at least in part on the indication that the issue has been resolved (See at least: [0351] The action control function (B1) of the error recovery section switches among the following three levels (M1) to (M3) of an error recovery operation mode according the degree of error seriousness, and assists a procedure for restart of production by synchronizing respective robots and respective production facilities with one another from an arbitrary stop position after recovery from an error state in each of the modes (M1) to (M3).).
Therefore, it would have been obvious to one of ordinary skill in the art at the time of the invention to modify Shang in view of Noda to teach wherein the processor is further configured to cause the robotically-controlled equipment to resume a previously-interrupted task based at least in part on the indication that the issue has been resolved so that the system can return to its task rather than waiting indefinitely.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Harry Oh whose telephone number is (571)270-5912. The examiner can normally be reached on Monday-Thursday, 9:00-3:00.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Abby Lin can be reached on (571) 270-3976. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/HARRY Y OH/Primary Examiner, Art Unit 3657