DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
Joint Inventors
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Priority
Acknowledgment is made of applicant’s claim for foreign priority under 35 U.S.C. 119 (a)-(d). A certified copy of this document has been placed in the file wrapper as of 05/30/2025. As such, the effective filing date of the instant application is considered 11/18/2021, coinciding with the filing date of the Japanese application to which foreign priority was requested.
Response to Amendment
Claims 1 and 3-10 have been amended. Claim 2 has been canceled and no claims have been added. The claim objection has been withdrawn in view of amendment.
Response to Arguments
Applicant's arguments filed 07/22/2025 have been fully considered but they are not persuasive.
Examiner acknowledges Applicant’s arguments with respect to the 35 U.S.C. 102(a)(1) rejection and finds them unconvincing.
Applicant contends Vu does not disclose the movement of the tool in advance to a first axis coordinate. Examiner respectfully disagrees, and points to at least Vu 0074, which states “a predicted future state of a robot can be derived from the current instantaneous state and knowledge of a robot's geometry and kinematics. This is also true of the end effector. If, for example, an identified end effector is known to extend along a linear axis, its current state may be projected into the future based on this constraint (along with, if the robot and/or its arm is moving, the current robot state as well).”.
Applicant further contends that the filtration discussed in Vu does not encompass the filtration process described in the instant application. Examiner respectfully disagrees with this characterization of Vu and points to at least Vu 0077, which states “the robot may provide an interface to obtain joint positions that are not safety-rated, in which case the joint positions can be verified against images from sensors 102 (using, for example, safety-rated software). For example, received joint positions may be combined with static 3D models of each link and any end effectors to generate a 3D model of the entire robot 402. This 3D image can be used to remove any objects in the sensing data that are part of the robot itself. If the joint positions are correct, this will fully eliminate all object data attributed to the robot 402. If, however, the joint positions are incorrect, the true position of robot 402 will diverge from the model, and some parts of the detected robot will not be removed. Those points will then appear as a foreign object in the new cycle. In the previous cycle, it can be assumed that the joint positions were correct because otherwise robot 402 would have been halted. Since the base joint of the robot does not move, at least one of the divergent points must be close to the robot.”. This filtering of link position data attributed to the robot teaches the broadly claimed “[extraction of] distribution data which is the distribution data at the first axis coordinate according to the tool condition”.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claim(s) 1 and 3-10 is/are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Vu et al. (US20210205995, referred to as Vu).
Regarding claim 1: Vu discloses: A tool check device for robot arm, ([0009] the invention relates to a system for identifying an end effector of a robot in a processing environment. In various embodiments, the system comprises at least one sensor for digitally recording visual information, at sensor(s) being positioned to record at least one image of the end effector in an arbitrary orientation; a computer memory for storing a digital model of each of a plurality of end effectors; and a processor configured to identify the end effector from the at least one recorded image and the stored digital models. The processor may be configured to computationally generate, from at least one recorded image, a 3D spatial representation of the end effector. In various embodiments, the processor is further configured to generate a 3D voxel-grid volumetric representation of the end effector.) comprising: a movement control unit configured to control a robot arm such that a tool to be attached to the robot arm is arranged at a first axis coordinate, of a three-dimensional coordinate system, according to a tool condition concerning at least either of a type or a state of the tool in an inspection space defined as the three-dimensional coordinate system; ([0072] The voxel-level or point cloud representation may be compared to the stored models—directly, if the models are represented as, or converted to, image data, or at a higher level of representation, e.g., as a CAD model. The comparison may be performed during the configuration process, as described above, and can be assisted by knowledge of robot configuration and its reported position and orientation (which typically includes the position and orientation of the end effector). That is, the end effector may be identified solely from image information or from this information combined with position and orientation information, which constrains the search space. Alternatively, the volumetric end-effector representation may be analyzed by a neural network that has been trained using labeled images of many end effectors in many spatial configurations. [0074] a predicted future state of a robot can be derived from the current instantaneous state and knowledge of a robot's geometry and kinematics. This is also true of the end effector. If, for example, an identified end effector is known to extend along a linear axis, its current state may be projected into the future based on this constraint (along with, if the robot and/or its arm is moving, the current robot state as well).) a distribution data acquisition unit configured to acquire distribution data, of an object in the inspection space, indicated by a combination of a second axis coordinate and a third axis coordinate of the three-dimensional coordinate system, after the control by the movement control unit, the distribution data indicating distribution of the object in a photographed image of the object which is generated by a 3D camera whose photographing range is the inspection space; and ([0070] Object analysis module 415 can further be trained to identify the end effector 445 from among a library or database of end-effector models stored in the memory of OMS 410. Based on an object-level representation of the robot 402, object analysis module 415 may semantically identify the robot arm and isolate, in the digital image of the robot captured by sensors 102, the terminal region of the robot arm. Object analysis module 415 may then perform segmentation and voxel-grid dissection to isolate 3D point cloud data corresponding to the end effector from background pixels and the remainder of the robot arm. [0023] The sensors 102 may be conventional optical sensors such as cameras, e.g., 3D time-of-flight cameras, stereo vision cameras, or 3D LIDAR sensors or radar-based sensors, ideally with high frame rates (e.g., between 25 FPS and 100 FPS). The mode of operation of the sensors 102 is not critical so long as a 3D representation of the workspace 100 is obtainable from images or other data obtained by the sensors 102. As shown in the figure, sensors 102 collectively cover and can monitor the workspace 100, which includes a robot 106 controlled by a conventional robot controller 108.) a determination unit configured to determine whether the tool condition is satisfied, wherein the distribution data acquisition unit is configured to perform, on the distribution data, a filtering process for extracting corresponding distribution data which is the distribution data at the first axis coordinate according to the tool condition, and wherein the determination unit is configured to determine whether the tool condition is satisfied based on the corresponding distribution data. ([0077] the robot may provide an interface to obtain joint positions that are not safety-rated, in which case the joint positions can be verified against images from sensors 102 (using, for example, safety-rated software). For example, received joint positions may be combined with static 3D models of each link and any end effectors to generate a 3D model of the entire robot 402. This 3D image can be used to remove any objects in the sensing data that are part of the robot itself. If the joint positions are correct, this will fully eliminate all object data attributed to the robot 402. If, however, the joint positions are incorrect, the true position of robot 402 will diverge from the model, and some parts of the detected robot will not be removed. Those points will then appear as a foreign object in the new cycle. In the previous cycle, it can be assumed that the joint positions were correct because otherwise robot 402 would have been halted. Since the base joint of the robot does not move, at least one of the divergent points must be close to the robot.)
Regarding claim 3: Vu discloses: The tool check device for the robot arm according to claim 1
Vu further discloses: wherein the distribution data is data associating each of a plurality of pixels forming a photographed image indicated by the combination of the second axis coordinate and the third axis coordinate with a luminance value correlated with a distance from the object to the 3D camera. ([0023] Refer first to FIG. 1, which illustrates a representative 3D workspace 100 monitored by a plurality of sensors representatively indicated at 102 1, 102 2, 102 3. The sensors 102 may be conventional optical sensors such as cameras, e.g., 3D time-of-flight cameras, stereo vision cameras, or 3D LIDAR sensors or radar-based sensors, ideally with high frame rates (e.g., between 25 FPS and 100 FPS). The mode of operation of the sensors 102 is not critical so long as a 3D representation of the workspace 100 is obtainable from images or other data obtained by the sensors 102. As shown in the figure, sensors 102 collectively cover and can monitor the workspace 100, which includes a robot 106 controlled by a conventional robot controller 108.)
Regarding claim 4: Vu discloses: The tool check device for the robot arm according to claim 3
Vu further discloses: wherein the distribution data acquisition unit is configured to perform a binarization process on the photographed image data generated by the 3D camera, by using a large luminance threshold and a small luminance threshold according to the tool condition, and is configured to acquire corresponding distribution data which is the distribution data at the first axis coordinate according to the tool condition. ([0041] if the sensed light level in a given voxel is insufficient to definitively establish emptiness or the presence of a boundary, the voxel is marked as unknown. The signal and threshold value may depend on the type of sensor being used. In the case of an intensity-based 3D sensor (for example, a time-of-flight camera) the threshold value can be a signal intensity, which may be attenuated by objects in the workspace of low reflectivity. In the case of a stereo vision system, the threshold may be the ability to resolve individual objects in the field of view. Other signal and threshold value combinations can be utilized depending on the type of sensor used.)
Regarding claim 5: Vu discloses: The tool check device for the robot arm according to claim 3
Vu further discloses: a posture acquisition unit configured to acquire posture data indicating a posture of the 3D camera in the inspection space, wherein the distribution data acquisition unit is configured to acquire the distribution data in the photographed image which is a partial region, of an original photographed image photographed by the 3D camera, determined based on the posture data. ([0024] For ease of illustration, FIG. 2 shows two sensors 202 1, 202 2 and their zones of coverage 205 1, 205 2 within the workspace 200 in two dimensions; similarly, only the 2D footprint 210 of a 3D object is shown. The portions of the coverage zones 205 between the object boundary and the sensors 200 are marked as unoccupied, because each sensor affirmatively detects no obstructions in this intervening space. The space at the object boundary is marked as occupied. In a coverage zone 205 beyond an object boundary, all space is marked as unknown; the corresponding sensor is configured to sense occupancy in this region but, because of the intervening object 210, cannot do so. [0025] With renewed reference to FIG. 1, data from each sensor 102 is received by a control system 112. The volume of space covered by each sensor—typically a solid cone—may be represented in any suitable fashion, e.g., the space may be divided into a 3D grid of small (5 cm, for example) cubes or “voxels” or other suitable form of volumetric representation. For example, workspace 100 may be represented using 2D or 3D ray tracing, where the intersections of the 2D or 3D rays emanating from the sensors 102 are used as the volume coordinates of the workspace 100. This ray tracing can be performed dynamically or via the use of precomputed volumes, where objects in the workspace 100 are previously identified and captured by control system 112. For convenience of presentation, the ensuing discussion assumes a voxel representation; control system 112 maintains an internal representation of the workspace 100 at the voxel level, with voxels marked as occupied, unoccupied, or unknown.)
Regarding claim 6: Vu discloses: The tool check device for the robot arm according to claim 1
Vu further discloses: wherein the determination unit is configured to determine whether the tool condition is satisfied, based on a distribution area of the object indicated by the distribution data. ([0072] the end effector may be identified solely from image information or from this information combined with position and orientation information, which constrains the search space. Alternatively, the volumetric end-effector representation may be analyzed by a neural network that has been trained using labeled images of many end effectors in many spatial configurations. [0073] The robot's current end effector is identified as the closest match to the entries in the end-effector database. If there is any chance the database may be incomplete, object analysis module 415 may report an identification failure if the degree of match to any entry does not exceed a predefined threshold.)
Regarding claim 7: Vu discloses: The tool check device for the robot arm according to claim 6
Vu further discloses: wherein the determination unit is configured to identify a limited region based on a centroid position of a distribution region indicated by the distribution data, and is configured to determine whether the tool condition is satisfied, based on the distribution area in the limited region. ([0072] the end effector may be identified solely from image information or from this information combined with position and orientation information, which constrains the search space. Alternatively, the volumetric end-effector representation may be analyzed by a neural network that has been trained using labeled images of many end effectors in many spatial configurations. [0073] The robot's current end effector is identified as the closest match to the entries in the end-effector database. If there is any chance the database may be incomplete, object analysis module 415 may report an identification failure if the degree of match to any entry does not exceed a predefined threshold.)
Regarding claim 8: Vu discloses: The tool check device for robot arm according to claim 1
Vu further discloses: wherein the determination unit is configured to determine whether the tool condition is satisfied, based on a relationship between the second axis coordinate and a length of a distribution region indicated by the distribution data in a third axis direction of the three- dimensional coordinate system. ([0072] the end effector may be identified solely from image information or from this information combined with position and orientation information, which constrains the search space. Alternatively, the volumetric end-effector representation may be analyzed by a neural network that has been trained using labeled images of many end effectors in many spatial configurations. [0073] The robot's current end effector is identified as the closest match to the entries in the end-effector database. If there is any chance the database may be incomplete, object analysis module 415 may report an identification failure if the degree of match to any entry does not exceed a predefined threshold.)
Regarding claim 9: Rejected using the same rationale as claim 1, however further directed to “A non-transitory computer readable storage medium”, which is further disclosed by Vu: A non-transitory computer readable storage medium ([0026] FIG. 3 illustrates, in greater detail, a representative embodiment of control system 112, which may be implemented on a general-purpose computer. The control system 112 includes a central processing unit (CPU) 305, system memory 310, and one or more non-volatile mass storage devices (such as one or more hard disks and/or optical storage units) 312.)
Regarding claim 10: Rejected using the same rationale as claim 1.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ATTICUS A CAMERON whose telephone number is 703-756-4535. The examiner can normally be reached M-F 8:30 am - 4:30 pm. Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Thomas Worden can be reached on 571-272-4876. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ATTICUS A CAMERON/ /JASON HOLLOWAY/ Primary Examiner, Art Unit 3658 Examiner, Art Unit 3658A
Prosecution Insights