DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA . In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
Joint Inventors
This application currently names joint inventors. In considering patentability of the claims, the Examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the Examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Response to Amendment
The amendments filed on 03/05/2026 have been entered. Claims 1-19 remain pending in the application.
Claim Rejections - 35 USC § 103
The text of those sections of Title 35, U.S. Code not included in this action can be found in a prior Office action.
Claims 1, 4, 7, 11, 14, 16, and 19 are rejected under 35 U.S.C. 103 as being obvious over Higo, US 20190308325 A1, herein referred to as Higo, and in view of Sasaki, US 20190392607 A1, herein referred to as Sasaki.
Regarding claim 1, Higo discloses determining a current position and attitude of a visual sensor of the robot relative to the 3D object (Paragraph 0029, 0048; imaging capturing unit is moved to a given location and has a specific orientation to capture the 3D objects, position and orientation of the 3D object is determined relative to the image capturing unit), acquiring a grabbing template of the 3D object, the grabbing template comprising a specified grabbing position and attitude of the visual sensor relative to the 3D object (Paragraphs 0029-0030, 0045, 0070; a suction target map is generated to determine specific suction zones on each object which correspond to specific positioning and orientation of the objects relative to the imaging unit, *examiner notes that under BRI, grabbing of an object can include vacuum suction of an object), judging whether the grabbing template further comprises at least one reference grabbing position and attitude of the visual sensor relative to the 3D object, wherein the reference grabbing position and attitude is generated on the basis of the specified grabbing position and attitude (Paragraphs 0029-0030, 0045; suction target map can contain areas of the object that can be suctioned, these areas that can be suctioned have their position and orientation known relative to the camera), and based on a judgment result, using the grabbing template and the current position and attitude to generate a grabbing position and attitude of the robot (Paragraph 0058; based on the suction target map, the robot actuates to the appropriate suction position), and actuating the robot to move to the generated grabbing position and attitude and grab the object (Paragraph 0058; based on the suction target map, the robot actuates to the appropriate suction position to grasp the object), but fails to disclose wherein the grabbing position includes two positions located on edges of the object and the robot contacts the objects at the two positions.
However, Sasaki, in an analogous field of endeavor, teaches wherein the grabbing position includes two positions located on edges of the object and the robot contacts the objects at the two positions (Paragraphs 0059-0061, 0081; the robot may utilize an imaged edge of an object and use that edge to grasp the item; the edge of the object includes multiple points; examiner notes that 0081 is the primary citation which is based on a sixth embodiment, however 0059-0061 are being used as supporting evidence for the ‘edge image’ recited in 0081 and is not a combination of the first and sixth embodiment). Therefore, from the teaching of Sasaki, it would have been obvious to one of ordinary skill in the art before the effective filing date to have modified, with a reasonable expectation for success, the robotic system of Higo to include wherein the grabbing position includes two positions located on edges of the object and the robot contacts the objects at the two positions, as taught/suggested by Sasaki. The motivation to do so would be to ensure that the robot has sufficient and secure gripping of an object due to better contact.
Regarding claim 4, Higo in view of Sasaki renders obvious all the limitations of claim 1. Higo further discloses the generating the grabbing template (Paragraph 0030; the suction target map is generated).
Regarding claim 7, Higo in view of Sasaki renders obvious all the limitations of claim 1. Higo further discloses acquiring a real image of the 3D object at a visual angle of a current position and attitude (Paragraph 0054; imaging unit may obtain an image of the object at a given position and orientation), acquiring an image template of the 3D object, the image template representing multiple virtual images of a virtual model of the 3D object at visual angles of multiple different positions and attitudes (Paragraph 0054; an image is compared to a template to determine similarities; the template is obtained by imaging an object in a multitude of positions and orientations, this can be considered a virtual model comprising virtual images as the template is stored), determining a degree of similarity between the real image and each of the multiple virtual images respectively (Paragraph 0054; similarities between the image template and the actual image may be determined), and generating the current position and attitude based on a corresponding position and attitude of a virtual image with the highest degree of similarity (Paragraph 0054; positioning and orientation may be determined based on image features matching the template, matching between two data points indicates the highest degree of similarity).
Regarding claims 11 and 14, the claim limitations are similar to those in claims 1 and 4 and are rejected using the same rationale as seen above in claims 1 and 4.
Regarding claim 16, the claim limitations are similar to those in claim 7 and are rejected using the same rationale as seen above in claim 7.
Regarding claim 19, a portion of the claim limitations are similar to those in claim 1 and are rejected using the same rationale as seen above in claim 1. Additionally, Higo discloses a processor (Fig. 3 item H11; system includes a CPU), and a memory (Fig. 3 item H14; system includes memory storage).
Claims 2-3 and 12-13 are rejected under 35 U.S.C. 103 as being obvious over Higo, in view of Sasaki, and further in view of Zhang et al., CN 110076662 A, herein referred to as Zhang.
Regarding claim 2, Higo in view of Sasaki renders obvious all the limitations of claim 1. Higo further discloses based on the judgment result, using the grabbing template and the current position and attitude to determine a target grabbing position and attitude of the visual sensor relative to the 3D object (Paragraph 0058; based on the suction target map, the robot actuates to the appropriate suction position), and converting the target grabbing position and attitude to the grabbing position and attitude of the robot by calibration (Paragraph 0085; calibration may be performed in advance to determine positional and orientational relationships between the imaging unit and the robot), but fails to disclose converting the target grabbing position and attitude to the grabbing position and attitude of the robot by hand-eye calibration. However, Zhang, in an analogous field of endeavor, teaches converting the target grabbing position and attitude to the grabbing position and attitude of the robot by hand-eye calibration (Paragraph 0118; a hand-eye Jacobian matrix may be used to establish a relationship between the camera and the robot). Therefore, from the teaching of Zhang, it would have been obvious to one of ordinary skill in the art before the effective filing date to have further modified, with a reasonable expectation for success, the robotic system of Higo and Sasaki to include converting the target grabbing position and attitude to the grabbing position and attitude of the robot by hand-eye calibration, as taught/suggested by Zhang. The motivation to do so would be to use a well-known method for calibration of the robot to determine spatial relationships between the imaging unit and the robot.
Regarding claim 3, Higo, in view of Sasaki, and further in view of Zhang renders obvious all the limitations of claim 2. Higo further discloses when the grabbing template further comprises at least one said reference grabbing position and attitude, determining a grabbing position and attitude with the shortest movement distance from the current position and attitude in the grabbing template, to serve as the target grabbing position and attitude (Paragraphs 0030 and 0055; the object with the smallest depth value obtained from the imaging unit is selected for suctioning), and when the grabbing template does not comprise the reference grabbing position and attitude, taking the specified grabbing position and attitude to be the target grabbing position and attitude (Paragraphs 0030 and 0055; the suction target map may be used to determine if an object has an area suitable for suctioning; if an object does not have a suitable area, the closest object which does have a suitable suction area is selected, such as those at the side of a pile which might not interfere with other objects).
Regarding claims 12-13, the claim limitations are similar to those in claims 2-3 and are rejected using the same rationale as seen above in claims 2-3.
Claims 5-6, 10, 15, and 18 are rejected under 35 U.S.C. 103 as being obvious over Higo, in view of Sasaki, and further in view of Tateno et al., US 20130245828 A1, herein referred to as Tateno.
Regarding claim 5, Higo in view of Sasaki renders obvious all the limitations of claim 4. Higo further discloses generating the grabbing template (Paragraph 0030; a suction target map is generated), but fails to disclose acquiring a specified virtual image of a virtual model of the 3D object at a visual angle of the specified grabbing position and attitude, simulating multiple different positions and attitudes of the visual sensor relative to the 3D object, and obtaining multiple virtual images of the virtual model of the 3D object at visual angles of the multiple different positions and attitudes, determining a degree of similarity between each of the multiple virtual images and the specified virtual image respectively, and taking a corresponding position and attitude of a virtual image with a degree of similarity higher than a preset threshold to be the reference grabbing position and attitude.
However, Tateno, in an analogous field of endeavor, teaches acquiring a specified virtual image of a virtual model of the 3D object at a visual angle of the specified grabbing position and attitude (Paragraph 0065; a virtual image may be obtained of a virtual model of an object), simulating multiple different positions and attitudes of the visual sensor relative to the 3D object, and obtaining multiple virtual images of the virtual model of the 3D object at visual angles of the multiple different positions and attitudes (Fig. 4, Paragraph 0065; multiple virtual images from different viewing positions and orientations may be obtained by the virtual imaging sensor), determining a degree of similarity between each of the multiple virtual images and the specified virtual image respectively (Paragraphs 0111-0112; viewpoints most similar to a determined virtual imaging viewpoint may be selected based on differences between the positioning and orientation), and taking a corresponding position and attitude of a virtual image with a degree of similarity higher than a preset threshold to be the reference grabbing position and attitude (Paragraphs 0111-0112; position and orientation similarities may be ranked based on the differences themselves; those with the lowest rank sum are selected; the ranks themselves may be considered pre-set thresholds as they are determined before selection).
Therefore, from the teaching of Tateno, it would have been obvious to one of ordinary skill in the art before the effective filing date to have further modified, with a reasonable expectation for success, the robotic system of Higo and Sasaki to include acquiring a specified virtual image of a virtual model of the 3D object at a visual angle of the specified grabbing position and attitude, simulating multiple different positions and attitudes of the visual sensor relative to the 3D object, and obtaining multiple virtual images of the virtual model of the 3D object at visual angles of the multiple different positions and attitudes, determining a degree of similarity between each of the multiple virtual images and the specified virtual image respectively, and taking a corresponding position and attitude of a virtual image with a degree of similarity higher than a preset threshold to be the reference grabbing position and attitude, as taught/suggested by Tateno. The motivation to do so would be to increase the accuracy of the system by evaluating different viewpoints of objects to be grabbed. This can allow the system to determine which objects have a higher likelihood of 1) being the correct object to be grasped and 2) correctly being grasped.
Regarding claim 6, Higo, in view of Sasaki, and further in view of Tateno renders obvious all the limitations of claim 5. Higo further discloses generating the grabbing template (Paragraph 0030; a suction target map is generated), but fails to disclose the degree of similarity comprises a degree of similarity between a characteristic of the virtual image and a characteristic of the specified virtual image. However, Tateno further teaches the degree of similarity comprises a degree of similarity between a characteristic of the virtual image and a characteristic of the specified virtual image (Paragraphs 0111-0112; viewpoints most similar to a determined virtual imaging viewpoint may be selected based on differences between the positioning and orientation; positioning and orientation of the viewpoints may be considered characteristics relative to an image taken at those viewpoints). Therefore, from the teaching of Tateno, it would have been obvious to one of ordinary skill in the art before the effective filing date to have further modified, with a reasonable expectation for success, the robotic system of Higo, Sasaki, and Tateno to include the degree of similarity comprises a degree of similarity between a characteristic of the virtual image and a characteristic of the specified virtual image, as taught/suggested by Tateno. The motivation to do so would be to allow the system to determine the best orientation of the robot for grabbing the object.
Regarding claim 10, Higo in view of Sasaki renders obvious all the limitations of claim 1. Higo further discloses establishing an Earth model with a specified grabbing point on a virtual model of the 3D object, wherein the current position and attitude, the specified grabbing position and attitude and the reference grabbing position and attitude are represented by position and attitude parameters in the Earth model (Paragraphs 0045, 0056; 6 DoF coordinate system may be established for the robot and imaging unit; objects may have an associated target suction map which indicates positions that the robot can suction), but fails to disclose establishing an Earth model with a specified grabbing point on a virtual model of the 3D object as a sphere center. However, Tateno teaches establishing an Earth model with a specified grabbing point on a virtual model of the 3D object as a sphere center (Fig. 4, Paragraph 0065; an object may be at the center of a virtual sphere for establishing viewpoints of the imaging unit; viewpoints have given positioning and orientation relative to the object). Therefore, from the teaching of Tateno, it would have been obvious to one of ordinary skill in the art before the effective filing date to have further modified, with a reasonable expectation for success, the robotic system of Higo and Sasaki to include establishing an Earth model with a specified grabbing point on a virtual model of the 3D object as a sphere center, as taught/suggested by Tateno. The motivation to do so would be to establish positioning and orientation around a given point. This can allow for easier representations in a given frame of reference.
Regarding claim 15, the claim limitations are similar to those in claim 5 and are rejected using the same rationale as seen above in claim 5.
Regarding claim 18, the claim limitations are similar to those in claim 10 and are rejected using the same rationale as seen above in claim 10.
Claims 8-9 and 17 are rejected under 35 U.S.C. 103 as being obvious over Higo, in view of Sasaki, and further in view of Lee et al., US 20210101286 A1, herein referred to as Lee.
Regarding claim 8, Higo in view of Sasaki renders obvious all the limitations of claim 7. Higo further discloses using a masking model to generate a mask of the 3D object based on the real image of the 3D object (Paragraph 0056; a mask may be applied to an image of the 3D object), and obtaining a characteristic of the real image of the 3D object based on the mask of the 3D object (Paragraph 0056; the mask of the 3D object results in areas that are suitable for suction; these areas correspond to pixels within the image mask), but fails to disclose using a Mask-RCNN model to generate a mask of the 3D object based on the real image of the 3D object. However, Lee, in an analogous field of endeavor, teaches using a Mask-RCNN model to generate a mask of the 3D object based on the real image of the 3D object (Paragraphs 0007-0008; an RCNN model may be used to generate a mask for object detection for grasping). Therefore, from the teaching of Lee, it would have been obvious to one of ordinary skill in the art before the effective filing date to have further modified, with a reasonable expectation for success, the robotic system of Higo and Sasaki to include using a Mask-RCNN model to generate a mask of the 3D object based on the real image of the 3D object, as taught/suggested by Lee. The motivation to do so would be to use a well-known method for object determination using image masking.
Regarding claim 9, Higo, in view of Sasaki, and further in view of Lee renders obvious all the limitations of claim 8. Higo further discloses the degree of similarity comprises a degree of similarity between the characteristic of the real image and a characteristic of the virtual image (Paragraph 0054; an image is compared to a template to determine similarities; the template comprises virtual images as the template is stored; similarities may be determined based on image features).
Regarding claim 17, the claim limitations are similar to those in claim 8 and are rejected using the same rationale as seen above in claim 8.
Response to Arguments
Applicant's arguments filed 03/05/2026 have been fully considered but they are not persuasive.
Applicant is arguing that the prior art combination fails to teach the claim limitations. Specifically, Applicant is arguing that the primary art of Higo only discloses performing a suction process in a vertical manner (specifically referring to Fig. 1). However, at least 0070 discloses that a suction position and orientation is input to the actuator control unit as a position (and orientation) when attempting to perform a suction process on an object. Additionally, Fig. 1 is only meant to show the general configuration of the system (see 0031) and is not intended to be a limiting embodiment on the functioning of the robotic arm; even further, Fig. 1 is a simplified figure and conclusion about the functionality of the robotic arm cannot be inferred from strictly an image. Furthermore, nothing in Higo discloses that the robot can only move vertically and/or side-to-side. In fact, at least 0045 discloses that 6 degrees of freedom are used for calibration of the actuation unit for performing a suction process, meaning that the orientation of the suction unit is taken into account and utilized when performing a suction process on an object. Examiner also notes that even if one were to consider the arm to only move vertically and side-to-side, the actuator being in a vertical orientation is in of itself an attitude of the actuator. This in effect means that any suction process being performed is at a given position and attitude.
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to CHRISTOPHER ALLEN BUKSA whose telephone number is (571)272-5346. The examiner can normally be reached M-F 7:30 AM-4:30 PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Thomas Worden can be reached at (571) 272-4876. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/CHRISTOPHER A BUKSA/Examiner, Art Unit 3658
Prosecution Insights