DETAILED ACTION
Remarks
This final office action is in response to the amendments filled on 04/07/2026. Claims 1, 14 and 15 are amended. Claims 1-20 are pending and examined below.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION. —The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claim(s) 14 is/are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor, or for pre-AIA the applicant regards as the invention.
Regarding claim 14, which recites “a method according to claim 1” is unclear and indefinite since claim 1 also mentioned a 3D physical object and a reference volume. It is not clear both physical objects and reference volume of claim 1 and 14 are same or different. Examiner suggests that Applicant make Claim 14 separate and incorporate the text of Claim 1 into it instead of referring to Claim 1 in order to avoid antecedent and indefinite issues.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1, 2, 4, 6, 7, 11, 12, 14-16 and 19 is/are rejected under 35 U.S.C. 103 as being unpatentable over NPL, title “Image-based Object Recognition and Dexterous Hand/ Arm Motion Planning Using RRTs for Grasping in Cluttered Scene”, by (“Hirano”), and in view of US 2022/0016765 (“Ku”), and further in view of US 2001/0005204 (“Matsumoto”).
Regarding claim 1, Hirano discloses a method for generating a technical instruction for handling a three-dimensional physical object present within a reference volume, the physical object comprising a 3D surface (see at least abstract; see also fig 4 and fig 8, where 3D surfaces of objects are shown. The measurements and detection are done in a real environment which is within a defined volume. Defined real world environment is interpreted as reference volume), the method comprising:
- obtaining at least two images of the physical object from a plurality of cameras positioned at different respective angles with respect to the physical object (see at least page 2, col 1, where “As shown in Fig. 1, at first , a pair of stereo images is taken at a starting position. Then the stereo cameras are moved to get other pair of images from different viewpoints. At the same time, by tracking feature points in the images, camera motion from the previous viewpoint is estimated.”);
- performing a measurement with respect to the plurality of first voxels (see at least page 3, col 2, section 3, at fig 6 first photograph showing the locus of points); and
- computing the technical instruction, said technical instruction preferably comprising a robot command, for the handling of the physical object based on the segmented voxel representation and the measurement (see at least fig 8, where robot hand grasping object; see also page 3, col 2, section III where, “Fig. 8 shows some variations of grasping attitude of the robot hand. One axis of the hand coordinate system {Xh, Yh, Zh} is settled to be parallel to the main axis of the object as shown in Fig 8. Once the orientation of the hand coordinate frame in the grasping attitude is decided, then the inverse kinematics problem is solved to decide collision-free arm attitude. This arm attitude corresponds to one goal point in the arm joint configuration space.”).
Hirano does not disclose the following limitations:
- generating, with respect to the 3D surface of the physical object, a voxel representation segmented based on the at least two images, said segmenting of the voxel representation comprising identifying a first segment component corresponding to a plurality of first voxels and a second segment component corresponding to a plurality of second voxels different from the plurality of first voxels; and
wherein said segmenting relates to at least one trained neural network (NN) being trained with respect to the 3D surface.
However, Ku discloses method wherein generating, with respect to the 3D surface of the physical object, a voxel representation segmented based on the at least two images, said segmenting of the voxel representation comprising identifying a (see at least fig 2, scene parsing module 320 and described in [0030] using machine learning; see also [0099] disclosing that a voxel representation of the image to determine a grasp location: " a predetermined grid (e.g., with a predetermined cell size, cell distribution, etc.) can be overlayed on an image and/or point cloud of the scene to subdivide the scene into a set of image segments and/or voxels. In this example, a candidate grasp can be determined for each grid cell (e.g., based on the image segment, using methods discussed above), wherein a final grasp can be selected from the resultant set of candidate grasps.").
wherein said segmenting relates to at least one trained neural network (NN) being trained with respect to the 3D surface (see at least [0030]).
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to modify the robot control as taught by Hirano with the scene parser using machine learning taught by Ku with a reasonable expectation of success in order for the robot to recognize an object and the grasping of the object in a reference space or volume. The teaching suggestion/motivation to combine is that by using a machine learning process to determine a grasping point, increases the probability of grasp success as taught by Ku in Para. [0094].
Hirano in view of Ku does not disclose the following limitation:
said segmenting of the voxel representation comprising identifying a first segment component corresponding to a plurality of first voxels and a second segment component corresponding to a plurality of second voxels different from the plurality of first voxels.
However, Matsumoto discloses method wherein said segmenting of the voxel representation comprising identifying a first segment component corresponding to a plurality of first voxels and a second segment component corresponding to a plurality of second voxels different from the plurality of first voxels (see at least [0374] and [0389]).
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to have modified Hirano in view of Ku to incorporate the teachings of Matsumoto by including the above feature for feature for determining accurate shape model of the object.
Regarding claim 2, Hirano further discloses a method wherein the technical instruction comprises a robot command, wherein the robot command is executable by means of a device comprising a robot element configured for handling the physical object (see at least fig 8).
Regarding claim 4, Matsumoto further discloses a method wherein said segmenting comprises identifying a third segment component corresponding to a plurality of third voxels comprised in the plurality of first voxels, wherein at least one of the first voxels does not belong to the third voxels (see at least [0374], where “an object image that is the average of the plurality of object images is segmented into a plurality of regions R at step S551.”; see also [0389], where “Referring to FIG. 36, cylindrical coordinate system voxel space 251 includes a plurality of voxels 39.”), wherein the measurement is performed with respect to the plurality of third voxels (see at least fig 31 and fig 36).
Regarding claim 6, Hirano further discloses a method wherein performing a 3D reconstruction of the 3D surface of the physical object based on the at least two images for obtaining a voxel representation (see at least page 2, section A), and
- obtaining said segmented voxel representation by projecting at least the first segment component with respect to said voxel representation (see at least fig 4 and fig 8).
Ku further discloses a method wherein the at least one trained NN comprises an instance segmentation NN, being a two-dimensional (2D) and/or 3D region-based convolutional neural network (R-CNN) or being a Mask R-CNN for segmenting the at least two images and/or a 3D-BoNet for segmenting the voxel representation (see at least [0030], [0049] and [0099]).
Regarding claim 7, Hirano further discloses a method wherein performing a 3D reconstruction of the 3D surface of the physical object based on the at least two images for obtaining a voxel representation (see at least fig 3), and
- obtaining said segmented voxel representation by projecting at least the first segment component with respect to said voxel representation (see at least fig 6).
Ku further discloses a method wherein the at least one trained NN comprises a semantic segmentation NN, being a 2D and/or 3D convolutional neural network, CNN, or being a 2D U-net for segmenting the at least two images and/or a PointNet++ for segmenting the voxel representation (see at least [0030], [0049] and [0099]).
Regarding claim 11, Ku further discloses a method wherein the handling comprises physically separating a sample from the physical object at a handling coordinate based on said measurement (see at least fig 1, [0033] and [0060]).
Regarding claim 12, Ku further discloses a method wherein a 3D approaching angle for reaching the handling coordinate on said physical object relates to a 3D sampling angle for separating the sample at the handling coordinate (see at least [0097]).
Regarding claim 14, as best understood in view of indefiniteness rejection explained above, Hirano in view of Ku and Matsumoto further discloses a device for handling a three-dimensional, 3D, physical object present within a reference volume, the physical object comprising a 3D surface, the device comprising a robot element, a processor and memory comprising instructions which, when executed by the processor, cause the device to execute a method according claim 1 for handling said 3D physical object (see citation on claim 1).
Regarding claim 15, Hirano in view of Ku and Matsumoto further discloses claim 15 (see citation on claim 1 and 2).
Regarding claim 16, Hirano in view of Ku and Matsumoto further discloses a non-transitory computer readable medium containing a computer executable software which when executed on a device, performs the method of claim 1 (the method of Hirano is perform on a computer. see citation on claim 1).
Regarding claim 19, Chen further discloses a method wherein obtaining of said segmented voxel representation comprises performing clustering with respect to a projected at least first segment component (see at least fig 3 , page 2 section 2 and page 3 section B).
Claim(s) 3 is/are rejected under 35 U.S.C. 103 as being unpatentable over Hirano, Ku and Matsumoto, as applied to claim 1 above, and further in view of US 2020/0211206 (“Wang”).
Regarding claim 3, Hirano in view of Ku and Matsumoto does not disclose claim 3. However, Wang discloses a method comprising: pre-processing of the at least two images based on a mask projection for distinguishing foreground from background, said mask projection being based at least partially on a mask-related 3D reconstruction of the 3D surface of the physical object (see at least [0115], where “The moving object mask is provided as a binary map to distinguish between static background and moving foreground in flow evaluation.”; see also [0025] and [0149]).
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to have modified Hirano in view of Ku and Matsumoto to incorporate the teachings of Wang by including the above feature for determining accurate geometry and 3D motion shape model of the object.
Claim(s) 5, 17 and 18 is/are rejected under 35 U.S.C. 103 as being unpatentable over Hirano, Ku and Matsumoto, as applied to claim 1 above, and further in view of US 2018/0286117 (“Redden”).
Regarding claim 5, Hirano in view of Ku and Matsumoto does not disclose claim 5. However, Redden discloses a method wherein the 3D surface of the physical object is a plant comprising, one or more leaves, corresponding to the first segment component (see at least [0013-18]).
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to have modified Hirano in view of Ku and Matsumoto to incorporate the teachings of Redden by including the above feature for providing wide variety of object recognition.
Regarding claim 17, Hirano in view of Ku and Matsumoto does not disclose claim 17. However, Redden further discloses a method wherein one or more stems corresponding to the third segment component (see at least fig 1A).
Regarding claim 18, Redden further discloses a method wherein the plant comprises soil and/or one or more roots, corresponding to the second segment component (see at least fig 1A, where skeleton segment include ground plane).
Claim(s) 8, 9 and 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Hirano, ku and Matsumoto, as applied to claim 1 above, and further in view of ARXIV.org, Cornell university library, title: “Vision-based Robotic Grasping from Object Localization, Object Pose Estimation to Grasp Estimation for Parallel Grippers: A Review” by (“Du”), submitted as IDS.
Regarding claim 8, Hirano in view of Ku and Matsumoto does not disclose claim 8. However, Du discloses a method wherein the measurement relates to counting with respect to the segmented voxel representation, and wherein the segmented voxel representation is obtained via semantic segmentation for counting clusters of voxels and/or instance segmentation for counting instances (see at least page 9, section 2.3).
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to have modified Hirano in view of Ku and Matsumoto to incorporate the teachings of Du by including the above feature for providing slip and damage free grasp during object handling by localizing object accurately by counting clusters of voxels.
Regarding claim 9, Hirano in view of Ku and Matsumoto does not disclose claim 9. However, Du further discloses a method wherein the measurement comprises determining any one or combination of: a number of elements, an area and a volume of said segment component based on counting with respect to the segmented voxel representation (see at least page 9, section 2.3 and fig 9).
Regarding claim 20, Hirano in view of Ku and Matsumoto does not disclose claim 20. However, Du further discloses a method wherein the measurement comprises determining any one or combination of: a height and an angle of said segment component with respect to a main direction comprised in the reference volume based on counting of a plurality of voxels associated with said segment component along the main direction representation (see at least page 9, section 2.3 and fig 15).
Claim(s) 10 is/are rejected under 35 U.S.C. 103 as being unpatentable over Hirano, Ku and Matsumoto, as applied to claim 1 above, and further in view of US 2021/0122586 (“Sun”).
Regarding claim 10, Hirano further discloses a method wherein the handling comprises (see at least fig 4 and fig 8).
Hirano in view of Ku and Matsumoto does not disclose the following limitation:
physically sorting the physical object according to respective physical destination locations corresponding to respective classes.
However, Sun discloses a method wherein physically sorting the physical object according to respective physical destination locations corresponding to respective classes (see at least [0045], where “use the routing information to sort the item to a corresponding destination, such as a pile, bin, or other set of items destined for the same next intermediate and/or final destination.”).
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to have modified Hirano in view of Ku and Matsumoto to incorporate the teachings of Sun by including the above feature for increasing efficiency by placing object based on final destination.
Claim(s) 13 is/are rejected under 35 U.S.C. 103 as being unpatentable over Hirano, Ku and Matsumoto, as applied to claim 12 above, and further in view of US 2022/0072707 (“Fan”).
Regarding claim 13, Hirano in view of Ku and Matsumoto does not disclose claim 13. However, Fan discloses a method comprising: actuating a robot element based on a robot command, wherein said actuating comprises (see at least fig 4):
-approaching, by the robot element, the 3D surface at a 3D approaching angle (see at least [0024] and [0061]); and
-separating, by the robot element, the sample from the physical object at the handling coordinate (see at least fig 4),
wherein the step of separating comprises surrounding, by two distal ends of the robot element, a receiving portion of the physical object at the 3D sampling angle, wherein the 3D sampling angle relates to an orientation of the two distal ends of the robot element with respect to a main plane of the receiving portion (see at least fig 4 and [0046]).
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to have modified Hirano in view of Ku and Matsumoto to incorporate the teachings of Fan by including the above feature for generating high quality grasp candidates without teaching manually.
Non-Statutory Double Patenting
The nonstatutory double patenting rejection is based on a judicially created doctrine grounded in public policy (a policy reflected in the statute) so as to prevent the unjustified or improper timewise extension of the “right to exclude” granted by a patent and to prevent possible harassment by multiple assignees. A nonstatutory double patenting rejection is appropriate where the conflicting claims are not identical, but at least one examined application claim is not patentably distinct from the reference claim(s) because the examined application claim is either anticipated by, or would have been obvious over, the reference claim(s). See, e.g., In re Berg, 140 F.3d 1428, 46 USPQ2d 1226 (Fed. Cir. 1998); In re Goodman, 11 F.3d 1046, 29 USPQ2d 2010 (Fed. Cir. 1993); In re Longi, 759 F.2d 887, 225 USPQ 645 (Fed. Cir. 1985); In re Van Ornum, 686 F.2d 937, 214 USPQ 761 (CCPA 1982); In re Vogel, 422 F.2d 438, 164 USPQ 619 (CCPA 1970); In re Thorington, 418 F.2d 528, 163 USPQ 644 (CCPA 1969).
A timely filed terminal disclaimer in compliance with 37 CFR 1.321(c) or 1.321(d) may be used to overcome an actual or provisional rejection based on nonstatutory double patenting provided the reference application or patent either is shown to be commonly owned with the examined application, or claims an invention made as a result of activities undertaken within the scope of a joint research agreement. See MPEP § 717.02 for applications subject to examination under the first inventor to file provisions of the AIA as explained in MPEP § 2159. See MPEP §§ 706.02(l) (1) - 706.02(l) (3) for applications not subject to examination under the first inventor to file provisions of the AIA . A terminal disclaimer must be signed in compliance with 37 CFR 1.321(b).
The USPTO Internet website contains terminal disclaimer forms which may be used. Please visit www.uspto.gov/patent/patents-forms. The filing date of the application in which the form is filed determines what form (e.g., PTO/SB/25, PTO/SB/26, PTO/AIA /25, or PTO/AIA /26) should be used. A web-based eTerminal Disclaimer may be filled out completely online using web-screens. An eTerminal Disclaimer that meets all requirements is auto-processed and approved immediately upon submission. For more information about eTerminal Disclaimers, refer to www.uspto.gov/patents/process/file/efs/guidance/eTD-info-I.jsp.
Claim(s) 1, 6, 7 and 14-16 is/are provisionally rejected on the ground of nonstatutory double patenting as being unpatentable over claim 1, 8, 9 and 10 of copending US Application No. 18/550,946.
This is a provisional non-statutory double patenting rejection since the claims directed to the reference patent have not in fact been patented. Although the claims at issue are not identical, they are not patentably distinct from each other. Table below shows the claim matching between two applications:
Application No.
claims
claim
claim
18/550,948
1,14,15,16
6
7
18/550,946
1,14
8
9,10
Response to Arguments
Applicant’s arguments with respect to claim 1-20 have been considered but are moot because the arguments do not apply to the new combination used in the current rejection that is due to the newly added claim amendments.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SOHANA TANJU KHAYER whose telephone number is (408)918-7597. The examiner can normally be reached on Monday - Thursday, 7 am-5.30 pm, PT.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Abby Lin can be reached on 571-270-3976. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see https://ppair-my.uspto.gov/pair/PrivatePair. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/SOHANA TANJU KHAYER/Primary Examiner, Art Unit 3657
Prosecution Insights