DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
Joint Inventors
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Foreign Priority
Receipt is acknowledged of certified copies of papers required by 37 CFR 1.55. As such, the effective filing date is April 18th, 2025 (the filing date of KR 10-2025-0050899, which applicant requested foreign priority to).
Information Disclosure Statement
The information disclosure statement (IDS) submitted on December 16th, 2025 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Claim Objections
Claims 2, 10-11 and 20 are objected to because of the following informalities:
Claim 2 Line 1: “wherein;” should be revised to “wherein:”
Claim 2 Line 3: “camera comprises” should be revised to “the camera comprises”
Claim 10 Line 3: “processor an a memory” should be revised to “processor and a memory”
Claim 11 Line 3: “camera comprises” should be revised to “the camera comprises”
Claim 20 Line 1: “wherein;” should be revised to “wherein:”
Claim 20 Line 3: “camera comprises” should be revised to “the camera comprises”
Appropriate correction is required.
Claim Rejections - 35 USC § 112
Claim 7 is rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 7 recites the limitations "the first disparity map" and “the second disparity map”. There is insufficient antecedent basis for this limitation in the claim.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1-20 are rejected under 35 U.S.C. 102(a)(1)/(a)(2) as being anticipated by Kollar et al. (US Patent Pub. No. 2023/0398692 A1), herein “Kollar”.
Regarding Claims 1, 10 and 19, Kollar discloses a method, apparatus, and computing device for determining a grasping point for an object in a manipulator system (See 0095, “[…] include producing grasp positions according to the OBB predictions. The method 900 may further include grasping the unknown object based on the grasp positions.”), comprising:
(per Claim 10 only) a computing device comprising a processor and a memory (See 0026, “Instructions executed at a processor (e.g., CPU 102) may be loaded from a program memory associated with the CPU […]”);
acquiring image data and RGB image data regarding a target object using a camera, wherein the camera is a component of a manipulator (See 0038, “The first camera 304 and the second camera 306 may be coupled to the robot 350 or may be in communication with the robot 350.” See also 0054, “[…] receives a video stream from the first camera 304 and the second camera 306. The video stream may include a 2D RGB left image from the first camera 304 and a 2D RGB right image from the second camera 306 to provide a stereo pair of video frame images. The video stream may include multiple frames, such as image frames.”); and
using a computing device:
determining a disparity map based on the image data (See 0004, “[…] generating a low-resolution disparity image based on the features extracted from each image of the synthetic stereo pair of images. The method further includes generating, by the neural network, a feature map based on the low-resolution disparity image and one of the synthetic stereo pair of images.” See also 0060, “[…] output of the SCVN 420 is a low-resolution disparity image 430. In this configuration, the low-resolution disparity image 430 is fed in with features extracted from the left stereo RGB image […]”);
determining detection information for the target object based on the RGB image data (See 0053, “[…] the robot perception module 310 may receive RGB video image data directly from the first camera 304 or the second camera 306 to perform oriented bounding box (OBB) labeling of unknown objects from images captured by the first camera 304 and the second camera 306 of the robot […]”);
determining matching depth information for the target object based on the disparity map and the RGB image data (See 0039, “[…] oriented bounding box (OBB) labeling of 3D objects within a scene is an important perception task in the area of autonomous agents […] a disparity (or horizontal difference in the pixel coordinates) of the single point can be directly mapped to depth. These aspects of the present disclosure rely on stereo vision to perform stereo matching for predicting depth images using a differentiable cost volume neural network that matches features in a pair of stereo images.” See also 0077, “[…] the SCVN 420 produces the low-resolution disparity image 430 at a quarter resolution, the feature extraction backbone 440 can combine the backbone and the left stereo RGB image 402 to produce a full resolution depth image. The same branch architecture as the previous heads is used to aggregate information across different scales of the full-resolution disparity prediction head […]”); and
determining a grasping point for the target object using the detection information and the matching depth information (See 0067, “[…] features are extracted from the ResNet stem, concatenated with the low-resolution disparity image 430 output of the SCVN 420, and fed into the feature extraction backbone […]” See also 0081, “[…] grasp positions 620 are produced from the OBBs 610 and are used, for example, by a classical planner to direct the robot 350 of FIG. 3 to grasp the unknown objects […]”).
Regarding Claims 2, 11 and 20, Kollar further discloses the object grasping point determining method of claim 1, apparatus of claim 10, and computing device of claim 19, wherein:
the image data comprises infrared (IR) image data (See 0059, “[…] a left stereo RGB image 402 and a right stereo RGB image […]” See also 0111, “[…] such as infrared (IR) […]”), and
camera comprises a stereo camera (See 0054, “[…] a stereo pair of video frame images.”).
Regarding Claims 3 and 12, Kollar further discloses the object grasping point determining method of claim 2 and apparatus of claim 11, wherein:
the target object comprises an object (See 0039 and 0053 as referenced above), and
the object comprises a transparent or highly reflective surface (See 0082, “[…] two classes of objects include: (1) optically easy, which is composed of opaque, non-reflective objects; and (2) optically hard, which is composed of optically challenging, transparent objects.”).
Regarding Claims 4 and 13, Kollar further discloses the object grasping point determining method of claim 2 and apparatus of claim 11, wherein:
the IR image data comprises first IR image data and second IR image data (See 0059 as referenced above), and
the determining the disparity map comprises inputting the first IR image data and the second IR image data into a stereo matching model to determine the disparity map (See 0056, “[…] 3D object manipulation method involves training of a neural network to rely on stereo vision for performing stereo matching to predict depth images using a differentiable cost volume (DCVS) neural network that matches features in a pair of stereo images.” See also 0062, “[…] performs learned stereo matching to generate the low-resolution disparity image […]”).
Regarding Claims 5 and 14, Kollar further discloses the object grasping point determining method of claim 4 and apparatus of claim 13, wherein:
the determining the disparity map comprises:
extracting a first feature map for the target object from the first IR image data (See 0059-0060, “[…] a left stereo RGB image 402 and a right stereo RGB image 404 are fed into a left feature extractor 410 and a right feature extractor […] left feature extractor 410 and the right feature extractor 414 are implemented using neural networks (e.g., Φ1 and Φr) trained to identify features of each image and output feature volumes […]”);
extracting a second feature map for the target object from the second IR image data (See 0059-0060 as referenced above);
generating a cost volume based on a similarity calculation for the first feature map and the second feature map (See 0060, “[…] output feature volumes ϕ1 412 and ϕ r 416 are fed into a stereo cost volume network (SCVN) 420, which performs approximate stereo matching between output feature volumes […]” See also 0063-0064, “[…] computes pixel-wise dot products between horizontally shifted versions of the output feature volumes […] horizontally searches for matches across the output feature volumes […]”); and
determining the disparity map based on pixel-wise matching of the first feature map and the second feature map performed using the cost volume (See 0063-0064 as referenced above and, “[…] feeds the resulting volume into a sequence of ResNet blocks, which outputs a volume of dimension Cc×Hϕ×Wϕ before performing a soft argmin along the first axis of the volume. The soft argmin operation approximately finds the disparity for each pixel by locating its best match.”).
Regarding Claim 6, Kollar further discloses the object grasping point determining method of claim 5, wherein the determining the disparity map based on the pixel-wise matching of the first feature map and the second feature map comprises optimizing pixel-wise matching at different resolutions based on a convolution calculation or a deconvolution calculation performed on the cost volume (See 0060 as referenced above. See 0062, “[…] using a lightweight Dilated ResNet-FPN as the feature extractor, to enable large receptive fields with a minimal amount of convolutional layers.” See 0064, “[…] feeds the resulting volume into a sequence of ResNet blocks […]“ See also 0068, “[…] the output prediction heads use an up-scaling branch, which aggregates different resolutions across the feature extractor.” See also 0077, “[…] same branch architecture as the previous heads is used to aggregate information across different scales of the full-resolution disparity prediction head […]”).
Regarding Claim 7, Kollar further discloses the object grasping point determining method of claim 4, wherein the stereo matching model is learned to minimize the loss function for matching the first disparity map and the second disparity map, which are respectively output by receiving the first IR image data and the second IR image data as inputs (See 0065, “[…] weights of Φl, Φr, and fcost are trained by minimizing an auxiliary depth reconstruction loss function. In particular, the loss function takes in a target disparity image Itarg,d of dimension H0×W0, downsamples it by a factor of H0/Hϕ and then computes the Huber loss […]” See also 0077, “[…] the same loss as the SCVN 420 is used, but enforced at full resolution. For example, the full-resolution disparity prediction head 480 is trained using a Huber loss function […]”).
Regarding Claims 8 and 15-16, Kollar further discloses the object grasping point determining method of claim 2 and apparatus of claims 11 and 15, wherein:
the determining the detection information comprises determining a first center point of the target object by inputting the RGB image data into an object detection model (See 0053, “[…] receive RGB video image data directly from the first camera 304 or the second camera 306 to perform oriented bounding box (OBB) labeling of unknown objects […]” See also 0069-0070, “[…] a Gaussian heatmap is predicted for each object […] the distance from the camera […] of the box centroid is regressed […]”), and
the determining the matching depth information comprises:
determining depth image data for the target object based on the disparity map (See 0039 and 0077 as referenced above);
converting the depth image data based on a coordinate system of the RGB image data (See 0070, “[…] pixel-wise offset from detected peak to the 8 box vertices projected on to the image.” See also 0077, “[…] full-resolution disparity image 482 can be converted into a 3D point cloud […]”); and
aligning depth image data closest to the target object among the converted depth image data to the RGB image data (See 0070 as referenced above and, “[…] predicted OBBs 462 are aligned based on principal axes sized in a fixed reference frame.” See also 0082, “[…] aligns the gripper with the largest principal axis of the OBBs 610. In the event of similar sized principal axes like a ball, the robot 350 favors grasping the object on the side closest to the robot […]”).
Regarding Claims 9 and 17-18, Kollar further discloses the object grasping point determining method of claim 8 and apparatus of claims 16-17, wherein the determining the grasping point for the target object comprises:
determining a second center point for the target object corresponding to the first center point based on the matching depth information (See 0070 as referenced above. See also 0063, “[…] difference in the x coordinates in the match, which is high for closer points in 3D space and low for farther points.” See also 0095, “[…] detecting keypoints of objects in the synthetic stereo pair of images detected from the feature map. The method 900 may also include planning an object grasp by a robot according to object grasp predictions […]” Examiner notes a plurality of points are determined for the object); and
determining the second center point as a grasping point for the target object based on verification of validity of the second center point (See 0070, 0082 and 0095 as referenced above), and
further comprising verifying the validity of the second center point (See 0071, “[…] the loss is only enforced when the Gaussian heatmaps have scored greater than 0.3 to prevent ambiguity in empty space.”),
wherein the verifying comprises comparing an average value of a depth of a 3D point cloud included within a predetermined distance from the second center point with a preset threshold value (See 0071, “[…] when the Gaussian heatmaps have scored greater than 0.3 […]” See also 0077, “[…] full-resolution disparity image 482 can be converted into a 3D point cloud […]”).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure: Correll et al. (US Patent Pub. No. 2019/0381670 A1), which is directed towards a robot gripper using 3D perception to determine physical object parameters for control and movement.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Bryant Tang whose telephone number is (571)270-0145. The examiner can normally be reached M-F 8-5 PST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Thomas Worden can be reached at (571)272-4876. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/BRYANT TANG/Examiner, Art Unit 3658
/THOMAS E WORDEN/Supervisory Patent Examiner, Art Unit 3658