Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
Applicant's arguments have been considered. The rejection below uses new references to address the limitations that the applicant argued.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 21-23, 26, 28, 30-33 and 42 are rejected under 35 U.S.C. 103 as being unpatentable over Grabner (20190147221) in view of Rhemann (20140241612) in view of Wiedemann (20090096790) in further view of Ueno (20190355140).
Regarding claim 21, Grabner teaches a method for determining a pose of an object, the method comprising: obtaining plural images of a field of view comprising one or more objects (pars. 9-10),
processing at least a first image of the plural images to identify a candidate object from among the one or more objects in the first image and to determine a search range corresponding to the candidate object (pars. 9-10 and par. 163);
Rhemann teaches wherein the plural images are obtained from plural viewpoints, and performing stereo matching between the first image of the plural images and a second image of the plural images to determine an accurate pose of the candidate object (fig. 1, pars. 2, 9 and 24, stereo matching of a stereo image pair taken from different viewpoints to find disparities and depths).
It would have been obvious before the effective filing date of the claimed invention to one of ordinary skill in the art to combine the stereo matching of Rhemann with the pose-determination method of Grabner in order to determine an accurate pose of an object from plural viewpoints.
Wiedemann teaches wherein processing the at least a first image comprises determining a coarse pose of the candidate object, the coarse pose comprising a coarse 3D orientation of the candidate object (pars. 13, 17-18 and 67-68 samples a range of object poses expressed as intervals of the spherical coordinates longitude, latitude and distance, and camera roll, to create views over the levels of an image pyramid)
obtains an initial coarse 3D object pose (pars. 14, 20 and 90), and
wherein the subsequent finer search is based at least in part on the coarse pose, the finer match being restricted to the child views of, and a close neighborhood of, the coarse-level match (pars. 20-21 and 80-81).
It would have been obvious before the effective filing date of the claimed invention to one of ordinary skill in the art to determine a coarse pose comprising a coarse 3D orientation as taught by Wiedemann in Rhemann and Grabner. The reason is start with less processing power.
Ueno teaches wherein an object-specific prior obtained from processing the first image is used to define the search range to which the stereo matching is limited (Ueno determines, for a detected object, a maximum disparity see FIGS. 2-3A and pars. 13-14).
It would have been obvious before the effective filing date of the invention to one of ordinary skill in the art to in include in Rhemann, Grabner and Wiedeann to ability to use an object-specific prior to bound the disparity search range of the stereo cost function as taught by Uneo. The reason is to reduce search space and memory.
Regarding claim 22, see Rhemann pars. 46-47 and Grabner par. 60, processing the second image to identify the candidate object in the second image.
Regarding claim 23, see Grabner pars. 143-145, proposing a candidate bounding box for each object and performing bounding box regression to determine bounding boxes.
Regarding claim 26, see Grabner pars. 163-165, which teach determining a 2D center (i.e., a 2D pixel-space origin) of the target object.
Regarding claim 28, see Rhemann par. 44, tiling the first image and performing the stereo matching for tiles that include the candidate object.
Regarding claim 30, see Rhemann pars. 46-47, locating the candidate object in the first image, defining a match region in the second image based on that location, and searching the match region.
Regarding claim 31, see Grabner pars. 71-72, defining a bounding box for each of a plurality of objects and selecting a matching bounding box based on a similarity score.
Regarding claim 32, see Rhemann par. 32, the match region determined based on a minimum and maximum depth of the scene.
Regarding claim 33, see Rhemann pars. 24 and 45-47, generating a match probability, selecting a disparity, and computing depths by triangulation using stereo calibration parameters.
Regarding claim 42, see the rejection of claim 21; Grabner further teaches at least one camera arranged to obtain the plural images, a robot arranged to pick up the objects from a target volume and place them in a target area, and a data processor connected to receive the images and configured to process the images and control the robot (pars. 171-174).
Claim 24 is rejected under 35 U.S.C. 103 as being unpatentable over Grabner (20190147221) in view of Rhemann (20140241612) in view of Wiedemann (20090096790) in view of Ueno (20190355140) and in further view of Koo (20230237765).
Regarding claim 24, Koo teaches wherein the bounding boxes comprise rotated 2D bounding boxes, a rotated 2D bounding box (pars. 18-19 and 70-72).
It would have been obvious before the effective filing date of the claimed invention to one of ordinary skill in the art to represent the bounding box of Grabner as a rotated 2D bounding box as taught by Koo in order to more tightly enclose the candidate object and to better represent its in-image orientation.
Claim 25 is rejected under 35 U.S.C. 103 as being unpatentable over Grabner (20190147221) in view of Rhemann (20140241612) in view of Wiedemann (20090096790) in view of Ueno (20190355140) and in further view of Gernoth (20190080149).
Regarding claim 25, Gernoth teaches wherein processing the at least a first image comprises determining an occlusion value for the candidate object, the occlusion value indicating a degree to which the object is occluded (pars. 43 and 57-58).
It would have been obvious before the effective filing date of the claimed invention to one of ordinary skill in the art to determine an occlusion value as taught by Gernoth in order to allow the system to determine the degree of occlusion of the candidate object.
Claim 27 is rejected under 35 U.S.C. 103 as being unpatentable over Grabner (20190147221) in view of Rhemann (20140241612) in view of Wiedemann (20090096790) in view of Ueno (20190355140) and in further view of Khamis (20200099920).
Regarding claim 27, Khamis teaches downsampling two of the plurality of images to provide corresponding downsampled images and performing stereo matching between the downsampled images (Khamis, see claim 1 which downsamples a first image and a second image to generate first and second reduced-resolution images and matches patches of the first reduced-resolution image to patches of the second reduced-resolution image to generate a coarse depth/disparity map).
It would have been obvious before the effective filing date of the claimed invention to one of ordinary skill in the art to downsample the two images and perform the stereo matching between the downsampled images as taught by Khamis in order to obtain a coarse disparity while reducing the computation of the stereo matching.
Claim 29 is rejected under 35 U.S.C. 103 as being unpatentable over Grabner (20190147221) in view of Rhemann (20140241612) in view of Wiedemann (20090096790) in view of Ueno (20190355140) and in further view of Chen (20190317519).
Regarding claim 29, see Chen pars. 46-50, locating the candidate object in the first and second image, calculating an approximate distance of the candidate object from the cameras, and generating the search range around the approximate distance; see also the rejection of claim 21.
It would have been obvious before the effective filing date of the claimed invention to one of ordinary skill in the art to calculate an approximate distance and generate the search range around it as taught by Chen in order to constrain the correspondence search to the expected location of the object.
Claim(s) 34, 35 and 39 are rejected under 35 U.S.C. 103 as being unpatentable over Grabner (20190147221) in view of Rhemann (20140241612) in view of Wiedemann (20090096790) in view of Ueno (20190355140) and in further view of Tremblay (20190228495).
Regarding claim 34, Tremblay teaches selecting one of the objects to pick and controlling a robot to pick the object (pars. 35 and 50); see also Rhemann par. 24 (robotic system).
It would have been obvious before the effective filing date of the claimed invention to one of ordinary skill in the art to control a robot to pick the selected object as taught by Tremblay in order to instruct the robot to act upon the object whose pose has been determined.
Regarding claim 35, see Grabner pars. 171-174, selecting the one of the plurality of objects based on processing 2D image data of the first image.
Regarding claim 39, see Tremblay figure 4 and pars. 50-51, determining a robot pose to grip the object, determining whether the robot can grip the object, and if so controlling the robot to grip the object using the robot pose.
Claim 36 is rejected under 35 U.S.C. 103 as being unpatentable over Grabner (20190147221) in view of Rhemann (20140241612) in view of Wiedemann (20090096790) in view of Ueno (20190355140) in view of Tremblay (20190228495) and in further view of Truebenbach (20190389062).
Regarding claim 36, Truebenbach teaches selecting the one of the objects based on a multi-step selection, by identifying one or more candidate objects for selection by a robot, validating a feasibility of grasping a first candidate object, and, if the feasibility is not validated, selecting a different grasping point, a different path, or a second candidate object (pars. 4-6).
It would have been obvious before the effective filing date of the claimed invention to one of ordinary skill in the art to select the object based on a multi-step selection as taught by Truebenbach in order to increase the probability of a successful pick.
Claim 41 is rejected under 35 U.S.C. 103 as being unpatentable over Grabner (20190147221) in view of Rhemann (20140241612) in view of Wiedemann (20090096790) in view of Ueno (20190355140) in view of Tremblay (20190228495) and in further view of Ban (20070274812).
Regarding claim 41, Ban teaches determining and storing information for each of a plurality of piled workpieces, judging whether the stacked condition has changed as a result of a picking operation, and, for a workpiece whose surroundings are unchanged, controlling the robot to pick that workpiece using the stored information from the previous operation (pars. 9-15 and 23-27).
It would have been obvious before the effective filing date of the claimed invention to one of ordinary skill in the art to determine and cache the poses of a plurality of objects, acquire further images to detect changes, and pick an object that is not affected by the changes using its corresponding cached pose as taught by Ban in order to avoid redundant pose computation and reduce cycle time.
Allowable Subject Matter
Claims 37, 38 and 40 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
The prior art of record does not teach: for claim 37, a multi-step sorting comprising sorting based on one or more of mask size, occlusion classification, and the closeness between the coarse pose and one or more preferred robot orientations; for claim 38, a multi-step sorting comprising sorting based on a closeness of a 2D shape corresponding to a robot gripper gripping the object to an edge of a mask of the object or to a mask of one or more other objects; and for claim 40, determining the pose of the object asynchronously with controlling the robot to pick the object.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure: Rzeszutek (8867826) teaches coarse-to-fine disparity estimation on a stereo image pyramid formed by subsampling both stereo images.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to HADI AKHAVANNIK whose telephone number is (571)272-8622. The examiner can normally be reached 9 AM - 5 PM Monday to Friday.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Henok Shiferaw can be reached at (571) 272-4637. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/HADI AKHAVANNIK/ Primary Examiner, Art Unit 2676
Prosecution Insights