DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-4, 6-7, and 13-16 is/are rejected under 35 U.S.C. 103 as being unpatentable over Kanemoto (WO 2023/034533 A1) in view of Su (US 2023/0290090) and further in view of Zhang (“Finding Planes and Clusters from 3D Line Segments for 3D Motion Determination”, published 2004). Relative to claims 1-4, 6-7, and 13-14, Kanemoto discloses:
claim 1) A computer implemented method for improving computer vision based object identification for automated robotic picking operations (Para. 0009), the computer implemented method comprising:
obtaining a first image of a pick area from a first camera (cameras, such as 320-1), the pick area associated with a robotic picking unit (100)(Para. 0019), the first camera capturing the first image from a first perspective, the first image comprising two dimensional (2D) data and three dimensional (3D) data (cameras capture first image such as image 400-1; Para. 0048; 0021)(Fig. 4A);
obtaining a second image of the pick area from a second camera (such as Ref. 302-2), the second camera capturing the second image from a second perspective, the
second perspective is different than the first perspective, the second image comprising 2D data and 3D data (second camera 320-2 can capture second image from a different perspective, Para. 0048)(Fig. 4B);
processing the first image by executing an object detection algorithm on the first image to generate a first pixel mask, the first pixel mask indicating a plurality of detected objects in the first image (Para. 0049; 0056, key points are generated from pixel);
processing the second image by executing an object detection algorithm on the second image to generate a second pixel mask, the second pixel mask indicating a plurality of detected objects in the second image (Para. 0049; 0056);
extracting depth data from the 3D data associated with the first image for each detected object associated with the first pixel mask (see 3D depth values and/or surface normals for 3D points, Para. 0049);
extracting depth data from the 3D data associated with the second image for each detected object associated with the second pixel mask (Para. 0049, depth values are extracted for both first and second images);
grouping criteria configured to group shapes, contours, and/or edges belonging to the same physical object (Para. 0050, see for instance, ..the detected lines 406a-406d are part of an object having a rectangular shape”); and
providing pick instructions to the robotic picking unit based on the corresponding detection information (Para. 0071);
claim 2) the first camera (320-1) is positioned such that the first image comprises a top-down view of the pick objects within the pick area and the second camera (320-2) is positioned such that the second image comprises a side view of the pick objects within the pick area (Para. 0048).
Kanemoto does not expressly disclose:
claim 1) computing a plane for each of the plurality of detected objects using the extracted 3D data; and providing pick instructions to the robotic picking unit based on the at least one set of corresponding planes;
claim 3) computing a plane comprises fitting a plane to the depth data associated with each detected object;
claim 4) computing a plane comprises determining a location of at least one edge of the plane in the 3D data;
claim 5) the grouping criteria comprise planes which share a common edge in the 3D data;
claim 6) computing each plane comprises computing an orthogonal to the plane; or
claim 7) the grouping criteria comprise identifying planes having orthogonals which are perpendicular to each other.
Su teaches:
claim 1) computing a plane for each of the plurality of detected objects using the extracted 3D data (Para. 0043-0044, 3D features are extracted from video, feature points such as edges, points, and planes are detected using the object recognition algorithm),
claim 3) computing a plane comprises fitting a plane to the depth data associated with each detected object (identifies planes as feature points from the 3D depth scan and uses those plane features (walls, floor, ceiling) derived from the depth data to build the 2D or 3D layout; Para. 0044);
claim 4) computing a plane comprises determining a location of at least one edge of the plane in the 3D data (see extracted feature points 204 may include edges captured in the 3D scan (e.g., walls, windows, etc.), and the positions of the edges are used when generating the 2D or 3D layout);
claim 6) computing each plane comprises computing an orthogonal to the plane (Para. 0065, identifies floor plane and distinguishes floor plane as orthogonal to wall planes); and
claim 7) the grouping criteria comprise identifying planes having orthogonals which are perpendicular to each other (Para. 0065).
Su teaches the: computing a plane for each of the plurality of detected objects using the extracted 3D data, fitting a plane to the depth data associated with each object, determining a location of at least one edge of the plane, computing an orthogonal to the plane, and identifying planes having orthogonals that are perpendicular to each other as described above, for the purpose of providing a system and method that can analyze a point cloud or mesh from a device used to scan a 3D space that is clear and more accurate. (Para. 0029; 0030; 0032).
Zhang teaches: identifying at least one set of corresponding planes, the corresponding planes comprising planes satisfying a grouping criteria, the grouping criteria configured to group planes belonging to the same physical object (Zhang identifies sets of corresponding surface regions from 3D line segments satisfying a grouping criterion, the corresponding surface regions correspond to the corresponding planes belonging to the same physical object. See Page 3, Para. 2-3, which provide that segments live in a neighborhood in space and groups segments based on proximity. Proximity of the 3D line segments allows for grouping the scene into clusters each constituting a geometrical compact entity, e.g., a physical object),
for the purpose of navigating a mobile robot in an unknown indoor scene where an arbitrary number of rigid mobile objects may be present. This technology of identifying 3D line segments and clustering them into coplanar facets is well-known in the art.)
It would have been obvious to one of ordinary skill in the art on or before the time of the filing to modify the method of Kanemoto with computing a plane for each of the plurality of detected objects using the extracted 3D data, as taught is Su, for the purpose of providing a system and method that can analyze a point cloud or mesh from scan of a 3D space that is clear and more accurate.
It would have been further obvious to one of ordinary skill in the art on or before the time of the filing to modify the method of Kanemoto in view of Su, to include identifying a set of corresponding planes that comprise planes satisfying a grouping criteria, the grouping criteria groups planes belonging to the same physical object described above, as taught in Zhang, for the purpose of providing a method of navigating a mobile robot in an unknown indoor scene where an arbitrary number of rigid mobile objects may be present.
Relative to claim 1, Kanemoto in view of Su and Zhang does not expressly disclose: providing pick instructions to the robotic picking unit based on the at least one set of corresponding planes.
Kanemoto in view of Su and Zhang teaches: providing pick instructions to the robotic picking unit based on the at least one set of corresponding planes as an obvious matter of design choice.
Kanemoto provides picking instructions to a robotic unit based on detection mask information representing the regions of the scene containing target objects (Para. 0071, Kanemoto). Zhang teaches identifying planes within such regions and clustering them into corresponding planes belonging to a same physical object (Page 3 of Zhang).
Kanemoto, modified as above, can be modified to identify and cluster planes belonging to a same physical object, as taught in Zhang, to allow the system to determine stable pick surfaces and to guide the robot to approach directions more accurately.
Therefore, it would have been obvious to one of ordinary skill in the art on or before the time of the filing to modify Kanemoto in view of Su and Zhang so that the picking instructions are based on the at least one set of corresponding planes to allow the system to determine stable pick surfaces and to approach directions more accurately.
Relative to claims 13-14, Kanemoto in view of Su and Zhang discloses all claim limitations mentioned above, but does not expressly disclose:
claim 13) computing a plane comprises computing a first set of planes associated with the first image data and a second set of planes associated with the second image data; or
claim 14) identifying comprises identifying at least one plane in the first set and at least one plane in the second set that satisfy the grouping criteria.
Kanemoto in view of Su and Zhang teaches: computing a plane comprises computing a first set of planes associated with the first image data and a second set of planes associated with the second image data; and
identifying comprises identifying at least one plane in the first set and at least one plane in the second set that satisfy the grouping criteria, as an obvious matter of design choice.
Kanemoto teaches acquiring first and second image data of an area from different viewpoints and generating respective depth data for each view (Kanemoto, Para. 0048-0049)(Fig. 4A-4B).
Su teaches processing depth data to identify multiple planes representing surfaces in the scene (Para. 0054).
It would have been obvious to one of ordinary skill in the art, to apply Su’s plane extraction to each of Kanemoto’ s first and second depth datasets, to provide a first set of planes associated with the first image data, and a second set of planes associated with the second image data.
Further, once the planes are identified in each view, a person of ordinary skill would have found it obvious to identify corresponding planes across first and second sets, that satisfy a common grouping criteria (e.g., a similar pose, location, etc., with the same object), so as to treat at least one plane in the first set and at least one plane in the second set as representing the same object surface.
Applying the plane extraction to each of the first and second depth datasets to compute a first and second set of planes associated with the image data, and to identify corresponding planes across first and second sets that satisfy a common grouping criteria described above, is routine optimization to one of ordinary skill. For instance, applying the plane extraction to each of the datasets, and identifying corresponding planes that satisfy a common grouping criteria mentioned above, may improve 3D reconstruction of the area and provide better pose estimation for the robot. See MPEP §2144.05
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the method of Kanemoto in view of Su and Zhang, so that computing a plane comprises computing a first set of planes associated with the first image data, and a second set of planes associated with the second image data; and identifying at least one plane in the first and second set that satisfy the grouping criteria since this is considered routine optimization which may improve 3D reconstruction of the area and to provide better pose estimation for the robot.
Claim(s) 15-16 is/are rejected under 35 U.S.C. 103 as being unpatentable over Kanemoto (WO 2023/034533 A1) in view of Su (US 2023/0290090) and further in view of Zhang (“Finding Planes and Clusters from 3D Line Segments for 3D Motion Determination”, published 2004). Relative to claims 15-16, Kanemoto discloses:
a computing system for improving computer vision based object identification for automated robotic picking operations (Para. 0013; 0021), the computing system comprising: at least one computing processor; and memory comprising instructions (Para. 0025), and a non-transitory computer readable medium comprising instructions (Para. 0025) that when executed by a processor enable the processor to:
obtain a first image of a pick area from a first camera, the pick area associated with a robotic picking unit, the first camera capturing the first image from a first perspective, the first image comprising two dimensional (2D) data and three dimensional (3D) data (Para. 0048);
obtain a second image of the pick area from a second camera, the second camera capturing the second image from a second perspective, the second perspective is different than the first perspective, the second image comprising 2D data and 3D data (Para. 0048);
process the first image by executing an object detection algorithm on the first image to generate a first pixel mask, the first pixel mask indicating a plurality of detected objects in the first image (Para. 0049);
process the second image by executing an object detection algorithm on the second image to generate a second pixel mask, the second pixel mask indicating a plurality of detected objects in the second image (Para. 0049);
extract depth data from the 3D data associated with the first image for each detected object associated with the first pixel mask (Para. 0049);
extract depth data from the 3D data associated with the second image for each detected object associated with the second pixel mask (Para. 0049); and
provide pick instructions based on the detected features to the robotic picking unit (Para. 0071).
Kanemoto does not expressly disclose:
computing a plane for each of the plurality of detected objects using the extracted 3D data;
identifying at least one set of corresponding planes, the corresponding planes comprising planes satisfying a grouping criteria, the grouping criteria configured to group planes belonging to the same physical object; or
providing pick instructions to the robotic picking unit based on the at least one set of corresponding planes.
Su teaches: computing a plane for each of the plurality of detected objects using the extracted 3D data (Para. 0043-0044, 3D features are extracted from video, feature points such as edges, points, and planes are detected using the object recognition algorithm), for the purpose of providing a system and method that can analyze a point cloud or mesh from a device used to scan a 3D space that is clear and more accurate.
Zhang teaches: identifying at least one set of corresponding planes, the corresponding planes comprising planes satisfying a grouping criteria, the grouping criteria configured to group planes belonging to the same physical object (Zhang identifies sets of corresponding surface regions from 3D line segments satisfying a grouping criterion, Page 3, Para. 2-3), for the purpose of navigating a mobile robot in an unknown indoor scene where an arbitrary number of rigid mobile objects may be present.
It would have been obvious to one of ordinary skill in the art on or before the time of the filing to modify the method of Kanemoto with computing a plane for each of the plurality of detected objects using the extracted 3D data, as taught is Su, for the purpose of providing a system and method that can analyze a point cloud or mesh from a device used to scan a 3D space that is clear and more accurate.
It would have been obvious to one of ordinary skill in the art on or before the time of the filing to modify the method of Kanemoto in view of Su, to include identifying a set of corresponding planes that comprise planes satisfying a grouping criteria, the grouping criteria groups planes belonging to the same physical object described above, as taught in Zhang, for the purpose of providing a method of navigating a mobile robot in an unknown indoor scene where an arbitrary number of rigid mobile objects may be present.
Relative to claims 15-16, Kanemoto in view of Su and Zhang does not expressly disclose: providing pick instructions to the robotic picking unit based on the at least one set of corresponding planes.
Kanemoto in view of Su and Zhang teaches: providing pick instructions to the robotic picking unit based on the at least one set of corresponding planes as an obvious matter of design choice.
Kanemoto provides picking instructions to a robotic unit based on detection mask information representing the regions of the scene containing target objects (Para. 0071, Kanemoto). Zhang teaches identifying planes within such regions and clustering them into corresponding planes belonging to a same physical object (Page 3 of Zhang).
Kanemoto, modified as above, can be modified to identify and cluster planes belonging to a same physical object, as taught in Zhang, to allow the system to determine stable pick surfaces and to guide the robot to approach directions more accurately.
Therefore, it would have been obvious to one of ordinary skill in the art on or before the time of the filing to modify Kanemoto in view of Su and Zhang so that the picking instructions are based on the at least one set of corresponding planes to allow the system to determine stable pick surfaces and to approach directions more accurately.
Claim(s) 5, 8-9, and 11, is/are rejected under 35 U.S.C. 103 as being unpatentable over Kanemoto in view of Su and Zhang as applied to claim 1 above, and further in view of Srivastava et al (US PG. Pub. 2024/0095709).
Relative to claims 5, 8-9, and 11, Kanemoto in view of Su and Zhang discloses all claim limitations mentioned above, but does not expressly disclose:
claim 5) the grouping criteria comprise planes which share a common edge in the 3D data;
claim 8) the grouping criteria are obtained from a database of previously defined criteria;
claim 9) the grouping criteria comprise a trained model and wherein the grouping criteria dynamically change over time as the trained model is updated; or
claim 11) the grouping criteria comprise a first grouping criteria and second grouping criteria and wherein the grouping criteria can be changed between the first grouping criteria and the second grouping criteria depending on characteristics of the objects in the pick area.
Srivastava teaches:
claim 5) the grouping criteria comprise planes which share a common edge in the 3D data (Para. 0089; 0105);
claim 8) the grouping criteria are obtained from a database of previously defined criteria (Para. 0090, a trained segmentation classifier embodies a database of previously learned grouping criteria);
claim 9) the grouping criteria comprise a trained model, and the grouping criteria dynamically change over time as the trained model is updated (Para. 0090, the grouping criteria embodied by the trained segmentation classifier are dynamically updated when the model is updated on new training data; Para. 0092, segmentation classifier can be trained on real data; and
claim 11) the grouping criteria comprise a first grouping criteria and second grouping criteria and wherein the grouping criteria can be changed between the first grouping criteria and the second grouping criteria depending on characteristics of the objects in the pick area (region masks include corresponding planes belonging to a same object based on grouping criteria, including determining a line that divides adjoining items based on a planar or elevation view, and extending the line through the orthogonal plane or along the vertical plane to segment the volume, Para. 0089; the region masks are determined by segmenting the geometric representation of the item set, Para. 0088, the masks define items for the grouping criteria, region masks may include a single item, or a first grouping, or multiple items, a second grouping, Para. 0087, so that when the system scans the scene in real time, Para. 0090, the first or second grouping criteria may be applied depending on whether the item is a single can or a 6-pack of cans).
Srivastava teaches: the grouping criteria comprising planes sharing a common edge; grouping criteria is obtained from a database; the grouping criteria comprise a trained model; and grouping criteria comprise a first grouping criteria and second grouping criteria as mentioned above, for the purpose of providing a new and useful system and method for item recognition from scenes using computer vision, that improves item segmentation and identification accuracy, and segments images more efficiently (Para. 0001; 0026-0027).
It would have been obvious to one of ordinary skill in the art on or before the time of the filing to modify the system of Kanemoto in view of Su and Zhang the grouping criteria comprising planes sharing a common edge; grouping criteria is obtained from a database; the grouping criteria comprise a trained model; and grouping criteria comprise a first grouping criteria and second grouping criteria described above, as taught in Srivastava, for the purpose of providing a new and useful system and method for item recognition from scenes using computer vision, that improves item segmentation and identification accuracy, and segments images more efficiently.
Claim(s) 10 and 12 is/are rejected under 35 U.S.C. 103 as being unpatentable over Kanemoto in view of Su and Zhang, as applied to claim 1 above, and further in view of Merkle et al (US PG. Pub. 2023/0182315).
Relative to claims 10 and 12, Kanemoto in view of Su and Zhang discloses all claim limitations mentioned above, but does not expressly disclose:
the grouping criteria comprises input from a user indicating how the planes should be grouped; or
the grouping criteria comprise a set of prototypes, each prototype indicative of expected characteristics of planes for an object, the set of prototypes comprising a plurality of prototypes associated with different objects.
Merkle teaches:
the grouping criteria comprises input from a user indicating how the planes should be grouped (Para. 0142, user or another computing device has knowledge about the types of objects; 0160); and
the grouping criteria comprise a set of prototypes (see set of prototypes), each prototype indicative of expected characteristics of planes for an object (features associated with a stored prototype for the 3D object), the set of prototypes comprising a plurality of prototypes associated with different objects (Para. 0140),
for the purpose of providing an improved system and method of picking objects from a stack using a robot that can determine a next object to be picked, and can perform complex and/or dynamic motions with speed, agility and efficiency (Para. 0002; 0121).
It would have been obvious to one of ordinary skill in the art on or before the time of the filing to modify the method of Kanemoto in view of Su and Zhang with the grouping criteria comprise a set of prototypes, each prototype indicative of expected characteristics of planes for an object, the set of prototypes comprising a plurality of prototypes associated with different objects, taught in Merkle for the purpose of providing an improved system and method of picking objects from a stack using a robot that can determine a next object to be picked, and can perform complex and/or dynamic motions with speed, agility and efficiency.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to YOLANDA RENEE CUMBESS whose telephone number is (571)270-5527. The examiner can normally be reached M-F 10-6.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Gene Crawford can be reached at 571-272-6911. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/YOLANDA R CUMBESS/Primary Examiner, Art Unit 3651
Prosecution Insights