Notice of Pre-AIA or AIA Status
1. The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
2. This communication is responsive to Application No. 18/368,927 and the amendments filed on 5/6/2026.
3. Claims 1-23 are presented for examination.
Information Disclosure Statement
4. The information disclosure statements (IDS) submitted on 9/27/2023, 6/4/2025, and 9/18/2025 have been fully considered by the Examiner.
Response to Arguments
5. Applicant’s arguments with respect to the rejection of claim(s) 1-20 under 35 U.S.C. 103 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Regarding independent claim 1, the Examiner agrees that the combination of US 20220203547 A1 to Majumdar, US 20250353173 A1 to Noda, and US 20230202774 A1 to Chen fails to teach all of the amended limitations of the claim. However, in light of the amendments and the Applicant’s remarks, an updated search was conducted, and a new ground of rejection concerning claim 1 has been determined, in which will be described later.
Regarding independent claims 9 and 17, as these claims contain similar limitations to claim 1, are still rejected for similar reasons as claim 1 is, in which will be described later.
Regarding dependent claims 2-8, 10-16, and 18-20, as all of these claims depend from either claims 1, 9, or 17, are still rejected, in which will be described later.
Claim Rejections - 35 USC § 103
6. In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
7. The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
8. Claim(s) 1, 4, 7, 8, 9, 12, 15, 16, and 17 is/are rejected under 35 U.S.C. 103 as being unpatentable over Majumdar et al. (US 20220203547 A1 hereinafter Majumdar) in view of Shiratsuchi et al. (WO 2022085408 A1 hereinafter Shiratsuchi) and Chen et al. (US 20230202774 A1 hereinafter Chen).
Regarding Claim 1, Majumdar teaches a computer-implemented method ([0074] via “… at least some of the features or functionalities of the various embodiments disclosed herein may be implemented on one or more general-purpose computers ….”) comprising:
accessing information indicating a position of one or more objects in an environment ([0058] via “At step 303, the process comprises determining features associated with the identified objects. Features may comprise at least one of observable intrinsic features, extrinsic features, and unobservable intrinsic features. … Extrinsic object features may comprise two dimensional (2D) object location, three dimensional (3D) object location, … etc.”), the information represented using one or more point clouds associated with the one or more objects ([0058] via “Extrinsic object features may be computed from analysis of the pick area data as a whole. For example, for each detected object, a relative location from the edges or from other objects may be computed based on where in the full pick area data set (e.g. a 3D point cloud), data associated with each object is located.”);
determining a grasp pose of the one or more objects ([0059] via “At step 304, the process comprises computing a pick plan based on at least one of the identifying objects step 302 (e.g. pick shapes) and at least one feature of the determining features step 303. A computed pick plan may comprise at least one of a pick sequence or order in which each object will be picked, instructions or pick coordinates for each pick, and end effector controls associated with each planned pick.”);
determining a placement of the one or more objects ([0057] via “For example, pick objects may be further classified based on their determined placement location such as a first group of pick objects to be placed at a first location, a second group of pick objects to be placed at a second location, and so on.”); and
causing an autonomous robot to manipulate the one or more objects based on the grasp pose of the one or more objects and the placement of the one or more objects ([0031] via “The robotic picking unit 114 may pick objects from one portion (e.g. a pallet) of a pick area 102 and place them at another portion (e.g. a conveyor) of the pick area 102. The robotic picking unit 114 may comprise a robotic arm and an end effector 124 attached to the robotic arm.”), ([0064] via “At step 305, the process comprises providing pick instructions based on the computed pick plan. The pick instructions may be provided to a robotic picking unit associated with the pick area. … For example, pick instructions may be provided on a pick by pick basis as the computed pick plan is executed by a robotic picking unit such that the robotic picking unit is being provided instructions for one picking action at a time.”).
Majumdar is silent on determining a grasp pose of the one or more objects by using one or more neural networks to generate a contact mask based, at least in part, on an indication of the ability of the one or more objects to be grasped; and determining a placement of the one or more objects by using the one or more neural networks to generate a placement mask based, at least in part, on the information indicating available placement location and orientation pairs of the one or more objects.
However, Shiratsuchi teaches determining a grasp pose of the one or more objects by using one or more neural networks to generate a contact mask based, at least in part, on an indication of the ability of the one or more objects to be grasped (Page 3 paragraph 3 via “Specifically, the target shape information is obtained by acquiring and calculating the image information or the distance information of the object 70 obtained by the visual sensor as the measuring device 60 as a point cloud.”), (Page 12 paragraph 6 via “As shown in FIG. 12, it is exemplified to learn a network that outputs a gripping point, a gripping force, and a gripping stability by inputting target shape information (before deformation).”), (Page 13 paragraph 4 via “As shown in FIG. 13, when the neural network 41 acquired in the above process is applied as the grip point candidate generation unit 32a and the target shape information is input, the grip point candidate generation unit 32a generates a plurality of grip point candidates and grip stability. It is generated and output to the gripping point determination unit 36. The gripping point determination unit 36 selects and outputs one gripping point candidate using the gripping stability.”), (Note: The Examiner interprets the grip point candidates using point cloud information of Shiratsuchi as the contact mask, as the contact mask is described within paragraphs [0085], [0086], and [0095] of the specification of the instant application. The Examiner also interprets the grip point candidates of Shiratsuchi as indicating the ability to grasp an object.).
Further, Chen teaches determining a placement of the one or more objects by using the one or more neural networks to generate a placement mask based, at least in part, on the information indicating available placement location and orientation pairs of the one or more objects ([0036] via “It is noted that, in this embodiment, the processing unit 13 may use a neural network technology to identify each existing object 4 from the space 3D point cloud, so a number of the space occupation data piece (s) generated by the processing unit 13 will be equal to a number of the existing object(s) 4. … If there is no existing object 4 placed in the accommodation space 20, the processing unit 13 will not generate any space occupation data piece in step S3.”), ([0037] via “In step S4, the processing unit 13 generates a first cross-section status data piece based on the space occupation data piece(s) that respectively correspond to the existing object(s) 4. … The first cross-section status data piece corresponds to a first cross section L1 of the accommodation space 20 that is parallel to a bottom surface of the container 2, and is indicative of an unoccupied area (i.e., an area not occupied by any of the existing object (s) 4) of the first cross section L1.”), (Note: The Examiner interprets the determined placement location of objects into the accommodation space using point cloud information of Chen as the placement mask, as the placement mask is described within paragraphs [0071], [0086], and [0095] of the specification of the instant application. The Examiner also notes that Chen is both able to determine space occupation data of already placed objects and determine unoccupied spaces based on the already placed objects, the unoccupied spaces indicating available placement locations.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Shiratsuchi wherein determining a grasp pose of the one or more objects by using one or more neural networks to generate a contact mask based, at least in part, on an indication of the ability of the one or more objects to be grasped. Doing so decreases the amount of time spent determining a grasp pose of the one or more objects, increasing the overall grasping efficiency, as stated by Shiratsuchi (Page 13 paragraph 5 via “According to the present embodiment, a gripping point generation algorithm that corrects the modeling error acquired in the actual work can be acquired by learning, and as a result, the calculation cost for calculating the gripping point candidate is reduced, and the gripping point is calculated. Since the time to do is shortened, a special effect of increasing production efficiency can be obtained.”).
In addition, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Chen wherein determining a placement of the one or more objects by using the one or more neural networks to generate a placement mask based, at least in part, on the information indicating available placement location and orientation pairs of the one or more objects. Doing so ensures that objects are placed in the placement area not in areas already occupied by other objects, as stated by Chen ([0042] via “In step S6, the processing unit 13 controls the holding unit 11 to place the to-be-packed object 3 into the accommodation space 20 at a position corresponding to the unoccupied area of the first cross section L1. To be specific, the processing unit 13
controls the holding unit 11 to place the to-be-packed object 3 into the accommodation space 20 at a position corresponding to the target region.”).
Regarding Claim 4, modified reference Majumdar teaches the computer-implemented method of claim 1, but is silent on wherein the indication of the ability of the one or more objects in the environment to be grasped comprises one or more portions of image data.
However, Shiratsuchi teaches wherein the indication of the ability of the one or more objects in the environment to be grasped comprises one or more portions of image data (Page 3 paragraph 3 via “Specifically, the target shape information is obtained by acquiring and calculating the image information or the distance information of the object 70 obtained by the visual sensor as the measuring device 60 as a point cloud.”), (Page 13 paragraph 4 via “As shown in FIG. 13, when the neural network 41 acquired in the above process is applied as the grip point candidate generation unit 32a and the target shape information is input, the grip point candidate generation unit 32a generates a plurality of grip point candidates and grip stability.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Shiratsuchi wherein the indication of the ability of the one or more objects in the environment to be grasped comprises one or more portions of image data. Doing so incorporates a method of determining grasp poses for objects in an environment that decreases the amount of time spent determining the grasp pose, increasing the overall grasping efficiency, as stated by Shiratsuchi (Page 13 paragraph 5 via “According to the present embodiment, a gripping point generation algorithm that corrects the modeling error acquired in the actual work can be acquired by learning, and as a result, the calculation cost for calculating the gripping point candidate is reduced, and the gripping point is calculated. Since the time to do is shortened, a special effect of increasing production efficiency can be obtained.”).
Regarding Claim 7, modified reference Majumdar teaches the computer-implemented method of claim 1, further comprising performing the grasp pose of the one or more objects in the environment using the one or more autonomous robots ([0031] via “The robotic picking unit 114 may pick objects from one portion (e.g. a pallet) of a pick area 102 and place them at another portion (e.g. a conveyor) of the pick area 102. The robotic picking unit 114 may comprise a robotic arm and an end effector 124 attached to the robotic arm.”).
Regarding Claim 8, modified reference Majumdar teaches the computer-implemented method of claim 1, further comprising performing the placement of the one or more objects in the environment using the one or more autonomous robots ([0031] via “The robotic picking unit 114 may pick objects from one portion (e.g. a pallet) of a pick area 102 and place them at another portion (e.g. a conveyor) of the pick area 102. The robotic picking unit 114 may comprise a robotic arm and an end effector 124 attached to the robotic arm.”).
Regarding Claim 9, Majumdar teaches a non-transitory computer readable storage medium storing thereon executable instructions that ([0082] via “Because such information and program instructions may be employed to implement one or more systems or methods described herein, at least some network device embodiments may include nontransitory machine-readable storage media, which, for example, may be configured or designed to store program instructions, state information, and the like for performing various operations described herein.”), as a result of being executed by one or more processors of a computer system ([0083] via “Computing device 20 includes processors 21 that may run software that carry out one or more functions or applications of embodiments, such as for example a client application 24.”), cause the computer system to:
access information indicating a position of one or more objects in an environment ([0058] via “At step 303, the process comprises determining features associated with the identified objects. Features may comprise at least one of observable intrinsic features, extrinsic features, and unobservable intrinsic features. … Extrinsic object features may comprise two dimensional (2D) object location, three dimensional (3D) object location, … etc.”), the information represented using one or more point clouds associated with the one or more objects ([0058] via “Extrinsic object features may be computed from analysis of the pick area data as a whole. For example, for each detected object, a relative location from the edges or from other objects may be computed based on where in the full pick area data set (e.g. a 3D point cloud), data associated with each object is located.”);
determine a grasp pose of the one or more objects ([0059] via “At step 304, the process comprises computing a pick plan based on at least one of the identifying objects step 302 (e.g. pick shapes) and at least one feature of the determining features step 303. A computed pick plan may comprise at least one of a pick sequence or order in which each object will be picked, instructions or pick coordinates for each pick, and end effector controls associated with each planned pick.”);
determine a placement of the one or more objects ([0057] via “For example, pick objects may be further classified based on their determined placement location such as a first group of pick objects to be placed at a first location, a second group of pick objects to be placed at a second location, and so on.”); and
causing an autonomous machine to manipulate the one or more objects based on the grasp pose of the one or more objects and the placement of the one or more objects ([0031] via “The robotic picking unit 114 may pick objects from one portion (e.g. a pallet) of a pick area 102 and place them at another portion (e.g. a conveyor) of the pick area 102. The robotic picking unit 114 may comprise a robotic arm and an end effector 124 attached to the robotic arm.”), ([0064] via “At step 305, the process comprises providing pick instructions based on the computed pick plan. The pick instructions may be provided to a robotic picking unit associated with the pick area. … For example, pick instructions may be provided on a pick by pick basis as the computed pick plan is executed by a robotic picking unit such that the robotic picking unit is being provided instructions for one picking action at a time.”).
Majumdar is silent on to determine a grasp pose of the one or more objects by using one or more neural networks to generate a contact mask based, at least in part, on an indication of the ability of the one or more objects to be grasped; and determine a placement of the one or more objects by using the one or more neural networks to generate a placement mask based, at least in part, on the information indicating available placement location and orientation pairs of the one or more objects.
However, Shiratsuchi teaches to determine a grasp pose of the one or more objects by using one or more neural networks to generate a contact mask based, at least in part, on an indication of the ability of the one or more objects to be grasped (Page 3 paragraph 3 via “Specifically, the target shape information is obtained by acquiring and calculating the image information or the distance information of the object 70 obtained by the visual sensor as the measuring device 60 as a point cloud.”), (Page 12 paragraph 6 via “As shown in FIG. 12, it is exemplified to learn a network that outputs a gripping point, a gripping force, and a gripping stability by inputting target shape information (before deformation).”), (Page 13 paragraph 4 via “As shown in FIG. 13, when the neural network 41 acquired in the above process is applied as the grip point candidate generation unit 32a and the target shape information is input, the grip point candidate generation unit 32a generates a plurality of grip point candidates and grip stability. It is generated and output to the gripping point determination unit 36. The gripping point determination unit 36 selects and outputs one gripping point candidate using the gripping stability.”), (Note: The Examiner interprets the grip point candidates using point cloud information of Shiratsuchi as the contact mask, as the contact mask is described within paragraphs [0085], [0086], and [0095] of the specification of the instant application. The Examiner also interprets the grip point candidates of Shiratsuchi as indicating the ability to grasp an object.).
Further, Chen teaches to determine a placement of the one or more objects by using the one or more neural networks to generate a placement mask based, at least in part, on the information indicating available placement location and orientation pairs of the one or more objects ([0036] via “It is noted that, in this embodiment, the processing unit 13 may use a neural network technology to identify each existing object 4 from the space 3D point cloud, so a number of the space occupation data piece (s) generated by the processing unit 13 will be equal to a number of the existing object(s) 4. … If there is no existing object 4 placed in the accommodation space 20, the processing unit 13 will not generate any space occupation data piece in step S3.”), ([0037] via “In step S4, the processing unit 13 generates a first cross-section status data piece based on the space occupation data piece(s) that respectively correspond to the existing object(s) 4. … The first cross-section status data piece corresponds to a first cross section L1 of the accommodation space 20 that is parallel to a bottom surface of the container 2, and is indicative of an unoccupied area (i.e., an area not occupied by any of the existing object (s) 4) of the first cross section L1.”), (Note: The Examiner interprets the determined placement location of objects into the accommodation space using point cloud information of Chen as the placement mask, as the placement mask is described within paragraphs [0071], [0086], and [0095] of the specification of the instant application. The Examiner also notes that Chen is both able to determine space occupation data of already placed objects and determine unoccupied spaces based on the already placed objects, the unoccupied spaces indicating available placement locations.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Shiratsuchi wherein to determine a grasp pose of the one or more objects by using one or more neural networks to generate a contact mask based, at least in part, on an indication of the ability of the one or more objects to be grasped. Doing so decreases the amount of time spent determining a grasp pose of the one or more objects, increasing the overall grasping efficiency, as stated by Shiratsuchi (Page 13 paragraph 5 via “According to the present embodiment, a gripping point generation algorithm that corrects the modeling error acquired in the actual work can be acquired by learning, and as a result, the calculation cost for calculating the gripping point candidate is reduced, and the gripping point is calculated. Since the time to do is shortened, a special effect of increasing production efficiency can be obtained.”).
In addition, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Chen wherein to determine a placement of the one or more objects by using the one or more neural networks to generate a placement mask based, at least in part, on the information indicating available placement location and orientation pairs of the one or more objects. Doing so ensures that objects are placed in the placement area not in areas already occupied by other objects, as stated by Chen ([0042] via “In step S6, the processing unit 13 controls the holding unit 11 to place the to-be-packed object 3 into the accommodation space 20 at a position corresponding to the unoccupied area of the first cross section L1. To be specific, the processing unit 13
controls the holding unit 11 to place the to-be-packed object 3 into the accommodation space 20 at a position corresponding to the target region.”).
Regarding Claim 12, modified reference Majumdar teaches the non-transitory computer readable storage medium of claim 9, but is silent on wherein the indication of the ability of the one or more objects in the environment to be grasped comprises one or more portions of image data.
However, Shiratsuchi teaches wherein the indication of the ability of the one or more objects in the environment to be grasped comprises one or more portions of image data (Page 3 paragraph 3 via “Specifically, the target shape information is obtained by acquiring and calculating the image information or the distance information of the object 70 obtained by the visual sensor as the measuring device 60 as a point cloud.”), (Page 13 paragraph 4 via “As shown in FIG. 13, when the neural network 41 acquired in the above process is applied as the grip point candidate generation unit 32a and the target shape information is input, the grip point candidate generation unit 32a generates a plurality of grip point candidates and grip stability.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Shiratsuchi wherein the indication of the ability of the one or more objects in the environment to be grasped comprises one or more portions of image data. Doing so incorporates a method of determining grasp poses for objects in an environment that decreases the amount of time spent determining the grasp pose, increasing the overall grasping efficiency, as stated by Shiratsuchi (Page 13 paragraph 5 via “According to the present embodiment, a gripping point generation algorithm that corrects the modeling error acquired in the actual work can be acquired by learning, and as a result, the calculation cost for calculating the gripping point candidate is reduced, and the gripping point is calculated. Since the time to do is shortened, a special effect of increasing production efficiency can be obtained.”).
Regarding Claim 15, modified reference Majumdar teaches the non-transitory computer readable storage medium of claim 9, wherein the computer system is to further perform the grasp pose of the one or more objects in the environment using the one or more autonomous machines ([0031] via “The robotic picking unit 114 may pick objects from one portion (e.g. a pallet) of a pick area 102 and place them at another portion (e.g. a conveyor) of the pick area 102. The robotic picking unit 114 may comprise a robotic arm and an end effector 124
attached to the robotic arm.”).
Regarding Claim 16, modified reference Majumdar teaches the non-transitory computer readable storage medium of claim 9, wherein the computer system is to further perform the placement of the one or more objects in the environment using the one or more autonomous machines ([0031] via “The robotic picking unit 114 may pick objects from one portion (e.g. a pallet) of a pick area 102 and place them at another portion (e.g. a conveyor) of the pick area 102. The robotic picking unit 114 may comprise a robotic arm and an end effector 124
attached to the robotic arm.”).
Regarding Claim 17, Majumdar teaches a system comprising: one or more processors ([0083] via “Computing device 20 includes processors 21 that may run software that carry out one or more functions or applications of embodiments, such as for example a client application 24.”) to:
access one or more point clouds indicating one or more objects in an environment ([0058] via “Extrinsic object features may be computed from analysis of the pick area data as a whole. For example, for each detected object, a relative location from the edges or from other objects may be computed based on where in the full pick area data set (e.g. a 3D point cloud), data associated with each object is located.”);
determine a grasp pose of the one or more objects ([0059] via “At step 304, the process comprises computing a pick plan based on at least one of the identifying objects step 302 (e.g. pick shapes) and at least one feature of the determining features step 303. A computed pick plan may comprise at least one of a pick sequence or order in which each object will be picked, instructions or pick coordinates for each pick, and end effector controls associated with each planned pick.”);
determine a placement of the one or more objects ([0057] via “For example, pick objects may be further classified based on their determined placement location such as a first group of pick objects to be placed at a first location, a second group of pick objects to be placed at a second location, and so on.”); and
cause an autonomous robot to manipulate the one or more objects based on the grasp pose of the one or more objects and the placement of the one or more objects ([0031] via “The robotic picking unit 114 may pick objects from one portion (e.g. a pallet) of a pick area 102 and place them at another portion (e.g. a conveyor) of the pick area 102. The robotic picking unit 114 may comprise a robotic arm and an end effector 124 attached to the robotic arm.”), ([0064] via “At step 305, the process comprises providing pick instructions based on the computed pick plan. The pick instructions may be provided to a robotic picking unit associated with the pick area. … For example, pick instructions may be provided on a pick by pick basis as the computed pick plan is executed by a robotic picking unit such that the robotic picking unit is being provided instructions for one picking action at a time.”).
Majumdar is silent on to determine whether the robot has the ability to grasp the one or more objects based, at least in part, on the one or more point clouds; determine a grasp pose of the one or more objects by using one or more neural networks to generate a contact mask based, at least in part, on the one or more point clouds; and determine a placement of the one or more objects by using the one or more neural networks to generate a placement mask based, at least in part, on the one or more point clouds.
However, Shiratsuchi teaches to determine whether the robot has the ability to grasp the one or more objects based, at least in part, on the one or more point clouds (Page 3 paragraph 3 via “Specifically, the target shape information is obtained by acquiring and calculating the image information or the distance information of the object 70 obtained by the visual sensor as the measuring device 60 as a point cloud.”), (Page 12 paragraph 6 via “As shown in FIG. 12, it is exemplified to learn a network that outputs a gripping point, a gripping force, and a gripping stability by inputting target shape information (before deformation).”), (Page 13 paragraph 4 via “As shown in FIG. 13, when the neural network 41 acquired in the above process is applied as the grip point candidate generation unit 32a and the target shape information is input, the grip point candidate generation unit 32a generates a plurality of grip point candidates and grip stability. It is generated and output to the gripping point determination unit 36. The gripping point determination unit 36 selects and outputs one gripping point candidate using the gripping stability.”); and
determine a grasp pose of the one or more objects by using one or more neural networks to generate a contact mask based, at least in part, on the one or more point clouds (Page 3 paragraph 3 via “Specifically, the target shape information is obtained by acquiring and calculating the image information or the distance information of the object 70 obtained by the visual sensor as the measuring device 60 as a point cloud.”), (Page 12 paragraph 6 via “As shown in FIG. 12, it is exemplified to learn a network that outputs a gripping point, a gripping force, and a gripping stability by inputting target shape information (before deformation).”), (Page 13 paragraph 4 via “As shown in FIG. 13, when the neural network 41 acquired in the above process is applied as the grip point candidate generation unit 32a and the target shape information is input, the grip point candidate generation unit 32a generates a plurality of grip point candidates and grip stability. It is generated and output to the gripping point determination unit 36. The gripping point determination unit 36 selects and outputs one gripping point candidate using the gripping stability.”), (Note: The Examiner interprets the grip point candidates using point cloud information of Shiratsuchi as the contact mask, as the contact mask is described within paragraphs [0085], [0086], and [0095] of the specification of the instant application. The Examiner also interprets the grip point candidates of Shiratsuchi as indicating the ability to grasp an object.).
Further, Chen teaches to determine a placement of the one or more objects by using the one or more neural networks to generate a placement mask based, at least in part, on the one or more point clouds ([0036] via “It is noted that, in this embodiment, the processing unit 13 may use a neural network technology to identify each existing object 4 from the space 3D point cloud, so a number of the space occupation data piece (s) generated by the processing unit 13 will be equal to a number of the existing object(s) 4. … If there is no existing object 4 placed in the accommodation space 20, the processing unit 13 will not generate any space occupation data piece in step S3.”), ([0037] via “In step S4, the processing unit 13 generates a first cross-section status data piece based on the space occupation data piece(s) that respectively correspond to the existing object(s) 4. … The first cross-section status data piece corresponds to a first cross section L1 of the accommodation space 20 that is parallel to a bottom surface of the container 2, and is indicative of an unoccupied area (i.e., an area not occupied by any of the existing object (s) 4) of the first cross section L1.”), (Note: The Examiner interprets the determined placement location of objects into the accommodation space using point cloud information of Chen as the placement mask, as the placement mask is described within paragraphs [0071], [0086], and [0095] of the specification of the instant application. The Examiner also notes that Chen is both able to determine space occupation data of already placed objects and determine unoccupied spaces based on the already placed objects, the unoccupied spaces indicating available placement locations.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Shiratsuchi wherein to determine whether the robot has the ability to grasp the one or more objects based, at least in part, on the one or more point clouds; and determine a grasp pose of the one or more objects by using one or more neural networks to generate a contact mask based, at least in part, on the one or more point clouds. Doing so decreases the amount of time spent determining a grasp pose of the one or more objects, increasing the overall grasping efficiency, as stated by Shiratsuchi (Page 13 paragraph 5 via “According to the present embodiment, a gripping point generation algorithm that corrects the modeling error acquired in the actual work can be acquired by learning, and as a result, the calculation cost for calculating the gripping point candidate is reduced, and the gripping point is calculated. Since the time to do is shortened, a special effect of increasing production efficiency can be obtained.”).
In addition, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Chen wherein to determine a placement of the one or more objects by using the one or more neural networks to generate a placement mask based, at least in part, on the one or more point clouds. Doing so ensures that objects are placed in the placement area not in areas already occupied by other objects, as stated by Chen ([0042] via “In step S6, the processing unit 13 controls the holding unit 11 to place the to-be-packed object 3 into the accommodation space 20 at a position corresponding to the unoccupied area of the first cross section L1. To be specific, the processing unit 13 controls the holding unit 11 to place the to-be-packed object 3 into the accommodation space 20 at a position corresponding to the target region.”).
9. Claim(s) 2 is/are rejected under 35 U.S.C. 103 as being unpatentable over Majumdar et al. (US 20220203547 A1 hereinafter Majumdar) in view of Shiratsuchi et al. (WO 2022085408 A1 hereinafter Shiratsuchi) and Chen et al. (US 20230202774 A1 hereinafter Chen), and further in view of Zizka et al. (US 20230415345 A1 hereinafter Zizka).
Regarding Claim 2, modified reference Majumdar teaches the computer-implemented method of claim 1, but is silent on wherein determining the grasp pose of the one or more objects is based on: one or more feature maps generated using the one or more point clouds; and one or more grasp task embeddings.
However, Zizka teaches wherein determining the grasp pose of the one or more objects is based on: one or more feature maps generated using the one or more point clouds ([0096] via “In another example, the apparatus 100 can receive a command from the master server to move an item from the box 120a to the box 120b, which can be carried out as shown in FIGS.
2A-2D. Upon receipt of the command, the apparatus 100 can acquire texture frame/information and point cloud information from its camera 160, feed the point cloud data (e.g., in a form of a depth map) and texture information to a neural network that can recognize potentially multiple instances of the item to be picked. The location(s) of the found items are then evaluated, for example, by checking whether they are accessible by the gripper 150 and the picker robot 125 without any collisions, …”); and
one or more grasp task embeddings ([0140] via “In some cases, the apparatus 500 can include an interface (e.g., a touchscreen) coupled to a processor of the apparatus 500 and/or to the master server to present order information (e.g., number of items to be picked for each order) to the operator to guide the operator when picking items for the boxes 520b and
520c.”), (Note: The Examiner interprets the grasp task embeddings as a maximum number of objects for the robot to grasp, as this term is defined in paragraph [0082] of the specification of the instant application. In the context of Zizka, the Examiner interprets the number of items to be picked as a maximum number of items for the robot to pick for the order, as the robot is not supposed to go over that amount.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Zizka wherein determining the grasp pose of the one or more objects is based on: one or more feature maps generated using the one or more point clouds; and one or more grasp task embeddings. Doing so locates objects and places them into the appropriate placement location based on given order information, as stated by Zizka ([0101] via “In yet another example of a typical operation that can be executed by the apparatus 100, the apparatus 100 can load a source box (e.g., the box 120a) based on an order received from the master server, and pick one or more items from the source box into a destination box (e.g., the box 120b). The apparatus 100 can then place the source box back on a shelf of the fulfillment center and move to the location of another source box that contains items for the order. The foregoing process can then be repeated until all the items are picked for the order.”).
10. Claim(s) 3 is/are rejected under 35 U.S.C. 103 as being unpatentable over Majumdar et al. (US 20220203547 A1 hereinafter Majumdar) in view of Shiratsuchi et al. (WO 2022085408 A1 hereinafter Shiratsuchi) and Chen et al. (US 20230202774 A1 hereinafter Chen), and further in view of Yu et al. (US 20230041343 A1 hereinafter Yu) and Sisbot et al. (US 9469028 B2 hereinafter Sisbot).
Regarding Claim 3, modified reference Majumdar teaches the computer-implemented method of claim 1, but is silent on wherein determining the placement of the one or more objects is based on: one or more feature maps generated using the one or more point clouds; and one or more placement task embeddings.
However, Yu teaches wherein determining the placement of the one or more objects is based on: one or more feature maps generated using the one or more point clouds ([0032] via “The storage devices 204 can also store object tracking data. In some embodiments, the object tracking data can include a log of scanned, manipulated, and/or transferred objects. In some embodiments, the object tracking data can include image data (e.g., a picture, point cloud, live video feed, etc.) of the objects at one or more locations (e.g., designated pickup or drop locations and/or conveyor belts) and/or placement locations/poses of the objects at the one or more locations.”).
Further, Sisbot teaches wherein determining the placement of the one or more objects is based on: one or more placement task embeddings (Col. 19 line 59 – Col. 20 line 10, where “The assessment module 268 may analyze the assessment criteria data 261 and determined if the threshold is exceeded. If the threshold is exceeded, the method 500 may return to step 508 where the assessment module 268 may select a different object transfer pose determined to be less risky. The method may cycle through in this manner a predetermined number of times attempting to identify an acceptable object transfer pose. If an acceptable object transfer pose is not identified, then the method 500 may end or the second communication unit 283 may ask for new instructions from the user 101 or the medical input provider 170. Alternatively, if no acceptable object transfer pose is identified after a predetermined number of attempts the pose system 199 may analyze the sensor data 249 to identify a flat or approximately flat surface in the user environment 198 and then determine actuations and movement vectors for the robot 190 to take to traverse to that surface and place the object 188 on the surface for the user 101 to pick up for themselves.”), (Note: The Examiner interprets the placement task embeddings as a number of discretized orientations of the object for placement, as this term is defined in paragraph [0082] of the specification of the instant application).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Yu wherein determining the placement of the one or more objects is based on: one or more feature maps generated using the one or more point clouds. Doing so incorporates a known method of mapping out a representation of the environment where the object(s) is/are located, as stated by Yu ([0038] via “The imaging devices 222 can generate a representation of the detected environment, such as a digital image, a depth map, and/or a point cloud, used for implementing machine/computer vision (e.g., for automatic inspection, robot guidance, or other robotic applications).”).
In addition, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Sisbot wherein determining the placement of the one or more objects is based on: one or more placement task embeddings. Doing so analyzes and determines the most appropriate placement pose for the object, as stated above by Sisbot.
11. Claim(s) 5, 13, and 19 is/are rejected under 35 U.S.C. 103 as being unpatentable over Majumdar et al. (US 20220203547 A1 hereinafter Majumdar) in view of Shiratsuchi et al. (WO 2022085408 A1 hereinafter Shiratsuchi) and Chen et al. (US 20230202774 A1 hereinafter Chen), and further in view of Sun et al. (US 20240017426 A1 hereinafter Sun).
Regarding Claim 5, modified reference Majumdar teaches the computer-implemented method of claim 1, but is silent on the method further determining one or more grasp parameters to be used to determine a grasp pose to grasp the one or more objects.
However, Sun teaches further determining one or more grasp parameters to be used to determine a grasp pose to grasp the one or more objects ([0112] via “Process 2000 is shown to include identifying a pre-grasp configuration for a robotic hand based on a target quantity of objects to be grasped (2010). For example, approaches to identifying a pre-grasp configuration such as the best-expectation pre-grasp and the maximum capability pre-grasp as described above can be used.”), ([0114] via “Process 2000 is also shown to include operating fingers of the robotic hand in accordance with the pre-grasp configuration (2030). … The pre-grasp configuration is intended for grasping the target quantity of objects with a high probability of success. Once hand 135 is oriented in accordance with the pre-grasp configuration, hand 135 is ready to grasp multiple objects as part of the transfer process.”), (Note: The Examiner interprets the target quantity of objects to be grasped of Sun as the grasp parameter.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Sun wherein the method further determines one or more grasp parameters to be used to determine a grasp pose to grasp the one or more objects. Doing so performs a grasp configuration of the robot hand resulting in the most likely configuration for a successful grasp of the object(s), as stated above by Sun in both paragraphs.
Regarding Claim 13, modified reference Majumdar teaches the non-transitory computer readable storage medium of claim 9, but is silent on wherein the computer system is further caused to determine one or more grasp parameters to be used to determine a grasp pose to grasp the one or more objects.
However, Sun teaches wherein the computer system is further caused to determine one or more grasp parameters to be used to determine a grasp pose to grasp the one or more objects ([0112] via “Process 2000 is shown to include identifying a pre-grasp configuration for a robotic hand based on a target quantity of objects to be grasped (2010). For example, approaches to identifying a pre-grasp configuration such as the best-expectation pre-grasp and the maximum capability pre-grasp as described above can be used.”), ([0114] via “Process 2000 is also shown to include operating fingers of the robotic hand in accordance with the pre-grasp configuration (2030). … The pre-grasp configuration is intended for grasping the target quantity of objects with a high probability of success. Once hand 135 is oriented in accordance with the pre-grasp configuration, hand 135 is ready to grasp multiple objects as part of the transfer process.”), (Note: The Examiner interprets the target quantity of objects to be grasped of Sun as the grasp parameter.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Sun wherein the computer system is further caused to determine one or more grasp parameters to be used to determine a grasp pose to grasp the one or more objects. Doing so performs a grasp configuration of the robot hand resulting in the most likely configuration for a successful grasp of the object(s), as stated above by Sun in both paragraphs.
Regarding Claim 19, modified reference Majumdar teaches the system of claim 17, but is silent on wherein the one or more processors are to determine one or more grasp parameters to be used to determine a grasp pose to grasp the one or more objects.
However, Sun teaches wherein the one or more processors are to determine one or more grasp parameters to be used to determine a grasp pose to grasp the one or more objects ([0112] via “Process 2000 is shown to include identifying a pre-grasp configuration for a robotic hand based on a target quantity of objects to be grasped (2010). For example, approaches to identifying a pre-grasp configuration such as the best-expectation pre-grasp and the maximum capability pre-grasp as described above can be used.”), ([0114] via “Process 2000 is also shown to include operating fingers of the robotic hand in accordance with the pre-grasp configuration (2030). … The pre-grasp configuration is intended for grasping the target quantity of objects with a high probability of success. Once hand 135 is oriented in accordance with the pre-grasp configuration, hand 135 is ready to grasp multiple objects as part of the transfer process.”), (Note: The Examiner interprets the target quantity of objects to be grasped of Sun as the grasp parameter.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Sun wherein the one or more processors are to determine one or more grasp parameters to be used to determine a grasp pose to grasp the one or more objects. Doing so performs a grasp configuration of the robot hand resulting in the most likely configuration for a successful grasp of the object(s), as stated above by Sun in both paragraphs.
12. Claim(s) 6, 14, and 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Majumdar et al. (US 20220203547 A1 hereinafter Majumdar) in view of Shiratsuchi et al. (WO 2022085408 A1 hereinafter Shiratsuchi) and Chen et al. (US 20230202774 A1 hereinafter Chen), and further in view of Boroushaki et al. (US 20220168899 A1 hereinafter Boroushaki).
Regarding Claim 6, modified reference Majumdar teaches the computer-implemented method of claim 1, but is silent on the method further comprising training the one or more neural networks to determine a grasp pose of one or more previously unseen objects and to determine a placement of the one or more previously unseen objects.
However, Boroushaki teaches training the one or more neural networks to determine a grasp pose of one or more previously unseen objects ([0077] via “The data may be a different type of data in another embodiment provided, for example, that the data provides a video representation of the area of interest and optionally depth information of features in the area of interest including any one or more of those that may be partially or fully occluding the target object.”), ([0083] via “At 635, if operation 630 determines that the target object is within reach of the robot, the system transitions to an RF-visual grasping operation (e.g., RF-visual grasping 805 in FIG. 8).”), ([0156] via “In accordance with one or more of the aforementioned embodiments, a control system and method are provided which locates a partially or fully occluded target object (generally, occluded objects) in an area of interest. … Model-based and deep-learning techniques may then be employed to move the robot into range relative to the target object and to provide access by the robot to the target object, which access may include, but is not limited to, performing a grasping operation for the target object while, for example, the object is still in the occluded state.”).
Further, Chen teaches training the one or more neural networks to determine a placement of the one or more previously unseen objects ([0036] via “It is noted that, in this embodiment, the processing unit 13 may use a neural network technology to identify each existing object 4 from the space 3D point cloud, so a number of the space occupation data piece (s) generated by the processing unit 13 will be equal to a number of the existing object(s) 4. … If there is no existing object 4 placed in the accommodation space 20, the processing unit 13 will not generate any space occupation data piece in step S3.”), ([0037] via “In step S4, the processing unit 13 generates a first cross-section status data piece based on the space occupation data piece(s) that respectively correspond to the existing object(s) 4. … The first cross-section status data piece corresponds to a first cross section L1 of the accommodation space 20 that is parallel to a bottom surface of the container 2, and is indicative of an unoccupied area (i.e., an area not occupied by any of the existing object (s) 4) of the first cross section L1.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Boroushaki wherein the method further comprises training the one or more neural networks to determine a grasp pose of one or more previously unseen objects. Doing so enables the robot to locate and subsequently grasp partially and fully occluded objects, as stated above by Boroushaki in paragraph [0156].
In addition, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Chen wherein the method further comprises training the one or more neural networks to determine a placement of the one or more previously unseen objects. Doing so ensures that objects are placed in the placement area not in areas already occupied by other objects, as stated by Chen ([0042] via “In step S6, the processing unit 13 controls the holding unit 11 to place the to-be-packed object 3 into the accommodation space 20 at a position corresponding to the unoccupied area of the first cross section L1. To be specific, the processing unit 13 controls the holding unit 11 to place the to-be-packed object 3 into the accommodation space 20 at a position corresponding to the target region.”).
Regarding Claim 14, modified reference Majumdar teaches the non-transitory computer readable storage medium of claim 9, but is silent on wherein the one or more neural networks are trained to determine a grasp pose of one or more previously unseen objects and determine a placement of the one or more previously unseen objects.
However, Boroushaki teaches wherein the one or more neural networks are trained to determine a grasp pose of one or more previously unseen objects ([0077] via “The data may be a different type of data in another embodiment provided, for example, that the data provides a video representation of the area of interest and optionally depth information of features in the area of interest including any one or more of those that may be partially or fully occluding the target object.”), ([0083] via “At 635, if operation 630 determines that the target object is within reach of the robot, the system transitions to an RF-visual grasping operation (e.g., RF-visual grasping 805 in FIG. 8).”), ([0156] via “In accordance with one or more of the aforementioned embodiments, a control system and method are provided which locates a partially or fully occluded target object (generally, occluded objects) in an area of interest. … Model-based and deep-learning techniques may then be employed to move the robot into range relative to the target object and to provide access by the robot to the target object, which access may include, but is not limited to, performing a grasping operation for the target object while, for example, the object is still in the occluded state.”).
Further, Chen teaches wherein the one or more neural networks are trained to determine a placement of the one or more previously unseen objects ([0036] via “It is noted that, in this embodiment, the processing unit 13 may use a neural network technology to identify each existing object 4 from the space 3D point cloud, so a number of the space occupation data piece (s) generated by the processing unit 13 will be equal to a number of the existing object(s) 4. … If there is no existing object 4 placed in the accommodation space 20, the processing unit 13 will not generate any space occupation data piece in step S3.”), ([0037] via “In step S4, the processing unit 13 generates a first cross-section status data piece based on the space occupation data piece(s) that respectively correspond to the existing object(s) 4. … The first cross-section status data piece corresponds to a first cross section L1 of the accommodation space 20 that is parallel to a bottom surface of the container 2, and is indicative of an unoccupied area (i.e., an area not occupied by any of the existing object (s) 4) of the first cross section L1.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Boroushaki wherein the one or more neural networks are trained to determine a grasp pose of one or more previously unseen objects. Doing so enables the robot to locate and subsequently grasp partially and fully occluded objects, as stated above by Boroushaki in paragraph [0156].
In addition, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Chen wherein the one or more neural networks are trained to determine a placement of the one or more previously unseen objects. Doing so ensures that objects are placed in the placement area not in areas already occupied by other objects, as stated by Chen ([0042] via “In step S6, the processing unit 13 controls the holding unit 11 to place the to-be-packed object 3 into the accommodation space 20 at a position corresponding to the unoccupied area of the first cross section L1. To be specific, the processing unit 13 controls the holding unit 11 to place the to-be-packed object 3 into the accommodation space 20 at a position corresponding to the target region.”).
Regarding Claim 20, modified reference Majumdar teaches the system of claim 17, but is silent on wherein the one or more neural networks are trained to determine a grasp pose of one or more previously unseen objects and determine a placement of the one or more previously unseen objects.
However, Boroushaki teaches wherein the one or more neural networks are trained to determine a grasp pose of one or more previously unseen objects ([0077] via “The data may be a different type of data in another embodiment provided, for example, that the data provides a video representation of the area of interest and optionally depth information of features in the area of interest including any one or more of those that may be partially or fully occluding the target object.”), ([0083] via “At 635, if operation 630 determines that the target object is within reach of the robot, the system transitions to an RF-visual grasping operation (e.g., RF-visual grasping 805 in FIG. 8).”), ([0156] via “In accordance with one or more of the aforementioned embodiments, a control system and method are provided which locates a partially or fully occluded target object (generally, occluded objects) in an area of interest. … Model-based and deep-learning techniques may then be employed to move the robot into range relative to the target object and to provide access by the robot to the target object, which access may include, but is not limited to, performing a grasping operation for the target object while, for example, the object is still in the occluded state.”).
Further, Chen teaches wherein the one or more neural networks are trained to determine a placement of the one or more previously unseen objects ([0036] via “It is noted that, in this embodiment, the processing unit 13 may use a neural network technology to identify each existing object 4 from the space 3D point cloud, so a number of the space occupation data piece (s) generated by the processing unit 13 will be equal to a number of the existing object(s) 4. … If there is no existing object 4 placed in the accommodation space 20, the processing unit 13 will not generate any space occupation data piece in step S3.”), ([0037] via “In step S4, the processing unit 13 generates a first cross-section status data piece based on the space occupation data piece(s) that respectively correspond to the existing object(s) 4. … The first cross-section status data piece corresponds to a first cross section L1 of the accommodation space 20 that is parallel to a bottom surface of the container 2, and is indicative of an unoccupied area (i.e., an area not occupied by any of the existing object (s) 4) of the first cross section L1.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Boroushaki wherein the one or more neural networks are trained to determine a grasp pose of one or more previously unseen objects. Doing so enables the robot to locate and subsequently grasp partially and fully occluded objects, as stated above by Boroushaki in paragraph [0156].
In addition, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Chen wherein the one or more neural networks are trained to determine a placement of the one or more previously unseen objects. Doing so ensures that objects are placed in the placement area not in areas already occupied by other objects, as stated by Chen ([0042] via “In step S6, the processing unit 13 controls the holding unit 11 to place the to-be-packed object 3 into the accommodation space 20 at a position corresponding to the unoccupied area of the first cross section L1. To be specific, the processing unit 13 controls the holding unit 11 to place the to-be-packed object 3 into the accommodation space 20 at a position corresponding to the target region.”).
13. Claim(s) 10 is/are rejected under 35 U.S.C. 103 as being unpatentable over Majumdar et al. (US 20220203547 A1 hereinafter Majumdar) in view of Shiratsuchi et al. (WO 2022085408 A1 hereinafter Shiratsuchi) and Chen et al. (US 20230202774 A1 hereinafter Chen), and further in view of Wiersma et al. (US 20240408766 A1 hereinafter Wiersma).
Regarding Claim 10, modified reference Majumdar teaches the non-transitory computer readable storage medium of claim 9, but is silent on wherein the grasp pose of the one or more objects is determined based on: one or more feature maps generated using the one or more point clouds; one or more contact points associated with the one or more objects; and one or more grasp task feature embeddings.
However, Wiersma teaches wherein the grasp pose of the one or more objects is determined based on: one or more feature maps generated using the one or more point clouds ([0068] via “A depth map may also be included as a learnable property map. It can help determine which object to pick first as it can show which objects lie at or near the top of the bin. During the dataset generation, the depth map can be directly determined from the pixel-aligned point cloud. A depth map may be created by normalizing the values from the point cloud to be between 0 and 1.”); and
one or more grasp task feature embeddings ([0063] via “Hence, a patch fitting algorithm is used to fit parts of a point cloud representing an object to be picked up by a gripper to a predetermined patch defining a surface having a predetermined orientation in the reference frame of the object and a predetermined curvature. Information associated with these patches may then be encoded into an object property map. This way, 3D information of an object may be encoded in (2D) object property map.”), (Note: The Examiner interprets the task feature embeddings as using point clouds to generate features of the object, as this term is defined within paragraph [0089] of the specification of the instant application.).
Further, Shiratsuchi teaches wherein the grasp pose of the one or more objects is determined based on: one or more contact points associated with the one or more objects (Page 3 paragraph 7 via “Next, the deformation evaluation unit 33 evaluates and outputs the expected shape deformation information as shown in FIG. 4 for each case of the plurality of grip point candidates generated by the grip point candidate generation unit 32. ... In order to evaluate the shape deformation information, it is possible to calculate the shape deformation information expected in the model in which each finger causes deformation with respect to the object 70, assuming gripping by point contact by each finger.”), (Page 4 paragraph 1 via “Further, for the point contact portions provided in the robot hand 20 as many as the number of fingers, an appropriate fixed gripping force Fi is assumed for the generated points i (i = 1, 2, 3 ...). The mechanical relationship in which deformation occurs with respect to force can be calculated at each point contact point.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Wiersma wherein the grasp pose of the one or more objects is determined based on: one or more feature maps generated using the one or more point clouds; one or more contact points associated with the one or more objects; and one or more grasp task feature embeddings. Doing so determines the three-dimensional information of the object to be grasped, as stated by Wiersma above in both citations.
In addition, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Shiratsuchi wherein the grasp pose of the one or more objects is determined based on: one or more contact points associated with the one or more objects. Doing so calculates how the grasp pose of the robot affects the one or more objects, such as by determining how much deformation of the one or more objects may occur, as stated above by Shiratsuchi on page 3 paragraph 7.
14. Claim(s) 11 is/are rejected under 35 U.S.C. 103 as being unpatentable over Majumdar et al. (US 20220203547 A1 hereinafter Majumdar) in view of Shiratsuchi et al. (WO 2022085408 A1 hereinafter Shiratsuchi) and Chen et al. (US 20230202774 A1 hereinafter Chen), and further in view of Yu et al. (US 20230041343 A1 hereinafter Yu).
Regarding Claim 11, modified reference Majumdar teaches the non-transitory computer readable storage medium of claim 9, but is silent on wherein the placement of the one or more objects is determined based on: one or more feature maps generated using the one or more point clouds; one or more contact points associated with the one or more objects; and one or more placement task feature embeddings.
However, Yu teaches wherein the placement of the one or more objects is determined based on: one or more feature maps generated using the one or more point clouds ([0032] via “The storage devices 204 can also store object tracking data. In some embodiments, the object tracking data can include a log of scanned, manipulated, and/or transferred objects. In some embodiments, the object tracking data can include image data (e.g., a picture, point cloud, live video feed, etc.) of the objects at one or more locations (e.g., designated pickup or drop locations and/or conveyor belts) and/or placement locations/poses of the objects at the one or more locations.”);
one or more contact points associated with the one or more objects ([0077] via “At block 613, the robotic system 100 can obtain additional data during the transfer of the object. For example, the robotic system 100 can obtain lateral dimensions of the object based on implementing an initial displacement to separate the edges of the grasped object from adjacent objects.”), (Note: The Examiner interprets the obtaining the dimensions of the object as the contact points.); and
one or more placement task feature embeddings ([0032] via “The storage devices 204
can also store object tracking data. In some embodiments, the object tracking data can include a log of scanned, manipulated, and/or transferred objects. In some embodiments, the object tracking data can include image data (e.g., a picture, point cloud, live video feed, etc.) of the objects at one or more locations (e.g., designated pickup or drop locations and/or conveyor belts) and/or placement locations/poses of the objects at the one or more locations.”), (Note: The Examiner interprets the task feature embeddings as using point clouds to generate features of the object, as this term is defined within paragraph [0089] of the specification of the instant application.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Yu wherein the placement of the one or more objects is determined based on: one or more feature maps generated using the one or more point clouds; one or more contact points associated with the one or more objects; and one or more placement task feature embeddings. Doing so captures multiple data points of the object to be placed, as stated above by Yu as a combination of both citations.
15. Claim(s) 18 is/are rejected under 35 U.S.C. 103 as being unpatentable over Majumdar et al. (US 20220203547 A1 hereinafter Majumdar) in view of Shiratsuchi et al. (WO 2022085408 A1 hereinafter Shiratsuchi) and Chen et al. (US 20230202774 A1 hereinafter Chen), and further in view of Wiersma et al. (US 20240408766 A1 hereinafter Wiersma) and Yu et al. (US 20230041343 A1 hereinafter Yu).
Regarding Claim 18, modified reference Majumdar teaches the system of claim 17, but is silent on wherein the grasp pose of the one or more objects and the placement of the one or more objects is determined based on: one or more feature maps generated using the one or more point clouds; one or more contact points associated with the one or more objects; and one or more task features.
However, Wiersma teaches wherein the grasp pose of the one or more objects is determined based on: one or more feature maps generated using the one or more point clouds ([0068] via “A depth map may also be included as a learnable property map. It can help determine which object to pick first as it can show which objects lie at or near the top of the bin. During the dataset generation, the depth map can be directly determined from the pixel-aligned point cloud. A depth map may be created by normalizing the values from the point cloud to be between 0 and 1.”); and
one or more task features ([0063] via “Hence, a patch fitting algorithm is used to fit parts of a point cloud representing an object to be picked up by a gripper to a predetermined patch defining a surface having a predetermined orientation in the reference frame of the object and a predetermined curvature. Information associated with these patches may then be encoded into an object property map. This way, 3D information of an object may be encoded in (2D) object property map.”), (Note: The Examiner interprets the task features as using point clouds to generate features of the object, as this term is defined within paragraph [0089] of the specification of the instant application.).
Further, Shiratsuchi teaches wherein the grasp pose of the one or more objects is determined based on: one or more contact points associated with the one or more objects (Page 3 paragraph 7 via “Next, the deformation evaluation unit 33 evaluates and outputs the expected shape deformation information as shown in FIG. 4 for each case of the plurality of grip point candidates generated by the grip point candidate generation unit 32. ... In order to evaluate the shape deformation information, it is possible to calculate the shape deformation information expected in the model in which each finger causes deformation with respect to the object 70, assuming gripping by point contact by each finger.”), (Page 4 paragraph 1 via “Further, for the point contact portions provided in the robot hand 20 as many as the number of fingers, an appropriate fixed gripping force Fi is assumed for the generated points i (i = 1, 2, 3 ...). The mechanical relationship in which deformation occurs with respect to force can be calculated at each point contact point.”).
Further, Yu teaches wherein the placement of the one or more objects is determined based on: one or more feature maps generated using the one or more point clouds ([0032] via “The storage devices 204 can also store object tracking data. In some embodiments, the object tracking data can include a log of scanned, manipulated, and/or transferred objects. In some embodiments, the object tracking data can include image data (e.g., a picture, point cloud, live video feed, etc.) of the objects at one or more locations (e.g., designated pickup or drop locations and/or conveyor belts) and/or placement locations/poses of the objects at the one or more locations.”);
one or more contact points associated with the one or more objects ([0077] via “At block 613, the robotic system 100 can obtain additional data during the transfer of the object. For example, the robotic system 100 can obtain lateral dimensions of the object based on implementing an initial displacement to separate the edges of the grasped object from adjacent objects.”), (Note: The Examiner interprets the obtaining the dimensions of the object as the contact points.); and
one or more task features ([0032] via “The storage devices 204 can also store object tracking data. In some embodiments, the object tracking data can include a log of scanned, manipulated, and/or transferred objects. In some embodiments, the object tracking data can include image data (e.g., a picture, point cloud, live video feed, etc.) of the objects at one or more locations (e.g., designated pickup or drop locations and/or conveyor belts) and/or placement locations/poses of the objects at the one or more locations.”), (Note: The Examiner interprets the task features as using point clouds to generate features of the object, as this term is defined within paragraph [0089] of the specification of the instant application.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Wiersma wherein the grasp pose of the one or more objects is determined based on: one or more feature maps generated using the one or more point clouds; and one or more task features. Doing so determines the three-dimensional information of the object to be grasped, as stated by Wiersma above in both citations.
In addition, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Shiratsuchi wherein the grasp pose of the one or more objects is determined based on: one or more contact points associated with the one or more objects. Doing so calculates how the grasp pose of the robot affects the one or more objects, such as by determining how much deformation of the one or more objects may occur, as stated above by Shiratsuchi on page 3 paragraph 7.
In addition, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Yu wherein the placement of the one or more objects is determined based on: one or more feature maps generated using the one or more point clouds; one or more contact points associated with the one or more objects; and one or more task features. Doing so captures multiple data points of the object to be placed, as stated above by Yu as a combination of both citations.
16. Claim(s) 21 and 22 is/are rejected under 35 U.S.C. 103 as being unpatentable over Majumdar et al. (US 20220203547 A1 hereinafter Majumdar) in view of Shiratsuchi et al. (WO 2022085408 A1 hereinafter Shiratsuchi) and Chen et al. (US 20230202774 A1 hereinafter Chen), and further in view of Li (US 20230125022 A1 hereinafter Li).
Regarding Claim 21, modified reference Majumdar teaches the computer-implemented method of claim 4, but is silent on wherein the indication of the ability of the one or more objects in the environment to be grasped is based, at least in part, on a comparison against a threshold value.
However, Li teaches wherein the indication of the ability of the one or more objects in the environment to be grasped is based, at least in part, on a comparison against a threshold value ([0091] via “Alternatively, the inference unit 54 sets a threshold of the score set according to the commonality of the image in the proximity of the above-described picking position, and defines all of the picking positions with commonality with the image exceeding the threshold as the picking positions with high possibility of successful picking determined by the findings of the teaching person, so that from among these picking positions as a more appropriate candidate group, the target workpieces Wo with a shallower depth of the picking position may be preferentially picked.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Li wherein the indication of the ability of the one or more objects in the environment to be grasped is based, at least in part, on a comparison against a threshold value. Doing so grasps the objects with picking positions that will most likely result in a successful grasp, as stated above by Li.
Regarding Claim 22, modified reference Majumdar teaches the non-transitory computer readable storage medium of claim 12, but is silent on wherein the indication of the ability of the one or more objects in the environment to be grasped is based, at least in part, on a comparison against a threshold value.
However, Li teaches wherein the indication of the ability of the one or more objects in the environment to be grasped is based, at least in part, on a comparison against a threshold value ([0091] via “Alternatively, the inference unit 54 sets a threshold of the score set according to the commonality of the image in the proximity of the above-described picking position, and defines all of the picking positions with commonality with the image exceeding the threshold as the picking positions with high possibility of successful picking determined by the findings of the teaching person, so that from among these picking positions as a more appropriate candidate group, the target workpieces Wo with a shallower depth of the picking position may be preferentially picked.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Li wherein the indication of the ability of the one or more objects in the environment to be grasped is based, at least in part, on a comparison against a threshold value. Doing so grasps the objects with picking positions that will most likely result in a successful grasp, as stated above by Li.
17. Claim(s) 23 is/are rejected under 35 U.S.C. 103 as being unpatentable over Majumdar et al. (US 20220203547 A1 hereinafter Majumdar) in view of Shiratsuchi et al. (WO 2022085408 A1 hereinafter Shiratsuchi) and Chen et al. (US 20230202774 A1 hereinafter Chen), and further in view of Peng et al. (WO 2023273179 A1 hereinafter Peng).
Regarding Claim 23, modified reference Majumdar teaches the system of claim 17, but is silent on wherein the ability of the robot to grasp the one or more objects is determined using: an encoder to generate one or more feature maps; a decoder to determine one or more points on the one or more objects from which the one or more objects can be grasped based, at least in part, on the one or more feature maps.
However, Peng teaches wherein the ability of the robot to grasp the one or more objects is determined using: an encoder to generate one or more feature maps (Page 4 paragraphs 7-8 via “Step 101, acquiring a depth map including a target object and grasping point information of the target object. Specifically, the depth map is captured by a 3D camera installed on the robotic arm (grabber).”);
a decoder to determine one or more points on the one or more objects from which the one or more objects can be grasped based, at least in part, on the one or more feature maps (Page 6 paragraph 7 via “Step 203, using the preset grasping point detection network to process the depth map and position information to obtain the grasping point information of the target object.”), (Note: See Steps 301 and 302 of Peng as well.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Peng wherein the ability of the robot to grasp the one or more objects is determined using: an encoder to generate one or more feature maps; a decoder to determine one or more points on the one or more objects from which the one or more objects can be grasped based, at least in part, on the one or more feature maps. Doing so uses acquired sensor data to determine graspable points on objects in order to grasp the object from the most suitable point, as stated by Peng (Page 6 paragraphs 11-12 via “Step 302 , using the grasping point detection network to select one three-dimensional point information from multiple three-dimensional point information of the target object as the grasping point information according to a preset selection rule. Specifically, the preset selection rule can be to select the center position of the target to pick up, or to select according to the object classification of the target object (such as selecting the handle of the cup, selecting the brim of the hat, etc.). After all the 3D point information is obtained, the most suitable 3D point information can be selected from all the 3D point information according to the selection rules, and at the same time, the grasping point information will be generated according to the position information of the 3D point information in the target object.”).
Examiner’s Note
18. The Examiner has cited particular paragraphs or columns and line numbers in the
references applied to the claims above for the convenience of the Applicant. Although the
specified citations are representative of the teachings of the art and are applied to specific
limitations within the individual claim, other passages and figures may apply as well. It is
respectfully requested of the Applicant in preparing responses, to fully consider the references
in their entirety as potentially teaching all or part of the claimed invention, as well as the
context of the passage as taught by the prior art or disclosed by the Examiner. See MPEP
2141.02 [R-07.2015] VI. A prior art reference must be considered in its entirety, i.e., as a whole,
including portions that would lead away from the claimed Invention. W.L. Gore & Associates,
Inc. v. Garlock, Inc., 721 F.2d 1540, 220 USPQ 303 (Fed. Cir. 1983), cert, denied, 469 U.S. 851
(1984). See also MPEP §2123.
Conclusion
19. Any inquiry concerning this communication or earlier communications from the
examiner should be directed to BYRON X KASPER whose telephone number is (571)272-3895.
The examiner can normally be reached Monday - Friday 8 am - 5 pm EST.
Examiner interviews are available via telephone, in-person, and video conferencing
using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is
encouraged to use the USPTO Automated Interview Request (AIR) at
http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s
supervisor, Adam Mott can be reached on (571) 270-5376. The fax phone number for the
organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be
obtained from Patent Center. Unpublished application information in Patent Center is available
to registered users. To file and manage patent submissions in Patent Center, visit:
https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for
more information about Patent Center and https://www.uspto.gov/patents/docx for
information about filing in DOCX format. For additional questions, contact the Electronic
Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO
Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/BYRON XAVIER KASPER/Examiner, Art Unit 3657
/ADAM R MOTT/Supervisory Patent Examiner, Art Unit 3657
Prosecution Insights