DETAILED ACTION
This is a non-final Office Action on the merits in response to communications filed by Applicant on July 6th, 2026. Claims 1 and 6-10 are currently pending and examined below.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
The amendments to the Claims filed on June 16th, 2026, have been entered. Claim 1 and 8 is currently amended and pending, claims 6-7 and 9 are original, unamended and pending, claim 10 is as previously presented, and claims 2-5 have been canceled.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 1, 6, 8, and 10 is/are rejected under 35 U.S.C. 103 as being unpatentable over US 10040199 B2 ("Ito") in view of US 11126844 B2 ("Hayashi") in further view of CN 112802105 A ("Duan") in further view of CN 111462232 A ("Zhu").
Regarding claim 1, Ito teaches a robot sorting method based on visual recognition, comprising (Ito: Figure 1, Abstract, “Provided is a processing apparatus for determining a work to be picked by a robot from a plurality of works, using an image, captured by an image capture device, of an area on which the plurality of works are placed. The apparatus selects a pickable candidate work based on the image, and determines a picking target work positioned in a partial area assigned with a highest priority among the partial areas where candidate works are respectively positioned. The apparatus selects a next pickable candidate work whose position and orientation have changed within allowable ranges before and after picking, and determines a next picking target work positioned in the partial area assigned with the highest priority among the partial areas where the next pickable candidate works are respectively positioned.”, Column 3 lines 22-46, “In accordance with a command from a processing apparatus 200, the measurement controller 110 for controlling the three-dimensional measurement apparatus 100 causes the light projector 120 to project pattern light on a pallet 400, and also causes the image capture device 130 to capture an area in the pallet 400 (container). The processing apparatus 200 is, for example, a computer apparatus. Note that the processing apparatus 200 may also function as the measurement controller 110. In the pallet 400, a number of works as picking targets of the picking system and as measurement targets of the three-dimensional measurement apparatus 100 are piled. The measurement controller 110 transmits the image capturing result of the image capture device 130 to the processing apparatus 200. The processing apparatus 200 recognizes the works in the pallet 400 and obtains position and orientation information based on data of an image captured by the image capture device 130, and selects a pickable candidate work. The information of the pickable candidate work is transmitted from the processing apparatus 200 to a robot 300, and the robot 300 moves a hand 310 and an arm 320 based on the information and picks the predetermined work. In the conventional measurement operation of the three-dimensional measurement apparatus, the above steps are repeatedly performed every time the robot 300 picks a work.”):
S1: acquiring a plurality of grating images in a visual field range (Ito: Column 3 lines 22-46, “In accordance with a command from a processing apparatus 200, the measurement controller 110 for controlling the three-dimensional measurement apparatus 100 causes the light projector 120 to project pattern light on a pallet 400, and also causes the image capture device 130 to capture an area in the pallet 400 (container).”, Column 5 lines 26-49, “The procedure of measurement and picking according to the embodiment will be described with reference to a flowchart shown in FIG. 5. In step S21, the three-dimensional measurement apparatus 100 performs normal three dimensional measurement. This three-dimensional measurement is implemented by, for example, a known technique of measuring the three-dimensional positions of works by projecting pattern light on the piled works and capturing the works, and a detailed description thereof will be omitted.”. One of ordinary skill in the art would recognize that projecting a pattern light and capturing an image is a grating image.),
S2: acquiring a posture matrix of each of visible workpieces in a grabbing space from the point cloud set information, wherein the visible workpieces comprise workpieces that are stacked with each other (Ito: Figure 2, Column 3 lines 22-46, “In accordance with a command from a processing apparatus 200, the measurement controller 110 for controlling the three-dimensional measurement apparatus 100 causes the light projector 120 to project pattern light on a pallet 400, and also causes the image capture device 130 to capture an area in the pallet 400 (container). The processing apparatus 200 is, for example, a computer apparatus. Note that the processing apparatus 200 may also function as the measurement controller 110. In the pallet 400, a number of works as picking targets of the picking system and as measurement targets of the three-dimensional measurement apparatus 100 are piled. The measurement controller 110 transmits the image capturing result of the image capture device 130 to the processing apparatus 200. The processing apparatus 200 recognizes the works in the pallet 400 and obtains position and orientation information based on data of an image captured by the image capture device 130, and selects a pickable candidate work.”, Column 3 lines 55-65, “In FIG. 2, 2a shows the orientation of the robot 300 when picking a work in the block at the upper right end among a number of works 410 piled in the area in the pallet 400.”, Column 6 lines 18-47, “The three-dimensional measurement apparatus 100 transfers the obtained captured data to the processing apparatus 200, as needed. The processing apparatus 200 performs the three-dimensional measurement calculation of the works based on the captured data, the by obtaining the positions and orientations of the works.”. The cited passages and figures clearly shows that there are a plurality of workpieces that are plied together on a pallet or other storage container. One of ordinary skill in the art would recognize that the workpieces are stacked together and the visible work pieces are those that are not completely occluded by other workpieces. Furthermore, the cited passages clearly shows that the pose of the workpieces are determined for every visible workpiece in the captured image);
and determining a target grabbing sequence (Ito: Column 7 line 64 – Colum 8 line 9, “The result of priority setting based on the distances is used to determine, if there exist a plurality of candidate works in one block, the picking sequence of the plurality of candidate works. Priorities among a plurality of candidate works of different blocks comply with priorities assigned to the blocks in which the respective candidate works are positioned, as in the example of the first embodiment. Consequently, in the example of FIG. 7, the picking sequence of the respective candidate works is set to "first candidate work 411[Wingdings font/0xE0]second candidate work 412[Wingdings font/0xE0]third candidate work 413[Wingdings font/0xE0]sixth candidate work 416[Wingdings font/0xE0]fifth candidate work 415[Wingdings font/0xE0]fourth candidate work 414[Wingdings font/0xE0]seventh candidate work 417".”),
and grabbing the workpieces according to the grabbing sequence (Ito: Column 3 lines 22-46, “In accordance with a command from a processing apparatus 200, the measurement controller 110 for controlling the three-dimensional measurement apparatus 100 causes the light projector 120 to project pattern light on a pallet 400, and also causes the image capture device 130 to capture an area in the pallet 400 (container). The processing apparatus 200 is, for example, a computer apparatus. Note that the processing apparatus 200 may also function as the measurement controller 110. In the pallet 400, a number of works as picking targets of the picking system and as measurement targets of the three-dimensional measurement apparatus 100 are piled. The measurement controller 110 transmits the image capturing result of the image capture device 130 to the processing apparatus 200. The processing apparatus 200 recognizes the works in the pallet 400 and obtains position and orientation information based on data of an image captured by the image capture device 130, and selects a pickable candidate work.”, Column 6 lines 18-47, “The three-dimensional measurement apparatus 100 transfers the obtained captured data to the processing apparatus 200, as needed. The processing apparatus 200 performs the three-dimensional measurement calculation of the works based on the captured data, the by obtaining the positions and orientations of the works. The processing apparatus 200 transfers the information to the robot 300, and the robot 300 performs a picking operation based on the information.”, Column 7 line 64 – Colum 8 line 9, “The result of priority setting based on the distances is used to determine, if there exist a plurality of candidate works in one block, the picking sequence of the plurality of candidate works. Priorities among a plurality of candidate works of different blocks comply with priorities assigned to the blocks in which the respective candidate works are positioned, as in the example of the first embodiment. Consequently, in the example of FIG. 7, the picking sequence of the respective candidate works is set to "first candidate work 411[Wingdings font/0xE0]second candidate work 412[Wingdings font/0xE0]third candidate work 413[Wingdings font/0xE0]sixth candidate work 416[Wingdings font/0xE0]fifth candidate work 415[Wingdings font/0xE0]fourth candidate work 414[Wingdings font/0xE0]seventh candidate work 417".”. The system is clearly configured to grasp the objects according to the determined sequence.);
S42: determining a target grabbing sequence for each of the workpieces according to the first priority, the second priority and the third priority of each of the workpieces (Ito: Column 7 line 64 – Colum 8 line 9, “The result of priority setting based on the distances is used to determine, if there exist a plurality of candidate works in one block, the picking sequence of the plurality of candidate works. Priorities among a plurality of candidate works of different blocks comply with priorities assigned to the blocks in which the respective candidate works are positioned, as in the example of the first embodiment. Consequently, in the example of FIG. 7, the picking sequence of the respective candidate works is set to "first candidate work 411[Wingdings font/0xE0]second candidate work 412[Wingdings font/0xE0]third candidate work 413[Wingdings font/0xE0]sixth candidate work 416[Wingdings font/0xE0]fifth candidate work 415[Wingdings font/0xE0]fourth candidate work 414[Wingdings font/0xE0]seventh candidate work 417".”. The cited passages clearly teach that the grasping order is determined based on a plurality of priorities.);
and grabbing the workpieces according to the target grabbing sequence (Ito: Column 3 lines 22-46, “In accordance with a command from a processing apparatus 200, the measurement controller 110 for controlling the three-dimensional measurement apparatus 100 causes the light projector 120 to project pattern light on a pallet 400, and also causes the image capture device 130 to capture an area in the pallet 400 (container). The processing apparatus 200 is, for example, a computer apparatus. Note that the processing apparatus 200 may also function as the measurement controller 110. In the pallet 400, a number of works as picking targets of the picking system and as measurement targets of the three-dimensional measurement apparatus 100 are piled. The measurement controller 110 transmits the image capturing result of the image capture device 130 to the processing apparatus 200. The processing apparatus 200 recognizes the works in the pallet 400 and obtains position and orientation information based on data of an image captured by the image capture device 130, and selects a pickable candidate work.”, Column 6 lines 18-47, “The three-dimensional measurement apparatus 100 transfers the obtained captured data to the processing apparatus 200, as needed. The processing apparatus 200 performs the three-dimensional measurement calculation of the works based on the captured data, the by obtaining the positions and orientations of the works. The processing apparatus 200 transfers the information to the robot 300, and the robot 300 performs a picking operation based on the information.”, Column 7 line 64 – Colum 8 line 9, “The result of priority setting based on the distances is used to determine, if there exist a plurality of candidate works in one block, the picking sequence of the plurality of candidate works. Priorities among a plurality of candidate works of different blocks comply with priorities assigned to the blocks in which the respective candidate works are positioned, as in the example of the first embodiment. Consequently, in the example of FIG. 7, the picking sequence of the respective candidate works is set to "first candidate work 411[Wingdings font/0xE0]second candidate work 412[Wingdings font/0xE0]third candidate work 413[Wingdings font/0xE0]sixth candidate work 416[Wingdings font/0xE0]fifth candidate work 415[Wingdings font/0xE0]fourth candidate work 414[Wingdings font/0xE0]seventh candidate work 417".”. The system is clearly configured to grasp the objects according to the determined sequence. ).
Ito does not teach and performing recognition and fusion to obtain point cloud set information;
S3: acquiring contour feature data of each of the workpieces according to the posture matrix, the contour feature data includes depth information,
a first stacking parameter,
and a second stacking parameter,
wherein the first stacking parameter is a placing state of a workpiece
and the second stacking parameter is the smallest one of the distances between a geometric center of one of the workpieces and each of workpiece edges of the workpieces; and
S4: comparing the contour feature data of the workpieces;
wherein S3 comprises: S31: determining a contour curve and depth information of each of the workpieces according to the posture matrix; and
S32: analyzing a stacking degree according to the contour curve, and calculating a stacking index;
wherein S32 comprises: A: determining workpiece edges of each of the workpieces according to the contour curve;
B: determining a placing state of the workpiece as a first stacking parameter according to the depth information and the workpiece edges, and calculating a geometric center of the workpiece; and
C: calculating distances between the geometric center and the workpiece edges, selecting a smallest one of the distances as a second stacking parameter, and determining the stacking index of the workpiece in combination with the first stacking parameter;
wherein, the placing state comprises a natural putting state or a stacking state;
wherein S4 comprises: S41: sequentially determining a first priority, a second priority and a third priority of each of the workpieces in the grabbing space according to the depth information, the first stacking parameter and the second stacking parameter;
S43: determining a product model according to the contour curve of the workpiece;
wherein in S41: the deeper the workpiece is, the lower the priority is; the workpiece in the natural putting state has a higher priority than the workpiece in stacking state; and the greater a numerical value of the second stacking parameter is, the higher the priority is;
and the first priority, the second priority and the third priority are sequentially decreased.
Hayashi, in the same field of endeavor, teaches and performing recognition and fusion to obtain point cloud set information (Hayashi: Column 5 lines 26-41, “In the second point cloud generation process, first, a second imaging process of imaging a plurality of phase shifting images PSM1 to PSM4 containing the parts PP with a plurality of phase shifting patterns PH1 to PH4 selectively projected one by one as the projection patterns is performed using the projection device 810. The plurality of phase shifting patterns PH1 to PH4 are sinusoidal banded patterns having dark and light parts. In the phase shifting method, generally, capturing of images is performed using the n phase shift patterns PH2 to PHn for n (integer) equal to or larger than three. The n phase shift patterns PH1 to PHn are sinusoidal patterns having phase sequentially shifted by 2π/n. In the example of FIG. 7, n=4. Note that, in the phase shifting method, it is not necessary to use the stereo camera, and only one of the two cameras forming the stereo camera may be used for imaging.”, Column 5 lines 42-52, “In the second point cloud generation process, then, a second analysis process is executed using the plurality of phase shifting images PSM1 to PSM4, and thereby, a second point cloud PG2 is generated. In the second analysis process, first, a distance image DMZ is generated according to the phase shifting method using the plurality of phase shifting images PSM1 to PSM4. The generation process of the distance image DMZ using the phase shifting method is well known, and the explanation thereof is omitted here. In the second point cloud generation process, a second point cloud PG2 is further generated using the distance image DMZ.”. The cited passages clearly teach that the system uses known techniques to create a point cloud from a grating image.);
S3: acquiring contour feature data of each of the workpieces according to the posture matrix, the contour feature data includes depth information (Column 5 lines 26-41, “In the second point cloud generation process, first, a second imaging process of imaging a plurality of phase shifting images PSM1 to PSM4 containing the parts PP with a plurality of phase shifting patterns PH1 to PH4 selectively projected one by one as the projection patterns is performed using the projection device 810. The plurality of phase shifting patterns PH1 to PH4 are sinusoidal banded patterns having dark and light parts. In the phase shifting method, generally, capturing of images is performed using the n phase shift patterns PH2 to PHn for n (integer) equal to or larger than three. The n phase shift patterns PH1 to PHn are sinusoidal patterns having phase sequentially shifted by 2π/n. In the example of FIG. 7, n=4. Note that, in the phase shifting method, it is not necessary to use the stereo camera, and only one of the two cameras forming the stereo camera may be used for imaging.”, Column 5 lines 42-52, “In the second point cloud generation process, then, a second analysis process is executed using the plurality of phase shifting images PSM1 to PSM4, and thereby, a second point cloud PG2 is generated. In the second analysis process, first, a distance image DMZ is generated according to the phase shifting method using the plurality of phase shifting images PSM1 to PSM4. The generation process of the distance image DMZ using the phase shifting method is well known, and the explanation thereof is omitted here. In the second point cloud generation process, a second point cloud PG2 is further generated using the distance image DMZ.”, Column 6 lines 43-48, “At step S190, if the first point cloud generation part 271 stops the process, the second point cloud generation part 272 continues execution of the second point cloud generation process. Then, at step S230, the object detection execution part 273 detects the three-dimensional shapes of the parts PP as objects using the second point cloud PG2.”, Column 6 lines 49-63, “As the method of detecting the three-dimensional shapes of the objects using the point cloud, e.g. the method described in JP-A-2017-182274 disclosed by the applicant of this application can be used. Or, the method described in "Model Globally, Match Locally: Efficient and Robust 3D Object Recognition", Bertram Drost et al., http://campar.in-.tum.de/pub/drost20l 0CVPR/drost201 0CVPR.pdf, the method described in "Bekutoru Pea Mattingu niyoru Kousinraina Sanjigen Ichi Shisei Ninshiki", Shuichi Akizuki, http://isl.sist.chukyou.ac.jp/Archives/vpm.pdf, or the like may be used. These methods are object detection methods using the point cloud and the three-dimensional model data of the objects. Note that the three-dimensional shapes of the objects may be detected from the point cloud without using the three-dimensional model data.”. The cited passages from Hayashi shows that the contour curve and the depth information of each article is determined based on the grating image.);
wherein S3 comprises: S31: determining a contour curve and depth information of each of the workpieces according to the posture matrix (Hayashi: Column 5 lines 26-41, “In the second point cloud generation process, first, a second imaging process of imaging a plurality of phase shifting images PSM1 to PSM4 containing the parts PP with a plurality of phase shifting patterns PH1 to PH4 selectively projected one by one as the projection patterns is performed using the projection device 810. The plurality of phase shifting patterns PH1 to PH4 are sinusoidal banded patterns having dark and light parts. In the phase shifting method, generally, capturing of images is performed using the n phase shift patterns PH2 to PHn for n (integer) equal to or larger than three. The n phase shift patterns PH1 to PHn are sinusoidal patterns having phase sequentially shifted by 2π/n. In the example of FIG. 7, n=4. Note that, in the phase shifting method, it is not necessary to use the stereo camera, and only one of the two cameras forming the stereo camera may be used for imaging.”, Column 5 lines 42-52, “In the second point cloud generation process, then, a second analysis process is executed using the plurality of phase shifting images PSM1 to PSM4, and thereby, a second point cloud PG2 is generated. In the second analysis process, first, a distance image DMZ is generated according to the phase shifting method using the plurality of phase shifting images PSM1 to PSM4. The generation process of the distance image DMZ using the phase shifting method is well known, and the explanation thereof is omitted here. In the second point cloud generation process, a second point cloud PG2 is further generated using the distance image DMZ.”, Column 6 lines 43-48, “At step S190, if the first point cloud generation part 271 stops the process, the second point cloud generation part 272 continues execution of the second point cloud generation process. Then, at step S230, the object detection execution part 273 detects the three-dimensional shapes of the parts PP as objects using the second point cloud PG2.”, Column 6 lines 49-63, “As the method of detecting the three-dimensional shapes of the objects using the point cloud, e.g. the method described in JP-A-2017-182274 disclosed by the applicant of this application can be used. Or, the method described in "Model Globally, Match Locally: Efficient and Robust 3D Object Recognition", Bertram Drost et al., http://campar.in-.tum.de/pub/drost20l 0CVPR/drost201 0CVPR.pdf, the method described in "Bekutoru Pea Mattingu niyoru Kousinraina Sanjigen Ichi Shisei Ninshiki", Shuichi Akizuki, http://isl.sist.chukyou.ac.jp/Archives/vpm.pdf, or the like may be used. These methods are object detection methods using the point cloud and the three-dimensional model data of the objects. Note that the three-dimensional shapes of the objects may be detected from the point cloud without using the three-dimensional model data.”. The cited passages show that the contour curve and depth information of each article is determined based on the grating image.);
S43: determining a product model according to the contour curve of the workpiece (Hayashi: Column 6 lines 43-48, “At step S190, if the first point cloud generation part 271 stops the process, the second point cloud generation part 272 continues execution of the second point cloud generation process. Then, at step S230, the object detection execution part 273 detects the three-dimensional shapes of the parts PP as objects using the second point cloud PG2.”, Column 6 lines 49-63, “As the method of detecting the three-dimensional shapes of the objects using the point cloud, e.g. the method described in JP-A-2017-182274 disclosed by the applicant of this application can be used. Or, the method described in "Model Globally, Match Locally: Efficient and Robust 3D Object Recognition", Bertram Drost et al., http://campar.in-.tum.de/pub/drost20l 0CVPR/drost201 0CVPR.pdf, the method described in "Bekutoru Pea Mattingu niyoru Kousinraina Sanjigen Ichi Shisei Ninshiki", Shuichi Akizuki, http://isl.sist.chukyou.ac.jp/Archives/vpm.pdf, or the like may be used. These methods are object detection methods using the point cloud and the three-dimensional model data of the objects. Note that the three-dimensional shapes of the objects may be detected from the point cloud without using the three-dimensional model data.”);
Ito teaches a robot sorting method based on visual recognition, comprising: S1: acquiring a plurality of grating images in a visual field range, S2: acquiring a posture matrix of each of visible workpieces in a grabbing space from the point cloud set information, wherein the visible workpieces comprise workpieces that are stacked with each other; and determining a target grabbing sequence to grab the workpieces. Ito does not teach and performing recognition and fusion to obtain point cloud set information; S3: acquiring contour feature data of each of the workpieces according to the posture matrix, the contour feature data includes depth information wherein S3 comprises: S31: determining a contour curve and depth information of each of the workpieces according to the posture matrix; S43: determining a product model according to the contour curve of the workpiece. Hayashi teaches and performing recognition and fusion to obtain point cloud set information; S3: acquiring contour feature data of each of the workpieces according to the posture matrix, the contour feature data includes depth information wherein S3 comprises: S31: determining a contour curve and depth information of each of the workpieces according to the posture matrix; S43: determining a product model according to the contour curve of the workpiece. A person of ordinary skill in the art would have had the technological capabilities required to have modified the method taught in Ito with and performing recognition and fusion to obtain point cloud set information; S3: acquiring contour feature data of each of the workpieces according to the posture matrix, the contour feature data includes depth information wherein S3 comprises: S31: determining a contour curve and depth information of each of the workpieces according to the posture matrix; S43: determining a product model according to the contour curve of the workpiece taught in Hayashi. Furthermore, the method taught in Ito is already configured to acquire depth information from a grating image created by projecting a pattern of lines on the objects. Hayashi teaches a similar method of generating a grating image and acquiring depth from the grating image to construct a point cloud. As such, the method taught in Ito is readily configurable with the method of constructing a point cloud taught in Hayashi and can be simply added to the method taught in Ito according to known methods. Additionally, Ito teaches all of the necessary components required to acquire contour information of the objects taught in Hayashi. Such modifications would not have changed or introduced new functionality. No inventive effort would have been required. The combination would have yielded the predictable result of a robot sorting method based on visual recognition, comprising: and performing recognition and fusion to obtain point cloud set information; S3: acquiring contour feature data of each of the workpieces according to the posture matrix, the contour feature data includes depth information wherein S3 comprises: S31: determining a contour curve and depth information of each of the workpieces according to the posture matrix; S43: determining a product model according to the contour curve of the workpiece.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filling date of the claimed invention, to have combine the robot sorting method taught in Ito with and performing recognition and fusion to obtain point cloud set information; S3: acquiring contour feature data of each of the workpieces according to the posture matrix, the contour feature data includes depth information wherein S3 comprises: S31: determining a contour curve and depth information of each of the workpieces according to the posture matrix; S43: determining a product model according to the contour curve of the workpiece taught in Hayashi with a reasonable expectation of success. One of ordinary skill in the art would have been motivated to make this modification because the combination would have yielded predictable results.
Ito in view of Hayashi does not teach the a first stacking parameter,
and a second stacking parameter,
wherein the first stacking parameter is a placing state of a workpiece
and the second stacking parameter is the smallest one of the distances between a geometric center of one of the workpieces and each of workpiece edges of the workpieces; and
S4: comparing the contour feature data of the workpieces;
S32: analyzing a stacking degree according to the contour curve, and calculating a stacking index;
wherein S32 comprises: A: determining workpiece edges of each of the workpieces according to the contour curve;
B: determining a placing state of the workpiece as a first stacking parameter according to the depth information and the workpiece edges, and calculating a geometric center of the workpiece; and
C: calculating distances between the geometric center and the workpiece edges, selecting a smallest one of the distances as a second stacking parameter, and determining the stacking index of the workpiece in combination with the first stacking parameter;
wherein, the placing state comprises a natural putting state or a stacking state;
wherein S4 comprises: S41: sequentially determining a first priority, a second priority and a third priority of each of the workpieces in the grabbing space according to the depth information, the first stacking parameter and the second stacking parameter;
wherein in S41: the deeper the workpiece is, the lower the priority is; the workpiece in the natural putting state has a higher priority than the workpiece in stacking state; and the greater a numerical value of the second stacking parameter is, the higher the priority is;
and the first priority, the second priority and the third priority are sequentially decreased.
Duan, in the same field of endeavor, teaches a first stacking parameter (Duan: ¶ 0083, “First, sample images corresponding to the three-dimensional sample area are acquired to identify multiple object objects contained in the sample images. The three-dimensional sample area contains multiple items to be grabbed, which serve as samples. The sample image corresponding to the three-dimensional sample region includes: a two-dimensional color image corresponding to the three-dimensional sample region obtained along a preset depth direction, and a depth image corresponding to the two-dimensional color image. For specific methods of obtaining the information, please refer to the corresponding description in step S110, which will not be repeated here. In determining multiple object objects contained in a sample image, the contours, boundary lines, and other information between each object can be identified through instance segmentation, and then the multiple object objects contained in the sample image can be segmented based on the recognition results.”, ¶ 0086, “In another alternative implementation, based on the exposure ratio of each item object, the area corresponding to the item object with an exposure ratio greater than a preset threshold is marked as a crawlable area, and the area corresponding to the item object with an exposure ratio not greater than the preset threshold is marked as an uncrawlable area. In some scenarios, the stacking relationship between items is not easy to determine, and items on the same layer may overlap each other. In such cases, it is difficult to accurately label the top item. For example, in supermarket product picking scenarios, the hierarchical relationship between products is not clear, and there may be overlapping between items on the same layer. Therefore, the requirements for the picking order are not strict, but the distinction between pickable and non-pickable items is more stringent. In this case, items with little surface exposure, or items that may cause other items in the scene to fly out after being grabbed, should not be marked as grabbable items. Accordingly, in the above scenario, a labeling threshold can be set, such as 85%. If an item's exposed surface area is greater than 85%, it is marked as a graspable item; if an item's exposed surface area is less than 85%, it is marked as a non-grabable item. Of course, the exposure ratio of an item can be quantified not only by the exposed surface area but also by the exposed volume. This invention does not limit the specific details.”. The cited passages teach that the first stacking parameter of the object is set based on both the depth information and how much of the object is exposed (i.e. not occluded).),
wherein the first stacking parameter is a placing state of a workpiece (Duan: ¶ 0083, “First, sample images corresponding to the three-dimensional sample area are acquired to identify multiple object objects contained in the sample images. The three-dimensional sample area contains multiple items to be grabbed, which serve as samples. The sample image corresponding to the three-dimensional sample region includes: a two-dimensional color image corresponding to the three-dimensional sample region obtained along a preset depth direction, and a depth image corresponding to the two-dimensional color image. For specific methods of obtaining the information, please refer to the corresponding description in step S110, which will not be repeated here. In determining multiple object objects contained in a sample image, the contours, boundary lines, and other information between each object can be identified through instance segmentation, and then the multiple object objects contained in the sample image can be segmented based on the recognition results.”, ¶ 0086, “In another alternative implementation, based on the exposure ratio of each item object, the area corresponding to the item object with an exposure ratio greater than a preset threshold is marked as a crawlable area, and the area corresponding to the item object with an exposure ratio not greater than the preset threshold is marked as an uncrawlable area. In some scenarios, the stacking relationship between items is not easy to determine, and items on the same layer may overlap each other. In such cases, it is difficult to accurately label the top item. For example, in supermarket product picking scenarios, the hierarchical relationship between products is not clear, and there may be overlapping between items on the same layer. Therefore, the requirements for the picking order are not strict, but the distinction between pickable and non-pickable items is more stringent. In this case, items with little surface exposure, or items that may cause other items in the scene to fly out after being grabbed, should not be marked as grabbable items. Accordingly, in the above scenario, a labeling threshold can be set, such as 85%. If an item's exposed surface area is greater than 85%, it is marked as a graspable item; if an item's exposed surface area is less than 85%, it is marked as a non-grabable item. Of course, the exposure ratio of an item can be quantified not only by the exposed surface area but also by the exposed volume. This invention does not limit the specific details.”. The cited passages teach that the stacking state of the object is set based on both the depth information and how much of the object is exposed (i.e. not occluded).) and
S4: comparing the contour feature data of the workpieces (Duan: ¶ 0083, “In determining multiple object objects contained in a sample image, the contours, boundary lines, and other information between each object can be identified through instance segmentation, and then the multiple object objects contained in the sample image can be segmented based on the recognition results.”, ¶ 0086, “In another alternative implementation, based on the exposure ratio of each item object, the area corresponding to the item object with an exposure ratio greater than a preset threshold is marked as a crawlable area, and the area corresponding to the item object with an exposure ratio not greater than the preset threshold is marked as an uncrawlable area. In some scenarios, the stacking relationship between items is not easy to determine, and items on the same layer may overlap each other. In such cases, it is difficult to accurately label the top item. For example, in supermarket product picking scenarios, the hierarchical relationship between products is not clear, and there may be overlapping between items on the same layer. Therefore, the requirements for the picking order are not strict, but the distinction between pickable and non-pickable items is more stringent. In this case, items with little surface exposure, or items that may cause other items in the scene to fly out after being grabbed, should not be marked as grabbable items. Accordingly, in the above scenario, a labeling threshold can be set, such as 85%. If an item's exposed surface area is greater than 85%, it is marked as a graspable item; if an item's exposed surface area is less than 85%, it is marked as a non-grabable item. Of course, the exposure ratio of an item can be quantified not only by the exposed surface area but also by the exposed volume. This invention does not limit the specific details.”, ¶ 0087, “In another alternative implementation, the contact area contained in each item object is determined according to the shape and/or type of each item object. The area corresponding to the item object whose contact area is not occluded is marked as a grabbable area, and the area corresponding to the item object whose contact area is occluded is marked as a non-grabable area. The contact area refers to the force-bearing area of the object that is easy to grasp. For example, for items with metal parts, in order to prevent damage to the parts, it is necessary to grasp a specific area of the metal parts. This specific area is the contact area, which is usually a relatively firm and not easily detached area of the metal parts. Therefore, when labeling such items, it is also necessary to determine whether the contact area is completely exposed and unobstructed. If an item has a large exposed surface area but the contact area is obstructed, it should still be marked as an ungrabable item.”. The cited passages clearly show that the contour information of each object is compared in order to determine a grasping order. The cited passage discloses using the exposed area of each object or the available contact area of each object, both of which are defined, in part, by the contours of the object.);
S32: analyzing a stacking degree according to the contour curve, and calculating a stacking index (Duan: ¶ 0015, “Determine the stacking order of each item object along the preset depth direction, mark the area corresponding to the top-level item object as a graspable area, and mark the area corresponding to the bottom-level item object as an ungraspable area; and/or”, ¶ 0016, “Based on the exposure ratio of each item, areas corresponding to item objects with exposure ratios greater than a preset threshold are marked as crawlable areas, while areas corresponding to item objects with exposure ratios not greater than the preset threshold are marked as uncrawlable areas; and/or,”, ¶ 0017, “Based on the shape and/or type of each item, determine the contact area contained in each item object, mark the area corresponding to the item object whose contact area is not occluded as the grabbable area, and mark the area corresponding to the item object whose contact area is occluded as the non-grabable area.”, ¶ 0083, “First, sample images corresponding to the three-dimensional sample area are acquired to identify multiple object objects contained in the sample images. The three-dimensional sample area contains multiple items to be grabbed, which serve as samples. The sample image corresponding to the three-dimensional sample region includes: a two-dimensional color image corresponding to the three-dimensional sample region obtained along a preset depth direction, and a depth image corresponding to the two-dimensional color image. For specific methods of obtaining the information, please refer to the corresponding description in step S110, which will not be repeated here. In determining multiple object objects contained in a sample image, the contours, boundary lines, and other information between each object can be identified through instance segmentation, and then the multiple object objects contained in the sample image can be segmented based on the recognition results.”, ¶ 0086, “In another alternative implementation, based on the exposure ratio of each item object, the area corresponding to the item object with an exposure ratio greater than a preset threshold is marked as a crawlable area, and the area corresponding to the item object with an exposure ratio not greater than the preset threshold is marked as an uncrawlable area. In some scenarios, the stacking relationship between items is not easy to determine, and items on the same layer may overlap each other. In such cases, it is difficult to accurately label the top item. For example, in supermarket product picking scenarios, the hierarchical relationship between products is not clear, and there may be overlapping between items on the same layer. Therefore, the requirements for the picking order are not strict, but the distinction between pickable and non-pickable items is more stringent. In this case, items with little surface exposure, or items that may cause other items in the scene to fly out after being grabbed, should not be marked as grabbable items. Accordingly, in the above scenario, a labeling threshold can be set, such as 85%. If an item's exposed surface area is greater than 85%, it is marked as a graspable item; if an item's exposed surface area is less than 85%, it is marked as a non-grabable item. Of course, the exposure ratio of an item can be quantified not only by the exposed surface area but also by the exposed volume. This invention does not limit the specific details.”, ¶ 0087, “In another alternative implementation, the contact area contained in each item object is determined according to the shape and/or type of each item object. The area corresponding to the item object whose contact area is not occluded is marked as a grabbable area, and the area corresponding to the item object whose contact area is occluded is marked as a non-grabable area. The contact area refers to the force-bearing area of the object that is easy to grasp. For example, for items with metal parts, in order to prevent damage to the parts, it is necessary to grasp a specific area of the metal parts. This specific area is the contact area, which is usually a relatively firm and not easily detached area of the metal parts. Therefore, when labeling such items, it is also necessary to determine whether the contact area is completely exposed and unobstructed. If an item has a large exposed surface area but the contact area is obstructed, it should still be marked as an ungrabable item.”. The cited passages clearly teach that the system is configured to determine a stacking degree and stacking index based, in part, on the contour curve of each object.);
wherein S32 comprises: A: determining workpiece edges of each of the workpieces according to the contour curve (Duan: ¶ 0083, “First, sample images corresponding to the three-dimensional sample area are acquired to identify multiple object objects contained in the sample images. The three-dimensional sample area contains multiple items to be grabbed, which serve as samples. The sample image corresponding to the three-dimensional sample region includes: a two-dimensional color image corresponding to the three-dimensional sample region obtained along a preset depth direction, and a depth image corresponding to the two-dimensional color image. For specific methods of obtaining the information, please refer to the corresponding description in step S110, which will not be repeated here. In determining multiple object objects contained in a sample image, the contours, boundary lines, and other information between each object can be identified through instance segmentation, and then the multiple object objects contained in the sample image can be segmented based on the recognition results.”. The edges of the objects are clearly determined.);
B: determining a placing state of the workpiece as a first stacking parameter according to the depth information and the workpiece edges (Duan: ¶ 0083, “First, sample images corresponding to the three-dimensional sample area are acquired to identify multiple object objects contained in the sample images. The three-dimensional sample area contains multiple items to be grabbed, which serve as samples. The sample image corresponding to the three-dimensional sample region includes: a two-dimensional color image corresponding to the three-dimensional sample region obtained along a preset depth direction, and a depth image corresponding to the two-dimensional color image. For specific methods of obtaining the information, please refer to the corresponding description in step S110, which will not be repeated here. In determining multiple object objects contained in a sample image, the contours, boundary lines, and other information between each object can be identified through instance segmentation, and then the multiple object objects contained in the sample image can be segmented based on the recognition results.”, ¶ 0086, “In another alternative implementation, based on the exposure ratio of each item object, the area corresponding to the item object with an exposure ratio greater than a preset threshold is marked as a crawlable area, and the area corresponding to the item object with an exposure ratio not greater than the preset threshold is marked as an uncrawlable area. In some scenarios, the stacking relationship between items is not easy to determine, and items on the same layer may overlap each other. In such cases, it is difficult to accurately label the top item. For example, in supermarket product picking scenarios, the hierarchical relationship between products is not clear, and there may be overlapping between items on the same layer. Therefore, the requirements for the picking order are not strict, but the distinction between pickable and non-pickable items is more stringent. In this case, items with little surface exposure, or items that may cause other items in the scene to fly out after being grabbed, should not be marked as grabbable items. Accordingly, in the above scenario, a labeling threshold can be set, such as 85%. If an item's exposed surface area is greater than 85%, it is marked as a graspable item; if an item's exposed surface area is less than 85%, it is marked as a non-grabable item. Of course, the exposure ratio of an item can be quantified not only by the exposed surface area but also by the exposed volume. This invention does not limit the specific details.”. The cited passages teach that the state of the object is set based on both the depth information and how much of the object is exposed (i.e. not occluded).),
wherein, the placing state comprises a natural putting state or a stacking state wherein, the placing state comprises a natural putting state or a stacking state (Duan: ¶ 0083, “First, sample images corresponding to the three-dimensional sample area are acquired to identify multiple object objects contained in the sample images. The three-dimensional sample area contains multiple items to be grabbed, which serve as samples. The sample image corresponding to the three-dimensional sample region includes: a two-dimensional color image corresponding to the three-dimensional sample region obtained along a preset depth direction, and a depth image corresponding to the two-dimensional color image. For specific methods of obtaining the information, please refer to the corresponding description in step S110, which will not be repeated here. In determining multiple object objects contained in a sample image, the contours, boundary lines, and other information between each object can be identified through instance segmentation, and then the multiple object objects contained in the sample image can be segmented based on the recognition results.”, ¶ 0086, “In another alternative implementation, based on the exposure ratio of each item object, the area corresponding to the item object with an exposure ratio greater than a preset threshold is marked as a crawlable area, and the area corresponding to the item object with an exposure ratio not greater than the preset threshold is marked as an uncrawlable area. In some scenarios, the stacking relationship between items is not easy to determine, and items on the same layer may overlap each other. In such cases, it is difficult to accurately label the top item. For example, in supermarket product picking scenarios, the hierarchical relationship between products is not clear, and there may be overlapping between items on the same layer. Therefore, the requirements for the picking order are not strict, but the distinction between pickable and non-pickable items is more stringent. In this case, items with little surface exposure, or items that may cause other items in the scene to fly out after being grabbed, should not be marked as grabbable items. Accordingly, in the above scenario, a labeling threshold can be set, such as 85%. If an item's exposed surface area is greater than 85%, it is marked as a graspable item; if an item's exposed surface area is less than 85%, it is marked as a non-grabable item. Of course, the exposure ratio of an item can be quantified not only by the exposed surface area but also by the exposed volume. This invention does not limit the specific details.”. One of ordinary skill in the art would recognize that objects that have an exposed surface area greater than the threshold do not have an object(s) stacked on top of it and are therefore grabbable (i.e. a natural putting state as no objects are on top of it). Objects have an exposed surface area less than the threshold have an object(s) stacked on top of it and are therefore not grabbable (i.e. a stacking state as objects are on top of it).);
wherein S4 comprises: S41: sequentially determining a first priority, a second priority and a third priority of each of the workpieces in the grabbing space according to the depth information, the first stacking parameter and the second stacking parameter (Duan: ¶ 0015, “Determine the stacking order of each item object along the preset depth direction, mark the area corresponding to the top-level item object as a graspable area, and mark the area corresponding to the bottom-level item object as an ungraspable area; and/or”, ¶ 0016, “Based on the exposure ratio of each item, areas corresponding to item objects with exposure ratios greater than a preset threshold are marked as crawlable areas, while areas corresponding to item objects with exposure ratios not greater than the preset threshold are marked as uncrawlable areas; and/or,”, ¶ 0017, “Based on the shape and/or type of each item, determine the contact area contained in each item object, mark the area corresponding to the item object whose contact area is not occluded as the grabbable area, and mark the area corresponding to the item object whose contact area is occluded as the non-grabable area.”, ¶ 0083, “First, sample images corresponding to the three-dimensional sample area are acquired to identify multiple object objects contained in the sample images. The three-dimensional sample area contains multiple items to be grabbed, which serve as samples. The sample image corresponding to the three-dimensional sample region includes: a two-dimensional color image corresponding to the three-dimensional sample region obtained along a preset depth direction, and a depth image corresponding to the two-dimensional color image. For specific methods of obtaining the information, please refer to the corresponding description in step S110, which will not be repeated here. In determining multiple object objects contained in a sample image, the contours, boundary lines, and other information between each object can be identified through instance segmentation, and then the multiple object objects contained in the sample image can be segmented based on the recognition results.”, ¶ 0086, “In another alternative implementation, based on the exposure ratio of each item object, the area corresponding to the item object with an exposure ratio greater than a preset threshold is marked as a crawlable area, and the area corresponding to the item object with an exposure ratio not greater than the preset threshold is marked as an uncrawlable area. In some scenarios, the stacking relationship between items is not easy to determine, and items on the same layer may overlap each other. In such cases, it is difficult to accurately label the top item. For example, in supermarket product picking scenarios, the hierarchical relationship between products is not clear, and there may be overlapping between items on the same layer. Therefore, the requirements for the picking order are not strict, but the distinction between pickable and non-pickable items is more stringent. In this case, items with little surface exposure, or items that may cause other items in the scene to fly out after being grabbed, should not be marked as grabbable items. Accordingly, in the above scenario, a labeling threshold can be set, such as 85%. If an item's exposed surface area is greater than 85%, it is marked as a graspable item; if an item's exposed surface area is less than 85%, it is marked as a non-grabable item. Of course, the exposure ratio of an item can be quantified not only by the exposed surface area but also by the exposed volume. This invention does not limit the specific details.”, ¶ 0087, “In another alternative implementation, the contact area contained in each item object is determined according to the shape and/or type of each item object. The area corresponding to the item object whose contact area is not occluded is marked as a grabbable area, and the area corresponding to the item object whose contact area is occluded is marked as a non-grabable area. The contact area refers to the force-bearing area of the object that is easy to grasp. For example, for items with metal parts, in order to prevent damage to the parts, it is necessary to grasp a specific area of the metal parts. This specific area is the contact area, which is usually a relatively firm and not easily detached area of the metal parts. Therefore, when labeling such items, it is also necessary to determine whether the contact area is completely exposed and unobstructed. If an item has a large exposed surface area but the contact area is obstructed, it should still be marked as an ungrabable item.”);
wherein in S41: the deeper the workpiece is, the lower the priority is (Duan: ¶ 0085, “In one optional implementation, the stacking order of each item object along a preset depth direction is determined, the area corresponding to the item object at the top level is marked as a grabbable area, and the area corresponding to the item object at the bottom level is marked as an ungrabable area. Conventional instance segmentation algorithms do not distinguish whether items in the scene are grabbable; that is, they require a complete and accurate instance mask for all items in the scene. Therefore, if traditional instance segmentation algorithms are directly applied to the identification of graspable areas, it will cause the crushed items at the bottom to be identified as graspable items or the irrelevant background items to be identified as graspable items, thus causing identification errors. To prevent the aforementioned problems, this implementation determines the stacking order of each item object along a preset depth direction, thereby marking the area corresponding to the top-level item object as a graspable area and the area corresponding to the bottom-level item object as an ungraspable area, thus avoiding anomalies caused by the robot grasping the bottom-level items. For example, in a cardboard box destacking scenario, the destacking should proceed layer by layer from the top to the bottom, and the lower cardboard boxes should not be grabbed before the lower cardboard boxes have been completely grabbed. Therefore, in similar scenarios, only the topmost cardboard box is marked as a grabbable object, while all other cardboard boxes are marked as non-grabable objects. This labeling method can accurately distinguish between items on the top layer and those not on the top layer, thus providing accurate pixel-level item positioning.”. The cited passage clearly shows that the deeper the object, the lower the priority.);
the workpiece in the natural putting state has a higher priority than the workpiece in stacking state (Duan: ¶ 0086, “In another alternative implementation, based on the exposure ratio of each item object, the area corresponding to the item object with an exposure ratio greater than a preset threshold is marked as a crawlable area, and the area corresponding to the item object with an exposure ratio not greater than the preset threshold is marked as an uncrawlable area. In some scenarios, the stacking relationship between items is not easy to determine, and items on the same layer may overlap each other. In such cases, it is difficult to accurately label the top item. For example, in supermarket product picking scenarios, the hierarchical relationship between products is not clear, and there may be overlapping between items on the same layer. Therefore, the requirements for the picking order are not strict, but the distinction between pickable and non-pickable items is more stringent. In this case, items with little surface exposure, or items that may cause other items in the scene to fly out after being grabbed, should not be marked as grabbable items. Accordingly, in the above scenario, a labeling threshold can be set, such as 85%. If an item's exposed surface area is greater than 85%, it is marked as a graspable item; if an item's exposed surface area is less than 85%, it is marked as a non-grabable item. Of course, the exposure ratio of an item can be quantified not only by the exposed surface area but also by the exposed volume. This invention does not limit the specific details.”. One of ordinary skill in the art would recognize that objects that have an exposed surface area greater than the threshold do not have an object(s) stacked on top of it and are therefore grabbable (i.e. a natural putting state as no objects are on top of it). Objects have an exposed surface area less than the threshold have an object(s) stacked on top of it and are therefore not grabbable (i.e. a stacking state as objects are on top of it). Additionally, it is obvious that the objects with an exposed surface area above the threshold has a higher priority than those that do not.);
and the first priority, the second priority and the third priority are sequentially decreased (Duan: ¶ 0015, “Determine the stacking order of each item object along the preset depth direction, mark the area corresponding to the top-level item object as a graspable area, and mark the area corresponding to the bottom-level item object as an ungraspable area; and/or”, ¶ 0016, “Based on the exposure ratio of each item, areas corresponding to item objects with exposure ratios greater than a preset threshold are marked as crawlable areas, while areas corresponding to item objects with exposure ratios not greater than the preset threshold are marked as uncrawlable areas; and/or,”, ¶ 0017, “Based on the shape and/or type of each item, determine the contact area contained in each item object, mark the area corresponding to the item object whose contact area is not occluded as the grabbable area, and mark the area corresponding to the item object whose contact area is occluded as the non-grabable area.”, ¶ 0083, ¶ 0085, ¶ 0086, ¶ 0087. It is obvious from the cited passages that the priorities decrease sequentially in the order they are listed, i.e. objects at the top of the pile are, in an unstacked state (i.e. a natural placing state), and have the largest smallest distance from its center point are given the highest priority, whereas objects at the top of the pile are, in an unstacked state (i.e. a natural placing state), and have the smallest distance from its center point are given a lower priority.).
Ito in view of Hayashi teaches a robot sorting method based on visual recognition, comprising: S1: acquiring a plurality of grating images in a visual field range, and performing recognition and fusion to obtain point cloud set information; S2: acquiring a posture matrix of each of workpieces in a grabbing space from the point cloud set information; S3: acquiring contour feature data of each of the workpieces according to the posture matrix, the contour feature data includes depth information and determining a target grabbing sequence to grab the workpieces. Ito in view of Hayashi does not teach a first stacking parameter, and a second stacking parameter, wherein the first stacking parameter is a placing state of a workpiece S4: comparing the contour feature data of the workpieces; S32: analyzing a stacking degree according to the contour curve, and calculating a stacking index; wherein S32 comprises: A: determining workpiece edges of each of the workpieces according to the contour curve; B: determining a placing state of the workpiece as a first stacking parameter according to the depth information and the workpiece edges; wherein, the placing state comprises a natural putting state or a stacking state; wherein S4 comprises: S41: sequentially determining a first priority, a second priority and a third priority of each of the workpieces in the grabbing space according to the depth information, the first stacking parameter and the second stacking parameter; wherein in S41: the deeper the workpiece is, the lower the priority is; the workpiece in the natural putting state has a higher priority than the workpiece in stacking state; and the first priority, the second priority and the third priority are sequentially decreased. Duan teaches a first stacking parameter, and a second stacking parameter, wherein the first stacking parameter is a placing state of a workpiece S4: comparing the contour feature data of the workpieces; S32: analyzing a stacking degree according to the contour curve, and calculating a stacking index; wherein S32 comprises: A: determining workpiece edges of each of the workpieces according to the contour curve; B: determining a placing state of the workpiece as a first stacking parameter according to the depth information and the workpiece edges; wherein, the placing state comprises a natural putting state or a stacking state; wherein S4 comprises: S41: sequentially determining a first priority, a second priority and a third priority of each of the workpieces in the grabbing space according to the depth information, the first stacking parameter and the second stacking parameter; wherein in S41: the deeper the workpiece is, the lower the priority is; the workpiece in the natural putting state has a higher priority than the workpiece in stacking state; and the first priority, the second priority and the third priority are sequentially decreased. A person of ordinary skill in the art would have had the technological capabilities required to have modified the method taught in Ito in view of Hayashi with a first stacking parameter, and a second stacking parameter, wherein the first stacking parameter is a placing state of a workpiece S4: comparing the contour feature data of the workpieces; S32: analyzing a stacking degree according to the contour curve, and calculating a stacking index; wherein S32 comprises: A: determining workpiece edges of each of the workpieces according to the contour curve; B: determining a placing state of the workpiece as a first stacking parameter according to the depth information and the workpiece edges; wherein, the placing state comprises a natural putting state or a stacking state; wherein S4 comprises: S41: sequentially determining a first priority, a second priority and a third priority of each of the workpieces in the grabbing space according to the depth information, the first stacking parameter and the second stacking parameter; wherein in S41: the deeper the workpiece is, the lower the priority is; the workpiece in the natural putting state has a higher priority than the workpiece in stacking state; and the first priority, the second priority and the third priority are sequentially decreased taught in Duan. Furthermore, the method taught in Ito in view of Hayashi is already configured to determine the contour information of each object, the depth information for each object, and a grasping order for the objects. As such, the method taught in Ito in view of Hayashi could be easily modified to use the contour information and the stacking parameters when determining the grasping sequence using the methods taught in Duan. Such a modification would not have changed or introduced new functionality. No inventive effort would have been required. The combination would have yielded the predictable result of a robot sorting method based on visual recognition, comprising: a first stacking parameter, and a second stacking parameter, wherein the first stacking parameter is a placing state of a workpiece S4: comparing the contour feature data of the workpieces; S32: analyzing a stacking degree according to the contour curve, and calculating a stacking index; wherein S32 comprises: A: determining workpiece edges of each of the workpieces according to the contour curve; B: determining a placing state of the workpiece as a first stacking parameter according to the depth information and the workpiece edges; wherein, the placing state comprises a natural putting state or a stacking state; wherein S4 comprises: S41: sequentially determining a first priority, a second priority and a third priority of each of the workpieces in the grabbing space according to the depth information, the first stacking parameter and the second stacking parameter; wherein in S41: the deeper the workpiece is, the lower the priority is; the workpiece in the natural putting state has a higher priority than the workpiece in stacking state; and the first priority, the second priority and the third priority are sequentially decreased.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filling date of the claimed invention, to have combine the robot sorting method taught in Ito in view of Hayashi with a first stacking parameter, and a second stacking parameter, wherein the first stacking parameter is a placing state of a workpiece S4: comparing the contour feature data of the workpieces; S32: analyzing a stacking degree according to the contour curve, and calculating a stacking index; wherein S32 comprises: A: determining workpiece edges of each of the workpieces according to the contour curve; B: determining a placing state of the workpiece as a first stacking parameter according to the depth information and the workpiece edges; wherein, the placing state comprises a natural putting state or a stacking state; wherein S4 comprises: S41: sequentially determining a first priority, a second priority and a third priority of each of the workpieces in the grabbing space according to the depth information, the first stacking parameter and the second stacking parameter; wherein in S41: the deeper the workpiece is, the lower the priority is; the workpiece in the natural putting state has a higher priority than the workpiece in stacking state; and the first priority, the second priority and the third priority are sequentially decreased taught in Duan with a reasonable expectation of success. One of ordinary skill in the art would have been motivated to make this modification because the combination would have yielded predictable results.
Ito in view of Hayashi in further view of Duan does not teach and a second stacking parameter, and the second stacking parameter is the smallest one of the distances between a geometric center of one of the workpieces and each of workpiece edges of the workpieces;
and calculating a geometric center of the workpiece; and
C: calculating distances between the geometric center and the workpiece edges, selecting a smallest one of the distances as a second stacking parameter, and determining the stacking index of the workpiece in combination with the first stacking parameter;
and the greater a numerical value of the second stacking parameter is, the higher the priority is;
Zhu, in the same field of endeavor, teaches and a second stacking parameter, and the second stacking parameter is the smallest one of the distances between a geometric center of one of the workpieces and each of workpiece edges of the workpieces (Zhu: ¶ 0010, “Determine the geometric center of the object based on the separation position;”, ¶ 0014, “The preset size includes half, the first endpoint is the point on the edge of the object that is closest to the geometric center at the separation position, and the second endpoint is located on the edge of the object and on the extension line of the line connecting the first endpoint and the geometric center.”, ¶ 0025, “Furthermore, the step of determining the geometric center of the object based on the separation position includes the following steps:”, ¶ 0026, “Using the endpoints of the two ends of the third pixel line segment as the initial point and the termination point respectively, query the pixel grayscale value of the next pixel point in the direction from the initial point to the termination point.”, ¶ 0027 “If the pixel grayscale value is 1, record it as the first point. Continue the query from the first point to the end point until the second point is obtained and recorded.”, ¶ 0028, “If the pixel grayscale value is 0, continue searching towards the end point until a pixel with a grayscale value of 1 is found, and record it as the first point. Starting from the first point, continue searching towards the end point until the second point is found and recorded.”, ¶ 0029, “Based on the first point and the second point, the midpoint between the first point and the second point is determined as the geometric center;” ¶ 0031, “Furthermore, the step of determining the first endpoint and the second endpoint based on the geometric center and the separation position includes the following steps:”, ¶ 0032, “Select one segment of the third pixel line segment that is divided by the geometric center as the first line segment;”, ¶ 0033, “The first line segment is rotated several times along the geometric center at a first preset angle until the first line segment returns to its initial position, and the length of the second line segment is calculated during each first rotation, wherein the length of the second line segment is the length between the object edge on the first line segment and the geometric center;”, ¶ 0034, “Determine two adjacent second line segments whose sum of lengths is the smallest, and rotate one of the second line segments several times along the geometric center toward the other second line segment at a second preset angle, and calculate the length of the third line segment during each second rotation, where the length of the third line segment is the length between the object edge on the second line segment and the geometric center.”, ¶ 0035, “Take the endpoint of the third line segment with the shortest length as the first endpoint, and determine the second endpoint;”. The cited passages clearly shows that the geometric center of the object is determined, a distance from the center to the edges of the object is determined, and then the smallest of these distances is selected. Additionally this process is used to control the robot when grasping an object.);
and calculating a geometric center of the workpiece (Zhu: ¶ 0010, “Determine the geometric center of the object based on the separation position;”, ¶ 0014, “The preset size includes half, the first endpoint is the point on the edge of the object that is closest to the geometric center at the separation position, and the second endpoint is located on the edge of the object and on the extension line of the line connecting the first endpoint and the geometric center.”, ¶ 0025, “Furthermore, the step of determining the geometric center of the object based on the separation position includes the following steps:”, ¶ 0026, “Using the endpoints of the two ends of the third pixel line segment as the initial point and the termination point respectively, query the pixel grayscale value of the next pixel point in the direction from the initial point to the termination point.”, ¶ 0027 “If the pixel grayscale value is 1, record it as the first point. Continue the query from the first point to the end point until the second point is obtained and recorded.”, ¶ 0028, “If the pixel grayscale value is 0, continue searching towards the end point until a pixel with a grayscale value of 1 is found, and record it as the first point. Starting from the first point, continue searching towards the end point until the second point is found and recorded.”, ¶ 0029, “Based on the first point and the second point, the midpoint between the first point and the second point is determined as the geometric center;”. The system is clearly configured to determine the geometric center of the object.); and
C: calculating distances between the geometric center and the workpiece edges, selecting a smallest one of the distances as a second stacking parameter, and determining the stacking index of the workpiece in combination with the first stacking parameter Zhu: ¶ 0035, “Take the endpoint of the third line segment with the shortest length as the first endpoint, and determine the second endpoint;”. The cited passage shows that the smallest distance is used as a control parameter.);
and the greater a numerical value of the second stacking parameter is, the higher the priority is (Zhu: ¶ 0031, “Furthermore, the step of determining the first endpoint and the second endpoint based on the geometric center and the separation position includes the following steps:”, ¶ 0032, “Select one segment of the third pixel line segment that is divided by the geometric center as the first line segment;”, ¶ 0033, “The first line segment is rotated several times along the geometric center at a first preset angle until the first line segment returns to its initial position, and the length of the second line segment is calculated during each first rotation, wherein the length of the second line segment is the length between the object edge on the first line segment and the geometric center;”, ¶ 0034, “Determine two adjacent second line segments whose sum of lengths is the smallest, and rotate one of the second line segments several times along the geometric center toward the other second line segment at a second preset angle, and calculate the length of the third line segment during each second rotation, where the length of the third line segment is the length between the object edge on the second line segment and the geometric center.”, ¶ 0035, “Take the endpoint of the third line segment with the shortest length as the first endpoint, and determine the second endpoint;”).
Ito in view of Hayashi in further view of Duan teaches a system and method for a robotic sorting image. The method is configured to capture a plurality of grating images of a plurality of workpieces and performing recognition and fusion to obtain point cloud set information, acquire a posture matrix of each of workpieces in a grabbing space from the point cloud set information, acquire contour feature data of each of the workpieces according to the posture matrix, the contour feature data includes depth information, a first stacking parameter, and a second stacking parameter, wherein the first stacking parameter is a placing state of a workpiece, compare the contour feature data of the workpieces and determining a target grabbing sequence to grab the workpieces. The combination of Ito in view of Hayashi in further view of Duan is configured to determine the priority to grasp each object based on depth information, contour information, and the stacking state of each object. Additionally, The combination of Ito in view of Hayashi in further view of Duan determines a plurality of geometric information for each object. Zhu teaches a system and method for robotic grasping tasks. The method is configured to, using an image, determine the geometric center of an object and the smallest distance from the geometric center to an edge of the object. The method then uses this information in order to grasp the object. As the combination of Ito in view of Hayashi in further view of Duan teaches using various geometric parameters in addition to the stacking state and depth information to determine the grasping priorities, a person of ordinary skill in the art would have been able to combine the method taught Ito in view of Hayashi in further view of Duan with the determination of the geometric center and the smallest distance as taught in Zhu such that these parameters taught in Zhu are used in the setting of the grasping privities. Therefore, the combination of Ito in view of Hayashi in further view of Duan in further view of Zhu teaches the limitations “C: calculating distances between the geometric center and the workpiece edges, selecting a smallest one of the distances as a second stacking parameter, and determining the stacking index of the workpiece in combination with the first stacking parameter” and “the greater a numerical value of the second stacking parameter is, the higher the priority is”.
Ito in view of Hayashi in further view of Duan teaches a robot sorting method based on visual recognition, comprising: S1: acquiring a plurality of grating images in a visual field range, and performing recognition and fusion to obtain point cloud set information; S2: acquiring a posture matrix of each of workpieces in a grabbing space from the point cloud set information; S3: acquiring contour feature data of each of the workpieces according to the posture matrix, the contour feature data includes depth information, a first stacking parameter, and a second stacking parameter, wherein the first stacking parameter is a placing state of a workpiece S4: comparing the contour feature data of the workpieces and determining a target grabbing sequence to grab the workpieces. Ito in view of Hayashi in further view of Duan does not teach a second stacking parameter, and the second stacking parameter is the smallest one of the distances between a geometric center of one of the workpieces and each of workpiece edges of the workpieces; and calculating a geometric center of the workpiece; and C: calculating distances between the geometric center and the workpiece edges, selecting a smallest one of the distances as a second stacking parameter, and determining the stacking index of the workpiece in combination with the first stacking parameter; and the greater a numerical value of the second stacking parameter is, the higher the priority is. Zhu teaches a second stacking parameter, and the second stacking parameter is the smallest one of the distances between a geometric center of one of the workpieces and each of workpiece edges of the workpieces; and calculating a geometric center of the workpiece; and C: calculating distances between the geometric center and the workpiece edges, selecting a smallest one of the distances as a second stacking parameter, and determining the stacking index of the workpiece in combination with the first stacking parameter; and the greater a numerical value of the second stacking parameter is, the higher the priority is. A person of ordinary skill in the art would have had the technological capabilities required to have modified the method taught in Ito in view of Hayashi in further view of Duan with a second stacking parameter, and the second stacking parameter is the smallest one of the distances between a geometric center of one of the workpieces and each of workpiece edges of the workpieces; and calculating a geometric center of the workpiece; and C: calculating distances between the geometric center and the workpiece edges, selecting a smallest one of the distances as a second stacking parameter, and determining the stacking index of the workpiece in combination with the first stacking parameter; and the greater a numerical value of the second stacking parameter is, the higher the priority is taught in Zhu. Furthermore, the method taught in Ito in view of Hayashi in further view of Duan teaches determining the shape and contours of each object, the pose of each object, and several metric used to determine the stacking order. As such, the method taught in Ito in view of Hayashi in further view of Duan teaches all of the components necessary to determine the geometric center and smallest distance to the edge as taught in Zhu. Additionally, the method taught in Ito in view of Hayashi in further view of Duan teaches multiple metrics used to determine the grasping order, including various area based metrics. As such the method can be easily modified to include a metric based on the smallest distance from the center as taught in Zhu. Such modifications would not change or introduce new functionality. No inventive effort would have been required. The combination would have yielded the predictable result of sorting method based on visual recognition, comprising: a second stacking parameter, and the second stacking parameter is the smallest one of the distances between a geometric center of one of the workpieces and each of workpiece edges of the workpieces.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filling date of the claimed invention, to have combine the robot sorting method taught in Ito in view of Hayashi in further view of Duan with a second stacking parameter, and the second stacking parameter is the smallest one of the distances between a geometric center of one of the workpieces and each of workpiece edges of the workpieces taught in Zhu with a reasonable expectation of success. One of ordinary skill in the art would have been motivated to make this modification because the combination would have yielded predictable results.
Regarding claim 6, Ito in view of Hayashi in further view of Duan in further view of Zhu teaches wherein S1 comprises: S11: projecting multiple sets of gratings with different phases to an area to be sorted in the visual field range (Hayashi: Column 5 lines 26-41, “In the second point cloud generation process, first, a second imaging process of imaging a plurality of phase shifting images PSM1 to PSM4 containing the parts PP with a plurality of phase shifting patterns PH1 to PH4 selectively projected one by one as the projection patterns is performed using the projection device 810. The plurality of phase shifting patterns PH1 to PH4 are sinusoidal banded patterns having dark and light parts. In the phase shifting method, generally, capturing of images is performed using the n phase shift patterns PH2 to PHn for n (integer) equal to or larger than three. The n phase shift patterns PH1 to PHn are sinusoidal patterns having phase sequentially shifted by 2π/n. In the example of FIG. 7, n=4. Note that, in the phase shifting method, it is not necessary to use the stereo camera, and only one of the two cameras forming the stereo camera may be used for imaging.”, Column 5 lines 42-52, “In the second point cloud generation process, then, a second analysis process is executed using the plurality of phase shifting images PSM1 to PSM4, and thereby, a second point cloud PG2 is generated. In the second analysis process, first, a distance image DMZ is generated according to the phase shifting method using the plurality of phase shifting images PSM1 to PSM4. The generation process of the distance image DMZ using the phase shifting method is well known, and the explanation thereof is omitted here. In the second point cloud generation process, a second point cloud PG2 is further generated using the distance image DMZ.”),
and acquiring corresponding grating images (Hayashi: Column 5 lines 26-41, Column 5 lines 42-52); and
S12: identifying the grating images ((Hayashi: Column 5 lines 26-41, Column 5 lines 42-52)),
calculating coordinates of each feature point in the grating images by a triangulation location method, and integrating the coordinates to obtain the point cloud set information (Hayashi: Column 5 lines 26-41, Column 5 lines 42-52. The cited passage clearly states that the point cloud is generated based on the multiple grating images generated by projecting a pattern at different phases according to methods known in the art. One of ordinary skill in the art would have recognized that feature based methods of image processing are commonly used to reduce computational loads and had the capability to perform such a method. Additionally feature based imaging techniques are known in the art and would be considered part of the known methods referred to in Hayashi.).
Regarding claim 8, Ito in view of Hayashi in further view of Duan in further view of Zhu teaches wherein in S43, the grabbing the workpieces according to the target grabbing sequence, comprises: determining the posture matrix of a current target grabbing workpiece according to the target grabbing sequence, (Ito: Column 3 lines 22-46, “In accordance with a command from a processing apparatus 200, the measurement controller 110 for controlling the three-dimensional measurement apparatus 100 causes the light projector 120 to project pattern light on a pallet 400, and also causes the image capture device 130 to capture an area in the pallet 400 (container). The processing apparatus 200 is, for example, a computer apparatus. Note that the processing apparatus 200 may also function as the measurement controller 110. In the pallet 400, a number of works as picking targets of the picking system and as measurement targets of the three-dimensional measurement apparatus 100 are piled. The measurement controller 110 transmits the image capturing result of the image capture device 130 to the processing apparatus 200. The processing apparatus 200 recognizes the works in the pallet 400 and obtains position and orientation information based on data of an image captured by the image capture device 130, and selects a pickable candidate work.”, Column 6 lines 18-47, “The three-dimensional measurement apparatus 100 transfers the obtained captured data to the processing apparatus 200, as needed. The processing apparatus 200 performs the three-dimensional measurement calculation of the works based on the captured data, the by obtaining the positions and orientations of the works. The processing apparatus 200 transfers the information to the robot 300, and the robot 300 performs a picking operation based on the information.”)
converting the posture matrix into target grabbing coordinates in a manipulator coordinate system in combination with a hand-eye calibration matrix (Zhu: ¶ 0112, “Step S61: Based on the grasping pose, obtain the corresponding grasping pose parameters in the camera coordinate system of the camera; in this embodiment, the camera can be a camera that captures preset images, such as an RGBD camera.”, ¶ 0114, “Step S62: Obtain the preset rotation and translation matrix between the camera coordinate system and the robot arm coordinate system;”, ¶ 0115, “Specifically, the camera coordinate system and the robot arm coordinate system can be calibrated by hand and eye using the nine-point calibration method to obtain and save the preset rotation and translation matrix;”, ¶ 0116, “Step S63: Based on the grasping pose parameters and the preset rotation and translation matrix, the robot arm performs a grasping operation on the object.”),
and sending the target grabbing coordinates to a manipulator (Ito: Column 6 lines 18-47, “The three-dimensional measurement apparatus 100 transfers the obtained captured data to the processing apparatus 200, as needed. The processing apparatus 200 performs the three-dimensional measurement calculation of the works based on the captured data, the by obtaining the positions and orientations of the works. The processing apparatus 200 transfers the information to the robot 300, and the robot 300 performs a picking operation based on the information.”, Zhu: ¶ 0116, “Step S63: Based on the grasping pose parameters and the preset rotation and translation matrix, the robot arm performs a grasping operation on the object.”);
and grabbing the target grabbing workpiece according to the target grabbing coordinates by the manipulator (Ito: Column 6 lines 18-47, “The three-dimensional measurement apparatus 100 transfers the obtained captured data to the processing apparatus 200, as needed. The processing apparatus 200 performs the three-dimensional measurement calculation of the works based on the captured data, the by obtaining the positions and orientations of the works. The processing apparatus 200 transfers the information to the robot 300, and the robot 300 performs a picking operation based on the information.”, Column 7 line 64 – Colum 8 line 9, “The result of priority setting based on the distances is used to determine, if there exist a plurality of candidate works in one block, the picking sequence of the plurality of candidate works. Priorities among a plurality of candidate works of different blocks comply with priorities assigned to the blocks in which the respective candidate works are positioned, as in the example of the first embodiment. Consequently, in the example of FIG. 7, the picking sequence of the respective candidate works is set to "first candidate work 411[Wingdings font/0xE0]second candidate work 412[Wingdings font/0xE0]third candidate work 413[Wingdings font/0xE0]sixth candidate work 416[Wingdings font/0xE0]fifth candidate work 415[Wingdings font/0xE0]fourth candidate work 414[Wingdings font/0xE0]seventh candidate work 417".”).
Regarding claim 10, Ito in view of Hayashi in further view of Duan in further view of Zhu teaches a non-transitory computer-readable storage medium storing a computer program thereon, wherein the computer program is used to realize the robot sorting method based on visual recognition according to claim 1 (Ito: Column 8 line 42 – Column 9 line 3, “Embodiment(s) of the present invention can also be realized by a computer of a system or apparatus that reads out and executes computer executable instructions (e.g., one or more programs) recorded on a storage medium (which may also be referred to more fully as a 'non-transitory computer-readable storage medium') to perform the functions of one or more of the above-described embodiment(s) and/or that includes one or more circuits (e.g., application specific integrated circuit (ASIC)) for performing the functions of one or more of the above-described embodiment(s), and by a method performed by the computer of the system or apparatus by, for example, reading out and executing the computer executable instructions from the storage medium to perform the functions of one or more of the above described embodiment(s) and/or controlling the one or more circuits to perform the functions of one or more of the above-described embodiment(s). The computer may comprise one or more processors (e.g., central processing unit (CPU), micro processing unit (MPU)) and may include a network of separate computers or separate processors to read out and execute the computer executable instructions. The computer executable instructions may be provided to the computer, for example, from a network or the storage medium. The storage medium may include, for example, one or more of a hard disk, a random-access memory (RAM), a read only memory (ROM), a storage of distributed computing systems, an optical disk (such as a compact disc (CD), digital versatile disc (DVD), or Blu-ray Disc (BD)™), a flash memory device, a memory card, and the like.”).
Claim(s) 7 is/are rejected under 35 U.S.C. 103 as being unpatentable over US 10040199 B2 ("Ito") in view of US 11126844 B2 ("Hayashi") in further view of CN 112802105 A ("Duan") in further view of CN 111462232 A ("Zhu") in further view of US 2020/0194117 A1 ("Krieger").
Regarding claim 7, Ito in view of Hayashi in further view of Duan in further view of Zhu teaches wherein S2 comprises: S21: setting the grabbing space according to a shape and a structure of an area to be sorted (Ito: Column 4 line 58 – Column 5 line 10, “The procedure of assigning the priorities of the blocks, which has been explained, will be described with reference to a flowchart shown in FIG. 4. The priorities of the blocks may be assigned using an apparatus outside the picking system. In this embodiment, assume that the processing apparatus 200 performs an operation of assigning the priorities of the blocks. When the operation of assigning the example of FIG. 2, the area in the pallet 400 is equally divided into 12 rectangular blocks in the vertical and horizontal directions. In step S12, the processing apparatus 200 obtains the orientation of the robot 300, when viewed from the three-dimensional measurement apparatus 100, at the time of picking of a work in each block. The orientation of the robot 300 may be obtained by actual image capturing by the three-dimensional measurement apparatus 100, or by simulation using a three-dimensional CAD model or the like.”, Column 5 lines 22-35, “In step S13, based on the orientation of the robot 300 obtained in step S12, the processing apparatus 200 assigns the priorities of all the blocks by setting a higher priority to a block which prevents the robot from shielding the works in the remaining blocks as much as possible. At this time, criteria for assigning the priorities of the blocks may include setting criteria such as the distance to a conveyance destination of a work in addition to a criterion that the works in the remaining blocks are not shielded as much as possible. As shown in FIG. by assigning the priorities of the blocks 4, before measurement, for example, at the time of installation of the three-dimensional measurement apparatus, it is possible to quickly pick a candidate work while preventing the robot from shielding the remaining candidate works as much as possible at the time of picking.”. The cited passages clearly shows that the system is configured to set the grabbing space according to the area of the pallet.);
S22: screening the point cloud set information according to coordinate information of the grabbing space to obtain a target point cloud set (Ito: Column 4 line 58 – Column 5 line 10, “The procedure of assigning the priorities of the blocks, which has been explained, will be described with reference to a flowchart shown in FIG. 4. The priorities of the blocks may be assigned using an apparatus outside the picking system. In this embodiment, assume that the processing apparatus 200 performs an operation of assigning the priorities of the blocks. When the operation of assigning the example of FIG. 2, the area in the pallet 400 is equally divided into 12 rectangular blocks in the vertical and horizontal directions. In step S12, the processing apparatus 200 obtains the orientation of the robot 300, when viewed from the three-dimensional measurement apparatus 100, at the time of picking of a work in each block. The orientation of the robot 300 may be obtained by actual image capturing by the three-dimensional measurement apparatus 100, or by simulation using a three-dimensional CAD model or the like.”. Hayashi: Column 5 lines 26-41, “In the second point cloud generation process, first, a second imaging process of imaging a plurality of phase shifting images PSM1 to PSM4 containing the parts PP with a plurality of phase shifting patterns PH1 to PH4 selectively projected one by one as the projection patterns is performed using the projection device 810. The plurality of phase shifting patterns PH1 to PH4 are sinusoidal banded patterns having dark and light parts. In the phase shifting method, generally, capturing of images is performed using the n phase shift patterns PH2 to PHn for n (integer) equal to or larger than three. The n phase shift patterns PH1 to PHn are sinusoidal patterns having phase sequentially shifted by 2π/n. In the example of FIG. 7, n=4. Note that, in the phase shifting method, it is not necessary to use the stereo camera, and only one of the two cameras forming the stereo camera may be used for imaging.”, Column 5 lines 42-52, “In the second point cloud generation process, then, a second analysis process is executed using the plurality of phase shifting images PSM1 to PSM4, and thereby, a second point cloud PG2 is generated. In the second analysis process, first, a distance image DMZ is generated according to the phase shifting method using the plurality of phase shifting images PSM1 to PSM4. The generation process of the distance image DMZ using the phase shifting method is well known, and the explanation thereof is omitted here. In the second point cloud generation process, a second point cloud PG2 is further generated using the distance image DMZ.”. The combination of Ito iv view of Hayashi clearly shows that the point cloud is screened to determine a target point cloud.); and
determining each visible workpiece in the grabbing space (Duan: ¶ 0086, “In another alternative implementation, based on the exposure ratio of each item object, the area corresponding to the item object with an exposure ratio greater than a preset threshold is marked as a crawlable area, and the area corresponding to the item object with an exposure ratio not greater than the preset threshold is marked as an uncrawlable area. In some scenarios, the stacking relationship between items is not easy to determine, and items on the same layer may overlap each other. In such cases, it is difficult to accurately label the top item. For example, in supermarket product picking scenarios, the hierarchical relationship between products is not clear, and there may be overlapping between items on the same layer. Therefore, the requirements for the picking order are not strict, but the distinction between pickable and non-pickable items is more stringent. In this case, items with little surface exposure, or items that may cause other items in the scene to fly out after being grabbed, should not be marked as grabbable items. Accordingly, in the above scenario, a labeling threshold can be set, such as 85%. If an item's exposed surface area is greater than 85%, it is marked as a graspable item; if an item's exposed surface area is less than 85%, it is marked as a non-grabable item. Of course, the exposure ratio of an item can be quantified not only by the exposed surface area but also by the exposed volume. This invention does not limit the specific details.”. The method is clearly configured to determine what objects are visible.),
and calculating the posture matrix of each visible workpiece in a camera coordinate system (Ito: Column 3 lines 22-46, “In accordance with a command from a processing apparatus 200, the measurement controller 110 for controlling the three-dimensional measurement apparatus 100 causes the light projector 120 to project pattern light on a pallet 400, and also causes the image capture device 130 to capture an area in the pallet 400 (container). The processing apparatus 200 is, for example, a computer apparatus. Note that the processing apparatus 200 may also function as the measurement controller 110. In the pallet 400, a number of works as picking targets of the picking system and as measurement targets of the three-dimensional measurement apparatus 100 are piled. The measurement controller 110 transmits the image capturing result of the image capture device 130 to the processing apparatus 200. The processing apparatus 200 recognizes the works in the pallet 400 and obtains position and orientation information based on data of an image captured by the image capture device 130, and selects a pickable candidate work.”, Column 6 lines 18-47, “The three-dimensional measurement apparatus 100 transfers the obtained captured data to the processing apparatus 200, as needed. The processing apparatus 200 performs the three-dimensional measurement calculation of the works based on the captured data, the by obtaining the positions and orientations of the works.”).
Ito in view of Hayashi in further view of Duan in further view of Zhu does not teach S23: performing outlier filtration on the target point cloud set,
Krieger, in the same field of endeavor, teaches S23: performing outlier filtration on the target point cloud set (Krieger: ¶ 0098, “After acquiring camera images 240, the flow 234 can include generating a 3D point cloud 242. The previously acquired images from the mobile camera 116 (e.g., camera images 240) can be used to generate the 3D point cloud 242. For example, the mobile platform 110 (or the computing device 106) can perform a point cloud registration using the 3D images (depicting multiple scenes) to construct a 3D model using any suitable technique or combinations of techniques. For example, the mobile platform 110 can use the Iterative Closest Point (“ICP”) algorithm. In such an example, the ICP algorithm can be implemented using the point cloud library (“PCL”), and pointmatcher, and can include prior noise removal with a PCL passthrough filter. In some embodiments, a point cloud color segmentation can be applied to extract a point cloud corresponding to the patient from the reconstructed scene, for example by removing background items such as a table and a supporting frame.”. The cited passage clearly teaches a method of filtering noise in a point cloud.).
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filling date of the claimed invention, to have combine the robot sorting method taught in Ito in view of Hayashi in further view of Duan in further view of Zhu with S23: performing outlier filtration on the target point cloud set taught in Krieger with a reasonable expectation of success. One of ordinary skill in the art would have been motivated to make this modification because the combination would have required the simple addition of a known algorithm to a readily configurable system. The system taught in Ito in view of Hayashi in further view of Duan in further view of Zhu already teaches determining a point cloud. As such the filtering method taught in Krieger can be simply added to the system taught in Ito in view of Hayashi in further view of Duan in further view of Zhu according to methods known in the art. Such a modification would not change or introduce new functionality. No inventive effort would have been required.
Claim(s) 9 is/are rejected under 35 U.S.C. 103 as being unpatentable over US 10040199 B2 ("Ito") in view of US 11126844 B2 ("Hayashi") in further view of CN 112802105 A ("Duan") in further view of CN 111462232 A ("Zhu") in further view of US 2023/0125022 A1 ("Li").
Regarding claim 9, Ito in view of Hayashi in further view of Duan in further view of Zhu does not teach wherein the manipulator is a magnetic suction manipulator.
Li, in the same field of endeavor, teaches wherein the manipulator is a magnetic suction manipulator (Li: ¶ 0033, “The picking hand 21 may have any configuration that can hold the workpieces W one by one. As an example, the picking hand 21 may have a suction pad 211 for suctioning a workpiece W, as illustrated in FIG. 1. In this way, the picking hand 21 may be a suction hand for suctioning a workpiece using air tightness, but may be an attraction hand with a strong attraction force which does not requires air tightness. The picking hand 21 may have a pair of gripping fingers 212 or three or more gripping fingers 212 for pinching and holding a workpiece W as an alternative enclosed by the two-dot chain line in FIG. 1, or may have a plurality of suction pads 211 (not illustrated). Alternatively, the picking hand 21 may have a magnetic hand (not illustrated) configured to hold a workpiece made of iron or the like with a magnetic force.”).
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filling date of the claimed invention, to have combine the robot sorting method taught in Ito in view of Hayashi in further view of Duan in further view of Zhu in further view of Zhu with wherein the manipulator is a magnetic suction manipulator taught in Li with a reasonable expectation of success. One of ordinary skill in the art would have been motivated to make this modification because the modification would have required the simple substitution of a known manipulator for another. A person of ordinary skill in the art would have had the technological capabilities to have modified the robot taught in Ito in view of Hayashi in further view of Duan in further view of Zhu in further view of Zhu with a magnetic manipulator as taught in Li. Additionally such a manipulator would have been known to one of ordinary skill in the art. The manipulator of the robot taught in Ito in view of Hayashi in further view of Duan in further view of Zhu in further view of Zhu can be easily substituted for the magnetic manipulator taught in Li according to methods known in the art. Such a modification would not have changed or introduced new functionality. No inventive effort would have been required.
Response to Arguments
Applicant's arguments filed June 16th, 2026 have been fully considered but they are not persuasive.
Regarding Applicant’s arguments on pages 6-14, Applicant argues that the prior art on record fails to teach the limitations of the amended independent claim 1.
Specifically on Pages 6-10, Applicant argues that the primary reference Hayashi fails to teach the limitations “performing recognition and fusion to obtain point cloud set information;”, “S2: acquiring a posture matrix of each of visible workpieces in a grabbing space from the point cloud set information, wherein, the visible workpieces comprise workpieces that are stacked with each other;”, “S3: acquiring contour feature data of each of the workpieces according to the posture matrix, the contour feature data includes depth information, a first stacking parameter, and a second stacking parameter, wherein the first stacking parameter is a placing state of a workpiece and the second stacking parameter is the smallest one of the distances between a geometric center of one of the workpieces and each of workpiece edges of the workpieces;”, and “S4: comparing the contour feature data of the workpieces, determining a target grabbing sequence”.
The Examiner respectfully disagrees. In response to applicant's arguments against the references individually, one cannot show nonobviousness by attacking references individually where the rejections are based on combinations of references. See In re Keller, 642 F.2d 413, 208 USPQ 871 (CCPA 1981); In re Merck & Co., 800 F.2d 1091, 231 USPQ 375 (Fed. Cir. 1986). As was stated in the previous Final Office Action mailed April 16th, 2026 and above in the 35 U.S.C. § 103 rejection section, the primary reference Ito teaches a robot sorting method based on visual recognition, comprising (Ito: Figure 1, Abstract, Column 3 lines 22-46): S1: acquiring a plurality of grating images in a visual field range (Ito: Column 3 lines 22-46, Column 5 lines 26-49), S2: acquiring a posture matrix of each of visible workpieces in a grabbing space from the point cloud set information, wherein, the visible workpieces comprise workpieces that are stacked with each other (Ito: Figure 2, Column 3 lines 22-46, Column 3 lines 55-65, Column 6 lines 18-47); and determining a target grabbing sequence (Ito: Column 7 line 64 – Colum 8 line 9), and grabbing the workpieces according to the grabbing sequence (Ito: Column 3 lines 22-46, Column 6 lines 18-47, Column 7 line 64 – Colum 8 line 9); S42: determining a target grabbing sequence for each of the workpieces according to the first priority, the second priority and the third priority of each of the workpieces (Ito: Column 7 line 64 – Colum 8 line 9); and grabbing the workpieces according to the target grabbing sequence (Ito: Column 3 lines 22-46, Column 6 lines 18-47, Column 7 line 64 – Colum 8 line 9). Ito clearly teaches a robotic system that is configured to capture a plurality of grating images in a visual range. The system then uses these grating images to determine the position and orientation (i.e. the pose) of each object in the container using these pose matrices. Additionally, the objects in the container are clearly piled on one another, wherein the visible workpieces are those stacked in such a way as to not be fully occluded by other workpieces. The robot then determines a target grabbing sequence and controls the robot to grasp the objects according to the sequence. Furthermore, as is clear from the previous Final Office Action mailed April 16th, 2026 and above in the 35 U.S.C. § 103 rejection section, that Ito was not relied upon to teach the limitations “performing recognition and fusion to obtain point cloud set information”, “S3: acquiring contour feature data of each of the workpieces according to the posture matrix, the contour feature data includes depth information, a first stacking parameter, and a second stacking parameter, wherein the first stacking parameter is a placing state of a workpiece and the second stacking parameter is the smallest one of the distances between a geometric center of one of the workpieces and each of workpiece edges of the workpieces;”, and “S4: comparing the contour feature data of the workpieces, determining a target grabbing sequence”. The secondary reference Hayashi teaches performing recognition and fusion to obtain point cloud set information (Hayashi: Column 5 lines 26-41, Column 5 lines 42-52); S3: acquiring contour feature data of each of the workpieces according to the posture matrix, the contour feature data includes depth information (Column 5 lines 26-41, Column 5 lines 42-52, Column 6 lines 43-48, Column 6 lines 49-63); wherein S3 comprises: S31: determining a contour curve and depth information of each of the workpieces according to the posture matrix (Hayashi: Column 5 lines 26-41, Column 5 lines 42-52, Column 6 lines 43-48, Column 6 lines 49-63); S43: determining a product model according to the contour curve of the workpiece (Hayashi: Column 6 lines 43-48). The cited passages of Hayashi clearly shows that Hayashi, which relates to a robotic system configured to gasp objects, teaches generating a point cloud by capturing a plurality of grating images and performing a recognition and fusion process to generate said point cloud. The secondary reference Duan teaches a first stacking parameter (Duan: ¶ 0083, ¶ 0086), wherein the first stacking parameter is a placing state of a workpiece (Duan: ¶ 0083, ¶ 0086) and S4: comparing the contour feature data of the workpieces (Duan: ¶ 0083, ¶ 0086, ¶ 0087); S32: analyzing a stacking degree according to the contour curve, and calculating a stacking index (Duan: ¶ 0015, ¶ 0016, ¶ 0017, ¶ 0083, ¶ 0086, ¶ 0087); wherein S32 comprises: A: determining workpiece edges of each of the workpieces according to the contour curve (Duan: ¶ 0083); B: determining a placing state of the workpiece as a first stacking parameter according to the depth information and the workpiece edges (Duan: ¶ 0083, ¶ 0086), wherein, the placing state comprises a natural putting state or a stacking state wherein, the placing state comprises a natural putting state or a stacking state (Duan: ¶ 0083, ¶ 0086); wherein S4 comprises: S41: sequentially determining a first priority, a second priority and a third priority of each of the workpieces in the grabbing space according to the depth information, the first stacking parameter and the second stacking parameter (Duan: ¶ 0015, ¶ 0016, ¶ 0017, ¶ 0083, ¶ 0086, ¶ 0087); wherein in S41: the deeper the workpiece is, the lower the priority is (Duan: ¶ 0085); the workpiece in the natural putting state has a higher priority than the workpiece in stacking state (Duan: ¶ 0086); and the first priority, the second priority and the third priority are sequentially decreased (Duan: ¶ 0015, ¶ 0016, ¶ 0017, ¶ 0083, ¶ 0085, ¶ 0086, ¶ 0087). Duan teaches a system and method for grasping objects using a robot. The system is configured capture an image of a container containing a plurality of workpieces in a pile. The system is configured to determine various parameters for each workpieces. Said parameters include the depth of each workpiece, contour information of each workpiece, the stacking state of each workpiece (i.e. the degree to which the workpiece in question is exposed), amongst other geometric parameters. The system is then configured to use these parameters to determine the grasping priority for each object and control the robot to grasps the objects according to the priorities. the method taught in Ito in view of Hayashi is already configured to determine the contour information of each object, the depth information for each object, and a grasping order for the objects. As such, the method taught in Ito in view of Hayashi could be easily modified to use the contour information and the stacking parameters when determining the grasping sequence as taught in Duan. The secondary reference Zhu teaches a second stacking parameter, and the second stacking parameter is the smallest one of the distances between a geometric center of one of the workpieces and each of workpiece edges of the workpieces (Zhu: ¶ 0010, ¶ 0014, ¶ 0025, ¶ 0026, ¶ 0027, ¶ 0028, ¶ 0029, ¶ 0031, ¶ 0032, ¶ 0033, ¶ 0034, ¶ 0035); and calculating a geometric center of the workpiece (Zhu: ¶ 0010, ¶ 0014, ¶ 0025, ¶ 0026, ¶ 0027, ¶ 0028, ¶ 0029); and C: calculating distances between the geometric center and the workpiece edges, selecting a smallest one of the distances as a second stacking parameter, and determining the stacking index of the workpiece in combination with the first stacking parameter Zhu: ¶ 0035); and the greater a numerical value of the second stacking parameter is, the higher the priority is (Zhu: ¶ 0031, ¶ 0032, ¶ 0033, ¶ 0034, ¶ 0035). Zhu teaches a system and method for grasping object using a robot. The system is configured to capture an image of the object to be grasped, determine the geometric center form the image, determine the shortest length to an edge of the object from the geometric center, and use this information to grasp the object. The method taught in Ito in view of Hayashi in further view of Duan teaches determining the shape and contours of each object, the pose of each object, and several metric used to determine the stacking order. Additionally, the method taught in Ito in view of Hayashi in further view of Duan teaches multiple metrics used to determine the grasping order, including various area based metrics. As such the method can be easily modified to include a metric based on the smallest distance from the center as taught in Zhu. Additionally, Applicant argues that that Ito does not teach the above limitations because Ito determines a priority for workpieces that are completely unshielded by stacking as and performs picking according to the priorities, whereas the present invention determines a priority for all workpieces visible in the image (including workpieces that are partially shielded due to stacking). In response to applicant's argument that the references fail to show certain features of the invention, it is noted that the features upon which applicant relies (i.e., that the present invention determines a priority for all workpieces visible in the image (including workpieces that are partially shielded due to stacking) are not recited in the rejected claim(s). The claims only recite that a posture matrix is determined for all visible workpieces (i.e. the limitation “S2: acquiring a posture matrix of each of visible workpieces in a grabbing space from the point cloud set information, wherein, the visible workpieces comprise workpieces that are stacked with each other”) and does not recite a limitation specifying that the priority is determined for all visible workpieces. Furthermore, even if the limitations could only be interpreted as determining the priority for all visible workpieces under the broadest reasonable interpretation, the secondary reference Duan teaches determining a grasping priority for all visible workpieces whose exposure ration is above a threshold. As is clearly stated in ¶ 0086 of Duan, the system is configured to determine if the exposure ration of the visible workpiece is above a threshold, and if so, determine a grasping priority based on the exposure ratio and other metrics. Additionally, the cited passage of Duan specifies that the exposure ration threshold can be set, and the 85% is a non-limiting example. One of ordinary skill in the art would recognize that, even if the exposure ratio threshold could only be 85% under the broadest reasonable interpretation, any object having an exposure ratio of less than 100% would be at least partially occluded. As the method is configured to set a priority for the objects whose exposure ratio is greater than or equal to the threshold, Duan clearly can set a grasping priority for objects that are at least partially occluded. Therefore, the combination of Ito in view of Hayashi in further view of Duan in further view of Zhu teaches the above limitations.
Regarding Applicant’s arguments on Pages 10-11, Applicant argues that the secondary reference Hayashi does not teach the limitations “S2: acquiring a posture matrix of each of visible workpieces in a grabbing space from the point cloud set information, wherein, the visible workpieces comprise workpieces that are stacked with each other;”, “S3: acquiring contour feature data of each of the workpieces according to the posture matrix, the contour feature data includes depth information, a first stacking parameter, and a second stacking parameter, wherein the first stacking parameter is a placing state of a workpiece and the second stacking parameter is the smallest one of the distances between a geometric center of one of the workpieces and each of workpiece edges of the workpieces;”, and “S4: comparing the contour feature data of the workpieces, determining a target grabbing sequence”.
The Examiner respectfully disagrees. In response to applicant's arguments against the references individually, one cannot show nonobviousness by attacking references individually where the rejections are based on combinations of references. See In re Keller, 642 F.2d 413, 208 USPQ 871 (CCPA 1981); In re Merck & Co., 800 F.2d 1091, 231 USPQ 375 (Fed. Cir. 1986). As was stated in the previous Final Office Action mailed April 16th, 2026 and above in the 35 U.S.C. § 103 rejection section, Hayashi was not relied upon to teach the limitations “S2: acquiring a posture matrix of each of workpieces in a grabbing space from the point cloud set information”, “"S3: acquiring contour feature data of each of the workpieces according to the posture matrix, the contour feature data includes depth information, a first stacking parameter, wherein the first stacking parameter is a placing state of a workpiece”, and “S4: comparing the contour feature data of the workpieces, determining a target grabbing sequence, and grabbing the workpieces according to the target grabbing sequence”. The primary reference Ito teaches a robot sorting method, wherein the robot is configured to capture a plurality of grating image of a container containing a plurality of objects. The robot the determines the pose of each object based on the grating images. Using these poses, the robot then determines a grabbing sequence of the objects and grabs the object according to the sequence. The secondary reference Hayashi teaches performing recognition and fusion to obtain point cloud set information (Hayashi: Column 5 lines 26-41, Column 5 lines 42-52); S3: acquiring contour feature data of each of the workpieces according to the posture matrix, the contour feature data includes depth information (Column 5 lines 26-41, Column 5 lines 42-52, Column 6 lines 43-48, Column 6 lines 49-63); wherein S3 comprises: S31: determining a contour curve and depth information of each of the workpieces according to the posture matrix (Hayashi: Column 5 lines 26-41, Column 5 lines 42-52, Column 6 lines 43-48, Column 6 lines 49-63); S43: determining a product model according to the contour curve of the workpiece (Hayashi: Column 6 lines 43-48). The cited passages of Hayashi clearly shows that Hayashi, which relates to a robotic system configured to gasp objects, teaches generating a point cloud by capturing a plurality of grating images and performing a recognition and fusion process to generate said point cloud. The secondary reference Duan teaches a system and method for grasping objects using a robot. The system is configured capture an image of a container containing a plurality of workpieces in a pile. The system is configured to determine various parameters for each workpieces. Said parameters include the depth of each workpiece, contour information of each workpiece, the stacking state of each workpiece (i.e. the degree to which the workpiece in question is exposed), amongst other geometric parameters. The system is then configured to use these parameters to determine the grasping priority for each object and control the robot to grasps the objects according to the priorities. The secondary reference Zhu teaches a system and method for grasping object using a robot. The system is configured to capture an image of the object to be grasped, determine the geometric center form the image, determine the shortest length to an edge of the object from the geometric center, and use this information to grasp the object. Therefore, because Hayashi was not relied upon to teach the above limitations and said limitations are taught by the other cited references in the combination, the combination of Ito in view of Hayashi in further view of Duan in further view of Zhu teaches the limitations “S2: acquiring a posture matrix of each of visible workpieces in a grabbing space from the point cloud set information, wherein, the visible workpieces comprise workpieces that are stacked with each other;”, “S3: acquiring contour feature data of each of the workpieces according to the posture matrix, the contour feature data includes depth information, a first stacking parameter, and a second stacking parameter, wherein the first stacking parameter is a placing state of a workpiece and the second stacking parameter is the smallest one of the distances between a geometric center of one of the workpieces and each of workpiece edges of the workpieces;”, and “S4: comparing the contour feature data of the workpieces, determining a target grabbing sequence”.
Specifically on Pages 11-12, Applicant argues that the secondary reference Duan does not teach the limitations “S3: acquiring contour feature data of each of the workpieces according to the posture matrix, the contour feature data includes depth information, a first stacking parameter, and a second stacking parameter, wherein the first stacking parameter is a placing state of a workpiece and the second stacking parameter is the smallest one of the distances between a geometric center of one of the workpieces and each of workpiece edges of the workpieces;” and “S4: comparing the contour feature data of the workpieces, determining a target grabbing sequence” because Duan does not set a priority for all visible workpieces, including those that are partially occluded, and only those that are fully visible.
The Examiner respectfully disagrees. The secondary reference Duan teaches determining a grasping priority for all visible workpieces whose exposure ration is above a threshold. As is clearly stated in ¶ 0086 of Duan, the system is configured to determine if the exposure ration of the visible workpiece is above a threshold, and if so, determine a grasping priority based on the exposure ratio and other metrics. Additionally, the cited passage of Duan specifies that the exposure ration threshold can be set, and the 85% is a non-limiting example. One of ordinary skill in the art would recognize that, even if the exposure ratio threshold could only be 85% under the broadest reasonable interpretation, any object having an exposure ratio of less than 100% would be at least partially occluded. As the method is configured to set a priority for the objects whose exposure ratio is greater than or equal to the threshold, Duan clearly can set a grasping priority for objects that are at least partially occluded. Therefore, the combination of Ito in view of Hayashi in further view of Duan in further view of Zhu teaches the above limitations.
Specifically on Page 12, Applicant argues that the secondary reference Zhu fails to teach the limitation “S3: acquiring contour feature data of each of the workpieces according to the posture matrix, the contour feature data includes depth information, a first stacking parameter, and a second stacking parameter, wherein the first stacking parameter is a placing state of a workpiece and the second stacking parameter is the smallest one of the distances between a geometric center of one of the workpieces and each of workpiece edges of the workpieces;”.
The Examiner respectfully disagrees. In response to applicant's arguments against the references individually, one cannot show nonobviousness by attacking references individually where the rejections are based on combinations of references. See In re Keller, 642 F.2d 413, 208 USPQ 871 (CCPA 1981); In re Merck & Co., 800 F.2d 1091, 231 USPQ 375 (Fed. Cir. 1986). As was stated in the previous Final Office Action mailed April 16th, 2026 and above in the 35 U.S.C. § 103 rejection section, Hayashi was not relied upon to teach the limitations “S3: acquiring contour feature data of each of the workpieces according to the posture matrix, the contour feature data includes depth information, a first stacking parameter, wherein the first stacking parameter is a placing state of a workpiece”. The primary reference Ito teaches a robot sorting method, wherein the robot is configured to capture a plurality of grating image of a container containing a plurality of objects. The robot the determines the pose of each object based on the grating images. Using these poses, the robot then determines a grabbing sequence of the objects and grabs the object according to the sequence. Hayashi teaches a system and method for a robotic sorting task. The method is configured to generate a point cloud by capturing a plurality of grating images and performing a recognition and fusion process to generate said point cloud. The method then causes the robot to grasp the object based on this information. The secondary reference Duan teaches a system and method for grasping objects using a robot. The system is configured capture an image of a container containing a plurality of workpieces in a pile. The system is configured to determine various parameters for each workpieces. Said parameters include the depth of each workpiece, contour information of each workpiece, the stacking state of each workpiece (i.e. the degree to which the workpiece in question is exposed), amongst other geometric parameters. The system is then configured to use these parameters to determine the grasping priority for each object and control the robot to grasps the objects according to the priorities. The secondary reference Zhu teaches a second stacking parameter, and the second stacking parameter is the smallest one of the distances between a geometric center of one of the workpieces and each of workpiece edges of the workpieces (Zhu: ¶ 0010, ¶ 0014, ¶ 0025, ¶ 0026, ¶ 0027, ¶ 0028, ¶ 0029, ¶ 0031, ¶ 0032, ¶ 0033, ¶ 0034, ¶ 0035); and calculating a geometric center of the workpiece (Zhu: ¶ 0010, ¶ 0014, ¶ 0025, ¶ 0026, ¶ 0027, ¶ 0028, ¶ 0029); and C: calculating distances between the geometric center and the workpiece edges, selecting a smallest one of the distances as a second stacking parameter, and determining the stacking index of the workpiece in combination with the first stacking parameter Zhu: ¶ 0035); and the greater a numerical value of the second stacking parameter is, the higher the priority is (Zhu: ¶ 0031, ¶ 0032, ¶ 0033, ¶ 0034, ¶ 0035). Zhu teaches a system and method for grasping object using a robot. The system is configured to capture an image of the object to be grasped, determine the geometric center form the image, determine the shortest length to an edge of the object from the geometric center, and use this information to grasp the object. The method taught in Ito in view of Hayashi in further view of Duan teaches determining the shape and contours of each object, the pose of each object, and several metric used to determine the stacking order. Additionally, the method taught in Ito in view of Hayashi in further view of Duan teaches multiple metrics used to determine the grasping order, including various area based metrics. As such the method can be easily modified to include a metric based on the smallest distance from the center as taught in Zhu. Therefore, because Hayashi was not relied upon to teach the above limitations and said limitations are taught by the other cited references in the combination, the combination of Ito in view of Hayashi in further view of Duan in further view of Zhu teaches the limitation “S3: acquiring contour feature data of each of the workpieces according to the posture matrix, the contour feature data includes depth information, a first stacking parameter, and a second stacking parameter, wherein the first stacking parameter is a placing state of a workpiece and the second stacking parameter is the smallest one of the distances between a geometric center of one of the workpieces and each of workpiece edges of the workpieces”.
Therefore, based on the above reasons and the above 35 U.S.C. § 103 rejection, the 35 U.S.C. § 103 rejection of independent claim 1 is maintained.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Noah W Stiebritz whose telephone number is (571)272-3414. The examiner can normally be reached Monday thru Friday 7-5 EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Ramon Mercado can be reached at (571) 270-5744. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/N.W.S./Examiner, Art Unit 3658
/Ramon A. Mercado/Supervisory Patent Examiner, Art Unit 3658
Prosecution Insights