DETAILED ACTION
This is a non-final Office Action on the merits in response to communications filed by Applicant on March 23rd, 2026. Claims 1-10 are currently pending and examined below.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
The amendments to the Claims filed on March 23rd, 2026 have been entered. Claim 1 is currently amended and pending, claims 2-9 are original, unamended and pending, and claim 10 is as previously presented.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 1-6, 8, and 10 is/are rejected under 35 U.S.C. 103 as being unpatentable over US 10040199 B2 ("Ito") in view of US 11126844 B2 ("Hayashi") in further view of CN 112802105 A ("Duan") in further view of CN 111462232 A ("Zhu").
Regarding claim 1, Ito teaches a robot sorting method based on visual recognition, comprising (Ito: Figure 1, Abstract, “Provided is a processing apparatus for determining a work to be picked by a robot from a plurality of works, using an image, captured by an image capture device, of an area on which the plurality of works are placed. The apparatus selects a pickable candidate work based on the image, and determines a picking target work positioned in a partial area assigned with a highest priority among the partial areas where candidate works are respectively positioned. The apparatus selects a next pickable candidate work whose position and orientation have changed within allowable ranges before and after picking, and determines a next picking target work positioned in the partial area assigned with the highest priority among the partial areas where the next pickable candidate works are respectively positioned.”, Column 3 lines 22-46, “In accordance with a command from a processing apparatus 200, the measurement controller 110 for controlling the three-dimensional measurement apparatus 100 causes the light projector 120 to project pattern light on a pallet 400, and also causes the image capture device 130 to capture an area in the pallet 400 (container). The processing apparatus 200 is, for example, a computer apparatus. Note that the processing apparatus 200 may also function as the measurement controller 110. In the pallet 400, a number of works as picking targets of the picking system and as measurement targets of the three-dimensional measurement apparatus 100 are piled. The measurement controller 110 transmits the image capturing result of the image capture device 130 to the processing apparatus 200. The processing apparatus 200 recognizes the works in the pallet 400 and obtains position and orientation information based on data of an image captured by the image capture device 130, and selects a pickable candidate work. The information of the pickable candidate work is transmitted from the processing apparatus 200 to a robot 300, and the robot 300 moves a hand 310 and an arm 320 based on the information and picks the predetermined work. In the conventional measurement operation of the three-dimensional measurement apparatus, the above steps are repeatedly performed every time the robot 300 picks a work.”):
S1: acquiring a plurality of grating images in a visual field range (Ito: Column 3 lines 22-46, “In accordance with a command from a processing apparatus 200, the measurement controller 110 for controlling the three-dimensional measurement apparatus 100 causes the light projector 120 to project pattern light on a pallet 400, and also causes the image capture device 130 to capture an area in the pallet 400 (container).”, Column 5 lines 26-49, “The procedure of measurement and picking according to the embodiment will be described with reference to a flowchart shown in FIG. 5. In step S21, the three-dimensional measurement apparatus 100 performs normal three dimensional measurement. This three-dimensional measurement is implemented by, for example, a known technique of measuring the three-dimensional positions of works by projecting pattern light on the piled works and capturing the works, and a detailed description thereof will be omitted.”. One of ordinary skill in the art would recognize that projecting a pattern light and capturing an image is a grating image.),
S2: acquiring a posture matrix of every workpiece in a grabbing space from the point cloud set information (Column 3 lines 22-46, “In accordance with a command from a processing apparatus 200, the measurement controller 110 for controlling the three-dimensional measurement apparatus 100 causes the light projector 120 to project pattern light on a pallet 400, and also causes the image capture device 130 to capture an area in the pallet 400 (container). The processing apparatus 200 is, for example, a computer apparatus. Note that the processing apparatus 200 may also function as the measurement controller 110. In the pallet 400, a number of works as picking targets of the picking system and as measurement targets of the three-dimensional measurement apparatus 100 are piled. The measurement controller 110 transmits the image capturing result of the image capture device 130 to the processing apparatus 200. The processing apparatus 200 recognizes the works in the pallet 400 and obtains position and orientation information based on data of an image captured by the image capture device 130, and selects a pickable candidate work.”, Column 6 lines 18-47, “The three-dimensional measurement apparatus 100 transfers the obtained captured data to the processing apparatus 200, as needed. The processing apparatus 200 performs the three-dimensional measurement calculation of the works based on the captured data, the by obtaining the positions and orientations of the works.”);
and determining a target grabbing sequence (Ito: Column 7 line 64 – Colum 8 line 9, “The result of priority setting based on the distances is used to determine, if there exist a plurality of candidate works in one block, the picking sequence of the plurality of candidate works. Priorities among a plurality of candidate works of different blocks comply with priorities assigned to the blocks in which the respective candidate works are positioned, as in the example of the first embodiment. Consequently, in the example of FIG. 7, the picking sequence of the respective candidate works is set to "first candidate work 411[Wingdings font/0xE0]second candidate work 412[Wingdings font/0xE0]third candidate work 413[Wingdings font/0xE0]sixth candidate work 416[Wingdings font/0xE0]fifth candidate work 415[Wingdings font/0xE0]fourth candidate work 414[Wingdings font/0xE0]seventh candidate work 417".”),
and grabbing the workpieces according to the grabbing sequence (Ito: Column 3 lines 22-46, “In accordance with a command from a processing apparatus 200, the measurement controller 110 for controlling the three-dimensional measurement apparatus 100 causes the light projector 120 to project pattern light on a pallet 400, and also causes the image capture device 130 to capture an area in the pallet 400 (container). The processing apparatus 200 is, for example, a computer apparatus. Note that the processing apparatus 200 may also function as the measurement controller 110. In the pallet 400, a number of works as picking targets of the picking system and as measurement targets of the three-dimensional measurement apparatus 100 are piled. The measurement controller 110 transmits the image capturing result of the image capture device 130 to the processing apparatus 200. The processing apparatus 200 recognizes the works in the pallet 400 and obtains position and orientation information based on data of an image captured by the image capture device 130, and selects a pickable candidate work.”, Column 6 lines 18-47, “The three-dimensional measurement apparatus 100 transfers the obtained captured data to the processing apparatus 200, as needed. The processing apparatus 200 performs the three-dimensional measurement calculation of the works based on the captured data, the by obtaining the positions and orientations of the works. The processing apparatus 200 transfers the information to the robot 300, and the robot 300 performs a picking operation based on the information.”, Column 7 line 64 – Colum 8 line 9, “The result of priority setting based on the distances is used to determine, if there exist a plurality of candidate works in one block, the picking sequence of the plurality of candidate works. Priorities among a plurality of candidate works of different blocks comply with priorities assigned to the blocks in which the respective candidate works are positioned, as in the example of the first embodiment. Consequently, in the example of FIG. 7, the picking sequence of the respective candidate works is set to "first candidate work 411[Wingdings font/0xE0]second candidate work 412[Wingdings font/0xE0]third candidate work 413[Wingdings font/0xE0]sixth candidate work 416[Wingdings font/0xE0]fifth candidate work 415[Wingdings font/0xE0]fourth candidate work 414[Wingdings font/0xE0]seventh candidate work 417".”. The system is clearly configured to grasp the objects according to the determined sequence.).
Ito does not teach and performing recognition and fusion to obtain point cloud set information;
S3: acquiring contour feature data of each of the workpieces according to the posture matrix, the contour feature data includes depth information,
a first stacking parameter,
and a second stacking parameter,
wherein the first stacking parameter is a placing state of a workpiece
and the second stacking parameter is the smallest one of the distances between a geometric center of one of the workpieces and each of workpiece edges of the workpieces; and
S4: comparing the contour feature data of the workpieces.
Hayashi, in the same field of endeavor, teaches and performing recognition and fusion to obtain point cloud set information (Hayashi: Column 5 lines 26-41, “In the second point cloud generation process, first, a second imaging process of imaging a plurality of phase shifting images PSM1 to PSM4 containing the parts PP with a plurality of phase shifting patterns PH1 to PH4 selectively projected one by one as the projection patterns is performed using the projection device 810. The plurality of phase shifting patterns PH1 to PH4 are sinusoidal banded patterns having dark and light parts. In the phase shifting method, generally, capturing of images is performed using the n phase shift patterns PH2 to PHn for n (integer) equal to or larger than three. The n phase shift patterns PH1 to PHn are sinusoidal patterns having phase sequentially shifted by 2π/n. In the example of FIG. 7, n=4. Note that, in the phase shifting method, it is not necessary to use the stereo camera, and only one of the two cameras forming the stereo camera may be used for imaging.”, Column 5 lines 42-52, “In the second point cloud generation process, then, a second analysis process is executed using the plurality of phase shifting images PSM1 to PSM4, and thereby, a second point cloud PG2 is generated. In the second analysis process, first, a distance image DMZ is generated according to the phase shifting method using the plurality of phase shifting images PSM1 to PSM4. The generation process of the distance image DMZ using the phase shifting method is well known, and the explanation thereof is omitted here. In the second point cloud generation process, a second point cloud PG2 is further generated using the distance image DMZ.”. The cited passages clearly teach that the system uses known techniques to create a point cloud from a grating image.);
S3: acquiring contour feature data of each of the workpieces according to the posture matrix, the contour feature data includes depth information (Column 5 lines 26-41, “In the second point cloud generation process, first, a second imaging process of imaging a plurality of phase shifting images PSM1 to PSM4 containing the parts PP with a plurality of phase shifting patterns PH1 to PH4 selectively projected one by one as the projection patterns is performed using the projection device 810. The plurality of phase shifting patterns PH1 to PH4 are sinusoidal banded patterns having dark and light parts. In the phase shifting method, generally, capturing of images is performed using the n phase shift patterns PH2 to PHn for n (integer) equal to or larger than three. The n phase shift patterns PH1 to PHn are sinusoidal patterns having phase sequentially shifted by 2π/n. In the example of FIG. 7, n=4. Note that, in the phase shifting method, it is not necessary to use the stereo camera, and only one of the two cameras forming the stereo camera may be used for imaging.”, Column 5 lines 42-52, “In the second point cloud generation process, then, a second analysis process is executed using the plurality of phase shifting images PSM1 to PSM4, and thereby, a second point cloud PG2 is generated. In the second analysis process, first, a distance image DMZ is generated according to the phase shifting method using the plurality of phase shifting images PSM1 to PSM4. The generation process of the distance image DMZ using the phase shifting method is well known, and the explanation thereof is omitted here. In the second point cloud generation process, a second point cloud PG2 is further generated using the distance image DMZ.”, Column 6 lines 43-48, “At step S190, if the first point cloud generation part 271 stops the process, the second point cloud generation part 272 continues execution of the second point cloud generation process. Then, at step S230, the object detection execution part 273 detects the three-dimensional shapes of the parts PP as objects using the second point cloud PG2.”, Column 6 lines 49-63, “As the method of detecting the three-dimensional shapes of the objects using the point cloud, e.g. the method described in JP-A-2017-182274 disclosed by the applicant of this application can be used. Or, the method described in "Model Globally, Match Locally: Efficient and Robust 3D Object Recognition", Bertram Drost et al., http://campar.in-.tum.de/pub/drost20l 0CVPR/drost201 0CVPR.pdf, the method described in "Bekutoru Pea Mattingu niyoru Kousinraina Sanjigen Ichi Shisei Ninshiki", Shuichi Akizuki, http://isl.sist.chukyou.ac.jp/Archives/vpm.pdf, or the like may be used. These methods are object detection methods using the point cloud and the three-dimensional model data of the objects. Note that the three-dimensional shapes of the objects may be detected from the point cloud without using the three-dimensional model data.”. The cited passages from Hayashi shows that the contour curve and the depth information of each article is determined based on the grating image.),
Ito teaches a robot sorting method based on visual recognition, comprising: S1: acquiring a plurality of grating images in a visual field range, S2: acquiring a posture matrix of each of workpieces in a grabbing space from the point cloud set information; and determining a target grabbing sequence to grab the workpieces. Ito does not teach and performing recognition and fusion to obtain point cloud set information; S3: acquiring contour feature data of each of the workpieces according to the posture matrix, the contour feature data includes depth information Hayashi teaches and performing recognition and fusion to obtain point cloud set information; S3: acquiring contour feature data of each of the workpieces according to the posture matrix, the contour feature data includes depth information. A person of ordinary skill in the art would have had the technological capabilities required to have modified the method taught in Ito with and performing recognition and fusion to obtain point cloud set information; S3: acquiring contour feature data of each of the workpieces according to the posture matrix, the contour feature data includes depth information taught in Hayashi. Furthermore, the method taught in Ito is already configured to acquire depth information from a grating image created by projecting a pattern of lines on the objects. Hayashi teaches a similar method of generating a grating image and acquiring depth from the grating image to construct a point cloud. As such, the method taught in Ito is readily configurable with the method of constructing a point cloud taught in Hayashi and can be simply added to the method taught in Ito according to known methods. Additionally, Ito teaches all of the necessary components required to acquire contour information of the objects taught in Hayashi. Such modifications would not have changed or introduced new functionality. No inventive effort would have been required. The combination would have yielded the predictable result of a robot sorting method based on visual recognition, comprising: and performing recognition and fusion to obtain point cloud set information; S3: acquiring contour feature data of each of the workpieces according to the posture matrix, the contour feature data includes depth information.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filling date of the claimed invention, to have combine the robot sorting method taught in Ito with and performing recognition and fusion to obtain point cloud set information; S3: acquiring contour feature data of each of the workpieces according to the posture matrix, the contour feature data includes depth information taught in Hayashi with a reasonable expectation of success. One of ordinary skill in the art would have been motivated to make this modification because the combination would have yielded predictable results.
Ito in view of Hayashi does not teach the a first stacking parameter,
and a second stacking parameter,
wherein the first stacking parameter is a placing state of a workpiece
and the second stacking parameter is the smallest one of the distances between a geometric center of one of the workpieces and each of workpiece edges of the workpieces; and
S4: comparing the contour feature data of the workpieces.
Duan, in the same field of endeavor, teaches a first stacking parameter (Duan: ¶ 0083, “First, sample images corresponding to the three-dimensional sample area are acquired to identify multiple object objects contained in the sample images. The three-dimensional sample area contains multiple items to be grabbed, which serve as samples. The sample image corresponding to the three-dimensional sample region includes: a two-dimensional color image corresponding to the three-dimensional sample region obtained along a preset depth direction, and a depth image corresponding to the two-dimensional color image. For specific methods of obtaining the information, please refer to the corresponding description in step S110, which will not be repeated here. In determining multiple object objects contained in a sample image, the contours, boundary lines, and other information between each object can be identified through instance segmentation, and then the multiple object objects contained in the sample image can be segmented based on the recognition results.”, ¶ 0086, “In another alternative implementation, based on the exposure ratio of each item object, the area corresponding to the item object with an exposure ratio greater than a preset threshold is marked as a crawlable area, and the area corresponding to the item object with an exposure ratio not greater than the preset threshold is marked as an uncrawlable area. In some scenarios, the stacking relationship between items is not easy to determine, and items on the same layer may overlap each other. In such cases, it is difficult to accurately label the top item. For example, in supermarket product picking scenarios, the hierarchical relationship between products is not clear, and there may be overlapping between items on the same layer. Therefore, the requirements for the picking order are not strict, but the distinction between pickable and non-pickable items is more stringent. In this case, items with little surface exposure, or items that may cause other items in the scene to fly out after being grabbed, should not be marked as grabbable items. Accordingly, in the above scenario, a labeling threshold can be set, such as 85%. If an item's exposed surface area is greater than 85%, it is marked as a graspable item; if an item's exposed surface area is less than 85%, it is marked as a non-grabable item. Of course, the exposure ratio of an item can be quantified not only by the exposed surface area but also by the exposed volume. This invention does not limit the specific details.”. The cited passages teach that the first stacking parameter of the object is set based on both the depth information and how much of the object is exposed (i.e. not occluded).),
wherein the first stacking parameter is a placing state of a workpiece (Duan: ¶ 0083, “First, sample images corresponding to the three-dimensional sample area are acquired to identify multiple object objects contained in the sample images. The three-dimensional sample area contains multiple items to be grabbed, which serve as samples. The sample image corresponding to the three-dimensional sample region includes: a two-dimensional color image corresponding to the three-dimensional sample region obtained along a preset depth direction, and a depth image corresponding to the two-dimensional color image. For specific methods of obtaining the information, please refer to the corresponding description in step S110, which will not be repeated here. In determining multiple object objects contained in a sample image, the contours, boundary lines, and other information between each object can be identified through instance segmentation, and then the multiple object objects contained in the sample image can be segmented based on the recognition results.”, ¶ 0086, “In another alternative implementation, based on the exposure ratio of each item object, the area corresponding to the item object with an exposure ratio greater than a preset threshold is marked as a crawlable area, and the area corresponding to the item object with an exposure ratio not greater than the preset threshold is marked as an uncrawlable area. In some scenarios, the stacking relationship between items is not easy to determine, and items on the same layer may overlap each other. In such cases, it is difficult to accurately label the top item. For example, in supermarket product picking scenarios, the hierarchical relationship between products is not clear, and there may be overlapping between items on the same layer. Therefore, the requirements for the picking order are not strict, but the distinction between pickable and non-pickable items is more stringent. In this case, items with little surface exposure, or items that may cause other items in the scene to fly out after being grabbed, should not be marked as grabbable items. Accordingly, in the above scenario, a labeling threshold can be set, such as 85%. If an item's exposed surface area is greater than 85%, it is marked as a graspable item; if an item's exposed surface area is less than 85%, it is marked as a non-grabable item. Of course, the exposure ratio of an item can be quantified not only by the exposed surface area but also by the exposed volume. This invention does not limit the specific details.”. The cited passages teach that the stacking state of the object is set based on both the depth information and how much of the object is exposed (i.e. not occluded).) and
S4: comparing the contour feature data of the workpieces (Duan: ¶ 0083, “In determining multiple object objects contained in a sample image, the contours, boundary lines, and other information between each object can be identified through instance segmentation, and then the multiple object objects contained in the sample image can be segmented based on the recognition results.”, ¶ 0086, “In another alternative implementation, based on the exposure ratio of each item object, the area corresponding to the item object with an exposure ratio greater than a preset threshold is marked as a crawlable area, and the area corresponding to the item object with an exposure ratio not greater than the preset threshold is marked as an uncrawlable area. In some scenarios, the stacking relationship between items is not easy to determine, and items on the same layer may overlap each other. In such cases, it is difficult to accurately label the top item. For example, in supermarket product picking scenarios, the hierarchical relationship between products is not clear, and there may be overlapping between items on the same layer. Therefore, the requirements for the picking order are not strict, but the distinction between pickable and non-pickable items is more stringent. In this case, items with little surface exposure, or items that may cause other items in the scene to fly out after being grabbed, should not be marked as grabbable items. Accordingly, in the above scenario, a labeling threshold can be set, such as 85%. If an item's exposed surface area is greater than 85%, it is marked as a graspable item; if an item's exposed surface area is less than 85%, it is marked as a non-grabable item. Of course, the exposure ratio of an item can be quantified not only by the exposed surface area but also by the exposed volume. This invention does not limit the specific details.”, ¶ 0087, “In another alternative implementation, the contact area contained in each item object is determined according to the shape and/or type of each item object. The area corresponding to the item object whose contact area is not occluded is marked as a grabbable area, and the area corresponding to the item object whose contact area is occluded is marked as a non-grabable area. The contact area refers to the force-bearing area of the object that is easy to grasp. For example, for items with metal parts, in order to prevent damage to the parts, it is necessary to grasp a specific area of the metal parts. This specific area is the contact area, which is usually a relatively firm and not easily detached area of the metal parts. Therefore, when labeling such items, it is also necessary to determine whether the contact area is completely exposed and unobstructed. If an item has a large exposed surface area but the contact area is obstructed, it should still be marked as an ungrabable item.”. The cited passages clearly show that the contour information of each object is compared in order to determine a grasping order. The cited passage discloses using the exposed area of each object or the available contact area of each object, both of which are defined, in part, by the contours of the object.).
Ito in view of Hayashi teaches a robot sorting method based on visual recognition, comprising: S1: acquiring a plurality of grating images in a visual field range, and performing recognition and fusion to obtain point cloud set information; S2: acquiring a posture matrix of each of workpieces in a grabbing space from the point cloud set information; S3: acquiring contour feature data of each of the workpieces according to the posture matrix, the contour feature data includes depth information and determining a target grabbing sequence to grab the workpieces. Ito in view of Hayashi does not teach a first stacking parameter, and a second stacking parameter, wherein the first stacking parameter is a placing state of a workpiece S4: comparing the contour feature data of the workpieces. Duan teaches a first stacking parameter, and a second stacking parameter, wherein the first stacking parameter is a placing state of a workpiece S4: comparing the contour feature data of the workpieces. A person of ordinary skill in the art would have had the technological capabilities required to have modified the method taught in Ito in view of Hayashi with a first stacking parameter, and a second stacking parameter, wherein the first stacking parameter is a placing state of a workpiece S4: comparing the contour feature data of the workpieces taught in Duan. Furthermore, the method taught in Ito in view of Hayashi is already configured to determine the contour information of each object and a grasping order for the objects. As such, the method taught in Ito in view of Hayashi could be easily modified to use the contour information when determining the grasping sequence using the methods taught in Duan. Such a modification would not have changed or introduced new functionality. No inventive effort would have been required. The combination would have yielded the predictable result of a robot sorting method based on visual recognition, comprising: a first stacking parameter, and a second stacking parameter, wherein the first stacking parameter is a placing state of a workpiece S4: comparing the contour feature data of the workpieces.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filling date of the claimed invention, to have combine the robot sorting method taught in Ito in view of Hayashi with a first stacking parameter, and a second stacking parameter, wherein the first stacking parameter is a placing state of a workpiece S4: comparing the contour feature data of the workpieces taught in Duan with a reasonable expectation of success. One of ordinary skill in the art would have been motivated to make this modification because the combination would have yielded predictable results.
Ito in view of Hayashi in further view of Duan does not teach and a second stacking parameter, and the second stacking parameter is the smallest one of the distances between a geometric center of one of the workpieces and each of workpiece edges of the workpieces.
Zhu, in the same field of endeavor, teaches and a second stacking parameter, and the second stacking parameter is the smallest one of the distances between a geometric center of one of the workpieces and each of workpiece edges of the workpieces (Zhu: ¶ 0010, “Determine the geometric center of the object based on the separation position;”, ¶ 0014, “The preset size includes half, the first endpoint is the point on the edge of the object that is closest to the geometric center at the separation position, and the second endpoint is located on the edge of the object and on the extension line of the line connecting the first endpoint and the geometric center.”, ¶ 0025, “Furthermore, the step of determining the geometric center of the object based on the separation position includes the following steps:”, ¶ 0026, “Using the endpoints of the two ends of the third pixel line segment as the initial point and the termination point respectively, query the pixel grayscale value of the next pixel point in the direction from the initial point to the termination point.”, ¶ 0027 “If the pixel grayscale value is 1, record it as the first point. Continue the query from the first point to the end point until the second point is obtained and recorded.”, ¶ 0028, “If the pixel grayscale value is 0, continue searching towards the end point until a pixel with a grayscale value of 1 is found, and record it as the first point. Starting from the first point, continue searching towards the end point until the second point is found and recorded.”, ¶ 0029, “Based on the first point and the second point, the midpoint between the first point and the second point is determined as the geometric center;” ¶ 0031, “Furthermore, the step of determining the first endpoint and the second endpoint based on the geometric center and the separation position includes the following steps:”, ¶ 0032, “Select one segment of the third pixel line segment that is divided by the geometric center as the first line segment;”, ¶ 0033, “The first line segment is rotated several times along the geometric center at a first preset angle until the first line segment returns to its initial position, and the length of the second line segment is calculated during each first rotation, wherein the length of the second line segment is the length between the object edge on the first line segment and the geometric center;”, ¶ 0034, “Determine two adjacent second line segments whose sum of lengths is the smallest, and rotate one of the second line segments several times along the geometric center toward the other second line segment at a second preset angle, and calculate the length of the third line segment during each second rotation, where the length of the third line segment is the length between the object edge on the second line segment and the geometric center.”, ¶ 0035, “Take the endpoint of the third line segment with the shortest length as the first endpoint, and determine the second endpoint;”. The cited passages clearly shows that the geometric center of the object is determined, a distance from the center to the edges of the object is determined, and then the smallest of these distances is selected. Additionally this process is used to control the robot when grasping an object.).
Ito in view of Hayashi in further view of Duan teaches a robot sorting method based on visual recognition, comprising: S1: acquiring a plurality of grating images in a visual field range, and performing recognition and fusion to obtain point cloud set information; S2: acquiring a posture matrix of each of workpieces in a grabbing space from the point cloud set information; S3: acquiring contour feature data of each of the workpieces according to the posture matrix, the contour feature data includes depth information, a first stacking parameter, and a second stacking parameter, wherein the first stacking parameter is a placing state of a workpiece S4: comparing the contour feature data of the workpieces and determining a target grabbing sequence to grab the workpieces. Ito in view of Hayashi in further view of Duan does not teach a second stacking parameter, and the second stacking parameter is the smallest one of the distances between a geometric center of one of the workpieces and each of workpiece edges of the workpieces. Zhu teaches a second stacking parameter, and the second stacking parameter is the smallest one of the distances between a geometric center of one of the workpieces and each of workpiece edges of the workpieces. A person of ordinary skill in the art would have had the technological capabilities required to have modified the method taught in Ito in view of Hayashi in further view of Duan with a second stacking parameter, and the second stacking parameter is the smallest one of the distances between a geometric center of one of the workpieces and each of workpiece edges of the workpieces taught in Zhu. Furthermore, the method taught in Ito in view of Hayashi in further view of Duan teaches determining the shape and contours of each object, the pose of each object, and several metric used to determine the stacking order. As such, the method taught in Ito in view of Hayashi in further view of Duan teaches all of the components necessary to determine the geometric center and smallest distance to the edge as taught in Zhu. Additionally, the method taught in Ito in view of Hayashi in further view of Duan teaches multiple metrics used to determine the grasping order, including various area based metrics. As such the method can be easily modified to include a metric based on the smallest distance from the center as taught in Zhu. Such modifications would not change or introduce new functionality. No inventive effort would have been required. The combination would have yielded the predictable result of sorting method based on visual recognition, comprising: a second stacking parameter, and the second stacking parameter is the smallest one of the distances between a geometric center of one of the workpieces and each of workpiece edges of the workpieces.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filling date of the claimed invention, to have combine the robot sorting method taught in Ito in view of Hayashi in further view of Duan with a second stacking parameter, and the second stacking parameter is the smallest one of the distances between a geometric center of one of the workpieces and each of workpiece edges of the workpieces taught in Zhu with a reasonable expectation of success. One of ordinary skill in the art would have been motivated to make this modification because the combination would have yielded predictable results.
Regarding claim 2, Ito in view of Hayashi in further view of Duan in further view of Zhu teaches wherein S3 comprises: S31: determining a contour curve and depth information of each of the workpieces according to the posture matrix (Hayashi: Column 5 lines 26-41, “In the second point cloud generation process, first, a second imaging process of imaging a plurality of phase shifting images PSM1 to PSM4 containing the parts PP with a plurality of phase shifting patterns PH1 to PH4 selectively projected one by one as the projection patterns is performed using the projection device 810. The plurality of phase shifting patterns PH1 to PH4 are sinusoidal banded patterns having dark and light parts. In the phase shifting method, generally, capturing of images is performed using the n phase shift patterns PH2 to PHn for n (integer) equal to or larger than three. The n phase shift patterns PH1 to PHn are sinusoidal patterns having phase sequentially shifted by 2π/n. In the example of FIG. 7, n=4. Note that, in the phase shifting method, it is not necessary to use the stereo camera, and only one of the two cameras forming the stereo camera may be used for imaging.”, Column 5 lines 42-52, “In the second point cloud generation process, then, a second analysis process is executed using the plurality of phase shifting images PSM1 to PSM4, and thereby, a second point cloud PG2 is generated. In the second analysis process, first, a distance image DMZ is generated according to the phase shifting method using the plurality of phase shifting images PSM1 to PSM4. The generation process of the distance image DMZ using the phase shifting method is well known, and the explanation thereof is omitted here. In the second point cloud generation process, a second point cloud PG2 is further generated using the distance image DMZ.”, Column 6 lines 43-48, “At step S190, if the first point cloud generation part 271 stops the process, the second point cloud generation part 272 continues execution of the second point cloud generation process. Then, at step S230, the object detection execution part 273 detects the three-dimensional shapes of the parts PP as objects using the second point cloud PG2.”, Column 6 lines 49-63, “As the method of detecting the three-dimensional shapes of the objects using the point cloud, e.g. the method described in JP-A-2017-182274 disclosed by the applicant of this application can be used. Or, the method described in "Model Globally, Match Locally: Efficient and Robust 3D Object Recognition", Bertram Drost et al., http://campar.in-.tum.de/pub/drost20l 0CVPR/drost201 0CVPR.pdf, the method described in "Bekutoru Pea Mattingu niyoru Kousinraina Sanjigen Ichi Shisei Ninshiki", Shuichi Akizuki, http://isl.sist.chukyou.ac.jp/Archives/vpm.pdf, or the like may be used. These methods are object detection methods using the point cloud and the three-dimensional model data of the objects. Note that the three-dimensional shapes of the objects may be detected from the point cloud without using the three-dimensional model data.”. Duan: ¶ 0022, “Calculate the depth coordinate value of each grabbable object corresponding to the third coordinate axis, and calculate the three-dimensional pose information of each grabbable object based on the depth coordinate value.”, ¶ 0046, “Calculate the depth coordinate value of each grabbable object corresponding to the third coordinate axis, and calculate the three-dimensional pose information of each grabbable object based on the depth coordinate value.”, ¶ 0078, “Since the 3D pose information includes depth information, the grabbable objects can be sorted along a preset depth direction based on the 3D pose information. In this embodiment, the preset depth direction is consistent with the camera's shooting direction. For example, when the camera is shooting from above, the objects to be grabbed are sorted according to their height: the taller the object, the higher it is sorted, and the shorter the object is sorted, the lower it is sorted. Accordingly, when determining the grabbing order of each grabbable object based on the sorting results, the taller ones are grabbed first, and the shorter ones are grabbed later, thus ensuring that the robot grabs the items sequentially from top to bottom.”, ¶ 0083, “First, sample images corresponding to the three-dimensional sample area are acquired to identify multiple object objects contained in the sample images. The three-dimensional sample area contains multiple items to be grabbed, which serve as samples. The sample image corresponding to the three-dimensional sample region includes: a two-dimensional color image corresponding to the three-dimensional sample region obtained along a preset depth direction, and a depth image corresponding to the two-dimensional color image. For specific methods of obtaining the information, please refer to the corresponding description in step S110, which will not be repeated here. In determining multiple object objects contained in a sample image, the contours, boundary lines, and other information between each object can be identified through instance segmentation, and then the multiple object objects contained in the sample image can be segmented based on the recognition results.”. The cited passages from both Hayashi and Duan show that the contour curve and the depth information of each article is determined based on the grating image.); and
S32: analyzing a stacking degree according to the contour curve, and calculating a stacking index (Duan: ¶ 0015, “Determine the stacking order of each item object along the preset depth direction, mark the area corresponding to the top-level item object as a graspable area, and mark the area corresponding to the bottom-level item object as an ungraspable area; and/or”, ¶ 0016, “Based on the exposure ratio of each item, areas corresponding to item objects with exposure ratios greater than a preset threshold are marked as crawlable areas, while areas corresponding to item objects with exposure ratios not greater than the preset threshold are marked as uncrawlable areas; and/or,”, ¶ 0017, “Based on the shape and/or type of each item, determine the contact area contained in each item object, mark the area corresponding to the item object whose contact area is not occluded as the grabbable area, and mark the area corresponding to the item object whose contact area is occluded as the non-grabable area.”, ¶ 0083, “First, sample images corresponding to the three-dimensional sample area are acquired to identify multiple object objects contained in the sample images. The three-dimensional sample area contains multiple items to be grabbed, which serve as samples. The sample image corresponding to the three-dimensional sample region includes: a two-dimensional color image corresponding to the three-dimensional sample region obtained along a preset depth direction, and a depth image corresponding to the two-dimensional color image. For specific methods of obtaining the information, please refer to the corresponding description in step S110, which will not be repeated here. In determining multiple object objects contained in a sample image, the contours, boundary lines, and other information between each object can be identified through instance segmentation, and then the multiple object objects contained in the sample image can be segmented based on the recognition results.”, ¶ 0086, “In another alternative implementation, based on the exposure ratio of each item object, the area corresponding to the item object with an exposure ratio greater than a preset threshold is marked as a crawlable area, and the area corresponding to the item object with an exposure ratio not greater than the preset threshold is marked as an uncrawlable area. In some scenarios, the stacking relationship between items is not easy to determine, and items on the same layer may overlap each other. In such cases, it is difficult to accurately label the top item. For example, in supermarket product picking scenarios, the hierarchical relationship between products is not clear, and there may be overlapping between items on the same layer. Therefore, the requirements for the picking order are not strict, but the distinction between pickable and non-pickable items is more stringent. In this case, items with little surface exposure, or items that may cause other items in the scene to fly out after being grabbed, should not be marked as grabbable items. Accordingly, in the above scenario, a labeling threshold can be set, such as 85%. If an item's exposed surface area is greater than 85%, it is marked as a graspable item; if an item's exposed surface area is less than 85%, it is marked as a non-grabable item. Of course, the exposure ratio of an item can be quantified not only by the exposed surface area but also by the exposed volume. This invention does not limit the specific details.”, ¶ 0087, “In another alternative implementation, the contact area contained in each item object is determined according to the shape and/or type of each item object. The area corresponding to the item object whose contact area is not occluded is marked as a grabbable area, and the area corresponding to the item object whose contact area is occluded is marked as a non-grabable area. The contact area refers to the force-bearing area of the object that is easy to grasp. For example, for items with metal parts, in order to prevent damage to the parts, it is necessary to grasp a specific area of the metal parts. This specific area is the contact area, which is usually a relatively firm and not easily detached area of the metal parts. Therefore, when labeling such items, it is also necessary to determine whether the contact area is completely exposed and unobstructed. If an item has a large exposed surface area but the contact area is obstructed, it should still be marked as an ungrabable item.”. The cited passages clearly teach that the system is configured to determine a stacking degree and stacking index based, in part, on the contour curve of each object.).
Regarding claim 3, Ito in view of Hayashi in further view of Duan in further view of Zhu teaches wherein S32 comprises: A: determining workpiece edges of each of the workpieces according to the contour curve (Duan: ¶ 0083, “First, sample images corresponding to the three-dimensional sample area are acquired to identify multiple object objects contained in the sample images. The three-dimensional sample area contains multiple items to be grabbed, which serve as samples. The sample image corresponding to the three-dimensional sample region includes: a two-dimensional color image corresponding to the three-dimensional sample region obtained along a preset depth direction, and a depth image corresponding to the two-dimensional color image. For specific methods of obtaining the information, please refer to the corresponding description in step S110, which will not be repeated here. In determining multiple object objects contained in a sample image, the contours, boundary lines, and other information between each object can be identified through instance segmentation, and then the multiple object objects contained in the sample image can be segmented based on the recognition results.”. The edges of the objects are clearly determined.);
B: determining a placing state of the workpiece as a first stacking parameter according to the depth information and the workpiece edges (Duan: ¶ 0083, “First, sample images corresponding to the three-dimensional sample area are acquired to identify multiple object objects contained in the sample images. The three-dimensional sample area contains multiple items to be grabbed, which serve as samples. The sample image corresponding to the three-dimensional sample region includes: a two-dimensional color image corresponding to the three-dimensional sample region obtained along a preset depth direction, and a depth image corresponding to the two-dimensional color image. For specific methods of obtaining the information, please refer to the corresponding description in step S110, which will not be repeated here. In determining multiple object objects contained in a sample image, the contours, boundary lines, and other information between each object can be identified through instance segmentation, and then the multiple object objects contained in the sample image can be segmented based on the recognition results.”, ¶ 0086, “In another alternative implementation, based on the exposure ratio of each item object, the area corresponding to the item object with an exposure ratio greater than a preset threshold is marked as a crawlable area, and the area corresponding to the item object with an exposure ratio not greater than the preset threshold is marked as an uncrawlable area. In some scenarios, the stacking relationship between items is not easy to determine, and items on the same layer may overlap each other. In such cases, it is difficult to accurately label the top item. For example, in supermarket product picking scenarios, the hierarchical relationship between products is not clear, and there may be overlapping between items on the same layer. Therefore, the requirements for the picking order are not strict, but the distinction between pickable and non-pickable items is more stringent. In this case, items with little surface exposure, or items that may cause other items in the scene to fly out after being grabbed, should not be marked as grabbable items. Accordingly, in the above scenario, a labeling threshold can be set, such as 85%. If an item's exposed surface area is greater than 85%, it is marked as a graspable item; if an item's exposed surface area is less than 85%, it is marked as a non-grabable item. Of course, the exposure ratio of an item can be quantified not only by the exposed surface area but also by the exposed volume. This invention does not limit the specific details.”. The cited passages teach that the state of the object is set based on both the depth information and how much of the object is exposed (i.e. not occluded).),
and calculating a geometric center of the workpiece (Zhu: ¶ 0010, “Determine the geometric center of the object based on the separation position;”, ¶ 0014, “The preset size includes half, the first endpoint is the point on the edge of the object that is closest to the geometric center at the separation position, and the second endpoint is located on the edge of the object and on the extension line of the line connecting the first endpoint and the geometric center.”, ¶ 0025, “Furthermore, the step of determining the geometric center of the object based on the separation position includes the following steps:”, ¶ 0026, “Using the endpoints of the two ends of the third pixel line segment as the initial point and the termination point respectively, query the pixel grayscale value of the next pixel point in the direction from the initial point to the termination point.”, ¶ 0027 “If the pixel grayscale value is 1, record it as the first point. Continue the query from the first point to the end point until the second point is obtained and recorded.”, ¶ 0028, “If the pixel grayscale value is 0, continue searching towards the end point until a pixel with a grayscale value of 1 is found, and record it as the first point. Starting from the first point, continue searching towards the end point until the second point is found and recorded.”, ¶ 0029, “Based on the first point and the second point, the midpoint between the first point and the second point is determined as the geometric center;”. The system is clearly configured to determine the geometric center of the object.); and
C: calculating distances between the geometric center and the workpiece edges (Zhu: ¶ 0031, “Furthermore, the step of determining the first endpoint and the second endpoint based on the geometric center and the separation position includes the following steps:”, ¶ 0032, “Select one segment of the third pixel line segment that is divided by the geometric center as the first line segment;”, ¶ 0033, “The first line segment is rotated several times along the geometric center at a first preset angle until the first line segment returns to its initial position, and the length of the second line segment is calculated during each first rotation, wherein the length of the second line segment is the length between the object edge on the first line segment and the geometric center;”, ¶ 0034, “Determine two adjacent second line segments whose sum of lengths is the smallest, and rotate one of the second line segments several times along the geometric center toward the other second line segment at a second preset angle, and calculate the length of the third line segment during each second rotation, where the length of the third line segment is the length between the object edge on the second line segment and the geometric center.”, ¶ 0035, “Take the endpoint of the third line segment with the shortest length as the first endpoint, and determine the second endpoint;”. The cited passages clearly show that the distances from the center to the edges are determined.),
selecting a smallest one of the distances as a second stacking parameter (Zhu: ¶ 0035, “Take the endpoint of the third line segment with the shortest length as the first endpoint, and determine the second endpoint;”. The cited passage shows that the smallest distance is used as a control parameter.),
and determining the stacking index of the workpiece in combination with the first stacking parameter (Duan: ¶ 0015, “Determine the stacking order of each item object along the preset depth direction, mark the area corresponding to the top-level item object as a graspable area, and mark the area corresponding to the bottom-level item object as an ungraspable area; and/or”, ¶ 0016, “Based on the exposure ratio of each item, areas corresponding to item objects with exposure ratios greater than a preset threshold are marked as crawlable areas, while areas corresponding to item objects with exposure ratios not greater than the preset threshold are marked as uncrawlable areas; and/or,”, ¶ 0017, “Based on the shape and/or type of each item, determine the contact area contained in each item object, mark the area corresponding to the item object whose contact area is not occluded as the grabbable area, and mark the area corresponding to the item object whose contact area is occluded as the non-grabable area.”, ¶ 0083, “First, sample images corresponding to the three-dimensional sample area are acquired to identify multiple object objects contained in the sample images. The three-dimensional sample area contains multiple items to be grabbed, which serve as samples. The sample image corresponding to the three-dimensional sample region includes: a two-dimensional color image corresponding to the three-dimensional sample region obtained along a preset depth direction, and a depth image corresponding to the two-dimensional color image. For specific methods of obtaining the information, please refer to the corresponding description in step S110, which will not be repeated here. In determining multiple object objects contained in a sample image, the contours, boundary lines, and other information between each object can be identified through instance segmentation, and then the multiple object objects contained in the sample image can be segmented based on the recognition results.”, ¶ 0086, “In another alternative implementation, based on the exposure ratio of each item object, the area corresponding to the item object with an exposure ratio greater than a preset threshold is marked as a crawlable area, and the area corresponding to the item object with an exposure ratio not greater than the preset threshold is marked as an uncrawlable area. In some scenarios, the stacking relationship between items is not easy to determine, and items on the same layer may overlap each other. In such cases, it is difficult to accurately label the top item. For example, in supermarket product picking scenarios, the hierarchical relationship between products is not clear, and there may be overlapping between items on the same layer. Therefore, the requirements for the picking order are not strict, but the distinction between pickable and non-pickable items is more stringent. In this case, items with little surface exposure, or items that may cause other items in the scene to fly out after being grabbed, should not be marked as grabbable items. Accordingly, in the above scenario, a labeling threshold can be set, such as 85%. If an item's exposed surface area is greater than 85%, it is marked as a graspable item; if an item's exposed surface area is less than 85%, it is marked as a non-grabable item. Of course, the exposure ratio of an item can be quantified not only by the exposed surface area but also by the exposed volume. This invention does not limit the specific details.”, ¶ 0087, “In another alternative implementation, the contact area contained in each item object is determined according to the shape and/or type of each item object. The area corresponding to the item object whose contact area is not occluded is marked as a grabbable area, and the area corresponding to the item object whose contact area is occluded is marked as a non-grabable area. The contact area refers to the force-bearing area of the object that is easy to grasp. For example, for items with metal parts, in order to prevent damage to the parts, it is necessary to grasp a specific area of the metal parts. This specific area is the contact area, which is usually a relatively firm and not easily detached area of the metal parts. Therefore, when labeling such items, it is also necessary to determine whether the contact area is completely exposed and unobstructed. If an item has a large exposed surface area but the contact area is obstructed, it should still be marked as an ungrabable item.”. The cited passages clearly teach that the system is configured to determine a stacking degree and stacking index based, in part, on the contour curve of each object. The cited passage clearly teaches that the stacking index is determined based on the first stacking parameter.);
wherein, the placing state comprises a natural putting state or a stacking state (Duan: ¶ 0083, “First, sample images corresponding to the three-dimensional sample area are acquired to identify multiple object objects contained in the sample images. The three-dimensional sample area contains multiple items to be grabbed, which serve as samples. The sample image corresponding to the three-dimensional sample region includes: a two-dimensional color image corresponding to the three-dimensional sample region obtained along a preset depth direction, and a depth image corresponding to the two-dimensional color image. For specific methods of obtaining the information, please refer to the corresponding description in step S110, which will not be repeated here. In determining multiple object objects contained in a sample image, the contours, boundary lines, and other information between each object can be identified through instance segmentation, and then the multiple object objects contained in the sample image can be segmented based on the recognition results.”, ¶ 0086, “In another alternative implementation, based on the exposure ratio of each item object, the area corresponding to the item object with an exposure ratio greater than a preset threshold is marked as a crawlable area, and the area corresponding to the item object with an exposure ratio not greater than the preset threshold is marked as an uncrawlable area. In some scenarios, the stacking relationship between items is not easy to determine, and items on the same layer may overlap each other. In such cases, it is difficult to accurately label the top item. For example, in supermarket product picking scenarios, the hierarchical relationship between products is not clear, and there may be overlapping between items on the same layer. Therefore, the requirements for the picking order are not strict, but the distinction between pickable and non-pickable items is more stringent. In this case, items with little surface exposure, or items that may cause other items in the scene to fly out after being grabbed, should not be marked as grabbable items. Accordingly, in the above scenario, a labeling threshold can be set, such as 85%. If an item's exposed surface area is greater than 85%, it is marked as a graspable item; if an item's exposed surface area is less than 85%, it is marked as a non-grabable item. Of course, the exposure ratio of an item can be quantified not only by the exposed surface area but also by the exposed volume. This invention does not limit the specific details.”. One of ordinary skill in the art would recognize that objects that have an exposed surface area greater than the threshold do not have an object(s) stacked on top of it and are therefore grabbable (i.e. a natural putting state as no objects are on top of it). Objects have an exposed surface area less than the threshold have an object(s) stacked on top of it and are therefore not grabbable (i.e. a stacking state as objects are on top of it).).
Regarding claim 4, Ito in view of Hayashi in further view of Duan in further view of Zhu teaches wherein S4 comprises: S41: sequentially determining a first priority, a second priority and a third priority of each of the workpieces in the grabbing space according to the depth information, the first stacking parameter and the second stacking parameter (Duan: ¶ 0015, “Determine the stacking order of each item object along the preset depth direction, mark the area corresponding to the top-level item object as a graspable area, and mark the area corresponding to the bottom-level item object as an ungraspable area; and/or”, ¶ 0016, “Based on the exposure ratio of each item, areas corresponding to item objects with exposure ratios greater than a preset threshold are marked as crawlable areas, while areas corresponding to item objects with exposure ratios not greater than the preset threshold are marked as uncrawlable areas; and/or,”, ¶ 0017, “Based on the shape and/or type of each item, determine the contact area contained in each item object, mark the area corresponding to the item object whose contact area is not occluded as the grabbable area, and mark the area corresponding to the item object whose contact area is occluded as the non-grabable area.”, ¶ 0083, “First, sample images corresponding to the three-dimensional sample area are acquired to identify multiple object objects contained in the sample images. The three-dimensional sample area contains multiple items to be grabbed, which serve as samples. The sample image corresponding to the three-dimensional sample region includes: a two-dimensional color image corresponding to the three-dimensional sample region obtained along a preset depth direction, and a depth image corresponding to the two-dimensional color image. For specific methods of obtaining the information, please refer to the corresponding description in step S110, which will not be repeated here. In determining multiple object objects contained in a sample image, the contours, boundary lines, and other information between each object can be identified through instance segmentation, and then the multiple object objects contained in the sample image can be segmented based on the recognition results.”, ¶ 0086, “In another alternative implementation, based on the exposure ratio of each item object, the area corresponding to the item object with an exposure ratio greater than a preset threshold is marked as a crawlable area, and the area corresponding to the item object with an exposure ratio not greater than the preset threshold is marked as an uncrawlable area. In some scenarios, the stacking relationship between items is not easy to determine, and items on the same layer may overlap each other. In such cases, it is difficult to accurately label the top item. For example, in supermarket product picking scenarios, the hierarchical relationship between products is not clear, and there may be overlapping between items on the same layer. Therefore, the requirements for the picking order are not strict, but the distinction between pickable and non-pickable items is more stringent. In this case, items with little surface exposure, or items that may cause other items in the scene to fly out after being grabbed, should not be marked as grabbable items. Accordingly, in the above scenario, a labeling threshold can be set, such as 85%. If an item's exposed surface area is greater than 85%, it is marked as a graspable item; if an item's exposed surface area is less than 85%, it is marked as a non-grabable item. Of course, the exposure ratio of an item can be quantified not only by the exposed surface area but also by the exposed volume. This invention does not limit the specific details.”, ¶ 0087, “In another alternative implementation, the contact area contained in each item object is determined according to the shape and/or type of each item object. The area corresponding to the item object whose contact area is not occluded is marked as a grabbable area, and the area corresponding to the item object whose contact area is occluded is marked as a non-grabable area. The contact area refers to the force-bearing area of the object that is easy to grasp. For example, for items with metal parts, in order to prevent damage to the parts, it is necessary to grasp a specific area of the metal parts. This specific area is the contact area, which is usually a relatively firm and not easily detached area of the metal parts. Therefore, when labeling such items, it is also necessary to determine whether the contact area is completely exposed and unobstructed. If an item has a large exposed surface area but the contact area is obstructed, it should still be marked as an ungrabable item.”. Zhu: ¶ 0031, “Furthermore, the step of determining the first endpoint and the second endpoint based on the geometric center and the separation position includes the following steps:”, ¶ 0032, “Select one segment of the third pixel line segment that is divided by the geometric center as the first line segment;”, ¶ 0033, “The first line segment is rotated several times along the geometric center at a first preset angle until the first line segment returns to its initial position, and the length of the second line segment is calculated during each first rotation, wherein the length of the second line segment is the length between the object edge on the first line segment and the geometric center;”, ¶ 0034, “Determine two adjacent second line segments whose sum of lengths is the smallest, and rotate one of the second line segments several times along the geometric center toward the other second line segment at a second preset angle, and calculate the length of the third line segment during each second rotation, where the length of the third line segment is the length between the object edge on the second line segment and the geometric center.”, ¶ 0035, “Take the endpoint of the third line segment with the shortest length as the first endpoint, and determine the second endpoint;”. The cited passages of both references clearly show that the grasping sequence is determined based on the various defined priorities and depth information.);
S42: determining a target grabbing sequence for each of the workpieces according to the first priority, the second priority and the third priority of each of the workpieces (Ito: Column 7 line 64 – Colum 8 line 9, “The result of priority setting based on the distances is used to determine, if there exist a plurality of candidate works in one block, the picking sequence of the plurality of candidate works. Priorities among a plurality of candidate works of different blocks comply with priorities assigned to the blocks in which the respective candidate works are positioned, as in the example of the first embodiment. Consequently, in the example of FIG. 7, the picking sequence of the respective candidate works is set to "first candidate work 411[Wingdings font/0xE0]second candidate work 412[Wingdings font/0xE0]third candidate work 413[Wingdings font/0xE0]sixth candidate work 416[Wingdings font/0xE0]fifth candidate work 415[Wingdings font/0xE0]fourth candidate work 414[Wingdings font/0xE0]seventh candidate work 417".”. Duan: ¶ 0077, “Step S140: Sort each graspable object along a preset depth direction according to the three-dimensional pose information, and determine the grasping order of each graspable object according to the sorting result.”, ¶ 0083, ¶ 0086, ¶ 0087. The cited passages clearly teach that the grasping order is determined based on the various priotites.); and
S43: determining a product model according to the contour curve of the workpiece (Hayashi: Column 6 lines 43-48, “At step S190, if the first point cloud generation part 271 stops the process, the second point cloud generation part 272 continues execution of the second point cloud generation process. Then, at step S230, the object detection execution part 273 detects the three-dimensional shapes of the parts PP as objects using the second point cloud PG2.”, Column 6 lines 49-63, “As the method of detecting the three-dimensional shapes of the objects using the point cloud, e.g. the method described in JP-A-2017-182274 disclosed by the applicant of this application can be used. Or, the method described in "Model Globally, Match Locally: Efficient and Robust 3D Object Recognition", Bertram Drost et al., http://campar.in-.tum.de/pub/drost20l 0CVPR/drost201 0CVPR.pdf, the method described in "Bekutoru Pea Mattingu niyoru Kousinraina Sanjigen Ichi Shisei Ninshiki", Shuichi Akizuki, http://isl.sist.chukyou.ac.jp/Archives/vpm.pdf, or the like may be used. These methods are object detection methods using the point cloud and the three-dimensional model data of the objects. Note that the three-dimensional shapes of the objects may be detected from the point cloud without using the three-dimensional model data.”),
and grabbing the workpieces according to the target grabbing sequence (Ito: Column 3 lines 22-46, “In accordance with a command from a processing apparatus 200, the measurement controller 110 for controlling the three-dimensional measurement apparatus 100 causes the light projector 120 to project pattern light on a pallet 400, and also causes the image capture device 130 to capture an area in the pallet 400 (container). The processing apparatus 200 is, for example, a computer apparatus. Note that the processing apparatus 200 may also function as the measurement controller 110. In the pallet 400, a number of works as picking targets of the picking system and as measurement targets of the three-dimensional measurement apparatus 100 are piled. The measurement controller 110 transmits the image capturing result of the image capture device 130 to the processing apparatus 200. The processing apparatus 200 recognizes the works in the pallet 400 and obtains position and orientation information based on data of an image captured by the image capture device 130, and selects a pickable candidate work.”, Column 6 lines 18-47, “The three-dimensional measurement apparatus 100 transfers the obtained captured data to the processing apparatus 200, as needed. The processing apparatus 200 performs the three-dimensional measurement calculation of the works based on the captured data, the by obtaining the positions and orientations of the works. The processing apparatus 200 transfers the information to the robot 300, and the robot 300 performs a picking operation based on the information.”, Column 7 line 64 – Colum 8 line 9, “The result of priority setting based on the distances is used to determine, if there exist a plurality of candidate works in one block, the picking sequence of the plurality of candidate works. Priorities among a plurality of candidate works of different blocks comply with priorities assigned to the blocks in which the respective candidate works are positioned, as in the example of the first embodiment. Consequently, in the example of FIG. 7, the picking sequence of the respective candidate works is set to "first candidate work 411[Wingdings font/0xE0]second candidate work 412[Wingdings font/0xE0]third candidate work 413[Wingdings font/0xE0]sixth candidate work 416[Wingdings font/0xE0]fifth candidate work 415[Wingdings font/0xE0]fourth candidate work 414[Wingdings font/0xE0]seventh candidate work 417".”. The system is clearly configured to grasp the objects according to the determined sequence. ).
Regarding claim 5, Ito in view of Hayashi in further view of Duan in further view of Zhu teaches wherein in S41: the deeper the workpiece is, the lower the priority is (Duan: ¶ 0085, “In one optional implementation, the stacking order of each item object along a preset depth direction is determined, the area corresponding to the item object at the top level is marked as a grabbable area, and the area corresponding to the item object at the bottom level is marked as an ungrabable area. Conventional instance segmentation algorithms do not distinguish whether items in the scene are grabbable; that is, they require a complete and accurate instance mask for all items in the scene. Therefore, if traditional instance segmentation algorithms are directly applied to the identification of graspable areas, it will cause the crushed items at the bottom to be identified as graspable items or the irrelevant background items to be identified as graspable items, thus causing identification errors. To prevent the aforementioned problems, this implementation determines the stacking order of each item object along a preset depth direction, thereby marking the area corresponding to the top-level item object as a graspable area and the area corresponding to the bottom-level item object as an ungraspable area, thus avoiding anomalies caused by the robot grasping the bottom-level items. For example, in a cardboard box destacking scenario, the destacking should proceed layer by layer from the top to the bottom, and the lower cardboard boxes should not be grabbed before the lower cardboard boxes have been completely grabbed. Therefore, in similar scenarios, only the topmost cardboard box is marked as a grabbable object, while all other cardboard boxes are marked as non-grabable objects. This labeling method can accurately distinguish between items on the top layer and those not on the top layer, thus providing accurate pixel-level item positioning.”. The cited passage clearly shows that the deeper the object, the lower the priority.);
the workpiece in the natural putting state has a higher priority than the workpiece in stacking state (Duan: ¶ 0086, “In another alternative implementation, based on the exposure ratio of each item object, the area corresponding to the item object with an exposure ratio greater than a preset threshold is marked as a crawlable area, and the area corresponding to the item object with an exposure ratio not greater than the preset threshold is marked as an uncrawlable area. In some scenarios, the stacking relationship between items is not easy to determine, and items on the same layer may overlap each other. In such cases, it is difficult to accurately label the top item. For example, in supermarket product picking scenarios, the hierarchical relationship between products is not clear, and there may be overlapping between items on the same layer. Therefore, the requirements for the picking order are not strict, but the distinction between pickable and non-pickable items is more stringent. In this case, items with little surface exposure, or items that may cause other items in the scene to fly out after being grabbed, should not be marked as grabbable items. Accordingly, in the above scenario, a labeling threshold can be set, such as 85%. If an item's exposed surface area is greater than 85%, it is marked as a graspable item; if an item's exposed surface area is less than 85%, it is marked as a non-grabable item. Of course, the exposure ratio of an item can be quantified not only by the exposed surface area but also by the exposed volume. This invention does not limit the specific details.”. One of ordinary skill in the art would recognize that objects that have an exposed surface area greater than the threshold do not have an object(s) stacked on top of it and are therefore grabbable (i.e. a natural putting state as no objects are on top of it). Objects have an exposed surface area less than the threshold have an object(s) stacked on top of it and are therefore not grabbable (i.e. a stacking state as objects are on top of it). Additionally, it is obvious that the objects with an exposed surface area above the threshold has a higher priority than those that do not.);
and the greater a numerical value of the second stacking parameter is, the higher the priority is (Duan: ¶ 0087, “In another alternative implementation, the contact area contained in each item object is determined according to the shape and/or type of each item object. The area corresponding to the item object whose contact area is not occluded is marked as a grabbable area, and the area corresponding to the item object whose contact area is occluded is marked as a non-grabable area. The contact area refers to the force-bearing area of the object that is easy to grasp. For example, for items with metal parts, in order to prevent damage to the parts, it is necessary to grasp a specific area of the metal parts. This specific area is the contact area, which is usually a relatively firm and not easily detached area of the metal parts. Therefore, when labeling such items, it is also necessary to determine whether the contact area is completely exposed and unobstructed. If an item has a large exposed surface area but the contact area is obstructed, it should still be marked as an ungrabable item.”. Zhu: ¶ 0031, “Furthermore, the step of determining the first endpoint and the second endpoint based on the geometric center and the separation position includes the following steps:”, ¶ 0032, “Select one segment of the third pixel line segment that is divided by the geometric center as the first line segment;”, ¶ 0033, “The first line segment is rotated several times along the geometric center at a first preset angle until the first line segment returns to its initial position, and the length of the second line segment is calculated during each first rotation, wherein the length of the second line segment is the length between the object edge on the first line segment and the geometric center;”, ¶ 0034, “Determine two adjacent second line segments whose sum of lengths is the smallest, and rotate one of the second line segments several times along the geometric center toward the other second line segment at a second preset angle, and calculate the length of the third line segment during each second rotation, where the length of the third line segment is the length between the object edge on the second line segment and the geometric center.”, ¶ 0035, “Take the endpoint of the third line segment with the shortest length as the first endpoint, and determine the second endpoint;”. One of ordinary skill in the art would recognize that the higher the value of this parameter, the higher the priority.);
and the first priority, the second priority and the third priority are sequentially decreased (Duan: ¶ 0015, “Determine the stacking order of each item object along the preset depth direction, mark the area corresponding to the top-level item object as a graspable area, and mark the area corresponding to the bottom-level item object as an ungraspable area; and/or”, ¶ 0016, “Based on the exposure ratio of each item, areas corresponding to item objects with exposure ratios greater than a preset threshold are marked as crawlable areas, while areas corresponding to item objects with exposure ratios not greater than the preset threshold are marked as uncrawlable areas; and/or,”, ¶ 0017, “Based on the shape and/or type of each item, determine the contact area contained in each item object, mark the area corresponding to the item object whose contact area is not occluded as the grabbable area, and mark the area corresponding to the item object whose contact area is occluded as the non-grabable area.”, ¶ 0083, ¶ 0085, ¶ 0086, ¶ 0087. It is obvious from the cited passages that the priorities decrease sequentially in the order they are listed, i.e. objects at the top of the pile are, in an unstacked state (i.e. a natural placing state), and have the largest smallest distance from its center point are given the highest priority, whereas objects at the top of the pile are, in an unstacked state (i.e. a natural placing state), and have the smallest distance from its center point are given a lower priority.).
Regarding claim 6, Ito in view of Hayashi in further view of Duan in further view of Zhu teaches wherein S1 comprises: S11: projecting multiple sets of gratings with different phases to an area to be sorted in the visual field range (Hayashi: Column 5 lines 26-41, “In the second point cloud generation process, first, a second imaging process of imaging a plurality of phase shifting images PSM1 to PSM4 containing the parts PP with a plurality of phase shifting patterns PH1 to PH4 selectively projected one by one as the projection patterns is performed using the projection device 810. The plurality of phase shifting patterns PH1 to PH4 are sinusoidal banded patterns having dark and light parts. In the phase shifting method, generally, capturing of images is performed using the n phase shift patterns PH2 to PHn for n (integer) equal to or larger than three. The n phase shift patterns PH1 to PHn are sinusoidal patterns having phase sequentially shifted by 2π/n. In the example of FIG. 7, n=4. Note that, in the phase shifting method, it is not necessary to use the stereo camera, and only one of the two cameras forming the stereo camera may be used for imaging.”, Column 5 lines 42-52, “In the second point cloud generation process, then, a second analysis process is executed using the plurality of phase shifting images PSM1 to PSM4, and thereby, a second point cloud PG2 is generated. In the second analysis process, first, a distance image DMZ is generated according to the phase shifting method using the plurality of phase shifting images PSM1 to PSM4. The generation process of the distance image DMZ using the phase shifting method is well known, and the explanation thereof is omitted here. In the second point cloud generation process, a second point cloud PG2 is further generated using the distance image DMZ.”),
and acquiring corresponding grating images (Hayashi: Column 5 lines 26-41, Column 5 lines 42-52); and
S12: identifying the grating images ((Hayashi: Column 5 lines 26-41, Column 5 lines 42-52)),
calculating coordinates of each feature point in the grating images by a triangulation location method, and integrating the coordinates to obtain the point cloud set information (Hayashi: Column 5 lines 26-41, Column 5 lines 42-52. The cited passage clearly states that the point cloud is generated based on the multiple grating images generated by projecting a pattern at different phases according to methods known in the art. One of ordinary skill in the art would have recognized that feature based methods of image processing are commonly used to reduce computational loads and had the capability to perform such a method. Additionally feature based imaging techniques are known in the art and would be considered part of the known methods referred to in Hayashi.).
Regarding claim 8, Ito in view of Hayashi in further view of Duan in further view of Zhu teaches wherein in S43, the grabbing the workpieces according to the target grabbing sequence, comprises: determining the posture matrix of a current target grabbing workpiece according to the target grabbing sequence, (Ito: Column 3 lines 22-46, “In accordance with a command from a processing apparatus 200, the measurement controller 110 for controlling the three-dimensional measurement apparatus 100 causes the light projector 120 to project pattern light on a pallet 400, and also causes the image capture device 130 to capture an area in the pallet 400 (container). The processing apparatus 200 is, for example, a computer apparatus. Note that the processing apparatus 200 may also function as the measurement controller 110. In the pallet 400, a number of works as picking targets of the picking system and as measurement targets of the three-dimensional measurement apparatus 100 are piled. The measurement controller 110 transmits the image capturing result of the image capture device 130 to the processing apparatus 200. The processing apparatus 200 recognizes the works in the pallet 400 and obtains position and orientation information based on data of an image captured by the image capture device 130, and selects a pickable candidate work.”, Column 6 lines 18-47, “The three-dimensional measurement apparatus 100 transfers the obtained captured data to the processing apparatus 200, as needed. The processing apparatus 200 performs the three-dimensional measurement calculation of the works based on the captured data, the by obtaining the positions and orientations of the works. The processing apparatus 200 transfers the information to the robot 300, and the robot 300 performs a picking operation based on the information.”)
converting the posture matrix into target grabbing coordinates in a manipulator coordinate system in combination with a hand-eye calibration matrix (Zhu: ¶ 0112, “Step S61: Based on the grasping pose, obtain the corresponding grasping pose parameters in the camera coordinate system of the camera; in this embodiment, the camera can be a camera that captures preset images, such as an RGBD camera.”, ¶ 0114, “Step S62: Obtain the preset rotation and translation matrix between the camera coordinate system and the robot arm coordinate system;”, ¶ 0115, “Specifically, the camera coordinate system and the robot arm coordinate system can be calibrated by hand and eye using the nine-point calibration method to obtain and save the preset rotation and translation matrix;”, ¶ 0116, “Step S63: Based on the grasping pose parameters and the preset rotation and translation matrix, the robot arm performs a grasping operation on the object.”),
and sending the target grabbing coordinates to a manipulator (Ito: Column 6 lines 18-47, “The three-dimensional measurement apparatus 100 transfers the obtained captured data to the processing apparatus 200, as needed. The processing apparatus 200 performs the three-dimensional measurement calculation of the works based on the captured data, the by obtaining the positions and orientations of the works. The processing apparatus 200 transfers the information to the robot 300, and the robot 300 performs a picking operation based on the information.”, Zhu: ¶ 0116, “Step S63: Based on the grasping pose parameters and the preset rotation and translation matrix, the robot arm performs a grasping operation on the object.”);
and grabbing the target grabbing workpiece according to the target grabbing coordinates by the manipulator (Ito: Column 6 lines 18-47, “The three-dimensional measurement apparatus 100 transfers the obtained captured data to the processing apparatus 200, as needed. The processing apparatus 200 performs the three-dimensional measurement calculation of the works based on the captured data, the by obtaining the positions and orientations of the works. The processing apparatus 200 transfers the information to the robot 300, and the robot 300 performs a picking operation based on the information.”, Column 7 line 64 – Colum 8 line 9, “The result of priority setting based on the distances is used to determine, if there exist a plurality of candidate works in one block, the picking sequence of the plurality of candidate works. Priorities among a plurality of candidate works of different blocks comply with priorities assigned to the blocks in which the respective candidate works are positioned, as in the example of the first embodiment. Consequently, in the example of FIG. 7, the picking sequence of the respective candidate works is set to "first candidate work 411[Wingdings font/0xE0]second candidate work 412[Wingdings font/0xE0]third candidate work 413[Wingdings font/0xE0]sixth candidate work 416[Wingdings font/0xE0]fifth candidate work 415[Wingdings font/0xE0]fourth candidate work 414[Wingdings font/0xE0]seventh candidate work 417".”).
Regarding claim 10, Ito in view of Hayashi in further view of Duan in further view of Zhu teaches a non-transitory computer-readable storage medium storing a computer program thereon, wherein the computer program is used to realize the robot sorting method based on visual recognition according to claim 1 (Ito: Column 8 line 42 – Column 9 line 3, “Embodiment(s) of the present invention can also be realized by a computer of a system or apparatus that reads out and executes computer executable instructions (e.g., one or more programs) recorded on a storage medium (which may also be referred to more fully as a 'non-transitory computer-readable storage medium') to perform the functions of one or more of the above-described embodiment(s) and/or that includes one or more circuits (e.g., application specific integrated circuit (ASIC)) for performing the functions of one or more of the above-described embodiment(s), and by a method performed by the computer of the system or apparatus by, for example, reading out and executing the computer executable instructions from the storage medium to perform the functions of one or more of the above described embodiment(s) and/or controlling the one or more circuits to perform the functions of one or more of the above-described embodiment(s). The computer may comprise one or more processors (e.g., central processing unit (CPU), micro processing unit (MPU)) and may include a network of separate computers or separate processors to read out and execute the computer executable instructions. The computer executable instructions may be provided to the computer, for example, from a network or the storage medium. The storage medium may include, for example, one or more of a hard disk, a random-access memory (RAM), a read only memory (ROM), a storage of distributed computing systems, an optical disk (such as a compact disc (CD), digital versatile disc (DVD), or Blu-ray Disc (BD)™), a flash memory device, a memory card, and the like.”).
Claim(s) 7 is/are rejected under 35 U.S.C. 103 as being unpatentable over US 10040199 B2 ("Ito") in view of US 11126844 B2 ("Hayashi") in further view of CN 112802105 A ("Duan") in further view of CN 111462232 A ("Zhu") in further view of US 2020/0194117 A1 ("Krieger").
Regarding claim 7, Ito in view of Hayashi in further view of Duan in further view of Zhu teaches wherein S2 comprises: S21: setting the grabbing space according to a shape and a structure of an area to be sorted (Ito: Column 4 line 58 – Column 5 line 10, “The procedure of assigning the priorities of the blocks, which has been explained, will be described with reference to a flowchart shown in FIG. 4. The priorities of the blocks may be assigned using an apparatus outside the picking system. In this embodiment, assume that the processing apparatus 200 performs an operation of assigning the priorities of the blocks. When the operation of assigning the example of FIG. 2, the area in the pallet 400 is equally divided into 12 rectangular blocks in the vertical and horizontal directions. In step S12, the processing apparatus 200 obtains the orientation of the robot 300, when viewed from the three-dimensional measurement apparatus 100, at the time of picking of a work in each block. The orientation of the robot 300 may be obtained by actual image capturing by the three-dimensional measurement apparatus 100, or by simulation using a three-dimensional CAD model or the like.”, Column 5 lines 22-35, “In step S13, based on the orientation of the robot 300 obtained in step S12, the processing apparatus 200 assigns the priorities of all the blocks by setting a higher priority to a block which prevents the robot from shielding the works in the remaining blocks as much as possible. At this time, criteria for assigning the priorities of the blocks may include setting criteria such as the distance to a conveyance destination of a work in addition to a criterion that the works in the remaining blocks are not shielded as much as possible. As shown in FIG. by assigning the priorities of the blocks 4, before measurement, for example, at the time of installation of the three-dimensional measurement apparatus, it is possible to quickly pick a candidate work while preventing the robot from shielding the remaining candidate works as much as possible at the time of picking.”. The cited passages clearly shows that the system is configured to set the grabbing space according to the area of the pallet.);
S22: screening the point cloud set information according to coordinate information of the grabbing space to obtain a target point cloud set (Ito: Column 4 line 58 – Column 5 line 10, “The procedure of assigning the priorities of the blocks, which has been explained, will be described with reference to a flowchart shown in FIG. 4. The priorities of the blocks may be assigned using an apparatus outside the picking system. In this embodiment, assume that the processing apparatus 200 performs an operation of assigning the priorities of the blocks. When the operation of assigning the example of FIG. 2, the area in the pallet 400 is equally divided into 12 rectangular blocks in the vertical and horizontal directions. In step S12, the processing apparatus 200 obtains the orientation of the robot 300, when viewed from the three-dimensional measurement apparatus 100, at the time of picking of a work in each block. The orientation of the robot 300 may be obtained by actual image capturing by the three-dimensional measurement apparatus 100, or by simulation using a three-dimensional CAD model or the like.”. Hayashi: Column 5 lines 26-41, “In the second point cloud generation process, first, a second imaging process of imaging a plurality of phase shifting images PSM1 to PSM4 containing the parts PP with a plurality of phase shifting patterns PH1 to PH4 selectively projected one by one as the projection patterns is performed using the projection device 810. The plurality of phase shifting patterns PH1 to PH4 are sinusoidal banded patterns having dark and light parts. In the phase shifting method, generally, capturing of images is performed using the n phase shift patterns PH2 to PHn for n (integer) equal to or larger than three. The n phase shift patterns PH1 to PHn are sinusoidal patterns having phase sequentially shifted by 2π/n. In the example of FIG. 7, n=4. Note that, in the phase shifting method, it is not necessary to use the stereo camera, and only one of the two cameras forming the stereo camera may be used for imaging.”, Column 5 lines 42-52, “In the second point cloud generation process, then, a second analysis process is executed using the plurality of phase shifting images PSM1 to PSM4, and thereby, a second point cloud PG2 is generated. In the second analysis process, first, a distance image DMZ is generated according to the phase shifting method using the plurality of phase shifting images PSM1 to PSM4. The generation process of the distance image DMZ using the phase shifting method is well known, and the explanation thereof is omitted here. In the second point cloud generation process, a second point cloud PG2 is further generated using the distance image DMZ.”. The combination of Ito iv view of Hayashi clearly shows that the point cloud is screened to determine a target point cloud.); and
determining each visible workpiece in the grabbing space (Duan: ¶ 0086, “In another alternative implementation, based on the exposure ratio of each item object, the area corresponding to the item object with an exposure ratio greater than a preset threshold is marked as a crawlable area, and the area corresponding to the item object with an exposure ratio not greater than the preset threshold is marked as an uncrawlable area. In some scenarios, the stacking relationship between items is not easy to determine, and items on the same layer may overlap each other. In such cases, it is difficult to accurately label the top item. For example, in supermarket product picking scenarios, the hierarchical relationship between products is not clear, and there may be overlapping between items on the same layer. Therefore, the requirements for the picking order are not strict, but the distinction between pickable and non-pickable items is more stringent. In this case, items with little surface exposure, or items that may cause other items in the scene to fly out after being grabbed, should not be marked as grabbable items. Accordingly, in the above scenario, a labeling threshold can be set, such as 85%. If an item's exposed surface area is greater than 85%, it is marked as a graspable item; if an item's exposed surface area is less than 85%, it is marked as a non-grabable item. Of course, the exposure ratio of an item can be quantified not only by the exposed surface area but also by the exposed volume. This invention does not limit the specific details.”. The method is clearly configured to determine what objects are visible.),
and calculating the posture matrix of each visible workpiece in a camera coordinate system (Ito: Column 3 lines 22-46, “In accordance with a command from a processing apparatus 200, the measurement controller 110 for controlling the three-dimensional measurement apparatus 100 causes the light projector 120 to project pattern light on a pallet 400, and also causes the image capture device 130 to capture an area in the pallet 400 (container). The processing apparatus 200 is, for example, a computer apparatus. Note that the processing apparatus 200 may also function as the measurement controller 110. In the pallet 400, a number of works as picking targets of the picking system and as measurement targets of the three-dimensional measurement apparatus 100 are piled. The measurement controller 110 transmits the image capturing result of the image capture device 130 to the processing apparatus 200. The processing apparatus 200 recognizes the works in the pallet 400 and obtains position and orientation information based on data of an image captured by the image capture device 130, and selects a pickable candidate work.”, Column 6 lines 18-47, “The three-dimensional measurement apparatus 100 transfers the obtained captured data to the processing apparatus 200, as needed. The processing apparatus 200 performs the three-dimensional measurement calculation of the works based on the captured data, the by obtaining the positions and orientations of the works.”).
Ito in view of Hayashi in further view of Duan in further view of Zhu does not teach S23: performing outlier filtration on the target point cloud set,
Krieger, in the same field of endeavor, teaches S23: performing outlier filtration on the target point cloud set (Krieger: ¶ 0098, “After acquiring camera images 240, the flow 234 can include generating a 3D point cloud 242. The previously acquired images from the mobile camera 116 (e.g., camera images 240) can be used to generate the 3D point cloud 242. For example, the mobile platform 110 (or the computing device 106) can perform a point cloud registration using the 3D images (depicting multiple scenes) to construct a 3D model using any suitable technique or combinations of techniques. For example, the mobile platform 110 can use the Iterative Closest Point (“ICP”) algorithm. In such an example, the ICP algorithm can be implemented using the point cloud library (“PCL”), and pointmatcher, and can include prior noise removal with a PCL passthrough filter. In some embodiments, a point cloud color segmentation can be applied to extract a point cloud corresponding to the patient from the reconstructed scene, for example by removing background items such as a table and a supporting frame.”. The cited passage clearly teaches a method of filtering noise in a point cloud.).
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filling date of the claimed invention, to have combine the robot sorting method taught in Ito in view of Hayashi in further view of Duan in further view of Zhu with S23: performing outlier filtration on the target point cloud set taught in Krieger with a reasonable expectation of success. One of ordinary skill in the art would have been motivated to make this modification because the combination would have required the simple addition of a known algorithm to a readily configurable system. The system taught in Ito in view of Hayashi in further view of Duan in further view of Zhu already teaches determining a point cloud. As such the filtering method taught in Krieger can be simply added to the system taught in Ito in view of Hayashi in further view of Duan in further view of Zhu according to methods known in the art. Such a modification would not change or introduce new functionality. No inventive effort would have been required.
Claim(s) 9 is/are rejected under 35 U.S.C. 103 as being unpatentable over US 10040199 B2 ("Ito") in view of US 11126844 B2 ("Hayashi") in further view of CN 112802105 A ("Duan") in further view of CN 111462232 A ("Zhu") in further view of US 2023/0125022 A1 ("Li").
Regarding claim 9, Ito in view of Hayashi in further view of Duan in further view of Zhu does not teach wherein the manipulator is a magnetic suction manipulator.
Li, in the same field of endeavor, teaches wherein the manipulator is a magnetic suction manipulator (Li: ¶ 0033, “The picking hand 21 may have any configuration that can hold the workpieces W one by one. As an example, the picking hand 21 may have a suction pad 211 for suctioning a workpiece W, as illustrated in FIG. 1. In this way, the picking hand 21 may be a suction hand for suctioning a workpiece using air tightness, but may be an attraction hand with a strong attraction force which does not requires air tightness. The picking hand 21 may have a pair of gripping fingers 212 or three or more gripping fingers 212 for pinching and holding a workpiece W as an alternative enclosed by the two-dot chain line in FIG. 1, or may have a plurality of suction pads 211 (not illustrated). Alternatively, the picking hand 21 may have a magnetic hand (not illustrated) configured to hold a workpiece made of iron or the like with a magnetic force.”).
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filling date of the claimed invention, to have combine the robot sorting method taught in Ito in view of Hayashi in further view of Duan in further view of Zhu in further view of Zhu with wherein the manipulator is a magnetic suction manipulator taught in Li with a reasonable expectation of success. One of ordinary skill in the art would have been motivated to make this modification because the modification would have required the simple substitution of a known manipulator for another. A person of ordinary skill in the art would have had the technological capabilities to have modified the robot taught in Ito in view of Hayashi in further view of Duan in further view of Zhu in further view of Zhu with a magnetic manipulator as taught in Li. Additionally such a manipulator would have been known to one of ordinary skill in the art. The manipulator of the robot taught in Ito in view of Hayashi in further view of Duan in further view of Zhu in further view of Zhu can be easily substituted for the magnetic manipulator taught in Li according to methods known in the art. Such a modification would not have changed or introduced new functionality. No inventive effort would have been required.
Response to Arguments
Applicant’s arguments, see Page 6, filed March 23rd, 2026, with respect to the 35 U.S.C § 101 rejection of claims 1-3, 6-7, and 10 have been fully considered and are persuasive. The independent claim 1 has been amended to recite the limitation “and grabbing the workpieces according to the grabbing sequence”. Such a limitation is clearly an active control step of the system using the information generated by the abstract idea. Such a limitation is clearly indicative of integration into a practical application. Therefore, the 35 U.S.C § 101 rejection of claims 1-3, 6-7, and 10 has been withdrawn.
Applicant's arguments filed March 23rd, 2026, have been fully considered but they are not persuasive.
Regarding Applicant’s arguments on ages 6-14, Applicant argues that the prior art on record does not teach the limitations of the amended independent claim 1.
Specifically on Pages 6-9, Applicant argues that the primary reference Ito does not teach the limitations “performing recognition and fusion to obtain point cloud set information” and “S2: acquiring a posture matrix of every workpiece in a grabbing space from the point cloud set information”. The Examiner respectfully disagrees. In response to applicant's arguments against the references individually, one cannot show nonobviousness by attacking references individually where the rejections are based on combinations of references. See In re Keller, 642 F.2d 413, 208 USPQ 871 (CCPA 1981); In re Merck & Co., 800 F.2d 1091, 231 USPQ 375 (Fed. Cir. 1986). As was stated in the previous Non-Final Office Action mailed December 23rd, 2025 and above in the 35 U.S.C. § 103 rejection section, the primary reference Ito teaches a robot sorting method based on visual recognition, comprising (Ito: Figure 1, Abstract, Column 3 lines 22-46): S1: acquiring a plurality of grating images in a visual field range (Ito: Column 3 lines 22-46, Column 5 lines 26-49), S2: acquiring a posture matrix of every workpiece in a grabbing space from the point cloud set information (Column 3 lines 22-46, Column 6 lines 18-47); and determining a target grabbing sequence (Ito: Column 7 line 64 – Colum 8 line 9), and grabbing the workpieces according to the grabbing sequence (Ito: Column 3 lines 22-46, Column 6 lines 18-47, Column 7 line 64 – Colum 8 line 9). Ito clearly teaches a robotic system that is configured to capture a plurality of grating images in a visual range. The system then uses these grating images to determine the position and orientation (i.e. the pose) of each object in the container. Using these pose matrices. The robot then determines a target grabbing sequence and controls the robot to grasp the objects according to the sequence. Furthermore, as is clear from the previous Non-Final Office Action mailed December 23rd, 2025 and above in the 35 U.S.C. § 103 rejection section, that Ito was not relied upon to teach the limitation “performing recognition and fusion to obtain point cloud set information”. The secondary reference Hayashi was relied upon to teach performing recognition and fusion to obtain point cloud set information (Hayashi: Column 5 lines 26-41, Column 5 lines 42-52). The cited passages of Hayashi clearly shows that Hayashi, which relates to a robotic system configured to gasp objects, teaches generating a point cloud by capturing a plurality of grating images and performing a recognition and fusion process to generate said point cloud. Additionally, Applicant argues that Ito does not teach the above limitations because Ito selects, from all workpieces, only those that are completely unshielded by stacking as pickable candidate workpieces, and then assigns priorities to the candidate workpieces based on the predefined block division information of the pallet in which the candidate workpieces are located, and performs picking according to the priorities, whereas the present invention performs sequencing of all workpieces visible in the image (including workpieces that are partially shielded due to stacking), whereas Ito performs sequencing only on workpieces that are completely unshielded by stacking. In response to applicant's argument that the references fail to show certain features of the invention, it is noted that the features upon which applicant relies (i.e., that the present invention performs sequencing of all workpieces visible in the image (including workpieces that are partially shielded due to stacking), whereas Ito performs sequencing only on workpieces that are completely unshielded by stacking.) are not recited in the rejected claim(s). The claims only robot sorting method based on visual recognition, comprising: Si: acquiring a plurality of grating images in a visual field range, and performing recognition and fusion to obtain point cloud set information; S2: acquiring a posture matrix of every workpiece in a grabbing space from the point cloud set information; S3: acquiring contour feature data of each of the workpieces according to the posture matrix, the contour feature data includes depth information, a first stacking parameter, and a second stacking parameter, wherein the first stacking parameter is a placing state of a workpiece and the second stacking parameter is the smallest one of the distances between a geometric center of one of the workpieces and each of workpiece edges of the workpieces; and S4: comparing the contour feature data of the workpieces, determining a target grabbing sequence, and grabbing the workpieces according to the target grabbing sequence. None of the dependent claims teach or suggest that the system performs sequencing of all workpieces, even the ones partially shielded. Although the claims are interpreted in light of the specification, limitations from the specification are not read into the claims. See In re Van Geuns, 988 F.2d 1181, 26 USPQ2d 1057 (Fed. Cir. 1993). Lastly, on Page 8, Applicant stipulates that Ito does disclose acquiring images, obtaining the pose of workpieces based on the images, and picking the workpieces according to priority. Therefore, the combination of Ito in view of Hayashi in further view of Duan in further view of Zhu teaches the limitations “performing recognition and fusion to obtain point cloud set information” and “S2: acquiring a posture matrix of every workpiece in a grabbing space from the point cloud set information”.
Specifically on Pages 9-10, Applicant argues that the secondary reference Hayashi fails to teach the limitations “S2: acquiring a posture matrix of each of workpieces in a grabbing space from the point cloud set information”, “"S3: acquiring contour feature data of each of the workpieces according to the posture matrix, the contour feature data includes depth information, a first stacking parameter, and a second stacking parameter, wherein the first stacking parameter is a placing state of a workpiece and the second stacking parameter is the smallest one of the distances between a geometric center of one of the workpieces and each of workpiece edges of the workpieces;”, and “S4: comparing the contour feature data of the workpieces, determining a target grabbing sequence, and grabbing the workpieces according to the target grabbing sequence”. The Examiner respectfully disagrees. In response to applicant's arguments against the references individually, one cannot show nonobviousness by attacking references individually where the rejections are based on combinations of references. See In re Keller, 642 F.2d 413, 208 USPQ 871 (CCPA 1981); In re Merck & Co., 800 F.2d 1091, 231 USPQ 375 (Fed. Cir. 1986). As was stated in the previous Non-Final Office Action mailed December 23rd, 2025, above in the 35 U.S.C. § 103 rejection section, and above in response to Applicant’s prior arguments, Hayashi was not relied upon to teach the limitations “S2: acquiring a posture matrix of each of workpieces in a grabbing space from the point cloud set information”, “a first stacking parameter, and a second stacking parameter, wherein the first stacking parameter is a placing state of a workpiece and the second stacking parameter is the smallest one of the distances between a geometric center of one of the workpieces and each of workpiece edges of the workpieces;”, and “S4: comparing the contour feature data of the workpieces, determining a target grabbing sequence, and grabbing the workpieces according to the target grabbing sequence”. As was stated previously Ito teaches the limitations “S2: acquiring a posture matrix of each of workpieces in a grabbing space from the point cloud set information” and “determining a target grabbing sequence, and grabbing the workpieces according to the target grabbing sequence” a robot sorting method, wherein the robot is configured to capture a plurality of grating image of a container containing a plurality of objects. The robot the determines the pose of each object based on the grating images. Using these poses, the robot then determines a grabbing sequence of the objects and grabs the object according to the sequence. Applicant also stipulated that Ito teaches such on Page 8 of Applicant’s arguments. The secondary reference Hayashi was only relied on to teach and performing recognition and fusion to obtain point cloud set information (Hayashi: Column 5 lines 26-41, Column 5 lines 42-52); S3: acquiring contour feature data of each of the workpieces according to the posture matrix, the contour feature data includes depth information (Hayashi: Column 5 lines 26-41, Column 5 lines 42-52, Column 6 lines 49-63). The cited passages of Hayashi clearly shows that Hayashi is configured to generating a point cloud by capturing a plurality of grating images and performing a recognition and fusion process to generate said point cloud. The system then uses these point clouds to determine depth information as well as the 3D shapes of the objects (i.e. the contour information). The secondary reference Duan teaches a first stacking parameter (Duan: ¶ 0083, ¶ 0086), wherein the first stacking parameter is a placing state of a workpiece (Duan: ¶ 0083, ¶ 0086) and S4: comparing the contour feature data of the workpieces (Duan: ¶ 0083, ¶ 0086, ¶ 0087). Duan teaches a robot sorting method wherein the system is configured to capture images of a plurality of objects. The system uses these images to determine the contours, boundary lines, and other information the objects in the images. The system the determines based on the contour information, whether the objects are grabable or ungrabable. This is determined by determining how much of an object is occluded by other objects (i.e. if a significant portion of the object is covered by other objects) or if the area required to properly grasp the object is in accessible due to other robots blocking the robot from reaching the are required to properly grab the object. One of ordinary skill in the art would recognize that this method of determining if the objects are grabable or ungrabable is a method of determining a stacking state of the objects. The secondary reference Zhu teaches a second stacking parameter, and the second stacking parameter is the smallest one of the distances between a geometric center of one of the workpieces and each of workpiece edges of the workpieces (Zhu: ¶ 0010, ¶ 0014, ¶ 0025, ¶ 0026, ¶ 0027, ¶ 0028, ¶ 0029, ¶ 0031, ¶ 0032, ¶ 0033, ¶ 0035). The cited passages of Zhu clearly teaches that the geometric center of the object is determined. The system is the configured to determine a distance from the geometric center to the edges of the object and take the smallest of these distances. This information is then used to control the robot to grasp an object. One of ordinary skill in the art would recognize that this clearly teach the limitation “a second stacking parameter, and the second stacking parameter is the smallest one of the distances between a geometric center of one of the workpieces and each of workpiece edges of the workpieces”. Therefore, it is clear that the combination of Ito in view of Hayashi in further view of Duan in further view of Zhu teaches the limitations “S2: acquiring a posture matrix of each of workpieces in a grabbing space from the point cloud set information”, “"S3: acquiring contour feature data of each of the workpieces according to the posture matrix, the contour feature data includes depth information, a first stacking parameter, and a second stacking parameter, wherein the first stacking parameter is a placing state of a workpiece and the second stacking parameter is the smallest one of the distances between a geometric center of one of the workpieces and each of workpiece edges of the workpieces;”, and “S4: comparing the contour feature data of the workpieces, determining a target grabbing sequence, and grabbing the workpieces according to the target grabbing sequence”.
Specifically on Pages 10-12, Applicant argues that the secondary reference Duan does not teach the limitations “S2: acquiring a posture matrix of each of workpieces in a grabbing space from the point cloud set information”, “"S3: acquiring contour feature data of each of the workpieces according to the posture matrix, the contour feature data includes depth information, a first stacking parameter, and a second stacking parameter, wherein the first stacking parameter is a placing state of a workpiece and the second stacking parameter is the smallest one of the distances between a geometric center of one of the workpieces and each of workpiece edges of the workpieces;”, and “S4: comparing the contour feature data of the workpieces, determining a target grabbing sequence, and grabbing the workpieces according to the target grabbing sequence”. The Examiner respectfully disagrees. In response to applicant's arguments against the references individually, one cannot show nonobviousness by attacking references individually where the rejections are based on combinations of references. See In re Keller, 642 F.2d 413, 208 USPQ 871 (CCPA 1981); In re Merck & Co., 800 F.2d 1091, 231 USPQ 375 (Fed. Cir. 1986). As was stated in the previous Non-Final Office Action mailed December 23rd, 2025, above in the 35 U.S.C. § 103 rejection section, and above in response to Applicant’s prior arguments, Duan was not relied upon to teach the limitations “S2: acquiring a posture matrix of each of workpieces in a grabbing space from the point cloud set information”, “S3: acquiring contour feature data of each of the workpieces according to the posture matrix, the contour feature data includes depth information, a second stacking parameter, and the second stacking parameter is the smallest one of the distances between a geometric center of one of the workpieces and each of workpiece edges of the workpieces;”, and “determining a target grabbing sequence, and grabbing the workpieces according to the target grabbing sequence”. As was stated previously Ito teaches the limitations “S2: acquiring a posture matrix of each of workpieces in a grabbing space from the point cloud set information” and “determining a target grabbing sequence, and grabbing the workpieces according to the target grabbing sequence” a robot sorting method, wherein the robot is configured to capture a plurality of grating image of a container containing a plurality of objects. The robot the determines the pose of each object based on the grating images. Using these poses, the robot then determines a grabbing sequence of the objects and grabs the object according to the sequence. Applicant also stipulated that Ito teaches such on Page 8 of Applicant’s arguments. The secondary reference Hayashi teaches the limitations “S3: acquiring contour feature data of each of the workpieces according to the posture matrix, the contour feature data includes depth information”. Hayashi teaches a robotic grasping method wherein the system is configured to capture a plurality of grating images of objects. The system then preforms recognition and fusion on the plurality of grating images to generate a point cloud. The system then uses these point clouds to determine depth information as well as the 3D shapes of the objects (i.e. the contour information). The secondary reference Duan teaches a first stacking parameter (Duan: ¶ 0083, ¶ 0086), wherein the first stacking parameter is a placing state of a workpiece (Duan: ¶ 0083, ¶ 0086) and S4: comparing the contour feature data of the workpieces (Duan: ¶ 0083, ¶ 0086, ¶ 0087). Duan teaches a robot sorting method wherein the system is configured to capture images of a plurality of objects. The system uses these images to determine the contours, boundary lines, and other information the objects in the images. The system the determines based on the contour information, whether the objects are grabable or ungrabable. This is determined by determining how much of an object is occluded by other objects (i.e. if a significant portion of the object is covered by other objects) or if the area required to properly grasp the object is in accessible due to other robots blocking the robot from reaching the are required to properly grab the object. One of ordinary skill in the art would recognize that this method of determining if the objects are grabable or ungrabable is a method of determining a stacking state of the objects. the method taught in Ito in view of Hayashi is already configured to determine the contour information of each object and a grasping order for the objects. As such, the method taught in Ito in view of Hayashi could be easily modified to use the contour information when determining the grasping sequence using the methods taught in Duan. Such a modification would not have changed or introduced new functionality. No inventive effort would have been required. The secondary reference Zhu teaches a second stacking parameter, and the second stacking parameter is the smallest one of the distances between a geometric center of one of the workpieces and each of workpiece edges of the workpieces (Zhu: ¶ 0010, ¶ 0014, ¶ 0025, ¶ 0026, ¶ 0027, ¶ 0028, ¶ 0029, ¶ 0031, ¶ 0032, ¶ 0033, ¶ 0035). The cited passages of Zhu clearly teaches that the geometric center of the object is determined. The system is the configured to determine a distance from the geometric center to the edges of the object and take the smallest of these distances. This information is then used to control the robot to grasp an object. One of ordinary skill in the art would recognize that this clearly teach the limitation “a second stacking parameter, and the second stacking parameter is the smallest one of the distances between a geometric center of one of the workpieces and each of workpiece edges of the workpieces”. Therefore, it is clear that the combination of Ito in view of Hayashi in further view of Duan in further view of Zhu teaches the limitations “S2: acquiring a posture matrix of each of workpieces in a grabbing space from the point cloud set information”, “"S3: acquiring contour feature data of each of the workpieces according to the posture matrix, the contour feature data includes depth information, a first stacking parameter, and a second stacking parameter, wherein the first stacking parameter is a placing state of a workpiece and the second stacking parameter is the smallest one of the distances between a geometric center of one of the workpieces and each of workpiece edges of the workpieces;”, and “S4: comparing the contour feature data of the workpieces, determining a target grabbing sequence, and grabbing the workpieces according to the target grabbing sequence”.
Specifically on Pages 12-13, Applicant argues that the secondary reference Zhu does not teach the limitations “S2: acquiring a posture matrix of each of workpieces in a grabbing space from the point cloud set information”, “"S3: acquiring contour feature data of each of the workpieces according to the posture matrix, the contour feature data includes depth information, a first stacking parameter, and a second stacking parameter, wherein the first stacking parameter is a placing state of a workpiece and the second stacking parameter is the smallest one of the distances between a geometric center of one of the workpieces and each of workpiece edges of the workpieces;”, and “S4: comparing the contour feature data of the workpieces, determining a target grabbing sequence, and grabbing the workpieces according to the target grabbing sequence”. The Examiner respectfully disagrees. In response to applicant's arguments against the references individually, one cannot show nonobviousness by attacking references individually where the rejections are based on combinations of references. See In re Keller, 642 F.2d 413, 208 USPQ 871 (CCPA 1981); In re Merck & Co., 800 F.2d 1091, 231 USPQ 375 (Fed. Cir. 1986). As was stated in the previous Non-Final Office Action mailed December 23rd, 2025, above in the 35 U.S.C. § 103 rejection section, and above in response to Applicant’s prior arguments, Zhu was not relied upon to teach the limitations “S2: acquiring a posture matrix of each of workpieces in a grabbing space from the point cloud set information”, “"S3: acquiring contour feature data of each of the workpieces according to the posture matrix, the contour feature data includes depth information, a first stacking parameter, wherein the first stacking parameter is a placing state of a workpiece”, and “S4: comparing the contour feature data of the workpieces, determining a target grabbing sequence, and grabbing the workpieces according to the target grabbing sequence”. As was stated previously Ito teaches the limitations “S2: acquiring a posture matrix of each of workpieces in a grabbing space from the point cloud set information” and “determining a target grabbing sequence, and grabbing the workpieces according to the target grabbing sequence” a robot sorting method, wherein the robot is configured to capture a plurality of grating image of a container containing a plurality of objects. The robot the determines the pose of each object based on the grating images. Using these poses, the robot then determines a grabbing sequence of the objects and grabs the object according to the sequence. Applicant also stipulated that Ito teaches such on Page 8 of Applicant’s arguments. The secondary reference Hayashi teaches the limitations “S3: acquiring contour feature data of each of the workpieces according to the posture matrix, the contour feature data includes depth information”. Hayashi teaches a robotic grasping method wherein the system is configured to capture a plurality of grating images of objects. The system then preforms recognition and fusion on the plurality of grating images to generate a point cloud. The system then uses these point clouds to determine depth information as well as the 3D shapes of the objects (i.e. the contour information). Duan teaches the limitations “a first stacking parameter, wherein the first stacking parameter is a placing state of a workpiece” and “S4: comparing the contour feature data of the workpieces”. Duan teaches a robot sorting method wherein the system is configured to capture images of a plurality of objects. The system uses these images to determine the contours, boundary lines, and other information the objects in the images. The system the determines based on the contour information, whether the objects are grabable or ungrabable. This is determined by determining how much of an object is occluded by other objects (i.e. if a significant portion of the object is covered by other objects) or if the area required to properly grasp the object is in accessible due to other robots blocking the robot from reaching the are required to properly grab the object. One of ordinary skill in the art would recognize that this method of determining if the objects are grabable or ungrabable is a method of determining a stacking state of the objects. The secondary reference Zhu teaches a second stacking parameter, and the second stacking parameter is the smallest one of the distances between a geometric center of one of the workpieces and each of workpiece edges of the workpieces (Zhu: ¶ 0010, ¶ 0014, ¶ 0025, ¶ 0026, ¶ 0027, ¶ 0028, ¶ 0029, ¶ 0031, ¶ 0032, ¶ 0033, ¶ 0035). The cited passages of Zhu clearly teaches that the geometric center of the object is determined. The system is the configured to determine a distance from the geometric center to the edges of the object and take the smallest of these distances. This information is then used to control the robot to grasp an object. One of ordinary skill in the art would recognize that this clearly teach the limitation “a second stacking parameter, and the second stacking parameter is the smallest one of the distances between a geometric center of one of the workpieces and each of workpiece edges of the workpieces”. Furthermore, the method taught in Ito in view of Hayashi in further view of Duan teaches determining the shape and contours of each object, the pose of each object, and several metric used to determine the stacking order. As such, the method taught in Ito in view of Hayashi in further view of Duan teaches all of the components necessary to determine the geometric center and smallest distance to the edge as taught in Zhu. Additionally, the method taught in Ito in view of Hayashi in further view of Duan teaches multiple metrics used to determine the grasping order, including various area based metrics. As such the method can be easily modified to include a metric based on the smallest distance from the center as taught in Zhu. Such modifications would not change or introduce new functionality. No inventive effort would have been required. Therefore, it is clear that the combination of Ito in view of Hayashi in further view of Duan in further view of Zhu teaches the limitations “S2: acquiring a posture matrix of each of workpieces in a grabbing space from the point cloud set information”, “"S3: acquiring contour feature data of each of the workpieces according to the posture matrix, the contour feature data includes depth information, a first stacking parameter, and a second stacking parameter, wherein the first stacking parameter is a placing state of a workpiece and the second stacking parameter is the smallest one of the distances between a geometric center of one of the workpieces and each of workpiece edges of the workpieces;”, and “S4: comparing the contour feature data of the workpieces, determining a target grabbing sequence, and grabbing the workpieces according to the target grabbing sequence”.
Therefore, for the reasons stated above and in the 35 U.S.C. § 103 rejection section, the 35 U.S.C. § 103 rejection of claims 1-10 are maintained.
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Noah W Stiebritz whose telephone number is (571)272-3414. The examiner can normally be reached Monday thru Friday 7-5 EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Ramon Mercado can be reached at (571) 270-5744. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/N.W.S./Examiner, Art Unit 3658
/Ramon A. Mercado/Supervisory Patent Examiner, Art Unit 3658
Prosecution Insights