DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
This correspondence is in response to amendments filed on May 9, 2026. Claims 1-4, 6, 10, 13-15, and 17-19 are amended. Claims 5, 7, 11-12, and 16 are filed as previously presented. Claims 8 and 9 are cancelled. Amendments to the abstract obviate the objection set forth in the previous rejection, and as such the objection to the specification/abstract is withdrawn. Amendments to claims 1, 6, 13, 17, and 19 obviate the claim objections set forth in the previous rejection, and as such those objections are withdrawn. Examiner maintains the 112f claim interpretations of the previous correspondence. Amendments to claims 2-4, 6, 14-15, and 18 obviate the 112b rejections set forth in the previous rejection and as such those rejections are withdrawn. Claims 8 and 9 are cancelled and thus those 112b rejections set forth in the previous rejection are rendered moot. Amendments to Claim 18 obviate the 101 rejection set forth in the previous rejection and as such the rejection is withdrawn. Examiner responds to Applicant’s arguments below.
Response to Arguments
Applicant argues that cited prior art fails to disclose the previous features of claim 9 which were struck through, and as such does not teach the features of claim 1 which is amended to only include those struck-through features (Page 20 of Remarks). Applicant’s arguments with respect to the amended features have been considered but are moot because the new ground of rejection does not rely on the same combination of references applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Applicant further argues that Hickman does not disclose the sequence that requires from among multiple gripping point candidates already generated for the same current operating cycle, one candidate is selected by subsequently selecting the algorithm that generated that candidate (Page 20 of Remarks). Applicant further argues that Hickman does not teach the claimed “downstream selection of a gripping point determination algorithm from among algorithms that have already been executed” (Page 21 of Remarks). Applicant provides additional summary of claim and disclosure of Hickman to further these arguments on Pages 20 and 21 of Remarks. Examiner ascertains that there are two specific aspects of the argument that are unpersuasive. First, Examiner does not rely upon Hickman for any such teaching relating to gripping point determination. Examiner only relies on Hickman to teach methods of selecting one of multiple algorithms. In fact, Examiner explicitly acknowledges that Hickman does not teach any such gripping point determination (see Pages 24-25 of Non-Final Rejection mailed out on February 5, 2026). In response to applicant's argument that Hickman does not teach features for the gripping point determinations and only teaches general image processing, the test for obviousness is not whether the features of a secondary reference may be bodily incorporated into the structure of the primary reference; nor is it that the claimed invention must be expressly suggested in any one or all of the references. Rather, the test is what the combined teachings of the references would have suggested to those of ordinary skill in the art. See In re Keller, 642 F.2d 413, 208 USPQ 871 (CCPA 1981). Therefore, with respect to the contents which the algorithm selection analyzes, Hickman is only relied upon to teach the use of one or more algorithms to determine a best fitting solution to the problem at hand. When combined with Ku ‘601, which teaches the use of image analysis to determine the highest quality grasping candidate, the test for obvious renders a method which performs such grasp predictions for each of the one or more image analysis algorithms as taught by Hickman to arrive at the claimed invention. Therefore, the argument regarding the features of Hickman which were considered in making the rejection has been considered but is NOT PERSUASIVE.
Second, Examiner ascertains that the temporal ordering of the alleged sequence of steps is not expressed in the claim as the claim is currently written. That is, nothing in the claim suggests that the method steps should occur in any specified order. In response to applicant's argument that the references fail to show certain features of the invention, it is noted that the features upon which applicant relies (i.e., temporal ordering of method steps or “downstream selection of a gripping point determination algorithm”) are not recited in the rejected claim(s). Although the claims are interpreted in light of the specification, limitations from the specification are not read into the claims. See In re Van Geuns, 988 F.2d 1181, 26 USPQ2d 1057 (Fed. Cir. 1993). Thus, the argument which pertains to an ordered sequence of steps has been considered but is NOT PERSUASIVE.
Applicant further argues that Examiner relies on “blanket statements” and as such applies “Official Notice” in making the rejection and the combination of Ku ‘601 and Hickman would not have been obvious to one of ordinary skill in the art (see Pages 22 and 23 of Remarks).
In response to applicant's argument that Hickman addresses future or continued image-processing use in the machine-vision system and not the claimed selection of a gripping point determination algorithm, the test for obviousness is not whether the features of a secondary reference may be bodily incorporated into the structure of the primary reference; nor is it that the claimed invention must be expressly suggested in any one or all of the references. Rather, the test is what the combined teachings of the references would have suggested to those of ordinary skill in the art. See In re Keller, 642 F.2d 413, 208 USPQ 871 (CCPA 1981). That is, Ku ‘601 which uses a single image processing algorithm to produce a set of gripping point candidates would be reasonably modified by one of ordinary skill in the art to evaluate multiple such image processing algorithms in parallel as contemplated by Hickman to achieve the claimed invention.
In response to applicant's argument that the examiner's conclusion of obviousness is based upon improper hindsight reasoning (see Remarks Page 23), it must be recognized that any judgment on obviousness is in a sense necessarily a reconstruction based upon hindsight reasoning. But so long as it takes into account only knowledge which was within the level of ordinary skill at the time the claimed invention was made, and does not include knowledge gleaned only from the applicant's disclosure, such a reconstruction is proper. See In re McLaughlin, 443 F.2d 1392, 170 USPQ 209 (CCPA 1971). Examiner concludes that the combination of Ku ‘601 in view of Hickman is obtained by knowledge which is within the level of ordinary skill at the time of the claimed invention and does not rely on any such knowledge gleaned only from the applicant’s disclosure. As such, it is determined that the assessment for obviousness is proper.
In response to applicant’s argument that there is no teaching, suggestion, or motivation to combine the references, the examiner recognizes that obviousness may be established by combining or modifying the teachings of the prior art to produce the claimed invention where there is some teaching, suggestion, or motivation to do so found either in the references themselves or in the knowledge generally available to one of ordinary skill in the art. See In re Fine, 837 F.2d 1071, 5 USPQ2d 1596 (Fed. Cir. 1988), In re Jones, 958 F.2d 347, 21 USPQ2d 1941 (Fed. Cir. 1992), and KSR International Co. v. Teleflex, Inc., 550 U.S. 398, 82 USPQ2d 1385 (2007). In this case, Examiner relies on evidence from Hickman directly in rejecting the claimed features which are not contemplated by Ku ‘601. Examiner further expresses motivation to combine such teachings found directly within Hickman (see Page 25 of Non-Final Rejection mailed out on February 5, 2026).
Examiner furthers these conclusions of obviousness using rationale solicited in MPEP 2143.I. In the example of MPEP 2143.I(A) regarding a combination of prior art elements which Applicant argues on Page 22 of Remarks, Examiner ascertains that Ku ‘601 teaches each element claimed regarding for gripping point determinations and Hickman teaches each element claimed regarding selection of multiple algorithms. The only difference between the claimed invention and the prior art is that there is no actual combination of the elements in a single reference. Then, provided that Ku ‘601 teaches the gripping point selection via results of an image processing algorithm for scene detection (S100-S300 of Fig. 3 in Ku ‘601) and Hickman teaches a parallel processing of image analysis algorithms for scene detection (Figure 3 of Hickman), the combination of the gripping point selection algorithm of Ku ‘601 and the parallel processing of multiple image analysis algorithms of Hickman is a combination of teachings which may be performed by one of ordinary skill in the art which each element merely performs the same function as it does separately. One of ordinary skill in the art would find these results to be predictable because the combination is a mere modification of software processing which is within an ordinary skillset for the art. Thus, the combination as determined by Examiner is one which is gleaned only from the explicit methods and teachings of Ku ‘601 and Hickman.
In response to applicant's argument that Hickman is nonanalogous art, it has been held that a prior art reference must either be in the field of the inventor’s endeavor or, if not, then be reasonably pertinent to the particular problem with which the inventor was concerned, in order to be relied upon as a basis for rejection of the claimed invention. See In re Oetiker, 977 F.2d 1443, 24 USPQ2d 1443 (Fed. Cir. 1992). In this case, the image processing algorithms of Hickman, although disclosed for an alternative use, are directed to algorithm selection processes for robots. In fact, C9, L8-11 Hickman explicitly determines that a particular use of such machine vision system is applied to interacting with the environment, i.e., grasping and manipulating an object. Thus, Hickman is in the field of the inventor’s endeavor, or at the very least is reasonably pertinent to the particular algorithm selection process which the inventor is concerned.
As a reminder to Applicant, the factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Throughout the rejection below, Examiner determines the scope and contents of the prior art. Examiner then identifies the gaps between the primary reference and the claims at issue. In identifying these gaps, Examiner provides subsequent art which teaches said gaps and appropriately resolves the level of ordinary skill pertinent to the art. Examiner then further provides objective evidence, if not multiple points of objective evidence, to indicate motivation and rationale which indicates the combination as obvious.
Therefore, Applicant’s assertion that Examiner relies on improper Official Notice is NOT PERSUASIVE as each conclusion of obviousness is determined by relying on teachings of the prior art.
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action.
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because the claim limitation(s) uses a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier. Such claim limitations are:
“at least one detection unit which is designed to” in claims 1, 16, and 19; and
“a monitoring device which is designed to” in claim 14.
Regarding the at least one detection unit, Page 3 of Applicant’s specification recites, “The at least one detection unit is preferably designed as a camera, in particular a 3D camera, for example as a CCD camera. It is also conceivable for the at least one detection unit to be a laser scanner, ultrasonic sensor, or radar sensor. It is also conceivable for the detection device to comprise a plurality of detection units of different types.” Thus, such a detection unit will be considered as a camera(s), a laser scanner(s), an ultrasonic sensor(s), or a radar sensor(s), and any other such functional equivalent which captures an image of an item to be gripped when reviewing the prior art.
Regarding the monitoring device, Pages 19-20 of Applicant’s specification recites, “The monitoring device can, for example, comprise one or more cameras which are designed to capture the image of the gripping of the item at the target gripping point, in particular to capture whether the item has been reliably gripped and deposited again. In an embodiment of the end effector as a suction gripping apparatus, the monitoring device can, for example, comprise a vacuum sensor which is designed to monitor a negative pressure prevailing in the suction gripping apparatus. It is also conceivable for the monitoring device to comprise a weighing device which is designed to weigh a source container, in particular before and after the gripping of an item. It is also conceivable for the items to have an RFID tag. The monitoring device can then comprise an RFID detector.” Thus, the monitoring device will be considered as any such device inclusive of cameras, vacuum sensor, weighing device, RFID detector, or other functional equivalent which determines whether or not an item has been successfully gripped when reviewing the prior art.
Because these claim limitations are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, they are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof.
If applicant does not intend to have these limitations interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitations to avoid them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitations recites sufficient structure to perform the claimed function so as to avoid them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
This application includes one or more claim limitations that use the word “means” or “step” but are nonetheless not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph because the claim limitation(s) recite(s) sufficient structure, materials, or acts to entirely perform the recited function. Such claim limitations are:
“means of the detection device” in claims 1 and 19; and
“means of the monitoring device” in claim 14.
Examiner did not consider such devices as functions and thus will not interpret these uses of “means” as themselves invoking 112f interpretations. However, Examiner notes that such devices are related to or directly invoke 112f claim interpretations as being a generic unit or device which are designed to perform a desired detection or monitoring function (see above).
Because these claim limitations are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, they are not being interpreted to cover only the corresponding structure, material, or acts described in the specification as performing the claimed function, and equivalents thereof.
If applicant intends to have this/these limitation(s) interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation(s) to remove the structure, materials, or acts that performs the claimed function; or (2) present a sufficient showing that the claim limitation(s) does/do not recite sufficient structure, materials, or acts to perform the claimed function.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
NOTE: Wherein the claims have been addressed below regarding 103 rejections, any alternatives which have not been considered by the prior art of record and/or any such teachings which are not explicitly referenced by the prior art have been struck through.
Claims 1-13 and 15-19 are rejected under 35 U.S.C. 103 as being unpatentable over Ku et al. (US 2023/0256601 A1; hereinafter “Ku ‘601”) in view of Hickman et al. (US Patent No. 8,965,104; hereinafter “Hickman”) and further in view of Rohanimanesh et al. (US 2023/0081119 A1; hereinafter “Rohanimanesh”).
Regarding claim 1, Ku ‘601 teaches a computer-implemented method for controlling a handling system (“The method can be performed using a system of one or more computers in one or more locations configured to control a robot having grasping capabilities (an example of which is shown in FIG. 2) including one or more: robots 220, sensors, computing systems 230, sensors 240, and/or any other suitable components” [0021]. Thus, the method is computer-implemented and is configured to control a grasping robot, i.e., handling system.), the computer-implemented method comprising:
- at least one robot on which an end effector for gripping an item is arranged (“The robot 220 functions to manipulate an object. The robot can include one or more: end effectors 222, robotic arms 224, and/or any other suitable components” [0022]. Thus, there is a robot with an end effector which manipulates, i.e., grips, an object, i.e., item.);
- a detection device comprising at least one detection unit which is designed to capture an image of an item to be gripped (“The sensors 240 function to sample measurements of a physical scene. The sensors 240 can include: visual sensors (e.g., monocular cameras, stereo cameras, projected light systems, TOF systems, etc.), acoustic sensors, actuation feedback systems, and/or any other suitable sensors” [0024]. Thus, there are visual sensors which sample measurements of a physical scene, i.e., detection units which are designed to capture an image of the item(s) to be gripped. Such visual sensors which are listed satisfy the 112f claim interpretation of detection unit above, in which such sensors are inclusive of cameras.); and
- a control device for controlling the handling system, wherein the control device comprises a data processing system and a non-volatile memory device (“The computing system 230 functions to perform one or more steps of the method, but can additionally and/or alternatively provide any other suitable functionality. The computing system 230 can be local to the robot, remote, and/or otherwise located” [0025]. “Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non-transitory storage medium for execution by, or to control the operation of, data processing apparatus” [0084]. Thus, the computing system which controls the handling system according to the methods of the disclosure comprises computer programs stored on a non-transitory storage medium, i.e., non-volatile memory device, which are further executed by a data processing apparatus, i.e., data processing system.),
wherein the method comprising performing one or more control cycles (“All or portions of the method can be performed once, iteratively, repeatedly (e.g., for different objects, for different physical scenes, for different time frames, for different sensors), periodically, and/or otherwise performed” [0029]. Thus, provided that the method is performed once, iteratively, periodically, or repeatedly, the method performs one or more control cycles.), each control cycle comprising:
a) receiving image data that represent an image of at least one portion of the item to be gripped, which the image is captured by means of the detection device (“S100 functions to determine a measurement of a physical scene having a container 410 that contains one or more objects 420 to be grasped; example shown in FIG. 4. The measurement can include one measurement, multiple measurements, and/or any other suitable number of measurements. The measurement can be captured by a sensor, retrieved from a database, and/or otherwise determined. The measurement can be an image, depth information, point clouds, video, and/or any other suitable measurement” [0031]. Thus, the method determines a measurement of the container which contains objects to be grasped, and such a received measurement is an image which is captured by a sensor, i.e., the detection device.),
b) determining a target gripping point for the end effector on the item, comprising analyzing the image data (“Selecting a grasp S40 functions to determine a grasp proposal from the set based on a final score associated with each grasp proposal” [0067]. Thus, there is a selected grasp proposal, i.e., target gripping point, which is a result of determining a set of grasp proposals (see Fig. 3, S300) via analyzing the image data.), and
c) generating control signals which cause the at least one robot to grip the item at the target gripping point by means of the end effector (“Executing the grasp trajectory S50 functions to move the robotic arm and/or end effector to grasp an object in the physical scene based on the calculated grasp trajectory associated to the selected grasp proposal” [0069]. Thus, there is a step in the method which moves the end effector to grasp the object based on the selected grasp proposal, i.e., generates control signals causing the robot to grip the item at the target gripping point by means of the end effector.),
the determination of the target gripping point comprises:
b1) analyzing the image data by (“Determining a set of grasp proposals across the workspace S300 functions to determine grasp proposals within the workspace. Each grasp proposal can be associated with a virtual projection of the end effector onto the physical scene (e.g., a “window”), associated with an end effector pose (e.g., location and orientation; x, y, z position and α, β, γ orientation), and/or associated with any other suitable end effector attribute. Grasp proposals can be: predetermined, dynamically determined, randomly determined, and/or otherwise determined” [0033]. Thus, a set of gripping candidates using any of a variety of algorithms to project grasp proposals into the workspace. The workspace is determined in S100 and S200 based on a captured image (see [0031-0032]).); and
b2) selecting a gripping point candidate from the set Me of determined gripping point candidates as the target gripping point depending on one or more specified gripping point selection criteria (“Selecting a grasp S40 functions to determine a grasp proposal from the set based on a final score associated with each grasp proposal… In a first variant, S40 can include selecting a grasp proposal based on the preliminary score. In a second variant, S40 can include selecting a grasp proposal based on one or more heuristic scores. In a third variant, S40 can include selecting a grasp proposal based on a combination of the preliminary and heuristic scores. In a fourth variant, S40 can include selecting a grasp proposal based on the grasp score or a combination with preliminary score and/or heuristic score” [0067]. A target gripping point is determined in S40 depending on a preliminary score, one or more heuristic scores, or any combination of such scores.).
However, Ku ‘601 does not explicitly teach …analyzing the image data by two or more mutually independent gripping point determination algorithms… , wherein selecting the gripping point candidate as the target gripping point comprises:
b2.1) selecting one of the two or more gripping point determination algorithms as a target evaluation algorithm depending on at least one specified algorithm selection criterion, and
b2.2) selecting one of the gripping point candidates determined by the target evaluation algorithm as the target gripping point depending on the at least one gripping point selection criterion, or selecting the gripping point candidate determined by the target evaluation algorithm as the target gripping point,
wherein the at least one specified algorithm selection criterion comprises one or more of the following selection criteria:
- a calculation duration of the gripping point determination algorithms, wherein the gripping point determination algorithm is selected as the target evaluation algorithm which has determined a gripping point candidate the fastest;
- a probability of success when gripping the item at a gripping point candidate determined by the gripping point determination algorithm, wherein the gripping point determination algorithm which has the highest probability of success is selected as the target evaluation algorithm;
- a property or type of the employed end effector; and/or
a confidence value determined by the gripping point determination algorithm.
Hickman, in the same field of endeavor, teaches …analyzing the image data by two or more mutually independent (“As described above, to determine an appropriate image processing algorithm and corresponding parameter set for the robot 301, the cloud processing engine 302 shown in example 300 applies "n" different image processing algorithms 305, 306, 307 to the image included in the image data 304 received from the robot 301. Example 300 also shows each algorithm of the plurality of algorithms 305, 306, 307 being applied to the image multiple times. In operation, each application of the image processing algorithm to the image is executed with a different set of image processing parameters (which may include some similar or same parameters), and each application of a particular algorithm configured with a corresponding parameter set yields a different image processing result” (C 14, L 48-60). Thus, there are two or more mutually independent algorithms which analyze the image data.)…
Hickman does not explicitly apply the algorithms to gripping point determinations, only general image processing. However, given that Ku ‘601 teaches an analysis of gripping point candidates with each iteration of image analysis, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have modified the gripping point determinations of Ku ‘601 to include image analysis with a plurality of algorithms as taught by Hickman with a reasonable expectation of success. One of ordinary skill in the art would have been motivated to make this modification because a higher quality image processing results would lead to a higher quality grasp proposal due to higher quality attribute detections in the respective image of the object to be gripped (Hickman, (C2,L 63-C3,L14)). Such a modification would additionally be a combination of known methods which yield predictable results (see MPEP 2143.I(A)).
Hickman further teaches …wherein selecting the gripping point candidate as the target gripping point comprises:
b2.1) selecting one of the two or more gripping point determination algorithms as a target evaluation algorithm depending on at least one specified algorithm selection criterion (“After generating the plurality of quality scores 310, 313, and 316, the cloud processing engine 302 determines which quality score is the highest. After determining the highest quality score, the cloud processing engine 302 selects the image processing algorithm and the parameter set that was used to generate the image processing result having the highest score, and then the cloud processing engine 302 sends an indication of the selected algorithm and parameter set to the robot 301 via response 317. In situations where multiple image processing results have the highest score, the cloud processing engine 302 may be configured to select one of the multiple highest scoring results based in part on environmental data, task data, and/or object data (as previously described) received from the robot 301” (C16, L23-36). Thus, the target evaluation algorithm with the highest quality score depends on the algorithm selection criteria inclusive of environmental data, task data, and object data.)…
However, Ku ‘601 as modified by Hickman does not explicitly teach …b2.2) selecting one of the gripping point candidates determined by the target evaluation algorithm as the target gripping point depending on the at least one gripping point selection criterion, or selecting the gripping point candidate determined by the target evaluation algorithm as the target gripping point…
However, provided that the modification of Ku ‘601 in view of Hickman combines the gripping point determinations of Ku ‘601 with the algorithm determinations of Hickman such that each algorithm iteration provides a gripping point candidate which is reflected by a quality score (as taught by both Ku ‘601 and Hickman), it would be implied that the selected target algorithm would additionally select a target gripping point which has been determined by the selected target algorithm. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, that Ku ‘601 as modified by Hickman additionally teaches “selecting the gripping point candidate determined by the target evaluation algorithm as the target gripping point” with a reasonable expectation of success.
However, Ku ‘601 as modified by Hickman still does not teach … wherein the at least one specified algorithm selection criterion comprises one or more of the following selection criteria:
- a calculation duration of the gripping point determination algorithms, wherein the gripping point determination algorithm is selected as the target evaluation algorithm which has determined a gripping point candidate the fastest;
- a probability of success when gripping the item at a gripping point candidate determined by the gripping point determination algorithm, wherein the gripping point determination algorithm which has the highest probability of success is selected as the target evaluation algorithm;
- a property or type of the employed end effector; and/or
a confidence value determined by the gripping point determination algorithm.
Rohanimanesh, pertinent to the problem at hand, teaches … wherein the at least one specified algorithm selection criterion (As described in [0061-0070], the method includes an algorithm selection criteria related to a plurality of grasp prediction models, i.e., gripping point determination algorithm, which each predict a plurality of grasps for grasping a plurality of objects.) comprises one or more of the following selection criteria:
- a probability of success when gripping the item at a gripping point candidate determined by the gripping point determination algorithm, wherein the gripping point determination algorithm which has the highest probability of success is selected as the target evaluation algorithm (In the Markov Decision Process (MDP) for selecting the corresponding grasp prediction model, “the plurality of grasps are in an ordered sequence” (see [0061]). Per the teachings of [0066], k pixel positions for each of the plurality of grasp models having the highest success probability scores are identified and then selects the plurality of grasps by solving the MDP (see [0067]). As identified in [0062], the MDP is at least based on pick success. As identified in step (E) of the method ([0063]), the first grasp in the ordered sequence for performing the grasp is identified, i.e., the grasp with the highest probability of success. As identified in step (F) of the method ([0063]), there is further an identification of the end-effector corresponding to the first grasp, i.e., identification of the grasp prediction model which successfully predicted the highest probability of success.);
- a property or type of the employed end effector (As detailed above with regard to step (F) (see [0063]), the selection of the first grasp of the ordered sequence, i.e., the first grasp prediction model, is associated with a type of end effector employed for performing the grasp.); and/or
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have modified the algorithm selection process for gripping determinations as taught by Ku ‘601 in view of Hickman to include the success probability and end effector considerations as taught by Rohanimanesh with a reasonable expectation of success. One of ordinary skill in the art would have been motivated to make such a modification because by determining grasping points via unique algorithms which correspond to specified end effectors and selections based on a success probability, efficiency of picking tasks for highly diverse objects is increased, thereby optimizing and otherwise increasing throughput for robotic bin picking tasks (Rohanimanesh, [0001-0005]).
Regarding claim 2, Ku ‘601 as modified by Hickman further modified by Rohanimanesh teaches the computer-implemented method according to claim 1,
with Hickman further teaching …wherein target selection data are determined which contain information as to which of the two or more gripping point determination algorithms has determined the gripping point candidate selected as the target gripping point in a control cycle (“After determining a particular image processing algorithm and corresponding parameter set for execution by the robot's 301 machine vision system, the cloud processing engine 302 (alone or in combination with other cloud computing system components) may send the determined image processing algorithm and parameter set to the robot 301 via a response 317” (C12,L47-53). Thus, the selected image processing algorithm and parameter set are considered as the target selection data containing information as to which of the two or more algorithms has determined the selection in a control cycle based on a quality score.), and/or
Regarding claim 3, Ku ‘601 as modified by Hickman further modified by Rohanimanesh teaches the computer-implemented method according to claim 2,
with Hickman further teaching …wherein at least one of the target selection data or the selection frequency data are transmitted to at least one of an external computer, an external data network, a computer, or to a data network of a provider providing the gripping point determination algorithm which selected the target gripping point (“For example, in some embodiments, after the cloud processing engine 302 has selected a particular algorithm and corresponding parameter set based on the above-described multiple algorithm analysis process, the cloud processing engine 302 may send the selected algorithm and parameter set (or at least an indication of the selected algorithm and parameter set) to the machine vision knowledge base 303 along with the environmental data, task data, and/or object data (as previously described) that may have been received from the robot 301 along with the image data 304” (C18,L11-20). Thus, the target selection data is transmitted to an external knowledge base, i.e., data network.).
Regarding claim 4, Ku ‘601 as modified by Hickman further modified by Rohanimanesh teaches the computer-implemented method according to claim 3,
with Ku ‘601 further teaching …wherein the at least one gripping point selection criterion comprises one or more of the following criteria:
- at least one of a position, coordinates, or orientation of the gripping point candidate in a coordinate system of the handling system (“This process can determine a second score representative of how favorable and/or unfavorable a grasp proposal is based on the associated waypoints, grasp parameters (e.g., orientation, location, etc.) and/or other heuristics” [0059]. Thus, there is a heuristic measurement for grasp scoring which is based on the orientation and location of the gripping point candidate, i.e., grasp proposal.);
- a probability of success when gripping the item at the gripping point candidate, wherein the gripping point candidate is selected as the target gripping point which has the highest probability of success (“The heuristic score can be indicative of trajectory success, efficiency, speed, and/or any other suitable metric” [0059]. Thus, there is a heuristic based on success. Additionally, “In variants, the grasp can be selected based on: a probability of grasp success, a grasp execution speed, and/or otherwise selected. The probability of grasp success can be determined based on the scene features appearing within each grasp window, waypoints for the grasp (e.g., calculated using safety margins, etc.), and/or otherwise determined” [0018]. Thus, there is a passage which describes selection based on probability of success.);
- a property or type of the item to be gripped, wherein the property or type of the item is based on at least one of the item’s geometry, surface quality, or material properties (“In a fourth variant, S400 can include determining a preliminary score based on the material of the object at the grasp proposal. For example, S400 can include determining the material of the object (e.g., using RGB imagery), and assigning a more favorable score to grasp proposals with object material more suitable for grasping (e.g., material suitable for strong suction seal), such as nonporous surfaces or flat surfaces” [0043]. Thus, there is a scoring criteria which depends on a material property and/or geometry of the item to be gripped.);
- energy consumption to be expected when gripping the item at the gripping point candidate (“The heuristic score can be indicative of trajectory success, efficiency, speed, and/or any other suitable metric” [0059]. Thus, there is a heuristic based on efficiency and speed, i.e., energy consumption to be expected.); and/or
Regarding claim 5, Ku ‘601 as modified by Hickman further modified by Rohanimanesh teaches the computer-implemented method according to claim 4,
with Ku ‘601 further teaching …wherein selecting the gripping point candidate as the target gripping point comprises:
- receiving gripping point selection criteria data which represent a user-specified selection of one or more of the gripping point selection criteria, and selecting the gripping point candidate as the target gripping point depending on the selected gripping point selection criterion or selected gripping point selection criteria (“Selecting a grasp S40 functions to determine a grasp proposal from the set based on a final score associated with each grasp proposal. The final score can be a combined score (e.g., scaled, weighted) based on the preliminary score and the one or more heuristic scores, based only on the preliminary score, based only on one heuristic score, based only on multiple heuristic scores, based only on the grasp score, and/or based on any other suitable score or combination thereof” [0067]. Thus, the selection occurs based on a specified scoring criteria which can be based on a preliminary score, one or more heuristic scores, or the grasp score.); and/or
- receiving gripping point selection criteria weighting data which represents a user-specified weighting of the gripping point selection criteria, and selecting the gripping point candidate as the target gripping point depending on the weighted gripping point selection criteria (“Selecting a grasp S40 functions to determine a grasp proposal from the set based on a final score associated with each grasp proposal. The final score can be a combined score (e.g., scaled, weighted) based on the preliminary score and the one or more heuristic scores, based only on the preliminary score, based only on one heuristic score, based only on multiple heuristic scores, based only on the grasp score, and/or based on any other suitable score or combination thereof” [0067]. Thus, the selection occurs based on a specified scoring criteria which is weighted.).
Ku ‘601 does not explicitly state that such final score criteria are user-selected or received. However, given that such scoring criteria are given as alternative methods to be considered, it would have been obvious to one of ordinary skill in the art that the system would require a user-selection for the criteria based on design incentives and the goal of operation (see MPEP 2143.I(F)).
Regarding claim 6, Ku ‘601 as modified by Hickman further modified by Rohanimanesh teaches the computer-implemented method according claim 5,
with Ku ‘601 further teaching …wherein the determined gripping point candidates are evaluated depending on the gripping point selection criteria and sorted in a ranking list (“The system can for example rank and filter the predetermined grasps according to their scores” [0079]. Thus, the grasps are ranked according to their scores, i.e., gripping point selection criteria.),
wherein a gripping point candidate is selected as the target gripping point depending on a position of the gripping point candidate in the ranking list, wherein the uppermost gripping point candidate in the ranking list is selected as the target gripping point (“The system can then select a single predetermined grasp having the highest score” [0079]. Thus, the target gripping point is selected based on a position of the candidate in the ranking list wherein the candidate with the highest score, i.e., uppermost candidate in the list, is selected.).
Regarding claim 7, Ku ‘601 as modified by Hickman further modified by Rohanimanesh teaches the computer-implemented method according to claim 1,
with Hickman further teaching …wherein the determination of the target gripping point also comprising:
b0) selecting the two or more gripping point determination algorithms from a set Ma of available gripping point determination algorithms (“Likewise, the cloud computing system 302 could query the machine vision knowledge base 303 to select a set of candidate algorithms and/or parameter sets for use in a multiple algorithm analysis described herein” (C17,L47-50). Thus, the machine vision knowledge base, i.e., set Ma of available algorithms, is queried to select two or more algorithms to be used in the multiple algorithm analysis.),
wherein the two or more gripping point determination algorithms are selected from the set Ma depending on at least one specified algorithm selection criterion (“Alternatively, when the cloud processing engine 302 receives environmental data, object data, and/or task data associated with a particular image from the robot 301, the cloud processing engine 302 may search or query the machine vision knowledge base 303 to identify one or more candidate algorithms and parameter sets that are correlated with the same or similar environmental data, object data, and/or task data received from the robot 301 in the image data 304” (C19,L37-45). Thus, the algorithms are selected from the knowledge base, i.e., set Ma, based on environmental data, object data, and/or task data which will serve as the algorithm selection criterion.).
Regarding claim 10, Ku ‘601 as modified by Hickman further modified by Rohanimanesh teaches the computer-implemented method according to claim 1,
with Hickman further teaching …wherein the selection of the gripping point determination algorithm as the target evaluation algorithm comprises:
- receiving algorithm selection criteria data which represent a user-specified selection of one or more of the algorithm selection criteria, and selecting the gripping point determination algorithm as the target evaluation algorithm depending on the selected algorithm selection criterion or selected algorithm selection criteria (“Many different types of client devices may be configured to communicate with components of the cloud computing system 102 for the purpose of accessing data and executing applications provided by the cloud computing system 102. For example, a computer 112, a mobile device 114, a host 116, and a robot client 118 are shown as examples of the types of client devices that may be configured to communicate with the cloud computing system 102” (C5, L60-67). “Additionally, any of the client devices may also include a user-interface (UI) configured to allow a user to interact with the client device. For example, the robot client 118 may include various buttons and/or a touchscreen interface configured to receive commands from a human or provide output information to a human. As another example, the robot client 118 may also include a microphone configured to receive voice commands from a human. Furthermore, the robot client 118 may also include one or more interfaces that allow various types of user-interface devices to be connected to the robot client 118. For example, the mobile device 114, the computer 112, and/or the host 116 may be configured to run a user-interface for sending and receiving information to/from the robot client 118 or otherwise configuring and controlling the robot client 118” (C6, L49-63). Thus, client devices, inclusive of user interfaces which communicate with the robot through the cloud platform, are used to issue specific task commands, such as the object retrieval determinations described in the previous claims. Therefore, the algorithm selection criteria which determine the quality scores are based on a user-selected task and object data, and thus, the user-selection aids the determination of the target algorithm as previously described.); and/or
Regarding claim 11, Ku ‘601 as modified by Hickman further modified by Rohanimanesh teaches the computer-implemented method according to claim 10,
with Ku ‘601 in view of Hickman and further in view of Rohanimanesh further teaching …wherein the selection of the target evaluation algorithm comprises:
b2.1.1) selecting one of the two or more gripping point determination algorithms as a test algorithm, wherein the test algorithm is selected depending on at least one of the specified algorithm selection criteria (“In some embodiments, rather than selecting the image processing result having the highest quality score, the cloud processing engine 302 may instead select an image processing result having a quality score that meets or exceeds a minimum quality score threshold” (Hickman, (C16,L36-40)). Thus, an image processing result which meets a minimum threshold based on the quality score will be considered as the test algorithm.);
b2.1.2) specifying a gripping point rejection criterion or a plurality of gripping point rejection criteria (“Optionally adjusting each grasp proposal S500 functions to adjust each grasp proposal until a predetermined condition is satisfied” (Ku ‘601, [0046]). “In a third variant, S500 can include removing grasp proposals of the set that fail the condition from consideration for subsequent steps” (Ku ‘601, [0049]). Thus, the predetermined conditions may be considered as gripping point rejection criteria wherein a gripping point is rejected from further consideration when the condition is not met.); and
b2.1.3) checking whether the at least one gripping point candidate determined by the test algorithm meets the specified gripping point rejection criterion or meets one of the specified gripping point rejection criteria, wherein, when the at least one gripping point candidate does not meet a gripping point rejection criteria, the test algorithm is selected as the target evaluation algorithm (“There are multiple ways that the robot 301 can determine that its machine vision system is starting to degrade. For example, any set of one or more of the following conditions could be sufficient to trigger the process of obtaining an sending new image data (perhaps along with additional data) to the cloud processing system 302 for selecting a new parameter set or a new algorithm and parameter set: (i) the quality score of an image processing result falls below a threshold quality score…” (Hickman, (C17,L23-31)). Thus, when the quality score threshold is not met, i.e., a predetermined condition is not satisfied, a new target evaluation algorithm will be considered. Otherwise, the robot may continue to use the selected algorithm, and thus will determine the test algorithm as a target algorithm when the quality score does not fall beneath the threshold.).
Therefore, it would have been obvious to one of ordinary skill in the art to have modified the rejection criteria of Ku ‘601 to be inclusive of quality score criteria as determined by Hickman to reflect a test algorithm selection protocol with a reasonable expectation of success. One of ordinary skill in the art would have been motivated to combine such teachings because early detections of degradations of a given image processing algorithm would mitigate failures in the gripping determination system.
Regarding claim 12, Ku ‘601 as modified by Hickman further modified by Rohanimanesh teaches the computer-implemented method according to claim 11,
with Hickman further teaching …wherein when the at least one gripping point candidate meets the specified gripping point rejection criterion or one of the plurality of specified gripping point rejection criteria, steps b2.1.1) to b2.1.3) are repeated, wherein in step b2.1.1) a different gripping point determination algorithm is selected as a test algorithm (C17,L5-50 determines that when degradation in the quality result of a given algorithm occurs, a new algorithm is selected as the next algorithm, i.e., a test algorithm, and the process for degradation analysis may be repeated.).
Regarding claim 13, Ku ‘601 as modified by Hickman further modified by Rohanimanesh teaches the computer-implemented method according to claim 12,
with Ku ‘601 further teaching …wherein the specified gripping point rejection criterion is one of the following criteria, or wherein the plurality of specified gripping point rejection criteria comprise one or more of the following criteria:
a. the gripping point candidate cannot be approached by the at least one end effector; and/or b. the end effector approaching the gripping point candidate would lead to a collision of the end effector with another item with a given probability (“Optionally adjusting each grasp proposal S500 functions to adjust each grasp proposal until a predetermined condition is satisfied. The condition can be: a grasp vector does not collide with a predetermined workspace feature (e.g., lip of a box), a grasp vector is within an interval distance of a predetermined workspace feature, and/or any other suitable condition” [0046]. Thus, grasps which determine to be obstructed, i.e., unapproachable, and/or which would lead to a collision are adjusted, i.e., preliminarily rejected.).
Regarding claim 15, Ku ‘601 as modified by Hickman further modified by Rohanimanesh teaches the computer-implemented method of claim 1,
with Ku ‘601 further teaching …wherein the handling system comprises at least one of:
a plurality of end effectors which can be coupled to the at least one robot (“The robot 220 functions to manipulate an object. The robot can include one or more: end effectors 222, robotic arms 224, and/or any other suitable components. The end effector 222 can be: a suction cup, a gripper, and/or any other suitable end effector” [0022]. Thus, the robot may be coupled to a plurality of end effectors including a suction cup, gripper, or other such end effectors.), wherein either the selection of a gripping point candidate as the target gripping point or the selection of a gripping point determination algorithm as the target evaluation algorithm depends on which of the plurality of end effectors is coupled to the at least one robot (“The robot 220 and/or end effector 222 can be associated with an active surface (e.g., grasping region, contact region, etc.). The active surface can be associated with a projection of the end effector’s active surface (e.g., “a grasp window”) onto a plane (e.g., x-y plane, vertical plane, etc.). The grasp window (e.g., “window”) for each end effector, each grasp pose (e.g., the template grasp pose, etc.) can be known and/or predetermined for one or more end effector orientations (e.g., determined from the set of grasp proposals), but can alternatively be dynamically determined and/or otherwise determined. In a first example where a template grasp orientation is used for grasp proposal generation, the grasp window for the end effector can be determined once (e.g., based on the template grasp orientation) and reused for multiple grasps. In a second example, a different grasp window can be calculated for each grasp proposal (e.g., predetermined or determined based on scene information). However, the robot can be otherwise configured” [0023]. Thus, in each example, the selection of a gripping point candidate is determined based on which of the plurality of end effectors is coupled to the robot, as each end effector has a specific grasp window which may otherwise be manipulated according to the type of end effector.), or
Regarding claim 16, Ku ‘601 teaches a handling system, comprising:
- at least one robot on which an end effector for gripping an item is arranged (“The robot 220 functions to manipulate an object. The robot can include one or more: end effectors 222, robotic arms 224, and/or any other suitable components” [0022]. Thus, there is an end effector arranged on a robot which is for manipulating objects, i.e., gripping an item.);
- a detection device comprising at least one detection unit, or a camera, which is designed to capture an image of an item to be gripped (“The method can be performed using a system of one or more computers in one or more locations configured to control a robot having grasping capabilities (an example of which is shown in FIG. 2) including one or more: robots 220, sensors, computing systems 230, sensors 240, and/or any other suitable components” [0021]. “The measurement can be an image, depth information, point clouds, video, and/or any other suitable measurement. For example, S100 can include: moving a robot arm in front of a constrained volume containing objects, capturing an image (e.g., and/or depth information) with a camera, wherein the camera is mounted in front of the gripper” [0031]. Thus, there is a detection device having at least one detection unit, or camera, which is designed to capture an image of an item to be gripped, per the disclosed method and S100.); and
- a control device for controlling the handling system, wherein the control device comprises a data processing system and a non-volatile memory device, wherein a computer program is stored on the non-volatile memory device which comprises commands which, when executed by the data processing system, cause the data processing system to execute the method (“The computing system 230 functions to perform one or more steps of the method, but can additionally and/or alternatively provide any other suitable functionality. The computing system 230 can be local to the robot, remote, and/or otherwise located” [0025]. “Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non-transitory storage medium for execution by, or to control the operation of, data processing apparatus” [0084]. Thus, there is a control device with a data processing apparatus and a non-volatile memory which executes the disclosed methods.) according to claim 1 (Ku ‘601 as modified by Hickman further modified by Rohanimanesh).
Regarding claim 17, Ku ‘601 as modified by Hickman further modified by Rohanimanesh the handling system according to claim 16,
with Ku ‘601 further teaching wherein the end effector is a suction gripping apparatus, an elastomer suction gripper, or a vacuum gripper (“The end effector 222 can be: a suction cup” [0022]. Thus, the end effector is a suction gripping apparatus.).
Regarding claim 18, Ku ‘601 teaches a non-transitory computer readable storage medium having instructions stored thereon, such that when the instructions are read and executed by one or more processors, said one or more processors are configured to perform or execute the steps (“Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non-transitory storage medium for execution by, or to control the operation of, data processing apparatus” [0084]. Thus, there is a computer program product stored on a non-transitory medium which performs the disclosed methods.) of the computer-implemented method of claim 1 (Ku ‘601 as modified by Hickman further modified by Rohanimanesh).
Regarding claim 19, Ku ‘601 teaches a handling system (“The method can be performed using a system of one or more computers in one or more locations configured to control a robot having grasping capabilities (an example of which is shown in FIG. 2) including one or more: robots 220, sensors, computing systems 230, sensors 240, and/or any other suitable components” [0021]. Thus, the disclosure is directed to a grasping robot, i.e., handling system.) comprising:
- at least one robot on which an end effector for gripping an item is arranged (“The robot 220 functions to manipulate an object. The robot can include one or more: end effectors 222, robotic arms 224, and/or any other suitable components” [0022]. Thus, there is a robot with an end effector which manipulates, i.e., grips, an object, i.e., item.);
- a detection device comprising at least one detection unit which is designed to capture an image of an item to be gripped (“The sensors 240 function to sample measurements of a physical scene. The sensors 240 can include: visual sensors (e.g., monocular cameras, stereo cameras, projected light systems, TOF systems, etc.), acoustic sensors, actuation feedback systems, and/or any other suitable sensors” [0024]. Thus, there are visual sensors which sample measurements of a physical scene, i.e., detection units which are designed to capture an image of the item(s) to be gripped. Such visual sensors which are listed satisfy the 112f claim interpretation of detection unit above, in which such sensors are inclusive of cameras.); and
- a control device for controlling the handling system, wherein the control device comprises a data processing system and a non-volatile memory device (“The computing system 230 functions to perform one or more steps of the method, but can additionally and/or alternatively provide any other suitable functionality. The computing system 230 can be local to the robot, remote, and/or otherwise located” [0025]. “Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non-transitory storage medium for execution by, or to control the operation of, data processing apparatus” [0084]. Thus, the computing system which controls the handling system according to the methods of the disclosure comprises computer programs stored on a non-transitory storage medium, i.e., non-volatile memory device, which are further executed by a data processing apparatus, i.e., data processing system.),
wherein a computer program is stored on the non-volatile memory device which comprises commands which, when executed by the data processing system, cause the data processing system to execute one or more control cycles (“Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non-transitory storage medium for execution by, or to control the operation of, data processing apparatus” [0084]. Thus, there is a computer program product stored on a non-transitory medium which performs the disclosed methods. “All or portions of the method can be performed once, iteratively, repeatedly (e.g., for different objects, for different physical scenes, for different time frames, for different sensors), periodically, and/or otherwise performed” [0029]. Thus, provided that the method is performed once, iteratively, periodically, or repeatedly, the method performs one or more control cycles.), wherein each control cycle comprising:
a) receiving image data that represent an image of at least one portion of the item to be gripped, which the image is captured by means of the detection device (“S100 functions to determine a measurement of a physical scene having a container 410 that contains one or more objects 420 to be grasped; example shown in FIG. 4. The measurement can include one measurement, multiple measurements, and/or any other suitable number of measurements. The measurement can be captured by a sensor, retrieved from a database, and/or otherwise determined. The measurement can be an image, depth information, point clouds, video, and/or any other suitable measurement” [0031]. Thus, the method determines a measurement of the container which contains objects to be grasped, and such a received measurement is an image which is captured by a sensor, i.e., the detection device.),
b) determining a target gripping point for the end effector on the item, comprising analyzing the image data (“Selecting a grasp S40 functions to determine a grasp proposal from the set based on a final score associated with each grasp proposal” [0067]. Thus, there is a selected grasp proposal, i.e., target gripping point, which is a result of determining a set of grasp proposals (see Fig. 3, S300) via analyzing the image data.), and
c) generating control signals which cause the at least one robot to grip the item at the target gripping point by means of the end effector (“Executing the grasp trajectory S50 functions to move the robotic arm and/or end effector to grasp an object in the physical scene based on the calculated grasp trajectory associated to the selected grasp proposal” [0069]. Thus, there is a step in the method which moves the end effector to grasp the object based on the selected grasp proposal, i.e., generates control signals causing the robot to grip the item at the target gripping point by means of the end effector.),
the determination of the target gripping point comprises:
b1) analyzing the image data by (“Determining a set of grasp proposals across the workspace S300 functions to determine grasp proposals within the workspace. Each grasp proposal can be associated with a virtual projection of the end effector onto the physical scene (e.g., a “window”), associated with an end effector pose (e.g., location and orientation; x, y, z position and α, β, γ orientation), and/or associated with any other suitable end effector attribute. Grasp proposals can be: predetermined, dynamically determined, randomly determined, and/or otherwise determined” [0033]. Thus, a set of gripping candidates using any of a variety of algorithms to project grasp proposals into the workspace. The workspace is determined in S100 and S200 based on a captured image (see [0031-0032]).); and
b2) selecting a gripping point candidate from the set Me of determined gripping point candidates as the target gripping point depending on one or more specified gripping point selection criteria (“Selecting a grasp S40 functions to determine a grasp proposal from the set based on a final score associated with each grasp proposal… In a first variant, S40 can include selecting a grasp proposal based on the preliminary score. In a second variant, S40 can include selecting a grasp proposal based on one or more heuristic scores. In a third variant, S40 can include selecting a grasp proposal based on a combination of the preliminary and heuristic scores. In a fourth variant, S40 can include selecting a grasp proposal based on the grasp score or a combination with preliminary score and/or heuristic score” [0067]. A target gripping point is determined in S40 depending on a preliminary score, one or more heuristic scores, or any combination of such scores.).
However, Ku ‘601 does not explicitly teach …analyzing the image data by two or more mutually independent gripping point determination algorithms… , wherein selecting the gripping point candidate as the target gripping point comprises:
b2.1) selecting one of the two or more gripping point determination algorithms as a target evaluation algorithm depending on at least one specified algorithm selection criterion, and
b2.2) selecting one of the gripping point candidates determined by the target evaluation algorithm as the target gripping point depending on the at least one gripping point selection criterion, or selecting the gripping point candidate determined by the target evaluation algorithm as the target gripping point,
wherein the at least one specified algorithm selection criterion comprises one or more of the following selection criteria:
- a calculation duration of the gripping point determination algorithms, wherein the gripping point determination algorithm is selected as the target evaluation algorithm which has determined a gripping point candidate the fastest;
- a probability of success when gripping the item at a gripping point candidate determined by the gripping point determination algorithm, wherein the gripping point determination algorithm which has the highest probability of success is selected as the target evaluation algorithm;
- a property or type of the employed end effector; and/or
a confidence value determined by the gripping point determination algorithm.
Hickman, in the same field of endeavor, teaches …analyzing the image data by two or more mutually independent (“As described above, to determine an appropriate image processing algorithm and corresponding parameter set for the robot 301, the cloud processing engine 302 shown in example 300 applies "n" different image processing algorithms 305, 306, 307 to the image included in the image data 304 received from the robot 301. Example 300 also shows each algorithm of the plurality of algorithms 305, 306, 307 being applied to the image multiple times. In operation, each application of the image processing algorithm to the image is executed with a different set of image processing parameters (which may include some similar or same parameters), and each application of a particular algorithm configured with a corresponding parameter set yields a different image processing result” (C 14, L 48-60). Thus, there are two or more mutually independent algorithms which analyze the image data.)…
Hickman does not explicitly apply the algorithms to gripping point determinations, only general image processing. However, given that Ku ‘601 teaches an analysis of gripping point candidates with each iteration of image analysis, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have modified the gripping point determinations of Ku ‘601 to include image analysis with a plurality of algorithms as taught by Hickman with a reasonable expectation of success. One of ordinary skill in the art would have been motivated to make this modification because a higher quality image processing results would lead to a higher quality grasp proposal due to higher quality attribute detections in the respective image of the object to be gripped (Hickman, (C2,L 63-C3,L14)). Such a modification would additionally be a combination of known methods which yield predictable results (see MPEP 2143.I(A)).
Hickman further teaches …wherein selecting the gripping point candidate as the target gripping point comprises:
b2.1) selecting one of the two or more gripping point determination algorithms as a target evaluation algorithm depending on at least one specified algorithm selection criterion (“After generating the plurality of quality scores 310, 313, and 316, the cloud processing engine 302 determines which quality score is the highest. After determining the highest quality score, the cloud processing engine 302 selects the image processing algorithm and the parameter set that was used to generate the image processing result having the highest score, and then the cloud processing engine 302 sends an indication of the selected algorithm and parameter set to the robot 301 via response 317. In situations where multiple image processing results have the highest score, the cloud processing engine 302 may be configured to select one of the multiple highest scoring results based in part on environmental data, task data, and/or object data (as previously described) received from the robot 301” (C16, L23-36). Thus, the target evaluation algorithm with the highest quality score depends on the algorithm selection criteria inclusive of environmental data, task data, and object data.)…
However, Ku ‘601 as modified by Hickman does not explicitly teach …b2.2) selecting one of the gripping point candidates determined by the target evaluation algorithm as the target gripping point depending on the at least one gripping point selection criterion, or selecting the gripping point candidate determined by the target evaluation algorithm as the target gripping point…
However, provided that the modification of Ku ‘601 in view of Hickman combines the gripping point determinations of Ku ‘601 with the algorithm determinations of Hickman such that each algorithm iteration provides a gripping point candidate which is reflected by a quality score (as taught by both Ku ‘601 and Hickman), it would be implied that the selected target algorithm would additionally select a target gripping point which has been determined by the selected target algorithm. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, that Ku ‘601 as modified by Hickman additionally teaches “selecting the gripping point candidate determined by the target evaluation algorithm as the target gripping point” with a reasonable expectation of success.
However, Ku ‘601 as modified by Hickman still does not teach … wherein the at least one specified algorithm selection criterion comprises one or more of the following selection criteria:
- a calculation duration of the gripping point determination algorithms, wherein the gripping point determination algorithm is selected as the target evaluation algorithm which has determined a gripping point candidate the fastest;
- a probability of success when gripping the item at a gripping point candidate determined by the gripping point determination algorithm, wherein the gripping point determination algorithm which has the highest probability of success is selected as the target evaluation algorithm;
- a property or type of the employed end effector; and/or
a confidence value determined by the gripping point determination algorithm.
Rohanimanesh, pertinent to the problem at hand, teaches … wherein the at least one specified algorithm selection criterion (As described in [0061-0070], the method includes an algorithm selection criteria related to a plurality of grasp prediction models, i.e., gripping point determination algorithm, which each predict a plurality of grasps for grasping a plurality of objects.) comprises one or more of the following selection criteria:
- a probability of success when gripping the item at a gripping point candidate determined by the gripping point determination algorithm, wherein the gripping point determination algorithm which has the highest probability of success is selected as the target evaluation algorithm (In the Markov Decision Process (MDP) for selecting the corresponding grasp prediction model, “the plurality of grasps are in an ordered sequence” (see [0061]). Per the teachings of [0066], k pixel positions for each of the plurality of grasp models having the highest success probability scores are identified and then selects the plurality of grasps by solving the MDP (see [0067]). As identified in [0062], the MDP is at least based on pick success. As identified in step (E) of the method ([0063]), the first grasp in the ordered sequence for performing the grasp is identified, i.e., the grasp with the highest probability of success. As identified in step (F) of the method ([0063]), there is further an identification of the end-effector corresponding to the first grasp, i.e., identification of the grasp prediction model which successfully predicted the highest probability of success.);
- a property or type of the employed end effector (As detailed above with regard to step (F) (see [0063]), the selection of the first grasp of the ordered sequence, i.e., the first grasp prediction model, is associated with a type of end effector employed for performing the grasp.); and/or
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have modified the algorithm selection process for gripping determinations as taught by Ku ‘601 in view of Hickman to include the success probability and end effector considerations as taught by Rohanimanesh with a reasonable expectation of success. One of ordinary skill in the art would have been motivated to make such a modification because by determining grasping points via unique algorithms which correspond to specified end effectors and selections based on a success probability, efficiency of picking tasks for highly diverse objects is increased, thereby optimizing and otherwise increasing throughput for robotic bin picking tasks (Rohanimanesh, [0001-0005]).
Claim 14 is rejected under 35 U.S.C. 103 as being unpatentable over Ku ‘601 in view of Hickman, further in view of Rohanimanesh, and further in view of Ku et al. (US 2022/0016767 A1; hereinafter “Ku ‘767”).
Regarding claim 14, Ku ‘601 as modified by Hickman further modified by Rohanimanesh teaches the computer-implemented method according to claim 1.
However, Ku ‘601 as modified does not teach …wherein the handling system comprises a monitoring device which is designed to monitor the gripping of the item at the target gripping point,
the method comprising receiving gripping success data generated by means of the monitoring device, which represent a gripping success when gripping the item at the target gripping point,
wherein a probability of success is determined from the gripping success data, which represents a gripping success to be expected when gripping an item at a gripping point candidate determined by the gripping point determination algorithm which has determined the gripping point candidate selected as the target gripping point,
wherein the probability of success in a subsequent control cycle forms a gripping point selection criterion and/or an algorithm selection criterion.
Ku ‘767, pertinent to the problem at hand, teaches … wherein the handling system comprises a monitoring device which is designed to monitor the gripping of the item at the target gripping point (“Actuation feedback sensors of the actuation feedback system preferably function to enable control of the robot arm (and/or joints therein) and/or the end effector, but can additionally or alternatively be used to determine the outcome (e.g., success or failure) of a grasp attempt” [0032]. Thus, actuation feedback sensors monitor the gripping of an item, specifically the gripping outcome of a grasp attempt at a target gripping point.),
the method comprising receiving gripping success data generated by means of the monitoring device, which represent a gripping success when gripping the item at the target gripping point, wherein a probability of success is determined from the gripping success data, which represents a gripping success to be expected when gripping an item at a gripping point candidate determined by the gripping point determination algorithm which has determined the gripping point candidate selected as the target gripping point (“The predetermined grasp probability score can be determined based on the number of grasp attempt successes and the number of grasp attempt failures (e.g., success divided by total grasp outcomes, failure divided by total grasp outcomes, etc.)” [0040]. Thus, there is a number of grasp attempt successes, i.e., gripping success data, which represent a gripping success when gripping the item at the target gripping point. Subsequently, the grasp probability score, i.e., probability of success, is determined by such attempt successes and determines the likelihood of success to be expected when gripping an item at a gripping point candidate which is selected as the target gripping point (see Fig. 3 and associated descriptions for more details).),
wherein the probability of success in a subsequent control cycle forms a gripping point selection criterion and/or an algorithm selection criterion (“In this variant, the grasp probability score for each subregion is predetermined, wherein the grasp locations for each subregion (determined in S400) can be weighted, retained, and/or removed to calculate the graspability score based on the subregion's grasp probability. The subregion's grasp probability score is preferably determined based on empirical grasping data (e.g., historical success/failure of grasps within the given subregion; success rate for the given subregion; etc.) but can be otherwise determined” [0088]. “Selecting a candidate grasp location S500 can function to select a grasp location that is most likely to result in a grasp success. The candidate grasp location is preferably the grasp location corresponding to the highest graspability score” [0108]. Thus, the probability of success determines a graspability score which is the selection criteria for selecting a candidate grasp location.).
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have modified the gripping point selection criteria of Ku ‘601 to include the probabilities of success as taught by Ku ‘767 with a reasonable expectation of success. One of ordinary skill in the art would have been motivated to make such a modification because by basing the selection criteria on empirical success data, the grasping methods can function to increase the accuracy of grasping an object or additionally increase efficiency or speed at which the object may be grasped (Ku ‘767, [0015]).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. US 2013/0346348 A1 teaches a process for selecting a visual-model class or type based on confidence measures. US 2020/0198129 A1 teaches an algorithm selection criteria based on algorithm speed.
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SIDNEY L MOLNAR whose telephone number is (571)272-2276. The examiner can normally be reached 9 A.M. to 4 P.M. EST Monday-Friday.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Jonathan (Wade) Miles can be reached at (571) 270-7777. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/S.L.M./Examiner, Art Unit 3656
/WADE MILES/Supervisory Patent Examiner, Art Unit 3656
Prosecution Insights