DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Specification
The title of the invention is not descriptive. A new title is required that is clearly indicative of the invention to which the claims are directed.
The following title is suggested: Generating Corrected Training Patches For Determining Grip Success Probability of Robot Hand
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action.
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because the claim limitation(s) uses a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier. Such claim limitation(s) is/are:
an image acquirer, an image compositor, and a patch image generator which are considered nonce terms and/or otherwise fail to denote structure to one of ordinary skill in the art; moreover these elements do not recite structural elements for performing the recited functions.
Because this/these claim limitation(s) is/are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, it/they is/are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof.
If applicant does not intend to have this/these limitation(s) interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation(s) to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation(s) recite(s) sufficient structure to perform the claimed function so as to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claim 1, 8, and 11-13 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Chu {F. J. Chu, R. Xu, and P. A. Vela, “Real-world multiobject, multigrasp detection,” IEEE Robot. Autom. Lett., vol. 3, no. 4, pp. 3355–3362, Oct. 2018, doi: 10.1109/LRA.2018.2852777}.
Claim 1
In regards to claim 1, Chu discloses an image processing device, comprising:
a processor {see Section V Experiments and Evaluation, Fig. 3 structural diagram of the system, and Tables III-VI which are experimental results obtained by a processor implementing the disclosed methods} configured to:
output a correction amount for correcting a position of a target range with respect to one component included in a first patch image when the first patch image is input, the target range being set for the one component, out of a plurality of components included in a stored component image representing the plurality of components stored in a container, the first patch image being cut from an image within the target range
{see Fig. 2, copied below, and Section III Problem Statement. The target range corresponds to the grasp representation in Fig. 2(a) in which the graph representation (target range) describes the location, orientation, and opening distance of a parallel plate gripper and is well within the BRI of “target range” as defined in [0104] of the instant specification as published. The “correction amount” is shown in Fig. 2(b) and is the graph rectangle which has been oriented (corrected) as further discussed in Section III. As to plural components stored in a container see Fig. 1 while noting that Chu is directed to solving the same problem of picking objects from a container whose orientations are random as further discussed in Section I Introduction};
PNG
media_image1.png
326
516
media_image1.png
Greyscale
generate a second patch image including the one component, the second patch image being an image within a range obtained by correcting the target range by the correction amount and cut from the stored component image {see the bounding box grasp configuration in Section III, equation (1) and Section IV Approach including proposal bounding box in A Grasp Proposals. See also IIIC Multigrasp Detection including refining the proposal bounding box to a non-oriented grasp bounding bow (x, y, w, h)}; and
calculate a grip success probability in the case of trying to grip the one component included in the second patch image by a robot hand located in a range where the second patch image is set {see Section IIIA in which the Grasp Proposal Network outputs probability of grasp proposal. See also VB Single-Object Multi-Grasp that calculates grip success probabilities to produce a rank ordered link of grasp candidates and Section VC. See also Table V}.
Claim 8
In regards to claim 8, Chu discloses wherein: the processor is configured to calculate the grip success probability from the second patch image using a convolutional neural network {See Section I, deep convolutional neural network, Fig. 3. Section IIIA in which the Grasp Proposal Network outputs probability of grasp proposal. See also VB Single-Object Multi-Grasp and Section VII discussing the high classification performance of CNNs for improved grasp detection outcomes of the disclosed techniques.
Claim 11
In regards to claim 11, Chu discloses a component gripping system, comprising:
the image processing device according to claim 1 {see above mapping of claim}; and
a robot hand, the image processing device causing being configured to cause the robot hand to grip the component at a position determined based on the calculated grip success probability {see VD Evaluation Metric, Section VI Results in which the highest output score of all grasp candidates is chosen as the final output to achieve high accuracy. Tables I-VI, and Fig. 5c showing the robot and robot hand used for the physical grasping test}.
Claim 12
The rejection of device claim 1 above applies mutatis mutandis to the corresponding limitations of method claim 12 while noting that the rejection above cites to both device and method disclosures.
Claim 13
In regards to claim 13, Chu discloses a component gripping method, comprising:
outputting a correction amount for correcting a position of a target range with respect to one component included in a first patch image when the first patch image is input, the target range being set for the one component, out of a plurality of components included in a stored component image representing the plurality of components stored in a container, the first patch image being cut from an image within the target range {see mapping of claim 1};
generating a second patch image including the one component, the second patch image being an image in a range obtained by correcting the target range by the correction amount and cut from the stored component image {see mapping of claim 1};
calculating a grip success probability in the case of trying to grip the one component included in the second patch image by a robot hand located in a range where the second patch image is set {see mapping of claim 1}; and
causing the robot hand to grip the component at a position determined based on the grip success probability {see mapping of claim 11}.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim 10 is rejected under 35 U.S.C. 103 as being unpatentable over Chu and Mirbach (US 2015/0235351 A1).
Claim 10
In regards to claim 10, Chu discloses composited/combined image that combines a color image with a depth image and a patch image generator configured to generate the first patch image from the stored component image and inputting the first patch image to processor {see above mapping of claim 1}. Chu, however, is not relied upon to disclose the highly conventional process of forming such data such as that expressed in claim 10.
Mirbach is an analogous reference that solves a problem that is reasonably pertinent to the specific problem faced by the inventor which is how to derive a combined/composited dataset that combines image and depth information.
Mirbach teaches
an image acquirer configured to acquire a luminance image representing the plurality of components and a depth image representing the plurality of components {Fig. 1, [0044]-[0047] sensor 12 includes a depth sensor producing a depth image and a camera 20 having the same field of view as the depth sensor to generate intensity (luminance/greyscale) images}; and
an image compositor configured to generate the stored component image by combining the luminance image and the depth image acquired by the image acquirer {see [0050]-[0059] including the bilateral filter that combines depth and intensity/luminance images}.
It would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains to have modified Chu which already discloses composited/combined image that combines a color image with a depth image such that it uses the highly conventional process of forming such data including an image acquirer configured to acquire a luminance image representing the plurality of components and a depth image representing the plurality of components; and an image compositor configured to generate the stored component image by combining the luminance image and the depth image acquired by the image acquirer as taught by Mirbach because Mirbach motivates doing so in [0048] in that camera image may be used to correct areas of the depth image containing no or unreliable depth values by taking the intensity/luminance image as a guidance image, because there is a reasonable expectation of success and/or because doing so merely combines prior art elements according to known methods to yield predictable results.
Allowable Subject Matter
Claims 2-7, 9, and 14-20 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
The following is an examiner’s statement of reasons for allowance:
Although Chu discloses the broadly worded-features of claim 1, neither Chu nor the other prior art of record discloses or fairly suggests the machine learning method of claim 2 including wherein: the processor is configured to learn a relationship of the first patch image and the correction amount, using a position difference between a position determination mask representing a proper position of the component in the target range and the component included in the first patch image as training data and in combination with the remaining elements of claim 1.
Claims 3-7 and 14-20 are allowable due to their dependency upon claim 2.
Although Chu discloses the broadly worded-features of claim 1, neither Chu nor the other prior art of record discloses or fairly suggests the features of claim 9 including wherein: the processor is configured to weight a feature map output from the convolutional neural network by adding an attention mask to the feature map, and the attention mask represents to pay attention to a region extending in a gripping direction in which the robot hand grips the component and passing through a center of the second patch image and a region orthogonal to the gripping direction and passing through the center of the second patch image and in combination with the remaining elements of claim 1 and intervening claim 8.
Any comments considered necessary by applicant must be submitted no later than the payment of the issue fee and, to avoid processing delays, should preferably accompany the issue fee. Such submissions should be clearly labeled “Comments on Statement of Reasons for Allowance.”
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
G. C. Nandi, P. Agarwal, P. Gupta and A. Singh, "Deep Learning Based Intelligent Robot Grasping Strategy," 2018 IEEE 14th International Conference on Control and Automation (ICCA), Anchorage, AK, USA, 2018, pp. 1064-1069, doi: 10.1109/ICCA.2018.8444265 disclosing machine learning for detecting grip success and selects the best rectangle (patch image) for training the robot grasp controller. See Introduction, abstract, and sections 4.2.2, 4.5, and 4.6.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Michael R Cammarata whose telephone number is (571)272-0113. The examiner can normally be reached M-Th 7am-5pm EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Matthew Bella can be reached at 571-272-7778. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MICHAEL ROBERT CAMMARATA/Primary Examiner, Art Unit 2667