DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 05/08/2024 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claim 21 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
101 Analysis – Step 1
Claim 21 is directed to a system that communicates with a robot (i.e., a process). Therefore, claim 21 is within at least one of the four statutory categories.
101 Analysis – Step 2A, Prong I
Regarding Prong I of the Step 2A analysis in the 2019 PEG, the claims are to be analyzed to determine whether they recite subject matter that falls within one of the follow groups of abstract ideas: a) mathematical concepts, b) certain methods of organizing human activity, and/or c) mental processes.
Independent claim 21 includes limitations that recite an abstract idea (emphasized below) and will be used as a representative claim for the remainder of the 101 rejection. Claim 21 recites:
A computing system comprising:
a communication interface configured to communicate with a robot and with a camera; and at least one processing circuit configured, when an object is or has been in a field of view of the
camera, to:
receive image information, generated by the camera, representing the object;
identify one or more object recognition templates corresponding to an object or an object
type;
select a primary object template from among the one or more object recognition templates
based on matching the image information with the one or more object recognition templates;
generate a primary candidate region based on the primary object template;
determine at least one of:
(i) a subset of one or more remaining matching object recognition templates, or
(ii) an unmatched region of the image information that is adjacent to the primary candidate
region;
generate a safety volume list in response to a determination of the subset or the unmatched
region,
wherein the safety volume list includes at least one of:
(i) the unmatched region, or
(ii) one or more additional candidate regions based on the subset, wherein the one or more additional candidate regions estimate object boundary locations for the object or estimate locations in the field of view that are occupied by the object; and
perform motion planning for interaction between the robot and the object based on the primary candidate region and the safety volume list.
The examiner submits that the foregoing bolded limitation(s) constitute mental processes. For example, “identify…”, “select…”, “generate…”, and “determine…” in the context of this claim encompasses performing observation or judgment to obtain certain results which can be used to control the robot. Accordingly, the claim recites at least one abstract idea.
101 Analysis – Step 2A, Prong II
Regarding Prong II of the Step 2A analysis in the 2019 PEG, the claims are to be analyzed to determine whether the claim, as a whole, integrates the abstract into a practical application. As noted in the 2019 PEG, it must be determined whether any additional elements in the claim beyond the abstract idea integrate the exception into a practical application in a manner that imposes a meaningful limit on the judicial exception. The courts have indicated that additional elements merely using a computer to implement an abstract idea, adding insignificant extra solution activity, or generally linking use of a judicial exception to a particular technological environment or field of use do not integrate a judicial exception into a “practical application.”
In the present case, the additional limitations beyond the above-noted abstract idea are as follows (where the underlined portions are the “additional limitations” while the bolded portions continue to represent the “abstract idea”):
A method for autonomous solar installation, the method comprising:
A computing system comprising:
a communication interface configured to communicate with a robot and with a camera; and
at least one processing circuit configured, when an object is or has been in a field of view of the
camera, to:
receive image information, generated by the camera, representing the object;
identify one or more object recognition templates corresponding to an object or an object
type;
select a primary object template from among the one or more object recognition templates
based on matching the image information with the one or more object recognition templates;
generate a primary candidate region based on the primary object template;
determine at least one of:
(i) a subset of one or more remaining matching object recognition templates, or
(ii) an unmatched region of the image information that is adjacent to the primary candidate
region;
generate a safety volume list in response to a determination of the subset or the unmatched
region,
wherein the safety volume list includes at least one of:
(i) the unmatched region, or
(ii) one or more additional candidate regions based on the subset, wherein the one or more additional candidate regions estimate object boundary locations for the object or estimate locations in the field of view that are occupied by the object; and
perform motion planning for interaction between the robot and the object based on the primary candidate region and the safety volume list.
For the following reason(s), the examiner submits that the above identified additional limitations do not integrate the above-noted abstract idea into a practical application.
Regarding the additional limitations of “a communication interface configured to communicate …”, “at least one processing circuit…”, and “receive image information …”, the examiner submits that these limitations are insignificant extra-solution activities which does not integrate the abstract idea into a practical application. In general, the obtaining an image of an in-progress solar installation is recited at a high level of generality and amounts to mere data gathering, which is a form of insignificant extra-solution activity. Lastly, the “perform motion planning …” merely describes how to generally “apply” the otherwise mental processes in a generic or general purpose robot control environment. The robot control system is recited at a high level of generality.
Thus, taken alone, the additional elements do not integrate the abstract idea into a practical application. Further, looking at the additional limitation(s) as an ordered combination or as a whole, the limitation(s) add nothing that is not already present when looking at the elements taken individually. For instance, there is no indication that the additional elements, when considered as a whole, reflect an improvement in the functioning of a computer or an improvement to another technology or technical field, apply or use the above-noted judicial exception to effect a particular treatment or prophylaxis for a disease or medical condition, implement/use the above-noted judicial exception with a particular machine or manufacture that is integral to the claim, effect a transformation or reduction of a particular article to a different state or thing, or apply or use the judicial exception in some other meaningful way beyond generally linking the use of the judicial exception to a particular technological environment, such that the claim as a whole is not more than a drafting effort designed to monopolize the exception (MPEP § 2106.05). Accordingly, the additional limitation(s) do/does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea.
Analysis – Step 2B
Regarding Step 2B of the 2019 PEG, representative independent claim 1 does not include additional elements (considered both individually and as an ordered combination) that are sufficient to amount to significantly more than the judicial exception for the same reasons to those discussed above with respect to determining that the claim does not integrate the abstract idea into a practical application.
Further, a conclusion that an additional element is insignificant extra-solution activity in Step 2A should be re-evaluated in Step 2B to determine if they are more than what is well-understood, routine, conventional activity in the field. The additional limitations of “obtaining an image of an in-progress solar installation …” are well-understood, routine, and conventional activities and the specification does not provide any indication that controlling the robot is anything other than a conventional computer within a robot. MPEP 2106.05(d)(II), and the cases cited therein, including Intellectual Ventures I, LLC v. Symantec Corp., 838 F.3d 1307, 1321 (Fed. Cir. 2016), TLI Communications LLC v. AV Auto. LLC, 823 F.3d 607, 610 (Fed. Cir. 2016), and OIP Techs., Inc., v. Amazon.com, Inc., 788 F.3d 1359, 1363 (Fed. Cir. 2015), indicate that mere collection or receipt of data over a network is a well‐understood, routine, and conventional function when it is claimed in a merely generic manner.
Dependent claim(s) 22-37 do not recite any further limitations that cause the claim(s) to be patent eligible. Rather, the limitations of dependent claims are directed toward additional aspects of the judicial exception and/or well-understood, routine and conventional additional elements that do not integrate the judicial exception into a practical application. Therefore, dependent claims 22-37 are not patent eligible under the same rationale as provided for in the rejection of [independent claim].
Claim 38 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
101 Analysis – Step 1
Claim 38 is directed to a non-transitory computer-readable medium that when executed by at least one processing circuit of a computing system, causes the at least one processing circuit to communicates with a robot (i.e., a manufacture). Therefore, claim 38 is within at least one of the four statutory categories.
101 Analysis – Step 2A, Prong I
Regarding Prong I of the Step 2A analysis in the 2019 PEG, the claims are to be analyzed to determine whether they recite subject matter that falls within one of the follow groups of abstract ideas: a) mathematical concepts, b) certain methods of organizing human activity, and/or c) mental processes.
Independent claim 38 includes limitations that recite an abstract idea (emphasized below) and will be used as a representative claim for the remainder of the 101 rejection. Claim 38 recites:
A non-transitory computer-readable medium having instructions that, when executed by at least one processing circuit of a computing system, causes the at least one processing circuit to:
a communication interface configured to communicate with a robot and with a camera; and at least one processing circuit configured, when an object is or has been in a field of view of the
camera, to:
receive image information, generated by a camera, representing an object in a field of view of the camera, wherein the computing system is configured to communicate with: (i) a robot, and (ii) the camera;
identify one or more object recognition templates corresponding to an object or an object
type;
select, a primary object template from among the one or more object recognition templates
based on matching the image information with the one or more object recognition templates ;
generate a primary candidate region based on the primary object template;
determine at least one of:
(i) a subset of one or more remaining matching object recognition templates, or
(ii) an unmatched region of the image information that is adjacent to the primary candidate region;
generate a safety volume list in response to a determination of the subset, or the unmatched region,
wherein the safety volume list includes at least one of:
(i) the unmatched region, or
(ii) one or more additional candidate regions based on the subset, wherein the one or more additional candidate regions estimate object boundary locations for the object or estimate locations in the field of view that are occupied by the object; and
perform motion planning for interaction between the robot and the object based on the primary candidate region and the safety volume list.
The examiner submits that the foregoing bolded limitation(s) constitute mental processes. For example, “identify…”, “select…”, “generate…”, and “determine…” in the context of this claim encompasses performing observation or judgment to obtain certain results which can be used to control the robot. Accordingly, the claim recites at least one abstract idea.
101 Analysis – Step 2A, Prong II
Regarding Prong II of the Step 2A analysis in the 2019 PEG, the claims are to be analyzed to determine whether the claim, as a whole, integrates the abstract into a practical application. As noted in the 2019 PEG, it must be determined whether any additional elements in the claim beyond the abstract idea integrate the exception into a practical application in a manner that imposes a meaningful limit on the judicial exception. The courts have indicated that additional elements merely using a computer to implement an abstract idea, adding insignificant extra solution activity, or generally linking use of a judicial exception to a particular technological environment or field of use do not integrate a judicial exception into a “practical application.”
In the present case, the additional limitations beyond the above-noted abstract idea are as follows (where the underlined portions are the “additional limitations” while the bolded portions continue to represent the “abstract idea”):
A non-transitory computer-readable medium having instructions that, when executed by at least one processing circuit of a computing system, causes the at least one processing circuit to:
a communication interface configured to communicate with a robot and with a camera; and at least one processing circuit configured, when an object is or has been in a field of view of the
camera, to:
receive image information, generated by a camera, representing an object in a field of view of the camera, wherein the computing system is configured to communicate with: (i) a robot, and (ii) the camera;
identify one or more object recognition templates corresponding to an object or an object
type;
select, a primary object template from among the one or more object recognition templates
based on matching the image information with the one or more object recognition templates ;
generate a primary candidate region based on the primary object template;
determine at least one of:
(i) a subset of one or more remaining matching object recognition templates, or
(ii) an unmatched region of the image information that is adjacent to the primary candidate region;
generate a safety volume list in response to a determination of the subset, or the unmatched region,
wherein the safety volume list includes at least one of:
(i) the unmatched region, or
(ii) one or more additional candidate regions based on the subset, wherein the one or more additional candidate regions estimate object boundary locations for the object or estimate locations in the field of view that are occupied by the object; and
perform motion planning for interaction between the robot and the object based on the primary candidate region and the safety volume list.
For the following reason(s), the examiner submits that the above identified additional limitations do not integrate the above-noted abstract idea into a practical application.
Regarding the additional limitations of “a communication interface configured to communicate …”, “at least one processing circuit…”, and “receive image information …”, the examiner submits that these limitations are insignificant extra-solution activities which does not integrate the abstract idea into a practical application. In general, the obtaining an image of an in-progress solar installation is recited at a high level of generality and amounts to mere data gathering, which is a form of insignificant extra-solution activity. Lastly, the “perform motion planning …” merely describes how to generally “apply” the otherwise mental processes in a generic or general purpose robot control environment. The robot control system is recited at a high level of generality.
Thus, taken alone, the additional elements do not integrate the abstract idea into a practical application. Further, looking at the additional limitation(s) as an ordered combination or as a whole, the limitation(s) add nothing that is not already present when looking at the elements taken individually. For instance, there is no indication that the additional elements, when considered as a whole, reflect an improvement in the functioning of a computer or an improvement to another technology or technical field, apply or use the above-noted judicial exception to effect a particular treatment or prophylaxis for a disease or medical condition, implement/use the above-noted judicial exception with a particular machine or manufacture that is integral to the claim, effect a transformation or reduction of a particular article to a different state or thing, or apply or use the judicial exception in some other meaningful way beyond generally linking the use of the judicial exception to a particular technological environment, such that the claim as a whole is not more than a drafting effort designed to monopolize the exception (MPEP § 2106.05). Accordingly, the additional limitation(s) do/does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea.
Analysis – Step 2B
Regarding Step 2B of the 2019 PEG, representative independent claim 1 does not include additional elements (considered both individually and as an ordered combination) that are sufficient to amount to significantly more than the judicial exception for the same reasons to those discussed above with respect to determining that the claim does not integrate the abstract idea into a practical application.
Further, a conclusion that an additional element is insignificant extra-solution activity in Step 2A should be re-evaluated in Step 2B to determine if they are more than what is well-understood, routine, conventional activity in the field. The additional limitations of “obtaining an image of an in-progress solar installation …” are well-understood, routine, and conventional activities and the specification does not provide any indication that controlling the robot is anything other than a conventional computer within a robot. MPEP 2106.05(d)(II), and the cases cited therein, including Intellectual Ventures I, LLC v. Symantec Corp., 838 F.3d 1307, 1321 (Fed. Cir. 2016), TLI Communications LLC v. AV Auto. LLC, 823 F.3d 607, 610 (Fed. Cir. 2016), and OIP Techs., Inc., v. Amazon.com, Inc., 788 F.3d 1359, 1363 (Fed. Cir. 2015), indicate that mere collection or receipt of data over a network is a well‐understood, routine, and conventional function when it is claimed in a merely generic manner.
Dependent claim 39 do not recite any further limitations that cause the claim(s) to be patent eligible. Rather, the limitations of dependent claims are directed toward additional aspects of the judicial exception and/or well-understood, routine and conventional additional elements that do not integrate the judicial exception into a practical application. Therefore, dependent claim 39 is not patent eligible under the same rationale as provided for in the rejection of [independent claim].
Claim 40 is rejected under 35 U.S.C. 101 under similar rationale as claim 21 and 38.
Therefore, Claim(s) 21-40 are ineligible under 35 USC §101.
Double Patenting
The nonstatutory double patenting rejection is based on a judicially created doctrine grounded in public policy (a policy reflected in the statute) so as to prevent the unjustified or improper timewise extension of the “right to exclude” granted by a patent and to prevent possible harassment by multiple assignees. A nonstatutory double patenting rejection is appropriate where the conflicting claims are not identical, but at least one examined application claim is not patentably distinct from the reference claim(s) because the examined application claim is either anticipated by, or would have been obvious over, the reference claim(s). See, e.g., In re Berg, 140 F.3d 1428, 46 USPQ2d 1226 (Fed. Cir. 1998); In re Goodman, 11 F.3d 1046, 29 USPQ2d 2010 (Fed. Cir. 1993); In re Longi, 759 F.2d 887, 225 USPQ 645 (Fed. Cir. 1985); In re Van Ornum, 686 F.2d 937, 214 USPQ 761 (CCPA 1982); In re Vogel, 422 F.2d 438, 164 USPQ 619 (CCPA 1970); In re Thorington, 418 F.2d 528, 163 USPQ 644 (CCPA 1969).
A timely filed terminal disclaimer in compliance with 37 CFR 1.321(c) or 1.321(d) may be used to overcome an actual or provisional rejection based on nonstatutory double patenting provided the reference application or patent either is shown to be commonly owned with the examined application, or claims an invention made as a result of activities undertaken within the scope of a joint research agreement. See MPEP § 717.02 for applications subject to examination under the first inventor to file provisions of the AIA as explained in MPEP § 2159. See MPEP § 2146 et seq. for applications not subject to examination under the first inventor to file provisions of the AIA . A terminal disclaimer must be signed in compliance with 37 CFR 1.321(b).
The filing of a terminal disclaimer by itself is not a complete reply to a nonstatutory double patenting (NSDP) rejection. A complete reply requires that the terminal disclaimer be accompanied by a reply requesting reconsideration of the prior Office action. Even where the NSDP rejection is provisional the reply must be complete. See MPEP § 804, subsection I.B.1. For a reply to a non-final Office action, see 37 CFR 1.111(a). For a reply to final Office action, see 37 CFR 1.113(c). A request for reconsideration while not provided for in 37 CFR 1.113(c) may be filed after final for consideration. See MPEP §§ 706.07(e) and 714.13.
The USPTO Internet website contains terminal disclaimer forms which may be used. Please visit www.uspto.gov/patent/patents-forms. The actual filing date of the application in which the form is filed determines what form (e.g., PTO/SB/25, PTO/SB/26, PTO/AIA /25, or PTO/AIA /26) should be used. A web-based eTerminal Disclaimer may be filled out completely online using web-screens. An eTerminal Disclaimer that meets all requirements is auto-processed and approved immediately upon submission. For more information about eTerminal Disclaimers, refer to www.uspto.gov/patents/apply/applying-online/eterminal-disclaimer.
Claim 1 is rejected on the ground of nonstatutory double patenting as being unpatentable over claim 1 of U.S. Patent No. US 11900652 B2. Although the claims at issue are not identical, they are not patentably distinct from each other because selecting a primary object template from among the one or more object recognition templates based on matching the image information with the one or more object recognition templates is a broader limitation of selecting as a primary detection hypothesis, a detection hypothesis from among the set of one or more detection hypotheses, wherein the primary detection hypothesis is associated with a matching object recognition template of the set of one or more matching object recognition templates, wherein the detection hypothesis that is selected as the primary detection hypothesis has a confidence value which is highest among a set of one or more respective confidence values, wherein the set of one or more respective confidence values are associated with the set of one or more detection hypotheses, and indicate respective degrees by which the image information matches the set of one or more matching object recognition templates associated with the set of one or more detection hypotheses. And removing inherent and/or unnecessary limitation(s)/step(s) or adding an element and its function would be within the level of one of ordinary skill in the art. It is well settled that the adding or deleting of an element and its function(s) in the claim of the present application are an obvious expedient if the remaining elements perform the same function as before. In re Karlson, 136 USPQ 184 (CCPA 1963). Also note Ex parte Rainu, 168 USPQ 375 (Bd. App. 1969). Omission of a referenced element or step whose function is not needed would be obvious to one of ordinary skill in the art. Examiner further notes wherein although the claims are not identical (slightly broader), they are commensurate in scope to the claim limitations provided in the issued U.S. Patent, and likewise would anticipate the currently provided claim limitations.
Instant Application No. 18437946
U.S. Patent No. US 11900652 B2
21. A computing system comprising: a communication interface configured to communicate with a robot and with a camera; and at least one processing circuit configured, when an object is or has been in a field of view of the camera, to: receive image information, generated by the camera, representing the object; identify one or more object recognition templates corresponding to an object or an object type;
1. A computing system comprising: a communication interface configured to communicate with a robot and with a camera having a camera field of view ; and at least one processing circuit configured, when an object is or has been in the camera field of view, to: receive image information representing the object, wherein the image information is generated by the camera; identify a set of one or more matching object recognition templates, which are one or more object recognition templates that satisfy a predefined template matching condition when compared against the image information, wherein the set of one or more matching object recognition templates are associated with a set of one or more detection hypotheses, which are one or more respective estimates on which object or object type is represented by the image information;
select a primary object template from among the one or more object recognition templates based on matching the image information with the one or more object recognition templates;
select, as a primary detection hypothesis, a detection hypothesis from among the set of one or more detection hypotheses, wherein the primary detection hypothesis is associated with a matching object recognition template of the set of one or more matching object recognition templates, wherein the detection hypothesis that is selected as the primary detection hypothesis has a confidence value which is highest among a set of one or more respective confidence values, wherein the set of one or more respective confidence values are associated with the set of one or more detection hypotheses, and indicate respective degrees by which the image information matches the set of one or more matching object recognition templates associated with the set of one or more detection hypotheses;
generate a primary candidate region based on the primary object template; determine at least one of: (i) a subset of one or more remaining matching object recognition templates, or (ii) an unmatched region of the image information that is adjacent to the primary candidate region;
generate, as a primary candidate region, a candidate region which estimates object boundary locations for the object or estimates which locations in the camera field of view are occupied by the object, wherein the primary candidate region is generated based on the matching object recognition template associated with the primary detection hypothesis; determine, in addition to the matching object recognition template associated with the primary detection hypothesis, determine at least one of:(i) whether the set of one or more matching object recognition templates has, in addition to the matching object recognition template associated with the primary detection hypothesis, a subset of one or more remaining matching object recognition templates that also satisfy the predefined template matching condition when compared against the image information, or (ii) whether the image information has a portion representing an unmatched region which is adjacent to the primary candidate region and which fails to satisfy the predefined template matching condition;
generate a safety volume list in response to a determination of the subset or the unmatched region, wherein the safety volume list includes at least one of: (i) the unmatched region, or (ii) one or more additional candidate regions based on the subset, wherein the one or more additional candidate regions estimate object boundary locations for the object or estimate locations in the field of view that are occupied by the object;
generate a safety volume list in response to a determination that there is the subset of one or more remaining matching object recognition templates, or that the image information has the portion representing the unmatched region, generate a wherein the safety volume list, which is a list that describes at least one of:(i) the unmatched region, or (ii) one or more additional candidate regions that also estimate object boundary locations for the object or estimate which locations are occupied by the object, wherein the one or more additional candidate regions are generated based on the subset of one or more remaining matching object recognition templates;
and perform motion planning for interaction between the robot and the object based on the primary candidate region and the safety volume list.
and perform motion planning based on the primary candidate region and based on the safety volume list, wherein the motion planning is for robot interaction between the robot and the object for gripping or picking up the object and moving the object from the occupied location of the object to a destination location.
22. The computing system of claim 21, wherein the at least one processing circuit is further configured to determine a bounding region encompassing the primary candidate region and at least one of: (i) the one or more additional candidate regions or (ii) the unmatched region, and to perform the motion planning for a trajectory associated with a robot end effector apparatus of the robot based on the bounding region.
2. The computing system of claim 1, wherein the at least one processing circuit is configured to determine a bounding region which encompasses the primary candidate region and at least one of: (i) the one or more additional candidate regions or (ii) the unmatched region, wherein performing the motion planning includes determining a trajectory associated with a robot end effector apparatus based on the bounding region.
23. The computing system of claim 22, wherein the at least one processing circuit is further configured to perform the motion planning including determining robot gripping motion based on the primary candidate region.
3. The computing system of claim 2, wherein performing the motion planning includes determining robot gripping motion based on the primary candidate region, and determining the trajectory based on the bounding region.
24. The computing system of claim 21, wherein for the subset of one or more remaining matching object recognition templates, the at least one processing circuit is further configured to: determine whether a respective confidence value associated with each of the subset of one or more remaining matching object recognition templates is within a confidence similarity threshold relative to a confidence value associated with the primary object template, include, in the safety volume list, a respective candidate region associated with the each of the subset of one or more remaining matching object recognition templates, in response to a determination that the respective confidence value is within the confidence similarity threshold, and supplement the one or more additional regions of the safety volume list with the respective candidate region.
4. The computing system of claim 1, wherein the set of one or more detection hypotheses include, in addition to the primary detection hypothesis, a subset of one or more remaining detection hypotheses which are associated with the subset of one or more remaining matching object recognition templates, wherein the at least one processing circuit is configured, for each detection hypothesis of the subset of one or more remaining detection hypotheses, to: determine whether a respective confidence value associated with the detection hypothesis is within a predefined confidence similarity threshold relative to the confidence value associated with the primary detection hypothesis, wherein the at least one processing circuit is configured to include, in the safety volume list, a respective candidate region associated with the detection hypothesis in response to a determination that the respective confidence value associated with the detection hypothesis is within the predefined confidence similarity threshold relative to the confidence value associated with the primary detection hypothesis, such that the respective candidate region is part of the one or more additional regions of the safety volume list.
25. The computing system of claim 24, wherein each candidate region of the one or more additional candidate regions in the safety volume list is associated with a confidence value that is within the confidence similarity threshold.
5. The computing system of claim 4, wherein each candidate region of the one or more additional candidate regions in the safety volume list is associated with a respective detection hypothesis which has a confidence value that is within the predefined confidence similarity threshold relative to the confidence value associated with the primary detection hypothesis.
26. The computing system of claim 24, wherein each candidate region of the one or more additional candidate regions in the safety volume list is associated with a confidence value that is greater than or equal to a template matching threshold
6. The computing system of claim 4, wherein each candidate region of the one or more additional candidate regions in the safety volume list is associated with a respective detection hypothesis which has a confidence value that is greater than or equal to a predefined template matching threshold.
27. The computing system of claim 21, wherein the subset of one or more remaining matching object recognition templates includes a plurality of matching object recognition templates associated with a plurality of respective candidate regions, wherein the at least one processing circuit is further configured, for each candidate region of the plurality of candidate regions, to: determine a respective amount of overlap between the candidate region and the primary candidate region; determine whether the respective amount of overlap is equal to or exceeds an overlap threshold; and include the candidate region in the one or more additional candidate regions of the safety volume list, in response to the respective amount of overlap being equal to or exceeding the overlap threshold.
7. The computing system of claim 1, wherein the subset of one or more remaining matching object recognition templates include a plurality of matching object recognition templates associated with a plurality of respective candidate regions, wherein the at least one processing circuit is configured, for each candidate region of the plurality of candidate regions, to: determine a respective amount of overlap between the candidate region and the primary candidate region; determine whether the respective amount of overlap is equal to or exceeds a predefined overlap threshold, wherein the at least one processing circuit is configured to include the candidate region in the safety volume list in response to a determination that the amount of overlap is equal to or exceeds the predefined overlap threshold, such that the candidate region is part of the one or more additional candidate regions of the safety volume list.
28. The computing system of claim 21, wherein the image information includes 2D image information, and wherein the primary object template comprises a set of visual description information which is determined by the at least one processing circuit to satisfy a template matching condition when compared against the 2D image information.
8. The computing system of claim 1, wherein the image information includes 2D image information, and wherein the matching object recognition template associated with the primary detection hypothesis includes a set of visual description information which is determined by the at least one processing circuit to satisfy the predefined template matching condition when compared against the 2D image information.
29. The computing system of claim 28, wherein at least one object recognition template of the subset of one or more remaining object recognition templates has a set of visual description information that is determined by the at least one processing circuit to satisfy the template matching condition when compared against the 2D image information, and wherein the at least one processing circuit is further configured to generate the safety volume list based on at least one of the unmatched region, the one or more additional candidate regions, or the at least one object recognition template.
9. The computing system of claim 8, wherein at least one matching object recognition template of the subset of one or more remaining matching object recognition templates has a respective set of visual description information that is also determined by the at least one processing circuit to satisfy the predefined template matching condition when compared against the 2D image information, and wherein the at least one processing circuit is configured to generate the safety volume list based on the at least one matching object recognition template.
30. The computing system of claim 29, wherein the primary object template includes a respective set of structure description information that indicates a first object size, and wherein the at least one object recognition template includes a respective set of structure description information that indicates a second object size different than the first object size
10. The computing system of claim 9, wherein the matching object recognition template associated with the primary detection hypothesis includes a respective set of structure description information that indicates a first object size, and wherein the at least one matching object recognition template includes a respective set of structure description information that indicates a second object size different than the first object size.
31. The computing system of claim 28, wherein the image information further includes 3D image information, and wherein at least one object recognition template of the subset of one or more remaining object recognition templates has a respective set of structure description information that is determined by the at least one processing circuit to satisfy the template matching condition when compared against the 3D image information, and wherein the at least one processing circuit is further configured to generate the safety volume list based on at least one of the unmatched region, the one or more additional candidate regions, or the at least one object recognition template.
11. The computing system of claim 8, wherein the image information further includes 3D image information, and wherein at least one object recognition template of the subset of one or more remaining matching object recognition templates has a respective set of structure description information that is determined by the at least one processing circuit to satisfy the predefined template matching condition when compared against the 3D image information, and wherein the at least one processing circuit is configured to generate the safety volume list based on the at least one object recognition template.
32. The computing system of claim 28, wherein the at least one processing circuit is further configured, when the one or more object recognition templates are part of a plurality of object recognition templates stored in a template storage space, to: determine whether the plurality of object recognition templates has, in addition to the primary object template, at least one object recognition template that satisfies a template similarity condition when compared against the primary object template; and in response to a determination that the at least one object recognition template satisfies the template similarity condition, generate the safety volume list based on at least one of the unmatched region, the one or more additional candidate regions, or the at least one object recognition template.
12. The computing system of claim 8, wherein the matching object recognition template associated with the primary detection hypothesis is a first matching object recognition template among the set of one or more matching object recognition templates, wherein the at least one processing circuit is configured, when the set of one or more matching object recognition templates are part of a plurality of object recognition templates stored in a template storage space, to: determine whether the plurality of object recognition templates has, in addition to the first matching object recognition template, at least one object recognition template which satisfies a predefined template similarity condition when compared against the first matching object recognition template; and in response to a determination that the plurality of object recognition templates includes the at least one object recognition template which satisfies the predefined template similarity condition when compared against the first matching object recognition template, generate the safety volume list based on the at least one object recognition template.
33. The computing system of claim 21, wherein the primary candidate region represents a first manner of aligning the image information with the primary object template, and wherein the at least one processing circuit is further configured to include in the safety volume list another candidate region which represents a second manner of aligning the image information with the primary object template.
13. The computing system of claim 1, wherein the primary candidate region represents a first manner of aligning the image information with the matching object recognition template associated with the primary detection hypothesis, and wherein the at least one processing circuit is configured to include in the safety volume list another candidate region which represents a second manner of aligning the image information with the matching object recognition template.
34. The computing system of claim 21, wherein the at least one processing circuit is further configured to: identify a first set of image corners or a first set of image edges represented by the image information; identify a first image region located between the first set of image corners or the first set of image edges, wherein the primary object template is determined by the at least one processing circuit to satisfy a template matching condition when compared against the first image region, the primary object template being a first object recognition template among the one or more object recognition templates; identify, based on the image information, a second set of image corners or a second set of image edges, wherein the second set of image corners include at least one image corner which is part of the first set of image corners and include at least one image corner which is outside of the first image region, and wherein the second set of image edges include at least one image edge which is part of the first set of image edges and include at least one image edge which is outside the first image region; and identify a second image region located between the second set of image corners or the second set of image edges, wherein the second image region extends beyond the first image region, and wherein the one or more object recognition templates includes a second matching object recognition template, which is determined by the at least one processing circuit to satisfy the template matching condition when compared against the second image region, wherein the at least one processing circuit is further configured to generate the primary candidate region based on the first object recognition template, and to generate at least one candidate region in the safety volume list based on the second matching object recognition template.
14. The computing system of claim 1, wherein the at least one processing circuit is configured to: identify a first set of image corners or a first set of image edges represented by the image information; identify a first image region, which is an image region located between the first set of image corners or the first set of image edges, wherein the matching object recognition template associated with the primary detection hypothesis is determined by the at least one processing circuit to satisfy the predefined matching condition when compared against the first image region, the matching object recognition template being a first matching object recognition template among the set of one or more matching object recognition templates; identify, based on the image information, a second set of image corners or a second set of image edges, wherein the second set of image corners include at least one image corner which is part of the first set of image corners and include at least one image corner which is outside of the first image region, and wherein the second set of image edges include at least one image edge which is part of the first set of image edges and include at least one image edge which is outside the first image region; identify a second image region, which is an image region located between the second set of image corners or the second set of image edges, wherein the second image region extends beyond the first image region, and wherein the set of one or more matching object recognition templates includes a second matching object recognition template, which is determined by the at least one processing circuit to satisfy the predefined template matching condition when compared against the second image region, wherein the at least one processing circuit is configured to generate the primary candidate region based on the first matching object recognition template, and to generate at least one candidate region in the safety volume list based on the second matching object recognition template.
35. The computing system of claim 21, wherein the at least one processing circuit is further configured to generate a new object recognition template based on the unmatched region, in response to a determination that the image information has the unmatched region.
15. The computing system of claim 1, wherein the at least one processing circuit is configured, in response to a determination that the image information has the portion representing the unmatched region, to generate a new object recognition template based on the unmatched region.
36. The computing system of claim 21, wherein the primary candidate region represents a first orientation for an object shape described by the primary object template, and wherein the at least one processing circuit is further configured to add, to the safety volume list, a candidate region that represents a second orientation for the object shape, the second orientation being perpendicular to the first orientation.
16. The computing system of claim 1, wherein the primary candidate region is a region representing a first orientation for an object shape described by the matching object recognition template associated with the primary detection hypothesis, and wherein the at least one processing circuit is configured to add, to the safety volume list, a candidate region which represents a second orientation for the object shape, the second orientation being perpendicular to the first orientation.
37. The computing system of claim 21, wherein the at least one processing circuit is further configured to add, to the safety volume list, a candidate region that represents a maximum object height.
17. The computing system of claim 1, wherein the at least one processing circuit is configured to add, to the safety volume list, a candidate region which represents a predefined maximum object height.
38. A non-transitory computer-readable medium having instructions that, when executed by at least one processing circuit of a computing system, causes the at least one processing circuit to: receive image information, generated by a camera, representing an object in a field of view of the camera, wherein the computing system is configured to communicate with: (i) a robot, and (ii) the camera; identify one or more object recognition templates corresponding to an object or an object type; select, a primary object template from among the one or more object recognition templates based on matching the image information with the one or more object recognition templates ; generate a primary candidate region based on the primary object template; determine at least one of: (i) a subset of one or more remaining matching object recognition templates, or (ii) an unmatched region of the image information that is adjacent to the primary candidate region; generate a safety volume list in response to a determination of the subset, or the unmatched region, wherein the safety volume list includes at least one of: (i) the unmatched region, or (ii) one or more additional candidate regions based on the subset, wherein the one or more additional candidate regions estimate object boundary locations for the object or estimate locations in the field of view that are occupied by the object; and perform motion planning for interaction between the robot and the object based on the primary candidate region and the safety volume list.
18. A non-transitory computer-readable medium having instructions that, when executed by at least one processing circuit of a computing system, causes the at least one processing circuit to: receive image information by the at least one processing circuit of the computing system, wherein the computing system is configured to communicate with: (i) a robot, and (ii) a camera having a camera field of view, wherein the image information is for representing an object in the camera field of view, and is generated by the camera; identify a set of one or more matching object recognition templates, which are one or more object recognition templates that satisfy a predefined template matching condition when compared against the image information, wherein the set of one or more matching object recognition templates are associated with a set of one or more detection hypotheses, which are one or more respective estimates on which object or object type is represented by the image information; select, as a primary detection hypothesis, a detection hypothesis from among the set of one or more detection hypotheses, wherein the primary detection hypothesis is associated with a matching object recognition template of the set of one or more matching object recognition templates, wherein the detection hypothesis that is selected as the primary detection hypothesis has a confidence value which is highest among a set of one or more respective confidence values, wherein the set of one or more respective confidence values are associated with the set of one or more detection hypotheses, and indicate respective degrees by which the image information matches the set of one or more matching object recognition templates associated with the set of one or more detection hypotheses; generate, as a primary candidate region, a candidate region which estimates object boundary locations for the object or estimates which locations in the camera field of view are occupied by the object, wherein the primary candidate region is generated based on the matching object recognition template associated with the primary detection hypothesis; determine, in addition to the matching object recognition template associated with the primary detection hypothesis, determine at least one of:(i) whether the set of one or more matching object recognition templates has, in addition to the matching object recognition template associated with the primary detection hypothesis, a subset of one or more remaining matching object recognition templates that also satisfy the predefined template matching condition when compared against the image information, or (ii) whether the image information has a portion representing an unmatched region which is adjacent to the primary candidate region and which fails to satisfy the predefined template matching condition; generate a safety volume list in response to a determination that there is the subset of one or more remaining matching object recognition templates, or that the image information has the portion representing the unmatched region, generate a wherein the safety volume list, which is a list that describes at least one of:(i) the unmatched region, or (ii) one or more additional candidate regions that also estimate object boundary locations for the object or estimate which locations are occupied by the object, wherein the one or more additional candidate regions are generated based on the subset of one or more remaining matching object recognition templates; and perform motion planning based on the primary candidate region and based on the safety volume list, wherein the motion planning is for robot interaction between the robot and the object for gripping or picking up the object and moving the object from the occupied location of the object to a destination location.
39. The non-transitory computer-readable medium of claim 38, wherein the instructions, when executed by the at least one processing circuit, cause the at least one processing circuit to determine a bounding region encompassing the primary candidate region and at least one of: (i) the one or more additional candidate regions or (ii) the unmatched region, and wherein the instructions further cause the at least one processing circuit to perform the motion planning for a trajectory associated with a robot end effector apparatus of the robot based on the bounding region.
19. The non-transitory computer-readable medium of claim 18, wherein the instructions, when executed by the at least one processing circuit, cause the at least one processing circuit to determine a bounding region which encompasses the primary candidate region and at least one of: (i) the one or more additional candidate regions or (ii) the unmatched region, and wherein the instructions further cause the at least one processing circuit to perform the motion planning by determining a trajectory associated with a robot end effector apparatus based on the bounding region
40. A method performed by a computing system, the method comprising: receiving image information, at the computing system, wherein the computing system is configured to communicate with: (i) a robot, and (ii) a camera, wherein the image information represents an object in the camera field of view, and is generated by the camera; identifying one or more object recognition templates corresponding to an object or an object type; selecting a primary object template from among the one or more object recognition templates based on matching the image information with the one or more object recognition templates; generating a primary candidate region based on the primary object template; determining at least one of:(i) a subset of one or more remaining matching object recognition templates, or (ii) an unmatched region of the image information that is adjacent to the primary candidate region; generating a safety volume list comprising at least one of:(i) the unmatched region, or (ii) one or more additional candidate regions based on the subset, wherein the one or more additional candidate regions estimate object boundary locations for the object or estimate locations in the field of view that are occupied by the object; and performing motion planning for interaction between the robot and the object based on the primary candidate region and the safety volume list.
20. A method performed by a computing system, the method comprising: receiving image information by the computing system, wherein the computing system is configured to communicate with: (i) a robot, and (ii) a camera having a camera field of view, wherein the image information is for representing an object in the camera field of view, and is generated by the camera; identifying a set of one or more matching object recognition templates, which are one or more object recognition templates that satisfy a predefined template matching condition when compared against the image information, wherein the set of one or more matching object recognition templates are associated with a set of one or more detection hypotheses, which are one or more respective estimates on which object or object type is represented by the image information; selecting, as a primary detection hypothesis, a detection hypothesis from among the set of one or more detection hypotheses, wherein the primary detection hypothesis is associated with a matching object recognition template of the set of one or more matching object recognition templates, wherein the detection hypothesis that is selected as the primary detection hypothesis has a confidence value which is highest among a set of one or more respective confidence values, wherein the set of one or more respective confidence values are associated with the set of one or more detection hypotheses, and indicate respective degrees by which the image information matches the set of one or more matching object recognition templates associated with the set of one or more detection hypotheses; generating, as a primary candidate region, a candidate region which estimates object boundary locations for the object or estimates which locations in the camera field of view are occupied by the object, wherein the primary candidate region is generated based on the matching object recognition template associated with the primary detection hypothesis; determining, in addition to the matching object recognition template associated with the primary detection hypothesis, determining at least one of:(i) that the set of one or more matching object recognition templates has, in addition to the matching object recognition template associated with the primary detection hypothesis, a subset of one or more remaining matching object recognition templates that also satisfy the predefined template matching condition when compared against the image information, or (ii) that the image information has a portion representing an unmatched region which is adjacent to the primary candidate region and which fails to satisfy the predefined template matching condition; generating a safety volume list, which is a list that describes at least one of:(i) the unmatched region, or (ii) one or more additional candidate regions that also estimate object boundary locations for the object or estimate which locations are occupied by the object, wherein the one or more additional candidate regions are generated based on the subset of one or more remaining matching object recognition templates; and performing motion planning based on the primary candidate region and based on the safety volume list, wherein the motion planning is for robot interaction between the robot and the object for gripping or picking up the object and moving the object from the occupied location of the object to a destination location.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claim(s) 21-23, 28, 29, 31, 38, and 40 are rejected under 35 U.S.C. 103 as being read upon Konishi (US 20190197727 A1).
Regarding Claim 21, Konishi teaches a computing system comprising:
a communication interface configured to communicate with a robot and with a camera (See at least Para [0041] “First, an example of a situation in which an embodiment is applied will be described using FIG. 1. FIG. 1 is a diagram illustrating an example of a situation in which an object recognition apparatus 1 according to an embodiment is applied. The object recognition apparatus 1 may be a system, installed in a production line or the like, that recognizes objects 2 within a tray 3 using an image obtained from an image capturing device 11. The objects 2 to be recognized are piled randomly in the tray 3. Here, the objects 2 may include multiple types of objects, or may be a single type of object having a different appearance depending on the viewpoint. By obtaining images from the image capturing device 11 at predetermined intervals of time and carrying out a template matching process, the object recognition apparatus 1 executes a process of recognizing the position and attitude of each object 2 included in the image captured by the image capturing device 11 (also called an “input image” hereinafter), and outputs a result of that process to a programmable logic controller (PLC) 4, a display 12, or the like. A recognition result, which is the output of the object recognition apparatus 1, is used in picking/robot control, control of processing devices or printing devices, the inspection, measurement, and so on of the objects 2, and so on.”); and
at least one processing circuit configured, when an object is or has been in a field of view of the camera, to:
receive image information, generated by the camera, representing the object (See at least Fig 1, [0045] “The image capturing device 11 is an image capturing device for providing a digital image of the objects 2 to the image processing device 10. A complementary MOS (CMOS) camera, a charge-coupled device (CCD) camera, or the like can be used favorably as the image capturing device 11. The characteristics of the input image, such as the resolution, color/black-and-white, still image/moving image, tone, data format, and so on, can be set as desired, and can be selected as appropriate in accordance with the types of objects 2, the purpose of the sensing, and so on…”, Para [0041]);
identify one or more object recognition templates corresponding to an object or an object type (See at least Para [0051] “The object recognition processing apparatus 30 may be an apparatus that recognizes an object in an image by carrying out a template matching process on an image obtained from the image capturing device 11 using a template created by the template creation apparatus 20 and then stored. The object recognition processing apparatus 30 includes an image obtainment unit 301, a threshold setting unit 302, a template information obtainment unit 303, a template matching unit 304, a candidate exclusion processing unit 305, and a recognition result output unit 306.”);
select a primary object template from among the one or more object recognition templates based on matching the image information with the one or more object recognition templates (See at least Para [0019] “Accordingly, of the candidates included in the recognition result, the candidates that should be excluded can be appropriately excluded while giving preference to candidates having higher scores.”, Para [0059] “The template matching unit 304 obtains a recognition result by detecting feature points in the input image and calculating a feature amount, and then carrying out a template matching process using the template information supplied from the template information obtainment unit 303 and the calculated feature amount. The feature amount calculated here is the same type of feature amount calculated by the template creation unit 202 when creating the template. The recognition result includes position parameters and attitude parameters pertaining to the candidates of the objects 2 recognized in the input image, a score indicating the degree to which image features match between the input image and the template, the template identifier, and so on. The template matching unit 304 is an example of a “template matching unit” according to an embodiment.”);
generate a primary candidate region based on the primary object template (See at least Para [0019] “Accordingly, of the candidates included in the recognition result, the candidates that should be excluded can be appropriately excluded while giving preference to candidates having higher scores.”);
determine at least one of:
(i) a subset of one or more remaining matching object recognition templates (See at least Para [0018] “In the above-described object recognition processing apparatus, the candidate exclusion processing unit may include: temporary storage configured to store the unexcluded candidate; a first unit configured to rearrange the plurality of candidates in order by a score used in the template matching process; and a second unit configured to obtain, in that order, one by one of the rearranged plurality of candidates,…”), or
(ii) an unmatched region of the image information that is adjacent to the primary candidate region (See at least Para [0062] “… However, if the calculated overlap value is greater than the threshold set by the threshold setting unit 302, the candidate exclusion processing unit 305 excludes the corresponding candidate from the recognition result. The candidate exclusion processing unit 305 carries out this process for all candidates included in the recognition result. The candidate exclusion processing unit 305 is an example of a “candidate exclusion processing unit” according to an embodiment. The overlap value being greater than the threshold is an example of a “predetermined condition” according to an embodiment.”);
generate a safety volume list in response to a determination of the subset or the unmatched region (See at least Para [0018] “In the above-described object recognition processing apparatus, the candidate exclusion processing unit may include: temporary storage configured to store the unexcluded candidate; a first unit configured to rearrange the plurality of candidates in order by a score used in the template matching process; and a second unit configured to obtain, in that order, one by one of the rearranged plurality of candidates,…”),
wherein the safety volume list includes at least one of:
(i) the unmatched region (See at least Para [0019] “According to an aspect, whether or not to exclude a candidate is determined in order of scores in the template matching process, and the candidates that remain without being excluded are stored in the temporary storage. Accordingly, of the candidates included in the recognition result, the candidates that should be excluded can be appropriately excluded while giving preference to candidates having higher scores.”), or
(ii) one or more additional candidate regions based on the subset, wherein the one or more additional candidate regions estimate object boundary locations for the object or estimate locations in the field of view that are occupied by the object (See at least Para [0106] “a first unit configured to rearrange the plurality of candidates in order by a score used in the template matching process; and”, Para [0018] “In the above-described object recognition processing apparatus, the candidate exclusion processing unit may include: temporary storage configured to store the unexcluded candidate; a first unit configured to rearrange the plurality of candidates in order by a score used in the template matching process; and a second unit configured to obtain, in that order, one by one of the rearranged plurality of candidates,…”); and
perform motion planning for interaction between the robot and the object based on the primary candidate region and the safety volume list (See at least Para [0041] “First, an example of a situation in which an embodiment is applied will be described using FIG. 1. FIG. 1 is a diagram illustrating an example of a situation in which an object recognition apparatus 1 according to an embodiment is applied. The object recognition apparatus 1 may be a system, installed in a production line or the like, that recognizes objects 2 within a tray 3 using an image obtained from an image capturing device 11. The objects 2 to be recognized are piled randomly in the tray 3. Here, the objects 2 may include multiple types of objects, or may be a single type of object having a different appearance depending on the viewpoint. By obtaining images from the image capturing device 11 at predetermined intervals of time and carrying out a template matching process, the object recognition apparatus 1 executes a process of recognizing the position and attitude of each object 2 included in the image captured by the image capturing device 11 (also called an “input image” hereinafter), and outputs a result of that process to a programmable logic controller (PLC) 4, a display 12, or the like. A recognition result, which is the output of the object recognition apparatus 1, is used in picking/robot control, control of processing devices or printing devices, the inspection, measurement, and so on of the objects 2, and so on.”).
Regarding Claim 22, Konishi teaches all the elements of claim 21. Konishi further teaches the computing system of claim 21, wherein the at least one processing circuit is further configured to determine a bounding region encompassing the primary candidate region and at least one of: (i) the one or more additional candidate regions (See at least Para [0015] “According to an aspect, using the degree of overlap among the candidates as the condition for excluding candidates makes it possible to appropriately exclude candidates that should be excluded, even if the objects to be recognized have different shapes, appearances, and so on.”) or (ii) the unmatched region, and to perform the motion planning for a trajectory associated with a robot end effector apparatus of the robot based on the bounding region (See at least Para [0041] “First, an example of a situation in which an embodiment is applied will be described using FIG. 1. FIG. 1 is a diagram illustrating an example of a situation in which an object recognition apparatus 1 according to an embodiment is applied. The object recognition apparatus 1 may be a system, installed in a production line or the like, that recognizes objects 2 within a tray 3 using an image obtained from an image capturing device 11. The objects 2 to be recognized are piled randomly in the tray 3. Here, the objects 2 may include multiple types of objects, or may be a single type of object having a different appearance depending on the viewpoint. By obtaining images from the image capturing device 11 at predetermined intervals of time and carrying out a template matching process, the object recognition apparatus 1 executes a process of recognizing the position and attitude of each object 2 included in the image captured by the image capturing device 11 (also called an “input image” hereinafter), and outputs a result of that process to a programmable logic controller (PLC) 4, a display 12, or the like. A recognition result, which is the output of the object recognition apparatus 1, is used in picking/robot control, control of processing devices or printing devices, the inspection, measurement, and so on of the objects 2, and so on.”).
Regarding Claim 23, Konishi teaches all the elements of claim 22. Konishi further teaches the computing system of claim 22, wherein the at least one processing circuit is further configured to perform the motion planning including determining robot gripping motion based on the primary candidate region (See at least Para [0041] “First, an example of a situation in which an embodiment is applied will be described using FIG. 1. FIG. 1 is a diagram illustrating an example of a situation in which an object recognition apparatus 1 according to an embodiment is applied. The object recognition apparatus 1 may be a system, installed in a production line or the like, that recognizes objects 2 within a tray 3 using an image obtained from an image capturing device 11. The objects 2 to be recognized are piled randomly in the tray 3. Here, the objects 2 may include multiple types of objects, or may be a single type of object having a different appearance depending on the viewpoint. By obtaining images from the image capturing device 11 at predetermined intervals of time and carrying out a template matching process, the object recognition apparatus 1 executes a process of recognizing the position and attitude of each object 2 included in the image captured by the image capturing device 11 (also called an “input image” hereinafter), and outputs a result of that process to a programmable logic controller (PLC) 4, a display 12, or the like. A recognition result, which is the output of the object recognition apparatus 1, is used in picking/robot control, control of processing devices or printing devices, the inspection, measurement, and so on of the objects 2, and so on.”).
Regarding Claim 28, Konishi teaches all the elements of claim 21. Konishi further teaches the computing system of claim 21, wherein the image information includes 2D image information, and wherein the primary object template comprises a set of visual description information which is determined by the at least one processing circuit to satisfy a template matching condition when compared against the 2D image information (See at least Para [0052] “The object data obtainment unit 201 obtains data expressing the shape of an object 2 to be recognized. Depending on the shape of the object 2 to be recognized, the object data obtainment unit 201 can obtain two-dimensional data expressing the two-dimensional shape of the object 2,…”, Para [0053] “… the template creation unit 202 may generate two-dimensional images of the object 2 from a plurality of viewpoints and create a template from the generated two-dimensional images…”).
Regarding Claim 29, Konishi teaches all the elements of claim 28. Konishi further teaches the computing system of claim 28, wherein at least one object recognition template of the subset of one or more remaining object recognition templates has a set of visual description information that is determined by the at least one processing circuit to satisfy the template matching condition when compared against the 2D image information, and wherein the at least one processing circuit is further configured to generate the safety volume list based on at least one of the unmatched region, the one or more additional candidate regions, or the at least one object recognition template (See at least Para [0052] “The object data obtainment unit 201 obtains data expressing the shape of an object 2 to be recognized. Depending on the shape of the object 2 to be recognized, the object data obtainment unit 201 can obtain two-dimensional data expressing the two-dimensional shape of the object 2,…”, Para [0053] “… the template creation unit 202 may generate two-dimensional images of the object 2 from a plurality of viewpoints and create a template from the generated two-dimensional images…”).
Regarding Claim 31, Konishi teaches all the elements of claim 28. Konishi further teaches the computing system of claim 28, wherein the image information further includes 3D image information, and wherein at least one object recognition template of the subset of one or more remaining object recognition templates has a respective set of structure description information that is determined by the at least one processing circuit to satisfy the template matching condition when compared against the 3D image information, and wherein the at least one processing circuit is further configured to generate the safety volume list based on at least one of the unmatched region, the one or more additional candidate regions, or the at least one object recognition template (See at least Para [0026] “In the above-described object recognition processing apparatus, the candidate exclusion processing unit may exclude a candidate, among the plurality of candidates, that meets a predetermined condition by generating a three-dimensional binary image for each of the plurality of candidates on the basis of the position and attitude of that candidate and using the three-dimensional binary images to find a degree of overlap for each of the candidates.”, Para [0052] “… or can obtain three-dimensional data expressing the three-dimensional shape of the object 2…”, Para [0053] “The template creation unit 202 creates a template using the data expressing the shape of the object 2, which has been obtained by the object data obtainment unit 201. If the object data obtainment unit 201 has obtained three-dimensional data of the object 2,…”).
Regarding Claim 38, Konishi teaches a non-transitory computer-readable medium having instructions that, when executed by at least one processing circuit of a computing system, causes the at least one processing circuit to:
receive image information, generated by a camera, representing an object in a field of view of the camera, wherein the computing system is configured to communicate with: (i) a robot, and (ii) the camera (See at least Para [0041] “First, an example of a situation in which an embodiment is applied will be described using FIG. 1. FIG. 1 is a diagram illustrating an example of a situation in which an object recognition apparatus 1 according to an embodiment is applied. The object recognition apparatus 1 may be a system, installed in a production line or the like, that recognizes objects 2 within a tray 3 using an image obtained from an image capturing device 11. The objects 2 to be recognized are piled randomly in the tray 3. Here, the objects 2 may include multiple types of objects, or may be a single type of object having a different appearance depending on the viewpoint. By obtaining images from the image capturing device 11 at predetermined intervals of time and carrying out a template matching process, the object recognition apparatus 1 executes a process of recognizing the position and attitude of each object 2 included in the image captured by the image capturing device 11 (also called an “input image” hereinafter), and outputs a result of that process to a programmable logic controller (PLC) 4, a display 12, or the like. A recognition result, which is the output of the object recognition apparatus 1, is used in picking/robot control, control of processing devices or printing devices, the inspection, measurement, and so on of the objects 2, and so on.”, Fig 1, [0045] “The image capturing device 11 is an image capturing device for providing a digital image of the objects 2 to the image processing device 10. A complementary MOS (CMOS) camera, a charge-coupled device (CCD) camera, or the like can be used favorably as the image capturing device 11. The characteristics of the input image, such as the resolution, color/black-and-white, still image/moving image, tone, data format, and so on, can be set as desired, and can be selected as appropriate in accordance with the types of objects 2, the purpose of the sensing, and so on…”);
identify one or more object recognition templates corresponding to an object or an object type
(See at least Para [0051] “The object recognition processing apparatus 30 may be an apparatus that recognizes an object in an image by carrying out a template matching process on an image obtained from the image capturing device 11 using a template created by the template creation apparatus 20 and then stored. The object recognition processing apparatus 30 includes an image obtainment unit 301, a threshold setting unit 302, a template information obtainment unit 303, a template matching unit 304, a candidate exclusion processing unit 305, and a recognition result output unit 306.”);
select, a primary object template from among the one or more object recognition templates
based on matching the image information with the one or more object recognition templates (See at least Para [0059] “The template matching unit 304 obtains a recognition result by detecting feature points in the input image and calculating a feature amount, and then carrying out a template matching process using the template information supplied from the template information obtainment unit 303 and the calculated feature amount. The feature amount calculated here is the same type of feature amount calculated by the template creation unit 202 when creating the template. The recognition result includes position parameters and attitude parameters pertaining to the candidates of the objects 2 recognized in the input image, a score indicating the degree to which image features match between the input image and the template, the template identifier, and so on. The template matching unit 304 is an example of a “template matching unit” according to an embodiment.”);
generate a primary candidate region based on the primary object template (See at least Para [0019] “Accordingly, of the candidates included in the recognition result, the candidates that should be excluded can be appropriately excluded while giving preference to candidates having higher scores.”);
determine at least one of:
(i) a subset of one or more remaining matching object recognition templates (See at least Para [0018] “In the above-described object recognition processing apparatus, the candidate exclusion processing unit may include: temporary storage configured to store the unexcluded candidate; a first unit configured to rearrange the plurality of candidates in order by a score used in the template matching process; and a second unit configured to obtain, in that order, one by one of the rearranged plurality of candidates,…”), or
(ii) an unmatched region of the image information that is adjacent to the primary candidate region (See at least Para [0062] “… However, if the calculated overlap value is greater than the threshold set by the threshold setting unit 302, the candidate exclusion processing unit 305 excludes the corresponding candidate from the recognition result. The candidate exclusion processing unit 305 carries out this process for all candidates included in the recognition result. The candidate exclusion processing unit 305 is an example of a “candidate exclusion processing unit” according to an embodiment. The overlap value being greater than the threshold is an example of a “predetermined condition” according to an embodiment.”);
generate a safety volume list in response to a determination of the subset, or the unmatched region,
wherein the safety volume list includes at least one of:
(i) the unmatched region (See at least Para [0019] “According to an aspect, whether or not to exclude a candidate is determined in order of scores in the template matching process, and the candidates that remain without being excluded are stored in the temporary storage. Accordingly, of the candidates included in the recognition result, the candidates that should be excluded can be appropriately excluded while giving preference to candidates having higher scores.”), or
(ii) one or more additional candidate regions based on the subset, wherein the one or more additional candidate regions estimate object boundary locations for the object or estimate locations in the field of view that are occupied by the object (See at least Para [0106] “a first unit configured to rearrange the plurality of candidates in order by a score used in the template matching process; and”, Para [0018] “In the above-described object recognition processing apparatus, the candidate exclusion processing unit may include: temporary storage configured to store the unexcluded candidate; a first unit configured to rearrange the plurality of candidates in order by a score used in the template matching process; and a second unit configured to obtain, in that order, one by one of the rearranged plurality of candidates,…”); and
perform motion planning for interaction between the robot and the object based on the primary candidate region and the safety volume list (See at least Para [0041] “First, an example of a situation in which an embodiment is applied will be described using FIG. 1. FIG. 1 is a diagram illustrating an example of a situation in which an object recognition apparatus 1 according to an embodiment is applied. The object recognition apparatus 1 may be a system, installed in a production line or the like, that recognizes objects 2 within a tray 3 using an image obtained from an image capturing device 11. The objects 2 to be recognized are piled randomly in the tray 3. Here, the objects 2 may include multiple types of objects, or may be a single type of object having a different appearance depending on the viewpoint. By obtaining images from the image capturing device 11 at predetermined intervals of time and carrying out a template matching process, the object recognition apparatus 1 executes a process of recognizing the position and attitude of each object 2 included in the image captured by the image capturing device 11 (also called an “input image” hereinafter), and outputs a result of that process to a programmable logic controller (PLC) 4, a display 12, or the like. A recognition result, which is the output of the object recognition apparatus 1, is used in picking/robot control, control of processing devices or printing devices, the inspection, measurement, and so on of the objects 2, and so on.”).
Regarding Claim 40, Konishi teaches a method performed by a computing system, the method comprising:
receiving image information, at the computing system, wherein the computing system is configured to communicate with: (i) a robot, and (ii) a camera, wherein the image information represents an object in the camera field of view, and is generated by the camera (See at least Para [0041] “First, an example of a situation in which an embodiment is applied will be described using FIG. 1. FIG. 1 is a diagram illustrating an example of a situation in which an object recognition apparatus 1 according to an embodiment is applied. The object recognition apparatus 1 may be a system, installed in a production line or the like, that recognizes objects 2 within a tray 3 using an image obtained from an image capturing device 11. The objects 2 to be recognized are piled randomly in the tray 3. Here, the objects 2 may include multiple types of objects, or may be a single type of object having a different appearance depending on the viewpoint. By obtaining images from the image capturing device 11 at predetermined intervals of time and carrying out a template matching process, the object recognition apparatus 1 executes a process of recognizing the position and attitude of each object 2 included in the image captured by the image capturing device 11 (also called an “input image” hereinafter), and outputs a result of that process to a programmable logic controller (PLC) 4, a display 12, or the like. A recognition result, which is the output of the object recognition apparatus 1, is used in picking/robot control, control of processing devices or printing devices, the inspection, measurement, and so on of the objects 2, and so on.”, Fig 1, [0045] “The image capturing device 11 is an image capturing device for providing a digital image of the objects 2 to the image processing device 10. A complementary MOS (CMOS) camera, a charge-coupled device (CCD) camera, or the like can be used favorably as the image capturing device 11. The characteristics of the input image, such as the resolution, color/black-and-white, still image/moving image, tone, data format, and so on, can be set as desired, and can be selected as appropriate in accordance with the types of objects 2, the purpose of the sensing, and so on…”);
identifying one or more object recognition templates corresponding to an object or an object type (See at least Para [0051] “The object recognition processing apparatus 30 may be an apparatus that recognizes an object in an image by carrying out a template matching process on an image obtained from the image capturing device 11 using a template created by the template creation apparatus 20 and then stored. The object recognition processing apparatus 30 includes an image obtainment unit 301, a threshold setting unit 302, a template information obtainment unit 303, a template matching unit 304, a candidate exclusion processing unit 305, and a recognition result output unit 306.”);
selecting a primary object template from among the one or more object recognition templates based on matching the image information with the one or more object recognition templates (See at least Para [0059] “The template matching unit 304 obtains a recognition result by detecting feature points in the input image and calculating a feature amount, and then carrying out a template matching process using the template information supplied from the template information obtainment unit 303 and the calculated feature amount. The feature amount calculated here is the same type of feature amount calculated by the template creation unit 202 when creating the template. The recognition result includes position parameters and attitude parameters pertaining to the candidates of the objects 2 recognized in the input image, a score indicating the degree to which image features match between the input image and the template, the template identifier, and so on. The template matching unit 304 is an example of a “template matching unit” according to an embodiment.”);
generating a primary candidate region based on the primary object template (See at
least Para [0019] “Accordingly, of the candidates included in the recognition result, the candidates that should be excluded can be appropriately excluded while giving preference to candidates having higher scores.”);
determining at least one of:
(i) a subset of one or more remaining matching object recognition templates (See at least Para [0018] “In the above-described object recognition processing apparatus, the candidate exclusion processing unit may include: temporary storage configured to store the unexcluded candidate; a first unit configured to rearrange the plurality of candidates in order by a score used in the template matching process; and a second unit configured to obtain, in that order, one by one of the rearranged plurality of candidates,…”), or
(ii) an unmatched region of the image information that is adjacent to the primary candidate region (See at least Para [0062] “… However, if the calculated overlap value is greater than the threshold set by the threshold setting unit 302, the candidate exclusion processing unit 305 excludes the corresponding candidate from the recognition result. The candidate exclusion processing unit 305 carries out this process for all candidates included in the recognition result. The candidate exclusion processing unit 305 is an example of a “candidate exclusion processing unit” according to an embodiment. The overlap value being greater than the threshold is an example of a “predetermined condition” according to an embodiment.”);
generating a safety volume list comprising at least one of:
(i) the unmatched region (See at least Para [0019] “According to an aspect, whether or not to exclude a candidate is determined in order of scores in the template matching process, and the candidates that remain without being excluded are stored in the temporary storage. Accordingly, of the candidates included in the recognition result, the candidates that should be excluded can be appropriately excluded while giving preference to candidates having higher scores.”), or
(ii) one or more additional candidate regions based on the subset, wherein the one or more additional candidate regions estimate object boundary locations for the object or estimate locations in the field of view that are occupied by the object (See at least Para [0106] “a first unit configured to rearrange the plurality of candidates in order by a score used in the template matching process; and”, Para [0018] “In the above-described object recognition processing apparatus, the candidate exclusion processing unit may include: temporary storage configured to store the unexcluded candidate; a first unit configured to rearrange the plurality of candidates in order by a score used in the template matching process; and a second unit configured to obtain, in that order, one by one of the rearranged plurality of candidates,…”); and
performing motion planning for interaction between the robot and the object based on
the primary candidate region and the safety volume list (See at least Para [0041] “First, an example of a situation in which an embodiment is applied will be described using FIG. 1. FIG. 1 is a diagram illustrating an example of a situation in which an object recognition apparatus 1 according to an embodiment is applied. The object recognition apparatus 1 may be a system, installed in a production line or the like, that recognizes objects 2 within a tray 3 using an image obtained from an image capturing device 11. The objects 2 to be recognized are piled randomly in the tray 3. Here, the objects 2 may include multiple types of objects, or may be a single type of object having a different appearance depending on the viewpoint. By obtaining images from the image capturing device 11 at predetermined intervals of time and carrying out a template matching process, the object recognition apparatus 1 executes a process of recognizing the position and attitude of each object 2 included in the image captured by the image capturing device 11 (also called an “input image” hereinafter), and outputs a result of that process to a programmable logic controller (PLC) 4, a display 12, or the like. A recognition result, which is the output of the object recognition apparatus 1, is used in picking/robot control, control of processing devices or printing devices, the inspection, measurement, and so on of the objects 2, and so on.”).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 24-27, 30, 37 and 39 are rejected under 35 U.S.C. 103 as being unpatentable over Konishi (US 20190197727 A1) in view of Hinterstoisser et al. (US 9327406 B1) (Hereinafter Hinterstoisser).
Regarding Claim 24. Konishi teaches all the elements of claim 21. Konishi further teaches the computing system of claim 21, wherein for the subset of one or more remaining matching object recognition templates, the at least one processing circuit is further configured to:
determine whether a respective confidence value associated with each of the subset of one or more remaining matching object recognition templates is within a confidence similarity threshold relative to a confidence value associated with the primary object template (See at least Para [0051] “… The object recognition processing apparatus 30 includes an image obtainment unit 301, a threshold setting unit 302 , Para [0057] “In another embodiment, the threshold setting unit 302 may set a threshold for an overlap value indicating the value of area over which the silhouette of one candidate overlaps with a cumulative image of the silhouettes of the remaining candidates not excluded.”) …
However, Konishi does not explicitly spell out …
include, in the safety volume list, a respective candidate region associated with the each of the subset of one or more remaining matching object recognition templates, in response to a determination that the respective confidence value is within the confidence similarity threshold, and
supplement the one or more additional regions of the safety volume list with the respective candidate region.
Hinterstoisser teaches …
include, in the safety volume list, a respective candidate region associated with the each of the subset of one or more remaining matching object recognition templates, in response to a determination that the respective confidence value is within the confidence similarity threshold (Col 4 lines 54-60 “Generally, within example implementations herein, the presence of these surface features within or on a boundary of a particular box hypothesis may raise or lower a confidence score of the particular box face, which may then be input to a global system for determining an optimal set of box hypotheses associated with the given image(s) and thereby facilitate an optimal segmentation.”, Col 18 lines 43-57 describes confidence value of hypotheses’ which is based on matching quality and image is segmented accordingly, Col 17 lines 49-63 describes threshold for substantial match, Col 18 lines 43-57 describes confidence value of hypotheses’ which is based on matching quality and image is segmented accordingly), and
supplement the one or more additional regions of the safety volume list with the respective candidate region (Col 17 lines 49-63 describes threshold for substantial match, Col 18 lines 43-57 describes confidence value which is based on matching quality and image is segmented accordingly).
Therefore, it would have obvious to one of the ordinary skill in the art before the effective filling date of the claimed invention to modify Konishi to include the teachings of Hinterstoisser and include the feature of in the safety volume list, a respective candidate region associated with the each of the subset of one or more remaining matching object recognition templates, in response to a determination that the respective confidence value is within the confidence similarity threshold and supplement the one or more additional regions of the safety volume list with the respective candidate region, thereby making it possible to perform image analysis more accurately and controlling the robot can be done precisely.
Regarding Claim 25, Konishi teaches all the elements of claim 24. Konishi further teaches the computing system of claim 24, wherein each candidate region of the one or more additional candidate regions in the safety volume list is associated with a confidence value that is within the confidence similarity threshold (See at least Para [0051] “… The object recognition processing apparatus 30 includes an image obtainment unit 301, a threshold setting unit 302 , Para [0057] “In another embodiment, the threshold setting unit 302 may set a threshold for an overlap value indicating the value of area over which the silhouette of one candidate overlaps with a cumulative image of the silhouettes of the remaining candidates not excluded.”, Para [0107] “a second unit configured to obtain, in that order, one by one of the rearranged plurality of candidates, compare a binary image generated on the basis of a position and attitude of the obtained candidate with a cumulative image of the binary images generated on the basis of the positions and attitudes of all the candidates stored in the temporary storage, and store the candidate in the temporary storage if a degree of overlap between the stated images is less than a predetermined threshold”).
Regarding Claim 26, computing system of claim 24, wherein each candidate region of the one or more additional candidate regions in the safety volume list is associated with a confidence value that is greater than or equal to a template matching threshold (See at least Para [0076] “If the calculated overlap value is less than or equal to the threshold set by the threshold setting unit 302 (step S610: Yes), the candidate exclusion processing unit 305 renders the silhouette of the candidate subject to the exclusion determination in the silhouette map image (step S611). However, if the calculated overlap value is greater than the threshold set by the threshold setting unit 302 (step S610: No), the candidate exclusion processing unit 305 excludes the corresponding candidate from the recognition result (step S612).”).
Regarding Claim 27, Konishi teaches all the elements of claim 21. Konishi further teaches the computing system of claim 21, wherein the subset of one or more remaining matching object recognition templates includes a plurality of matching object recognition templates associated with a plurality of respective candidate regions, wherein the at least one processing circuit is further configured, for each candidate region of the plurality of candidate regions, to:
determine a respective amount of overlap between the candidate region and the primary candidate region (See at least Para [0076] “If the calculated overlap value is less than or equal to the threshold set by the threshold setting unit 302 (step S610: Yes), the candidate exclusion processing unit 305 renders the silhouette of the candidate subject to the exclusion determination in the silhouette map image (step S611). However, if the calculated overlap value is greater than the threshold set by the threshold setting unit 302 (step S610: No), the candidate exclusion processing unit 305 excludes the corresponding candidate from the recognition result (step S612).”);
determine whether the respective amount of overlap is equal to or exceeds an overlap threshold (See at least Para [0076] “If the calculated overlap value is less than or equal to the threshold set by the threshold setting unit 302 (step S610: Yes), the candidate exclusion processing unit 305 renders the silhouette of the candidate subject to the exclusion determination in the silhouette map image (step S611). However, if the calculated overlap value is greater than the threshold set by the threshold setting unit 302 (step S610: No), the candidate exclusion processing unit 305 excludes the corresponding candidate from the recognition result (step S612).”); and …
However, Konishi does not explicitly spell out …
include the candidate region in the one or more additional candidate regions of the safety volume list, in response to the respective amount of overlap being equal to or exceeding the overlap threshold.
Hinterstoisser teaches …
include the candidate region in the one or more additional candidate regions of the safety volume list, in response to the respective amount of overlap being equal to or exceeding the overlap threshold (Col 21 lines 23-37 discloses the measurement of area of overlap that matches or within a predetermined threshold).
Therefore, it would have obvious to one of the ordinary skill in the art before the effective filling date of the claimed invention to modify Konishi to include the teachings of Hinterstoisser and include the feature of candidate region in the one or more additional candidate regions of the safety volume list in response to the respective amount of overlap being equal to or exceeding the overlap threshold, thereby making it possible to perform image analysis more accurately and controlling the robot can be done precisely.
Regarding Claim 30, Konishi teaches all the elements of claim 29.
However, Konishi does not explicitly spell out the computing system of claim 29, wherein the primary object template includes a respective set of structure description information that indicates a first object size, and wherein the at least one object recognition template includes a respective set of structure description information that indicates a second object size different than the first object size.
Hinterstoisser teaches the computing system of claim 29, wherein the primary object template includes a respective set of structure description information that indicates a first object size, and wherein the at least one object recognition template includes a respective set of structure description information that indicates a second object size different than the first object size (Fig 4A, 6A, and 6B disclose stack of boxes which shows volume which is construed as structure of the boxes, Col 16 lines 40-49 discloses sensors generating volumetric representation of the physical environment).
Therefore, it would have obvious to one of the ordinary skill in the art before the effective filling date of the claimed invention to modify Konishi to include the teachings of Hinterstoisser and include a feature of the primary object template includes a respective set of structure description information that indicates a first object size, and wherein the at least one object recognition template includes a respective set of structure description information that indicates a second object size different than the first object size, thereby making it possible to perform image analysis more accurately and controlling the robot can be done precisely.
Regarding Claim 37, Konishi teaches all the elements of claim 21.
However, Konishi does not explicitly spell out the computing system of claim 21, wherein the at
least one processing circuit is further configured to add, to the safety volume list, a candidate region that represents a maximum object height.
Hinterstoisser teaches the computing system of claim 21, wherein the at least one processing
circuit is further configured to add, to the safety volume list, a candidate region that represents a maximum object height (Col11 lines 62-67 Col 12 lines 1-5 discloses height maps).
Therefore, it would have obvious to one of the ordinary skill in the art before the effective filling date of the claimed invention to modify Konishi to include the teachings of Hinterstoisser and include a feature of processing circuit further configured to add to the safety volume list a candidate region that represents a maximum object height, thereby making it possible to perform image analysis more accurately and controlling the robot can be done precisely.
Regarding Claim 39, Konishi teaches all the elements of claim 38. Konishi further teaches … wherein the instructions further cause the at least one processing circuit to perform the motion planning for a trajectory associated with a robot end effector apparatus of the robot based on the bounding region (See at least Para [0041] “First, an example of a situation in which an embodiment is applied will be described using FIG. 1. FIG. 1 is a diagram illustrating an example of a situation in which an object recognition apparatus 1 according to an embodiment is applied. The object recognition apparatus 1 may be a system, installed in a production line or the like, that recognizes objects 2 within a tray 3 using an image obtained from an image capturing device 11. The objects 2 to be recognized are piled randomly in the tray 3. Here, the objects 2 may include multiple types of objects, or may be a single type of object having a different appearance depending on the viewpoint. By obtaining images from the image capturing device 11 at predetermined intervals of time and carrying out a template matching process, the object recognition apparatus 1 executes a process of recognizing the position and attitude of each object 2 included in the image captured by the image capturing device 11 (also called an “input image” hereinafter), and outputs a result of that process to a programmable logic controller (PLC) 4, a display 12, or the like. A recognition result, which is the output of the object recognition apparatus 1, is used in picking/robot control, control of processing devices or printing devices, the inspection, measurement, and so on of the objects 2, and so on.”).
However, Konishi does not explicitly spell out the non-transitory computer-readable medium of
claim 38, wherein the instructions, when executed by the at least one processing circuit, cause the at least one processing circuit to determine a bounding region encompassing the primary candidate region and at least one of: (i) the one or more additional candidate regions or (ii) the unmatched region, and …
Hinterstoisser teaches the non-transitory computer-readable medium of claim 38, wherein the
instructions, when executed by the at least one processing circuit, cause the at least one processing circuit (Col 16 lines 15-31, “…The program code may be stored on any type of computer readable medium, for example, such as a storage device including a disk or hard drive. The computer readable medium may include a non-transitory computer readable medium, for example, such as computer-readable media that stores data for short periods of time like register memory, processor cache and Random Access Memory (RAM). The computer readable medium may also include non-transitory media, such as secondary or persistent long term storage, like read only memory (ROM), optical or magnetic disks, compact-disc read only memory (CD-ROM), for example. The computer readable media may also be any other volatile or non-volatile storage systems. The computer readable medium may be considered a computer readable storage medium, a tangible storage device, or other article of manufacture”…) to determine a bounding region encompassing the primary candidate region and at least one of: (i) the one or more additional candidate regions (Col 4 lines 36-44, “The system can then identify surface features of the type(s) within various regions of the projected images or regions of the individual original images captured by the optical sensor. The identified features may indicate, for instance, that a particular current “box hypothesis” (e.g., an estimated region that corresponds to a box face) determined by the system is a valid box, where a box hypothesis represents estimated boundaries of distinct objects in the physical environment, such as estimated boundaries of a single box face of a box…”, describes estimation of regions of individual images captured by the optical sensor) or (ii) the unmatched region, and …
Therefore, it would have obvious to one of the ordinary skill in the art before the effective filling date of the claimed invention to modify Konishi to include the teachings of Hinterstoisser and include a feature of determining a bounding region encompassing the primary candidate region and at least one of the one or more additional candidate regions, thereby making it possible to perform image analysis more accurately and controlling the robot can be done precisely.
Claim(s) 33 is rejected under 35 U.S.C. 103 as being unpatentable over Konishi (US 20190197727 A1) in view of Katagiri et al. (US 10596707 B2) (Hereinafter Katagiri).
Regarding Claim 33, Konishi teaches all the elements of claim 21.
However, Konishi does not explicitly spell out the computing system of claim 21, wherein the
primary candidate region represents a first manner of aligning the image information with the primary object template, and
wherein the at least one processing circuit is further configured to include in the safety volume list another candidate region which represents a second manner of aligning the image information with the primary object template.
Katagiri teaches the computing system of claim 21, wherein the primary candidate region
represents a first manner of aligning the image information with the primary object template (Fig 2 item 24g, 24h discloses interference area setting and determination program, Para [0032] discloses interference determination for each workpiece which construed the claimed invention), and
wherein the at least one processing circuit is further configured to include in the safety volume list another candidate region which represents a second manner of aligning the image information with the primary object template (Fig 2 item 24g, 24h discloses interference area setting and determination program, Para [0032] discloses interference determination for each workpiece which construed the claimed invention).
Therefore, it would have obvious to one of the ordinary skill in the art before the effective filling date of the claimed invention to modify Konishi with the teachings of Katagiri and include a feature to include a second manner of aligning the image information to the safety volume list which will make it possible to perform image analysis more accurately and controlling the robot can be done precisely.
Claim(s) 35 is rejected under 35 U.S.C. 103 as being unpatentable over Konishi (US 20190197727 A1) in view of Mori et al. (US 2019/0224847 A1) (Hereinafter Mori).
Regarding Claim 35, Konishi teaches all the elements of claim 21. Konishi further teaches the computing system of claim 21, wherein the at least one processing circuit is further configured to generate a new object recognition template based on the unmatched region (Fig 3 item 202 template creation unit, Fig 4 Item S402 generate template),
However, Konishi does not explicitly spell out … in response to a determination that the image information has the unmatched region.
Mori teaches … in response to a determination that the image information has the unmatched region (“Para [0034] describes area other than the gripping target object which is construed as unmatched region”).
Therefore, it would have obvious to the ordinary skill in the art before the effective filling date of the claimed invention to modify Konishi with the teachings of Mori and include a feature that describes the unmatched region in order to generate a safety volume list which will make it possible to perform image analysis more accurately and controlling the robot can be done precisely.
Claim(s) 36 is rejected under 35 U.S.C. 103 as being unpatentable over Konishi (US 20190197727 A1) in view of Hinterstoisser et al. (US 9327406 B1) (Hereinafter Hinterstoisser), and further in view of Katagiri et al. (US 10596707 B2) (Hereinafter Katagiri).
Regarding Claim 36, Konishi teaches all the elements of claim 21.
However, Konishi does not explicitly spell out the computing system of claim 21, wherein the
primary candidate region represents a first orientation for an object shape described by the primary object template, and wherein the at least one processing circuit is further configured to add, to the safety volume list, a candidate region that represents a second orientation for the object shape, the second orientation being perpendicular to the first orientation.
Hinterstoisser teaches … the second orientation being perpendicular to the first orientation (Col 22 lines 45-50 discloses vertical tape).
Therefore, it would have obvious to the ordinary skill in the art before the effective filling date of the claimed invention to modify Konishi to include the teachings of Hinterstoisser and include a feature to add the second orientation which is being perpendicular to the first orientation for an object shape to the safety volume list which will make it possible to perform image analysis more accurately and controlling the robot can be done precisely.
Katagiri discloses the computing system of claim 21, wherein the primary candidate region
represents a first orientation for an object shape described by the primary object template, and wherein the at least one processing circuit is further configured to add, to the safety volume list, a candidate region that represents a second orientation for the object shape (Fig 2 item 24g, 24h, discloses interference area setting and determination program, Para [0032] discloses interference determination for each workpiece which constitutes the claimed invention),
Therefore, it would have obvious to the ordinary skill in the art before the effective filling date of the claimed invention to modify Konishi with the teachings of Katagiri and include a feature to add the second orientation for an object shape to the safety volume list which will make it possible to perform image analysis more accurately and controlling the robot can be done precisely.
Allowable Subject Matter
Claim 32 and 34 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims, and overcome the 101 and double patenting rejection set forth in this office action.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure:
US 20170177970 A1 (Kitajima) teaches controlling a robot using image processing through pattern matching where image edge is taken into consideration
US 20180126553 A1 (Corkum et al.) teaches estimating location of a target object within the range of an end effector
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SHAHEDA HOQUE whose telephone number is (571)270-5310. The examiner can normally be reached Monday-Friday 8:00 am- 5:00 pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Ramon Mercado can be reached at 571-270-5744. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/SHAHEDA HOQUE/Examiner, Art Unit 3658
/Ramon A. Mercado/Supervisory Patent Examiner, Art Unit 3658
Prosecution Insights