DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Status of Claims
Pending
55-76
Cancelled
1-54
35 U.S.C. 101
67-68, 73-75
35 U.S.C. 102
55, 57-61, 63, 65-68, 73-76
35 U.S.C. 103
56, 62, 64, 69-72
Priority
Applicant’s indication of National Stage information based on PCT/JP2021/048572 filed 12/27/2021 is acknowledged.
Information Disclosure Statement
The information disclosure statement(s) (IDS(s)) submitted on 11/07/2025, 09/27/2024, and 06/26/2024 is/are in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement(s) is/are being considered by the examiner.
Claim Objections
Claims 57 and 63 are objected to because of the following informalities:
Claim 57 recites: “The imaging system according to claim 55, wherein the first imaging apparatus and the second imaging apparatus are mounted to the robot.” However this should amended to read: “The imaging system according to claim 55, wherein the first imaging apparatus and the second imaging apparatus are mounted to a[[the]] robot.”
Claim 63 recites: “The imaging system according to claim 55, wherein the first imaging apparatus and the second imaging apparatus capture the object illustrated with the first illumination light and the second illumination light.” This should instead read: “The imaging system according to claim 55, wherein the first imaging apparatus and the second imaging apparatus capture the object illuminated
Appropriate correction is required.
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action.
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because the claim limitation(s) uses a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier.
Such claim limitation(s) is/are: first imaging apparatus, second imaging apparatus, a first filter, and a second filter in claim 55; a first illumination apparatus and a second illumination apparatus in claim 58; a process apparatus in claim 65; and the control apparatus in claim 74.
Because this/these claim limitation(s) is/are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, it/they is/are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof.
The specification discloses the corresponding structure for the above claim limitations in the following paragraphs (cited from PGPub version of application (US 2025/0214249 A1)): first imaging apparatus ([0047], “monocular camera”), second imaging apparatus ([0048], “stereo camera”), a first filter ([0202], “attenuating light component in the first wavelength bandwidth”), a second filter ([0203], “attenuating light component in the second wavelength bandwidth”), a first illumination apparatus ([0050], “projection apparatus 23”), a second illumination apparatus ([0201], “illumination light”), a process apparatus ([0212] “end effector”, “hand gripper”; [0213], “a tool”), and the control apparatus ([0059] “Configuration of Control Apparatus 3”; [0061], “a CPU”).
If applicant does not intend to have this/these limitation(s) interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation(s) to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation(s) recite(s) sufficient structure to perform the claimed function so as to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 67-68 and 73-75 are rejected under 35 U.S.C 101 because the claimed invention is directed to an abstract idea without significantly more.
Claim 67 is rejected under 35 U.S.C 101 because the claimed invention is directed to an abstract idea without significantly more. The claim recites:
“[Claim 55.] An imaging system that is used to control a robot for performing a processing on an object, wherein the imaging system comprises:
a first imaging apparatus;
a second imaging apparatus;
a first filter that attenuates light component in a first wavelength bandwidth of light from the object; and
a second filter that attenuates light component in a second wavelength bandwidth, which is different from the first wavelength bandwidth, of the light from the object, wherein
the first imaging apparatus acquires first image data of the object by optically receiving light from the object through the first filter, and
the second imaging apparatus acquires second image data of the object by optically receiving light from the object through the second filter.
[Claim 66.] A control system comprising:
the imaging system according to claim 55; and
a control apparatus.
[Claim 67.] The control system according to claim 66, wherein the control apparatus calculates position information, which includes at least one of a position and a pose of the object, based on the first image data and the second image data.”
These limitations, as drafted, are simple processes that, under their broadest reasonable interpretation, cover performance of the mind, but for the recitation of “an imaging system that is used to control a robot for performing a processing on an object, wherein the imaging system comprises: a first imaging apparatus; a second imaging apparatus; a first filter that attenuates light component in a first wavelength bandwidth of light from the object; and a second filter that attenuates light component in a second wavelength bandwidth, which is different from the first wavelength bandwidth, of the light from the object, wherein the first imaging apparatus acquires first image data of the object by optically receiving light from the object through the first filter, and the second imaging apparatus acquires second image data of the object by optically receiving light from the object through the second filter; a control system comprising: the imaging system according to claim 55; and a control apparatus; the control system according to claim 66, wherein the control apparatus”. That is, other than reciting the underlined and italicized limitations above, nothing in the claim elements preclude the steps from being performed in the mind. For example, a human can, in their mind, calculate position information, which includes at least one of a position and a pose of the object, based on the first image data and the second image data.
This judicial exception is not integrated into a practical application. The claim recites the additional elements underlined and italicized above. The an imaging system used to control a robot, a first imaging apparatus, a second imaging apparatus, a first filter, a second filter, a control system, and a control apparatus is/are recited at a high level of generality and merely link(s) the use of the abstract idea to a particular technological environment (see MPEP 2106.05(h)).
The acquires first image data of the object through the first filter and acquires second image data of the object through the second filter steps is/are recited at a high level of generality and amounts to mere data gathering, manipulation, and transmission, which is a form of insignificant extra-solution activity (see MPEP 2106.05(g)). Accordingly, even in combination, the additional elements do not integrate the abstract idea into a practical application because they do not impose any meaningful limits on practicing the abstract idea.
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. The additional element of an imaging system used to control a robot, a first imaging apparatus, a second imaging apparatus, a first filter, a second filter, a control system, and a control apparatus is/are no more than mere generic linking of the abstract idea to a technological environment, which cannot provide an inventive concept.
The additional element of acquires first image data of the object through the first filter and acquires second image data of the object through the second filter steps is/are mere data gathering, manipulation, and transmission, and is a well-understood, routine, and conventional function (see MPEP 2106.05(d) and see Versata Dev. Group, Inc. v. SAP Am., Inc., 793 F.3d 1306, 1334, 115 USPQ2d 1681, 1701 (Fed. Cir. 2015); OIP Techs., 788 F.3d at 1363, 115 USPQ2d at 1092-93), and thus is/are no more than insignificant extra-solution activity (see MPEP 2106.05(g) and see OIP Techs., 788 F.3d at 1362-63, 115 USPQ2d at 1092-93). Thus, the limitations do not provide an inventive concept, and the claim contains ineligible subject matter.
Claim(s) 68 recite(s) limitations that are no more that the abstract idea recited in claim(s) 67 . The claim(s) recite(s) calculating steps which can reasonably be performed in the human mind. The claim(s) recite(s) the control apparatus at a high level of generality to generically link the use of the abstract idea in a particular technological environment. Thus, the claim(s) contain(s) ineligible subject matter.
Claim(s) 73 and 74 recite(s) limitations that are no more that the abstract idea recited in claim(s) 66. The claims recite the content of claims 55 and 66, as described above. The claims further recite:
“[Claim 73.] The control system according to 66, wherein the control apparatus:
calculates a first position, which includes at least one of a position and a pose of the object, by performing a matching processing using the first image data and reference image data indicating a two-dimensional image that is a reference of the object; and
calculates a second position, which includes at least one of a position and a pose of the object, by performing a matching processing using the second image data and reference shape data indicating a three-dimensional image that is a reference of the object.
[Claim 74.] The control system according to claim 66, wherein the control apparatus:
performs an image processing on at least one of the first image data and the second image data;
is configured to calculate a first position, which includes at least one of a position and a pose of the object, based on the first image data on which the image processing has been performed; and
is configured to calculate a second position, which includes at least one of a position and a pose of the object, based on shape data generated from the second image data.”
The calculating steps can reasonably be performed in the human mind. The claim(s) recite(s) the control apparatus at a high level of generality to generically link the use of the abstract idea in a particular technological environment. Claim 74 recite(s) performing image processing steps which is/are mere data gathering, manipulation, and transmission, and is/are a well-understood, routine, and conventional function, and thus is/are no more than insignificant extra-solution activity. See MPEP 2106.05(g). Thus, the claim(s) contain(s) ineligible subject matter.
Claim(s) 75 recite(s) limitations that are no more that the abstract idea recited in claim(s) 74. The claim(s) recite(s) image processing including gamma correction, HDR processing, and de-noise processing steps which is/are mere data gathering, manipulation, and transmission, and is/are a well-understood, routine, and conventional function, and thus is/are no more than insignificant extra-solution activity. See MPEP 2106.05(g). Thus, the claim(s) contain(s) ineligible subject matter.
Claim(s) 69 recite(s) limitations that incorporate the abstract idea into a practical application. Thus, the claim(s) contain(s) eligible subject matter.
Claims 70-72 depend upon claim 69 and therefore recite limitations that incorporate the abstract idea into a practical application, and thus similarly contain eligible subject matter.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claim(s) 55, 57-61, 63, 65-68, 73-76 is/are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Konolige (US 2016/0288330 A1, hereinafter “Konolige”).
Regarding claim 55: Konolige teaches: An imaging system that is used to control a robot for performing a processing on an object, wherein the imaging system comprises ([0034] Methods, optical sensors, robotic systems improve depth sensing in environments with computer vision system using stereo image processing. light reflected off objects can capture detail of objects. [0044] methods and apparatuses to facilitate manipulation of other objects. [0051] robotic device. mechanical, sensing, and control systems, power supply):
a first imaging apparatus ([0037] optical sensor detecting light in wavelength bands (visible, IR). [0040] optical sensors capture images of environment from different viewpoints. stereoscopically image environment in multiple wavelength bands. stereo image. [0053] sensors attached to robotic arm);
a second imaging apparatus ([0037] optical sensor detecting light in wavelength bands (visible, IR). [0040] optical sensors capture images of environment from different viewpoints. stereoscopically image environment in multiple wavelength bands. [0053] sensors attached to robotic arm);
a first filter that attenuates light component in a first wavelength bandwidth of light from the object ([0091] optical sensor includes photodetectors to generate charge when exposed to incident beam of light whose wavelength is within visible spectrum (380 nm to 750 nm). [0094] photodetectors coupled to light filters. light filters overlaid on top of photodetectors so light incident on particular photodetector first passes through respective light filter. Each light filter act as band-pass filter that passes through light whose wavelength is within particular band, while blocking or attenuating light whose wavelength is outside of particular band. [0060] identify object); and
a second filter that attenuates light component in a second wavelength bandwidth, which is different from the first wavelength bandwidth, of the light from the object ([0091] optical sensor 402 captures light wavelength within visible spectrum (380 nm to 750 nm). [0092] other photodetectors generate charge when exposed to incident beam of light whose wavelength is within IR spectrum (750 nm to 3000 nm or 825 nm to 875 nm). [0094] light filters attenuating light whose wavelength is outside of particular band. [0060] identify object), wherein
the first imaging apparatus acquires first image data of the object by optically receiving light from the object through the first filter ([0092] other photodetectors generate charge when exposed to incident beam of light whose wavelength is within IR spectrum (750 nm to 3000 nm or 825 nm to 875 nm). [0099] optical sensor capture visible light image using its visible-sensing photodetectors and IR light image using its IR-sensing photodetectors. images used from first optical sensor and from second optical sensor. [0040] two or more optical sensors capture images of environment from different viewpoints. image environment in multiple wavelength bands. [0064] model of objects in 2D or 3D. [0121] optical sensors capture visible light image from visible light reflected off environment and IR light image from IR projected pattern reflected off environment), and
the second imaging apparatus acquires second image data of the object by optically receiving light from the object through the second filter ([0092] other photodetectors generate charge when exposed to incident beam of light whose wavelength is within IR spectrum (750 nm to 3000 nm or 825 nm to 875 nm). [0099] optical sensor capture visible light image using its visible-sensing photodetectors and IR light image using its IR-sensing photodetectors. images used from first optical sensor and from second optical sensor. [0040] two or more optical sensors capture images of environment from different viewpoints. image environment in multiple wavelength bands. [0064] model of objects in 2D or 3D. [0121] optical sensors capture visible light image from visible light reflected off environment and IR light image from IR projected pattern reflected off environment).
Regarding claim 57: Konolige further teaches: The imaging system according to claim 55, wherein the first imaging apparatus and the second imaging apparatus are mounted to the robot ([0053] sensing system may use one or more sensors attached to robotic arm. [0081] cameras located on robotic arm).
Regarding claim 58: Konolige further teaches: The imaging system according to claim 55 further comprising: a first illumination apparatus that illuminates the object with first illumination light including light component in the first wavelength bandwidth; and a second illumination apparatus that illuminates the object with second illumination light including light component in the second wavelength bandwidth ([0040] image environment in multiple wavelength bands. [0041] texture projector projects known structured-light pattern onto environment within particular IR light band. [0116] environment illuminated by visible light. sunlight or artificial lighting (incandescent light bulbs, fluorescent light bulbs, light-emitting diodes). illuminate objects within environment. [0158] Texture projector project IR projected pattern. first optical sensor and second optical sensor present and capture both visible light images and IR light images of environment. [0118] projected pattern detectable when reflected from objects. [0117] environment illuminated by IR projected pattern from texture projector (IR light emitter or IR projector). interrupting light with pattern of grids, dots, stripes, spots, horizontal bars, or vertical bars).
Regarding claim 59: Konolige further teaches: The imaging system according to claim 58, wherein the second illumination light is pattern light ([0035] projecting patterned light onto scene, capturing images with pattern. Pattern in particular IR wavelength band. [0041] texture projector projects known structured-light pattern onto environment. [0118] projected pattern reflected from objects).
Regarding claim 60: Konolige further teaches: The imaging system according to claim 58, wherein the second image data, which is acquired by capturing the object illuminated with the second illumination light, include information related to a shape of the object ([0041] texture projector projects known structured-light pattern onto environment within particular IR light band. [0116] environment illuminated by visible light. sunlight or artificial lighting (incandescent light bulbs, fluorescent light bulbs, light-emitting diodes). illuminate objects within environment. [0158] Texture projector project IR projected pattern. first optical sensor and second optical sensor present and capture both visible light images and IR light images of environment. [0044] implemented to find 3D geometry of objects in environment. [0146] identifying feature from image from one perspective and locating that same feature within another image from different perspective).
Regarding claim 61: Konolige further teaches: The imaging system according to claim 55, wherein the second imaging apparatus includes a stereo camera ([0034] computer vision system using stereo image processing. [0036] stereo image processing and structured light processing. [0095] visible light and IR light incident on optical sensor. images provided for stereo image processing and depth sensing. [0100] stereo image processing with two different types of light (IR light and visible light) to determine depth estimates).
Regarding claim 63: Konolige further teaches: The imaging system according to claim 55, wherein the first imaging apparatus and the second imaging apparatus capture the object illustrated with the first illumination light and the second illumination light ([0158] Texture projector placed at distance from wall and box, project IR projected pattern. first optical sensor and second optical sensor present and capture both visible light images and IR light images of environment. [0041] texture projector projects pattern within particular IR light band. [0116] environment illuminated by visible light. (incandescent, fluorescent light bulbs, LEDs). illuminate objects within environment. [0158] first optical sensor and second optical sensor present and capture both visible light images and IR light images of environment. [0118] projected pattern detectable when reflected from objects).
Regarding claim 65: Konolige further teaches: The imaging system according to claim 55, wherein a process apparatus that performs the process on the object is mounted to the robot, the robot is configured to move the process apparatus ([0047] sensors, computers, robotic arms. reconstructed environment used for identifying objects to pick up, determining pick positions for objects, or planning collision-free trajectories for robotic arms. [0044] methods and apparatuses to facilitate manipulation of other objects. implemented to find 3D geometry of objects in environment. [0049] robotic manipulator mounted on holonomic cart. robotic arm mounted on cart. gripping component for gripping objects).
Regarding claim 66: Konolige further teaches: A control system comprising ([0061] control system): the imaging system according to claim 55 ([0053] sensing system); and a control apparatus ([0053] sensing system. [0061] control system. [0047] sensors, computers, robotic arms).
Regarding claim 67: Konolige further teaches: The control system according to claim 66, wherein the control apparatus calculates position information ([0047] sensors, computers, robotic arms. reconstructed environment used for identifying objects to pick up, determining pick positions for objects, or planning collision-free trajectories for robotic arms. [0074] location of objects and surfaces in the environment),
which includes at least one of a position and a pose of the object, based on the first image data and the second image data ([0042] depth info of environment based on corresponding features between pair of visible light images captured by imaging system. using triangulation. visible light-based depth info. IR light-based depth info. combine depth info derived from both visible and IR light images to determine more accurate depth map. [0044] determine the 3D geometry of the environment or 3D geometry of objects in the environment. [0139] identifying second corresponding features of environment between first IR light image and second IR light image).
Regarding claim 68: Konolige further teaches: The control system according to claim 67, wherein the control apparatus calculates the position information, which includes at least one of the position and the pose of the object, based on the first image data and shape data that is acquired based on the second image data and that indicates a three-dimensional shape of the object ([0044] implemented to find 3D geometry of objects in environment. [0137] receiving, from second plurality of photodetectors of first optical sensor, first IR light image of environment as perceived from first viewpoint. [0138] receiving, from second plurality of photodetectors of second optical sensor, second IR light image of environment as perceived from second viewpoint. [0146] identifying feature from image from one perspective and locating that same feature within another image from different perspective. [0153] two optical sensors capture pair of visible light images and pair of IR light images. determine one correlation surface from pair of visible light images, and another correlation surface from pair of IR light images. combine two correlation surfaces to determine combined correlation surface).
Regarding claim 73: Konolige further teaches: The control system according to 66, wherein the control apparatus: calculates a first position, which includes at least one of a position and a pose of the object, by performing a matching processing using the first image data and reference image data indicating a two-dimensional image that is a reference of the object ([0099] images used from first optical sensor and from second optical sensor. [0040] two or more optical sensors capture images of environment from different viewpoints. image environment in multiple wavelength bands. [0064] model of objects in 2D or 3D. [0121] optical sensors capture visible light image from visible light reflected off environment and IR light image from IR projected pattern reflected off environment. [0042] depth info of environment based on corresponding features between pair of visible light images captured by imaging system. Using triangulation. Visible light-based depth info. IR light-based depth info. Combine depth info derived from both visible and IR light images to determine more accurate depth map); and
calculates a second position, which includes at least one of a position and a pose of the object, by performing a matching processing using the second image data and reference shape data indicating a three-dimensional image that is a reference of the object ([0042] depth info of environment based on corresponding features between pair of visible light images captured by imaging system. Using triangulation. Visible light-based depth info. IR light-based depth info. Combine depth info derived from both visible and IR light images to determine more accurate depth map. [0047] identifying objects to pick up, determining pick positions for objects, or planning collision-free trajectories for robotic arms. [0074] location of objects and surfaces in the environment. [0044] determine the 3D geometry of the environment or 3D geometry of objects in the environment. [0139] identifying second corresponding features of environment between first IR light image and second IR light image).
Regarding claim 74: Konolige further teaches: The control system according to claim 66, wherein the control apparatus: performs an image processing on at least one of the first image data and the second image data ([0099] optical sensor capture visible light image using its visible-sensing photodetectors and IR light image using its IR-sensing photodetectors. images used from first optical sensor and from second optical sensor. [0040] two or more optical sensors capture images of environment from different viewpoints. image environment in multiple wavelength bands. [0064] model of objects in 2D or 3D. [0121] optical sensors capture visible light image from visible light reflected off environment and IR light image from IR projected pattern reflected off environment);
is configured to calculate a first position, which includes at least one of a position and a pose of the object, based on the first image data on which the image processing has been performed ([0047] identifying objects to pick up, determining pick positions for objects, or planning collision-free trajectories for robotic arms. [0074] location of objects and surfaces in the environment. [0042] depth info of environment based on corresponding features between pair of visible light images captured by imaging system. using triangulation. visible light-based depth info. IR light-based depth info. combine depth info derived from both visible and IR light images to determine more accurate depth map. [0044] determine the 3D geometry of the environment or 3D geometry of objects in the environment. [0139] identifying second corresponding features of environment between first IR light image and second IR light image); and
is configured to calculate a second position, which includes at least one of a position and a pose of the object, based on shape data generated from the second image data ([0041] texture projector projects known structured-light pattern onto environment within particular IR light band. [0116] environment illuminated by visible light. sunlight or artificial lighting (incandescent light bulbs, fluorescent light bulbs, light-emitting diodes). illuminate objects within environment. [0158] Texture projector project IR projected pattern. first optical sensor and second optical sensor present and capture both visible light images and IR light images of environment. [0044] implemented to find 3D geometry of objects in environment. [0146] identifying feature from image from one perspective and locating that same feature within another image from different perspective).
Regarding claim 75: Konolige further teaches: The control system according to claim 74, wherein the image processing includes at least one of a gamma correction processing, a HDR (High-Dynamic-Range) processing, and a de-noise processing ([0066] virtual environment built up using 3D volumetric or surface model to integrate info (from different sensors). allow system to operate within larger environment. increase level of detail captured, help robotic device perform various tasks. integrating info yield finer detail than from single scan alone (by bringing down noise levels). enable better object detection, surface picking, or other applications).
Regarding claim 76: Konolige teaches: An imaging method that is used to control a robot for performing a processing on an object, wherein the imaging method comprises ([0034] Methods, optical sensors, robotic systems improve depth sensing in environments with computer vision system using stereo image processing. light reflected off objects can capture detail of objects. [0044] methods and apparatuses to facilitate manipulation of other objects. implemented to find 3D geometry of objects in environment):
acquiring first image data by using a first imaging apparatus to optically receive light from the object through a first filter that attenuates light component in a first wavelength bandwidth of light from the object ([0037] optical sensor detecting light in wavelength bands (visible, IR). [0040] two or more optical sensors capture images of environment from different viewpoints. stereoscopically image environment in multiple wavelength bands. stereo image processing to generate multiple depth maps based on pairs of images from different viewpoints in each wavelength band. [0091] optical sensor includes photodetectors to generate charge when exposed to incident beam of light whose wavelength is within visible spectrum (380 nm to 750 nm). [0092] other photodetectors generate charge when exposed to incident beam of light whose wavelength is within IR spectrum (750 nm to 3000 nm or 825 nm to 875 nm). [0094] photodetectors coupled to light filters. light filters overlaid on top of photodetectors so light incident on particular photodetector first passes through respective light filter. Each light filter act as band-pass filter that passes through light whose wavelength is within particular band, while blocking or attenuating light whose wavelength is outside of particular band), and
acquiring second image data by using a second imaging apparatus to optically receive light from the object through a second filter that attenuates light component in a second wavelength bandwidth, which is different from the first wavelength bandwidth, of the light from the object ([0037] optical sensor detecting light in wavelength bands (visible, IR). [0040] two or more optical sensors capture images of environment from different viewpoints. stereoscopically image environment in multiple wavelength bands. stereo image processing to generate multiple depth maps based on pairs of images from different viewpoints in each wavelength band. [0091] optical sensor includes photodetectors to generate charge when exposed to incident beam of light whose wavelength is within visible spectrum (380 nm to 750 nm). [0092] other photodetectors generate charge when exposed to incident beam of light whose wavelength is within IR spectrum (750 nm to 3000 nm or 825 nm to 875 nm). [0094] photodetectors coupled to light filters. light filters overlaid on top of photodetectors so light incident on particular photodetector first passes through respective light filter. Each light filter act as band-pass filter that passes through light whose wavelength is within particular band, while blocking or attenuating light whose wavelength is outside of particular band).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claim(s) 56, 62, 64, 69-72 is/are rejected under 35 U.S.C. 103 as being unpatentable over Konolige (US 2016/0288330 A1, hereinafter “Konolige”), and further in view of Taamazyan et al. (US 2022/0405506 A1, hereinafter “Taamazyan”).
Regarding claim 56: Konolige further teaches: The imaging system according to claim 55. However, Konolige does not explicitly teach, but Taamazyan teaches: wherein a time at which the imaging system captures the object to acquire the first image data is the same as a time at which the imaging system captures the object to acquire the second image data ([0128] multiple cameras (having different imaging modalities or sensitivity to different spectra) capture images; a single trigger used to control all cameras to capture images concurrently or simultaneously). See [0134] and [0152]).
Konolige and Taamazyan are analogous art to the claimed invention since they are from the similar field of robotic controls based on computer vision and image filtering. It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to modify the invention of Konolige with the aspects of Taamazyan to create, with a reasonable expectation for success, an imaging system wherein first image data and second image data are captured at the same time. The motivation for modification would have been to improve the detection and location (e.g., pose estimation) of small objects and larger objects, using the large number of correspondences and multiple viewpoints to perform filtering or smoothing to improve the accuracy of the dense image-to-object correspondences (Taamazyan, [0093], [0205]).
Regarding claim 62: Konolige further teaches: The imaging system according to claim 55. However, Konolige does not explicitly teach, but Taamazyan teaches: wherein the first imaging apparatus includes a monocular camera ([0099] multiple cameras. stereo camera used with monocular camera. [0118] main camera and support cameras. main camera includes a stereo camera include at least two monocular cameras spaced apart, with overlapping fields of view and parallel optical axes. support cameras may be stereo cameras, monocular cameras, or combinations thereof).
Konolige and Taamazyan are analogous art to the claimed invention since they are from the similar field of robotic controls based on computer vision and image filtering. It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to modify the invention of Konolige with the aspects of Taamazyan to create, with a reasonable expectation for success, an imaging system wherein one of the imaging apparatuses includes a monocular camera. The motivation for modification would have been to improve the detection and location (e.g., pose estimation) of small objects and larger objects, using the large number of correspondences and multiple viewpoints to perform filtering or smoothing to improve the accuracy of the dense image-to-object correspondences (Taamazyan, [0093], [0205]).
Regarding claim 64: Konolige further teaches: The imaging system according to claim 55. However, Konolige does not explicitly teach, but Taamazyan teaches: wherein a time at which the first imaging apparatus captures the object is the same as a time at which the second imaging apparatus captures the object ([0128] multiple cameras (having different imaging modalities or sensitivity to different spectra) capture images; a single trigger used to control all cameras to capture images concurrently or simultaneously). See [0134] and [0152]).
Konolige and Taamazyan are analogous art to the claimed invention since they are from the similar field of robotic controls based on computer vision and image filtering. It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to modify the invention of Konolige with the aspects of Taamazyan to create, with a reasonable expectation for success, an imaging system wherein first image data and second image data are captured at the same time. The motivation for modification would have been to improve the detection and location (e.g., pose estimation) of small objects and larger objects, using the large number of correspondences and multiple viewpoints to perform filtering or smoothing to improve the accuracy of the dense image-to-object correspondences (Taamazyan, [0093], [0205]).
Regarding claim 69: Konolige further teaches: The control system according to claim 67, wherein the control apparatus generates a control signal based on the position information [to control the robot] ([0053] sensors attached to robotic arm. [0046] process palletizing/un-palletizing objects include robotic devices to move objects. [0049] Robotic arm contain gripping component for gripping objects within environment. [0057] robotic arm picks up objects).
However, Konolige does not explicitly teach, but Taamazyan teaches: wherein the control apparatus generates a control signal based on the position information so that the robot moves closer to the object ([0059] controllers move end effector from first state to second state. [0068]-[0075] end effector approaches target object. capture images of objects in scene. retrieves 3D shape and pose of objects. motion plan for picking up target object from bin. optimize path taken by robot arm, approaches target object according to motion plan. attempts pick of target object when at pick location. determine whether pick was successful. If NO, feedback provided, process repeats to re-determine shape of end effector and motion plan using updated images of scene. Repeat process. [0078] robot arm progresses along motion path based on updated images of scene provided by cameras. [0079] updated images of scene after robot arm navigates. continue until robot arm achieves successful pick. updating of end effector based on updated images of scene. [0128] multiple cameras (with different imaging modalities) controlled to capture images in group (a single trigger used to control all cameras to capture images concurrently or substantially simultaneously)).
Konolige and Taamazyan are analogous art to the claimed invention since they are from the similar field of robotic controls based on computer vision and image filtering. It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to modify the invention of Konolige with the aspects of Taamazyan to create, with a reasonable expectation for success, an imaging system wherein control signals are generated based on position information to move the robot closer to the object. The motivation for modification would have been to improve the detection and location (e.g., pose estimation) of small objects and larger objects, using the large number of correspondences and multiple viewpoints to perform filtering or smoothing to improve the accuracy of the dense image-to-object correspondences (Taamazyan, [0093], [0205]).
Regarding claim 70: Konolige-Taamazyan further teach: The control system according to claim 69, wherein the control signal is generated based on the position information that is calculated based on the first image data, which is acquired during a relative movement between the first imaging system and the object, and the second image data, which is acquired during the relative movement (Konolige: [0047] identifying objects to pick up, pick positions, collision-free trajectories for robotic arms. [0042] images captured. use triangulation. visible light-based and IR light-based depth info. combine to determine depth map. Taamazyan: [0059] controllers move end effector from first state to second state. [0068]-[0075] end effector approaches target object. capture images of objects in scene. retrieves 3D shape and pose of objects. motion plan for picking up target object from bin. optimize path taken by robot arm, approaches target object according to motion plan. attempts pick of target object when at pick location. determine whether pick was successful. If NO, feedback provided, process repeats to re-determine shape of end effector and motion plan using updated images of scene. Repeat process. [0078] robot arm progresses along motion path based on updated images of scene provided by cameras. [0079] updated images of scene after robot arm navigates. continue until robot arm achieves successful pick. updating of end effector based on updated images of scene. [0128] multiple cameras (with different imaging modalities) controlled to capture images in group (a single trigger used to control all cameras to capture images concurrently or substantially simultaneously).
The motivation for modification would have been to improve the detection and location (e.g., pose estimation) of small objects and larger objects, using the large number of correspondences and multiple viewpoints to perform filtering or smoothing to improve the accuracy of the dense image-to-object correspondences (Taamazyan, [0093], [0205]).
Regarding claim 71: Konolige-Taamazyan further teach: The control system according to claim 69 wherein the position information is first position information, the control signal is a first control signal, and the control apparatus: calculates second position information, which includes at least one of a position and a pose of the object, based on third image data that is acquired by the first imaging apparatus capturing the object at a time different from a time at which the first image data is acquired; and generates a control signal, which is different from the first control signal, based on the second position information so that the robot moves closer to the object (Taamazyan: [0059] move end effector from first to second state. [0068]-[0075] end effector approaches target object. capture images of objects. retrieves 3D shape and pose. motion plan for picking up object. optimize path taken by robot arm, approaches target object. attempts pick up object. determine whether successful. If NO, feedback provided, process repeats to redetermine motion plan using updated images of scene. Repeat process. [0078] robot arm progresses along motion path based on updated images of scene. [0079] updated images of scene after robot arm navigates. continue until robot arm achieves successful pick. updating of end effector based on updated images. [0128] a single trigger used to control all cameras to capture images concurren