DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Receipt is acknowledged of certified copies of papers submitted under 35 U.S.C. 119(a)-(d), which papers have been placed of record in the file.
Information Disclosure Statement
The information disclosure statements (IDS) submitted on 12/20/2024 and 07/09/2025 have been considered by the examiner.
Claim Objections
Claims 1 and 8 are objected to because of the following informalities:
In claim 1, line 9, “wherein the processor acquires…” should read “wherein the processor: acquires…” in order to add a colon before the list of actions following the introductory clause.
In claim 1, lines 10-18, the listed actions should be separated by semicolons instead of commas such that “acquires,… plurality of marks, acquires,… by imaging the board, detects…captured image, and calculates… detected marks, detects… infrared image, and calculates… detected marks, and outputs… second index.” should read “acquires,… plurality of marks[[,]] ; acquires,… by imaging the board[[,]] ; detects…captured image[[,]] ; and calculates… detected marks[[,]] ; detects… infrared image[[,]] ; and calculates… detected marks[[,]] ; and outputs… second index.” in order to improve readability.
In claim 8, line 15, “wherein… the information processing device acquires…” should read “wherein… the information processing device: acquires…” in order to add a colon before the list of actions following the introductory clause.
In claim 8, lines 16-24, the listed actions should be separated by semicolons instead of commas such that “acquires,… plurality of marks, acquires,… imaging the board, detects… captured image, and calculates… detected marks, detects… infrared image, and calculates… detected marks, and outputs…second index” should read “acquires,… plurality of marks[[,]] ; acquires,… imaging the board[[,]] ; detects… captured image[[,]] ; and calculates… detected marks[[,]] ; detects… infrared image[[,]] ; and calculates… detected marks[[,]] ; and outputs…second index” in order to improve readability.
Appropriate correction is required.
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action.
Claims 6-9 recite limitations that use words like “means” (or “step”) or similar terms with functional language and do invoke 35 U.S.C. 112(f):
Claim 6; recites the limitation, “the projection device is able to project…” [Line 26].
Claim 7; recites the limitation, “the projection device projects…” [Line 32].
Claim 8; recites the limitation, “a projection device that projects a projection image onto a parcel being transported…,” [Line 9].
Claim 8; recites the limitation, “an information processing device that controls the projection device…,” [Line 12].
Claim 8; recites the limitation, “the information processing device acquires…,” [Lines 15-24].
Claim 9; recites the limitation, “an information processing device performs calibration related to a position of an imaging camera…,” [Lines 26-27].
Because this/these claim limitation(s) is/are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, it/they is/are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof.
After a careful analysis, as disclosed above, and a careful review of the specification the following limitations in claims 6-9:
“projection device” (Fig. 1, #105 called projector, Paragraph [0022] – “The projector 105 is a projection device that projects a projection image including a predetermined projection pattern to a parcel on the transport path based on an instruction from the sorting client 104. In the present embodiment, the "projection image" refers to an entire image projected by the projector 105 so as to be superimposed on an entire sorting area. The "projection pattern" refers to an image that is displayed in a superimposed manner corresponding to each parcel detected in the sorting area. Therefore, when parcels are detected in the sorting area, the projection image includes one or a plurality of projection patterns depending on the parcels.” Thus, the projection device does have sufficient structure associated with it wherein it is a projector.
“information processing device” (Fig. 2, #200 called information processing device, Paragraph [0027] – “Fig. 2 is a diagram showing an example of a hardware configuration of an information processing device that can be used as the sorting management server 101, the link server 102, or the sorting client 104 according to the present embodiment.” Paragraph [0119] – “An information processing device (104, 200) according to the present disclosure is an information processing device that performs calibration related to a position of an imaging camera 107 that captures a captured image used for detection of a parcel 300 being transported and a position of an infrared camera 106 that captures an infrared image used for detection of the parcel 300, the information processing device including a processor 201 and a memory 202. The processor 201 : acquires, from the imaging camera 107, the captured image obtained by imaging a board 820 having a plurality of marks 823; acquires, from the infrared camera 106, the infrared image obtained by imaging the board 820; detects the marks 823 from the captured image, and calculates a first index 911 related to calibration of the captured image based on the number of the detected marks 823; detects the marks 823 from the infrared image, and calculates a second index 921 related to calibration of the infrared image based on the number of the detected marks 823; and outputs information indicating the first index 911 and the second index 921.” See also Paragraphs [0120-0125]. Thus, the information processing device does have sufficient structure associated with it wherein a processor carries out the functions.
If applicant does not intend to have this/these limitation(s) interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation(s) to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation(s) recite(s) sufficient structure to perform the claimed function so as to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed
invention is not identically disclosed as set forth in section 102 of this title, if the
differences between the claimed invention and the prior art are such that the claimed
invention as a whole would have been obvious before the effective filing date of the
claimed invention to a person having ordinary skill in the art to which the claimed
invention pertains. Patentability shall not be negated by the manner in which the
invention was made.
Claims 1, 2, and 4 are rejected under 35 U.S.C. 103 as being unpatentable over ISLAM (US 20230025684 A1), hereinafter referenced as ISLAM (A) in view of CLAVEAU (US 20170287166 A1), hereinafter referenced as CLAVEAU.
Regarding claim 1, ISLAM (A) teaches an information processing device (Fig. 1C, #110 called computing system, Paragraph [0036]) that performs calibration (Fig. 1C, Paragraph [0036] – ISLAM (A) discloses the computing system 110 may be configured to perform camera calibration for both the camera 170 and the camera 180.)
related to a position of an imaging camera (Fig. 1C, #170/180 called cameras, Paragraph [0030] - ISLAM (A) discloses the computing system 110 may be dedicated to performing the camera calibration, and may communicate the calibration information to another computing system that is dedicated to controlling the robot 150. For instance, the robot 150 may be positioned based on images generated by the camera 170 and on the camera calibration information.)
that captures a captured image used for detection of a parcel being transported (Fig. 1C, Paragraph [0036] – ISLAM (A) discloses the robot control system 110 of FIG. 1C may be configured to both receive images from camera 170 and receive images from camera 180. In some cases, the computing system 110 may be configured to use an image generated by the camera 170 and an image generated by the camera 180 to determine object structure information, which may describe a three-dimensional structure of an object [wherein an object is a parcel] captured by both images.)
and a position of an infrared camera (Fig. 1C, #170/180 called cameras, Paragraph [0030] – ISLAM (A) discloses the robot 150 may be positioned based on images generated by the camera 170 and on the camera calibration information. Paragraph [0035] – ISLAM (A) discloses cameras 170, 180 may be the same type of camera, or may be different types of cameras. Paragraph [0029] – ISLAM (A) discloses the camera 170 may be any type of image sensing device that is configured to generate or otherwise acquire an image (or, more generally, image data) that represents a scene in a camera field of view. The camera 170 may be, e.g., a color image camera, a grayscale image camera, a depth-sensing camera (e.g., a time-of-flight (TOF) or structured light camera), or any other camera (the term “or” in this disclosure may be used to refer to “and/or”) [wherein any other camera can be an infrared camera].)
that captures an infrared image used for detection of the parcel (Fig. 1C, Paragraph [0036] – ISLAM (A) discloses the computing system 110 may be configured to use an image generated by the camera 170 and an image generated by the camera 180 to determine object structure information, which may describe a three-dimensional structure of an object [wherein an object is a parcel] captured by both images. Paragraph [0029] – ISLAM (A) discloses camera 170 may be, e.g., a color image camera, a grayscale image camera, a depth-sensing camera (e.g., a time-of-flight (TOF) or structured light camera), or any other camera (the term “or” in this disclosure may be used to refer to “and/or”) [wherein any other camera can be an infrared camera].),
the information processing device (Fig. 1C, #110 called computing system, Paragraph [0036]) comprising:
a processor (Fig. 2, #111 called control circuit, Paragraph [0037] – ISLAM (A) discloses the computing system 110 can include a control circuit 111, a communication interface 113, and a non-transitory computer-readable medium 115 (e.g., memory). In an embodiment, the control circuit 111 may include one or more processors, a programmable logic circuit (PLC) or a programmable logic array (PLA), a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), or any other control circuit.);
and a memory (Fig. 2, #115 called non-transitory computer-readable medium, Paragraph [0037] – ISLAM (A) discloses computing system 110 can include a control circuit 111, a communication interface 113, and a non-transitory computer-readable medium 115 (e.g., memory).),
wherein the processor (Fig. 2, #111 called control circuit, Paragraph [0037])
acquires, from the imaging camera (Fig. 1C, #170/180 called cameras, Paragraph [0036] – ISLAM (A) discloses the computing system 110 may be configured to use an image generated by the camera 170 and an image generated by the camera 180 to determine object structure information, which may describe a three-dimensional structure of an object captured by both images.),
the captured image obtained by imaging a board having a plurality of marks (Fig. 1C, Paragraph [0036] – ISLAM (A) discloses the computing system 110 may control both cameras 170, 180 to capture respective images of the calibration pattern 160 in order to perform camera calibration. See also Fig. 4A-4C, #460 called calibration pattern, Paragraph [0061].),
acquires, from the infrared camera (Fig. 1C, #170/180 called cameras, Paragraph [0036] – ISLAM (A) discloses the computing system 110 may be configured to use an image generated by the camera 170 and an image generated by the camera 180 to determine object structure information, which may describe a three-dimensional structure of an object captured by both images. See also Paragraph [0029].),
the infrared image obtained by imaging the board (Fig. 1C, Paragraph [0036] – ISLAM (A) discloses the computing system 110 may control both cameras 170, 180 to capture respective images of the calibration pattern 160 in order to perform camera calibration. See also Fig. 4A-4C, #460 called calibration pattern, Paragraph [0061].),
detects the marks from the captured image (Fig. 5, Paragraph [0067] – ISLAM (A) discloses the computing system 110 determines a plurality of image coordinates (also referred to as image pattern element locations) that indicate or otherwise represent respective locations at which the plurality of pattern elements appear in the calibration image.),
Although ISLAM (A) further teaches detects the marks from the infrared image (Fig. 5, Paragraph [0067] – ISLAM (A) discloses the computing system 110 determines a plurality of image coordinates (also referred to as image pattern element locations) that indicate or otherwise represent respective locations at which the plurality of pattern elements appear in the calibration image. See also Paragraph [0029].),
ISLAM (A) fails to explicitly teach and calculates a first index related to calibration of the captured image based on the number of the detected marks, and calculates a second index related to calibration of the infrared image based on the number of the detected marks, and outputs information indicating the first index and the second index.
However, CLAVEAU explicitly teaches and calculates a first index related to calibration of the captured image (Fig. 13, Paragraph [0140] – CLAVEAU discloses the method 300 can involve, for each multi-camera bin, a step 310 of obtaining, based on the multi-camera reference images stored in the multi-camera bin, estimated values for the extrinsic parameters of each camera associated with the multi-camera bin. Paragraph [0141] – CLAVEAU further discloses method 300 can also include a step 312 of obtaining, based on the estimated values of the extrinsic parameters [wherein estimated values of the extrinsic parameters are a calculated first index] of all the cameras of the network, calibrated values of the extrinsic parameters for each camera of the network in a same global reference frame.)
based on the number of the detected marks (Fig. 13, Paragraph [0131] – CLAVEAU discloses an exemplary image quality metric for qualifying the target images can be a weighted sum or average of the following quality factors: the number of points or fiducial markers detected on the target image [wherein the number of fiducial markers are the number of the detected marks]; the level of blur in the image where the points or fiducial markers have been detected; the saturation of the image; and the contrast of the image. Paragraph [0132] – CLAVEAU further discloses each qualified target image can be assigned to a volume bin, and/or an angle bin and/or a multi-camera bin, and therefore be used to compute either or both of intrinsic and extrinsic camera parameters.),
and calculates a second index related to calibration of the infrared image (Fig. 13, Paragraph [0140] – CLAVEAU discloses the method 300 can involve, for each multi-camera bin, a step 310 of obtaining, based on the multi-camera reference images stored in the multi-camera bin, estimated values for the extrinsic parameters of each camera associated with the multi-camera bin. Paragraph [0141] – CLAVEAU further discloses method 300 can also include a step 312 of obtaining, based on the estimated values of the extrinsic parameters [wherein estimated values of the extrinsic parameters are a calculated second index] of all the cameras of the network, calibrated values of the extrinsic parameters for each camera of the network in a same global reference frame. Paragraph [0083] – CLAVEAU discloses cameras that can benefit from the present techniques can operate in various regions of the electromagnetic spectrum including, without limitation, the ultraviolet, visible, near-infrared (NIR), short-wavelength infrared (SWIR), mid-wavelength infrared (MWIR), long-wavelength infrared (LWIR), and terahertz (THz) wavelength ranges.)
based on the number of the detected marks (Fig. 13, Paragraph [0131] – CLAVEAU discloses an exemplary image quality metric for qualifying the target images can be a weighted sum or average of the following quality factors: the number of points or fiducial markers detected on the target image [wherein the number of fiducial markers are the number of the detected marks]; the level of blur in the image where the points or fiducial markers have been detected; the saturation of the image; and the contrast of the image. Paragraph [0132] – CLAVEAU further discloses each qualified target image can be assigned to a volume bin, and/or an angle bin and/or a multi-camera bin, and therefore be used to compute either or both of intrinsic and extrinsic camera parameters.),
and outputs information indicating the first index and the second index (Fig. 13, Paragraph [0144] – CLAVEAU discloses the extrinsic calibration calculations [wherein extrinsic calibration calculations include the first and second index] are performed iteratively and refined with each successive addition of a new set of reference images. The process can also allow the user to track which cameras have been extrinsically calibrated and which ones have not been or are currently being extrinsically calibrated. In some implementations, all this information about the calibration process can be displayed in real-time to the user via the display of a tablet computer affixed to the calibration target held by the user.).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date the claimed invention was made to combine the teachings of ISLAM (A) of having an information processing device that performs calibration related to a position of an imaging camera that captures a captured image used for detection of a parcel being transported and a position of an infrared camera that captures an infrared image used for detection of the parcel, the information processing device comprising: a processor; and a memory, wherein the processor acquires, from the imaging camera, the captured image obtained by imaging a board having a plurality of marks, acquires, from the infrared camera, the infrared image obtained by imaging the board, detects the marks from the captured image, detects the marks from the infrared image, with the teachings of CLAVEAU having and calculates a first index related to calibration of the captured image based on the number of the detected marks, and calculates a second index related to calibration of the infrared image based on the number of the detected marks, and outputs information indicating the first index and the second index.
Wherein ISLAM (A)’s information processing device wherein having and calculates a first index related to calibration of the captured image based on the number of the detected marks, and calculates a second index related to calibration of the infrared image based on the number of the detected marks, and outputs information indicating the first index and the second index.
The motivation behind this modification would have been to provide an enhanced method of performing camera calibration that improves accuracy and execution time of the image analysis process, since both ISLAM (A) and CLAVEAU relate to camera calibration systems and methods, wherein ISLAM (A) relates to improving an accuracy of stereo camera calibration, and more specifically to effectively measuring how much error there is in an estimate (e.g., estimated transformation function) used by the stereo camera calibration, so that the stereo camera calibration can be improved by reducing the error in such an estimate, and CLAVEAU discloses an interactive camera calibration method and an image acquisition and analysis scheme, thus improving the efficiency and execution time of the image acquisition and analysis process. Please see ISLAM (A) (US 20230025684 A1), Paragraph [0102], and CLAVEAU (US 20170287166 A1), Paragraph [0117].
Regarding claim 2, ISLAM (A) in view of CLAVEAU teach the information processing device according to claim 1,
ISLAM (A) fails to explicitly teach wherein the processor sets a predetermined first threshold to be compared with the first index and a predetermined second threshold to be compared with the second index, and outputs a first index image indicating whether the first index is equal to or greater than the first threshold and a second index image indicating whether the second index is equal to or greater than the second threshold.
However, CLAVEAU explicitly teaches wherein the processor (Fig. 17, #80 called processing unit, Paragraph [0158] – CLAVEAU discloses processing unit 80 can include or be coupled to a computer readable memory 82 storing computer executable instructions thereon that, when executed by the processing unit 80 can perform various steps of the calibration methods described herein (see, e.g., FIGS. 6 and 13). Such steps can include, without limitation, defining volume, angle and multi-camera bins as defined above; acquiring, identifying and qualifying target images for various poses of a calibration target 20; assigning the qualified target poses as reference images in the various volume, angle and multi-camera bins; performing camera calibration to estimate the intrinsic and extrinsic parameters of the various cameras 44a to 44d; and validating the calibration results.)
sets a predetermined first threshold to be compared with the first index (Fig. 3, Paragraph [0143] – CLAVEAU discloses one possible approach to assess the completion level of the extrinsic calibration is to continuously or repeatedly (e.g., periodically) compute the average reprojection error of target points in the reference images and then compare the computed error with a predetermined threshold below which extrinsic calibration [wherein extrinsic calibration is the first index] is considered complete or satisfactory.)
and a predetermined second threshold to be compared with the second index (Fig. 3, Paragraph [0143] – CLAVEAU discloses one possible approach to assess the completion level of the extrinsic calibration is to continuously or repeatedly (e.g., periodically) compute the average reprojection error of target points in the reference images and then compare the computed error with a predetermined threshold below which extrinsic calibration [wherein extrinsic calibration is the second index] is considered complete or satisfactory.),
and outputs a first index image indicating whether the first index is equal to or greater than the first threshold (Fig. 3, Paragraph [0143] – CLAVEAU discloses the method 300 of FIG. 13 can allow the user to monitor the progress of the extrinsic camera calibration. One possible approach to assess the completion level of the extrinsic calibration is to continuously or repeatedly (e.g., periodically) compute the average reprojection error of target points in the reference images and then compare the computed error with a predetermined threshold below which extrinsic calibration is considered complete or satisfactory. Paragraph [0144] – CLAVEAU further discloses the process can also allow the user to track which cameras have been extrinsically calibrated and which ones have not been or are currently being extrinsically calibrated. In some implementations, all this information about the calibration process can be displayed in real-time to the user via the display of a tablet computer affixed to the calibration target held by the user.)
and a second index image indicating whether the second index is equal to or greater than the second threshold (Fig. 3, Paragraph [0143] – CLAVEAU discloses the method 300 of FIG. 13 can allow the user to monitor the progress of the extrinsic camera calibration. One possible approach to assess the completion level of the extrinsic calibration is to continuously or repeatedly (e.g., periodically) compute the average reprojection error of target points in the reference images and then compare the computed error with a predetermined threshold below which extrinsic calibration is considered complete or satisfactory. Paragraph [0144] – CLAVEAU further discloses the process can also allow the user to track which cameras have been extrinsically calibrated and which ones have not been or are currently being extrinsically calibrated. In some implementations, all this information about the calibration process can be displayed in real-time to the user via the display of a tablet computer affixed to the calibration target held by the user.).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date the claimed invention was made to combine the teachings of ISLAM (A) in view of CLAVEAU of having an information processing device that performs calibration related to a position of an imaging camera that captures a captured image used for detection of a parcel being transported and a position of an infrared camera that captures an infrared image used for detection of the parcel, the information processing device comprising: a processor; and a memory, with the teachings of CLAVEAU having wherein the processor sets a predetermined first threshold to be compared with the first index and a predetermined second threshold to be compared with the second index, and outputs a first index image indicating whether the first index is equal to or greater than the first threshold and a second index image indicating whether the second index is equal to or greater than the second threshold.
Wherein ISLAM (A)’s information processing device wherein the processor sets a predetermined first threshold to be compared with the first index and a predetermined second threshold to be compared with the second index, and outputs a first index image indicating whether the first index is equal to or greater than the first threshold and a second index image indicating whether the second index is equal to or greater than the second threshold.
The motivation behind this modification would have been to provide an enhanced method of performing camera calibration that improves accuracy and execution time of the image analysis process, since both ISLAM (A) and CLAVEAU relate to camera calibration systems and methods, wherein ISLAM (A) relates to improving an accuracy of stereo camera calibration, and more specifically to effectively measuring how much error there is in an estimate (e.g., estimated transformation function) used by the stereo camera calibration, so that the stereo camera calibration can be improved by reducing the error in such an estimate, and CLAVEAU discloses an interactive camera calibration method and an image acquisition and analysis scheme, thus improving the efficiency and execution time of the image acquisition and analysis process. Please see ISLAM (A) (US 20230025684 A1), Paragraph [0102], and CLAVEAU (US 20170287166 A1), Paragraph [0117].
Regarding claim 4, ISLAM (A) in view of CLAVEAU teach the information processing device according to claim 2,
ISLAM (A) fails to explicitly teach wherein the processor sets the first threshold such that the first index falls below the first threshold when the number of the marks detected from the captured image is less than a predetermined number, and sets the second threshold such that the second index falls below the second threshold when the number of the marks detected from the infrared image is less than a predetermined number.
However, CLAVEAU explicitly teaches wherein the processor (Fig. 17, #80 called processing unit, Paragraph [0158]) sets the first threshold (Fig. 13, Paragraph [0131] – CLAVEAU discloses if each evaluated image quality parameter meets a respective specific or preset quality criterion or threshold [wherein a preset quality criterion or threshold is the first threshold], the identifying step 306 can include a step of classifying the target image as a qualified target image.)
such that the first index falls below the first threshold when the number of the marks detected from the captured image is less than a predetermined number (Fig. 13, Paragraph [0131] – CLAVEAU discloses an exemplary image quality metric for qualifying the target images can be a weighted sum or average of the following quality factors: the number of points or fiducial markers detected on the target image [wherein the number of points or fiducial markers is the number of marks detected from the captured image]; the level of blur in the image where the points or fiducial markers have been detected; the saturation of the image; and the contrast of the image.),
and sets the second threshold (Fig. 13, Paragraph [0131] – CLAVEAU discloses if each evaluated image quality parameter meets a respective specific or preset quality criterion or threshold [wherein a preset quality criterion or threshold is the second threshold], the identifying step 306 can include a step of classifying the target image as a qualified target image.)
such that the second index falls below the second threshold when the number of the marks detected from the infrared image is less than a predetermined number (Fig. 13, Paragraph [0131] – CLAVEAU discloses an exemplary image quality metric for qualifying the target images can be a weighted sum or average of the following quality factors: the number of points or fiducial markers detected on the target image [wherein the number of points or fiducial markers is the number of marks detected from the infrared image]; the level of blur in the image where the points or fiducial markers have been detected; the saturation of the image; and the contrast of the image. Paragraph [0083] – CLAVEAU discloses cameras that can benefit from the present techniques can operate in various regions of the electromagnetic spectrum including, without limitation, the ultraviolet, visible, near-infrared (NIR), short-wavelength infrared (SWIR), mid-wavelength infrared (MWIR), long-wavelength infrared (LWIR), and terahertz (THz) wavelength ranges.).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date the claimed invention was made to combine the teachings of ISLAM (A) in view of CLAVEAU of having an information processing device that performs calibration related to a position of an imaging camera that captures a captured image used for detection of a parcel being transported and a position of an infrared camera that captures an infrared image used for detection of the parcel, the information processing device comprising: a processor; and a memory, with the teachings of CLAVEAU having wherein the processor sets the first threshold such that the first index falls below the first threshold when the number of the marks detected from the captured image is less than a predetermined number, and sets the second threshold such that the second index falls below the second threshold when the number of the marks detected from the infrared image is less than a predetermined number.
Wherein ISLAM (A)’s information processing device wherein the processor sets the first threshold such that the first index falls below the first threshold when the number of the marks detected from the captured image is less than a predetermined number, and sets the second threshold such that the second index falls below the second threshold when the number of the marks detected from the infrared image is less than a predetermined number.
The motivation behind this modification would have been to provide an enhanced method of performing camera calibration that improves accuracy and execution time of the image analysis process, since both ISLAM (A) and CLAVEAU relate to camera calibration systems and methods, wherein ISLAM (A) relates to improving an accuracy of stereo camera calibration, and more specifically to effectively measuring how much error there is in an estimate (e.g., estimated transformation function) used by the stereo camera calibration, so that the stereo camera calibration can be improved by reducing the error in such an estimate, and CLAVEAU discloses an interactive camera calibration method and an image acquisition and analysis scheme, thus improving the efficiency and execution time of the image acquisition and analysis process. Please see ISLAM (A) (US 20230025684 A1), Paragraph [0102], and CLAVEAU (US 20170287166 A1), Paragraph [0117].
Claim 3 is rejected under 35 U.S.C. 103 as being unpatentable over ISLAM (US 20230025684 A1), hereinafter referenced as ISLAM (A) in view of CLAVEAU (US 20170287166 A1), hereinafter referenced as CLAVEAU, in further view of ISLAM (US 20200306977 A1), hereinafter referenced as ISLAM (B).
Regarding claim 3, ISLAM (A) in view of CLAVEAU teach the information processing device according to claim 2,
Although ISLAM (A) explicitly teaches wherein the processor (Fig. 2, #111 called control circuit, Paragraph [0039] – ISLAM (A) discloses the control circuit 111 may include one or more processors configured to execute the computer-executable instructions to perform the camera calibration.) outputs,
ISLAM (A) fails to explicitly teach an image different from an image output when the first index is less than the first threshold or when the second index is less than the second threshold.
However, CLAVEAU explicitly teaches an image different from an image output when the first index is less than the first threshold (Fig. 3, Paragraph [0143] – CLAVEAU discloses the method 300 of FIG. 13 can allow the user to monitor the progress of the extrinsic camera calibration. One possible approach to assess the completion level of the extrinsic calibration is to continuously or repeatedly (e.g., periodically) compute the average reprojection error of target points in the reference images and then compare the computed error with a predetermined threshold below [wherein below is less] which extrinsic calibration [wherein extrinsic calibration includes the first index] is considered complete or satisfactory.)
or when the second index is less than the second threshold (Fig. 3, Paragraph [0143] – CLAVEAU discloses the method 300 of FIG. 13 can allow the user to monitor the progress of the extrinsic camera calibration. One possible approach to assess the completion level of the extrinsic calibration is to continuously or repeatedly (e.g., periodically) compute the average reprojection error of target points in the reference images and then compare the computed error with a predetermined threshold below [wherein below is less] which extrinsic calibration [wherein extrinsic calibration includes the second index] is considered complete or satisfactory.).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date the claimed invention was made to combine the teachings of ISLAM (A) in view of CLAVEAU of having an information processing device that performs calibration related to a position of an imaging camera that captures a captured image used for detection of a parcel being transported and a position of an infrared camera that captures an infrared image used for detection of the parcel, the information processing device comprising: a processor; and a memory, with the teachings of CLAVEAU having an image different from an image output when the first index is less than the first threshold or when the second index is less than the second threshold.
Wherein ISLAM (A)’s information processing device wherein the processor outputs an image different from an image output when the first index is less than the first threshold or when the second index is less than the second threshold.
The motivation behind this modification would have been to provide an enhanced method of performing camera calibration that improves accuracy and execution time of the image analysis process, since both ISLAM (A) and CLAVEAU relate to camera calibration systems and methods, wherein ISLAM (A) relates to improving an accuracy of stereo camera calibration, and more specifically to effectively measuring how much error there is in an estimate (e.g., estimated transformation function) used by the stereo camera calibration, so that the stereo camera calibration can be improved by reducing the error in such an estimate, and CLAVEAU discloses an interactive camera calibration method and an image acquisition and analysis scheme, thus improving the efficiency and execution time of the image acquisition and analysis process. Please see ISLAM (A) (US 20230025684 A1), Paragraph [0102], and CLAVEAU (US 20170287166 A1), Paragraph [0117].
ISLAM (A) in view of CLAVEAU fail to explicitly teach when the first index is equal to or greater than the first threshold and the second index is equal to or greater than the second threshold,
However, ISLAM (B) explicitly teaches when the first index is equal to or greater than the first threshold (Fig. 12B, Paragraph [0130] – ISLAM(B) discloses step 1215 may involve outputting the notification to a user interface device, such as an electronic display in communication with the robot controller 110. The electronic display may display, for instance, the at least one deviation parameter value [wherein the deviation parameter value includes the first index], or an indication that the at least one deviation parameter value exceeds the defined deviation threshold.)
and the second index is equal to or greater than the second threshold (Fig. 12B, Paragraph [0130] – ISLAM(B) discloses step 1215 may involve outputting the notification to a user interface device, such as an electronic display in communication with the robot controller 110. The electronic display may display, for instance, the at least one deviation parameter value [wherein the deviation parameter value includes the second index], or an indication that the at least one deviation parameter value exceeds the defined deviation threshold.),
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date the claimed invention was made to combine the teachings of ISLAM (A) in view of CLAVEAU of having an information processing device that performs calibration related to a position of an imaging camera that captures a captured image used for detection of a parcel being transported and a position of an infrared camera that captures an infrared image used for detection of the parcel, the information processing device comprising: a processor; and a memory, wherein the processor acquires, from the imaging camera, the captured image obtained by imaging a board having a plurality of marks, acquires, from the infrared camera, the infrared image obtained by imaging the board, and calculates a first index related to calibration of the captured image based on the number of the detected marks, and calculates a second index related to calibration of the infrared image based on the number of the detected marks, and outputs information indicating the first index and the second index, with the teachings of ISLAM (B) having when the first index is equal to or greater than the first threshold and the second index is equal to or greater than the second threshold.
Wherein ISLAM (A)’s information processing device wherein having when the first index is equal to or greater than the first threshold and the second index is equal to or greater than the second threshold.
The motivation behind this modification would have been to provide an enhanced method of performing camera calibration that improves accuracy and reduces error, since both ISLAM (A) and ISLAM (B) relate to camera calibration systems and methods, wherein ISLAM (A) relates to improving an accuracy of stereo camera calibration, and more specifically to effectively measuring how much error there is in an estimate (e.g., estimated transformation function) used by the stereo camera calibration, so that the stereo camera calibration can be improved by reducing the error in such an estimate, and ISLAM (B) is directed to a method and control system for verifying and updating calibration information for robot control; an automatic verification of the calibration information and/or update of the calibration information are performed to ensure that the robot operates based on correct information about one or more properties associated with the camera or any other element of the robot operation system. Please see ISLAM (A) (US 20230025684 A1), Paragraph [0102], and ISLAM (B) (US 20200306977 A1), Paragraph [0028].
Claim 5 is rejected under 35 U.S.C. 103 as being unpatentable over ISLAM (US 20230025684 A1), hereinafter referenced as ISLAM (A) in view of CLAVEAU (US 20170287166 A1), hereinafter referenced as CLAVEAU, in further view of WANG (US 20220402444 A1), hereinafter referenced as WANG.
Regarding claim 5, ISLAM (A) in view of CLAVEAU teach the information processing device according to claim 1,
Although ISLAM (A) further teaches wherein the board (Fig. 4A, #460 called calibration pattern, Paragraph [0061]) has a flat surface (Fig. 4A, Paragraph [0061] – ISLAM (A) discloses the calibration pattern 460 may be printed on a flat calibration board.)
along a surface on which the parcel is transported (Fig. 4B, Paragraph [0101] – ISLAM (A) discloses the image may be generated by the camera (e.g., 470) during a robot operation, such as a de-palletization operation or a bin picking operation. In some scenarios, the image may capture or otherwise represent an object with which there is to be robot interaction. For example, the object may be a package to be de-palletized, or a component to be gripped [wherein the object is a parcel].)
ISLAM (A) fails to explicitly teach and wherein the processor calculates the first index based on the number of the marks detected from the flat surface in the captured image and the number of the marks detected from the inclined surface in the captured image, and calculates the second index based on the number of the marks detected from the flat surface in the infrared image and the number of the marks detected from the inclined surface in the infrared image.
However, CLAVEAU explicitly teaches and wherein the processor (Fig. 17, #80 called processing unit, Paragraph [0158])
calculates the first index (Fig. 13, Paragraph [0140] – CLAVEAU discloses the method 300 can involve, for each multi-camera bin, a step 310 of obtaining, based on the multi-camera reference images stored in the multi-camera bin, estimated values for the extrinsic parameters of each camera associated with the multi-camera bin. Paragraph [0141] – CLAVEAU further discloses method 300 can also include a step 312 of obtaining, based on the estimated values of the extrinsic parameters [wherein estimated values of the extrinsic parameters are a calculated first index] of all the cameras of the network, calibrated values of the extrinsic parameters for each camera of the network in a same global reference frame.)
based on the number of the marks detected from the flat surface in the captured image (Fig. 13, Paragraph [0131] – CLAVEAU discloses the identifying step 306 can include a step of searching the calibration target in each target image acquired by each camera, which can involve looking for one or more fiducial features present on the calibration target [wherein the calibration target is the flat surface]. As mentioned above regarding the intrinsic calibration method, an exemplary image quality metric for qualifying the target images can be a weighted sum or average of the following quality factors: the number of points or fiducial markers detected on the target image [wherein the number of points or fiducial markers is the number of marks detected from the captured image]; the level of blur in the image where the points or fiducial markers have been detected; the saturation of the image; and the contrast of the image.)
and the number of the marks detected from the inclined surface in the captured image (Fig. 12A, Paragraph [0093] – CLAVEAU discloses it can further be desirable or even necessary that the calibration target be captured in different orientations with respect to each camera and, more particularly, that target poses with an out-of-plane angle be part of the capture session. More specifically, it can be advantageous or required not to limit the reference images to fronto-parallel poses, that is, target poses where the exposed surface of the calibration target is perpendicular to the optical axis of the camera. Paragraph [0131] – CLAVEAU discloses an exemplary image quality metric for qualifying the target images can be a weighted sum or average of the following quality factors: the number of points or fiducial markers detected on the target image [wherein the number of points or fiducial markers is the number of marks detected from the captured image].),
and calculates the second index (Fig. 13, Paragraph [0140] – CLAVEAU discloses the method 300 can involve, for each multi-camera bin, a step 310 of obtaining, based on the multi-camera reference images stored in the multi-camera bin, estimated values for the extrinsic parameters of each camera associated with the multi-camera bin. Paragraph [0141] – CLAVEAU further discloses method 300 can also include a step 312 of obtaining, based on the estimated values of the extrinsic parameters [wherein estimated values of the extrinsic parameters are a calculated second index] of all the cameras of the network, calibrated values of the extrinsic parameters for each camera of the network in a same global reference frame.)
based on the number of the marks detected from the flat surface (Fig. 13, Paragraph [0131] – CLAVEAU discloses the identifying step 306 can include a step of searching the calibration target in each target image acquired by each camera, which can involve looking for one or more fiducial features present on the calibration target [wherein the calibration target is the flat surface]. As mentioned above regarding the intrinsic calibration method, an exemplary image quality metric for qualifying the target images can be a weighted sum or average of the following quality factors: the number of points or fiducial markers detected on the target image [wherein the number of points or fiducial markers is the number of marks detected from the captured image]; the level of blur in the image where the points or fiducial markers have been detected; the saturation of the image; and the contrast of the image.)
in the infrared image (Fig. 8, Paragraph [0083] – CLAVEAU discloses cameras that can benefit from the present techniques can operate in various regions of the electromagnetic spectrum including, without limitation, the ultraviolet, visible, near-infrared (NIR), short-wavelength infrared (SWIR), mid-wavelength infrared (MWIR), long-wavelength infrared (LWIR), and terahertz (THz) wavelength ranges.)
and the number of the marks detected from the inclined surface (Fig. 12A, Paragraph [0093] – CLAVEAU discloses it can further be desirable or even necessary that the calibration target be captured in different orientations with respect to each camera and, more particularly, that target poses with an out-of-plane angle be part of the capture session. More specifically, it can be advantageous or required not to limit the reference images to fronto-parallel poses, that is, target poses where the exposed surface of the calibration target is perpendicular to the optical axis of the camera. Paragraph [0131] – CLAVEAU discloses an exemplary image quality metric for qualifying the target images can be a weighted sum or average of the following quality factors: the number of points or fiducial markers detected on the target image [wherein the number of points or fiducial markers is the number of marks detected from the image].)
in the infrared image (Fig. 8, Paragraph [0083] – CLAVEAU discloses cameras that can benefit from the present techniques can operate in various regions of the electromagnetic spectrum including, without limitation, the ultraviolet, visible, near-infrared (NIR), short-wavelength infrared (SWIR), mid-wavelength infrared (MWIR), long-wavelength infrared (LWIR), and terahertz (THz) wavelength ranges.).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date the claimed invention was made to combine the teachings of ISLAM (A) in view of CLAVEAU of having an information processing device that performs calibration related to a position of an imaging camera that captures a captured image used for detection of a parcel being transported and a position of an infrared camera that captures an infrared image used for detection of the parcel, the information processing device comprising: a processor; and a memory, with the teachings of CLAVEAU having and wherein the processor calculates the first index based on the number of the marks detected from the flat surface in the captured image and the number of the marks detected from the inclined surface in the captured image, and calculates the second index based on the number of the marks detected from the flat surface in the infrared image and the number of the marks detected from the inclined surface in the infrared image.
Wherein ISLAM (A)’s information processing device having and wherein the processor calculates the first index based on the number of the marks detected from the flat surface in the captured image and the number of the marks detected from the inclined surface in the captured image, and calculates the second index based on the number of the marks detected from the flat surface in the infrared image and the number of the marks detected from the inclined surface in the infrared image.
The motivation behind this modification would have been to provide an enhanced method of performing camera calibration that improves accuracy and execution time of the image analysis process, since both ISLAM (A) and CLAVEAU relate to camera calibration systems and methods, wherein ISLAM (A) relates to improving an accuracy of stereo camera calibration, and more specifically to effectively measuring how much error there is in an estimate (e.g., estimated transformation function) used by the stereo camera calibration, so that the stereo camera calibration can be improved by reducing the error in such an estimate, and CLAVEAU discloses an interactive camera calibration method and an image acquisition and analysis scheme, thus improving the efficiency and execution time of the image acquisition and analysis process. Please see ISLAM (A) (US 20230025684 A1), Paragraph [0102], and CLAVEAU (US 20170287166 A1), Paragraph [0117].
ISLAM (A) in view of CLAVEAU fail to explicitly teach and an inclined surface extending from one side of the flat surface while inclining in a height direction,
However, WANG explicitly teaches and an inclined surface extending from one side of the flat surface while inclining in a height direction (Fig. 2G, Paragraph [0084] – WANG discloses the combined sensor calibration target 255 illustrated in FIG. 2G is cubic and this includes six surfaces. Paragraph [0086] – WANG further discloses the first surface 260A also joins with the third surface 260C [wherein the first surface 260A is an inclined surface extending in the height direction relative to third surface 260C.]),
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date the claimed invention was made to combine the teachings of ISLAM (A) in view of CLAVEAU of having an information processing device that performs calibration related to a position of an imaging camera that captures a captured image used for detection of a parcel being transported and a position of an infrared camera that captures an infrared image used for detection of the parcel, the information processing device comprising: a processor; and a memory, with the teachings of WANG having and an inclined surface extending from one side of the flat surface while inclining in a height direction.
Wherein ISLAM (A)’s information processing device wherein having and an inclined surface extending from one side of the flat surface while inclining in a height direction.
The motivation behind this modification would have been to provide an enhanced method of performing camera calibration that improves accuracy and runtime-efficiency, since both ISLAM (A) and WANG relate to calibration techniques, wherein ISLAM (A) relates to improving an accuracy of stereo camera calibration, and more specifically to effectively measuring how much error there is in an estimate (e.g., estimated transformation function) used by the stereo camera calibration, so that the stereo camera calibration can be improved by reducing the error in such an estimate, and WANG pertains to use of combined sensor calibration targets that are polyhedral and that are used to calibrate multiple types of vehicle sensors and improve runtime-efficiency, space-efficiency, comprehensiveness of calibration, and consistency of vehicle sensor calibration over prior calibration techniques. Please see ISLAM (A) (US 20230025684 A1), Paragraph [0102], and WANG (US 20220402444 A1), Paragraph [0033].
Claims 6 and 7 are rejected under 35 U.S.C. 103 as being unpatentable over ISLAM (US 20230025684 A1), hereinafter referenced as ISLAM (A) in view of CLAVEAU (US 20170287166 A1), hereinafter referenced as CLAVEAU, in further view of GRUNDHÖFER (US 20180376116 A1), hereinafter referenced as GRUNDHÖFER.
Regarding claim 6, ISLAM (A) in view of CLAVEAU teach the information processing device according to claim 1,
ISLAM (A) in view of CLAVEAU fail to explicitly teach wherein the parcel is a parcel onto which a projection image is projected from a projection device, and wherein the processor projects a range in which the projection device is able to project a projection image as a range in which the board is to be installed.
However, GRUNDHÖFER explicitly teaches wherein the parcel (Fig. 1, #161 called projection surface, Paragraph [0046]) is a parcel onto which a projection image is projected from a projection device (Fig. 1, Paragraph [0036] – GRUNDHÖFER discloses projection surface 161 may be substantially any type of opaque surface or object. Paragraph [0046] – GRUNDHÖFER further discloses the laser projector 110 [wherein laser projector 110 is a projection device] is used to project light patterns, images, or the like onto the projection surface 161 of the scene 160.),
and wherein the processor (Fig. 3, #203 called processing elements, Paragraph [0047] – GRUNDHÖFER discloses laser projector 110 may include a mirror assembly 201, a laser source 202, one or more processing elements 203.)
projects a range in which the projection device is able to project a projection image as a range in which the board is to be installed (Fig. 1, Paragraph [0054] – GRUNDHÖFER discloses the laser projector 110 projects a calibration pattern 150 onto the scene 160. With reference to FIGS. 4A-4I, the calibration pattern 150 may include one or more blobs or regions with substantially constant properties, or properties that are varied, within a prescribed range of values. Paragraph [0064] – GRUNDHÖFER further discloses a plurality of calibration patterns may be projected in a predetermined sequence and one or more calibration images 208 for each calibration pattern may be captured. See also Paragraph [0061].).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date the claimed invention was made to combine the teachings of ISLAM (A) in view of CLAVEAU of having an information processing device that performs calibration related to a position of an imaging camera that captures a captured image used for detection of a parcel being transported and a position of an infrared camera that captures an infrared image used for detection of the parcel, the information processing device comprising: a processor; and a memory, with the teachings of GRUNDHÖFER of having wherein the parcel is a parcel onto which a projection image is projected from a projection device, and wherein the processor projects a range in which the projection device is able to project a projection image as a range in which the board is to be installed.
Wherein ISLAM (A)’s information processing device wherein the parcel is a parcel onto which a projection image is projected from a projection device, and wherein the processor projects a range in which the projection device is able to project a projection image as a range in which the board is to be installed.
The motivation behind this modification would have been to provide an enhanced method of performing calibration that improves accuracy and sensitivity, since both ISLAM (A) and GRUNDHÖFER relate to calibration systems and methods, wherein ISLAM (A) relates to improving an accuracy of stereo camera calibration, and more specifically to effectively measuring how much error there is in an estimate (e.g., estimated transformation function) used by the stereo camera calibration, so that the stereo camera calibration can be improved by reducing the error in such an estimate, and GRUNDHÖFER relates generally to methods and systems for calibrating one or more projectors; because the laser projector can be accurately calibrated allowing the input image to be projected at desired locations, the laser projector can be used for presentations that require highly accurate projection. Please see ISLAM (A) (US 20230025684 A1), Paragraph [0102], and GRUNDHÖFER (US 20180376116 A1), Paragraph [0032].
Regarding claim 7, ISLAM (A) in view of CLAVEAU teach the information processing device according to claim 1,
ISLAM (A) in view of CLAVEAU fail to explicitly teach wherein the parcel is a parcel onto which a projection image is projected from a projection device, wherein the projection device projects a predetermined image onto the board, and wherein the processor calculates a correction parameter used for calibration between the projection device and the imaging camera based on the captured image of the board onto which the predetermined image is projected, and calculates a correction parameter used for calibration between the projection device and the infrared camera based on the infrared image of the board onto which the predetermined image is projected.
However, GRUNDHÖFER explicitly teaches wherein the parcel is a parcel onto which a projection image is projected from a projection device (Fig. 1, Paragraph [0036] – GRUNDHÖFER discloses projection surface 161 may be substantially any type of opaque surface or object. Paragraph [0046] – GRUNDHÖFER further discloses the laser projector 110 [wherein laser projector 110 is a projection device] is used to project light patterns, images, or the like onto the projection surface 161 of the scene 160.),
wherein the projection device (Fig. 1, #110 called laser projector, Paragraph [0046]) projects a predetermined image onto the board (Fig. 1, Paragraph [0054] – GRUNDHÖFER discloses the laser projector 110 projects a calibration pattern 150 onto the scene 160. With reference to FIGS. 4A-4I, the calibration pattern 150 is a predefined, structured light arrangement having several pattern elements 151. For example, the calibration pattern 150 may include one or more blobs or regions with substantially constant properties, or properties that are varied, within a prescribed range of values.),
and wherein the processor (Fig. 3, #301 called processing elements, Paragraph [0039]) calculates a correction parameter used for calibration (Fig. 7, Paragraph [0081] – GRUNDHÖFER discloses the processor element 301 calculates the warping required to map each detected pattern element 151 in the estimated projection matrix onto the intended projection location of the virtual image plane of the laser projector 110.)
between the projection device (Fig. 1, #110 called laser projector, Paragraph [0046]) and the imaging camera (Fig. 1, #120a-b called cameras, Paragraph [0072)
based on the captured image of the board onto which the predetermined image is projected (Fig. 7, Paragraph [0081] – GRUNDHÖFER discloses the processor element 301 calculates the warping required to map each detected pattern element 151 in the estimated projection matrix onto the intended projection location of the virtual image plane of the laser projector 110. That is, the amount of distortion needed to be applied to an input image so that pattern element 151 impinges the projection surface 161 at the desired location. When the amount of warping for each pattern element 151 is determined, the method 700 proceeds to create a look up table in operation 703. The look up table may include distortion amounts for those pattern elements 151 detected by the cameras 120a, 120b.),
and calculates a correction parameter used for calibration (Fig. 7, Paragraph [0081] – GRUNDHÖFER discloses the processor element 301 calculates the warping required to map each detected pattern element 151 in the estimated projection matrix onto the intended projection location of the virtual image plane of the laser projector 110.)
between the projection device (Fig. 1, #110 called laser projector, Paragraph [0046]) and the infrared camera (Fig. 1, #120a-b called cameras, Paragraph [0072] – GRUNDHÖFER discloses one or more cameras capable of depth perception may be used, such as a 3D stereoscopic camera, KINECT-type depth camera, 3D cameras, or any other type of active or passive depth-detection camera, such as time of flight based 3D cameras or infrared 3D cameras.)
based on the infrared image of the board onto which the predetermined image is projected (Fig. 7, Paragraph [0081] – GRUNDHÖFER discloses the processor element 301 calculates the warping required to map each detected pattern element 151 in the estimated projection matrix onto the intended projection location of the virtual image plane of the laser projector 110. That is, the amount of distortion needed to be applied to an input image so that pattern element 151 impinges the projection surface 161 at the desired location. When the amount of warping for each pattern element 151 is determined, the method 700 proceeds to create a look up table in operation 703. The look up table may include distortion amounts for those pattern elements 151 detected by the cameras 120a, 120b.).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date the claimed invention was made to combine the teachings of ISLAM (A) in view of CLAVEAU of having an information processing device that performs calibration related to a position of an imaging camera that captures a captured image used for detection of a parcel being transported and a position of an infrared camera that captures an infrared image used for detection of the parcel, the information processing device comprising: a processor; and a memory, with the teachings of GRUNDHÖFER of having wherein the parcel is a parcel onto which a projection image is projected from a projection device, wherein the projection device projects a predetermined image onto the board, and wherein the processor calculates a correction parameter used for calibration between the projection device and the imaging camera based on the captured image of the board onto which the predetermined image is projected, and calculates a correction parameter used for calibration between the projection device and the infrared camera based on the infrared image of the board onto which the predetermined image is projected.
Wherein ISLAM (A)’s information processing device wherein the parcel is a parcel onto which a projection image is projected from a projection device, wherein the projection device projects a predetermined image onto the board, and wherein the processor calculates a correction parameter used for calibration between the projection device and the imaging camera based on the captured image of the board onto which the predetermined image is projected, and calculates a correction parameter used for calibration between the projection device and the infrared camera based on the infrared image of the board onto which the predetermined image is projected.
The motivation behind this modification would have been to provide an enhanced method of performing calibration that improves accuracy and sensitivity, since both ISLAM (A) and GRUNDHÖFER relate to calibration systems and methods, wherein ISLAM (A) relates to improving an accuracy of stereo camera calibration, and more specifically to effectively measuring how much error there is in an estimate (e.g., estimated transformation function) used by the stereo camera calibration, so that the stereo camera calibration can be improved by reducing the error in such an estimate, and GRUNDHÖFER relates generally to methods and systems for calibrating one or more projectors; because the laser projector can be accurately calibrated allowing the input image to be projected at desired locations, the laser projector can be used for presentations that require highly accurate projection. Please see ISLAM (A) (US 20230025684 A1), Paragraph [0102], and GRUNDHÖFER (US 20180376116 A1), Paragraph [0032].
Claims 8 and 9 are rejected under 35 U.S.C. 103 as being unpatentable over DAL MUTTO (US 20190364206 A1), hereinafter referenced as DAL MUTTO in view of CLAVEAU (US 20170287166 A1), hereinafter referenced as CLAVEAU.
Regarding claim 8, DAL MUTTO teaches a parcel detection system (Fig. 1D, #100 called depth camera system, Paragraph [0059] – DAL MUTTO discloses FIG. 1D is a block diagram of a stereo depth camera system according to one embodiment of the present invention. See also Fig. 1C, Paragraph [0053].) comprising:
a projection device (Fig. 1D, #106 called projection source, Paragraph [0059] – DAL MUTTO discloses depth camera system 100 shown in FIG. 1D includes a first camera 102, a second camera 104, a projection source 106 (or illumination source or active projection system).)
that projects a projection image onto a parcel being transported (Fig. 1D, Paragraph [0073] – DAL MUTTO discloses projection source 106 according to embodiments of the present invention may be configured to emit visible light (e.g., light within the spectrum visible to humans and/or other animals) or invisible light (e.g., infrared light) toward the scene imaged by the cameras 102 and 104. See also Paragraph [0075].);
an imaging camera (Fig. 1C, #100i, 100j, 100k cameras, Paragraph [0053] – DAL MUTTO discloses three cameras CAM1, CAM2, and CAM3 (respectively labeled 100i, 100j, and 100k) are configured to capture overlapping images different portions of objects 10 on conveyor system 12. See also Paragraph [0051, 0060].)
that captures a captured image used for detecting the parcel (Fig. 1C, Paragraph [0053] – DAL MUTTO discloses capture of images may be triggered by a triggering system, which may include a start trigger 28, which detects when an object 10 [wherein object 10 is the parcel] has entered the fields of view of the cameras 100i, 100j, and 100k.);
an infrared camera (Fig. 1C, #100i, 100j, 100k cameras, Paragraph [0051] – DAL MUTTO discloses each of the individual cameras 100 is a standard (e.g., commercial off-the-shelf) digital camera that includes a lens and an image sensor. In various embodiments, the image sensor may be a color image sensor (e.g., a visible light or red-green-blue or RGB sensor in, for example, a Bayer filter layout, where 25% of the pixels detect red light, 50% of the pixels detect green light, and 25% of the pixels detect blue light), an infrared (IR) image sensor, or a combination color and infrared (RGB-IR) sensor (e.g., a layout where 25% of the pixels detect red light, 25% of the pixels detect green light, 25% of the pixels detect blue light, and 25% of the pixels detect infrared light).)
that captures an infrared image used for detecting the parcel (Fig. 1C, Paragraph [0053] – DAL MUTTO discloses capture of images may be triggered by a triggering system, which may include a start trigger 28, which detects when an object 10 [wherein object 10 is the parcel] has entered the fields of view of the cameras 100i, 100j, and 100k.);
and an information processing device (Fig. 1D, #108 called host processor, Paragraph [0059] – DAL MUTTO discloses depth camera system 100 shown in FIG. 1D includes a first camera 102, a second camera 104, a projection source 106 (or illumination source or active projection system), and a host processor 108 and memory 110. See also Fig. 1C, #24 called controller, Paragraph [0060].)
that controls the projection device, the imaging camera, and the infrared camera (Fig. 1D, Paragraph [0060] – DAL MUTTO discloses cameras 102 and 104, projection source 106 [wherein the projection source 106 is the projection device], and a communication component (e.g., a USB connection or a network adapter 116), and processing the two-dimensional images captured by the cameras 102 and 104 of the three depth cameras 100 [wherein the depth cameras 100 can be imaging or infrared cameras] may be performed by a shared processor or shared collection of processors [wherein the shared processor or collection of processors is an information processing device that controls the projection device, imaging camera, and infrared camera] in communication with the depth cameras 100 using their respective communication components or network adapters 116.),
wherein in calibration related to a position of the imaging camera and a position of the infrared camera (Fig. 1D, Paragraph [0102] – DAL MUTTO discloses each camera 100 includes two or more standard 2-D cameras (e.g., cameras 102, 104, and 105) that are rigidly mounted and calibrated with respect to one another (e.g., in the case of stereoscopic cameras). Accordingly, the images captured by the separate standard 2-D cameras of the stereoscopic camera can be used to provide additional data regarding the position and orientation of each camera 100.),
the information processing device (Fig. 1C, #24 called controller, Paragraph [0060]) acquires,
from the imaging camera (Fig. 1C, #100i, 100j, 100k cameras, Paragraph [0053]), the captured image obtained by imaging a board having a plurality of marks (Fig. 4, Paragraph [0091] – DAL MUTTO discloses in operation 410, the controller 24 controls the first and second cameras (e.g., CAM A 100A and CAM B 100B) of a camera group to capture first and second images (e.g., capturing the images substantially simultaneously), respectively of a first scene where a calibration target [wherein a calibration target is a board having a plurality of marks] is located in the fields of view (e.g., 101A and 101B) of both cameras (e.g., CAM A 100A and CAM B 100B). See also Fig. 3B, Paragraph [0089].),
acquires, from the infrared camera (Fig. 1C, #100i, 100j, 100k cameras, Paragraph [0051]), the infrared image obtained by imaging the board (Fig. 4, Paragraph [0091] – DAL MUTTO discloses in operation 410, the controller 24 controls the first and second cameras (e.g., CAM A 100A and CAM B 100B) of a camera group to capture first and second images (e.g., capturing the images substantially simultaneously), respectively of a first scene where a calibration target [wherein a calibration target is a board] is located in the fields of view (e.g., 101A and 101B) of both cameras (e.g., CAM A 100A and CAM B 100B). See also Fig. 3B, Paragraph [0089].),
detects the marks from the captured image (Fig. 2, Paragraph [0081] – DAL MUTTO discloses each of the fiducial markers on the target is different, thereby communicating information about the orientation of the calibration target with respect to the camera.),
Although DAL MUTTO further teaches detects the marks from the infrared image (Fig. 2, Paragraph [0081] – DAL MUTTO discloses each of the fiducial markers on the target is different, thereby communicating information about the orientation of the calibration target with respect to the camera. Paragraph [0063] – DAL MUTTO further discloses the system includes one or more visible light cameras (e.g., RGB cameras) and, separately, one or more invisible light cameras (e.g., infrared cameras, where an IR band-pass filter is located across all over the pixels).),
DAL MUTTO fails to explicitly teach and calculates a first index related to calibration of the captured image based on the number of the detected marks, and calculates a second index related to calibration of the infrared image based on the number of the detected marks, and outputs information indicating the first index and the second index.
However, CLAVEAU explicitly teaches and calculates a first index related to calibration of the captured image (Fig. 13, Paragraph [0140] – CLAVEAU discloses the method 300 can involve, for each multi-camera bin, a step 310 of obtaining, based on the multi-camera reference images stored in the multi-camera bin, estimated values for the extrinsic parameters of each camera associated with the multi-camera bin. Paragraph [0141] – CLAVEAU further discloses method 300 can also include a step 312 of obtaining, based on the estimated values of the extrinsic parameters [wherein estimated values of the extrinsic parameters are a calculated first index] of all the cameras of the network, calibrated values of the extrinsic parameters for each camera of the network in a same global reference frame.)
based on the number of the detected marks (Fig. 13, Paragraph [0131] – CLAVEAU discloses an exemplary image quality metric for qualifying the target images can be a weighted sum or average of the following quality factors: the number of points or fiducial markers detected on the target image [wherein the number of fiducial markers are the number of the detected marks]; the level of blur in the image where the points or fiducial markers have been detected; the saturation of the image; and the contrast of the image. Paragraph [0132] – CLAVEAU further discloses each qualified target image can be assigned to a volume bin, and/or an angle bin and/or a multi-camera bin, and therefore be used to compute either or both of intrinsic and extrinsic camera parameters.),
and calculates a second index related to calibration of the infrared image (Fig. 13, Paragraph [0140] – CLAVEAU discloses the method 300 can involve, for each multi-camera bin, a step 310 of obtaining, based on the multi-camera reference images stored in the multi-camera bin, estimated values for the extrinsic parameters of each camera associated with the multi-camera bin. Paragraph [0141] – CLAVEAU further discloses method 300 can also include a step 312 of obtaining, based on the estimated values of the extrinsic parameters [wherein estimated values of the extrinsic parameters are a calculated second index] of all the cameras of the network, calibrated values of the extrinsic parameters for each camera of the network in a same global reference frame. Paragraph [0083] – CLAVEAU discloses cameras that can benefit from the present techniques can operate in various regions of the electromagnetic spectrum including, without limitation, the ultraviolet, visible, near-infrared (NIR), short-wavelength infrared (SWIR), mid-wavelength infrared (MWIR), long-wavelength infrared (LWIR), and terahertz (THz) wavelength ranges.)
based on the number of the detected marks (Fig. 13, Paragraph [0131] – CLAVEAU discloses an exemplary image quality metric for qualifying the target images can be a weighted sum or average of the following quality factors: the number of points or fiducial markers detected on the target image [wherein the number of fiducial markers are the number of the detected marks]; the level of blur in the image where the points or fiducial markers have been detected; the saturation of the image; and the contrast of the image. Paragraph [0132] – CLAVEAU further discloses each qualified target image can be assigned to a volume bin, and/or an angle bin and/or a multi-camera bin, and therefore be used to compute either or both of intrinsic and extrinsic camera parameters.),
and outputs information indicating the first index and the second index (Fig. 13, Paragraph [0144] – CLAVEAU discloses the extrinsic calibration calculations [wherein extrinsic calibration calculations include the first and second index] are performed iteratively and refined with each successive addition of a new set of reference images. The process can also allow the user to track which cameras have been extrinsically calibrated and which ones have not been or are currently being extrinsically calibrated. In some implementations, all this information about the calibration process can be displayed in real-time to the user via the display of a tablet computer affixed to the calibration target held by the user.).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date the claimed invention was made to combine the teachings of DAL MUTTO of having a parcel detection system comprising: a projection device that projects a projection image onto a parcel being transported; an imaging camera that captures a captured image used for detecting the parcel; an infrared camera that captures an infrared image used for detecting the parcel; and an information processing device that controls the projection device, the imaging camera, and the infrared camera, wherein in calibration related to a position of the imaging camera and a position of the infrared camera, the information processing device acquires, from the imaging camera, the captured image obtained by imaging a board having a plurality of marks, acquires, from the infrared camera, the infrared image obtained by imaging the board, detects the marks from the captured image, detects the marks from the infrared image, and outputs information indicating the first index and the second index, with the teachings of CLAVEAU having and calculates a first index related to calibration of the captured image based on the number of the detected marks, and calculates a second index related to calibration of the infrared image based on the number of the detected marks, and outputs information indicating the first index and the second index.
Wherein DAL MUTTO’s parcel detection system wherein having and calculates a first index related to calibration of the captured image based on the number of the detected marks, and calculates a second index related to calibration of the infrared image based on the number of the detected marks, and outputs information indicating the first index and the second index.
The motivation behind this modification would have been to provide a detection system with enhanced calibration detection and improved execution time for image acquisition and analysis, since both DAL MUTTO and CLAVEAU relate to camera calibration systems and methods, wherein DAL MUTTO discloses systems and methods for placing multiple cameras and calibrating these cameras with respect to one another, including estimating the 3-D poses of the cameras; use of a sufficiently bright backlit calibration target can also improve the ability of the system to generate meaningful poses when the calibration target is moved in the scene, and CLAVEAU discloses an interactive camera calibration method and an image acquisition and analysis scheme, thus improving the efficiency and execution time of the image acquisition and analysis process. Please see DAL MUTTO (US 20190364206 A1), Paragraph [0118], and CLAVEAU (US 20170287166 A1), Paragraph [0117].
Regarding claim 9, DAL MUTTO teaches a calibration method (Fig. 8, Paragraph [0123] – DAL MUTTO discloses FIG. 8 is a flowchart of a method 800 for calibrating two camera groups with respect to one another.)
in which an information processing device (Fig. 1C, Paragraph [0055] – DAL MUTTO discloses the controller 24 and the coordinating server 30 may include one or more processors (e.g., central processing units, graphics processing units, field programmable gate arrays, and application specific integrated circuits) coupled with memory (e.g., dynamic memory and/or persistent memory) storing instructions that configure the computing devices to perform particular specific functions as described herein.)
performs calibration related to a position of an imaging camera (Fig. 1C, #100i, 100j, 100k cameras, Paragraph [0053]) that captures a captured image used for detection of a parcel being transported (Fig. 8, Paragraph [0123] – DAL MUTTO discloses in operation 810, the first camera group 130ABC is calibrated by controlling a camera group (e.g., a controller 24) to calibrate the group based on images captured of a calibration target 200 carried by a portion of the conveyor system 12 that is visible to the first camera group (e.g., camera group 130ABC as shown in FIG. 7).)
and a position of an infrared camera (Fig. 1C, #100i, 100j, 100k cameras, Paragraph [0053]) that captures an infrared image (Fig. 1C, Paragraph [0063] – DAL MUTTO discloses the system includes one or more visible light cameras (e.g., RGB cameras) and, separately, one or more invisible light cameras (e.g., infrared cameras, where an IR band-pass filter is located across all over the pixels).)
used for detection of the parcel (Fig. 8, Paragraph [0123] – DAL MUTTO discloses in operation 810, the first camera group 130ABC is calibrated by controlling a camera group (e.g., a controller 24) to calibrate the group based on images captured of a calibration target 200 carried by a portion of the conveyor system 12 that is visible to the first camera group (e.g., camera group 130ABC as shown in FIG. 7). Paragraph [0053] – DAL MUTTO discloses capture of images may be triggered by a triggering system, which may include a start trigger 28, which detects when an object 10 [wherein object 10 is the parcel] has entered the fields of view of the cameras 100i, 100j, and 100k.),
the calibration method comprising: acquiring, from the imaging camera (Fig. 1C, #100i, 100j, 100k cameras, Paragraph [0053]), the captured image obtained by imaging the board (Fig. 4, Paragraph [0091] – DAL MUTTO discloses in operation 410, the controller 24 controls the first and second cameras (e.g., CAM A 100A and CAM B 100B) of a camera group to capture first and second images (e.g., capturing the images substantially simultaneously), respectively of a first scene where a calibration target [wherein a calibration target is the board] is located in the fields of view (e.g., 101A and 101B) of both cameras (e.g., CAM A 100A and CAM B 100B). See also Fig. 3B, Paragraph [0089].);
acquiring, from the infrared camera (Fig. 1C, #100i, 100j, 100k cameras, Paragraph [0053]), the infrared image obtained by imaging the board (Fig. 4, Paragraph [0091] – DAL MUTTO discloses in operation 410, the controller 24 controls the first and second cameras (e.g., CAM A 100A and CAM B 100B) of a camera group to capture first and second images (e.g., capturing the images substantially simultaneously), respectively of a first scene where a calibration target [wherein a calibration target is the board] is located in the fields of view (e.g., 101A and 101B) of both cameras (e.g., CAM A 100A and CAM B 100B). Paragraph [0063] – DAL MUTTO further discloses the system includes one or more visible light cameras (e.g., RGB cameras) and, separately, one or more invisible light cameras (e.g., infrared cameras, where an IR band-pass filter is located across all over the pixels). See also Fig. 3B, Paragraph [0089].);
detecting the marks from the captured image (Fig. 2, Paragraph [0081] – DAL MUTTO discloses each of the fiducial markers on the target is different, thereby communicating information about the orientation of the calibration target with respect to the camera.),
Although DAL MUTTO further teaches detecting the marks from the infrared image (Fig. 2, Paragraph [0081] – DAL MUTTO discloses each of the fiducial markers on the target is different, thereby communicating information about the orientation of the calibration target with respect to the camera. Paragraph [0063] – DAL MUTTO further discloses the system includes one or more visible light cameras (e.g., RGB cameras) and, separately, one or more invisible light cameras (e.g., infrared cameras, where an IR band-pass filter is located across all over the pixels).),
DAL MUTTO fails to explicitly teach placing a board having a plurality of marks by an installation worker; and calculating a first index related to calibration of the captured image based on the number of the detected marks; and calculating a second index related to calibration of the infrared image based on the number of the detected marks; and displaying information indicating the first index and the second index.
However, CLAVEAU explicitly teaches placing a board having a plurality of marks by an installation worker (Fig. 1B, Paragraph [0101] – CLAVEAU discloses the calibration target is portable and is intended to be held and moved within the scene by an operator such that the camera gradually captures images of the calibration target from varying distances and/or orientations relative to the camera.);
and calculating a first index related to calibration of the captured image (Fig. 13, Paragraph [0140] – CLAVEAU discloses the method 300 can involve, for each multi-camera bin, a step 310 of obtaining, based on the multi-camera reference images stored in the multi-camera bin, estimated values for the extrinsic parameters of each camera associated with the multi-camera bin. Paragraph [0141] – CLAVEAU further discloses method 300 can also include a step 312 of obtaining, based on the estimated values of the extrinsic parameters [wherein estimated values of the extrinsic parameters are a calculated first index] of all the cameras of the network, calibrated values of the extrinsic parameters for each camera of the network in a same global reference frame.)
based on the number of the detected marks (Fig. 13, Paragraph [0131] – CLAVEAU discloses an exemplary image quality metric for qualifying the target images can be a weighted sum or average of the following quality factors: the number of points or fiducial markers detected on the target image [wherein the number of fiducial markers are the number of the detected marks]; the level of blur in the image where the points or fiducial markers have been detected; the saturation of the image; and the contrast of the image. Paragraph [0132] – CLAVEAU further discloses each qualified target image can be assigned to a volume bin, and/or an angle bin and/or a multi-camera bin, and therefore be used to compute either or both of intrinsic and extrinsic camera parameters.);
and calculating a second index related to calibration of the infrared image (Fig. 13, Paragraph [0140] – CLAVEAU discloses the method 300 can involve, for each multi-camera bin, a step 310 of obtaining, based on the multi-camera reference images stored in the multi-camera bin, estimated values for the extrinsic parameters of each camera associated with the multi-camera bin. Paragraph [0141] – CLAVEAU further discloses method 300 can also include a step 312 of obtaining, based on the estimated values of the extrinsic parameters [wherein estimated values of the extrinsic parameters are a calculated second index] of all the cameras of the network, calibrated values of the extrinsic parameters for each camera of the network in a same global reference frame. Paragraph [0083] – CLAVEAU discloses cameras that can benefit from the present techniques can operate in various regions of the electromagnetic spectrum including, without limitation, the ultraviolet, visible, near-infrared (NIR), short-wavelength infrared (SWIR), mid-wavelength infrared (MWIR), long-wavelength infrared (LWIR), and terahertz (THz) wavelength ranges.)
based on the number of the detected marks (Fig. 13, Paragraph [0131] – CLAVEAU discloses an exemplary image quality metric for qualifying the target images can be a weighted sum or average of the following quality factors: the number of points or fiducial markers detected on the target image [wherein the number of fiducial markers are the number of the detected marks]; the level of blur in the image where the points or fiducial markers have been detected; the saturation of the image; and the contrast of the image. Paragraph [0132] – CLAVEAU further discloses each qualified target image can be assigned to a volume bin, and/or an angle bin and/or a multi-camera bin, and therefore be used to compute either or both of intrinsic and extrinsic camera parameters.);
and displaying information indicating the first index and the second index (Fig. 13, Paragraph [0144] – CLAVEAU discloses the extrinsic calibration calculations [wherein extrinsic calibration calculations include the first and second index] are performed iteratively and refined with each successive addition of a new set of reference images. The process can also allow the user to track which cameras have been extrinsically calibrated and which ones have not been or are currently being extrinsically calibrated. In some implementations, all this information about the calibration process can be displayed in real-time to the user via the display of a tablet computer affixed to the calibration target held by the user.).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date the claimed invention was made to combine the teachings of DAL MUTTO of having a calibration method in which an information processing device performs calibration related to a position of an imaging camera that captures a captured image used for detection of a parcel being transported and a position of an infrared camera that captures an infrared image used for detection of the parcel, the calibration method comprising: acquiring, from the imaging camera, the captured image obtained by imaging the board; acquiring, from the infrared camera, the infrared image obtained by imaging the board; detecting the marks from the captured image, detecting the marks from the infrared image, with the teachings of CLAVEAU having placing a board having a plurality of marks by an installation worker; and calculating a first index related to calibration of the captured image based on the number of the detected marks; and calculating a second index related to calibration of the infrared image based on the number of the detected marks; and displaying information indicating the first index and the second index.
Wherein DAL MUTTO’s calibration method wherein having placing a board having a plurality of marks by an installation worker; and calculating a first index related to calibration of the captured image based on the number of the detected marks; and calculating a second index related to calibration of the infrared image based on the number of the detected marks; and displaying information indicating the first index and the second index.
The motivation behind this modification would have been to provide a calibration method with enhanced calibration detection and improved execution time for image acquisition and analysis, since both DAL MUTTO and CLAVEAU relate to camera calibration systems and methods, wherein DAL MUTTO discloses systems and methods for placing multiple cameras and calibrating these cameras with respect to one another, including estimating the 3-D poses of the cameras; use of a sufficiently bright backlit calibration target can also improve the ability of the system to generate meaningful poses when the calibration target is moved in the scene, and CLAVEAU discloses an interactive camera calibration method and an image acquisition and analysis scheme, thus improving the efficiency and execution time of the image acquisition and analysis process. Please see DAL MUTTO (US 20190364206 A1), Paragraph [0118], and CLAVEAU (US 20170287166 A1), Paragraph [0117].
Conclusion
Listed below are the prior arts made of record and not relied upon but are considered pertinent to applicant’s disclosure.
MYOKAN et al. (US 20220148225 A1) - A jig holds an imaging apparatus including a plurality of cameras with different optical axis orientations and a chart including a plurality of planes with different angles and changes the orientation of the imaging apparatus relative to the chart. A calibration apparatus obtains camera parameters of the imaging apparatus by sequentially acquiring captured images captured by adjacent cameras when these cameras have obtained predetermined fields-of-view relative to the chart and extracting images of feature points of chart patterns....… Fig. 1, Abstract.
HAIN et al. (US 20210225032 A1) - Provided is a method for generating pose transformation data between first and second rigidly mounted digital cameras having non-coincident fields of view. The method includes obtaining a first plurality of images of a first calibration object, obtaining a second plurality of images of the first or a second calibration object, generating first and second object point data, generating first and second calibration data of the first and second digital cameras, determining first pose data between a first frame of reference of the first digital camera and the frame of reference of the first calibration object, determining second pose data between the frame of reference of the second digital camera and the frame of reference of the first or second calibration objects, and calculating the pose transformation data between the pose of the first digital camera and the pose of the second digital camera......… Fig. 1, Abstract.
DING et al. (US 20210248781 A1) - A system is disclosed for providing extrinsic calibration of a camera to a relative working environment of a programmable motion device that includes an end-effector. The system includes a fiducial located at or near the end-effector, at least one camera system for viewing the fiducial as the programmable motion device moves in at least three degrees of freedom, and for capturing a plurality of images containing the fiducial, and a calibration system for analyzing the plurality of images to determine a fiducial location with respect to the camera to permit calibration of the camera with the programmable motion device......… Fig. 1, Abstract.
RYDSTRÖM et al. (US 20210215475 A1) - Imaging system based on light triangulation for capturing information on three dimensional characteristics of an object by means of one or more cameras. A calibration target object is within respective field of view of said cameras so that the cameras are able to detect light reflected from a surface structure of the calibration target object comprising one or more regular right pyramidal recesses and one or more regular right pyramids, with their respective bases in the same plane and their respective apexes at the same orthogonal distance from that same plane. The base of at least one of said regular right pyramidal recesses shares at least one side with the base of at least one of said regular right pyramids, such that each pair of lateral faces sharing side are located in a common plane.....… Fig. 1, Abstract.
DRUMHELLER et al. (US 20230099417 A1) - A dynamic dimensioning system includes at least one camera, a range finder, and a tachometer. A computer may be used to perform a dynamic calibration operation for the cameras coupled over a communications network. The dynamic calibration operation includes a calibration estimate routine configured to generate default configuration parameters selected by a user from among a plurality of pre-defined user inputs via a graphical user interface, and a calibration refinement routine configured to refine the default configuration parameters to generate a completed set of calibration parameters that are set for the at least one camera. The dynamic calibration operation may be performed without first performing any static calibration operation......… Fig. 1, Abstract.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to BEZAWIT N SHIMELES whose telephone number is (571)272-7663. The examiner can normally be reached M-F 7:30am-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Chineyere Wills-Burns can be reached at (571) 272-9752. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/BEZAWIT NOLAWI SHIMELES/Examiner, Art Unit 2673
/CHINEYERE WILLS-BURNS/Supervisory Patent Examiner, Art Unit 2673
Prosecution Insights