DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendments
Applicant’s remarks, see page 1, filed 06/08/2026, with respect to the drawing objections, have been fully considered and are persuasive in light of corrected drawing sheets. Thus, the drawing objection has been withdrawn.
Applicant’s remarks, see page 1, filed 06/08/2026, with respect to the specification objections for a typographical error, have been fully considered and are persuasive in light of suggested amendments. Thus, the specification objection has been withdrawn.
Applicant’s remarks, see page 1, filed 06/08/2026, with respect to the claim objections of claims 3, 4, and 10, have been fully considered and are persuasive in light of claim amendments. Thus, the claim objections have been withdrawn.
Response to Arguments
Applicant’s arguments, see remarks, filed 06/08/2026, with respect to claims 1-4 and 6-21, have been fully considered, but are moot because the arguments do not apply to the current references and current combinations of references being used in the current rejection.
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action.
Claims 11-13 recite limitations that use words like “means” (or “step”) or similar terms with functional language but do not invoke 35 U.S.C. 112(f):
Claim 11; recites the limitation, “a computer subsystem having at least one processor and memory storing computer instructions that, when executed by the at least one processor, cause the thermal-augmented image object detection system to…” [Lines 4-6].
Claims 12 & 13: recite the limitation, “computer subsystem is further configured so that…” [Line 2].
Such claim limitation(s) is/are:
(i) “computer subsystem” has a structure associated with it a computer.
Because this/these claim limitation(s) is/are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, it/they is/are not being interpreted to cover only the corresponding structure, material, or acts described in the specification as performing the claimed function, and equivalents thereof.
If applicant does intend to have this/these limitation(s) interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation(s) to remove the structure, materials, or acts that performs the claimed function; or (2) present a sufficient showing that the claim limitations do not recite sufficient structure to perform the claimed function.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-4, 6, 7, 11, 15-18, and 21 are rejected under 35 U.S.C. 103 as being unpatentable over HUYNH (US 11106903 B1), hereinafter referenced as HUYNH in view of MEHTA (US 20180330474 A1), hereinafter referenced as MEHTA.
Regarding claim 1, HUYNH teaches a method of detecting an object within an image (Fig. 11, Col. 15, Lines [26-27] – HUYNH discloses Fig. 11 depicts a flow chart showing an example process 1100 for object detection in image data), comprising:
obtaining a thermal image (Fig. 11, Col. 15, Lines [40-41] – HUYNH discloses at action 1110, the detector 118 may receive an input frame of image data. Lines [46-47] – HUYNH discloses input frame of image data may be, for example, an NIR frame of image data. See also Fig. 1, Col. 4, Lines [17-50].)
captured by a thermal image sensor (Fig. 10, Col. 15, Lines [22-25] – HUYNH discloses although, in the preceding example, NIR and RGB domains are provided by way of example, any image domains (e.g., thermal IR, UV, visible, etc.) may instead be used in accordance with the techniques described herein.),
projecting the upsampled thermal image into a visible light color space to generate a projected thermal image (Fig. 1, Col. 16, Lines [25-28] – HUYNH discloses the generator 106 may transform the feature data from an non-target domain (e.g., NIR) into a target domain (e.g., RGB). Fig. 7, Col. 12, Lines [7-17] – HUYNH discloses the generators described herein (e.g., generators 702, 705) may comprise an encoder and a decoder. The decoder may construct the output, transformed image from the latent feature data using a series of deconvolution (transpose convolution) layers. The deconvolution may include an up-sampling rate/deconvolution stride in order to recover a target output resolution.);
Although HUYNH further teaches and detecting an object within the projected thermal image through inputting the projected thermal image into a visible light object detector (Fig. 8, Col. 13, Lines [49-54] – HUYNH discloses a detector (e.g., an SSD) trained using a dataset of annotated RGB images to detect objects in the RGB domain may receive NIR image data during runtime (e.g., during inference), transform the NIR image data to synthetic RGB image data using generator 802 and detect objects in the synthetic RGB image data. See also Fig. 11, Col. 16, Lines [29-45].)
configured to detect objects within image data within the visible light color space (Fig. 11, Col. 16, Lines [30-45] – HUYNH discloses at action 1180, the generator 106 may pass the synthetic feature data (e.g., a synthetic RGB feature vector) to the next layer of the detector (e.g., an SSD) for further processing. Processing may proceed from action 1180 to action 1190, “Generate bounding box for feature data”. At action 1190, the detector may output a bounding box corresponding to the particular feature data extracted at action 1120. The bounding box may identify a grouping of contiguous pixel addresses within the frame of image data received at action 1110. In various examples, the detector may also classify an object within the bounding box. For example, if the detector is trained to detect dogs, the detector may output classification data indicating a confidence value that an object depicted within the bounding box is a dog.).
HUYNH fails to explicitly teach wherein a super-resolution technique is used to generate an upsampled thermal image based on a single thermal image where the thermal image is used as the single thermal image;
However, MEHTA explicitly teaches wherein a super-resolution technique is used (Fig. 2, Paragraph [0065] – MEHTA discloses various embodiments of the disclosure may advantageously provide super-resolution of images in applications in which the original images may natively have a low resolution, low definition, and/or low signal-to-noise ratio (SNR), such as images captured by infrared imaging sensors (e.g., infrared imaging sensors responsive to thermal infrared, near infrared, or infrared radiation in other wavebands), time-of-flight imaging sensors, millimeter wave imaging sensors, LIDAR sensors, or other imaging sensors that operate in modalities and/or spectra that make it difficult to obtain native high resolution images.)
to generate an upsampled thermal image (Fig. 2, Paragraph [0068] – MEHTA discloses at step 204, original image 305 is resized to a larger size (e.g., to have a larger image dimension) to generate a resized image 310. In some embodiments, upscaling (e.g., by bilinear interpolation, by bicubic interpolation, etc.), upsampling, or other linear higher-order, or non-linear interpolation techniques may be performed on original image 305 to resize original image 305 to a larger dimension than its native image dimension.)
based on a single thermal image where the thermal image is used as the single thermal image (Fig. 2, Paragraph [0036] – MEHTA discloses the imaging sensor 130 may comprise a long-wave IR (LWIR) (e.g., a thermal IR) imaging sensor having a focal plane array (FPA) of detectors responsive to thermal IR radiation including LWIR radiation. Paragraph [0067] – MEHTA discloses at step 202, an original image (such as an original image 305 of FIG. 3) is received, for example, at processing component 110 from imaging sensor 130, or from externally captured and/or stored images or videos via image interface 134.);
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date the claimed invention was made to combine the teachings of HUYNH of having a method of detecting an object within an image, comprising: obtaining a thermal image captured by a thermal image sensor, projecting the upsampled thermal image into a visible light color space to generate a projected thermal image; and detecting an object within the projected thermal image through inputting the projected thermal image into a visible light object detector configured to detect objects within image data within the visible light color space, with the teachings of MEHTA having wherein a super-resolution technique is used to generate an upsampled thermal image based on a single thermal image where the thermal image is used as the single thermal image.
Wherein having HUYNH’s method wherein a super-resolution technique is used to generate an upsampled thermal image based on a single thermal image where the thermal image is used as the single thermal image.
The motivation behind the modification would have been to obtain an enhanced method of image object detection that can reduce computational resources and produce higher quality resolution-enhanced images, since both HUYNH and MEHTA relate to image processing devices and methods that utilize thermal images, wherein HUYNH discloses transformation of a feature vector in one domain to another is a one-to-one transformation which may save computational resources; using the techniques described herein traditional RGB-based image detectors (e.g., those trained in the RGB domain) may be used to detect objects when the input image data is in another domain (e.g., infrared) using the GAN-trained generators described herein, while MEHTA relates generally to imaging processing and more particularly, for example, to enhancement of the resolution of images wherein the main advantages of the self-similarity based approach (also referred to herein as the iterative collaborative filtering approach) for single image super-resolution disclosed herein in accordance with various embodiments of the disclosure include (1) no external data or training required; (2) superior performance; (3) support of non-integral scaling factors (images can be upscale to any size, without any change in parameters); (4) no border pruning effect, unlike most of the dictionary based algorithms; and (5) robustness to noise. Please see HUYNH (US 11106903 B1), Col. 16-17, Lines [46-67, 1-3 respectively], and MEHTA (US 20180330474 A1), Paragraph [0063-0065].
Regarding claim 2, HUYNH in view of MEHTA teach the method of claim 1,
HUYNH further teaches wherein the visible light object detector (Fig. 1, #118 called detector, Col. 4, Lines [17-20])
is configured to detect objects within a visible light image (Fig. 1, #118 called detector, Col. 4, Lines [17-20] - HUYNH discloses detector 118 may be trained to detect objects in a particular domain or modality. For example, detector 118 may be effective to detect objects in RGB image data.)
captured by a visible light sensor (Fig. 1, #130 called camera, Col. 5, Lines [29-33] – HUYNH discloses camera 130 may comprise one or more image sensors effective to generate image data representing a scene. In various examples, camera 130 may be effective to capture visible (e.g., RGB) image data and/or infrared image data.),
the projected thermal image (Fig. 1, Col. 4, Lines [23-28] – HUYNH discloses a generator 106 may be trained using a GAN 120, in accordance with various techniques described in further detail below, to transform feature data extracted from image data of a first domain into feature data in a second domain for which the classifier of detector 118 has been trained. See also Paragraph [0067])
captured by the thermal image sensor (Fig. 1, #130 called camera, Col. 5, Lines [29-33] – HUYNH discloses camera 130 may comprise one or more image sensors effective to generate image data representing a scene. In various examples, camera 130 may be effective to capture visible (e.g., RGB) image data and/or infrared image data.),
or both the visible light image and the projected thermal image (Fig. 1, Col. 5, Lines [3-5] – HUYNH discloses the system depicted in FIG. 1 is able to perform object detection in two different domains (e.g., in the visible, RGB domain, and in the NIR domain).).
Regarding claim 3, HUYNH in view of MEHTA teach the method of claim 1,
HUYNH further teaches further comprising: capturing a visible light image (Fig. 1, Col. 5, Lines [29-33] – HUYNH discloses camera 130 may comprise one or more image sensors effective to generate image data representing a scene. In various examples, camera 130 may be effective to capture visible (e.g., RGB) image data and/or infrared image data.)
and inputting the visible light image into the visible light object detector in order to detect one or more objects located within the visible light image (Fig. 1, Col. 4, Lines [17-20] - HUYNH discloses detector 118 may be trained to detect objects in a particular domain or modality. For example, detector 118 may be effective to detect objects in RGB image data.).
Regarding claim 4, HUYNH in view of MEHTA teach the method of claim 3,
HUYNH further teaches wherein the visible light image includes at least three visible light channels (Fig. 1, Col. 16, Lines [59-60] – HUYNH discloses RGB image data includes three channels),
and wherein at least three visible light channels (Fig. 1, Col. 16, Lines [59-60]) include one or more of the following: a red channel, a green channel, a blue channel (Fig. 1, Col. 2, Lines [24-26] - HUYNH discloses datasets typically comprise RGB (red, green, blue) image data representing image data in the visible light spectrum.).
Regarding claim 6, HUYNH in view of MEHTA teach the method of claim 1,
HUYNH further teaches further comprising: wherein the upsampled thermal image (Fig. 7, Col. 12, Lines [7-17] – HUYNH discloses the generators described herein (e.g., generators 702, 705) may comprise an encoder and a decoder. The decoder may construct the output, transformed image from the latent feature data using a series of deconvolution (transpose convolution) layers. The deconvolution may include an up-sampling rate/deconvolution stride in order to recover a target output resolution.)
is projected to the visible light color space (Fig. 1, Col. 16, Lines [25-28] – HUYNH discloses the generator 106 may transform the feature data from an non-target domain (e.g., NIR) into a target domain (e.g., RGB).)
using a projection function (Figs. 7-8, Col. 3, Lines [17-20] – HUYNH discloses the generator [wherein the generator is performing projection] is trained to map data from a latent space to a particular data distribution of interest (e.g., from RGB image data to near infrared image data). Col. 3, Lines [53-56] – HUYNH further discloses image data may be transformed to and from the far infrared domain, the ultraviolet domain, thermal infrared domain, visible spectrum, etc.)
to generate the projected thermal image (Fig. 1, Col. 16, Lines [25-28] – HUYNH discloses the generator 106 may transform the feature data from a non-target domain (e.g., NIR) into a target domain (e.g., RGB)),
and wherein the projected thermal image is used as the upsampled thermal image for object detection (Fig. 7, Col. 12, Lines [7-17] – HUYNH discloses the generators described herein (e.g., generators 702, 705) may comprise an encoder and a decoder. The decoder may construct the output, transformed image from the latent feature data using a series of deconvolution (transpose convolution) layers. The deconvolution may include an up-sampling rate/deconvolution stride in order to recover a target output resolution. Fig. 8, Col. 13, Lines [49-54] – HUYNH further discloses a detector (e.g., an SSD) trained using a dataset of annotated RGB images to detect objects in the RGB domain may receive NIR image data during runtime (e.g., during inference), transform the NIR image data to synthetic RGB image data using generator 802 and detect objects in the synthetic RGB image data.).
Regarding claim 7, HUYNH in view of MEHTA teach the method of claim 6,
HUYNH further teaches wherein the projected thermal image (Fig. 10, Col. 15, Lines [6-14] – HUYNH discloses depicted in FIG. 10, an NIR image may be received. The NIR image may be encoded to extract feature data in the NIR domain, and the feature data may be sent to detector 1008 to generate first bounding boxes and/or respective first classifications. In parallel, the NIR feature data may be passed to generator 1002. Generator 1002 may be trained in accordance with the various techniques described above (e.g., using one or more GANs) to transform the NIR feature data to synthetic RGB feature data.)
is combined with a visible light image in order to detect an object visible (Fig. 10, Col. 15, Lines [11-25] – HUYNH discloses generator 1002 may be trained in accordance with the various techniques described above (e.g., using one or more GANs) to transform the NIR feature data to synthetic RGB feature data. The synthetic RGB feature data may be passed to detector 1006 to generate second bounding boxes and/or respective second classifications. The results from detector 1006 and detector 1008 may be fused using non-maximal suppression to find the bounding boxes and/or classifications from detectors 1006, 1008 with the highest confidence scores that are more likely to represent a more accurate location of the object-of-interest. Although, in the preceding example, NIR and RGB domains are provided by way of example, any image domains (e.g., thermal IR, UV, visible, etc.) may instead be used in accordance with the techniques described herein.)
within a shared field of view (Fig. 6, Col. 11, Lines [22-35] – HUYNH discloses architecture 600 may also include one or more sensors 630 such as, for example, one or more position sensors, image sensors, and/or motion sensors. An image sensor 632 is shown in FIG. 6. Some examples of the architecture 600 may include multiple image sensors 632. For example, a panoramic camera system may comprise multiple image sensors 632 resulting in multiple images and/or video frames that may be stitched and may be blended to form a seamless panoramic output. An example of an image sensor 632 may be camera 130 shown and described in FIG. 1. As described, camera 130 may be configured to capture color information, IR image data, image geometry information, and/or ambient light information.),
and wherein the shared field of view is a field of view defined by overlapping a field of view of the thermal image sensor and a field of view of a visible light sensor used to capture the visible light image (Fig. 6, Col. 11, Lines [22-35] – HUYNH discloses architecture 600 may also include one or more sensors 630 such as, for example, one or more position sensors, image sensors, and/or motion sensors. An image sensor 632 is shown in FIG. 6. Some examples of the architecture 600 may include multiple image sensors 632. For example, a panoramic camera system may comprise multiple image sensors 632 resulting in multiple images and/or video frames that may be stitched and may be blended to form a seamless panoramic output. An example of an image sensor 632 may be camera 130 shown and described in FIG. 1. As described, camera 130 may be configured to capture color information, IR image data, image geometry information, and/or ambient light information.).
Regarding claim 11, HUYNH teaches a thermal-augmented image object detection system (Fig. 1, Col. 5, Lines [3-5] - HUYNH discloses the system depicted in FIG. 1 is able to perform object detection in two different domains (e.g., in the visible, RGB domain, and in the NIR domain). Col. 3, Lines [54-56] – HUYNH further discloses image data may be transformed to and from the far infrared domain, the ultraviolet domain, thermal infrared domain, visible spectrum, etc.), comprising:
a visible light sensor configured to capture a visible light image (Fig. 1, Col. 5, Lines [29-33] – HUYNH discloses camera 130 may comprise one or more image sensors effective to generate image data representing a scene. In various examples, camera 130 may be effective to capture visible (e.g., RGB) image data and/or infrared image data.);
a thermal image sensor configured to capture a thermal image (Fig. 1, Col. 5, Lines [29-33] – HUYNH discloses camera 130 may comprise one or more image sensors effective to generate image data representing a scene. In various examples, camera 130 may be effective to capture visible (e.g., RGB) image data and/or infrared image data.);
a computer subsystem (Fig. 1, Col. 3, Lines [64-65] – HUYNH discloses system 100 may comprise a detector 118, a GAN 120, a generator 106. Col. 5, Lines [15-17] – HUYNH further discloses computing device(s) 102 may be effective to implement detector 118, GAN 120, and/or generator 106.)
having at least one processor and memory storing computer instructions that, when executed by the at least one processor (Fig. 1, Col. 5, Lines [22-28] – HUYNH discloses computing device(s) 102 may be configured in communication with a non-transitory, computer-readable memory 103. Non-transitory, computer-readable memory 103 may be effective to store one or more instructions that, when executed by at least one processor of computing device(s) 102, program the at least one processor to perform the various techniques described herein.),
cause the thermal-augmented image object detection system (Fig. 1, #100 called system, Col. 3-4, Lines [61-67, and 1-16 respectively]) to:
project the thermal image into a visible light color space to generate a projected thermal image (Fig. 1, Col. 16, Lines [25-28] – HUYNH discloses the generator 106 may transform the feature data from an non-target domain (e.g., NIR) into a target domain (e.g., RGB).);
and detect an object within a shared field of view (Fig. 6, Col. 11, Lines [22-35] – HUYNH discloses architecture 600 may also include one or more sensors 630 such as, for example, one or more position sensors, image sensors, and/or motion sensors. An image sensor 632 is shown in FIG. 6. Some examples of the architecture 600 may include multiple image sensors 632. For example, a panoramic camera system may comprise multiple image sensors 632 resulting in multiple images and/or video frames that may be stitched and may be blended to form a seamless panoramic output. An example of an image sensor 632 may be camera 130 shown and described in FIG. 1. As described, camera 130 may be configured to capture color information, IR image data, image geometry information, and/or ambient light information.)
by inputting one or both of the visible light image (Fig. 11, Col. 15, Lines [40-41] – HUYNH discloses at action 1110, the detector 118 may receive an input frame of image data.)
and the projected thermal image (Fig. 11, Col. 16, Lines [23-28] – HUYNH discloses processing may continue from action 1160 to action 1170, “Transform feature data to synthetic feature data in target domain.” As described above in reference to FIG. 1, the generator 106 may transform the feature data from an non-target domain (e.g., NIR) into a target domain (e.g., RGB).)
into a visible light object detector configured to detect objects within image data within the visible light color space (Fig. 11, Col. 16, Lines [29-34] – HUYNH discloses processing may proceed from action 1170 to action 1180, “Pass feature data to subsequent layer of detector”. At action 1180, the generator 106 may pass the synthetic feature data (e.g., a synthetic RGB feature vector) to the next layer of the detector (e.g., an SSD) for further processing. Lines [41-42] – HUYNH further discloses the detector may also classify an object within the bounding box.).
HUYNH fails to explicitly teach upsample a thermal image using a super-resolution technique, wherein the super-resolution technique is used to generate an upsampled thermal image based on a single thermal image where the thermal image is used as the single thermal image;
However, MEHTA explicitly teaches upsample a thermal image (Fig. 2, Paragraph [0068] – MEHTA discloses at step 204, original image 305 is resized to a larger size (e.g., to have a larger image dimension) to generate a resized image 310. In some embodiments, upscaling (e.g., by bilinear interpolation, by bicubic interpolation, etc.), upsampling, or other linear higher-order, or non-linear interpolation techniques may be performed on original image 305 to resize original image 305 to a larger dimension than its native image dimension.)
using a super-resolution technique (Fig. 2, Paragraph [0065] – MEHTA discloses various embodiments of the disclosure may advantageously provide super-resolution of images in applications in which the original images may natively have a low resolution, low definition, and/or low signal-to-noise ratio (SNR), such as images captured by infrared imaging sensors (e.g., infrared imaging sensors responsive to thermal infrared, near infrared, or infrared radiation in other wavebands), time-of-flight imaging sensors, millimeter wave imaging sensors, LIDAR sensors, or other imaging sensors that operate in modalities and/or spectra that make it difficult to obtain native high resolution images.),
wherein the super-resolution technique is used (Fig. 2, Paragraph [0065] – MEHTA discloses various embodiments of the disclosure may advantageously provide super-resolution of images in applications in which the original images may natively have a low resolution, low definition, and/or low signal-to-noise ratio (SNR), such as images captured by infrared imaging sensors (e.g., infrared imaging sensors responsive to thermal infrared, near infrared, or infrared radiation in other wavebands), time-of-flight imaging sensors, millimeter wave imaging sensors, LIDAR sensors, or other imaging sensors that operate in modalities and/or spectra that make it difficult to obtain native high resolution images)
to generate an upsampled thermal image (Fig. 2, Paragraph [0068] – MEHTA discloses at step 204, original image 305 is resized to a larger size (e.g., to have a larger image dimension) to generate a resized image 310. In some embodiments, upscaling (e.g., by bilinear interpolation, by bicubic interpolation, etc.), upsampling, or other linear higher-order, or non-linear interpolation techniques may be performed on original image 305 to resize original image 305 to a larger dimension than its native image dimension.)
based on a single thermal image where the thermal image is used as the single thermal image (Fig. 2, Paragraph [0036] – MEHTA discloses the imaging sensor 130 may comprise a long-wave IR (LWIR) (e.g., a thermal IR) imaging sensor having a focal plane array (FPA) of detectors responsive to thermal IR radiation including LWIR radiation. Paragraph [0067] – MEHTA discloses at step 202, an original image (such as an original image 305 of FIG. 3) is received, for example, at processing component 110 from imaging sensor 130, or from externally captured and/or stored images or videos via image interface 134.);
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date the claimed invention was made to combine the teachings of HUYNH of having a thermal-augmented image object detection system, comprising: a visible light sensor configured to capture a visible light image; a thermal image sensor configured to capture a thermal image; a computer subsystem having at least one processor and memory storing computer instructions that, when executed by the at least one processor, cause the thermal-augmented image object detection system to: project the thermal image into a visible light color space to generate a projected thermal image; and detect an object within a shared field of view by inputting one or both of the visible light image and the projected thermal image into a visible light object detector configured to detect objects within image data within the visible light color space, with the teachings of MEHTA having upsample a thermal image using a super-resolution technique, wherein the super-resolution technique is used to generate an upsampled thermal image based on a single thermal image where the thermal image is used as the single thermal image;
Wherein HUYNH’s system wherein having upsample a thermal image using a super-resolution technique, wherein the super-resolution technique is used to generate an upsampled thermal image based on a single thermal image where the thermal image is used as the single thermal image;
The motivation behind the modification would have been to obtain an enhanced system for image object detection that can reduce computational resources and produce higher quality resolution-enhanced images, since both HUYNH and MEHTA relate to image processing devices and methods that utilize thermal images, wherein HUYNH discloses transformation of a feature vector in one domain to another is a one-to-one transformation which may save computational resources; using the techniques described herein traditional RGB-based image detectors (e.g., those trained in the RGB domain) may be used to detect objects when the input image data is in another domain (e.g., infrared) using the GAN-trained generators described herein, while MEHTA relates generally to imaging processing and more particularly, for example, to enhancement of the resolution of images wherein the main advantages of the self-similarity based approach (also referred to herein as the iterative collaborative filtering approach) for single image super-resolution disclosed herein in accordance with various embodiments of the disclosure include (1) no external data or training required; (2) superior performance; (3) support of non-integral scaling factors (images can be upscale to any size, without any change in parameters); (4) no border pruning effect, unlike most of the dictionary based algorithms; and (5) robustness to noise. Please see HUYNH (US 11106903 B1), Col. 16-17, Lines [46-67, 1-3 respectively], and MEHTA (US 20180330474 A1), Paragraph [0063-0065].
Regarding claim 15, HUYNH in view of MEHTA teach the thermal-augmented image object detection system of claim 11,
HUYNH further teaches wherein the visible light image includes at least three visible light channels (Fig. 1, Col. 16, Lines [59-60] – HUYNH discloses RGB image data includes three channels).
Regarding claim 16, HUYNH in view of MEHTA teach the thermal-augmented image object detection system of claim 15,
HUYNH further teaches wherein at least three visible light channels (Fig. 1, Col. 16, Lines [59-60]) include one or more of the following: a red channel, a green channel, a blue channel (Fig. 1, Col. 2, Lines [24-26] - HUYNH discloses datasets typically comprise RGB (red, green, blue) image data representing image data in the visible light spectrum.).
Regarding claim 17, HUYNH in view of MEHTA teach the thermal-augmented image object detection system of claim 11,
HUYNH further teaches wherein the upsampled thermal image (Fig. 7, Col. 12, Lines [7-17] – HUYNH discloses the generators described herein (e.g., generators 702, 705) may comprise an encoder and a decoder. The decoder may construct the output, transformed image from the latent feature data using a series of deconvolution (transpose convolution) layers. The deconvolution may include an up-sampling rate/deconvolution stride in order to recover a target output resolution.)
is projected to the visible light color space (Fig. 1, Col. 16, Lines [25-28] – HUYNH discloses the generator 106 may transform the feature data from an non-target domain (e.g., NIR) into a target domain (e.g., RGB).)
using a projection function (Figs. 7-8, Col. 3, Lines [17-20] – HUYNH discloses the generator [wherein the generator is performing projection] is trained to map data from a latent space to a particular data distribution of interest (e.g., from RGB image data to near infrared image data). Col. 3, Lines [53-56] – HUYNH further discloses image data may be transformed to and from the far infrared domain, the ultraviolet domain, thermal infrared domain, visible spectrum, etc.)
to generate a projected thermal image (Fig. 1, Col. 16, Lines [25-28] – HUYNH discloses the generator 106 may transform the feature data from a non-target domain (e.g., NIR) into a target domain (e.g., RGB)),
and wherein the projected thermal image is used as the upsampled thermal image for object detection (Fig. 7, Col. 12, Lines [7-17] – HUYNH discloses the generators described herein (e.g., generators 702, 705) may comprise an encoder and a decoder. The decoder may construct the output, transformed image from the latent feature data using a series of deconvolution (transpose convolution) layers. The deconvolution may include an up-sampling rate/deconvolution stride in order to recover a target output resolution. Fig. 8, Col. 13, Lines [49-54] – HUYNH further discloses a detector (e.g., an SSD) trained using a dataset of annotated RGB images to detect objects in the RGB domain may receive NIR image data during runtime (e.g., during inference), transform the NIR image data to synthetic RGB image data using generator 802 and detect objects in the synthetic RGB image data.).
Regarding claim 18, HUYNH in view of MEHTA teach the thermal-augmented image object detection system of claim 17,
HUYNH further teaches wherein the projected thermal image (Fig. 10, Col. 15, Lines [6-14] – HUYNH discloses depicted in FIG. 10, an NIR image may be received. The NIR image may be encoded to extract feature data in the NIR domain, and the feature data may be sent to detector 1008 to generate first bounding boxes and/or respective first classifications. In parallel, the NIR feature data may be passed to generator 1002. Generator 1002 may be trained in accordance with the various techniques described above (e.g., using one or more GANs) to transform the NIR feature data to synthetic RGB feature data.)
is combined with a visible light image in order to detect an object visible (Fig. 10, Col. 15, Lines [11-25] – HUYNH discloses generator 1002 may be trained in accordance with the various techniques described above (e.g., using one or more GANs) to transform the NIR feature data to synthetic RGB feature data. The synthetic RGB feature data may be passed to detector 1006 to generate second bounding boxes and/or respective second classifications. The results from detector 1006 and detector 1008 may be fused using non-maximal suppression to find the bounding boxes and/or classifications from detectors 1006, 1008 with the highest confidence scores that are more likely to represent a more accurate location of the object-of-interest. Although, in the preceding example, NIR and RGB domains are provided by way of example, any image domains (e.g., thermal IR, UV, visible, etc.) may instead be used in accordance with the techniques described herein.)
within a shared field of view (Fig. 6, Col. 11, Lines [22-35] – HUYNH discloses architecture 600 may also include one or more sensors 630 such as, for example, one or more position sensors, image sensors, and/or motion sensors. An image sensor 632 is shown in FIG. 6. Some examples of the architecture 600 may include multiple image sensors 632. For example, a panoramic camera system may comprise multiple image sensors 632 resulting in multiple images and/or video frames that may be stitched and may be blended to form a seamless panoramic output. An example of an image sensor 632 may be camera 130 shown and described in FIG. 1. As described, camera 130 may be configured to capture color information, IR image data, image geometry information, and/or ambient light information.),
and wherein the shared field of view is a field of view defined by overlapping a field of view of the thermal image sensor and a field of view of a visible light sensor used to capture the visible light image (Fig. 6, Col. 11, Lines [22-35] – HUYNH discloses architecture 600 may also include one or more sensors 630 such as, for example, one or more position sensors, image sensors, and/or motion sensors. An image sensor 632 is shown in FIG. 6. Some examples of the architecture 600 may include multiple image sensors 632. For example, a panoramic camera system may comprise multiple image sensors 632 resulting in multiple images and/or video frames that may be stitched and may be blended to form a seamless panoramic output. An example of an image sensor 632 may be camera 130 shown and described in FIG. 1. As described, camera 130 may be configured to capture color information, IR image data, image geometry information, and/or ambient light information.).
Regarding claim 21, HUYNH in view of MEHTA teach the method of claim 1,
HUYNH further teaches down-sampling the single thermal image to obtain a down-sampled image (Fig. 7, Col. 12, Lines 7-12] – HUYNH discloses the generators described herein (e.g., generators 702, 705) may comprise an encoder and a decoder. The encoder may extract latent feature data from input image data by applying consecutive convolution operations, optionally down-sampling the feature data with a convolution stride larger than one.),
and upsampling the down-sampled image to obtain an upsampled image (Fig. 7, Col. 12, Lines [7-16] – HUYNH discloses the generators described herein (e.g., generators 702, 705) may comprise an encoder and a decoder. The encoder may extract latent feature data from input image data by applying consecutive convolution operations, optionally down-sampling the feature data with a convolution stride larger than one. The decoder may construct the output, transformed image from the latent feature data using a series of deconvolution (transpose convolution) layers. The deconvolution may include an up-sampling rate/deconvolution stride in order to recover a target output resolution.),
wherein the upsampled thermal image (Fig. 7, Col. 12, Lines [7-17] – HUYNH discloses the generators described herein (e.g., generators 702, 705) may comprise an encoder and a decoder. The decoder may construct the output, transformed image from the latent feature data using a series of deconvolution (transpose convolution) layers. The deconvolution may include an up-sampling rate/deconvolution stride in order to recover a target output resolution.)
is projected to the visible light color space (Fig. 1, Col. 16, Lines [25-28] – HUYNH discloses the generator 106 may transform the feature data from an non-target domain (e.g., NIR) into a target domain (e.g., RGB).)
to generate the projected thermal image (Fig. 1, Col. 16, Lines [25-28] – HUYNH discloses the generator 106 may transform the feature data from a non-target domain (e.g., NIR) into a target domain (e.g., RGB). See also Col. 4, Lines [23-50].).
HUYNH fails to explicitly teach wherein the super-resolution technique includes:
However, MEHTA explicitly teaches wherein the super-resolution technique (Fig. 2, Paragraph [0065] – MEHTA discloses various embodiments of the disclosure may advantageously provide super-resolution of images in applications in which the original images may natively have a low resolution, low definition, and/or low signal-to-noise ratio (SNR)) includes:
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date the claimed invention was made to combine the teachings of HUYNH in view of MEHTA of having a method of detecting an object within an image, comprising: obtaining a thermal image captured by a thermal image sensor, wherein a super-resolution technique is used to generate an upsampled thermal image based on a single thermal image where the thermal image is used as the single thermal image; with the teachings of MEHTA of having wherein the super-resolution technique includes:
Wherein HUYNH’s method wherein the super-resolution technique includes: down-sampling the single thermal image to obtain a down-sampled image, and upsampling the down-sampled image to obtain an upsampled image, wherein the upsampled image is projected into the visible light color space to generate the projected thermal image.
The motivation behind the modification would have been to obtain an enhanced method of image object detection that can reduce computational resources and produce higher quality resolution-enhanced images, since both HUYNH and MEHTA relate to image processing devices and methods that utilize thermal images, wherein HUYNH discloses transformation of a feature vector in one domain to another is a one-to-one transformation which may save computational resources; using the techniques described herein traditional RGB-based image detectors (e.g., those trained in the RGB domain) may be used to detect objects when the input image data is in another domain (e.g., infrared) using the GAN-trained generators described herein, while MEHTA relates generally to imaging processing and more particularly, for example, to enhancement of the resolution of images wherein the main advantages of the self-similarity based approach (also referred to herein as the iterative collaborative filtering approach) for single image super-resolution disclosed herein in accordance with various embodiments of the disclosure include (1) no external data or training required; (2) superior performance; (3) support of non-integral scaling factors (images can be upscale to any size, without any change in parameters); (4) no border pruning effect, unlike most of the dictionary based algorithms; and (5) robustness to noise. Please see HUYNH (US 11106903 B1), Col. 16-17, Lines [46-67, 1-3 respectively], and MEHTA (US 20180330474 A1), Paragraph [0063-0065].
Claims 8, 10, 12-14 are rejected under 35 U.S.C. 103 as being unpatentable over HUYNH (US 11106903 B1), hereinafter referenced as HUYNH in view of MEHTA (US 20180330474 A1), hereinafter referenced as MEHTA in further view of BLEYER (US 20210160441 A1), hereinafter referenced as BLEYER.
Regarding claim 8, HUYNH in view of MEHTA teach the method of claim 1,
Although HUYNH further teaches further comprising: capturing a visible light image using a visible light sensor (Fig. 1, Col. 5, Lines [29-33] – HUYNH discloses camera 130 may comprise one or more image sensors effective to generate image data representing a scene. In various examples, camera 130 may be effective to capture visible (e.g., RGB) image data and/or infrared image data.);
HUYNH in view of MEHTA fail to explicitly teach and determining whether the visible light is degraded, wherein the upsampled thermal image is used for object detection when it is determined that the visible light image is degraded, and wherein the visible light image is used for object detection when it is determined that the visible light image is not degraded.
However, BLEYER explicitly teaches and determining whether the visible light is degraded (Fig. 12, Paragraph [0152] – BLEYER discloses a light sensor may be used to initially determine the light conditions of the environment. If the light sensor indicates that the light is low, then an attempt may be made to use the low light cameras to generate the passthrough image. Alternatively, if the light sensor indicates the light conditions are too low, then the embodiments may refrain from using either one of the visible light cameras or the low light cameras),
wherein the upsampled thermal image (Fig. 3B, Paragraph [0070]- BLEYER discloses the camera distortion correction 330 may include optimizations to correct for distortions related to barrel distortion, pincushion distortion, flare, ghosts, spherical aberrations, chromatic aberrations, coma aberrations, astigmatisms, shutter speed, resolution [wherein resolution correction is upsampling], brightness abilities, intensity abilities, and/or exposure properties)
is used for object detection when it is determined that the visible light image is degraded (Fig. 12, Paragraph [0153] – BLEYER discloses in response to determining that neither one of the first camera or the second camera is usable, there is an act (act 1215) of causing the third camera to generate a thermal image of the environment. Paragraph [0154] – BLEYER further discloses method 1200 then includes an act (act 1220) of performing planar reprojection on the thermal image. The planar reprojection is performed by selecting, relative to the computer system/HMD, a perspective distance at which to project the thermal image so as to provide depth for the thermal image.),
and wherein the visible light image is used for object detection when it is determined that the visible light image is not degraded (Fig. 6A, Paragraph [0108] – BLEYER discloses FIG. 6A shows another example of a lighted environment 600. This particular environment includes a hot and steaming pot 605. Because the lighted environment 600 is sufficiently lit (i.e. has a threshold degree of illuminance to enable the visible light cameras to accurately determine depth, position, and so forth), the HMD's visible light cameras may be used to generate visible light passthrough images, as was described in FIG. 4C. See also Fig. 4C, Paragraph [0103].).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date the claimed invention was made to combine the teachings of HUYNH in view of MEHTA of having a method of detecting an object within an image, comprising: obtaining a thermal image captured by a thermal image sensor, wherein a super-resolution technique is used to generate an upsampled thermal image based on a single thermal image where the thermal image is used as the single thermal image; projecting the upsampled thermal image into a visible light color space to generate a projected thermal image; and detecting an object within the projected thermal image through inputting the projected thermal image into a visible light object detector configured to detect objects within image data within the visible light color space, with the teachings of BLEYER of having and determining whether the visible light is degraded, wherein the upsampled thermal image is used for object detection when it is determined that the visible light image is degraded, and wherein the visible light image is used for object detection when it is determined that the visible light image is not degraded.
Wherein having HUYNH’s method wherein having and determining whether the visible light is degraded, wherein the upsampled thermal image is used for object detection when it is determined that the visible light image is degraded, and wherein the visible light image is used for object detection when it is determined that the visible light image is not degraded.
The motivation behind the modification would have been to obtain an enhanced method of image object detection that can reduce computational resources and improve the user’s experience, since both HUYNH and BLEYER relate to image processing devices and methods that utilize thermal images, wherein HUYNH discloses transformation of a feature vector in one domain to another is a one-to-one transformation which may save computational resources; using the techniques described herein traditional RGB-based image detectors (e.g., those trained in the RGB domain) may be used to detect objects when the input image data is in another domain (e.g., infrared) using the GAN-trained generators described herein, while BLEYER relates generally to systems, methods, and devices (e.g., hardware storage devices, wearable devices, etc.) that enhance passthrough visualizations wherein embodiments provide a synergistic effect by combining multiple different types of data to provide substantial benefits beyond which any one of those data types would be capable of providing; additionally, by providing an enhanced passthrough visualization, the embodiments directly and substantially improve the user's experience. Please see HUYNH (US 11106903 B1), Col. 16-17, Lines [46-67, 1-3 respectively], and BLEYER (US 20210160441 A1), Paragraph [0035-0036].
Regarding claim 10, HUYNH teaches a method of detecting an object within a thermal image (Fig. 11, Col. 15, Lines [26-27] – HUYNH discloses Fig. 11 depicts a flow chart showing an example process 1100 for object detection in image data. Col. 15, Lines [22-25] – HUYNH discloses although, in the preceding example, NIR and RGB domains are provided by way of example, any image domains (e.g., thermal IR, UV, visible, etc.) may instead be used in accordance with the techniques described herein.), comprising:
obtaining a visible light image (Fig. 11, Col. 15, Lines [40-41] – HUYNH discloses at action 1110, the detector 118 may receive an input frame of image data.)
captured by a visible light sensor (Fig. 1, Col. 5, Lines [29-33] – HUYNH discloses camera 130 may comprise one or more image sensors effective to generate image data representing a scene. In various examples, camera 130 may be effective to capture visible (e.g., RGB) image data and/or infrared image data.);
projecting the upsampled thermal image into a visible light color space to generate a projected thermal image (Fig. 1, Col. 16, Lines [25-28] – HUYNH discloses the generator 106 may transform the feature data from an non-target domain (e.g., NIR) into a target domain (e.g., RGB). Fig. 7, Col. 12, Lines [7-17] – HUYNH discloses the generators described herein (e.g., generators 702, 705) may comprise an encoder and a decoder. The decoder may construct the output, transformed image from the latent feature data using a series of deconvolution (transpose convolution) layers. The deconvolution may include an up-sampling rate/deconvolution stride in order to recover a target output resolution.);
Although HUYNH further teaches and detecting an object within the projected thermal image through inputting the projected thermal image into a visible light object detector (Fig. 8, Col. 13, Lines [49-54] – HUYNH discloses a detector (e.g., an SSD) trained using a dataset of annotated RGB images to detect objects in the RGB domain may receive NIR image data during runtime (e.g., during inference), transform the NIR image data to synthetic RGB image data using generator 802 and detect objects in the synthetic RGB image data. See also Fig. 11, Col. 16, Lines [29-45].);
HUYNH fails to explicitly teach upsampling a thermal image using a super-resolution technique, wherein the super-resolution technique is used to generate an upsampled thermal image based on a single thermal image where the thermal image is used as the single thermal image;
However, MEHTA explicitly teaches upsampling a thermal image (Fig. 2, Paragraph [0068] – MEHTA discloses at step 204, original image 305 is resized to a larger size (e.g., to have a larger image dimension) to generate a resized image 310. In some embodiments, upscaling (e.g., by bilinear interpolation, by bicubic interpolation, etc.), upsampling, or other linear higher-order, or non-linear interpolation techniques may be performed on original image 305 to resize original image 305 to a larger dimension than its native image dimension.)
using a super-resolution technique (Fig. 2, Paragraph [0065] – MEHTA discloses various embodiments of the disclosure may advantageously provide super-resolution of images in applications in which the original images may natively have a low resolution, low definition, and/or low signal-to-noise ratio (SNR), such as images captured by infrared imaging sensors (e.g., infrared imaging sensors responsive to thermal infrared, near infrared, or infrared radiation in other wavebands), time-of-flight imaging sensors, millimeter wave imaging sensors, LIDAR sensors, or other imaging sensors that operate in modalities and/or spectra that make it difficult to obtain native high resolution images.),
wherein the super-resolution technique is used (Fig. 2, Paragraph [0065] – MEHTA discloses various embodiments of the disclosure may advantageously provide super-resolution of images in applications in which the original images may natively have a low resolution, low definition, and/or low signal-to-noise ratio (SNR), such as images captured by infrared imaging sensors (e.g., infrared imaging sensors responsive to thermal infrared, near infrared, or infrared radiation in other wavebands), time-of-flight imaging sensors, millimeter wave imaging sensors, LIDAR sensors, or other imaging sensors that operate in modalities and/or spectra that make it difficult to obtain native high resolution images)
to generate an upsampled thermal image (Fig. 2, Paragraph [0068] – MEHTA discloses at step 204, original image 305 is resized to a larger size (e.g., to have a larger image dimension) to generate a resized image 310. In some embodiments, upscaling (e.g., by bilinear interpolation, by bicubic interpolation, etc.), upsampling, or other linear higher-order, or non-linear interpolation techniques may be performed on original image 305 to resize original image 305 to a larger dimension than its native image dimension.)
based on a single thermal image where the thermal image is used as the single thermal image (Fig. 2, Paragraph [0036] – MEHTA discloses the imaging sensor 130 may comprise a long-wave IR (LWIR) (e.g., a thermal IR) imaging sensor having a focal plane array (FPA) of detectors responsive to thermal IR radiation including LWIR radiation. Paragraph [0067] – MEHTA discloses at step 202, an original image (such as an original image 305 of FIG. 3) is received, for example, at processing component 110 from imaging sensor 130, or from externally captured and/or stored images or videos via image interface 134.);
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date the claimed invention was made to combine the teachings of HUYNH of having a method of detecting an object within a thermal image, comprising: obtaining a visible light image captured by a visible light sensor; projecting the upsampled thermal image into a visible light color space to generate a projected thermal image; and detecting an object within the projected thermal image through inputting the projected thermal image into a visible light object detector; with the teachings of MEHTA having upsampling a thermal image using a super-resolution technique, wherein the super-resolution technique is used to generate an upsampled thermal image based on a single thermal image where the thermal image is used as the single thermal image;
Wherein having HUYNH’s method wherein having upsampling a thermal image using a super-resolution technique, wherein the super-resolution technique is used to generate an upsampled thermal image based on a single thermal image where the thermal image is used as the single thermal image;
The motivation behind the modification would have been to obtain an enhanced method of image object detection that can reduce computational resources and produce higher quality resolution-enhanced images, since both HUYNH and MEHTA relate to image processing devices and methods that utilize thermal images, wherein HUYNH discloses transformation of a feature vector in one domain to another is a one-to-one transformation which may save computational resources; using the techniques described herein traditional RGB-based image detectors (e.g., those trained in the RGB domain) may be used to detect objects when the input image data is in another domain (e.g., infrared) using the GAN-trained generators described herein, while MEHTA relates generally to imaging processing and more particularly, for example, to enhancement of the resolution of images wherein the main advantages of the self-similarity based approach (also referred to herein as the iterative collaborative filtering approach) for single image super-resolution disclosed herein in accordance with various embodiments of the disclosure include (1) no external data or training required; (2) superior performance; (3) support of non-integral scaling factors (images can be upscale to any size, without any change in parameters); (4) no border pruning effect, unlike most of the dictionary based algorithms; and (5) robustness to noise. Please see HUYNH (US 11106903 B1), Col. 16-17, Lines [46-67, 1-3 respectively], and MEHTA (US 20180330474 A1), Paragraph [0063-0065].
HUNYH in view of MEHTA fail to explicitly teach determining whether the visible light is degraded; when it is determined that the visible light image is degraded: and when it is determined that the visible light image is not degraded, detecting an object within the visible light image through inputting the visible light image into the object detector.
However, BLEYER explicitly teaches determining whether the visible light is degraded (Fig. 12, Paragraph [0152] – BLEYER discloses a light sensor may be used to initially determine the light conditions of the environment. If the light sensor indicates that the light is low, then an attempt may be made to use the low light cameras to generate the passthrough image. Alternatively, if the light sensor indicates the light conditions are too low, then the embodiments may refrain from using either one of the visible light cameras or the low light cameras);
when it is determined that the visible light image is degraded (Fig. 12, Paragraph [0152] – BLEYER discloses a light sensor may be used to initially determine the light conditions of the environment. If the light sensor indicates that the light is low, then an attempt may be made to use the low light cameras to generate the passthrough image. Alternatively, if the light sensor indicates the light conditions are too low, then the embodiments may refrain from using either one of the visible light cameras or the low light cameras. Paragraph [0153] – BLEYER further discloses in response to determining that neither one of the first camera or the second camera is usable, there is an act (act 1215) of causing the third camera to generate a thermal image of the environment.):
and when it is determined that the visible light image is not degraded (Fig. 6A, Paragraph [0108] – BLEYER discloses FIG. 6A shows another example of a lighted environment 600. This particular environment includes a hot and steaming pot 605. Because the lighted environment 600 is sufficiently lit (i.e. has a threshold degree of illuminance to enable the visible light cameras to accurately determine depth, position, and so forth), the HMD's visible light cameras may be used to generate visible light passthrough images, as was described in FIG. 4C. See also Fig. 4C, Paragraph [0103].),
detecting an object within the visible light image through inputting the visible light image into the object detector (Fig. 6A, Paragraph [0108] – BLEYER discloses FIG. 6A shows another example of a lighted environment 600. This particular environment includes a hot and steaming pot 605. Because the lighted environment 600 is sufficiently lit (i.e. has a threshold degree of illuminance to enable the visible light cameras to accurately determine depth, position, and so forth), the HMD's visible light cameras may be used to generate visible light passthrough images, as was described in FIG. 4C. Fig. 1, Paragraph [0039] – BLEYER further discloses HMD 100 can use the scanning sensor(s) 105 to scan environments, map environments, capture environmental data, and/or generate images of any kind of environment (e.g., by generating a 3D representation of the environment or by generating a “passthrough” visualization) [wherein the HMD is a visible light object detector]. Scanning sensor(s) 105 may comprise any number or any type of scanning devices, without limit. See also Fig. 4C, Paragraph [0103].).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date the claimed invention was made to combine the teachings of HUYNH in view of MEHTA of having a method of detecting an object within a thermal image, comprising: obtaining a visible light image captured by a visible light sensor; projecting the upsampled thermal image into a visible light color space to generate a projected thermal image; and detecting an object within the projected thermal image through inputting the projected thermal image into a visible light object detector; upsampling a thermal image using a super-resolution technique, wherein the super-resolution technique is used to generate an upsampled thermal image based on a single thermal image where the thermal image is used as the single thermal image; with the teachings of BLEYER of having determining whether the visible light is degraded; when it is determined that the visible light image is degraded: and when it is determined that the visible light image is not degraded, detecting an object within the visible light image through inputting the visible light image into the object detector.
Wherein HUYNH’s method wherein having determining whether the visible light is degraded; when it is determined that the visible light image is degraded: upsampling a thermal image using a super-resolution technique, and when it is determined that the visible light image is not degraded, detecting an object within the visible light image through inputting the visible light image into the object detector.
The motivation behind the modification would have been to obtain an enhanced method of image object detection that can reduce computational resources and improve the user’s experience, since both HUYNH and BLEYER relate to image processing devices and methods that utilize thermal images, wherein HUYNH discloses transformation of a feature vector in one domain to another is a one-to-one transformation which may save computational resources; using the techniques described herein traditional RGB-based image detectors (e.g., those trained in the RGB domain) may be used to detect objects when the input image data is in another domain (e.g., infrared) using the GAN-trained generators described herein, while BLEYER relates generally to systems, methods, and devices (e.g., hardware storage devices, wearable devices, etc.) that enhance passthrough visualizations wherein embodiments provide a synergistic effect by combining multiple different types of data to provide substantial benefits beyond which any one of those data types would be capable of providing; additionally, by providing an enhanced passthrough visualization, the embodiments directly and substantially improve the user's experience. Please see HUYNH (US 11106903 B1), Col. 16-17, Lines [46-67, 1-3 respectively], and BLEYER (US 20210160441 A1), Paragraph [0035-0036].
Regarding claim 12, HUYNH in view of MEHTA teach the thermal-augmented image object detection system of claim 11,
HUYNH further teaches wherein the computer subsystem is further configured so that, when the computer instructions are executed by the at least one processor (Fig. 1, Col. 5, Lines [22-28] – HUYNH discloses computing device(s) 102 may be configured in communication with a non-transitory, computer-readable memory 103. Non-transitory, computer-readable memory 103 may be effective to store one or more instructions that, when executed by at least one processor of computing device(s) 102, program the at least one processor to perform the various techniques described herein.),
and to input the upsampled thermal image (Fig. 7, Col. 12, Lines [7-17] – HUYNH discloses the generators described herein (e.g., generators 702, 705) may comprise an encoder and a decoder. The decoder may construct the output, transformed image from the latent feature data using a series of deconvolution (transpose convolution) layers. The deconvolution may include an up-sampling rate/deconvolution stride in order to recover a target output resolution.)
into the visible light object detector in order to detect objects (Fig. 8, Col. 13, Lines [49-54] – HUYNH further discloses a detector (e.g., an SSD) trained using a dataset of annotated RGB images to detect objects in the RGB domain may receive NIR image data during runtime (e.g., during inference), transform the NIR image data to synthetic RGB image data using generator 802 and detect objects in the synthetic RGB image data.).
HUYNH fails to explicitly teach to generate the upsampled thermal image
However, MEHTA explicitly teaches to generate the upsampled thermal image (Fig. 2, Paragraph [0068] – MEHTA discloses at step 204, original image 305 is resized to a larger size (e.g., to have a larger image dimension) to generate a resized image 310. In some embodiments, upscaling (e.g., by bilinear interpolation, by bicubic interpolation, etc.), upsampling, or other linear higher-order, or non-linear interpolation techniques may be performed on original image 305 to resize original image 305 to a larger dimension than its native image dimension. See also Paragraph [0065].)
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date the claimed invention was made to combine the teachings of HUYNH in view of MEHTA of having a thermal-augmented image object detection system, comprising: a visible light sensor configured to capture a visible light image; a thermal image sensor configured to capture a thermal image; a computer subsystem having at least one processor and memory storing computer instructions that, when executed by the at least one processor, cause the thermal-augmented image object detection system to: project the thermal image into a visible light color space to generate a projected thermal image; with the teachings of MEHTA having to generate the upsampled thermal image.
Wherein HUYNH’s system wherein having to generate the upsampled thermal image.
The motivation behind the modification would have been to obtain an enhanced system for image object detection that can reduce computational resources and produce higher quality resolution-enhanced images, since both HUYNH and MEHTA relate to image processing devices and methods that utilize thermal images, wherein HUYNH discloses transformation of a feature vector in one domain to another is a one-to-one transformation which may save computational resources; using the techniques described herein traditional RGB-based image detectors (e.g., those trained in the RGB domain) may be used to detect objects when the input image data is in another domain (e.g., infrared) using the GAN-trained generators described herein, while MEHTA relates generally to imaging processing and more particularly, for example, to enhancement of the resolution of images wherein the main advantages of the self-similarity based approach (also referred to herein as the iterative collaborative filtering approach) for single image super-resolution disclosed herein in accordance with various embodiments of the disclosure include (1) no external data or training required; (2) superior performance; (3) support of non-integral scaling factors (images can be upscale to any size, without any change in parameters); (4) no border pruning effect, unlike most of the dictionary based algorithms; and (5) robustness to noise. Please see HUYNH (US 11106903 B1), Col. 16-17, Lines [46-67, 1-3 respectively], and MEHTA (US 20180330474 A1), Paragraph [0063-0065].
HUYNH in view of MEHTA fail to explicitly teach the thermal-augmented image object detection system determines whether the visible light is degraded; and, when it is determined that the visible light image is degraded,
However, BLEYER explicitly teaches the thermal-augmented image object detection system determines whether the visible light is degraded (Fig. 12, Paragraph [0152] – BLEYER discloses a light sensor may be used to initially determine the light conditions of the environment. If the light sensor indicates that the light is low, then an attempt may be made to use the low light cameras to generate the passthrough image. Alternatively, if the light sensor indicates the light conditions are too low, then the embodiments may refrain from using either one of the visible light cameras or the low light cameras);
and, when it is determined that the visible light image is degraded (Fig. 12, Paragraph [0152] – BLEYER discloses a light sensor may be used to initially determine the light conditions of the environment. If the light sensor indicates that the light is low, then an attempt may be made to use the low light cameras to generate the passthrough image. Alternatively, if the light sensor indicates the light conditions are too low, then the embodiments may refrain from using either one of the visible light cameras or the low light cameras. Paragraph [0153] – BLEYER discloses in response to determining that neither one of the first camera or the second camera is usable, there is an act (act 1215) of causing the third camera to generate a thermal image of the environment.),
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date the claimed invention was made to combine the teachings of HUYNH in view of MEHTA of having a thermal-augmented image object detection system, comprising: a visible light sensor configured to capture a visible light image; a thermal image sensor configured to capture a thermal image; a computer subsystem having at least one processor and memory storing computer instructions that, when executed by the at least one processor, cause the thermal-augmented image object detection system to: project the thermal image into a visible light color space to generate a projected thermal image; with the teachings of BLEYER having the thermal-augmented image object detection system determines whether the visible light is degraded; and, when it is determined that the visible light image is degraded.
Wherein HUYNH’s system wherein having the thermal-augmented image object detection system determines whether the visible light is degraded; and, when it is determined that the visible light image is degraded, to generate the upsampled thermal image.
The motivation behind the modification would have been to obtain an enhanced system for image object detection that can reduce computational resources and improve the user’s experience, since both HUYNH and BLEYER relate to image processing devices and methods that utilize thermal images, wherein HUYNH discloses transformation of a feature vector in one domain to another is a one-to-one transformation which may save computational resources; using the techniques described herein traditional RGB-based image detectors (e.g., those trained in the RGB domain) may be used to detect objects when the input image data is in another domain (e.g., infrared) using the GAN-trained generators described herein, while BLEYER relates generally to systems, methods, and devices (e.g., hardware storage devices, wearable devices, etc.) that enhance passthrough visualizations wherein embodiments provide a synergistic effect by combining multiple different types of data to provide substantial benefits beyond which any one of those data types would be capable of providing; additionally, by providing an enhanced passthrough visualization, the embodiments directly and substantially improve the user's experience. Please see HUYNH (US 11106903 B1), Col. 16-17, Lines [46-67, 1-3 respectively], and BLEYER (US 20210160441 A1), Paragraph [0035-0036].
Regarding claim 13, HUYNH and MEHTA in view of BLEYER teach thermal-augmented image object detection system of claim 12,
HUYNH further teaches wherein the computer subsystem is further configured so that, when the computer instructions are executed by the at least one processor (Fig. 1, Col. 5, Lines [22-28] – HUYNH discloses computing device(s) 102 may be configured in communication with a non-transitory, computer-readable memory 103. Non-transitory, computer-readable memory 103 may be effective to store one or more instructions that, when executed by at least one processor of computing device(s) 102, program the at least one processor to perform the various techniques described herein.),
the thermal-augmented image object detection system inputs the visible light image into the visible light object detector in order to detect objects (Fig. 1, Col. 4, Lines [17-20] - HUYNH discloses detector 118 may be trained to detect objects in a particular domain or modality. For example, detector 118 may be effective to detect objects in RGB image data. See also Col. 5, Lines [29-33].).
Regarding claim 14, HUYNH and MEHTA in view of BLEYER teach thermal-augmented image object detection system of claim 12,
HUYNH further teaches wherein the shared field of view is a field of view defined by overlapping of a field of view of the thermal image sensor and a field of view of a visible light sensor used to capture the visible light image (Fig. 6, Col. 11, Lines [22-35] – HUYNH discloses architecture 600 may also include one or more sensors 630 such as, for example, one or more position sensors, image sensors, and/or motion sensors. An image sensor 632 is shown in FIG. 6. Some examples of the architecture 600 may include multiple image sensors 632. For example, a panoramic camera system may comprise multiple image sensors 632 resulting in multiple images and/or video frames that may be stitched and may be blended to form a seamless panoramic output. An example of an image sensor 632 may be camera 130 shown and described in FIG. 1. As described, camera 130 may be configured to capture color information, IR image data, image geometry information, and/or ambient light information.).
Claim 9 is rejected under 35 U.S.C. 103 as being unpatentable over HUYNH (US 11106903 B1), hereinafter referenced as HUYNH in view of MEHTA (US 20180330474 A1), hereinafter referenced as MEHTA in further view of MOHR (US 20230306747 A1), hereinafter referenced as MOHR.
Regarding claim 9, HUYNH in view of MEHTA teach the method of claim 1,
Although HUYNH further teaches wherein the visible light object detector (Fig. 11, Col. 15, Lines [40-41] – HUYNH discloses at action 1110, the detector 118 may receive an input frame of image data. Col. 5, Lines [29-33] – HUYNH discloses camera 130 may comprise one or more image sensors effective to generate image data representing a scene. In various examples, camera 130 may be effective to capture visible (e.g., RGB) image data and/or infrared image data.).
HUYNH in view of MEHTA fail to explicitly teach wherein the visible light object detector includes a bidirectional feature pyramid network and an object detector head.
However, MOHR explicitly teaches wherein the visible light object detector includes a bidirectional feature pyramid network and an object detector head (Fig. 3, Paragraph [0063] – MOHR discloses the object detection unit 104 employs a weighted bi-directional feature pyramid network (BiFPN), which allows easy and fast multi-scale feature fusion and a compound scaling method that uniformly scales the resolution, depth, and width for all backbone, feature network, and box/class prediction networks at the same time. Paragraph [0045] – MOHR further discloses a type of object detection machine learning model configured to perform a plurality of optimizations and/or operations for detecting the objects in the environment, such as the use of a weighted Bi-directional Feature Pyramid Network (BiFPN) for allowing easier and faster multi-scale feature combinations, and/or a compound scaling method for uniformly scaling the resolution, depth and width for all feature networks and box/class prediction networks at the same time. See also Paragraph [0040].).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date the claimed invention was made to combine the teachings of HUYNH in view of MEHTA of having a method of detecting an object within an image, comprising: detecting an object within the projected thermal image through inputting the projected thermal image into a visible light object detector configured to detect objects within image data within the visible light color space, with the teachings of MOHR having wherein the visible light object detector includes a bidirectional feature pyramid network and an object detector head.
Wherein having HUYNH’s method wherein the visible light object detector includes a bidirectional feature pyramid network and an object detector head.
The motivation behind the modification would have been to obtain an enhanced method of image object detection that can reduce computational resources and perform optimizations and/or operations for object detection, since both HUYNH and MOHR relate to image processing devices and methods for object detection, wherein HUYNH discloses transformation of a feature vector in one domain to another is a one-to-one transformation which may save computational resources; using the techniques described herein traditional RGB-based image detectors (e.g., those trained in the RGB domain) may be used to detect objects when the input image data is in another domain (e.g., infrared) using the GAN-trained generators described herein, while MOHR has an object detection unit configured to perform a plurality of optimizations and/or operations for detecting the objects in the environment, such as the use of a weighted Bi-directional Feature Pyramid Network (BiFPN) for allowing easier and faster multi-scale feature combinations, and/or a compound scaling method for uniformly scaling the resolution, depth and width for all feature networks and box/class prediction networks at the same time. Please see HUYNH (US 11106903 B1), Col. 16-17, Lines [46-67, 1-3 respectively], and MOHR (US 20230306747 A1), Paragraph [0045].
- Claims 19 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over HUYNH (US 11106903 B1), hereinafter referenced as HUYNH in view of MEHTA (US 20180330474 A1), hereinafter referenced as MEHTA in further view of CHOI (US 20220080829 A1), hereinafter referenced as CHOI.
Regarding claim 19, HUYNH in view of MEHTA teach the thermal-augmented image object detection system of claim 11,
HUYNH in view of MEHTA fail to explicitly teach a vehicle having vehicle electronics that include the thermal-augmented image object detection system of claim 11.
However, CHOI explicitly teaches a vehicle having vehicle electronics (Fig. 2, Paragraph [0059] – CHOI discloses a vehicle 10 may include a user interface device 200, an object detection device 210, a communication device 220, a driving operation device 230, a main ECU 240, a driving control device 250, an autonomous device 260, a sensing unit 270, and a location data generation device 280)
that include the thermal-augmented image object detection system of claim 11 (Fig. 8, Paragraph [0207] – CHOI discloses the vehicle according to the present disclosure may include the first camera 510, the second camera 520, and the vehicle image processing device 600 [wherein the vehicle image processing device 600 is the thermal-augmented image object detection system]. Fig. 32, Paragraph [0311] – CHOI further discloses when an AR function is executed in S101, and it is checked whether an illumination is less than a predetermined value in S102, the vehicle image processing device 600 may run a thermal imaging (FIR) camera depending on the illumination and obtain an image in S105).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date the claimed invention was made to combine the teachings of HUYNH in view of MEHTA of having a thermal-augmented image object detection system, with the teachings of CHOI having a vehicle having vehicle electronics that include the thermal-augmented image object detection system.
Wherein HUYNH’s system wherein having a vehicle having vehicle electronics that include the thermal-augmented image object detection system.
The motivation behind the modification would have been to obtain an enhanced system for image object detection that can reduce computational resources and improve driver visibility, since both HUYNH and CHOI relate to image processing devices and methods for object detection, wherein HUYNH discloses transformation of a feature vector in one domain to another is a one-to-one transformation which may save computational resources; using the techniques described herein traditional RGB-based image detectors (e.g., those trained in the RGB domain) may be used to detect objects when the input image data is in another domain (e.g., infrared) using the GAN-trained generators described herein, while CHOI has a vehicle image processing method comprising generating augmented reality data based on the extracted target information so that a visual object corresponding to the target is displayed; the present disclosure has an effect of allowing passengers to get more conveniently necessary information from the screen by achieving the above-described objects and promoting safe driving and user convenience. Please see HUYNH (US 11106903 B1), Col. 16-17, Lines [46-67, 1-3 respectively], and CHOI (US 20220080829 A1), Paragraph [0022, 0367].
Regarding claim 20, HUYNH in view of MEHTA teach the thermal-augmented image object detection system of claim 11,
HUYNH in view of MEHTA fail to explicitly teach an advanced driver assistance system (ADAS) for a vehicle, comprising the thermal-augmented image object detection system of claim 11.
However, CHOI explicitly teaches an advanced driver assistance system (ADAS) (Fig. 2, Paragraph [0091] – CHOI discloses the autonomous device 260 can implement at least one advanced driver assistance system (ADAS) function) for a vehicle (Fig. 2, Paragraph [0059] – CHOI discloses a vehicle 10 may include… an autonomous communication device 260).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date the claimed invention was made to combine the teachings of HUYNH in view of MEHTA of having a thermal-augmented image object detection system, with the teachings of CHOI having an advanced driver assistance system (ADAS) for a vehicle.
Wherein HUYNH’s system wherein having an advanced driver assistance system (ADAS) for a vehicle.
The motivation behind the modification would have been to obtain an enhanced system for image object detection that can reduce computational resources and improve driver visibility, since both HUYNH and CHOI relate to image processing devices and methods for object detection, wherein HUYNH discloses transformation of a feature vector in one domain to another is a one-to-one transformation which may save computational resources; using the techniques described herein traditional RGB-based image detectors (e.g., those trained in the RGB domain) may be used to detect objects when the input image data is in another domain (e.g., infrared) using the GAN-trained generators described herein, while CHOI has a vehicle image processing method comprising generating augmented reality data based on the extracted target information so that a visual object corresponding to the target is displayed; the present disclosure has an effect of allowing passengers to get more conveniently necessary information from the screen by achieving the above-described objects and promoting safe driving and user convenience. Please see HUYNH (US 11106903 B1), Col. 16-17, Lines [46-67, 1-3 respectively], and CHOI (US 20220080829 A1), Paragraph [0022, 0367].
Conclusion
Listed below are the prior arts made of record and not relied upon but are considered pertinent to applicant’s disclosure.
Luan et al. (US 20220215528 A1) - An example three-dimensional (3D) printer may include a camera to capture a low-resolution thermal image of a build material bed. The 3D printer may include an interpolation engine to generate an interpolated thermal image based on the low-resolution thermal image. The 3D printer may also include a correction engine to enhance fine details of the interpolated thermal image without distorting thermal values from portions of the interpolated thermal image without fine details to produce an enhanced thermal image…. Fig. 3, Abstract.
Mojaver et al. (US 20220132052 A1) - A system for monitoring a human operator of critical equipment comprises an imaging module; a biometric measurement module; a risk detection module; and a risk response module. The imaging module includes a multi-spectral light source configured to emit light in a first spectral wavelength range for illuminating at least a portion of the human operator; a camera configured to detect light received from the human operator in a second spectral wavelength range; an imaging data generator configured to generate image data based on the emitted light and detected light. The biometric measurement module is configured to receive the image data; and based on the image data, perform at least one biometric measurement on the human operator. The risk detection module is configured to establish a safety risk associated with the human operator; and the risk response module is configured to based on the safety risk generate a risk response.…. Fig. 3, Abstract.
Paul et al. (US 20220094896 A1) - An example computing system comprises a processor and a storage device holding instructions executable by the processor to receive a thermal image acquired via a thermal imaging system, each pixel of the thermal image comprising an intensity level, and generate a histogram via binning pixels by intensity level. The instructions are further executable to, based at least on the histogram, determine a subset of pixels to colorize, colorize the subset of pixels to produce a selectively colorized image, and output the selectively colorized image...…. Fig. 1, Abstract.
Weng et al. (US 20200342275 A1) - The present disclosure provides a target-image acquisition method. The target-image acquisition method includes acquiring a visible-light image and an infrared (IR) image of a target, captured at a same time point by a photographing device; weighting and fusing the visible-light image and the IR image to obtain a fused image; and obtaining an image of the target according to the fused image. The present disclosure also provides a photographing device and an unmanned aerial vehicle (UAV) using the method above...…. Fig. 1, 2, Abstract.
Schulte et al. (US 20180300884 A1) - An infrared (IR) imaging module may capture a background image in response to receiving IR radiation from a background of a scene and determine background calibration terms using the background image. The determined background calibration terms may be scale factors and/or offsets that equalize the pixel values of the background image to a baseline, value. IR imaging device may use the background calibration terms to capture images that have the baseline value for pixels corresponding to IR radiation received from the background and higher values (or lower values) for pixels corresponding to IR radiation received from a foreground. Such images may be used to count people and generate a heat map. The background calibration terms may be updated periodically, with the update period being increased at least for some pixels or a pixel area when a person is detected. Fig. 1, Abstract.
Nguyen et al. (US 20130188058 A1) - Systems, methods, and devices for thermal detection. A thermal detection device includes a visual camera, a thermal detector, a controller, a user interface, a display, and a removable and rechargeable battery pack. The thermal detection device also includes a plurality of additional software and hardware modules configured to perform or execute various functions and operations of the thermal detection device. An output from the visual camera and an output from the thermal detector are combined by the controller or the plurality of additional modules to generate a combined image for display on the display... Fig. 6, 9, Abstract.
Wolff et al. (US 7620265 B1) - A methodology for forming a composite color image fusion from a set of N gray level images takes advantage of the natural decomposition of color spaces into 2-D chromaticity planes and 1-D intensity. This is applied to the color fusion of thermal infrared and reflective domain (e.g., visible) images whereby chromaticity representation of this fusion is invariant to changes in reflective illumination.... Fig. 1, Abstract.
Mankowski et al. (US 20130286236 A1) – Enhanced passthrough images are generated and displayed. A current visibility condition of an environment is determined. Based on the current visibility condition, a first camera or a second camera, which detect light spanning different ranges of illuminance, is selected to generate a passthrough image of the environment. The selected camera is then caused to generate the passthrough image. Additionally, a third camera, which is structured to detect long wave infrared radiation, is caused to generate a thermal image of the environment. Parallax correction is performed by aligning coordinates of the thermal image with corresponding coordinates identified within the passthrough image... Fig. 7, 12, Abstract
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to BEZAWIT N SHIMELES whose telephone number is (571)272-7663. The examiner can normally be reached M-F 7:30am-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Chineyere Wills-Burns can be reached at (571) 272-9752. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/BEZAWIT NOLAWI SHIMELES/Examiner, Art Unit 2673
/CHINEYERE WILLS-BURNS/Supervisory Patent Examiner, Art Unit 2673