Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 12/17/2024 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1-10,13-17 are rejected under 35 U.S.C. 103 as being unpatentable over Brodeur et al (US 8611586 B1, hereinafter Brodeur), Hall (US 9196056 B2) and Samples (US 10776695 B1).
Regarding claim 1, Brodeur teaches A method comprising: obtaining a readout of a thermal imaging sensor, wherein the thermal imaging sensor includes a plurality of different sets of pixels, and wherein each respective set of pixels captures a (Col 4 Line 35-38 “The present invention uses a combination of methods to determine a reliable global threshold for detecting and extracting a target from the background clutter of a set of frames taken by a thermal camera”, Col 8 Line 6 “FIG. 3 shows a histogram of an LWIR image”, Claim 12 “finding a first band of pixel values of a current image having temperature values above the ambient temperature value, the first band of pixel values forming a histogram; (c) smoothing the histogram to estimate a first threshold; (d) forming a second band of pixel values of the current image; (e) clustering the second band of pixel values into a target class and a background class; (f) computing a second threshold value between the target class and the background class; (g) extracting the target based on the first and second thresholds;”);
generating a number of images using the readout, wherein each one of the images reflects a same scene using a corresponding one of (Col 4 Line 43-47 “Based on the ambient conditions, the present invention segments the image into two sets: warm or hot objects and background. A thermal band of warm or hot objects is often different (e.g. brighter) than the surrounding background”, where segmenting an image results in multiple images, and is referring to a LWIR image.);
identifying, within the images, an object that is represented in at least some of the images and is represented across at least some of the (Col 5 Line 26-30 “The ambient temperature of background clutter is metadata that is used in conjunction with a series of image frames for target detection and extraction. Knowledge of ambient temperature of a scene helps to isolate the background clutter from other objects in the scene, such as human targets”, where isolating objects in the scene corresponds to identifying an object represented in the images);
using one or more of the images, which reflect (Col 9 Line 53-60 “An illustration of this is shown in FIGS. 5a, 5b, and 5c, where the distributions of the background clutter take on forms different from the distribution of human targets. Three histograms of an LWIR image are shown on a semi-log scale. FIG. 5a shows the pixel distribution of the sky (see the values between two vertical lines); FIG. 5b shows the pixel distribution of the background clutter (see the values between two vertical lines; and FIG. 5c shows the pixel distribution of human targets (see the values between two vertical lines)”); and
PNG
media_image1.png
331
326
media_image1.png
Greyscale
PNG
media_image2.png
341
370
media_image2.png
Greyscale
colorizing a different thermal image of the scene, wherein colorizing the different thermal image of the scene includes colorizing the object as represented within that scene based on the determined type for the object(Col 9 Line 1-5 “Human and other warm targets in a field-of-view (FOV) of an imaging camera may be colorized using a threshold value. Objects exhibiting temperatures higher than the threshold value may be colorized according to saturation and hue look-up-tables (LUTs) stored in a memory disposed in the camera”, Col 13 Line 35-38 “When processing in real-time, a number of values used in these calculations may be unknown until a full frame is processed. In such a situation, values used in the aforementioned equations may be taken from the previous frame.”).
In related field of endeavor, Hall teaches different ranges of wavelengths (Col 4 Line 67 – Col 5 Line 3 “The thermal detector 12 may operate in a spectral band of either 3-5 microns or 8-12 microns, each of which are known to effectively propagate through the atmosphere.” [0085] of applicant’s specification states “the apparatus 200 thus disperses long-wave infra-red (LWIR) light having wavelengths spanning from about 8 microns to about 14 microns into a plurality of sub-wavebands”)
It would have been obvious to one of ordinary skill in the art prior to the time of filing to have modified Brodeur to identify temperature by different ranges of wavelengths as taught by Hall. Doing so would allow the use of bands which are known to effectively propagate through the atmosphere (Col 5 Line 1-3 “a spectral band of either 3-5 microns or 8-12 microns, each of which are known to effectively propagate through the atmosphere”)
Brodeur and Hall fails to explicitly teach determining a type of the object by matching the identified LWIR profile with a previously saved LWIR profile established for objects of that type. In related field of endeavor, Samples teaches determining a type of the object by matching the identified LWIR profile with a previously saved LWIR profile established for objects of that type (Col 19 Line 11-24 “match aspects of the content stream to at least one learned target object profile from a database of learned target object profiles to detect target objects within the content, and upon a determination that at least one of the detected target objects corresponds to the at least one learned target object profile: classify the at least one detected target object based on the at least one learned target object profile, and update the at least one learned target object profile with at least one aspect of the at least one detected target object.”)
It would have been obvious to one of ordinary skill in the art prior to the time of filing to have modified Brodeur to include determining a type of the object by matching the identified LWIR profile with a previously saved LWIR profile established for objects of that type as taught by Samples. Doing so would provide a way of classifying detected objects (Col 3 Line 30-35 “and upon a determination that at least one of the detected target objects corresponds to the at least one learned target object profile: classify the at least one detected target object based on the at least one learned target object profile”)
Regarding claim 2, Brodeur as modified by Hall and Samples teach the method of claim 1. Samples further teaches wherein a machine learning (ML) engine generated the previously saved LWIR profile (Col 19 Line 67 – Col 20 Line 3“the target object designations being associated with a plurality of learned target object profiles trained by the AI engine”).
It would have been obvious to one of ordinary skill in the art prior to the time of filing to have further modified Brodeur as modified by Hall and Samples to include wherein a machine learning (ML) engine generated the previously saved LWIR profile. Doing so would provide a way of classifying detected objects (Col 3 Line 30-35 “and upon a determination that at least one of the detected target objects corresponds to the at least one learned target object profile: classify the at least one detected target object based on the at least one learned target object profile”)
Regarding claim 3, Brodeur as modified by Hall and Samples teach the method of claim 2. Samples further teaches wherein the ML engine is tuned based on an environment classification of the scene (Col 21 Line 42-54 “The training content may be selected to be the same or similar to what AI engine 100 is likely to find during recognition stage 090. For example, if a user 005 elects to train AI engine 100 to detect a deer, training content will consist of pictures of deer. Accordingly, training content may be curated for the specific training user 005 desires to achieve. In some embodiments, AI engine 100 may filter the content to remove any unwanted objects or artifacts, or otherwise enhance quality, whether still or in motion, in order to better detect the target objects selected by user 005 for training. b. The training content may contain images in different conditions, such as, but not limited to:”, Col 21 Line 66- Col 22 Line 7 “AI engine 100 may encounter different weather conditions that must be accounted for, such as, but not limited to: 1. Foggy (FIG. 19); 2. Rainy; 3. Snowy; 4. Day (FIG. 17); 5. Night (FIG. 19-21); 6. Indoor; and 7. Outdoor (FIG. 17-21)”).
It would have been obvious to one of ordinary skill in the art prior to the time of filing to have further modified Brodeur as modified by Hall and Samples to include wherein the ML engine is tuned based on an environment classification of the scene. Doing so would allow the ML engine to be trained based on what the engine is likely to find during recognition (Col 21 Line 42-44 “The training content may be selected to be the same or similar to what AI engine 100 is likely to find during recognition stage 090”)
Regarding claim 4, Brodeur as modified by Hall and Samples teach the method of claim 3. Samples further teaches wherein the environment classification is one of an urban environment or a rural environment (Samples FIG. 17 depicts a rural environment, Col 21 Line 66- Col 22 Line 7 “AI engine 100 may encounter different weather conditions that must be accounted for, such as, but not limited to: 1. Foggy (FIG. 19); 2. Rainy; 3. Snowy; 4. Day (FIG. 17); 5. Night (FIG. 19-21); 6. Indoor; and 7. Outdoor (FIG. 17-21)”).
PNG
media_image3.png
319
412
media_image3.png
Greyscale
It would have been obvious to one of ordinary skill in the art prior to the time of filing to have further modified Brodeur as modified by Hall and Samples to include wherein the environment classification is one of an urban environment or a rural environment as taught by Samples. Doing so would allow the ML engine to be trained based on what the engine is likely to find during recognition (Col 21 Line 42-44 “The training content may be selected to be the same or similar to what AI engine 100 is likely to find during recognition stage 090”)
Regarding claim 5, Brodeur as modified by Hall and Samples teach the method of claim 3. Samples further teaches wherein the environment classification is an outdoor environment (Col 21 Line 66- Col 22 Line 7 “AI engine 100 may encounter different weather conditions that must be accounted for, such as, but not limited to: 1. Foggy (FIG. 19); 2. Rainy; 3. Snowy; 4. Day (FIG. 17); 5. Night (FIG. 19-21); 6. Indoor; and 7. Outdoor (FIG. 17-21)”).
It would have been obvious to one of ordinary skill in the art prior to the time of filing to have further modified Brodeur as modified by Hall and Samples to include wherein the environment classification is an outdoor environment as taught by Samples. Doing so would allow the ML engine to be trained based on what the engine is likely to find during recognition (Col 21 Line 42-44 “The training content may be selected to be the same or similar to what AI engine 100 is likely to find during recognition stage 090”)
Regarding claim 6, Brodeur as modified by Hall and Samples teach the method of claim 1. Hall further teaches wherein the different thermal image is generated by a different thermal imaging sensor (Col 2 Line 66 - Col 3 Line 1 “ the additional detector is an infrared detector or a visible light detector. In addition, the thermal detector may obtain thermal images of the scene in real time”)
It would have been obvious to one of ordinary skill in the art prior to the time of filing to have further modified Brodeur as modified by Hall and Samples to include wherein the different thermal image is generated by a different thermal imaging sensor as taught by Hall. Doing so would provide an additional thermal sensor utilized when temperature exceeds a threshold value (Col 3 Line 4-7 “Optionally, the controller may be configured to activate the additional detector to obtain the additional image when portions of the scene having a temperature exceeding a threshold value are identified by processing the thermal image.”)
Regarding claim 7, Brodeur as modified by Hall and Samples teach the method of claim 1. Brodeur further teaches wherein the identified LWIR profile is identified by generating a histogram for the object using the images (Col 2 Line 2-4 “finding a band of pixel values having temperature values above the ambient temperature value, the band of pixel values forming a histogram;”, Col 9 Line 53-61 “An illustration of this is shown in FIGS. 5a, 5b, and 5c, where the distributions of the background clutter take on forms different from the distribution of human targets. Three histograms of an LWIR image are shown on a semi-log scale. FIG. 5a shows the pixel distribution of the sky (see the values between two vertical lines); FIG. 5b shows the pixel distribution of the background clutter (see the values between two vertical lines; and FIG. 5c shows the pixel distribution of human targets (see the values between two vertical lines)”).
PNG
media_image4.png
347
376
media_image4.png
Greyscale
Regarding claim 8, Brodeur as modified by Hall and Samples teach the method of claim 1.
Brodeur further teaches wherein the number of images that are generated does not equal a number of the different sets of pixels. (Claim 12 “finding a first band of pixel values of a current image having temperature values above the ambient temperature value, the first band of pixel values forming a histogram; (c) smoothing the histogram to estimate a first threshold; (d) forming a second band of pixel values of the current image; (e) clustering the second band of pixel values into a target class and a background class; (f) computing a second threshold value between the target class and the background class; (g) extracting the target based on the first and second thresholds;”, Claim 13: “The method of claim 12 wherein step (g) includes combining the first and second thresholds, using a weighted average, to form a third threshold, and extracting the target using the third threshold”, Col 4 Line 43-47 “Based on the ambient conditions, the present invention segments the image into two sets: warm or hot objects and background. A thermal band of warm or hot objects is often different (e.g. brighter) than the surrounding background”)
Regarding claim 9, Brodeur as modified by Hall and Samples teach the method of claim 1. Brodeur further teaches wherein the LWIR profile that is identified for the object using the one or more images is based on a number of images that is equal to a number of different sets of pixels included in the thermal imaging sensor (Claim 2 “forming a first subset of pixel values of a current image defining a window of pixel values having a predetermined temperature variation”, Claim 4 “and selecting a second subset of pixel values of a current image having values above the minimum threshold value”, Col 4 Line 43-45 “Based on the ambient conditions, the present invention segments the image into two sets: warm or hot objects and background”, where there are two subsets of pixel values, and the image is segmented into two sets).
Regarding claim 10, Brodeur as modified by Hall and Samples teach the method of claim 1. Hall further teaches wherein a pixel pitch of pixels in the different sets of pixels is at least 8 microns (Col 4 Line 67 – Col 5 Line 2 “The thermal detector 12 may operate in a spectral band of either 3-5 microns or 8-12 microns.”)
It would have been obvious to one of ordinary skill in the art to have further modified Brodeur as modified by Hall and Samples to include wherein a pixel pitch of pixels in the different sets of pixels is at least 8 microns as taught by Hall. Doing so would cause the sensor to operate on a band known to propagate through the atmosphere (Col 5 Line 2-3 “each of which are known to effectively propagate through the atmosphere”)
Regarding claim 13, Brodeur teaches A computer system comprising: a processor system; and a storage system that stores instructions that are executable by the processor system to cause the computer system to (Col 9 Line 1-12 “Human and other warm targets in a field-of-view (FOV) of an imaging camera may be colorized using a threshold value. Objects exhibiting temperatures higher than the threshold value may be colorized according to saturation and hue look-up-tables (LUTs) stored in a memory disposed in the camera. A microprocessor or a microcontroller, embedded in the camera, may be used to execute, in parallel, an algorithm, as shown in FIG. 1, for determining the ambient temperature of the image, and other algorithms for determining a target threshold, as described below. The manner in which the present invention selects a first target threshold and, separately, selects a second target threshold is described below.”):
obtain a readout of a thermal imaging sensor, wherein the thermal imaging sensor includes a plurality of different sets of pixels, and wherein each respective set of pixels captures a different corresponding range of wavelengths of long-wave infra-red (LWIR) light (Col 4 Line 35-38 “The present invention uses a combination of methods to determine a reliable global threshold for detecting and extracting a target from the background clutter of a set of frames taken by a thermal camera”, Claim 12 “finding a first band of pixel values of a current image having temperature values above the ambient temperature value, the first band of pixel values forming a histogram; (c) smoothing the histogram to estimate a first threshold; (d) forming a second band of pixel values of the current image; (e) clustering the second band of pixel values into a target class and a background class; (f) computing a second threshold value between the target class and the background class; (g) extracting the target based on the first and second thresholds;”);
generate a number of images using the readout, wherein each one of the images reflects a same scene using a corresponding one of the different ranges of wavelengths of the LWIR light (Col 4 Line 43-47 “Based on the ambient conditions, the present invention segments the image into two sets: warm or hot objects and background. A thermal band of warm or hot objects is often different (e.g. brighter) than the surrounding background”, where segmenting an image results in multiple images.);
identify, within the images, an object that is represented in at least some of the images and is represented across at least some of the different ranges of wavelengths (Col 5 Line 26-30 “The ambient temperature of background clutter is metadata that is used in conjunction with a series of image frames for target detection and extraction. Knowledge of ambient temperature of a scene helps to isolate the background clutter from other objects in the scene, such as human targets”, where isolating objects in the scene corresponds to identifying an object represented in the images);
use the images, which reflect the different ranges of wavelengths, to identify an LWIR profile for the object (Col 9 Line 53-60 “An illustration of this is shown in FIGS. 5a, 5b, and 5c, where the distributions of the background clutter take on forms different from the distribution of human targets. Three histograms of an LWIR image are shown on a semi-log scale. FIG. 5a shows the pixel distribution of the sky (see the values between two vertical lines); FIG. 5b shows the pixel distribution of the background clutter (see the values between two vertical lines; and FIG. 5c shows the pixel distribution of human targets (see the values between two vertical lines)”);
and colorize a different thermal image of the scene, wherein colorizing the different thermal image of the scene includes colorizing the object as represented within that scene based on the determined type for the object (Col 9 Line 1-5 “Human and other warm targets in a field-of-view (FOV) of an imaging camera may be colorized using a threshold value. Objects exhibiting temperatures higher than the threshold value may be colorized according to saturation and hue look-up-tables (LUTs) stored in a memory disposed in the camera”, Col 13 Line 35-38 “When processing in real-time, a number of values used in these calculations may be unknown until a full frame is processed. In such a situation, values used in the aforementioned equations may be taken from the previous frame.”).
In related field of endeavor, Hall teaches different ranges of wavelengths (Col 4 Line 67 – Col 5 Line 3 “The thermal detector 12 may operate in a spectral band of either 3-5 microns or 8-12 microns, each of which are known to effectively propagate through the atmosphere.” [0085] of applicant’s specification states “the apparatus 200 thus disperses long-wave infra-red (LWIR) light having wavelengths spanning from about 8 microns to about 14 microns into a plurality of sub-wavebands”)
It would have been obvious to one of ordinary skill in the art prior to the time of filing to have modified Brodeur to identify temperature by different ranges of wavelengths as taught by Hall. Doing so would allow the use of bands which are known to effectively propagate through the atmosphere (Col 5 Line 1-3 “a spectral band of either 3-5 microns or 8-12 microns, each of which are known to effectively propagate through the atmosphere”)
Brodeur and Hall fail to explicitly teach determine a type of the object by matching the identified LWIR profile with a previously saved LWIR profile established for objects of that type. In related field of endeavor, Samples teaches determine a type of the object by matching the identified LWIR profile with a previously saved LWIR profile established for objects of that type (Col 19 Line 11-24 “match aspects of the content stream to at least one learned target object profile from a database of learned target object profiles to detect target objects within the content, and upon a determination that at least one of the detected target objects corresponds to the at least one learned target object profile: classify the at least one detected target object based on the at least one learned target object profile, and update the at least one learned target object profile with at least one aspect of the at least one detected target object.”)
It would have been obvious to one of ordinary skill in the art prior to the time of filing to have further modified Brodeur and Hall to include determine a type of the object by matching the identified LWIR profile with a previously saved LWIR profile established for objects of that type. Doing so would provide a way of classifying detected objects (Col 3 Line 30-35 “upon a determination that at least one of the detected target objects corresponds to the at least one learned target object profile: classify the at least one detected target object based on the at least one learned target object profile”)
Regarding claim 14, Brodeur as modified by Hall and Samples teach the computer system of claim 13. Brodeur further teaches wherein the number of images is an even number (Col 5 Line 18-20 “The global dynamic threshold yields segmentation of a target of interest, such as a human being, from the background clutter in the image”, where segmentation results in 2 images, one of a human being and one of background clutter)
Regarding claim 15, Brodeur as modified by Hall and Samples teach the computer system of claim 13. Samples further teaches wherein a machine learning (ML) engine determines the type of the object based on an environment classification the ML engine has with respect to the scene (Col 14 Line 28-33 “The target object classifications may be based on, for example, but not limited to, the zone with which the content is associated. Associating content with a zone, and defining target objects to be tracked within a zone, will be detailed with reference to FIGS. 6 and 7, FIG. 11, FIG. 12, and FIG. 13.”).
It would have been obvious to one of ordinary skill to have further modified Brodeur as modified by Hall and Samples to include wherein a machine learning (ML) engine determines the type of the object based on an environment classification the ML engine has with respect to the scene as taught by Samples. Doing so would allow the ML engine to be trained based on what the engine is likely to find during recognition (Col 21 Line 42-44 “The training content may be selected to be the same or similar to what AI engine 100 is likely to find during recognition stage 090”)
Regarding claim 16, Brodeur as modified by Hall and Samples teach the computer system of claim 13. Hall further teaches wherein the computer system includes a second thermal imaging sensor, the second thermal imaging sensor generating the different thermal image (Col 2 Line 66 - Col 3 Line 1 “the additional detector is an infrared detector or a visible light detector. In addition, the thermal detector may obtain thermal images of the scene in real time”)
It would have been obvious to one of ordinary skill in the art prior to the time of filing to have further modified Brodeur as modified by Hall and Samples to include wherein the computer system includes a second thermal imaging sensor, the second thermal imaging sensor generating the different thermal image as taught by Hall. Doing so would provide an additional thermal sensor utilized when temperature exceeds a threshold value (Col 3 Line 4-7 “Optionally, the controller may be configured to activate the additional detector to obtain the additional image when portions of the scene having a temperature exceeding a threshold value are identified by processing the thermal image.”)
Regarding claim 17, Brodeur as modified by Hall and Samples teach the computer system of claim 13. Brodeur further teaches wherein the number of different sets of pixels is a minimum of 4 sets (Claim 12: “ method for extracting a target from a series of images comprising the steps of: (a) estimating an ambient temperature value of pixels in the series of images; (b) finding a first band of pixel values of a current image having temperature values above the ambient temperature value, the first band of pixel values forming a histogram; (c) smoothing the histogram to estimate a first threshold; (d) forming a second band of pixel values of the current image; (e) clustering the second band of pixel values into a target class and a background class; (f) computing a second threshold value between the target class and the background class; (g) extracting the target based on the first and second thresholds”, where first band corresponds to a first set of pixels, second band corresponds to second set of pixels, clustering the second band into two classes corresponds to a third and fourth set of pixels, Claim 13: “The method of claim 12 wherein step (g) includes combining the first and second thresholds, using a weighted average, to form a third threshold, and extracting the target using the third threshold”, where a third threshold results in another set of pixels.)
Claims 11 and 12 are rejected under 35 U.S.C. 103 as being unpatentable over Brodeur as modified by Hall and Samples as applied to claim 1 above, and further in view of Tantalo et al (US 8755597 B1, hereinafter Tantalo).
Regarding claim 11, Brodeur as modified by Hall and Samples teach the method of claim 1, but fail to explicitly teach wherein colorizing the object includes assigning a single color to the object. In related field of endeavor, Tantalo teaches wherein colorizing the object includes assigning a single color to the object (Col 6 Line 6-11 “One approach to color mapping a scene is to assign the pixels of an image one of two colors, depending on whether the objects are cold (e.g., blue, green, etc.) or hot (e.g., red, amber, etc.). Alternatively, pixels that are cold may be assigned an identical color to those that are ambient, in order to minimize colorization of cool night skies”).
It would have been obvious to one of ordinary skill in the art prior to the time of filing to have further modified Brodeur as modified by Hall and Samples to include wherein colorizing the object includes assigning a single color to the object. Doing so would ensure the hue of objects is the same for all pixels (Col 13 Line 46-49 “This lightening process ensures that the hue provided by the addition of color is the same for all pixels, regardless of the final tonal character of the fused image.”)
Regarding claim 12, Brodeur as modified by Hall and Samples teach the method of claim 1. Brodeur further teaches wherein a second object is included in the scene, wherein the second object is of the same type as said object (Col 8 Line 6-7 “FIG. 3 shows a histogram of an LWIR image, representing five humans walking out from dense trees”), but Brodeur as modified by Hall and Samples fail to explicitly teach and wherein the second object is colorized using a same color as said object in the different thermal image. In related field of endeavor, Tantalo teaches the second object is colorized using a same color as said object in the different thermal image (Col 6 Line 6-11 “One approach to color mapping a scene is to assign the pixels of an image one of two colors, depending on whether the objects are cold (e.g., blue, green, etc.) or hot (e.g., red, amber, etc.). Alternatively, pixels that are cold may be assigned an identical color to those that are ambient, in order to minimize colorization of cool night skies”, where all warm objects are assigned the hot color, and all cold or ambient objects are assigned the cold color).
It would have been obvious to one of ordinary skill in the art prior to the time of filing to have further modified Brodeur as modified by Hall and Samples to include wherein the second object is colorized using a same color as said object in the different thermal image as taught by Tantalo. Doing so would minimize coloration of cool night skies (Col 6 Line 10-11 “ in order to minimize colorization of cool night skies”)
Claims 18-20 are rejected under 35 U.S.C. 103 as being unpatentable over Brodeur as modified by Paul et al (US 11375166 B2, hereinafter Paul), Hall and Samples.
Regarding claim 18, Brodeur teaches a processor system; and a storage system that stores instructions that are executable by the processor system to cause the ER system to (Col 9 Line 1-12 “Human and other warm targets in a field-of-view (FOV) of an imaging camera may be colorized using a threshold value. Objects exhibiting temperatures higher than the threshold value may be colorized according to saturation and hue look-up-tables (LUTs) stored in a memory disposed in the camera. A microprocessor or a microcontroller, embedded in the camera, may be used to execute, in parallel, an algorithm, as shown in FIG. 1, for determining the ambient temperature of the image, and other algorithms for determining a target threshold, as described below. The manner in which the present invention selects a first target threshold and, separately, selects a second target threshold is described below.”):
obtain a readout of a thermal imaging sensor, wherein the thermal imaging sensor includes a plurality of different sets of pixels, and wherein each respective set of pixels captures a (Col 4 Line 35-38 “The present invention uses a combination of methods to determine a reliable global threshold for detecting and extracting a target from the background clutter of a set of frames taken by a thermal camera”, Claim 12 “finding a first band of pixel values of a current image having temperature values above the ambient temperature value, the first band of pixel values forming a histogram; (c) smoothing the histogram to estimate a first threshold; (d) forming a second band of pixel values of the current image;”);
generate a number of images using the readout, wherein the number of images that are generated equals a number of the different sets of pixels, and wherein each one of the images reflects a same scene using a corresponding one of the(Col 4 Line 43-47 “Based on the ambient conditions, the present invention segments the image into two sets: warm or hot objects and background. A thermal band of warm or hot objects is often different (e.g. brighter) than the surrounding background”, where segmenting an image results in multiple images.);
identify, within the images, an object that is represented in at least some of the images and is represented across at least some of the (Col 5 Line 26-30 “The ambient temperature of background clutter is metadata that is used in conjunction with a series of image frames for target detection and extraction. Knowledge of ambient temperature of a scene helps to isolate the background clutter from other objects in the scene, such as human targets”, where isolating objects in the scene corresponds to identifying an object represented in the images);
use the images, which reflect the (Col 9 Line 53-60 “An illustration of this is shown in FIGS. 5a, 5b, and 5c, where the distributions of the background clutter take on forms different from the distribution of human targets. Three histograms of an LWIR image are shown on a semi-log scale. FIG. 5a shows the pixel distribution of the sky (see the values between two vertical lines); FIG. 5b shows the pixel distribution of the background clutter (see the values between two vertical lines; and FIG. 5c shows the pixel distribution of human targets (see the values between two vertical lines)”);
PNG
media_image1.png
331
326
media_image1.png
Greyscale
PNG
media_image2.png
341
370
media_image2.png
Greyscale
colorize a different thermal image of the scene, wherein colorizing the different thermal image of the scene includes colorizing the object as represented within that scene based on the determined type for the object (Col 9 Line 1-5 “Human and other warm targets in a field-of-view (FOV) of an imaging camera may be colorized using a threshold value. Objects exhibiting temperatures higher than the threshold value may be colorized according to saturation and hue look-up-tables (LUTs) stored in a memory disposed in the camera”, Col 13 Line 35-38 “When processing in real-time, a number of values used in these calculations may be unknown until a full frame is processed. In such a situation, values used in the aforementioned equations may be taken from the previous frame.”).
Brodeur fails to explicitly teach different ranges of wavelengths. In related field of endeavor, Hall teaches different ranges of wavelengths (Col 4 Line 67 – Col 5 Line 3 “The thermal detector 12 may operate in a spectral band of either 3-5 microns or 8-12 microns, each of which are known to effectively propagate through the atmosphere.” [0085] of applicant’s specification states “the apparatus 200 thus disperses long-wave infra-red (LWIR) light having wavelengths spanning from about 8 microns to about 14 microns into a plurality of sub-wavebands”)
It would have been obvious to one of ordinary skill in the art prior to the time of filing to have modified Brodeur to identify temperature by different ranges of wavelengths as taught by Hall. Doing so would allow the use of bands which are known to effectively propagate through the atmosphere (Col 5 Line 1-3 “a spectral band of either 3-5 microns or 8-12 microns, each of which are known to effectively propagate through the atmosphere”)
Brodeur as modified by Hall fails to explicitly teach an extended reality (ER) system. In related field of endeavor, Paul teaches an extended reality system (Col 3 Line 6-19 “an example thermal imaging display system in the form of a head-mounted display (HMD) device 100 is described with reference to FIG. 1. HMD device 100 comprises a thermal imaging camera 102 and low-light imaging camera 104. HMD device 100 further comprises processing circuitry 112 configured to process images from cameras 102, 104, and to display processed images on near-eye displays 108a, 108b. HMD device 100 may capture real-time video from thermal imaging camera 102 and/or low-light imaging camera 104 output real-time video to near-eye displays 108a, 108b. Visor 106 and near-eye displays 108a, 108b may be at least partially transparent to allow a user to view real-world objects.”)
It would have been obvious to one of ordinary skill in the art prior to the time of filing to have modified Brodeur to include an extended reality (ER) system as taught by Paul. Doing so would provide an ER system for use which allows the user to view real-world objects while using the thermal imaging system (Col 3 Line 17-19 “Visor 106 and near-eye displays 108a, 108b may be at least partially transparent to allow a user to view real-world objects.”)
Brodeur as modified by Paul and Hall fails to explicitly teach determine a type of the object by matching the identified LWIR profile with a previously saved LWIR profile established for objects of that type. In related field of endeavor, Samples teaches determine a type of the object by matching the identified LWIR profile with a previously saved LWIR profile established for objects of that type (Col 19 Line 11-24 “match aspects of the content stream to at least one learned target object profile from a database of learned target object profiles to detect target objects within the content, and upon a determination that at least one of the detected target objects corresponds to the at least one learned target object profile: classify the at least one detected target object based on the at least one learned target object profile, and update the at least one learned target object profile with at least one aspect of the at least one detected target object.”)
It would have been obvious to one of ordinary skill in the art prior to the time of filing to have further modified Brodeur and Paul to include determine a type of the object by matching the identified LWIR profile with a previously saved LWIR profile established for objects of that type as taught by Samples. Doing so would provide a way of classifying detected objects (Col 3 Line 30-35 “and upon a determination that at least one of the detected target objects corresponds to the at least one learned target object profile: classify the at least one detected target object based on the at least one learned target object profile”)
Regarding claim 19, Brodeur as modified by Paul, Hall and Samples teach the ER system of claim 18. Brodeur further teaches wherein the number of images is at least 2 (Col 4 Line 43-47 “Based on the ambient conditions, the present invention segments the image into two sets: warm or hot objects and background. A thermal band of warm or hot objects is often different (e.g. brighter) than the surrounding background”, where segmenting an image results in multiple images)
Regarding claim 20, Brodeur as modified by Paul, Hall and Samples teach the ER system of claim 18. Samples further teaches wherein a machine learning (ML) engine determines the type of the object based on an environment classification the ML engine has with respect to the scene (Col 19 Line 67 – Col 20 Line 3 “the target object designations being associated with a plurality of learned target object profiles trained by the AI engine”, Col 14 Line 19-25 “AI engine 100 may receive or retrieve data from content module 055 during an input stage. The content 085 may then be processed in accordance to target object classifications associated with the content. The target object classifications may be based on, for example, but not limited to, the zone with which the content is associated.”, Col 26 Line 43-52 “zones may be grouped by, but not be limited to: Living Room Outdoor Sector 1 Indoor Sector 1 Backyard Driveway Office Building Shed Grand Canyon”, where zone corresponds to an environment classification).
It would have been obvious to one of ordinary skill in the art prior to the time of filing to have further modified Brodeur as modified by Paul, Hall and Samples to include wherein a machine learning (ML) engine determines the type of the object based on an environment classification the ML engine has with respect to the scene as taught by Samples. Doing so would provide a way of classifying detected objects (Col 3 Line 30-35 “and upon a determination that at least one of the detected target objects corresponds to the at least one learned target object profile: classify the at least one detected target object based on the at least one learned target object profile”).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JOHN PATRICK GOCO whose telephone number is (571)272-5872. The examiner can normally be reached M-Th, 7:00 am - 5:00 pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kee Tung can be reached at (571)272-7794. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/KEE M TUNG/Supervisory Patent Examiner, Art Unit 2611
/J.P.G./ Examiner, Art Unit 2611
Prosecution Insights