DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on July 16th, 2026 has been entered.
Claim Status
Claims 1-20 were pending for examination in the amendments filed for Application No. 18/484,306 filed December 9th, 2025. In the remarks and amendments received on July 16th, 2026, claims 1-2, 5, 10-11, 14, and 19 are amended, no claims are cancelled, and no claims are added. Accordingly, claims 1-20 are currently pending for examination.
Response to Arguments
Applicant’s arguments filed July 16th, 2026, with respect to the rejection of claims 1, 2, and 5, have been fully considered but are moot because the arguments do not apply to the new combination of references, facilitated by Applicant’s newly submitted amendments being used in the current rejection.
Claim Objections
Claim 11 objected to because of the following informalities: "a corresponding IR image" should be "a corresponding successive IR image".
Appropriate correction is required.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claim(s) 1-3, 9-12, and 18 is/are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Hiskens et al. (US-20230064450-A1).
Regarding claim 1, Hiskens teaches:
A processor comprising:
one or more processing units (“The video streams are fed to processing hardware 130. The processing hardware 130 may be a single general-purpose processor,” Para [0013]) to:
generate a frame of infrared (IR) image data (an infrared image 126… are received from the respective cameras,” Para [0017]) and a frame of color image data (“an RGB image 128 are received from the respective cameras” Para [0017]) representing a common time slice (“a pair of concurrently captured infrared and RGB images,” Para [0014]) and having at least partially overlapping fields of view (“At step 170 a scene is captured by the infrared camera and the RGB camera,” Para [0022]);
and generate a frame of synthesized color image data colorizing the frame of IR image data using the frame of color image data (“the colorizing module 152 colors each matched feature (from the infrared image) in the initial output color image 150 based on the corresponding colors of the matching features from the RGB image 128,” Para [0021], where the “initial output color image” is “monochromatic” or “grayscale” (Para [0020]) and may not truly be “color[ed]”) based at least on a determination that one or more first classified regions of the frame of color image data (Fig. 3, “features” 144A , 144B, 144C) correspond to one or more second classified regions (Fig. 3, “features” 142A, 142B, 142C) of at least one of the frame of synthesized color image data or the frame of IR image data (Fig. 3, “features” from the color image data and the infrared data are matched, which then informs the colorization of the infrared image, see steps 146, 152, and 154).
PNG
media_image1.png
757
529
media_image1.png
Greyscale
Regarding claim 2, the rejection of claim 1 is incorporated herein. Hiskens teaches the processor of claim 1, and further teaches:
the one or more processing units further to colorize, for each successive frame of one or more frames of a video feed of a monitoring system (“When a pair of concurrently captured infrared and RGB images have been pre-processed,” Para [0014], and these images come from infrared and RGB videos, respectively), a corresponding successive frame of IR image data using a corresponding successive frame of color image data representing a corresponding common time slice as the successive frame of IR image data (“the embodiments described above are useful for providing video output that conveys a blend of infrared and RGB information to a user,” Para [0023]; that is, the processor colorizes each infrared image and outputs the resulting video).
Regarding claim 3, the rejection of claim 1 is incorporated herein. Hiskens teaches the processor of claim 1, and further teaches:
the one or more processing units further to transfer at least one of one or more colors or one or more color statistics of the one or more first classified regions of the frame of color image data to the one or more second classified regions of at least one of the frame of synthesized color image data or the frame of IR image data (“the colorizing module 152 colors each matched feature (from the infrared image) in the initial output color image 150 based on the corresponding colors of the matching features from the RGB image 128,” Para [0021]).
Regarding claim 9, the rejection of claim 1 is incorporated herein. Hiskens teaches the processor of claim 1, and further teaches:
wherein the processor is comprised in at least one of:
a control system for an autonomous or semi-autonomous machine;
a perception system for an autonomous or semi-autonomous machine (“a partially blended color image will provide the driver or autonomous controls of the vehicle the clarity of objects in the scene via the infrared camera 120 and the color of the currently lit stop light via the RGB camera 122,” Para [0016]);
a system for performing simulation operations;
a system for performing digital twin operations;
a system for performing light transport simulation; a system for performing collaborative content creation for 3D assets;
a system for performing deep learning operations;
a system for performing real-time streaming;
a system implemented using an edge device;
a system implemented using a robot;
a system for performing conversational Al operations;
a system for generating synthetic data;
a system incorporating one or more virtual machines (VMs);
a system implemented at least partially in a data center;
or a system implemented at least partially using cloud computing resources.
Examiner’s Note: Claim 9 as recited is treated as a “field of use” or “intended use” limitation and therefore carries no patentable weight although it has been examined in view of Hiskens. The processor as recited has been examined as evidenced in claim 1 above. With respect to the enumerated environments that said processor is “comprised in”, the specification as disclosed merely mentions these environments as preferred intended use environments without specific details that warrant said processor comprised in these environments resulted in a novel and non-obvious structural change to the processor. Reference to MPEP 2112.01 is also made for applicant’s attention.
Regarding claims 10-12 and 18, the rejections of claims 1-3 apply, mutatis mutandis, to claims 10-12 and 18.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 4 and 13 is/are rejected under 35 U.S.C. 103 as being unpatentable over Hiskens as applied to claims 1 and 10 above, and further in view of Qu et al., "Manga Colorization", ACM Transactions on Graphics (TOG), Volume 25, Issue 3, hereinafter referred to as "Qu".
Regarding claim 4, the rejection of claim 1 is incorporated herein. Hiskens teaches the processor of claim 1, but is not relied upon to teach the following limitations as further claimed. Qu, however, further teaches the one or more processing units further to colorize at least a portion of the IR image data (Qu, Fig. 1(d)) based at least on propagating one or more colors of the one or more classified regions of the frame of color image data (Hiskens, Fig. 3, colors from matching features between infrared and RGB images) as one or more seed colors to the one or more second classified regions of at least one of the frame of synthesized color image data or the frame of IR image data (Qu, Fig. 1(a), the lines indicating a color in a specific region of the image Fig. 1(a)), and using the one or more seed color as input designating one or more target regions to colorize (Qu, Fig 1(d), a result from the color input of Fig. 1(a) where segmented regions are fully colored).
PNG
media_image2.png
369
1084
media_image2.png
Greyscale
Qu is considered to be analogous to the claimed invention because they are in the same field of colorizing black and white or infrared images with color. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have incorporated the teachings of Qu into Hiskens for the benefit of complete colorization with a small amount of input.
Regarding claim 13, the rejection of claim 4 applies, mutatis mutandis, to claim 13.
Claim(s) 5 and 14 is/are rejected under 35 U.S.C. 103 as being unpatentable over Hiskens as applied to claims 1 and 10 above, and further in view of Kanazawa (US-20220301227-A1).
Regarding claim 5, the rejection of claim 1 is incorporated herein. Hiskens teaches the processor of claim 1, and further teaches:
the one or more processing units further to transfer at least one of one or more colors or one or more color statistics least one of the frame of synthesized color image data or the frame of IR image data (Fig. 3, “features” 142A, 142B, 142C are colored).
Hiskens is not relied upon to teach wherein these “classified regions” are “one or more segmented body parts of a plurality of segmented body parts”. Kanazawa however, further teaches “the machine learning model may be trained to perform part segmentation to annotate one or more parts of a person depicted in the input image and to colorize the input image” (Para [0033]), where this input image is “grayscale” (Para [0030]), where the “segmented body parts of a plurality of segmented body parts” are Kanazawa’s “one or more parts of a person depicted in the input image”.
Kanazawa is considered to be analogous to the claimed invention because they are both in the field of image colorization. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have incorporated the teachings of Kanazawa into Hiskens for the benefit of more accurate human colorization (when/if a human is in the scene being imaged).
Regarding claim 14, the rejection of claim 5 applies, mutatis mutandis, to claim 14.
Claim(s) 6-8 and 15-17 is/are rejected under 35 U.S.C. 103 as being unpatentable over Hiskens as applied to claims 1 and 10 above, and further in view of Li et al., "Automatic Example-Based Image Colorization Using Location-Aware Cross-Scale Matching," in IEEE Transactions on Image Processing, vol. 28, no. 9, pp. 4606-4619, hereinafter referred to as "Li".
Regarding claim 6, the rejection of claim 1 is incorporated herein. Hiskens teaches the processor of claim 1, but is not relied upon to teach the following limitations as further claimed. Li, however, teaches the one or more processing units further to transfer at least one of one or more colors or one or more color statistics of the one or more first classified regions of the frame of color image data (Fig. 4(b), the “reference image”) to the one or more second classified regions of the synthesized color image data (Figs. 4(d) and (e)) based at least on determining that a difference (“location violation”) between the one or more first classified regions of the frame of color image data (Fig. 4(b)) and the one or more second classified regions of the synthesized color image data (Fig. 4(e)) exceeds a threshold (“semantically incorrect colorization is related to up-down location violation (i.e. grass should not appear above the sky), and such location based “knowledge” can be automatically learnt from the single reference image,” pg. 4611, Section III, Part B).
PNG
media_image3.png
366
896
media_image3.png
Greyscale
Li is considered to be analogous to the claimed invention because they are in the same field of transferring color from an RGB image to an infrared or black and white image. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date to have incorporated the teachings of Li into Hiskens for the benefit of accurate colorized infrared or black and white images.
Regarding claim 7, the rejection of claim 1 is incorporated herein. Hiskens teaches the processor of claim 1, but is not relied upon to teach the following limitations as further claimed. Li, however, teaches the one or more processing units further to encode the frame of synthesized color image data (Fig. 4(d)) into a latent representation (transferring color to transform Fig. 4(a) into 4(d)), modify one or more dimensions of the latent representation to generate a modified latent representation (Fig. 4(g), the modified image of Fig. 4(d)), and decode the modified latent representation (Fig. 4(h), the changed colors are now shown with the coloring errors fixed).
It would have been obvious to one of ordinary skill in the art before the effective filing date to have incorporated the teachings of Li into Hiskens for the benefit of accurate colorized infrared or black and white images.
Regarding claim 8, the rejection of claim 1 is incorporated herein. Hiskens teaches the processor of claim 1, but is not relied upon to teach the following limitations as further claimed. Li, however, teaches the one or more processing units further to determine to modify one or more dimensions of a latent representation of the frame of synthesized color image data (Figs. 4(d) to (g), where coloring errors are fixed “from the single reference image”) based at least on determining that a difference (“location violation”) between one or more segmented regions of the frame of color image data (Fig. 4(b)) and one or more corresponding segmented regions of the frame of synthesized color image data (Fig. 4(e)) exceeds a threshold (“semantically incorrect colorization is related to up-down location violation (i.e. grass should not appear above the sky), and such location based “knowledge” can be automatically learnt from the single reference image,” pg. 4611, Section III, Part B).
It would have been obvious to one of ordinary skill in the art before the effective filing date to have incorporated the teachings of Li into Hiskens for the benefit of accurate colorized infrared or black and white images.
Regarding claims 15-17, the rejection of claims 6-8 applies, mutatis mutandis, to these claims.
Claim(s) 19-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Hiskens and further in view of Leinen (DE-102021123275-B3).
Regarding claim 19, Hiskens teaches the limitations of claim 19 as these limitations correspond to the limitations of claim 1, see the rejection above for specific citations. Hiskens is not relied upon to teach the following limitation. Leinen, however, further teaches:
a video feed of an occupant monitoring system (Leinen, “the invention comprises a method for providing color image data by which at least one object in a predetermined environment is displayed in color. The environment may in particular be the interior of a vehicle or the passenger compartment of a motor vehicle,” Para [0013]).
Leinen is considered to be analogous to the claimed invention because they are both in the field of infrared or black and white image colorization in the context of vehicles. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have incorporated the teachings of Leinen into Hiskens for the benefit of more accurately viewing occupants inside a vehicle.
Regarding claim 20, the rejection of claim 19 is incorporated herein. Hiskens teaches the method of claim 20, as evidenced by the prior art citations used in the rejection of claim 9.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Ungureanu et al. (US-20230055204-A1) teaches a method for colorizing or re-colorizing images.
Du et al. (US-20210366087-A1) teaches a method for colorizing an image that contains a human.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to RACHEL A OMETZ whose telephone number is (571)272-2535. The examiner can normally be reached 8:30am-5:30pm ET Monday-Thursday, 7:30am-3:30pm ET every other Friday.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Vu Le can be reached at 571-272-7332. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Rachel Anne Ometz/Examiner, Art Unit 2668 9/8/26
Rachel.ometz@uspto.gov
/VU LE/Supervisory Patent Examiner, Art Unit 2668
Prosecution Insights