DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on May 14th, 2026 has been entered.
Claim Status
Claims 1-20 were pending for examination in the amendments for Application No. 18/484,122, filed December 9th, 2025. In the remarks and amendments received on May 14th, 2026, claims 1, 3-5, 9, and 12-18 are amended, no claims are cancelled, and no claims are added. Accordingly, claims 1-20 are currently pending for examination in the application.
Response to Amendment
Applicant’s amendments filed May 14th, 2026, have overcome the objections previously set forth in the Final Rejection Office Action mailed January 20th, 2026. Accordingly, the objections are withdrawn.
Response to Arguments
Applicant’s arguments, see pgs. 9-10, filed May 14th, 2026, with respect to the rejection(s) of claim(s) 1, 11, and 19 have been fully considered and are persuasive. Therefore, the rejection has been withdrawn. However, upon further consideration, a new ground(s) of rejection is made in view of the teachings of Xue, see the full rejection below.
Applicant’s arguments, see pgs. 10-12, filed May 14th, 2026, with respect to claims have been fully considered and are persuasive. The rejection regarding claims 2 and 12 has been withdrawn.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1, 3, 5, and 10 is/are rejected under 35 U.S.C. 103 as being unpatentable over Li et al., "I2V-GAN: Unpaired Infrared-to-Visible Video Translation", arXiv:2108.00913, 2021, hereinafter referred to as Li, and further in view of Klomp et al. (DE-102018216806-A1) and Xue et al., “Exemplar-Based Image Colorization with A Learning Framework”, arXiv:2209.05775v1, hereinafter referred to as Xue.
Regarding claim 1, Li teaches a processor comprising:
one or more processing units to: generate infrared image data (“an infrared-to-visible (I2V) video translation method,” using a GAN network i.e., processing units, see Abstract)
generate, based at least on applying a representation of the infrared image data to a generator of a generative adversarial network (Fig. 1, “I2V-GAN”), RGB image data corresponding to the infrared image data (Fig. 1, “translate videos from the source domain X to the target domain Y”, where X is the infrared domain and Y is the RGB/visible light domain).
PNG
media_image1.png
530
589
media_image1.png
Greyscale
Li fails to teach the following limitations as further claimed. However, Klomp further teaches generat[ing] infrared image data representing an interior space (“taking infrared images of a user, in particular a driver of a motor vehicle,” Para [0017]) of an ego-machine (“main use case for such a camera is in automated driving, where it checks whether the driver can resume the driving task,” Para [0003]).
Li and Klomp are not relied upon to teach the following limitations. Xue, however, further teaches:
identify one or more segmented regions of the synthesized color image data (“We then introduce the CRF (Conditional Random Fields) [43], which is a good structure to describe probabilities in a graph, to further optimize the color transferring image and remove those possible wrong matchings in terms of the spatial consistency,” pg. 5, Section “C. Colorization from Reference Image”; also see Fig. 1, “Color transferring” to “Post-processing”); and
PNG
media_image2.png
619
940
media_image2.png
Greyscale
modify a fill for the one or more segmented regions of the synthesized color image data (pg. 6, Fig. 4a and 4b, the post-processing operation modifies a fill/a color for the flower).
PNG
media_image3.png
291
455
media_image3.png
Greyscale
Klomp is considered to be analogous to the claimed invention because they are both in the same field of colorizing infrared images representing cabin spaces of vehicles. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have incorporated the teachings of Klomp into Li for the benefit of better detecting the driver in a vehicle regardless of the lighting inside or outside of the vehicle.
Xue is considered to be analogous to the claimed invention because they are both in the field of colorizing black-and-white/infrared images. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have incorporated the teachings of Xue into Li and Klomp for the benefit of more realistic and refined final colorized output images.
Regarding claim 3, the rejection of claim 1 is incorporated herein. Li in view of Klomp and Xue teach the processor of claim 1, and Xue further teaches:
the one or more processing units further to modify the fill for the one or more segmented regions of the synthesized color image data based at least on an energy function that encourages spatial color continuity (“We then introduce the CRF (Conditional Random Fields) [43], which is a good structure to describe probabilities in a graph, to further optimize the color transferring image and remove those possible wrong matchings in terms of the spatial consistency,” pg. 5, Section “C. Colorization from Reference Image”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have incorporated the teachings of Xue into Li and Klomp for the benefit of more realistic and refined final colorized output images.
Regarding claim 5, Li in view of Klomp and Xue teaches the processor of claim 1, and Li further teaches the one or more processing units further to train the generative adversarial network (Fig. 3, “I2V-GAN network architecture”) based at least on training a first branch that chains the generator (Fig. 3, “GY”) of the synthesized color image data (Fig. 3, “ȳt+1”) followed by a generator of infrared image data (Fig. 3,”GX”), and a second branch that chains the generator of the infrared image data followed by the generator of the synthesized color image data (Fig. 3 caption, “The opposite direction Y
→
X is similar”, I.E. the positions of the generators can be swapped).
Regarding claim 10, Li in view of Klomp and Xue teach the processor of claim 1, and Li further teaches wherein the processor is comprised in at least one of:
a control system for an autonomous or semi-autonomous machine;
a perception system for an autonomous or semi-autonomous machine;
a system for performing simulation operations;
a system for performing digital twin operations;
a system for performing light transport simulation;
a system for performing collaborative content creation for 3D assets;
a system for performing deep learning operations (Fig. 8, the flower life cycle in images is created using I2V-GAN, where GANs are a type of deep learning architecture);
PNG
media_image4.png
591
602
media_image4.png
Greyscale
a system for performing real-time streaming;
a system implemented using an edge device;
a system implemented using a robot; a system for performing conversational AI operations;
a system for generating synthetic data;
a system incorporating one or more virtual machines (VMs);
a system implemented at least partially in a data center; or
a system implemented at least partially using cloud computing resources.
(Examiner’s Note: Claim 10 as recited is treated as a “field of use” or “intended use” limitation and therefore carries no patentable weight although it has been examined in view of Li above. The processor as recited has been examined as evidenced in claim 1 above. With respect to the enumerated environments that said processor is “comprised in”, the specification as disclosed merely mentions these environments as preferred intended use environments without specific details that warrant said processor comprised in these environments resulted in a novel and non-obvious structural change to the processor. Reference to MPEP 2112.01 is also made for applicant’s attention).
Claim(s) 6 is/are rejected under 35 U.S.C. 103 as being unpatentable over Li in view of Klomp and Xue as applied to claim 1 above, and further in view of Niu et al., "Electrical Equipment Identification Method With Synthetic Data Using Edge-Oriented Generative Adversarial Network", in IEEE Access, vol. 8, pp. 136487-136497, 2020, hereinafter referred to as Niu, and Gafni et al. (US-20240221235-A1).
Regarding claim 6, Li in view of Klomp and Xue teaches the processor of claim 1, but fails to teach the following limitations as further claimed. Niu and Gafni, however, further teaches the one or more processing units further to train the generative adversarial network (Niu, Fig. 1, “edge-oriented GAN training”) based at least on emphasizing loss for detected edge pixels (Niu, Fig. 1, identified in “edge feature data”) using higher weights than for detected non-edge pixels (Gafni, “employ a weighted binary cross-entropy face loss over the segmentation face parts classes, emphasizing higher importance for face parts, and (2) include the face parts edges as part of the semantic segmentation edge map,” Para [0039]).
PNG
media_image5.png
548
1008
media_image5.png
Greyscale
Niu is considered to be analogous to the claimed invention because they are in the same field of GAN networks that use edge maps to create sharper output images. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have incorporated the teachings of Niu into Li, Klomp, and Xue for the benefit of sharper output images.
Gafni is considered to be analogous to the claimed invention because they are in the same field of feature emphasis using machine learning. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have incorporated the teachings of Gafni into Li, Klomp, and Xue for the benefit of sharper output images.
Claim(s) 7 is/are rejected under 35 U.S.C. 103 as being unpatentable over Li in view of Klomp and Xue as applied to claim 1 above, and further in view of Niu et al., "Electrical Equipment Identification Method With Synthetic Data Using Edge-Oriented Generative Adversarial Network", in IEEE Access, vol. 8, pp. 136487-136497, 2020, hereinafter referred to as Niu.
Regarding claim 7, Li in view of Klomp and Xue teaches the processor of claim 1, but fails to teach the following limitations as further claimed. Niu, however, further teaches wherein the applying of the representation of the infrared image data to the generator of the generative adversarial network (Niu, Fig. 1, “edge-oriented GAN training”) comprises applying an edge map (Niu, Fig. 1, “edge feature data”) extracted from the infrared image data (Niu, Fig. 1, “real infrared image data”) to the generator (Niu, Fig. 1, “edge feature data” is input into “Edge-oriented GAN training”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have incorporated the teachings of Niu into Li, Klomp, and Xue for the benefit of sharper output images.
Claim(s) 8 is/are rejected under 35 U.S.C. 103 as being unpatentable over Li in view of Klomp and Xue as applied to claim 1 above, and further in view of Niu et al., "Electrical Equipment Identification Method With Synthetic Data Using Edge-Oriented Generative Adversarial Network", in IEEE Access, vol. 8, pp. 136487-136497, 2020, hereinafter referred to as Niu, and Stein (US-20170154225-A1).
Regarding claim 8, Li in view of Klomp and Xue teaches the processor of claim 1, but fails to teach the following limitations as further claimed. However, Niu and Stein further teach the one or more processing units further to extract an edge map from the infrared image data (Niu, Fig. 1, “Edge feature data” from the “Real infrared image data”) and pass the edge map over a wireless communication channel to the generator (Stein, “the neural network (or aspects of the neural network) may be provided via one or more servers located remotely from vehicle 200 and accessible over a network via wireless transceiver 172,” Para [0172]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have incorporated the teachings of Niu into Li, Klomp, and Xue for the benefit of sharper output images.
Stein is considered analogous to the claimed invention because they are in the same field of using wireless machine learning to determine outputs from an input image. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have incorporated the teachings of Stein into Li, Klomp, Xue, and Niu for the benefit of less bandwidth consumed by the machine learning model.
Claim(s) 11, 13, 15, and 18-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Li in view of Xue.
Regarding claim 11, Li teaches:
A system comprising one or more processing units to:
generate, based at least on applying a representation of infrared image data to a generator of a generative adversarial network (Fig. 1, “I2V-GAN”), color image data corresponding to the infrared image data (Fig. 1, “translate videos from the source domain X to the target domain Y”, where X is the infrared domain and Y is the RGB/visible light domain).
Li is not relied upon to teach the following limitation. Xue, however, further teaches:
and modify a fill for one or more segmented regions of the color image data (pg. 6, Fig. 4a and 4b, the post-processing operation modifies a fill/a color for the flower).
Xue is considered to be analogous to the claimed invention because they are both in the field of colorizing black-and-white/infrared images. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have incorporated the teachings of Xue into Li for the benefit of more realistic and refined final colorized output images.
Regarding claim 13, the rejection of claim 11 is incorporated herein. Li in view of Xue teaches the processor of claim 11, and Xue further teaches:
the one or more processing units further to modify the fill for the one or more segmented regions of the color image data based at least on an energy function that encourages spatial color continuity (“We then introduce the CRF (Conditional Random Fields) [43], which is a good structure to describe probabilities in a graph, to further optimize the color transferring image and remove those possible wrong matchings in terms of the spatial consistency,” pg. 5, Section “C. Colorization from Reference Image”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have incorporated the teachings of Xue into Li for the benefit of more realistic and refined final colorized output images.
Regarding claim 15, the rejection of claim 11 is incorporated herein. Li in view of Xue teaches the processor of claim 11, and Li teaches:
the one or more processing units further to train the generative adversarial network (Fig. 3, “I2V-GAN network architecture”) based at least on training a first branch that chains the generator (Fig. 3, “GY”) of the color image data (Fig. 3, “ȳt+1”) followed by a generator of infrared image data (Fig. 3,”GX”), and a second branch that chains the generator of the infrared image data followed by the generator of the color image data (Fig. 3 caption, “The opposite direction Y → X is similar”, I.E. the positions of the generators can be swapped).
Regarding claim 18, the rejection of claim 11 is incorporated herein. Li in view of Xue teaches the system of claim 11, and Li further teaches:
wherein the system is comprised in at least one of:
a control system for an autonomous or semi-autonomous machine;
a perception system for an autonomous or semi-autonomous machine;
a system for performing simulation operations;
a system for performing digital twin operations;
a system for performing light transport simulation;
a system for performing collaborative content creation for 3D assets;
a system for performing deep learning operations (Fig. 8, the flower life cycle in images is created using I2V-GAN, where GANs are a type of deep learning architecture);
a system for performing real-time streaming;
a system for generating or presenting one or more of augmented reality content, virtual reality content, or mixed reality content;
a system implemented using an edge device;
a system implemented using a robot;
a system for performing conversational AI operations;
a system for generating synthetic data;
a system incorporating one or more virtual machines (VMs);
a system implemented at least partially in a data center;
or a system implemented at least partially using cloud computing resources.
(Examiner’s Note: Claim 18 as recited is treated as a “field of use” or “intended use” limitation and therefore carries no patentable weight although it has been examined in view of Li and Xue above. The system as recited has been examined as evidenced in claim 11 above. With respect to the enumerated environments that said system is “comprised in”, the specification as disclosed merely mentions these environments as preferred intended use environments without specific details that warrant said system comprised in these environments resulted in a novel and non-obvious structural change to the system. Reference to MPEP 2112.01 is also made for applicant’s attention).
Regarding claim 19, the rejection of claim 11 applies, mutatis mutandis, to claim 19.
Regarding claim 20, the rejection of claim 19 is incorporated herein. Li in view of Xue teaches the method of claim 19, and Li further teaches:
wherein the method is performed by at least one of:
a control system for an autonomous or semi-autonomous machine;
a perception system for an autonomous or semi-autonomous machine;
a system for performing simulation operations;
a system for performing real-time streaming;
a system for generating or presenting one or more of augmented reality content, virtual reality content, or mixed reality content,
a system for performing digital twin operations;
a system for performing deep learning operations (Fig. 8, the flower life cycle in images is created using I2V-GAN, where GANs are a type of deep learning architecture);
a system implemented using an edge device;
a system implemented using a robot;
a system incorporating one or more virtual machines (VMs);
a system implemented at least partially in a data center;
a system for performing light transport simulation;
a system for performing collaborative content creation for 3D assets;
a system for generating synthetic data; or
a system implemented at least partially using cloud computing resources.
(Examiner’s Note: Claim 20 as recited is treated as a “field of use” or “intended use” limitation and therefore carries no patentable weight although it has been examined in view of Li and Xue above. The method as recited has been examined as evidenced in claim 19 above. With respect to the enumerated environments that said method is “performed by at least one of”, the specification as disclosed merely mentions these environments as preferred intended use environments without specific details that warrant said method is performed by these environments resulted in a novel and non-obvious structural change to the method. Reference to MPEP 2112.01 is also made for applicant’s attention).
Claim(s) 14 is/are rejected under 35 U.S.C. 103 as being unpatentable over Li in view of Xue as applied to claim 11 above, and further in view of Zhang et al., “TV-GAN: Generative Adversarial Network Based Thermal to Visible Face Recognition”, arXiv:1712.02514v1, 2017, hereinafter referred to as Zhang.
Regarding claim 14, the rejection of claim 11 is incorporated herein. Li in view of Xue teaches the system of claim 11, but are not relied upon to teach the following limitations. Zhang, however, further teaches:
the one or more processing units further to modify the fill for the one or more segmented regions of the color image data using one or more colors selected from a predetermined range of candidate colors (Fig. 3, the generator learns realistic human skin tones from Y image(s) input into the discriminator) corresponding to a person of at least one of
PNG
media_image6.png
594
1026
media_image6.png
Greyscale
PNG
media_image7.png
536
1017
media_image7.png
Greyscale
Zhang is considered to be analogous to the claimed invention because they are both in the same field of generating an image with realistic colors from black-and-white images. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have incorporated the teachings of Zhang into Li and Xue for the benefit of more accurate colorization from the GAN generator.
Claim(s) 16 is/are rejected under 35 U.S.C. 103 as being unpatentable over Li in view of Xue as applied to claim 11 above, and further in view of Niu et al., "Electrical Equipment Identification Method With Synthetic Data Using Edge-Oriented Generative Adversarial Network", in IEEE Access, vol. 8, pp. 136487-136497, 2020, hereinafter referred to as Niu, and further in view of Gafni et al. (US-20240221235-A1).
Regarding claim 16, the rejection of claim 11 is incorporated herein. Li in view of Xue teaches the system of claim 11, but is not relied upon to teach the following limitations. Niu and Gafni, however, further teach:
the one or more processing units further to train the generative adversarial network (Niu, Fig. 1, “edge-oriented GAN training”) based at least on emphasizing loss for detected edge pixels (Niu, Fig. 1, identified in “edge feature data”) using higher weights than for detected non-edge pixels (Gafni, “employ a weighted binary cross-entropy face loss over the segmentation face parts classes, emphasizing higher importance for face parts, and (2) include the face parts edges as part of the semantic segmentation edge map,” Para [0039]).
PNG
media_image5.png
548
1008
media_image5.png
Greyscale
Niu is considered to be analogous to the claimed invention because they are in the same field of GAN networks that use edge maps to create sharper output images. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have incorporated the teachings of Niu into Li and Xue for the benefit of sharper output images.
Gafni is considered to be analogous to the claimed invention because they are in the same field of feature emphasis using machine learning. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have incorporated the teachings of Gafni into Li and Xue for the benefit of sharper output images.
Claim(s) 17 is/are rejected under 35 U.S.C. 103 as being unpatentable over Li in view of Xue as applied to claim 11 above, and further in view of Niu.
Regarding claim 17, the rejection of claim 11 is incorporated herein. Li in view of Xue teaches the system of claim 11, but are not relied upon to teach the following limitations. Niu, however, further teaches:
wherein the applying of the representation of the infrared image data to the generator of the generative adversarial network (Niu, Fig. 1, “edge-oriented GAN training”) comprises applying an edge map (Niu, Fig. 1, “edge feature data”) comprises applying an edge map (Niu, Fig. 1, “edge feature data”) extracted from the infrared image data (Niu, Fig. 1, “real infrared image data”) to the generator (Niu, Fig. 1, “edge feature data” is input into “Edge-oriented GAN training”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have incorporated the teachings of Niu into Li and Xue for the benefit of sharper output images.
Allowable Subject Matter
Claims are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Examiner’s Note: The color, lighting, and weather condition of claims 2 and 12 are interpreted as all being outside the ego-machine.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Karacan et al., “Learning to Generate Images of Outdoor Scenes from Attributes and Semantic Layouts”, arXiv:1612.00215v1, teaches a method for generating outdoor scenes with different weather conditions.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to RACHEL A OMETZ whose telephone number is (571)272-2535. The examiner can normally be reached 8:30am-5:30pm ET Monday-Thursday, 7:30am-3:30pm ET every other Friday.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Vu Le can be reached at 571-272-7332. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Rachel Anne Ometz/Examiner, Art Unit 2668 7/27/26
Rachel.ometz@uspto.gov
/VU LE/Supervisory Patent Examiner, Art Unit 2668
Prosecution Insights