METHOD FOR GENERATING A COLORED IMAGE, COMPUTER PROGRAM PRODUCT AND SYSTEM FOR GENERATING A COLORED IMAGE

§101 §102 §103

DETAILED ACTION Notice of Pre-AIA or AIA Status The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA . Information Disclosure Statement The information disclosure statement (IDS) submitted on August 22, 2025, September 30, 2025, April 11, 2026 and August 14, 2026 are in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner. Claim Objections Claim 1 is objected to because of the following informalities: Claim 1, line 10 “on the one hand” …. “on the other hand” should be removed. Appropriate correction is required. Claim Rejections - 35 USC § 101 35 U.S.C. 101 reads as follows: Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title. Claim 15 is rejected under 35 U.S.C. 101 because the claimed invention is directed to non-statutory subject matter. The claim(s) does/do not fall within at least one of the four categories of patent eligible subject matter because data per se and/or computer programs do not fall into one of the four categories of statutory invention (machine, process, manufacture, composition). More specifically, claims are eligible for patent protection under 35 USC § 101 if they are in one of the four statutory categories and not directed to a judicial exception to patentability (i.e. laws of nature, natural phenomena, and abstract ideas). Regarding claim 15, the claim is drawn toward a “computer program product.” As described in MPEP 2106, data per se and computer programs do not fall into one of the four statutory categories. Therefore, since claim 15 is drawn towards a program, the claim is not eligible for patent protection. Claim Rejections - 35 USC § 102 In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status. The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action: A person shall be entitled to a patent unless – (a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention. (a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention. Claim(s) 1-11 and 14-17 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by U.S. Publication No. 2022/0094847 to Yang et al. (hereinafter Yang). Regarding independent claim 1, Yang discloses A method for generating a colored image that contains an object arranged in an environment (abstract, “An apparatus includes a camera and a processor circuit. The camera may be configured to capture color images in response to visible light and monochrome infrared images in response to infrared light. The processor circuit may be configured to extract color features from the color images and add color to corresponding monochrome features detected in the monochrome infrared images.” Paragraph 0073, “ In various embodiments, the process (or method) 300 may be utilized to add color, obtained during daytime operation, to a monochrome infrared image captured during nighttime operation. ”), comprising the following method steps: - capturing a first image using a camera, wherein the first image contains a first set of first pixels, which are in color (paragraph 0077, “In an example, a day mode 402 may start in a step 404 where the process 400 may periodically capture color (e.g., RGB, etc.) images.”), and wherein the first image shows only the environment (paragraph 0076, “ In various embodiments, the process (or method) 400 may be utilized to add color, obtained during daytime operation, to a monochrome infrared image captured during nighttime operation. In some applications, a user may have particular objects of interest and/or objects (e.g., vehicles, people, etc.) may vary (come and go) over time, while the scene may remain otherwise static.”); - capturing a second image using the camera, wherein the second image contains a second set of second pixels, which are not in color (paragraph 0079, “In an example, a night mode 420 may start in a step 422 where the process 400 may periodically capture monochrome infrared images. ”), and wherein the second image contains the environment, on the one hand, and an object arranged in the environment, on the other hand (paragraph 0076, “ In various embodiments, the process (or method) 400 may be utilized to add color, obtained during daytime operation, to a monochrome infrared image captured during nighttime operation. In some applications, a user may have particular objects of interest and/or objects (e.g., vehicles, people, etc.) may vary (come and go) over time, while the scene may remain otherwise static.”); - determining a first subset of the set of second pixels of the second image using a processor unit, wherein the first subset of the set of second pixels depicts the object (paragraph 0079, “In a step 424, the process 400 may use an artificial intelligence (AI) algorithm to detect objects of interest in the night or low light scene viewed by the smart IP camera. In an example, each region of interest (ROI) with an object class of interest may be generated by a respective object detector. ” see Figure 8, element 424), and determining a second subset of the set of second pixels of the second image using the processor unit, wherein the second subset of the set of second pixels depicts the environment (paragraph 0076, “In some applications, a user may have particular objects of interest and/or objects (e.g., vehicles, people, etc.) may vary (come and go) over time, while the scene may remain otherwise static.” The environment is read as everything that is not a detected object); - determining a subset of the set of first pixels of the first image using the processor unit (Figure 8, element 406), wherein the subset comprises only pixels of the first image that correspond to corresponding pixels of the set of second pixels of the second image, wherein the corresponding pixels of the set of second pixels of the second image contain only the environment (paragraph 0077, “In a step 406, the process 400 may use an artificial intelligence (AI) algorithm to detect objects of interest in the scene viewed by the smart IP camera. In an example, regions of interest (ROIs) with an object class of interest may be generated by a respective object detector(s). ”); - inputting both the subset of the set of first pixels of the first image and the second subset of the set of second pixels of the second image into the processor unit as input variables (Figure 8, element 428 pulls from both the data sets in 406 and 424); - computing color information for at least one pixel of the second subset of the set of second pixels of the second image on the basis of at least one pixel of the subset of the set of first pixels of the first image using the processor unit (Figure 8, element 430; paragraph 0079, “ In a step 428, the process 400 may use an artificial intelligence (AI) algorithm to smartly select full color features in the modified database that correspond to the extracted objects and/or features from the monochrome infrared images. In a step 430, the process 400 may use another AI algorithm to restore (e.g., colorize) each IR image back to a full color image using the full color information associated with the objects and/or color features in the modified database. ”); and - generating the colored image that contains an object arranged in the environment on the basis of the second image and on the basis of the computed color information of the at least one pixel of the second subset of the set of second pixels of the second image using the processor unit (paragraph 0079, “ In a step 428, the process 400 may use an artificial intelligence (AI) algorithm to smartly select full color features in the modified database that correspond to the extracted objects and/or features from the monochrome infrared images. In a step 430, the process 400 may use another AI algorithm to restore (e.g., colorize) each IR image back to a full color image using the full color information associated with the objects and/or color features in the modified database. ”). Regarding dependent claim 2, the rejection of claim 1 is incorporated herein. Additionally, Yang further discloses wherein - the first image is captured at a first time using the camera (Figure 8, element 404, “fetch RGB images of the scene in day mode”), and wherein - the second image is captured at a second time, different from the first time, using the camera (Figure 8, element 422, “Fetch IR images of the scene in night mode”). Regarding dependent claim 3, the rejection of claim 1 is incorporated herein. Additionally, Yang further discloses wherein - the first image is captured using the camera such that each pixel of the first set of first pixels comprises color information (paragaprh 0077, “In an example, a day mode 402 may start in a step 404 where the process 400 may periodically capture color (e.g., RGB, etc.) images.” RGB is read as the color inormation) and location information with regard to its position in the first image (paragraph 0077, “In an example, regions of interest (ROIs) with an object class of interest may be generated by a respective object detector(s).” the ROI data is read as location information ); and/or wherein - the second image is captured using the camera such that each pixel of the second set of second pixels comprises grayscale information and location information with regard to its position in the second image (NOTE: not analyzed as the limitation is not required due to “and/or”). Regarding dependent claim 4, the rejection of claim 1 is incorporated herein. Additionally, Yang further discloses wherein the computing is carried out such that color information is computed for each pixel of the second subset of the set of second pixels of the second image using the processor unit (paragraph 0079, “In a step 426, the process 400 may feed the ROI images into a feature extractor using an artificial intelligence (AI) algorithm to extract information about one or more features of the objects in the monochrome infrared (IR) images of the scene monitored by the smart IP camera. In an example, the AI algorithm may be implemented using a segmentation neural network (e.g., DeepLabv3/FCN, etc.). In a step 428, the process 400 may use an artificial intelligence (AI) algorithm to smartly select full color features in the modified database that correspond to the extracted objects and/or features from the monochrome infrared images. In a step 430, the process 400 may use another AI algorithm to restore (e.g., colorize) each IR image back to a full color image using the full color information associated with the objects and/or color features in the modified database. In an example, the AI algorithm may be implemented using a fully convolutional neural network (FCN) (e.g., UNet/Pix2Pix, etc.). The restored full color images may be presented by the smart IP camera as an output of the night mode 420.” Paragraph 0080, “The processor/camera circuit 902 generally incorporates hardware and/or software/firmware that may be configured to implement the circuits and processes described above in connection with FIG. 1 through FIG. 8.”). Regarding dependent claim 5, the rejection of claim 1 is incorporated herein. Additionally, Yang further discloses wherein the computing is carried out on the basis of the subset of the set of first pixels of the first image (See Figure 6, element 216 pulls from the data from the day mode (i.e. the first image pixels) image) Regarding dependent claim 6, the rejection of claim 1 is incorporated herein. Additionally, Yang further discloses wherein the method comprises the following method steps: - loading color information with regard to the object for the first subset of the set of second pixels of the second image from a storage unit into the processor unit (Figure 6, color data is stored in the database) ; and - computing color information for at least one pixel of the first subset of the set of second pixels of the second image on the basis of the loaded color information using the processor unit, wherein the colored image that contains the object arranged in the environment is generated using the color information for the at least one pixel of the first subset of the set of second pixels of the second image (Figure 6, elements 208, 216, and 218; paragraph 0072, “). In a step 216, the process 200 may use an artificial intelligence (AI) algorithm to smartly select full color features in the database that correspond to the extracted features from the monochrome infrared images.”… “In a step 218, the process 200 may use another AI algorithm to restore (e.g., colorize) each IR image back to a full color image using the full color information associated with the color features in the database. ”). Regarding dependent claim 7, the rejection of claim 6 is incorporated herein. Additionally, Yang further discloses wherein the computing is carried out on the basis of the loaded color information using the processor unit such that the color information is computed for each pixel of the first subset of the set of second pixels of the second image (figure 3, element 102), wherein the colored image that contains the object arranged in the environment is generated using the color information for each pixel of the first subset of the set of second pixels of the second image (Figure 6, elements 208, 216, and 218; paragraph 0072, “). In a step 216, the process 200 may use an artificial intelligence (AI) algorithm to smartly select full color features in the database that correspond to the extracted features from the monochrome infrared images.”). Regarding dependent claim 8, the rejection of claim 1 is incorporated herein. Additionally, Yang further discloses wherein the method comprises one of the following method steps: a) the computing is carried out (i) using a mathematical model, (ii) using machine learning and/or (iii) using an artificial neural network (paragraph 0079, “In a step 424, the process 400 may use an artificial intelligence (AI) algorithm to detect objects of interest in the night or low light scene viewed by the smart IP camera. In an example, each region of interest (ROI) with an object class of interest may be generated by a respective object detector. In an example, the AI algorithm may be implemented using an object detection neural network (e.g., YOLO/SSD, etc.) to select the ROI images.”); b) the computing is carried out (i) using a single mathematical model and/or (ii) using a single artificial neural network. Regarding dependent claim 9, the rejection of claim 1 is incorporated herein. Additionally, Yang further discloses wherein the generated colored image that contains the object arranged in the environment is displayed on a display unit (paragraph 0079, “ In a step 428, the process 400 may use an artificial intelligence (AI) algorithm to smartly select full color features in the modified database that correspond to the extracted objects and/or features from the monochrome infrared images. In a step 430, the process 400 may use another AI algorithm to restore (e.g., colorize) each IR image back to a full color image using the full color information associated with the objects and/or color features in the modified database. In an example, the AI algorithm may be implemented using a fully convolutional neural network (FCN) (e.g., UNet/Pix2Pix, etc.). The restored full color images may be presented by the smart IP camera as an output of the night mode 420.”) Regarding dependent claim 10, the rejection of claim 1 is incorporated herein. Additionally, Yang further discloses herein the method comprises the following method steps: - capturing the first image using the camera comprises capturing multiple first images (paragraph 0077, “n an example, a day mode 402 may start in a step 404 where the process 400 may periodically capture color (e.g., RGB, etc.) images.”), wherein each image of the multiple first images shows only the environment (paragraph 0076, “In some applications, a user may have particular objects of interest and/or objects (e.g., vehicles, people, etc.) may vary (come and go) over time, while the scene may remain otherwise static. ”) and wherein each image of the multiple first images contains a first set of first pixels, which are in color (paragraph 0077, “n an example, a day mode 402 may start in a step 404 where the process 400 may periodically capture color (e.g., RGB, etc.) images.”); and - determining one image of the multiple first images using the processor unit, wherein the subset of the set of first pixels is determined such that a subset of the set of first pixels of the determined image of the multiple first images is determined (Figure 8, element 404 and 406). Regarding dependent claim 11, the rejection of claim 10 is incorporated herein. Additionally, Yang further discloses wherein the image of the multiple first images is determined such that the image is computed using at least two images of the multiple first images using the processor unit (paragraph 0071, “ the process 200 may periodically capture color (e.g., RGB, etc.) images”). Regarding dependent claim 14, the rejection of claim 1 is incorporated herein. Additionally, Yang further discloses wherein, after the first image has been captured using the camera, the first image is stored in an image storage unit and is loaded into the processor unit in order to determine the subset of the set of first pixels of the first image (paragraph 0040, “. The capture device 104 may have the memory 126 to store the raw data and/or the processed bitstream. For example, the capture device 104 may implement a frame memory and/or buffer 126 to store (e.g., provide temporary storage and/or cache) one or more of the video frames (e.g., the digital video signal). In some embodiments, the processor/logic 124 may perform analysis and/or correction on the video frames stored in the memory/buffer 126 of the capture device 104.”) Regarding independent claim 15, Yang discloses A computer program product containing a program code that is able to be loaded into a processor unit of a system (paragraph 0056, “The processor 150 may be configured to execute computer readable instructions. In one example, the computer readable instructions may be stored by the circuit 154. In some embodiments, the computer readable instructions may comprise controller operations.”) for generating a colored image that contains an object arranged in an environment and, when it is executed, controls the system such that a method as claimed in at least one of the preceding claims is carried out (Figure 3; abstract, “An apparatus includes a camera and a processor circuit. The camera may be configured to capture color images in response to visible light and monochrome infrared images in response to infrared light. The processor circuit may be configured to extract color features from the color images and add color to corresponding monochrome features detected in the monochrome infrared images.”). Regarding dependent claim 16, the rejection of claim 15 is incorporated herein. Additionally, Yang further discloses a system for generating a colored image that contains an object arranged in an environment (Figure 3; abstract, “An apparatus includes a camera and a processor circuit. The camera may be configured to capture color images in response to visible light and monochrome infrared images in response to infrared light. The processor circuit may be configured to extract color features from the color images and add color to corresponding monochrome features detected in the monochrome infrared images.”), comprising a camera for capturing images (Figure 3, element 104, “capture device;” paragraph 0035, “The lens(es) 108 may be attached to the capture device 104. In an example, the capture device 104 may comprise a block (or circuit) 122, a block (or circuit) 124, and a block (or circuit) 126. In one example, the circuit 122 may comprise a color (e.g., RGB) image sensor and a first lens 108, and an infrared (IR) image sensor and a second lens 108. In another example, the circuit 122 may be implemented using an RGB-IR image sensor with a single lens 108. ”), and comprising a processor unit into which a computer program product as claimed in claim 15 is loaded (Figure 3, element 102, “processor/SOC;” paragraph 0056, “The processor 150 may be configured to execute computer readable instructions. In one example, the computer readable instructions may be stored by the circuit 154. In some embodiments, the computer readable instructions may comprise controller operations.”). Regarding dependent claim 17, the rejection of claim 16 is incorporated herein. Additionally, Yang further discloses the system as claimed in claim 16, wherein the system has at least one of the following features: a storage unit (paragraph 0091, “The storage interface 936 may be configured to manage one or more types of storage and/or data access. In one example, the storage interface 936 may implement a direct memory access (DMA) engine and/or a graphics direct memory access (GDMA). In another example, the storage interface 936 may implement a secure digital (SD) card interface (e.g., to connect to the removable media 920). ”); and a display unit (paragraph 0095, “The video output module 950 may be configured to send video data. For example, the processor/camera circuit 902 may be connected to an external device (e.g., a TV, a monitor, a laptop computer, a tablet computing device, etc.). The video output module 950 may implement a high-definition multimedia interface (HDMI), a PAL/NTSC interface, an LCD/TV/Parallel interface and/or a DisplayPort interface.”). Claim Rejections - 35 USC § 103 The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action: A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made. Claim(s) 12 is rejected under 35 U.S.C. 103 as being unpatentable over Yang as applied to claim 1 above, and further in view of U.S. Publication No. 2014/0104432 to Lee et al. (hereinafter Lee). Regarding dependent claim 12, the rejection of claim 1 is incorporated herein. Additionally, Yang fails to explicitly disclose wherein each pixel of the first set of first pixels of the first image is assigned exactly one corresponding pixel of the second set of second pixels of the second image using the processor unit. However, Lee discloses wherein each pixel of the first set of first pixels of the first image is assigned exactly one corresponding pixel of the second set of second pixels of the second image using the processor unit (paragraph 0041, “When the current ambient illuminance belongs to the nighttime illuminance range (S21), the control apparatus 13 replaces a background area of a black and white visible image Ivis 33 from the visible imaging camera 11 with the color background image 32 (S24). Further, the control apparatus 13 provides the display apparatus 12 with a mixed image Irep 34, which is a result of the replacement (S25).” See also Figure 3). Yang is directed toward, “ The camera may be configured to capture color images in response to visible light and monochrome infrared images in response to infrared light. The processor circuit may be configured to extract color features from the color images and add color to corresponding monochrome features detected in the monochrome infrared images (abstract).” Lee is directed toward, “A method of controlling a surveillance system having a visible imaging camera for obtaining a visible image of an object and a display apparatus, includes providing a color visible image, from the visible imaging camera, to the display apparatus and generating a color background image by extracting a background area from the color visible image when a current ambient illuminance belongs to a daytime illuminance range, and replacing a background area of a black and white visible image, from the visible imaging camera, with the color background image and providing a mixed image as a result of the replacement to the display apparatus when the current illuminance belongs to the nighttime illuminance range (abstract).” As can be easily seen by one of ordinary skill in the art before the effective filing date of the claimed invention, Yang and Lee are directed toward similar methods of endeavor of color image generation. Further, one or ordinary skill in the art before the effective filing date of the claimed invention would easily understand mapping pixels 1 to 1 would be beneficial in terms of simple processing and efficient mapping. Said differently, there is no additional math or manipulation required to determine the pixel mapping. Thus, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teaching of Lee to ensure the mapping between pixels in both images have a distinct and simple relationship between them. Allowable Subject Matter Claim 13 is objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims. The following is a statement of reasons for the indication of allowable subject matter: the closest prior arts of record teach methods of generating colored images of a scene from a black and white image using reference data. However, none of them alone or in any combination teaches determining if a pixel in the second subset satisfies a predefined quality criteria, selecting this pixel, and blending the pixel with a corresponding pixel in the first set of pixels from the first image. The closest prior art Yang discloses at Figure 6, element 216, “select best color features in database.” Thus, Yang does disclose selecting appropriate color features between the images, however, there is no quality thresholding, or utilizing the two pixel values to generate the final value. Thus, Yang fails to disclose determining if a pixel in the second subset satisfies a predefined quality criteria, selecting this pixel, and blending the pixel with a corresponding pixel in the first set of pixels from the first image. Conclusion The prior art made of record and not relied upon is considered pertinent to applicant's disclosure: U.S. Patent No. 11,514,261 to Liao et al. discloses, “an image colorization solution. The solution includes determining a similarity between contents presented in a grayscale source image and a color reference image and determining a col or target image corresponding to the source image based on the similarity (abstract).” U.S. Patent No. 11,854,119 to Khodadedeh et al. discloses, “Embodiments are disclosed for automatic object re-colorization in images. In some embodiments, a method of automatic object re-colorization includes receiving a request to recolor an object in an image, the request including an object identifier and a color identifier (abstract)” Contact Any inquiry concerning this communication or earlier communications from the examiner should be directed to Courtney J. Windsor whose telephone number is (571)272-3956. The examiner can normally be reached Monday - Friday 8:00 - 4:00. Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice. If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, John Villecco can be reached at 571-272-7319. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300. Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000. COURTNEY J. WINDSOR Primary Examiner Art Unit 2661