CTNF 18/781,432 CTNF 96846 Notice of Pre-AIA or AIA Status 07-03-aia AIA 15-10-aia The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA. Priority 02-27 AIA Acknowledgment is made of applicant’s claim for foreign priority under 35 U.S.C. 119 (a)-(d). The certified copy has been filed in parent Application No. IN202441017810 , filed on 03/12/2024 . Information Disclosure Statement The information disclosure statement (IDS) submitted on 07/23/2024 and 02/28/2025 are being considered by the examiner. Claim Rejections - 35 USC § 103 07-06 AIA 15-10-15 In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status. 07-20-aia AIA The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action: A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made. 07-103 AIA The text of those sections of Title 35, U.S. Code not included in this action can be found in a prior Office action. 07-21-aia AIA Claim (s) 1-4, 10-13, 19-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Afifi ( When Color Constancy Goes Wrong: Correcting Improperly White-Balanced Images ), and further in view of Mayle (US 9576221 B2) . Regarding claims 1, 10 and 19, Afifi discloses at least one processor (page 1, col 2: The WB transform is applied to the camera’s raw-RGB image and is often one of the first steps in the in-camera processing pipeline.) ; a memory configured to store instructions which, when executed by the at least one processor, cause the device to (page 4, col 2: We also seek compact representation as these features represent the bulk of information that will need to be stored in memory.) : receive an input image to be corrected (page 3, col 2: Fig. 3 provides an overviews our framework. Given an incorrectly white-balanced sRGB image, denoted as Iin, our goal is to compute a mapping M that can transform the input image’s colors to appear as if the WB was correctly applied) ; assign, using the AI module, weights from among zero and non-zero magnitudes to each template image based on an extent and a type of correction to be applied to each template image (fig. 1: Two incorrectly white-balanced images produced by different cameras and attempts to correct them using (1) a linear WB correction, (2) correction by first applying a gamma linearization ( [3], [17]), and (3) our results. Also shown is the ground truth image produced by the camera using the correct WB.) ; and apply the assigned weights to the plurality of template images to obtain a corrected output image (fig. 3, (corrected images)) . Afifi does not explicitly disclose but implicitly discloses generate a plurality of template images based on the input image, wherein each template image from among the plurality of template images is generated based on at least one of an image feature transformation (fig. 4: Example of rendering sRGB training images. Working directly from the raw-RGB camera image, we render sRGB output images using the camera’s pre-defined white balance settings and different picture styles. A target white balance sRGB image is also rendered using the color rendition chart in the scene to provide the ground truth.) ; provide the plurality of template images to an artificial intelligence (AI) module. In a similar field of endeavor of object recognition, Mayle teaches generate a plurality of template images based on the input image, wherein each template image from among the plurality of template images is generated based on at least one of an image feature transformation (col 14, lines 36-45: Preferably one classifier is trained per template image. However, a particular template image may be treated as multiple templates, one for each (color) channel. In a present embodiment hereof, test images and template images are pre-processed to produce three separate images filtered to a specific color channel (preferably Y, U and V). The system then attempts to match on each of these separate channels for each template. It should be appreciated that not every logo or image will have information on each channel e.g. a black and white image will not have U or V.) ; provide the plurality of template images to an artificial intelligence (AI) module (col 14, lines 36-45: Preferably one classifier is trained per template image. However, a particular template image may be treated as multiple templates, one for each (color) channel. In a present embodiment hereof, test images and template images are pre-processed to produce three separate images filtered to a specific color channel (preferably Y, U and V). The system then attempts to match on each of these separate channels for each template. It should be appreciated that not every logo or image will have information on each channel e.g. a black and white image will not have U or V.) . It would have been obvious to one of ordinary skill in the art before the effective filing date of this invention to combine the known system of white balancing, as disclosed by Afifi, with known technique of template creation, as taught by Mayle, in order to yield the predictable results of ensuring objective color accuracy, eliminating light-source contamination, and enabling consistent data comparison. Regarding claim 2 and 11, Afifi discloses generating a transformation matrix by assigning a weight to each color contribution component of each template image from among the plurality of template images (page 4, col 2: We have examined several different color transformation mappings and found the polynomial kernel function proposed by Hong et al. [24] provided the best results for our task (additional details are given in the supplemental materials). Based on [24], Φ:[R, G, B]T → [R, G, B, RG, RB, GB, R2, G2, B2, RGB, 1]T and M(i) is represented as a 3×11 matrix.) . Regarding claim 3 and 12, Afifi discloses performing a matrix multiplication between the transformation matrix and the plurality of template images (page 5, col 2: As shown in Fig. 3, the final color transformation is generated based on correction transformations associated with training images taken from different cameras and render styles (see supplemental materials for a study of the effect of having different picture styles on the results). By blending the mapping functions from images produced by a wide range of different cameras and their different photofinishing styles, we can interpret M correction as mapping the input image to a meta-camera’s output composed from the most similar images to the input.) . Regarding claim 4, 13, and 20, Afifi discloses generating each template image from among the plurality of template images based on at least a non-linear combination of one or more channels from among red-green-blue (RGB) color channels ("page 3, col 2: Our method relies on a large set of n training images expressed as It={I(1)t,…,I(n)t} that have been generated using the incorrect WB settings. Each training image has a corresponding correct white-balanced image (or ground truth image), denoted as I(i)gt. Note that multiple training images may share the same target ground truth image. Section 3.2 details how we generated this dataset. For each pair of training image I(i)t and its ground truth image I(i)gt, we compute a nonlinear color correction matrix M(i) that maps the incorrect image’s colors to its target ground truth image’s colors. The details of this mapping are discussed in Section 3.3. page 4, col 1: Because the images are in the camera’s raw-RGB format, we can convert them to sRGB output emulating different WB settings and picture styles on the camera. To do this, we use the Adobe Camera Raw feature in Photoshop to render different sRGB images using different WB presets in the camera. In addition, each incorrect WB can be rendered with different camera picture styles (e.g., Vivid, Standard, Neutral, Landscape). Depending on the make and model of the camera, a single raw-RGB image can be rendered to more than 25 different camera-specific sRGB images. These images make up our training images {I(1)t,…,I(n)t}.") . 07-21-aia AIA Claim (s) 5 and 14 is/are rejected under 35 U.S.C. 103 as being unpatentable over Afifi ( When Color Constancy Goes Wrong: Correcting Improperly White-Balanced Images ), in view of Mayle (US 9576221 B2), and further in view of Saad (US 20230306714 A1) . Regarding claims 5 and 14, Afif does not disclose but Mayle teaches wherein each template image from among the one or more template images comprises at least one attribute (col 14, lines 36-45: Preferably one classifier is trained per template image. However, a particular template image may be treated as multiple templates, one for each (color) channel. In a present embodiment hereof, test images and template images are pre-processed to produce three separate images filtered to a specific color channel (preferably Y, U and V). The system then attempts to match on each of these separate channels for each template. It should be appreciated that not every logo or image will have information on each channel e.g. a black and white image will not have U or V.) . It would have been obvious to one of ordinary skill in the art before the effective filing date of this invention to combine the known system of white balancing, as disclosed by Afifi, with known technique of template creation, as taught by Mayle, in order to yield the predictable results of ensuring objective color accuracy, eliminating light-source contamination, and enabling consistent data comparison. Afifi and Mayle do not disclose or teach extracting, using the AI module, semantic segmentation information included in the input image; and generating one or more template images corresponding to one or more attributes included in the semantic segmentation information. In a similar field of endeavor of chromatic undertone detection, Saad teaches extracting, using the AI module, semantic segmentation information included in the input image ([0028] Examples of types of segmentation include panoptic segmentation and semantic segmentation. Segmenting of an image can isolate items, portions of items, or portions of a person that are known to provide more effective measures of the undertone of interest. For example, in cosmetic selection, it is known that certain parts of the body provide more readily discernible undertones for an individual's complexion. The underside of the wrist is known to be one such area. Segmentation can be used to selectively identify a person's wrist from the person's hand, arm, etc. Color warmth classification provides a numerical indication of a color warmth category in the image in the form of an image warmth profile.) ; and generating one or more template images corresponding to one or more attributes included in the semantic segmentation information ([0028] Examples of types of segmentation include panoptic segmentation and semantic segmentation. Segmenting of an image can isolate items, portions of items, or portions of a person that are known to provide more effective measures of the undertone of interest. For example, in cosmetic selection, it is known that certain parts of the body provide more readily discernible undertones for an individual's complexion. The underside of the wrist is known to be one such area. Segmentation can be used to selectively identify a person's wrist from the person's hand, arm, etc. Color warmth classification provides a numerical indication of a color warmth category in the image in the form of an image warmth profile.) . It would have been obvious to one of ordinary skill in the art before the effective filing date of this invention to combine the known system of white balancing with template creation, as disclosed by Afifi and Mayle, with known technique of semantic segmentation, as taught by Saad, in order to yield the predictable results of improving white balance by classifying objects at the pixel level, allowing the algorithm to apply targeted color corrections based on context rather than global averages . 07-21-aia AIA Claim (s) 6, 7, 15, 16 is/are rejected under 35 U.S.C. 103 as being unpatentable over Afifi ( When Color Constancy Goes Wrong: Correcting Improperly White-Balanced Images ), in view of Mayle (US 9576221 B2), and further in view of Rowell (US 20190208181 A1) . Regarding claims 6 and 15, Afifi and Mayle do not disclose but in a similar field of endeavor of 3d digital image, Rowell teaches generating the plurality of template images by performing at least one pixel shift operation on a plurality of pixels included in the input image ([0185] In some embodiments, the camera calibration module 221 modifies one or more coordinates of the position data included in the first set of image data to shift the alignment of color data included in the right and left image frames to a plurality of shift positions.) , wherein the at least one pixel shift operation comprises at least one from among a shift right operation, a shift left operation, a shift up operation, and a shift down operation ([0185] In some embodiments, the camera calibration module 221 modifies one or more coordinates of the position data included in the first set of image data to shift the alignment of color data included in the right and left image frames to a plurality of shift positions.) . It would have been obvious to one of ordinary skill in the art before the effective filing date of this invention to combine the known system of white balancing with template creation, as disclosed by Afifi and Mayle, with known technique of pixel shifting, as taught by Rowell, in order to yield the predictable results of capturing the full RGB value at every single photosite, and yields highly accurate, artifact-free color data that simplifies precise white point calibration and cross-spectrum analysis. Regarding claims 7 and 16, Afifi discloses wherein the image feature transformation comprises at least one of a color kernel transform, an attribute kernel transform, and a shifted pixel kernel transform (page 5, col 2: As shown in Fig. 3, the final color transformation is generated based on correction transformations associated with training images taken from different cameras and render styles (see supplemental materials for a study of the effect of having different picture styles on the results).) . Rowell teaches and a shifted pixel kernel transform ([0185] In some embodiments, the camera calibration module 221 modifies one or more coordinates of the position data included in the first set of image data to shift the alignment of color data included in the right and left image frames to a plurality of shift positions.) . It would have been obvious to one of ordinary skill in the art before the effective filing date of this invention to combine the known system of white balancing with template creation, as disclosed by Afifi and Mayle, with known technique of pixel shifting, as taught by Rowell, in order to yield the predictable results of capturing the full RGB value at every single photosite, and yields highly accurate, artifact-free color data that simplifies precise white point calibration and cross-spectrum analysis . 07-21-aia AIA Claim (s) 8, 9, 17, 18 is/are rejected under 35 U.S.C. 103 as being unpatentable over Afifi ( When Color Constancy Goes Wrong: Correcting Improperly White-Balanced Images ), in view of Mayle (US 9576221 B2), and further in view of Abdelhamed (US 12488504 B2) . Regarding claims 8 and 17, Afifi does not explicitly disclose but Mayle teaches providing, to the AI module, a plurality of sample template images corresponding to a sample image (col 14, lines 36-45: Preferably one classifier is trained per template image. However, a particular template image may be treated as multiple templates, one for each (color) channel. In a present embodiment hereof, test images and template images are pre-processed to produce three separate images filtered to a specific color channel (preferably Y, U and V). The system then attempts to match on each of these separate channels for each template. It should be appreciated that not every logo or image will have information on each channel e.g. a black and white image will not have U or V.) ; generating predicted weights for color contribution components corresponding to the plurality of sample template images (col 14, lines 30-35: The ground truth (e.g., as determined above) or a subset thereof for a particular template image may be used to train a classifier for that template image (e.g., at 410 in FIG. 4(a)). The classifier can then be used to predict which of the other candidate matches for this template are true positives are which are false positives. Preferably one classifier is trained per template image. However, a particular template image may be treated as multiple templates, one for each (color) channel. In a present embodiment hereof, test images and template images are pre-processed to produce three separate images filtered to a specific color channel (preferably Y, U and V).) ; applying the predicted weights to the color contribution components corresponding to the plurality of sample template images to generate a predicted output image ("col 19, lines 61-70: Then the libsvm model may loaded (e.g., from the database(s)) and be applied using the predict function to compute the predicted class of the candidate match. The template image associated with the match is known, and so the identifier of that template image may be used to access and obtain the corresponding classifier for that template image from the database(s) 104. ") . It would have been obvious to one of ordinary skill in the art before the effective filing date of this invention to combine the known system of white balancing, as disclosed by Afifi, with known technique of template creation, as taught by Mayle, in order to yield the predictable results of ensuring objective color accuracy, eliminating light-source contamination, and enabling consistent data comparison. Afifi and Mayle do not disclose comparing the predicted output image with a ground truth image to calculate a training loss; and training the AI module based on the calculated training loss. In a similar field of endeavor of color transformation, Abdelhamed teaches comparing the predicted output image with a ground truth image to calculate a training loss (col 11, lines 20-25: A difference between the estimated first color transform and a ground-truth color transform may be computed as a loss of the first neural network and the loss may be back-propagated to the first neural network 40 to update node weights of the first neural network 40.) ; and training the AI module based on the calculated training loss (col 11, lines 20-25: A difference between the estimated first color transform and a ground-truth color transform may be computed as a loss of the first neural network and the loss may be back-propagated to the first neural network 40 to update node weights of the first neural network 40.) . It would have been obvious to one of ordinary skill in the art before the effective filing date of this invention to combine the known system of white balancing with template creation, as disclosed by Afifi and Mayle, with known technique of training loss, as taught by Abdelhamed, in order to yield the predictable results of correcting color casts and accurately predict scene illumination by optimizing algorithms for color constancy and perceptual accuracy. Regarding claims 9 and 18, Afifi and Mayle do not disclose but Abdelhamed teaches performing a loss calculation until the training loss is below a predetermined threshold value (col 15, lines 2-8: A difference between the estimated color transform and a ground-truth color transform may be computed as a loss of the transform estimator, and the transform estimator may be trained until the loss reaches a predetermined minimum value, or converges into a constant value with a preset margin.) . It would have been obvious to one of ordinary skill in the art before the effective filing date of this invention to combine the known system of white balancing with template creation, as disclosed by Afifi and Mayle, with known technique of training loss, as taught by Abdelhamed, in order to yield the predictable results of correcting color casts and accurately predict scene illumination by optimizing algorithms for color constancy and perceptual accuracy . Conclusion 07-96 AIA The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. US 20230098058 A1 with regards to claim 8: "[0056] The image processing module 154 applies a single WB correction that corrects for a fixed color temperature (e.g., 5500 K) in addition to generating small images (e.g. 384×384) with a predefined set of color temperatures (e.g., 2850K, 3800K) and/or tints that correlate to the common lighting conditions (e.g., incandescent, fluorescent). Given these images, the image processing module 154 predicts blending weights 412 to produce the final AWB result 404. [0060] The DNN is used to predict the values of {W.sub.i}, where the DNN takes as input the small images rendered with predefined WB settings, and learns to produce proper weighting maps {W.sub.i}.". Any inquiry concerning this communication or earlier communications from the examiner should be directed to AHMED A NASHER whose telephone number is (571)272-1885. The examiner can normally be reached Mon - Fri 0800 - 1700. Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice. If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Andrew Moyer can be reached at (571) 272-9523. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300. Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000. /AHMED A NASHER/Examiner, Art Unit 2675 /ANDREW M MOYER/Supervisory Patent Examiner, Art Unit 2675 Application/Control Number: 18/781,432 Page 2 Art Unit: 2675 Application/Control Number: 18/781,432 Page 3 Art Unit: 2675 Application/Control Number: 18/781,432 Page 4 Art Unit: 2675 Application/Control Number: 18/781,432 Page 5 Art Unit: 2675 Application/Control Number: 18/781,432 Page 6 Art Unit: 2675 Application/Control Number: 18/781,432 Page 7 Art Unit: 2675 Application/Control Number: 18/781,432 Page 8 Art Unit: 2675 Application/Control Number: 18/781,432 Page 9 Art Unit: 2675 Application/Control Number: 18/781,432 Page 10 Art Unit: 2675 Application/Control Number: 18/781,432 Page 11 Art Unit: 2675 Application/Control Number: 18/781,432 Page 12 Art Unit: 2675 Application/Control Number: 18/781,432 Page 13 Art Unit: 2675
Prosecution Insights