DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1–3, 5, 9, 11–12, 14–15, and 18–20 are rejected under 35 U.S.C. 103 as being unpatentable over Srinivasamurthy et al. (US 2019/0139189 A1) (hereafter, “Srinivasamurthy”) in view of Ratnasingam (Ratnasingam, Sivalogeswaran. "Deep camera: A fully convolutional neural network for image signal processing." Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops. 2019) (hereafter, “Ratnasingam”).
Regarding claim 1, Srinivasamurthy discloses a method [a method of image processing, para 0009] comprising, by a computing system [processing system 1800 can be configured to perform some or all of the functions, operations, and methods disclosed herein ... camera(s) 101, display 102, and hard buttons 103, are components of processing system 1800, para 0042] associated with an image sensor [camera 101 can include a housing 111 retaining a lens 112 and a sensor panel 121, para 0041] comprising a quad-Bayer color filter array [filter array 140 has a Quadra spectral pattern, which is discussed with reference to Figure 8 below, para 0047]: accessing image-sensor data generated by the image sensor [Figure 3-5; sensor panel 121 can include a plurality of camera pixels 171 (e.g., millions of camera pixels) ... sensor panel 121 (also called an image sensor and/or a pixel array) can include ... a spectral filter array 140 (also called a “color filter array” or a “filter array”), para 0046, 0043], wherein the quad-Bayer color filter array comprises sixteen sets of filters [Figure 8; quadra spectral pattern 800 is characterized by a repeating group of sixteen spectral units 701: eight green units 701 a arranged in two diagonal clusters of four, four blue units 701 b arranged in a single cluster, and four red units 701 c arranged in a single cluster, para 0052], each corresponding to a pixel location within a quad-Bayer pattern [a spectral unit can represent a spectral filter or a spectral channel of an image pixel, para 0050]; splitting the image-sensor data into sixteen packed channels [each spectral unit 701 a-701 e can represent a spectral channel (also called “channel”) of an image pixel, para 0060], wherein each of the sixteen packed channels corresponds to one of the sixteen sets of filters [each spectral unit 701 can represent a filter 141 of filter array 140 ... each camera pixel 171 can include a filter 141 configured to admit a single spectral channel. For example, if camera 101 includes a Quadra filter array 140, then the four camera pixels with filters 141 corresponding to blue spectral unit cluster 702 would capture blue channel light, but not green or red channel light, para 0055, 0057]; producing three channels of interpolated pixels by processing the sixteen packed channels [using a machine learning model] [processing system 1800 can perform multi-channel interpolation (also called full-color interpolation) ... processing system 1800 can interpolate a blue channel for the image pixel by finding the average channel value of four neighboring blue channel image pixels. Similarly, processing system 1800 can estimate a green channel for the image pixel by finding the average channel value of four neighboring green channel image pixels, para 0070, 0072], wherein the three channels comprise red, green, and blue channels [processing system 1800 can interpolate until each image pixel includes a channel value for each channel of a predetermined color space ... when an image exists as a multi-channel image, each image pixel can have multiple channels corresponding to a desired color space (e.g., three spectral channels for RGB color space, para 0073, 0067]; and constructing an output image using the three channels of the interpolated pixels [after multi-channel interpolation, processing system 1800 can store the multi-channel image in a stable state (e.g., as a JPEG), para 0073].
Srinivasamurthy fails to explicitly disclose [producing three channels of interpolated pixels by processing the sixteen packed channels] using a machine learning model.
However, Ratnasingam teaches [producing three channels of interpolated pixels by processing the sixteen packed channels] using a machine-learning model [Figure 2; the neural network architecture that we used to implement ISP pipeline ... learn the entire processing ... demosaicing, color transform, pg. 4, 3.1 Network Architecture, left column, lines 1-2; lines 5-8 ... demosaicing is performed to interpolate the missing red, green, or blue values in the Bayer color filter array, pg. 1, Introduction, right column, lines 8-9],
It would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to modify Srinivasamurthy’s reference by incorporating the teachings of Ratnasingam to perform better image signal processing compared to conventional methods, as recognized by Ratnasingam [pg. 1, Abstract].
Further, one skilled in the art could have combined the elements as described above with known method with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Ratnasingam with Srinivasamurthy to obtain the invention as specified in claim 1.
Regarding claim 2, which claim 1 is incorporated, Srinivasamurthy fails to explicitly disclose wherein the machine-learning model comprises a series of neural networks (NNs).
However, Ratnasingam teaches wherein the machine-learning model comprises a series of neural networks (NNs) [Figure 2; the network consists of four parallel connections with one main path and three short connections ... each parallel connection is concatenated to the main path followed by a 1X1 convolution, pg. 4, 3.1 Network Architecture, left column, lines 23-25; lines 37-38].
It would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to modify Srinivasamurthy’s reference by incorporating the teachings of Ratnasingam to improve image processing compared to pipelines that perform processing sequentially, as recognized by Ratnasingam [pg. 1, Abstract].
Further, one skilled in the art could have combined the elements as described above with known method with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Ratnasingam with Srinivasamurthy to obtain the invention as specified in claim 2.
Regarding claim 3, which claim 1 is incorporated, Srinivasamurthy fails to explicitly disclose wherein the machine-learning model comprises a series of convolutional neural networks (CNNs).
However, Ratnasingam teaches wherein the machine-learning model comprises a series of convolutional neural networks (CNNs) [Figure 2; the network consists of four parallel connections with one main path and three short connections ... each parallel connection is concatenated to the main path followed by a 1X1 convolution ... except 1X1 convolutional layers, all the other convolutional layers were performed with 3X3 kernels with stride of 1, pg. 4, 3.1 Network Architecture, left column, lines 23-25; lines 37-38; lines 39-42].
It would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to modify Srinivasamurthy’s reference by incorporating the teachings of Ratnasingam to improve image processing compared to pipelines that perform processing sequentially, as recognized by Ratnasingam.
Further, one skilled in the art could have combined the elements as described above with known method with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Ratnasingam with Srinivasamurthy to obtain the invention as specified in claim 3.
Regarding claim 5, which claim 1 is incorporated, Srinivasamurthy fails to explicitly disclose wherein the machine-learning model is trained with training images selected from a corpus of images.
However, Ratnasingam teaches wherein the machine-learning model is trained with training images selected from a corpus of images [Ratnasingam, this created 272000 raw images of different noise levels and different exposure conditions (low light and high light images). We split the images by randomly assigning the images to training (240000), test (16000), and validation (16000) sets, pg. 6, 3.5 Generation of Bayer image data, left column, lines 26-29].
It would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to modify Srinivasamurthy’s reference by incorporating the teachings of Ratnasingam to use a set of training images that can accurately represent the real world image statistics, as recognized by Ratnasingam [pg. 5, 3.3 Data set].
Further, one skilled in the art could have combined the elements as described above with known method with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Ratnasingam with Srinivasamurthy to obtain the invention as specified in claim 5.
Regarding claim 9, which claim 5 is incorporated, Srinivasamurthy discloses splitting the recovered image-sensor data into sixteen packed channels [each spectral unit 701 a-701 e can represent a spectral channel (also called “channel”) of an image pixel, para 0060]; producing three channels of interpolated pixels by processing the sixteen packed channels [using the machine learning model] [processing system 1800 can perform multi-channel interpolation (also called full-color interpolation) ... processing system 1800 can interpolate a blue channel for the image pixel by finding the average channel value of four neighboring blue channel image pixels. Similarly, processing system 1800 can estimate a green channel for the image pixel by finding the average channel value of four neighboring green channel image pixels, para 0070, 0072].
Srinivasamurthy fails to explicitly disclose recovering image-sensor data from the training image; [producing three channels of interpolated pixels by processing the sixteen packed channel] using the machine learning model; and updating parameters of the machine-learning model based on a comparison between the training image and a constructed image using the produced three channels of the interpolated pixels.
However, Ratnasingam teaches recovering image-sensor data from the training image [we trained our neural network end-to-end using the raw CFA image responses as input, pg. 6, 3.6 Training, left column, line 37-38] from the training image [the raw Bayer CFA images were generated from a database of images, pg. 5, 3.5 Generation of Bayer image data, right column, line 47-38 ... we split the images by randomly assigning the images to training (240000), pg. 6, 3.5 Generation of Bayer image data, left column, line 28-29]; [producing three channels of interpolated pixels by processing the sixteen packed channel] using the machine learning model [Figure 2; the neural network architecture that we used to implement ISP pipeline ... learn the entire processing ... demosaicing, color transform, pg. 4, 3.1 Network Architecture, left column, lines 1-2; lines 5-8 ... demosaicing is performed to interpolate the missing red, green, or blue values in the Bayer color filter array, pg. 1, Introduction, right column, lines 8-9]; and updating parameters of the machine-learning model [the advantage of using a CNN to implement the entire ISP pipeline is that the parameters of the CNN can be optimized in an end-to-end manner by minimizing a single loss function that carefully measures the accuracy of the reconstructed output image, pg. 3, 3 CNN for image signal processing, right column, line 46-50 ... we used the Adam optimizer, pg. 6, 3.6 Training, left column, line 40-41 ... we halve the learning rate if the loss calculated on the validation set did not improve for 100 epochs. This was required to reach the optimum point in the space spanned by the loss function, pg. 6, 3.6 Training, left column, line 47- right column, line 1] based on a comparison between the training image and a constructed image using the produced three channels of the interpolated pixels [Rat, also important to have the appropriate loss function that accurately measures the perceptual quality of an image. Reconstruction of a raw sensor image into an RGB image can be formulated as follows: y=f(x)+n where x ∈ ℝN denotes the reconstructed RGB image, y ∈ ℝN denotes the observed raw Bayer CFA image data, and n denotes the noise from various sources, pg. 4, 3.2 Loss function, right column, line 10-17].
It would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to modify Srinivasamurthy’s reference by incorporating the teachings of Ratnasingam to achieve better image quality, as recognized by Ratnasingam [pg. 2, Introduction, line 52-54].
Further, one skilled in the art could have combined the elements as described above with known method with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Ratnasingam with Srinivasamurthy to obtain the invention as specified in claim 9.
Regarding claim 11, Srinivasamurthy discloses one or more computer-readable non-transitory storage media embodying software that is operable on a computing system [any and all of the methods, functions, and operations described in the present application can be fully embodied in the form of tangible and/or non-transitory machine readable code saved in memory 1802 ... processors 1801 are configured to perform a certain function, method, or operation at least when one of the one or more of the distinct processors is capable of executing code, stored on memory 1802 embodying the function, method, or operation, para 0161, 0159] associated with an image sensor [camera 101 can include a housing 111 retaining a lens 112 and a sensor panel 121, para 0041] comprising a quad-Bayer color filter array [filter array 140 has a Quadra spectral pattern, which is discussed with reference to Figure 8 below, para 0047] when executed to: access image-sensor data generated by the image sensor [Figure 3-5; sensor panel 121 can include a plurality of camera pixels 171 (e.g., millions of camera pixels) ... sensor panel 121 (also called an image sensor and/or a pixel array) can include ... a spectral filter array 140 (also called a “color filter array” or a “filter array”), para 0046, 0043], wherein the quad-Bayer color filter array comprises sixteen sets of filters [Figure 8; quadra spectral pattern 800 is characterized by a repeating group of sixteen spectral units 701: eight green units 701 a arranged in two diagonal clusters of four, four blue units 701 b arranged in a single cluster, and four red units 701 c arranged in a single cluster, para 0052], each corresponding to a pixel location within a quad-Bayer pattern [a spectral unit can represent a spectral filter or a spectral channel of an image pixel, para 0050]; split the image-sensor data into sixteen packed channels [each spectral unit 701 a-701 e can represent a spectral channel (also called “channel”) of an image pixel, para 0060], wherein each of the sixteen packed channels corresponds to one of the sixteen sets of filters [each spectral unit 701 can represent a filter 141 of filter array 140 ... each camera pixel 171 can include a filter 141 configured to admit a single spectral channel. For example, if camera 101 includes a Quadra filter array 140, then the four camera pixels with filters 141 corresponding to blue spectral unit cluster 702 would capture blue channel light, but not green or red channel light, para 0055, 0057]; produce three channels of interpolated pixels by processing the sixteen packed channels [using a machine learning model] [processing system 1800 can perform multi-channel interpolation (also called full-color interpolation) ... processing system 1800 can interpolate a blue channel for the image pixel by finding the average channel value of four neighboring blue channel image pixels. Similarly, processing system 1800 can estimate a green channel for the image pixel by finding the average channel value of four neighboring green channel image pixels, para 0070, 0072], wherein the three channels comprise red, green, and blue channels [processing system 1800 can interpolate until each image pixel includes a channel value for each channel of a predetermined color space ... when an image exists as a multi-channel image, each image pixel can have multiple channels corresponding to a desired color space (e.g., three spectral channels for RGB color space, para 0073, 0067]; and construct an output image using the three channels of the interpolated pixels [after multi-channel interpolation, processing system 1800 can store the multi-channel image in a stable state (e.g., as a JPEG), para 0073].
Srinivasamurthy fails to explicitly disclose [produce three channels of interpolated pixels by processing the sixteen packed channels] using a machine learning model.
However, Ratnasingam teaches [produce three channels of interpolated pixels by processing the sixteen packed channels] using a machine-learning model [Figure 2; the neural network architecture that we used to implement ISP pipeline ... learn the entire processing ... demosaicing, color transform, pg. 4, 3.1 Network Architecture, left column, lines 1-2; lines 5-8 ... demosaicing is performed to interpolate the missing red, green, or blue values in the Bayer color filter array, pg. 1, Introduction, right column, lines 8-9],
It would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to modify Srinivasamurthy’s reference by incorporating the teachings of Ratnasingam to perform better image signal processing compared to conventional methods, as recognized by Ratnasingam [pg. 1, Abstract].
Further, one skilled in the art could have combined the elements as described above with known method with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Ratnasingam with Srinivasamurthy to obtain the invention as specified in claim 11.
Regarding claim 12, (drawn to a computer-readable non-transitory storage media) the proposed combination of Srinivasamurthy in view of Ratnasingam explained in the rejection of method claim 3 renders obvious the steps of the computer-readable non-transitory storage media claim 12, because these steps occur in the operation of the method as discussed above. Thus, the arguments similar to that presented above for claim 3 is equally applicable to claim 12.
Regarding claim 14, (drawn to a computer-readable non-transitory storage media) the proposed combination of Srinivasamurthy in view of Ratnasingam explained in the rejection of method claim 5 renders obvious the steps of the computer-readable non-transitory storage media claim 14, because these steps occur in the operation of the method as discussed above. Thus, the arguments similar to that presented above for claim 5 is equally applicable to claim 14.
Regarding claim 15, (drawn to a computer-readable non-transitory storage media) the proposed combination of Srinivasamurthy in view of Ratnasingam explained in the rejection of method claim 6 renders obvious the steps of the computer-readable non-transitory storage media claim 15, because these steps occur in the operation of the method as discussed above. Thus, the arguments similar to that presented above for claim 6 is equally applicable to claim 15.
Regarding claim 18, (drawn to a computing system) the proposed combination of Srinivasamurthy in view of Ratnasingam explained in the rejection of method claim 9 renders obvious the steps of the computing system claim 18, because these steps occur in the operation of the method as discussed above. Thus, the arguments similar to that presented above for claim 9 is equally applicable to claim 18.
Regarding claim 19, Srinivasamurthy discloses a computing system [Figure 18; processing system 1800 can be configured to perform some or all of the functions, operations, and methods disclosed herein, para 0042] comprising: one or more processors [processors 1801 are configured to perform a certain function, method, or operation at least when one of the one or more of the distinct processors is capable of executing code, stored on memory 1802 embodying the function, method, or operation, para 0159]; an image sensor [processing system 1800 can include ... one or more sensors 1804, para 0157] comprising a quad-Bayer color filter array [sensors 1804 can include camera 101 ... each camera pixel 171 can include a filter 141 configured to admit a single spectral channel. For example, if camera 101 includes a Quadra filter array 140, para 0163, 0057]; and one or more computer-readable non-transitory storage media coupled to one or more of the processors and comprising instructions operable when executed by one or more of the processors [examples of memory 1802 include a non-transitory computer-readable media ... any and all of the methods, functions, and operations described in the present application can be fully embodied in the form of tangible and/or non-transitory machine readable code saved in memory 1802., para 0161] to cause the system to: access image-sensor data generated by the image sensor [Figure 3-5; sensor panel 121 can include a plurality of camera pixels 171 (e.g., millions of camera pixels) ... sensor panel 121 (also called an image sensor and/or a pixel array) can include ... a spectral filter array 140 (also called a “color filter array” or a “filter array”), para 0046, 0043], wherein the quad-Bayer color filter array comprises sixteen sets of filters [Figure 8; quadra spectral pattern 800 is characterized by a repeating group of sixteen spectral units 701: eight green units 701 a arranged in two diagonal clusters of four, four blue units 701 b arranged in a single cluster, and four red units 701 c arranged in a single cluster, para 0052], each corresponding to a pixel location within a quad-Bayer pattern [a spectral unit can represent a spectral filter or a spectral channel of an image pixel, para 0050]; split the image-sensor data into sixteen packed channels [each spectral unit 701 a-701 e can represent a spectral channel (also called “channel”) of an image pixel, para 0060], wherein each of the sixteen packed channels corresponds to one of the sixteen sets of filters [each spectral unit 701 can represent a filter 141 of filter array 140 ... each camera pixel 171 can include a filter 141 configured to admit a single spectral channel. For example, if camera 101 includes a Quadra filter array 140, then the four camera pixels with filters 141 corresponding to blue spectral unit cluster 702 would capture blue channel light, but not green or red channel light, para 0055, 0057]; produce three channels of interpolated pixels by processing the sixteen packed channels [using a machine learning model] [processing system 1800 can perform multi-channel interpolation (also called full-color interpolation) ... processing system 1800 can interpolate a blue channel for the image pixel by finding the average channel value of four neighboring blue channel image pixels. Similarly, processing system 1800 can estimate a green channel for the image pixel by finding the average channel value of four neighboring green channel image pixels, para 0070, 0072], wherein the three channels comprise red, green, and blue channels [processing system 1800 can interpolate until each image pixel includes a channel value for each channel of a predetermined color space ... when an image exists as a multi-channel image, each image pixel can have multiple channels corresponding to a desired color space (e.g., three spectral channels for RGB color space, para 0073, 0067]; and construct an output image using the three channels of the interpolated pixels [after multi-channel interpolation, processing system 1800 can store the multi-channel image in a stable state (e.g., as a JPEG), para 0073].
Srinivasamurthy fails to explicitly disclose [produce three channels of interpolated pixels by processing the sixteen packed channels] using a machine learning model.
However, Ratnasingam teaches [produce three channels of interpolated pixels by processing the sixteen packed channels] using a machine-learning model [Figure 2; the neural network architecture that we used to implement ISP pipeline ... learn the entire processing ... demosaicing, color transform, pg. 4, 3.1 Network Architecture, left column, lines 1-2; lines 5-8 ... demosaicing is performed to interpolate the missing red, green, or blue values in the Bayer color filter array, pg. 1, Introduction, right column, lines 8-9],
It would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to modify Srinivasamurthy’s reference by incorporating the teachings of Ratnasingam to perform better image signal processing compared to conventional methods, as recognized by Ratnasingam [pg. 1, Abstract].
Further, one skilled in the art could have combined the elements as described above with known method with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Ratnasingam with Srinivasamurthy to obtain the invention as specified in claim 19.
Regarding claim 20, (drawn to a computing system) the proposed combination of Srinivasamurthy in view of Ratnasingam explained in the rejection of method claim 3 renders obvious the steps of the computing system claim 20, because these steps occur in the operation of the method as discussed above. Thus, the arguments similar to that presented above for claim 3 is equally applicable to claim 20.
Claims 4 and 13 are rejected under 35 U.S.C. 103 as being unpatentable over Srinivasamurthy (US 2019/0139189 A1) in view of Ratnasingam ("Deep camera: A fully convolutional neural network for image signal processing."), as applied above, and further in view of Chih et al. (WO 2024/168589 A1) (hereafter, “Chih”).
Regarding claim 4, which claim 1 is incorporated, neither Srinivasamurthy nor Ratnasingam appears to explicitly disclose wherein the computing system is associated with an artificial-reality (AR) display.
However, Chih teaches wherein the computing system is associated with an artificial-reality (AR) display [Figure 3; the image capture system 300 can include or be part of an electronic device or system. For example, the image capture system 300 can include or be part of an electronic device or system, such as ... an extended reality (XR) device (e.g., a virtual reality (VR) device, an augmented reality (AR) device, or a mixed reality (MR) device) ... additional components of the image capture system 300 ... one or more display devices, para 0062, 0063].
It would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to modify Srinivasamurthy’s reference in view of Ratnasingam by incorporating the teachings of Chih to include a multimedia system that can capture and processing images in different conditions such as low lighting conditions, as recognized by Chih [¶003].
Further, one skilled in the art could have combined the elements as described above with known method with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Chih with Srinivasamurthy and Ratnasingam to obtain the invention as specified in claim 4.
Regarding claim 13, (drawn to a computer-readable non-transitory storage media) the proposed combination of Srinivasamurthy in view of Ratnasingam and further in view of Chih explained in the rejection of method claim 4 renders obvious the steps of the computer-readable non-transitory storage media claim 13, because these steps occur in the operation of the method as discussed above. Thus, the arguments similar to that presented above for claim 4 is equally applicable to claim 13.
Claims 6–8 and 16–17 are rejected under 35 U.S.C. 103 as being unpatentable over Srinivasamurthy (US 2019/0139189 A1) in view of Ratnasingam ("Deep camera: A fully convolutional neural network for image signal processing."), as applied above, and further in view of Leung et al. (US 8,738,553 B1) (hereafter, “Leung”).
Regarding claim 6, which claim 5 is incorporated, Srinivasamurthy fails to explicitly disclose wherein the corpus of images comprise randomly cropped images, and wherein each image in the corpus of images is associated with a score determined based on characteristics of the image.
However, Ratnasingam teaches wherein the corpus of images comprise randomly cropped images [we took four different crops of 240X220 pixels image. This created 272000 raw images of different noise levels and different exposure conditions (low light and high light images), pg. 6, 3.5 Generation of Bayer image data, left column, lines 25-28].
It would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to modify Srinivasamurthy’s reference by incorporating the teachings of Ratnasingam to include randomly cropped images to create a larger dataset for training, as recognized by Ratnasingam [pg. 6, 3.5 Generation of Bayer image data].
Neither Srinivasamurthy nor Ratnasingam appears to explicitly disclose wherein each image in the corpus of images is associated with a score determined based on characteristics of the image.
However, Leung teaches wherein each image in the corpus of images is associated with a score [the image quality subsystem 120 classifies the training image corresponding to the selected feature vector using the selected feature vector as an input to the image classification model. The output of the image classification model is an initial score 208 for the image, Col 6, line 66-Col 7, line 3] determined based on characteristics of the image [the image quality subsystem 120 obtains image feature values for each training image TI1-TIn ... image feature values can specify a value representing a color, texture, and/or other characteristics of an image ... the image quality subsystem 120 can represent the image feature values of each of the training images TI1-TIn as feature vectors FV1-FVn that correspond to each of the training images TI1-TIn, Col 6, line 44-48; line 52-55].
It would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to modify Srinivasamurthy’s reference in view of Ratnasingam by incorporating the teachings of Leung to determine which images are high quality versus low quality, as recognized by Leung [Col 9, line 43-45].
Further, one skilled in the art could have combined the elements as described above with known method with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Leung with Srinivasamurthy and Ratnasingam to obtain the invention as specified in claim 6.
Regarding claim 7, which claim 6 is incorporated, neither Srinivasamurthy nor Ratnasingam appears to explicitly disclose wherein a first image with a first score has a higher probability of being selected for the training than a second training image with a second score that is lower than the first score.
However, Leung teaches wherein a first image with a first score has a higher probability of being selected for the training than a second training image with a second score [the semi-random selection of training images can be subject to a selection requirement that specifies a likelihood that a negative and/or a positive image is selected from the set of training images. For example, the selection requirement can specify that a negative image be selected with a probability of 0.7, while a positive image is selected with a probability of 0.3. Thus, the semi-random number generator can select a negative image from the set of training images in 70% of all selections, while selecting a positive image the remaining 30% of the time, Col 10, line 41-50] that is lower than the first score [the initial score for the image is compared to a threshold quality score that indicates whether the image is classified as a positive ("high quality") image or a negative ("low quality") image. When the quality score meets or exceeds the threshold quality score, the image as classified as a positive image. When the quality score does not meet or exceed the threshold quality score, the image is classified as a negative image, Col 7, line 6-13].
It would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to modify Srinivasamurthy’s reference in view of Ratnasingam by incorporating the teachings of Leung to select images based on different selection criteria, as recognized by Leung [Col 10, line 32-50].
Further, one skilled in the art could have combined the elements as described above with known method with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Leung with Srinivasamurthy and Ratnasingam to obtain the invention as specified in claim 7.
Regarding claim 8, which claim 6 is incorporated, neither Srinivasamurthy nor Ratnasingam appears to explicitly disclose wherein the characteristics comprise spectral signal frequencies associated with the image.
However, Leung teaches wherein the characteristics comprise spectral signal frequencies associated with the image [Leung, each image is characterized by frequency distribution of the image feature values for the image, Col 11, line 67-Col 12, line 2].
It would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to modify Srinivasamurthy’s reference in view of Ratnasingam by incorporating the teachings of Leung to characterize an image irrespective of the image scale, as recognized by Leung [Col 12, line 3-4].
Further, one skilled in the art could have combined the elements as described above with known method with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Leung with Srinivasamurthy and Ratnasingam to obtain the invention as specified in claim 8.
Regarding claim 16, (drawn to a computer-readable non-transitory storage media) the proposed combination of Srinivasamurthy in view of Ratnasingam and further in view of Leung explained in the rejection of method claim 7 renders obvious the steps of the computer-readable non-transitory storage media claim 16, because these steps occur in the operation of the method as discussed above. Thus, the arguments similar to that presented above for claim 7 is equally applicable to claim 16.
Regarding claim 17, (drawn to a computer-readable non-transitory storage media) the proposed combination of Srinivasamurthy in view of Ratnasingam and further in view of Leung explained in the rejection of method claim 8 renders obvious the steps of the computer-readable non-transitory storage media claim 17, because these steps occur in the operation of the method as discussed above. Thus, the arguments similar to that presented above for claim 8 is equally applicable to claim 17.
Claim 10 is rejected under 35 U.S.C. 103 as being unpatentable over Srinivasamurthy (US 2019/0139189 A1) in view of Ratnasingam ("Deep camera: A fully convolutional neural network for image signal processing."), as applied above, and further in view of Watanabe (US 2018/0260934 A1) (hereafter, “Watanabe”).
Regarding claim 10, which claim 5 is incorporated, neither Srinivasamurthy nor Ratnasingam appears to explicitly disclose wherein the training images are pre-processed by amplifying red and blue pixels.
However, Watanabe teaches wherein the training images are pre-processed by amplifying red and blue pixels [white balance correction is performed by applying a gain to the R signal and the B signal, para 0040].
It would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to modify Srinivasamurthy’s reference in view of Ratnasingam by incorporating the teachings of Watanabe to increase the sensitives of the red and blue pixels, as recognized by Watanabe [¶0040].
Further, one skilled in the art could have combined the elements as described above with known method with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Watanabe with Srinivasamurthy and Ratnasingam to obtain the invention as specified in claim 10.
Conclusion
The art made of record and not relied upon is considered pertinent to applicant's disclosure:
Quad Bayer Joint Demosaicing and Denoising Based on Dual Encoder Network with Joint Residual Learning to Zheng et al. discloses a duel encoder network that performs demosaicing and denoising for quad Bayer color filter array.
Beyond Joint Demosaicking and Denoising: An Image Processing Pipeline for a Pixel-bin Image Sensor to Sharif et al. discloses a network for demosaiking and denoising for a pixel-bin image sensor.
US 2024/0144717 A1 to Feng et al. discloses a method of processing image data including de-mosaicing.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to TOLUWANI MARY-JANE IJASEUN whose telephone number is (571)270-1877. The examiner can normally be reached Monday - Friday 7:30AM-4PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Henok Shiferaw can be reached at (571) 272-4637. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/TOLUWANI MARY-JANE IJASEUN/Examiner, Art Unit 2676
/Henok Shiferaw/Supervisory Patent Examiner, Art Unit 2676