Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Notice to Applicants
This communication is in response to the Application filed on 10/11/2022.
Claims 1-22 are pending.
Claim Objections
Claim 3-10 and 12-22 are objected to because of the following informalities:
• In claim 3, line 1, "… claim 2" should be " … claim 2,". A comma needs to be after the claim number. Appropriate correction is required.
• In claim 4-10 and 12-22, line 1, "… claim X" should be "… claim X,". A comma needs to be after the claim number. Appropriate correction is required.
• In claim 10, line 1-2, “the video capture device a smartphone” should be " wherein the video capture device is a smartphone". Appropriate correction is required.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claim 3 and 13 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 3 recites the limitation "the image data" in line 2. It is unclear if “the image data” is referring back to “linear image data” in claim 2, “raw image data” in claim 1 or something else. Clarification/explanation is required.
Claim 13 recites the limitation "the image data" in line 2. It is unclear if “the image data” is referring back to “linear image data” in claim 12, “raw image data” in claim 11 or something else. Clarification/explanation is required.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim 1, 4-6, 10, 11, 14-16, 20 and 21 are rejected under 35 U.S.C. 103 as being unpatentable over SLABAUGH et al. (U.S. Publication No. 2021/0073957) (hereafter, "SLABAUGH") in view of Chen et al. (U.S. Publication No. 2021/0304359) (hereafter, "Chen") and further in view of Li et al. (U.S. Publication No. 2021/0248758) (hereafter, "Li").
Regarding claim 1, SLABAUGH teaches A video capture device, comprising ([0010] an image processor comprising a plurality of processing modules configured to operate in series to transform a raw image captured by a camera; [0053] The approach is applicable to both still photography and video): an image sensor ([0077] FIG. 11 shows an example of a camera configured to use the AISP to process images taken by an image sensor in the camera; [0081] an image may be captured by the camera sensor 2) configured to generate, from light incident on pixels of the image sensor ([0055] the RAW data passed into the RAW denoiser module 20 is an image formed using a color filter array (CFA) that captures light of specific colors at each pixel, for example, using the well-known Bayer pattern), a plurality of image frames of a captured scene ([0077] a camera configured to use the AISP to process images taken by an image sensor; [0053] the input to the pipeline may include a multi-frame (MF) burst of RAW images; [0068] Once the network is trained, it can be applied to mosaiced images to produce RGB images), each of the plurality of image frames comprising raw image data mosaiced according to a color filter pattern ([0055] the RAW data passed into the RAW denoiser module 20 is an image formed using a color filter array (CFA) that captures light of specific colors at each pixel, for example, using the well-known Bayer pattern. FIG. 3 (a) shows the standard Bayer pattern colour filter array on the sensor. This pattern has a recurring 2×2 mosaic that is tiled across the image. At each pixel, either a red 30, green 31 or blue color 32 is acquired. An image captured in this format is said to be mosaiced); Memory ([0078] Each physical device implementing an entity comprises a processor and a memory); and one or more processors configured to ([0077] FIG. 11 shows an example of a camera configured to use the AISP to process images taken by an image sensor in the camera. Such a camera 1 typically includes some onboard processing capability. This could be provided by the processor 4) process the plurality of image frames ([0049] A module configured to perform multi-frame noise reduction (MFNR), shown at 24, may combine a burst of RAW frames to achieve noise reduction ... The input to this module is a burst of RAW frames (typically between 6 and 12 frames)) to generate an output image frame, the one or more processors configured to ([0083] The stages in the AISP are based on deep convolutional neural networks, which take an image as input, and produce an output image; [0045] The output is an RGB image, where each pixel has a red, green, and blue color. This is a color image; Figure 2 shows generating an "RGB" between element 22 and 23 by processing “MF Raw” which are multi-frame burst of RAW images): de-noise with a multi-frame noise reduction algorithm ([0049] A module configured to perform multi-frame noise reduction (MFNR), shown at 24, may combine a burst of RAW frames to achieve noise reduction) ... of the output image frame ([0045] The output is an RGB image) … of the output image frame ([0045] The output is an RGB image); apply one or more image processing functions to the raw image data such that the output image frame comprises a partially developed image that is not tonally processed, the one or more image processing functions including at least a de-mosaicing function; and ([0045] 2) A demosaicing module, shown at 22. The demosaicing stage interpolates the values from the mosaic to produce an RGB image. The input to the demosaicing module is a RAW image, where each pixel has a red, green, or blue color. The output is an RGB image; [0046] The input to this module 23 is a RGB image with a large dynamic range and uncorrected colors (e.g. 1024 levels per color channel)) store the output image frame in the memory ([0018] The image processor may be configured to generate an output that is a compressed representation of an image input to the image processor. This may reduce the file size of the resulting image, allowing images to be stored more efficiently).
SLABAUGH does not expressly teach apply a resolution enhancement algorithm to at least two image frames of the plurality of image frames to increase resolution … process at least a relatively lower exposure frame of the plurality of image frames and a relatively higher exposure frame of the plurality of image frames to enhance dynamic range.
However, Chen teaches apply a resolution enhancement algorithm to at least two image frames of the plurality of image frames to increase resolution of … image ([0063] A method used to render a super-resolution image of a scene … capturing, in a burst sequence, multiple frames of an image of a scene, the multiple frames having respective, relative sub-pixel offsets of the image; performing super-resolution computations using the captured, multiple frames; [0029] The super-resolution computations ... Algorithms that support the super-resolution computations 302 may reside in the super-resolution manager 120 of the user device 102; [0039] FIG. 3 also illustrates a combining operation 312 that creates the super-resolution image 122 of the scene ... as part of the super-resolution computations 302, the user device 102 filters pixel signals from each frame of the multiple frames 202 to generate color-specific image planes corresponding to color channels; [0060] small motions of an image may generate necessary sub-pixel offsets or displacements to perform the super-resolution computations).
It would have been obvious before the effective filing date of the claimed invention to one having ordinary skill in the art to modify the device and method of SLABAUGH to incorporate the step/system of performing super-resolution using the multiple frames taught by Chen.
The suggestion/motivation for doing so would have been to improve the resolution for demosaicing techniques ([0003] These systems and techniques offer advantages over other systems and techniques that rely on demosaicing, providing the super-resolution image of the scene without detrimental artifacts). Further, one skilled in the art could have combined the elements as described above by known method with no change in their respective functions, and the combination would have yielded nothing more than predicted results.
The combination of SLABAUGH and Chen does not expressly teach process at least a relatively lower exposure frame of the plurality of image frames and a relatively higher exposure frame of the plurality of image frames to enhance dynamic range.
However, Li teaches process at least a relatively lower exposure frame of the plurality of image frames and a relatively higher exposure frame of the plurality of image frames to enhance dynamic range of … image ([0011] a method is configured to generate a high-dynamic-range (HDR) color image from a triple-exposure-time single-shot HDR color image sensor having a plurality of short-exposure-time pixels, a plurality of medium-exposure-time pixels, and a plurality of long-exposure-time pixels; [0043] each image frame captured by image sensor 120 contains both short-exposure-time image data and long-exposure time image data).
It would have been obvious before the effective filing date of the claimed invention to one having ordinary skill in the art to modify the device and method of combination of SLABAUGH and Chen to incorporate the step/system of generating a high dynamic-range image by using both short-exposure-time image data and long-exposure time image data taught by Li.
The suggestion/motivation for doing so would have been to improve the qualities of the output image for dynamic-range imaging ([0006] it may be regarded as an object of the present document to provide an improved image generation method for generating images based short-exposure and long-exposure pixel values; [0195] method 2700 looks forward and combines the first and second dual-exposure-time images according to an optimization of desirable qualities of the output image generated by method 2700). Further, one skilled in the art could have combined the elements as described above by known method with no change in their respective functions, and the combination would have yielded nothing more than predicted results. Therefore, it would have been obvious to combine SLABAUGH and Chen with Li to obtain the invention as specified in claim 1.
Regarding claim 4, the combination of SLABAUGH and Chen with Li teaches all the limitations of claim 1 above. SLABAUGH teaches wherein the output image frame is not a fully graded image ([0045] The output is an RGB image; [0046] The input to this module 23 is a RGB image with a large dynamic range and uncorrected colors (e.g. 1024 levels per color channel)).
Regarding claim 5, the combination of SLABAUGH and Chen with Li teaches all the limitations of claim 1 above. Chen teaches wherein the resolution enhancement algorithm is based on sub-pixel shifts between the at least two image frames ([0063] A method used to render a super-resolution image of a scene … capturing, in a burst sequence, multiple frames of an image of a scene, the multiple frames having respective, relative sub-pixel offsets of the image; performing super-resolution computations using the captured, multiple frames; [0029] Algorithms that support the super-resolution computations 302 may reside in the super-resolution manager 120 of the user device 102; [0060] super resolution-computations may rely on motion that is not induced through handheld movement (e.g., small motions of an image may generate necessary sub-pixel offsets or displacements to perform the super-resolution computations)).
Regarding claim 6, the combination of SLABAUGH and Chen with Li teaches all the limitations of claim 1 above. SLABAUGH teaches wherein the one or more processors are further configured to compress the output image frame ([0018] The image processor may be configured to generate an output that is a compressed representation of an image input to the image processor. This may reduce the file size of the resulting image, allowing images to be stored more efficiently).
Regarding claim 10, the combination of SLABAUGH and Chen with Li teaches all the limitations of claim 1 above. Chen teaches wherein the video capture device a smartphone ([0017] Although illustrated as a smartphone, the user device 102 may be another type of device that has image-capture capabilities, such as a tablet or a dedicated camera).
With respect to claim 11, arguments analogous to those presented for claim 1, are applicable.
With respect to claim 14, arguments analogous to those presented for claim 4, are applicable.
With respect to claim 15, arguments analogous to those presented for claim 5, are applicable.
With respect to claim 16, arguments analogous to those presented for claim 6, are applicable.
Regarding claim 20, the combination of SLABAUGH and Chen with Li teaches all the limitations of claim 16 above. SLABAUGH teaches further comprising decompressing the output image frame ([0002] A device that reverses the data compression process (decompression) to recreate data from the original data file as closely as possible is referred to as a decoder; [0070] The decoding device 112 may output the decoded video to a video destination device 122).
Regarding claim 21, the combination of SLABAUGH and Chen with Li teaches all the limitations of claim 11 above. SLABAUGH teaches further comprising, with a computing device that is separate from the electronic device ([0078] The transceiver 5 is capable of communicating over a network with other entities 10, 11. Those entities may be physically remote from the camera 1 … Entity 10 is a computing entity ... In practice they may each be provided by one or more physical devices such as servers and datastores ... Each physical device implementing an entity comprises a processor and a memory), applying tonal processing to the output image frame to generate a fully graded image ([0046] 3) An image equalizer module, shown at 23. The image equalizer performs dynamic range compression and tone mapping to adjust colors in the image. It may also handle vignette correction and white balancing. The input to this module 23 is a RGB image with a large dynamic range and uncorrected colors (e.g. 1024 levels per color channel). The output is a color corrected RGB image with a dynamic range suitable for display on standard devices (e.g. 256 levels per color channel); [0075] FIG. 9 illustrates an image equalizer result. FIG. 9(a) shows an image input to this stage (brightened for visualization). After demosaicing, the RGB image has a large dynamic range, and in this visualization some parts of the image are underexposed (water left of the plant) and others are overexposed (sky in the background). FIG. 9(b) shows the output of this stage. The image brightness has been corrected, and the colors and exposure are improved).
Claim 2, 3, 12, 13 and 22 are rejected under 35 U.S.C. 103 as being unpatentable over SLABAUGH et al. (U.S. Publication No. 2021/0073957) (hereafter, "SLABAUGH") in view of Chen et al. (U.S. Publication No. 2021/0304359) (hereafter, "Chen") and further in view of Li et al. (U.S. Publication No. 2021/0248758) (hereafter, "Li") and Gao et al. (U.S. Publication No. 2020/0293731) (hereafter, "Gao").
Regarding claim 2, the combination of SLABAUGH and Chen with Li teaches all the limitations of claim 1 above. SLABAUGH teaches wherein the one or more processors are further configured to apply a pre-emphasis function such that the output image frame ([0070] The output of the demosaicing stage is a clean, color RGB image. However, additional processing must be performed as the image will have a dynamic range that exceeds the standard 8-bit per color channel representation required for saving to a JPG file or displaying on a device (e.g. smartphone or standard monitor). The task of the image equalizer module 23 is to transform the image after demosaicing into a suitable 8-bit representation (i.e. dynamic range compression)).
SLABAUGH does not expressly teach comprises linear image data.
However, Gao teaches comprises linear image data ([0057] FIG. 2 shows an example of piece-wise linear tone mapping ... These components result in the curve 220, which is used to map the 12-bit raw pixel values (x axis, 230) to 8-bit tone mapped pixel values (y axis, 240) … for piece-wise linear tone mapping, this three-component technique as illustrated in curve 220 is applied to every pixel in the image; [0086] At step 310, the processing device performs local tone mapping based on the parameters determined in the previous steps (e.g., the parameters determined at steps 304 and 306). The tone mapping compresses a high dynamic range image (e.g., an HDR image) to a lower-dynamic range image, such as by compressing a 10 or 12-bit HDR image to an 8-bit image).
It would have been obvious before the effective filing date of the claimed invention to one having ordinary skill in the art to modify the device and method of combination of SLABAUGH to incorporate the step/system of performing the piece-wise linear tone mapping for compressing high dynamic range image to a lower-dynamic range image taught by Gao.
The suggestion/motivation for doing so would have been to improve data loss by using developed local tone mapping for the HDR image ([0101] The contrast-based techniques can be used to improve data loss (e.g., depending on the acquired images, symbology, etc.), to enhance over-exposed (e.g., bright) regions in addition to dark regions; [0046] The inventors have developed local tone mapping techniques to individually process each pixel in the HDR image to generate a lower-bit image. The techniques can reduce data loss, can have substantial benefits compared to other techniques). Further, one skilled in the art could have combined the elements as described above by known method with no change in their respective functions, and the combination would have yielded nothing more than predicted results. Therefore, it would have been obvious to combine SLABAUGH and Gao to obtain the invention as specified in claim 2.
Regarding claim 3, the combination of SLABAUGH, Chen, Li and Gao teaches all the limitations of claim 2 above. Gao teaches wherein the pre-emphasis function preserves dynamic range of the image data ([0086] At step 310, the processing device performs local tone mapping based on the parameters determined in the previous steps (e.g., the parameters determined at steps 304 and 306). The tone mapping compresses a high dynamic range image (e.g., an HDR image) to a lower-dynamic range image, such as by compressing a 10 or 12-bit HDR image to an 8-bit image; [0094] At step 312, the processing device can be configured to adjust the pixel intensity values calculated in step 310 … the processing device can be configured to stretch the intensity of the tone mapping image to fully utilize the 8-bit dynamic range for the resulting compressed image … the processing device can perform a linear stretch using the minimum and maximum pixel values determined in step 304).
With respect to claim 12, arguments analogous to those presented for claim 2, are applicable.
With respect to claim 13, arguments analogous to those presented for claim 3, are applicable.
Regarding claim 22, the combination of SLABAUGH and Chen with Li teaches all the limitations of claim 21 above. Chen teaches wherein the electronic device is a smartphone and ([0017] Although illustrated as a smartphone, the user device 102 may be another type of device that has image-capture capabilities, such as a tablet or a dedicated camera).
Chen does not expressly teach the computing device is a laptop or desktop computer.
However, Gao teaches the computing device is a laptop or desktop computer ([0127] a computer may be embodied in any of a number of forms, such as a rack-mounted computer, a desktop computer, a laptop computer, or a tablet computer; [0053] The image processing system 10 also includes a computer or processor 14).
It would have been obvious before the effective filing date of the claimed invention to one having ordinary skill in the art to modify the device and method of combination of Chen to incorporate the step/system of using the desktop or laptop computer for the tone mapping taught by Gao.
Motivation for this combination has been stated in claim 2.
Claim 7-9 and 17-19 are rejected under 35 U.S.C. 103 as being unpatentable over SLABAUGH et al. (U.S. Publication No. 2021/0073957) (hereafter, "SLABAUGH") in view of Chen et al. (U.S. Publication No. 2021/0304359) (hereafter, "Chen") and further in view of Li et al. (U.S. Publication No. 2021/0248758) (hereafter, "Li") and SIDDARAMANNA et al. (U.S. Publication No. 2022/0256169) (hereafter, "SIDDARAMANNA").
Regarding claim 7, the combination of SLABAUGH and Chen with Li teaches all the limitations of claim 6 above. The combination of SLABAUGH and Chen with Li does not expressly teach wherein the compression comprises application of a discrete-cosine transform (DCT)-based compression algorithm.
However, SIDDARAMANNA teaches wherein the compression comprises application of a discrete-cosine transform (DCT)-based compression algorithm ([0104] In the transform 340, the transform and quantization engine of the encoder 305 transforms the residual coding block into residual transform coefficients using a transform, such as a discrete cosine transform (DCT), a modified discrete cosine transform (MDCT), a discrete sine transform (DST), a fast Fourier transform (FFT), a wavelet transform, or a combination thereof. The transform 340 can be a lossy compression scheme).
It would have been obvious before the effective filing date of the claimed invention to one having ordinary skill in the art to modify the device and method of combination of SLABAUGH and Chen with Li to incorporate the step/system of using a discrete cosine transform (DCT) for compression taught by SIDDARAMANNA.
The suggestion/motivation for doing so would have been to improve video quality for video compression ([0096] A video encoder that can accurately estimate quad-tree partitioning of the LCU can perform efficient compression of video data; [0100] Compression of the video using optimized block partitioning 250 can reduce bit rate R while maintaining the same video quality (e.g., the same distortion D) relative to the same video compressed without optimized block partitioning 250. Similarly, compression of the video using optimized black partitioning 250 can increase video quality (e.g., reduce the distortion D) while maintaining the same bit rate R relative to the same video compressed without optimized block partitioning 250). Further, one skilled in the art could have combined the elements as described above by known method with no change in their respective functions, and the combination would have yielded nothing more than predicted results. Therefore, it would have been obvious to combine SLABAUGH, Chen and Li with SIDDARAMANNA to obtain the invention as specified in claim 7.
Regarding claim 8, the combination of SLABAUGH, Chen and Li with SIDDARAMANNA teaches all the limitations of claim 7 above. SIDDARAMANNA teaches wherein the application of the compression algorithm comprises application of a combination of Huffman and Golomb coding ([0104] Entropy coding 360 can be a lossless data compression scheme. The entropy coding 360 may include, for example, context-adaptive binary arithmetic coding (CABAC), arithmetic coding, Golomb coding, Huffman coding, range coding, Shannon coding, Shannon-Fano coding, Shannon-Fano-Elias coding, Tunstall coding, unary coding, universal coding, or a combination thereof).
Regarding claim 9, the combination of SLABAUGH, Chen and Li with SIDDARAMANNA teaches all the limitations of claim 6 above. SIDDARAMANNA teaches the compression comprises application of compression rate control ([0104] The transform 340 can be a lossy compression scheme. In the quantization 350, the transform and quantization engine of the encoder 305 quantizes the residual transform coefficients, reducing the bit rate R 365. A degree of quantization performed by the transform and quantization engine of the encoder 305 during the quantization 350 can be based on a quantization parameter (QP), and can be modified by adjusting the QP. The QP indicates a quantization step size for a video frame during quantization, and controls how much spatial detail is retained from the captured image after quantization. As the QP decreases, more image detail is retained during compression, leading to an increase in video quality and an increase in bit rate R 365. As the QP increases, more of the image detail is aggregated during compression, leading to a decrease in video quality and a decrease in bit rate R 365).
With respect to claim 17, arguments analogous to those presented for claim 7, are applicable.
With respect to claim 18, arguments analogous to those presented for claim 8, are applicable.
With respect to claim 19, arguments analogous to those presented for claim 9, are applicable.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DANIEL C. CHANG whose telephone number is (571)270-1277. The examiner can normally be reached Monday-Thursday and Alternate Fridays 8:00-5:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Chan S. Park can be reached on (571) 272-7409. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/DANIEL C CHANG/Examiner, Art Unit 2669
/CHAN S PARK/Supervisory Patent Examiner, Art Unit 2669
Prosecution Insights