DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submitted is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Priority
Acknowledgment is made of applicant’s claim for foreign priority under 35 U.S.C. 119 (a)-(d). The certified copy has been filed in parent Application No. KR10-2022-0144620 and KR10-2022-0114494, filed on 11/02/2022 and 09/08/2022 respectively.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The text of those sections of Title 35, U.S. Code not included in this action can be found in a prior Office action.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1 – 3, 7 – 15 are rejected under 35 U.S.C. 103 as being unpatentable over Yang et al. (See Machine Translation for KR 20190103047 A; hereafter referred to as Yang) in view of Kim et al. (US 20210004653 A1; hereafter referred to as Kim).
Regarding Claim 1, Yang teaches:
An image processing device comprising:
a memory storing one or more instructions (Yang, [0098] “The storage 140 may store a program for processing. The storage 140 may store a program for processing and controlling each signal in the signal processor 170, or may store a signal-processed video, audio, or data signal”); and
at least one processor configured to execute the one or more instructions (Yang, [0108] “the signal processor 170 may control overall operations of the image display apparatus 100”) to:
receive additional data to perform an image processing operation for input image data (Yang, [0222] “the signal processing unit 170m includes a quality calculating unit 632 for calculating information about an image type of an input image, an image quality setting unit 634 for setting image quality based on information about an image type, and According to the image quality, an image quality processing unit 635 which performs image quality processing may be provided”),
based on the additional data, (Yang, [0265] – [0267] “the quality calculator 632 and the quality learner 633 may perform quality calculation and quality learning, respectively, using a deep neural network… the quality learning unit 633 has a predetermined value in which a difference between the information about the input image type and the information about the image type of the calculated input image is a predetermined value while the information on the image type of the input image is input”), and
(Yang, [0457] “The quality learning unit 633 in the signal processing unit 170 may use the updated deep learning detector model in the next iteration. The quality learner 633 in the signal processor 170 may repeat this process until the performance converges or a predetermined number of iterations”).
However, Yang does not explicitly recite:
based on the additional data, determine a number of operations of a neural network model trained to perform the image processing operation on the input image data, and
based on the determined number of the operations of the neural network model, use the neural network model to generate output image data by performing the image processing operation on the input image data.
In the same field of endeavor, Kim teaches:
based on the additional data, determine a number of operations of a neural network model trained to perform the image processing operation on the input image data (Kim, [0087] “the controller 410 may control the plurality of neural network operators to perform the neural network operation on the image data split based on the size of the image and the number of the plurality of neural network operators”), and
based on the determined number of the operations of the neural network model, use the neural network model to generate output image data by performing the image processing operation on the input image data (Kim, [0087] “the controller 410 may control the plurality of neural network operators to perform the neural network operation on the image data split based on the size of the image and the number of the plurality of neural network operators and output the upscaled image data”).
Yang and Kim are considered analogous art as they are reasonably pertinent to the same field of endeavor of image processing. Therefore, it would have been obvious to one of the ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Yang with the invention of Kim to determine a number of operations of a neural network model trained to perform the image processing operation on the input image data and based on the determined number of the operations of the neural network model, use the neural network model to generate output image data; doing so the processor is capable of adaptively operating to perform neural network operations to reduce resources or power consumption of hardware of a terminal device required for image processing (Kim, [0006] – [0007]); thus, one of the ordinary skill in the art would have been motivated to combine the references.
Regarding Claim 2, Yang in view of Kim teaches the image processing device of claim 1, wherein the neural network model comprises a layer unit including a plurality of layers (Yang, [0460] “the quality learning unit 633 or the quality calculating unit 632 in the signal processing unit 170 may include the number and order of layers used, such as convolution, pooling, and fully connected layers, characteristics such as kernel size and stride of each layer, and deep learning”), and
wherein the at least one processor is further configured to, by repeating the layer unit same time as the number of operations to repeatedly perform the image processing operation same time as the number of operations performed on the input image data, generate the output image data (Yang, [0457] “The quality learning unit 633 in the signal processing unit 170 may use the updated deep learning detector model in the next iteration. The quality learner 633 in the signal processor 170 may repeat this process until the performance converges or a predetermined number of iterations”, Yang, [0301] “the image quality processing unit 635, after downscaling the image signal according to the original resolution of the image signal, performs image quality processing on the down-scaled image signal, up-scales the image signal subjected to the image quality processing, The upscaled video signal may be output”).
Regarding Claim 3, Yang in view of Kim teaches the image processing device of claim 1, wherein the at least one processor is further configured to:
apply to the neural network model a parameter corresponding to a round in which the neural network model performs the image processing operation, based on the number of operations (Kim, [0077]- [0078] “The controller 410 may include various processing circuitry and serve to set a parameter required for an operation of the operator 420. The operator 420 may include various processing circuitry and/or executable program elements and perform the neural network operation based on the set parameter… different parameters may be required for the neural network operation for each image frame. The ‘parameter’ may include a plurality of weight values used in an operation process of each neural network layer”; Yang, [0075] “The image display apparatus 100 may update a parameter for the deep neural network and calculate a resolution and a noise level of the received video signal based on the updated parameter. Accordingly, it is possible to accurately perform the original quality calculation of the video signal on a learning basis”), and
generate the output image data by performing the image processing operation on the input image data (Kim, [0084] “the controller 410 may control some of the plurality of neural network operators to perform the neural network operation on image data split based on the size of the image and the data processing capabilities of the plurality of neural network operators and output upscaled image data”; Yang, [0301] “the image quality processing unit 635, after downscaling the image signal according to the original resolution of the image signal, performs image quality processing on the down-scaled image signal, up-scales the image signal subjected to the image quality processing, The upscaled video signal may be output”).
The reasons for combining Yang and Kim are similar to that stated in the rejection of claim 1. In addition, this same reasoning is pertinent and applicable to the rejections of claims 7 – 10 below.
Regarding Claim 7, Yang in view of Kim teaches the image processing device of claim 1, wherein the additional data comprises resolution information of the input image data (Yang, [0074] “the image display apparatus 100 may calculate a resolution, a noise level, and the like of a received image signal using a deep neural network”), and
wherein the at least one processor is further configured to determine the number of operations, based on the resolution information (Kim, [0060] “The video processor 320 may include various video processing circuitry and perform various preset image processing operations on image data. The video processor 320 may output an output signal generated or combined by performing such image processing to the display 330 such that an image corresponding to the image data is displayed on the display 330”; Kim [0053] “The AI upscaler 224 may determine an upscale target of the second image 135 based on at least one of the difference information or the information related to the first image 115 included in the AI data. The upscale target may indicate, for example, to what degree of resolution the second image 135 needs to be upscaled. When the upscale target is determined, the AI upscaler 224 may AI upscale the second image 135 using a neural network operation to generate a third image 145 corresponding to the upscale target”).
Regarding Claim 8, Yang in view of Kim teaches the image processing device of claim 7, wherein the at least one processor is further configured to determine the number of operations when the resolution information corresponds to a first resolution, to be larger than when the resolution information corresponds to a second resolution which is lower than the first resolution (Kim, [0033] “when the original image 105 is received, the original image 105 may be AI downscaled 110 to generate the first image 115 of a predetermined resolution or a predetermined image quality. The AI downscaling 110 may be performed on an AI basis, and AI for the AI downscaling 110 may be trained connectively with AI for the AI upscaling 140 of the second image 135. This is because a difference between the original image 105 that is an AI encoding target and the third image 145 reconstructed through AI decoding increases when the AI for the AI downscaling 110 and the AI for the AI upscaling 140 may be trained separately”; Kim [0087] “the controller 410 may control the plurality of neural network operators to perform the neural network operation on the image data split based on the size of the image and the number of the plurality of neural network operators and output the upscaled image data”; Kim [0136]”).
Regarding Claim 9, Yang in view of Kim teaches the image processing device of claim 1, wherein the at least one processor is further configured to:
generate, from the input image data, reconstructed image data items having a smaller unit than a size of the input image data (Kim, [0084] “the controller 410 may control some of the plurality of neural network operators to perform the neural network operation on image data split based on the size of the image”), and
use the neural network model to repeatedly perform, on each of the reconstructed image data items, the image processing operation same time as the number of operations, and generate the output image data (Yang, [0457] “The quality learning unit 633 in the signal processing unit 170 may use the updated deep learning detector model in the next iteration. The quality learner 633 in the signal processor 170 may repeat this process until the performance converges or a predetermined number of iterations”; Kim, [0084] “the controller 410 may control some of the plurality of neural network operators to perform the neural network operation on image data split based on the size of the image and the data processing capabilities of the plurality of neural network operators and output upscaled image data”).
Regarding Claim 10, Yang in view of Kim teaches the image processing device of claim 1, wherein the at least one processor is further configured to:
generate corrected image data by performing motion correction on the input image data (Kim, [0029] “the encoding data may include mode information (e.g., prediction mode information, motion information, etc.) used for the first encoding 120 of the first image 115”), and
use the corrected image data and the neural network model to generate the output image data by performing the image processing operation on the input image data (Kim, [0046] “The first decoder 222 may reconstruct the second image 135 corresponding to the first image 115 based on the encoding data. The second image 135 generated by the first decoder 222 may be provided to the AI upscaler 224. According to an implementation example, first decoding related information such as mode information MODE INFORMATION (e.g., prediction mode information, motion information, etc.)”).
Regarding Claim 11, Yang teaches:
An operating method of an image processing device, the operating method comprising:
receiving additional data to perform an image processing operation for input image data (Yang, [0222] “the signal processing unit 170 includes a quality calculating unit 632 for calculating information about an image type of an input image, an image quality setting unit 634 for setting image quality based on information about an image type, and According to the image quality, an image quality processing unit 635 which performs image quality processing may be provided”),
based on the additional data, (Yang, [0265] – [0267] “the quality calculator 632 and the quality learner 633 may perform quality calculation and quality learning, respectively, using a deep neural network… the quality learning unit 633 has a predetermined value in which a difference between the information about the input image type and the information about the image type of the calculated input image is a predetermined value while the information on the image type of the input image is input”), and
using the neural network model, and, (Yang, [0457] “The quality learning unit 633 in the signal processing unit 170 may use the updated deep learning detector model in the next iteration. The quality learner 633 in the signal processor 170 may repeat this process until the performance converges or a predetermined number of iterations”).
However, Yang does not explicitly recite:
based on the additional data, determine a number of operations of a neural network model trained to perform the image processing operation on the input image data, and
using the neural network model, and, based on the number of operations, generating output image data by performing the image processing operation on the input image data.
In the same field of endeavor, Kim teaches:
based on the additional data, determine a number of operations of a neural network model trained to perform the image processing operation on the input image data (Kim, [0087] “the controller 410 may control the plurality of neural network operators to perform the neural network operation on the image data split based on the size of the image and the number of the plurality of neural network operators”), and
using the neural network model, and, based on the number of operations, generating output image data by performing the image processing operation on the input image data (Kim, [0087] “the controller 410 may control the plurality of neural network operators to perform the neural network operation on the image data split based on the size of the image and the number of the plurality of neural network operators and output the upscaled image data”).
Yang and Kim are considered analogous art as they are reasonably pertinent to the same field of endeavor of image processing. Therefore, it would have been obvious to one of the ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Yang with the invention of Kim to determine a number of operations of a neural network model trained to perform the image processing operation on the input image data and based on the determined number of the operations of the neural network model, use the neural network model to generate output image data; doing so the processor is capable of adaptively operating to perform neural network operations to reduce resources or power consumption of hardware of a terminal device required for image processing (Kim, [0006] – [0007]); thus, one of the ordinary skill in the art would have been motivated to combine the references.
Regarding Claim 12, Yang in view of Kim teaches the operating method of claim 11, wherein the neural network model comprises a layer unit including a plurality of layers (Yang, [0460] “the quality learning unit 633 or the quality calculating unit 632 in the signal processing unit 170 may include the number and order of layers used, such as convolution, pooling, and fully connected layers, characteristics such as kernel size and stride of each layer, and deep learning”), and
wherein the generating of the output image data comprises repeating the layer unit same time as the number of operations to repeatedly perform, on the input image data, the image processing operation same time as the number of operations (Yang, [0457] “The quality learning unit 633 in the signal processing unit 170 may use the updated deep learning detector model in the next iteration. The quality learner 633 in the signal processor 170 may repeat this process until the performance converges or a predetermined number of iterations”, Yang, [0301] “the image quality processing unit 635, after downscaling the image signal according to the original resolution of the image signal, performs image quality processing on the down-scaled image signal, up-scales the image signal subjected to the image quality processing, The upscaled video signal may be output”).
Regarding Claim 13, Yang in view of Kim teaches the operating method of claim 11, wherein the generating of the output image data comprises applying a parameter corresponding to a round in which the neural network model performs the image processing operation, based on the number of operations, to the neural network model (Kim, [0077]- [0078] “The controller 410 may include various processing circuitry and serve to set a parameter required for an operation of the operator 420. The operator 420 may include various processing circuitry and/or executable program elements and perform the neural network operation based on the set parameter… different parameters may be required for the neural network operation for each image frame. The ‘parameter’ may include a plurality of weight values used in an operation process of each neural network layer”; Yang, [0075] “The image display apparatus 100 may update a parameter for the deep neural network and calculate a resolution and a noise level of the received video signal based on the updated parameter. Accordingly, it is possible to accurately perform the original quality calculation of the video signal on a learning basis”).
The reasons for combining Yang and Kim are similar to that stated in the rejection of claim 11. In addition, this same reasoning is pertinent and applicable to the rejections of claims 14 – 15 below.
Regarding Claim 14, Yang in view of Kim teaches the operating method of claim 11, wherein the determining of the number of operations comprises determining the number of operations when resolution information corresponds to a first resolution, to be larger than when the resolution information corresponds to a second resolution which is lower than the first resolution (Kim, [0033] “when the original image 105 is received, the original image 105 may be AI downscaled 110 to generate the first image 115 of a predetermined resolution or a predetermined image quality. The AI downscaling 110 may be performed on an AI basis, and AI for the AI downscaling 110 may be trained connectively with AI for the AI upscaling 140 of the second image 135. This is because a difference between the original image 105 that is an AI encoding target and the third image 145 reconstructed through AI decoding increases when the AI for the AI downscaling 110 and the AI for the AI upscaling 140 may be trained separately”; Kim, [0087] “the controller 410 may control the plurality of neural network operators to perform the neural network operation on the image data split based on the size of the image and the number of the plurality of neural network operators and output the upscaled image data”; Kim [0136]”).
Regarding Claim 15, Yang in view of Kim teaches the operating method of claim 11, wherein the determining of the number of operations comprises generating, from the input image data, reconstructed image data items having a smaller unit than a size of the input image data (Kim, [0084] “the controller 410 may control some of the plurality of neural network operators to perform the neural network operation on image data split based on the size of the image”), and
wherein the generating of the output image data comprises, by using the neural network model to repeatedly perform, on each of the reconstructed image data items, the image processing operation same time as the number of operations, generating the output image data (Yang, [0457] “The quality learning unit 633 in the signal processing unit 170 may use the updated deep learning detector model in the next iteration. The quality learner 633 in the signal processor 170 may repeat this process until the performance converges or a predetermined number of iterations”; Kim, [0084] “the controller 410 may control some of the plurality of neural network operators to perform the neural network operation on image data split based on the size of the image and the data processing capabilities of the plurality of neural network operators and output upscaled image data”).
Claim 6 is rejected under 35 U.S.C. 103 as being unpatentable over Yang et al. (See Machine Translation for KR 20190103047 A; hereafter referred to as Yang) in view of Kim et al. (US 20210004653 A1; hereafter referred to as Kim) further in view of An et al. (US 20210150660 A1; hereafter referred to as An).
Regarding Claim 6, Yang in view of Kim teaches the method of claim 1, but does not explicitly recite:
use image data having a Bayer pattern as input image data, and use RGB image data as the output image data.
In the same field of endeavor, An teaches:
use image data having a Bayer pattern as input image data, and use RGB image data as the output image data (An, (Fig. 2, Raw data (Bayer image), RGB image data).
Yang, Kim and An are considered analogous art as they are reasonably pertinent to the same field of endeavor of image processing. Therefore, it would have been obvious to one of the ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Yang in view if Kim with the invention of An to use input data having Bayer patterns and RGB data as output image data; doing so sensing data provided with various filters can be converted into pixel data through the image processing process and transmitted to the controller for data analysis and quality assessment (An, [0126]); thus, one of the ordinary skill in the art would have been motivated to combine the references.
Allowable Subject Matter
Claims 4 – 5 are objected to as being dependent upon a rejected base claim but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
US 20220284548 A1 SYSTEM AND METHOD OF AUTOMATIC IMAGE ENHANCEMENT USING SYSTEM GENERATED FEEDBACK MECHANISM A method for automatically enhancing image data comprising receiving image data. Utilizing a neural network to analyze the image data to predict how each of a plurality of image enhancement processes will affect the image data. Utilizing the neural network to calculate a reward value for each of the plurality of enhancement processes that can be applied to the image data. Determining if the image data should be enhanced or not, wherein the determination is based on the predictions how each of the plurality of image enhancement processes will affect the image data.
US 20210104018 A1 METHOD AND APPARATUS FOR ENHANCING RESOLUTION OF IMAGE A method for enhancing the resolution of an image according to an embodiment of the present disclosure can include loading image data including a low resolution image and metadata of the image data, analyzing metadata including information related to an image processing artificial neural network to be applied to the low resolution image in the image data, selecting the image processing artificial neural network to be applied to the low resolution image from a plurality of image processing artificial neural networks, based on the metadata, and generating a high resolution image by processing the low resolution image according to the selected image processing artificial neural network.
Contact Information
Any inquiry concerning this communication or earlier communications from the examiner should be directed to VAISALI RAO KOPPOLU whose telephone number is (571)270-0273. The examiner can normally be reached Monday - Friday 8:30 - 5.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Jennifer Mehmood can be reached at (571) 272-2976. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
VAISALI RAO. KOPPOLU
Examiner
Art Unit 2664
/VAISALI RAO KOPPOLU/Examiner of Art Unit 2664
Prosecution Insights