DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's RCE submission filed on 06/26/2026 has been entered.
Priority
Receipt is acknowledged that application is a National Stage application of PCT/KR2022/011565 dated 08/04/2022. Receipt is acknowledged that application claims priority to foreign application with application number KR10-2021-0103284 dated 08/05/2021, application number KR10-2021-0106985 dated 08/12/2021, application number KR10-2021-0106986 08/12/2021. Copies of certified papers required by 37 CFR 1.55 have been received. Priority is acknowledged under 35 USC 119(e) and 37 CFR 1.78. Claims 8-20 have been afforded the benefit of filing date 08/05/2021.
Information Disclosure Statement
The information disclosure statement filed 12/16/2025 fails to comply with 37 CFR 1.98(a)(2), which requires a legible copy of each cited foreign patent document; each non-patent literature publication or that portion which caused it to be listed; and all other information or that portion which caused it to be listed. It has been placed in the application file, but the information referred to therein has not been considered.
A concise explanation and/or English language translation of the cited Office Action has not been provided. See 37 CFR 1.98.
Response to Amendment
The amendment filed 06/26/2026 has been entered. Applicant’s amendment to the title has overcome the objection previously set forth in the Final Office Action mailed 03/26/2025. Claims 8-20 remain pending in the application, with claims 1-7 having been previously cancelled.
Response to Arguments
In light of the amendments to the claims, the interpretations under 35 U.S.C. 112(f) previously set forth in the Final Office Action mailed 03/26/2025 have been withdrawn.
Applicant’s arguments with respect to the 35 U.S.C. 103 rejections have been considered but are moot because the new ground of rejection does not rely on any combination of references applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Claim Interpretation
Regarding claim 19, the claims are interpreted to require at least one element of the claimed list, in accordance with the plain English meaning of “at least one among” the listed elements.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 8, 10-12, 15, and 17-20 are rejected under 35 U.S.C. 103 as being unpatentable over Kim et al. (U.S. Patent No. 2022/0182537 A1), hereinafter Kim, in view of Lubana et al. (U.S. Patent No. 2021/0089803 A1), hereinafter Lubana.
Regarding claim 8, Kim teaches an image processing method of an image sensor (Kim, camera module w/ image sensor, para 31: “The IP network module 22 may receive first image data generated by the image sensor 11 of the camera module 10”; also see abstract), the method comprising the steps of: generating a first image data by light transmitted through a display panel (Kim, para 34: “the camera module 10 may generate the image data (e.g., first image data as described herein) based on light reflected from the subject, passing through the display 60, and reaching the optical lens of the image sensor”); and
outputting a second image data from the first image data by a learned deep learning neural network (Kim, para 31: “The IP network module 22 may receive first image data generated by the image sensor 11 of the camera module 10 and may generate second image data by performing the image processing operations on the first image data”; para 37: “The NN of the multilayered structure may be referred to as a deep neural network (DNN) or a deep learning architecture”; see also FIG 2 for the NN and output data in para 83) implemented in a hardware accelerator (Kim, para 26: “The neural network processor 20 may be implemented by a neural network operation accelerator, a coprocessor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a graphics processing unit (GPU), a neural processing unit (NPU), a tensor processing unit (TPU), or a multi-processor system-on-chip (MPSoC)”),
wherein the second image data is image data in which at least a portion of noise, which is a picture quality degradation phenomenon that occurs when the light transmits through a display panel (Kim, para 34: “A path of the light reflected from the subject may be changed by the pixels included in the display 60 while passing through the display 60 so that the image data obtained by capturing the subject may be distorted”), is removed (Kim, para 30: “denoise operation”; para 35: “removing the distortion in the image data generated by the UDC module”; see also para 48 where noise is removed from the first image data).
Kim teaches wherein the embodiments of the disclosed invention are modifiable (Kim, para 93: “FIGS. 8A to 8C are views illustrating modifiable embodiments of the image processing apparatus 1000 of FIG. 3”), but fails to explicitly teach wherein the hardware accelerator is formed inside the image sensor. However, Lubana teaches an image processing method of an image sensor wherein a hardware accelerator, including a deep learning neural network, is formed inside the image sensor (Lubana, para 55-56: “one or more modifications may be directed to establishing a deep learning accelerator in or near (i.e., at) the image sensor. Examples of a deep learning accelerator and/or other minimalistic image signal processing for machine vision applications are described below. In-sensor and near-sensor energy-efficient deep learning accelerators have the capability to enable the use of deep neural networks in embedded vision applications”, emphasis added). It would have been obvious to a person having ordinary skill in the art, before the effective filing date of the claimed invention, to have combined the in-sensor deep learning accelerator of Lubana with the method of Kim in order to increase the efficiency of the image processing method by decreasing energy consumption (Lubana, see citation above).
Regarding claim 10 (dependent on claim 8), Kim in view of Lubana teaches wherein the first image data is received from an image sensor disposed under the display panel (Kim, image sensor 11 of para 34 - para 34: “the camera module 10 may be disposed (e.g., located) under the display 60, such that the display 60 is located between the camera module 10 and an exterior environment”), and wherein the second image data is outputted to an image signal processor (Kim, See FIG 8C where the output of the IP network module, the neural network, is to another processor; see further wherein the processor may be an ISP in para 118: “may include, may be included in, and/or may be implemented by one or more instances of processors such as hardware including logic circuits…For example, a processor as described herein more specifically may include, but is not limited to…an Image Signal Processor (ISP)”; see para 105-107 regarding FIG 8C and para 93: “FIGS. 8A to 8C are views illustrating modifiable embodiments of the image processing apparatus 1000 of FIG. 3”).
Regarding claim 11, Kim teaches an image sensor (Kim, 4100 of FIG 8C, para 105: “camera module 4100”; para 93: “FIGS. 8A to 8C are views illustrating modifiable embodiments of the image processing apparatus 1000 of FIG. 3”) comprising:
an image sensing circuitry (Kim, para 105: “image sensor 4110”) configured to generate a first image data by light transmitted through a display panel (Kim, para 34: “the camera module 10 may generate the image data (e.g., first image data as described herein) based on light reflected from the subject, passing through the display 60, and reaching the optical lens of the image sensor 11”; FIG 8C is in accordance with FIGs 1 and 3, see para 45 and 93); and
a hardware accelerator comprising a deep learning neural network (Kim, para 26: “The neural network processor 20 may be implemented by a neural network operation accelerator, a coprocessor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a graphics processing unit (GPU), a neural processing unit (NPU), a tensor processing unit (TPU), or a multi-processor system-on-chip (MPSoC)”; the image sensor does not explicitly comprise the accelerator, see description below) configured to output a second image data from the first image data (Kim, para 31: “The IP network module 22 may receive first image data generated by the image sensor 11 of the camera module 10 and may generate second image data by performing the image processing operations on the first image data”; para 37: “The NN of the multilayered structure may be referred to as a deep neural network (DNN) or a deep learning architecture”; see also FIG 2 for the NN and output data in para 83);
wherein the second image data is image data from which at least a portion of noise, which is a picture quality degradation phenomenon that occurs when the light transmits through the display panel (Kim, para 34: “A path of the light reflected from the subject may be changed by the pixels included in the display 60 while passing through the display 60 so that the image data obtained by capturing the subject may be distorted”), is removed (Kim, para 30: “denoise operation”; para 35: “removing the distortion in the image data generated by the UDC module”; see also para 48 where noise is removed from the first image data).
Kim teaches wherein the embodiments of the disclosed invention are modifiable (Kim, para 93: “FIGS. 8A to 8C are views illustrating modifiable embodiments of the image processing apparatus 1000 of FIG. 3”), but fails to explicitly teach wherein the image sensor comprises the hardware accelerator and the hardware accelerator is formed inside the image sensor. However, Lubana teaches an image processing method using an image sensor wherein a hardware accelerator, including a deep learning neural network, is formed inside the image sensor (Lubana, para 55-56: “one or more modifications may be directed to establishing a deep learning accelerator in or near (i.e., at) the image sensor. Examples of a deep learning accelerator and/or other minimalistic image signal processing for machine vision applications are described below. In-sensor and near-sensor energy-efficient deep learning accelerators have the capability to enable the use of deep neural networks in embedded vision applications”). Therefore, the image sensor comprises the hardware accelerator. It would have been obvious to a person having ordinary skill in the art, before the effective filing date of the claimed invention, to have combined the in-sensor deep learning accelerator of Lubana with the image sensor of Kim in order to increase the efficiency of the image processing method by decreasing energy consumption (Lubana, see citation above).
Regarding claim 12 (dependent on claim 11), Kim in view of Lubana teaches comprising: an output interface circuitry (Kim, software of the IP network module that outputs the processed data, see next citation) configured to output the second image data to outside from the image sensor (Kim, See FIG 8C where the output of the IP network module, the neural network, is to another processor – para 107: “The IP network module 4120 may transmit the second image data IDTb to the main processor 4300”; see further wherein the processor may be an ISP in para 118: “may include, may be included in, and/or may be implemented by one or more instances of processors such as hardware including logic circuits…For example, a processor as described herein more specifically may include, but is not limited to…an Image Signal Processor (ISP)”; see para 105-107 regarding FIG 8C and para 93: “FIGS. 8A to 8C are views illustrating modifiable embodiments of the image processing apparatus 1000 of FIG. 3”), wherein the deep learning neural network outputs the second image data according to an output format of the output interface circuitry (Kim, See FIG 4A-4C where the output of the neural module changes depending on the subsequent image processing. In the relied upon embodiment, the output format is Bayer data; para 68: “For example, the IP network module 210b may remove noise and blur from the second tetra data IDTb and may convert the second tetra data IDTb into image data in the Bayer pattern”).
Regarding claim 15 (dependent on claim 11), Kim in view of Lubana teaches wherein the first image data and the second image data have different noise levels (Kim, since the neural network output of Kim, mapped to second image data, has noise removed from the first image data, the two image datasets have different noise levels; see further para 30 and 48).
Regarding claim 17 (dependent on claim 11), Kim in view of Lubana teaches wherein at least one of the first image data or the second image data is Bayer image data (Kim, output from the neural network, mapped to second image data, is Bayer data - see FIG 4B and para 70: “The main processor 300b may receive the Bayer data IDTc from the neural network processor 200b”).
Regarding claim 18 (dependent on claim 11), Kim in view of Lubana teaches wherein the second image data is outputted to an image signal processor outside the image sensor (Kim, See FIG 8C where the output of the IP network module, the neural network, is to another processor – para 107: “The IP network module 4120 may transmit the second image data IDTb to the main processor 4300”; see further wherein the processor may be an ISP in para 118: “may include, may be included in, and/or may be implemented by one or more instances of processors such as hardware including logic circuits…For example, a processor as described herein more specifically may include, but is not limited to…an Image Signal Processor (ISP)”; see para 105-107 regarding FIG 8C and para 93: “FIGS. 8A to 8C are views illustrating modifiable embodiments of the image processing apparatus 1000 of FIG. 3”).
Regarding claim 19 (dependent on claim 8), Kim in view of Lubana teaches wherein the noise comprises at least one among low intensity, blur, haze, reflection ghost, color separation, flare, fringe pattern, and yellowish phenomenon (Kim, para 89: “The raw image generated by the UDC module may include distortions such as blur, ghost, haze, and flare”; para 4: “Such image processing operations may include removing defects such as blur, ghost, flare, and haze included in image data generated by a camera module (also referred to herein as a camera)”).
Regarding claim 20 (dependent on claim 8), Kim in view of Lubana teaches wherein the first image data and the second image data have different noise levels (Kim, since the neural network output of Kim, mapped to second image data, has noise removed from the first image data, the two image datasets have different noise levels; see further para 30 and 48).
Claims 9 and 16 are rejected under 35 U.S.C. 103 as being unpatentable over Kim in view of Lubana, in further view of Do et al. (U.S. Patent No. 2022/0353401 A1), hereinafter Do.
Regarding claim 9 (dependent on claim 8), Kim in view of Lubana teaches wherein a training set of the deep learning neural network comprises a first image data generated by light transmitted through a display panel (Kim, images generated by the UDC, para 89: “the IP network module 22 may be learned (e.g., a neural network model used by the IP network module 22 may be trained) based on using the image (e.g., raw image) generated by the UDC module as the input data”), but fails to explicitly teach wherein a second image data generated by light not transmitted through a display panel (Kim teaches a second image data for training, but it is corrected UDC data, para 89).
However, Do teaches a similar method (Do, para 90: “the electronic device may correct the color around the light source to be similar to a color around the light source had the image been captured by a camera that was not an UDC”) including a training set wherein a second image data is generated by light not transmitted through a display panel (Do, para 90: “For example, the dataset for light source processing may include a pair of an input image obtained by capturing a scene (e.g., a scene including a light source with a preset brightness or greater) that satisfies a rainbow artifact occurring condition by a UDC and a ground truth image obtained by capturing the same scene by an externally exposed camera module”, emphasis added).
Thus, Kim and Do each disclose a method for removing noise from a UDC-captured image using training data to train a model. A person of ordinary skill in the art, before the effective filing date of the claimed invention, would have recognized that the training set of Kim could have been substituted for the training set of Do because both serve the purpose of utilizing a second training dataset with images that contain less UDC noise than that of the first training dataset. Furthermore, a person of ordinary skill in the art would have been able to carry out the substitution. Finally, the substitution achieves the predictable result of training a machine learned model for noise reduction in an image.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to substitute the training set of Kim in view of Lubana with the training set of Do according to known methods to yield the predictable result of teaching the model features of images captured by an externally exposed camera, compared to that of a UDC camera.
Regarding claim 16 (dependent on claim 11), Kim in view of Lubana teaches wherein a training set of the deep learning neural network comprises a first image data generated by light transmitted through a display panel (Kim, images generated by the UDC, para 89: “the IP network module 22 may be learned (e.g., a neural network model used by the IP network module 22 may be trained) based on using the image (e.g., raw image) generated by the UDC module as the input data”), but fails to explicitly teach a second image data generated by light not transmitted through a display panel (Kim teaches a second image data for training, but it is corrected UDC data, para 89).
However, Do teaches a similar method (Do, para 90: “the electronic device may correct the color around the light source to be similar to a color around the light source had the image been captured by a camera that was not an UDC”) including a training set wherein a second image data is generated by light not transmitted through a display panel (Do, para 90: “For example, the dataset for light source processing may include a pair of an input image obtained by capturing a scene (e.g., a scene including a light source with a preset brightness or greater) that satisfies a rainbow artifact occurring condition by a UDC and a ground truth image obtained by capturing the same scene by an externally exposed camera module”, emphasis added).
Thus, Kim and Do each disclose a system/method for removing noise from a UDC-captured image using training data to train a model. A person of ordinary skill in the art, before the effective filing date of the claimed invention, would have recognized that the training set of Kim could have been substituted for the training set of Do because both serve the purpose of utilizing a second training dataset with images that contain less UDC noise than that of the first training dataset. Furthermore, a person of ordinary skill in the art would have been able to carry out the substitution. Finally, the substitution achieves the predictable result of training a machine learned model for noise reduction in an image.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to substitute the training set of Kim in view of Lubana with the training set of Do according to known methods to yield the predictable result of teaching the model features of images captured by an externally exposed camera, compared to that of a UDC camera.
Claims 13-14 are rejected under 35 U.S.C. 103 as being unpatentable over Kim in view of Lubana, in further view of Roberts (U.S. Patent No. 2020/0174748 A1).
Regarding claim 13 (dependent on claim 11), Kim in view of Lubana teaches comprising: an alignment circuitry (Kim, pre-processor of S120 in FIG 5, see para 79-80; para 78: “FIG. 5 is a flowchart illustrating a method of operating the image processing systems 1200, 1200a, 1200b, and 1200c described above with reference to FIGS. 3 to 4C”) configured to output a third image data (Kim, referred to in this embodiment as second image data in para 80 of Kim, para 80: “Specifically, the image processing system 1200 may generate the second image data by (e.g., based on) performing a pre-processing operation”), wherein the deep learning neural network outputs the second image data from the third image data (Kim, Step 130-140 of FIG 5, mapped second image data, the output of the NN model in Kim, is referred to here as the third image data; see input and output data in para 82-83). However, Kim in view of Lubana fails to explicitly teach wherein the third image data is output by decomposing or rearranging at least a portion of the first image data (Kim only discloses BPC operation, LSC operation, crosstalk correction operation, and/or WB correction operation in para 80).
Roberts teaches a neural network (Roberts, abstract) for processing image data (Roberts, para 15). Roberts teaches a “presorter” (Roberts, para 41 and 304 in FIG 3), similar to the alignment circuitry mapped to the pre-processor of Kim. Roberts discloses wherein image data is output by decomposing or rearranging at least a portion of the first image data, or input data (Roberts, presorter process in FIG 4; para 59: “in step 400, the presorter may have rearranged the actual instances of input data into the sorted order in a memory (e.g., memory 204, presort buffer 306, etc.) and step 402 involves signaling the neural network processor”; para 60: “The neural network processor then uses the sorted order to control an order in which instances of input data from among the set of instances of input data are processed through the neural network (step 404)”; see also para 41). It would have been obvious to a person having ordinary skill in the art, before the effective filing date of the claimed invention, to have combined the rearranging of first image data, as taught by Roberts, with the image sensor of Kim in view of Lubana in order to increase efficiency of the neural network system (Roberts, para 58: “Recall that the described embodiments determine the sorted order to enable more efficient reuse of stored result values in the neural network—a goal that is met with even smaller improvements in an initial order for instances of input data”; see also para 28).
Regarding claim 14 (dependent on claim 13), Kim in view of Lubana and Roberts teaches wherein the alignment circuitry (Preprocessor of Kim, modified by Roberts - see claim 13 rejection) outputs the third image data according to an output format of an output interface circuitry (Kim, output interface circuitry is software of the IP network module that outputs the processed data; see FIG. 4A and para 61-63 where the output of the pre-processor is the same format, Tetra, as the output from the neural network).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant’s disclosure:
CN Patent No. 108921182 A (para 8: “FPGA feature extraction image sensor, which consists of a general purpose CCD/CMOS image sensor, an ARM controller, an FPGA convolutional neural network”)
CN Patent No. 110400250 A (para 26: “This embodiment uses an integrated circuit (ASIC) to replace the traditional AI chip and uses a stacking technology to integrate the ASIC into the image sensor”)
U.S. Patent No. 2022/0345592 A1 (Image sensor includes a PIM circuit including a neural network module, see abstract and claims 1-3)
Any inquiry concerning this communication or earlier communications from the examiner should be directed to EMMA E DRYDEN whose telephone number is (571)272-1179. The examiner can normally be reached M-F 9-5 EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, ANDREW BEE can be reached at (571) 270-5183. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/EMMA E DRYDEN/Examiner, Art Unit 2677
/ANDREW W BEE/Supervisory Patent Examiner, Art Unit 2677
Prosecution Insights