DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
This action is in response to Application filed on 06/03/2024
Application is a 371 of PCT/JP2022/041427 11/07/2022
Application claims a FP date of 12/10/2021
Claim 1 is independent
Claims 1-20 are pending
Priority
Acknowledgment is made of applicant’s claim for foreign priority under 35 U.S.C. 119 (a)-(d). The certified copy has been filed in the instant Application.
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 06/03/2024 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Specification
The title of the invention is not descriptive. A new title is required that is clearly indicative of the invention to which the claims are directed.
Drawings
The drawings are objected to as failing to comply with 37 CFR 1.84(p)(4) because reference character “1a” has been used to designate both Display area and Bezel in Fig 1. Corrected drawing sheets in compliance with 37 CFR 1.121(d) are required in reply to the Office action to avoid abandonment of the application. Any amended replacement drawing sheet should include all of the figures appearing on the immediate prior version of the sheet, even if only one figure is being amended. Each drawing sheet submitted after the filing date of an application must be labeled in the top margin as either “Replacement Sheet” or “New Sheet” pursuant to 37 CFR 1.121(d). If the changes are not accepted by the examiner, the applicant will be notified and informed of any required corrective action in the next Office action. The objection to the drawings will not be held in abeyance.
The drawings are objected to under 37 CFR 1.83(a). The drawings must show every feature of the invention specified in the claims. Therefore, the “data not caused by the image element” in Claim 1, “data caused by the display surface is caused by at least one of diffraction or reflection on the display surface” in Claim 4, “imaging information in proximity focus” in Claim 8, “positional relationship between an external subject and imaging element on a basis of the gyro signal” in Claim 13, “positional relationship with the display surface is changed and corrects the image signal on the basis of the positional relationship between the display and the imaging element along a time series” in Claim 17 and “acquires a shape change in the image signal with a lapse of time, and corrects the image signal on a basis of the shape change” in Claim 18 must be shown or the features canceled from the claims. No new matter should be entered.
Corrected drawing sheets in compliance with 37 CFR 1.121(d) are required in reply to the Office action to avoid abandonment of the application. Any amended replacement drawing sheet should include all of the figures appearing on the immediate prior version of the sheet, even if only one figure is being amended. The figure or figure number of an amended drawing should not be labeled as “amended.” If a drawing figure is to be canceled, the appropriate figure must be removed from the replacement sheet, and where necessary, the remaining figures must be renumbered and appropriate changes made to the brief description of the several views of the drawings for consistency. Additional replacement sheets may be necessary to show the renumbering of the remaining figures. Each drawing sheet submitted after the filing date of an application must be labeled in the top margin as either “Replacement Sheet” or “New Sheet” pursuant to 37 CFR 1.121(d). If the changes are not accepted by the examiner, the applicant will be notified and informed of any required corrective action in the next Office action. The objection to the drawings will not be held in abeyance.
Claim Rejections - 35 USC § 112
The following is a quotation of the first paragraph of 35 U.S.C. 112(a):
(a) IN GENERAL.—The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor or joint inventor of carrying out the invention.
The following is a quotation of the first paragraph of pre-AIA 35 U.S.C. 112:
The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor of carrying out his invention.
Claims 1-2, 13 and 18 are rejected under 35 U.S.C. 112(a) or 35 U.S.C. 112 (pre-AIA ), first paragraph, as failing to comply with the written description requirement. The claim(s) contains subject matter which was not described in the specification in such a way as to reasonably convey to one skilled in the relevant art that the inventor or a joint inventor, or for applications subject to pre-AIA 35 U.S.C. 112, the inventor(s), at the time the application was filed, had possession of the claimed invention. Claims 1 and 2 recite “data not caused by the imaging element”. Applicant fails to clearly describe what data is used that is not caused by the imaging element and provide examples in such a way so as to reasonably convey to one skilled in the art. Applicant merely discloses in ¶0066 of the Specification that the data maybe from a display driver or optical driver or gyro sensor. However, he does not discloses what data from display driver he is referring to. Data from display driver typically includes the rendered images and pixels sent to the screen and maybe metadata like version number and driver files. Since it is well known in the art that rendered images and pixels are caused by the imaging element, it further causes confusion in understanding what Applicant means by “data not caused by the imaging element”.
Claim 13 recite “corrects the image signal from a change in data acquired by the imaging element for a pixel for which an accurate pixel value has not been acquired in a single frame”. It is not clear what the correction is and what the Applicant is trying to claim when he recites “for which an accurate pixel has not been acquire in a single frame”. Through examination of the Specification does not throw light on this matter and Applicant fails to clearly describe what correction needs to be applied.
Claim 18 recite “a shape change in the image signal”. It is not clear what the change in image signal is and how this is used to correct the image signal. The instant specification or the drawing does not clearly disclose this so as to convey to one skilled in the art.
Claims 1-2, 13 and 18 are rejected under 35 U.S.C. 112(a) or 35 U.S.C. 112 (pre-AIA ), first paragraph, because the best mode contemplated by the inventor or a joint inventor, or for applications subject to pre-AIA 35 U.S.C. 112, the inventor(s) has not been disclosed. Evidence of concealment of the best mode is based upon the fact that Claims 1 and 2 recite “data not caused by the imaging element”. Applicant fails to clearly describe what data is used that is not caused by the imaging element and provide examples in such a way so as to reasonably convey to one skilled in the art. Applicant merely discloses in ¶0066 of the Specification that the data maybe from a display driver or optical driver or gyro sensor. However, he does not discloses what data from display driver he is referring to.
Claim 13 recite “corrects the image signal from a change in data acquired by the imaging element for a pixel for which an accurate pixel value has not been acquired in a single frame”. It is not clear what the correction is and what the Applicant is trying to claim when he recites “for which an accurate pixel has not been acquire in a single frame”. Through examination of the Specification does not throw light on this matter and Applicant fails to clearly describe what correction needs to be applied.
Claim 18 recite “a shape change in the image signal”. It is not clear what the change in image signal is and how this is used to correct the image signal. The instant specification or the drawing does not clearly disclose this so as to convey to one skilled in the art.
Claims -2, 13 and 18 are rejected under 35 U.S.C. 112(a) or 35 U.S.C. 112 (pre-AIA ), first paragraph, as failing to comply with the enablement requirement. The claim(s) contains subject matter which was not described in the specification in such a way as to enable one skilled in the art to which it pertains, or with which it is most nearly connected, to make and/or use the invention. Claims 1 and 2 recite “data not caused by the imaging element”. Applicant fails to clearly describe what data is used that is not caused by the imaging element and provide examples in such a way so as to reasonably convey to one skilled in the art. Applicant merely discloses in ¶0066 of the Specification that the data maybe from a display driver or optical driver or gyro sensor. However, he does not discloses what data from display driver he is referring to.
Claim 13 recite “corrects the image signal from a change in data acquired by the imaging element for a pixel for which an accurate pixel value has not been acquired in a single frame”. It is not clear what the correction is and what the Applicant is trying to claim when he recites “for which an accurate pixel has not been acquire in a single frame”. Through examination of the Specification does not throw light on this matter and Applicant fails to clearly describe what correction needs to be applied.
Claim 18 recite “a shape change in the image signal”. It is not clear what the change in image signal is and how this is used to correct the image signal. The instant specification or the drawing does not clearly disclose this so as to convey to one skilled in the art.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1-11, and 14-16 are rejected under 35 U.S.C. 102(a)(2) as being anticipated by Cha et al. (U.S. Patent Publication Number 2022/0046215 A1 which has a filing date of May 25, 2021).
Regarding Claim 1, Cha discloses an electronic device (In Fig 1, Cha discloses an electronic device), comprising:
a display surface (Fig 1- display 20) that displays information on a basis of light emitted from a light emitting element (In ¶0055, Cho discloses that the display may be n OLED or LCD or the like);
an imaging element disposed on an opposite side of the display surface (Fig 1, 2 – image sensor 100, 200; In ¶0062, Cho discloses that the first image sensor 100 may be covered by the display 20 and disposed in an opening formed in the back cover and in ¶0065, he disclose that the second image sensor 200 may be disposed on the rear surface); and
a processing circuit (Figs 5-9 – application processor 300) that processes a signal output from the imaging element (In ¶0076 Cha discloses that the first imaging signal S1 and second imaging signal S2 may be processed by the application processor 300), wherein the processing circuit acquires data not caused by the imaging element (Cho in ¶0075 discloses that the second image signal S2 is signal that does not pass through the display and has been interpreted and data not caused by imaging element. Examiner would like to state , on the record, that “data not caused by the imaging element has not been clearly defined in the claim, and the specification in the instant application does not give more guidance. Applicant discloses it to be data from a display driver. However, data from a display driver is well known to be rendered images and pixels are caused by the imaging element), and
corrects an image signal light-received and output by the imaging element using the acquired data not caused by the imaging element (In ¶0103, Cha discloses that the preprocessing circuit 311 corrects defective pixel and in ¶0104 he discloses that the image signal S1 outputted from the first image sensor may be corrected and in ¶0175 he further discloses that the signal S2 may be used under certain conditions to process the first signal S1 ).
Regarding Claim 2, Cha discloses wherein the data not caused by the imaging element is data caused by the display surface (Cho in ¶0075 discloses that the second image signal S2 is signal that does not pass through the display and has been interpreted and data not caused by imaging element.).
Regarding Claim 3, Cha discloses wherein the data caused by the display surface is caused by light emitted from the light emitting element (In ¶0056, Cho discloses that display 20 may be an organic light emitting diode display – since the display has been disclosed to be LED display, it is clear that signal S1 – that passes through the display is caused by light emitted from the light emitting element of the display).
Regarding Claim 4, Cha discloses wherein the data caused by the display surface is caused by at least one of diffraction or reflection on the display surface or light emitted from the light emitting element and directly reaching the imaging element (In ¶0058, Cho discloses that the first image sensor 100 may sense light outputted from the display 20 and reflected by the cover glass 10).
Regarding Claim 5, Cha discloses wherein the data caused by the display surface is caused by light incident on the display surface (In ¶0058, Cho discloses that the first image sensor 100 may sense light outputted from the display 20 and reflected by the cover glass 10; Further in the same paragraph, Cho discloses that the image sensor 100 may sense light incident through the display 20).
Regarding Claim 6, Cha discloses wherein the data caused by the display surface is caused by flare of light incident from an outside on the display surface (In ¶0058, Cho discloses that the first image sensor 100 may sense light outputted from the display 20 and reflected by the cover glass 10; Further in the same paragraph, Cho discloses that the image sensor 100 may sense light incident through the display 20 from the outside of the electronic device 1 and/or part of light outputted from the display 20. Since it is well known in the art that “flare” is caused by light incident from an outside – it is clear that Cho discloses all the elements of claim 6.).
Regarding Claim 7, Cha discloses further comprising a focus driver that controls a focus, wherein the processing circuit (Application processor 300) corrects the image signal on a basis of information acquired by the focus driver (In Fig 16 – Cho discloses an actuator 16; In ¶0191 Cho discloses that the actuator 1130 may move the optical lenses to a specific position so that an image sensor 1142 may be positioned at a focal length of the optical lenses for accurate sensing).
Regarding Claim 8, Cha discloses wherein the processing circuit (Application processor 300) acquires imaging information in proximity focus acquired by the focus driver (In Fig 16 – Cho discloses an actuator 16; In ¶0191 Cho discloses that the actuator 1130 may move the optical lenses to a specific position so that an image sensor 1142 may be positioned at a focal length of the optical lenses for accurate sensing), and
corrects the image signal with what is displayed in proximity to be corrected on a basis of imaging information in the proximity focus (In ¶0020, Cha discloses that the display may be further configured to output an image generated based on the corrected first image signal.).
Regarding Claim 9, Cha discloses wherein the processing circuit (Application processor 300) acquires a display signal to be displayed through the display surface, and
corrects the image signal on a basis of the display signal ().
Regarding Claim 10, Cha discloses wherein the processing circuit (Application processor 300, preprocessing circuit 311, post processing 321) acquires the display signal from a display driver that controls the light emitting element (In ¶0167, Cho discloses that the application processor 300 may provide the generated image signal to the display 20, the image signal generated by the display driving circuit may be transmitted to the display 20), or acquires the display signal input from an outside to correct the image signal ().
Regarding Claim 11, Cha discloses wherein the processing circuit corrects the image signal on a basis of a signal obtained by reducing resolution of the display signal (Cho discloses the display of the electronic device in Fig 1 and 3. He also discloses the area occupied by the camera 100 and 200 – and it is clear that the pixel area (140 and 240 of Fig 5) is sufficiently small as compared with a size of the display area and therefore it could be interpreted that the display signal has reduced resolution. Examiner has used an argument similar to the one presented by the Applicant in the instant application in ¶0082 of the printed publication).
Regarding Claim 14, Cha discloses wherein the correction includes correction processing related to a color of the image signal (Cha discloses this throughout – especially in ¶0012 where he discloses using color temperature information to correct the image).
Regarding Claim 15, Cha discloses wherein the color correction processing is correction processing at a linear matrix application timing (In ¶0111, Cha discloses that the demosaicing processor 322 may perform Bayer color interpolation), correction processing at a conversion timing to a YUV signal, or correction processing in color conversion using an LUT (In ¶0115, Cha discloses that the color processing may convert RGB into YCC color space).
Regarding Claim 16, Cha discloses wherein the correction further includes correction processing related to at least one of automatic white balance control, automatic exposure control, and automatic focus control (Throughout his application Cha discloses that preprocessing circuit 311 may include auto exposure (AE) 315 and auto white balance (AWB) evaluation calculator 316).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 12-13 and 17 are rejected under 35 U.S.C. 103 as being unpatentable over Cha et al. (U.S. Patent Publication Number 2022/0046215 A1 which has a filing date of May 25, 2021) in view of Tyagi et al. (U. S. Patent Publication Number 2023/0176612 A1 which has a filing date of Dec 08, 2021).
Regarding Claim 12, Cha fails to clearly disclose further comprising a gyro sensor; wherein the processing circuit corrects the image signal on a basis of a gyro signal output from the gyro sensor.
Instead in a similar endeavor, Tyagi discloses further comprising a gyro sensor (In ¶0068, Tyagi teaches that the image stabilization engine receives orientation signals from one or more inertial motion units. Further in ¶0085, he teaches that IMUs can comprise of ); wherein the processing circuit corrects the image signal on a basis of a gyro signal output from the gyro sensor (In ¶0069, Tyagi teaches that the electronic device processes the orientation data to correct the angular movement experienced by the imager 111 or camera 106 – and automatically adjusts the frame of the image being captured).
Cha and Tyagi are combinable because both are related to imaging devices.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to use the orientation sensor and information from the sensor to correct the images as taught by Tyagi in the imaging module disclosed by Cha.
The suggestion/motivation for doing so would have been to “correct the angular movement experienced by the imager” as disclosed by Tyagi in ¶0069.
Therefore, it would have been obvious to combine Cha and Tyagi to obtain the invention as specified in claim 12.
Regarding Claim 13, Cha in view of Tyagi discloses wherein the processing circuit (Chi: Application processor 300, preprocessing circuit 311, post processing 321; Tyagi: processor 109) estimates a positional relationship between an external subject and the imaging element on a basis of the gyro signal (Tyagi: Tyagi teaches this in Fig 10-13 and in ¶0126-¶0146 the “identifiable marker” positioned at location 1001 on the flexible display 121. In ¶0130 Tyagi teaches that when the housing 102 pivots, the identifiable marker 1001 translates and this translation is used to estimate the amount of deformation as a function how far the marker has translated), and corrects the image signal from a change in data acquired by the imaging element for a pixel for which an accurate pixel value has not been acquired in a single frame (Tyagi: Tyagi teaches this in Fig 10-13 and in ¶0126-¶0146).
Regarding Claim 17, Cha in view of Tyagi discloses wherein the imaging element is disposed so that a positional relationship with the display surface is changed (Tyagi: Tyagi teaches this in Fig 10-13 and in ¶0126-¶0146 the “identifiable marker” positioned at location 1001 on the flexible display 121. In ¶0130 Tyagi teaches that when the housing 102 pivots, the identifiable marker 1001 translates and this translation is used to estimate the amount of deformation as a function how far the marker has translated), and
the processing circuit corrects the image signal on a basis of the positional relationship between the display surface and the imaging element along a time series (Tyagi: Tyagi teaches this in Fig 10-13 and in ¶0126-¶0146; In ¶0069, Tyagi teaches that the stabilization engine continuously detects movement and vibration and automatically adjusts the frame of the image being captured so that successive frame see the same image details).
Claims 18-20 are rejected under 35 U.S.C. 103 as being unpatentable over Cha et al. (U.S. Patent Publication Number 2022/0046215 A1 which has a filing date of May 25, 2021) in view of Choi et al. (U. S. Patent Publication Number 2023/0176612 A1 which has an effective filing date of June 10, 2021).
Regarding Claim 18, Cha fails to clearly discloses wherein the processing circuit acquires a shape change in the image signal with a lapse of time, and corrects the image signal on a basis of the shape change.
Instead in a similar endeavor, Choi discloses wherein the processing circuit acquires (Choi – processor 120) a shape change in the image signal with a lapse of time, and corrects the image signal on a basis of the shape change (In ¶0014, Choi teaches that the electronic device operates adaptively to an environment change, thereby obtaining high quality images based on scenarios appropriate for respective environment. Choi teaches about frame analysis module 420 and frame correction module 430. In ¶0073-¶0088 (and throughout) Choi teaches that when the camera module 180 is disposed under the display 160, transmissivity of light may be reduces and therefore the frame acquiring module 410 acquires multiple frame based on the imaging parameter, and the frame compositing module 410 may extract identical regions and composite the multiple frames based on identical frame regions, thereby generating frame that has improved SNR).
Cha and Choi are combinable because both are related to imaging devices that have camera under the display.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to use the multiple frame of correct and improve SNR as taught by Choi in the imaging module disclosed by Cha.
The suggestion/motivation for doing so would have been to “perform frame correction based on identified attribute” as disclosed by Choi in ¶0012.
Therefore, it would have been obvious to combine Cha and Choi to obtain the invention as specified in claim 18.
Regarding Claim 19, Cha in view of Choi discloses wherein the processing circuit corrects the image signal using a learned model (Choi: In ¶0033, Choi teaches using artificial intelligence module generated by machine learning to perform the tasks disclosed in his disclosure).
Regarding Claim 20, Cha in view of Choi discloses wherein the learned model is a model learned using a display signal and the image signal as teacher data (Choi: In ¶0033, Choi also teaches that algorithms may include supervised learning and semi-supervised learning. Choi’s disclosure also teaches using signals captured by the imager (display signal) and sample frames (image signal) that are frames acquired by the device in real time through the camera module 180).
Reference Cited
The following prior art made of record but not relied upon is considered pertinent to applicant's disclosure.
Matsunaga (U.S. Patent Publication Number 2018/0338096 A1) discloses the apparatus (120) has a pixel value acquisition circuit (122) that acquires a pixel value from image data generated by an imaging device (104). A pixel specification circuit (124) specifies a flare-affected pixel that is affected by flare. The acquisition circuit sets a detection pixel that is arranged around targeted detection pixel in a prescribed area in a pixel arrangement including targeted detection pixel, a detection pixel that is a target of a flare determination, and sets as a surrounding detection pixel. The acquisition circuit acquires pixel values of multiple pixels that are adjacent to targeted detection pixel and pixel values that are adjacent to surrounding detection pixel. The specification circuit specifies a flare-affected pixel that is affected by flare from among pixels that are adjacent to targeted detection pixel, on basis of pixel values that are adjacent to targeted detection pixel and pixel values that are adjacent to surrounding detection pixel.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to PADMA HALIYUR whose telephone number is (571)272-3287. The examiner can normally be reached Monday-Friday 7AM - 4PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Twyler Haskins can be reached at 571-272-7406. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/PADMA HALIYUR/Primary Examiner, Art Unit 2639 October 17, 2025