Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Drawings
The drawings are objected to as failing to comply with 37 CFR 1.84(p)(5) because they do not include the following reference sign(s) mentioned in the description. Specifically, photodiode 224 is referenced in the specification (see [0023-0024]) but does not appear in any of the figures present in the drawings. Corrected drawing sheets in compliance with 37 CFR 1.121(d) are required in reply to the Office action to avoid abandonment of the application. Any amended replacement drawing sheet should include all of the figures appearing on the immediate prior version of the sheet, even if only one figure is being amended. Each drawing sheet submitted after the filing date of an application must be labeled in the top margin as either “Replacement Sheet” or “New Sheet” pursuant to 37 CFR 1.121(d). If the changes are not accepted by the examiner, the applicant will be notified and informed of any required corrective action in the next Office action. The objection to the drawings will not be held in abeyance.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claim 12 is rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 12 recites “subtracting a signal of one of the switchable pixels under the clear state from a signal of the one of the switchable pixels under the clear state”, where it is unclear regarding which image each set of switchable pixels belongs to (or rather if “the switchable pixels” are referring to the same image and therefore is just performing subtraction on itself).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-5, 9-11, and 14-17 are rejected as being unpatentable over Kwan et al. (“Debayering RGBW color filter arrays: A pansharpening approach”, Publication Year: 2017, DOI: 10.1109/UEMCON.2017.8248995; hereinafter “Kwan”) in view of Zhu (US 2025/0015121; hereinafter “Zhu”).
Regarding Claim 1, Kwan discloses a demosaicing method for an image sensor, comprising:
capturing a panchromatic image by using a pixel array comprising (Fig. 2, Kwan discloses generating a panchromatic image from the panchromatic pixels of a RGBW color filter pattern (note the “Panchromatic pixels” image in Fig. 2).);
capturing a chromic image by using the pixel array when the color pixels are under a color state and the clear pixels are under the clear state (Figs. 1-2, Kwan discloses using an RGBW color filter pattern to generate a chromic image consisting of clear pixels and color pixels (note the “Color component pixels” image in Fig. 2).);
performing a weighted average calculation and a signal subtraction on the chromic image and the panchromatic image to generate a plurality of first resolution chromic images, wherein a resolution of the first resolution chromic images is same as a resolution of the panchromatic image and the chromic image (Section II. New Pansharpening Approach to Demosaicing of RGBW CFAs, Fig. 2, Kwan discloses a demosaicing process applied to RGBW pattern color filter arrays, wherein an average process is applied to both a panchromatic image and an RBG chromic image such that both images are at the same reduced resolution (note that the chromic image undergoes a “weighted” average as it averages the red, blue, and green pixels). Consequently, the reduced resolution images are subtracted to produce a chromic image (i.e., the full-resolution “Chrominance-Luminance image” from Fig. 2).); and
fusing the plurality of the first resolution chromic images with the panchromatic image to generate a plurality of demosaiced chromic images (Fig. 2, Kwan discloses a final fusing process between the panchromatic luminance image and the chrominance-luminance image to generate a plurality of demosaiced image (i.e., the color image in Fig. 2).).
Kwan does not disclose capturing a panchromatic image (italicized for context) by using a pixel array comprising a plurality of switchable pixels when the switchable pixels are under a clear state.
Zhu discloses capturing a panchromatic image (italicized for context) by using a pixel array comprising a plurality of switchable pixels when the switchable pixels are under a clear state (Fig. 1, [0030], Zhu discloses a camera module consisting of a plurality of color filter regions and electrochromic regions, wherein the electrochromic regions can alternate between a filter state and a light-transmissive state (i.e., switchable pixels).).
[AltContent: textbox (Annotated Fig. 2 from Kwan to show specifically the combination of Kwan in view of Zhu, wherein the color pixels disclosed by Kwan are replaced by Zhu’s switchable pixel)]
PNG
media_image1.png
379
142
media_image1.png
Greyscale
[AltContent: arrow][AltContent: textbox (Zhu’s switchable pixel in a clear state)][AltContent: textbox (Zhu’s switchable pixel in a color state)][AltContent: arrow]Kwan and Zhu are considered to be analogous to the claimed invention as they are in the same field of utilizing RGBW color filter arrays for image processing. Therefore, it would have been obvious to one having ordinary skill in the art before the effective filling date of the claimed invention to modify the invention of Kwan such that the RGBW pattern used to generate panchromatic and chromatic images for image demosaicing is modified using the disclosure provided by Zhu, such that the color pixels present in the base RGBW pattern disclosed by Kwan are modified to be the switchable pixels disclosed by Zhu (wherein the switchable pixels can change between a color state and a clear state based on a change in voltage). The motivation for this combination being the ability to remove the interpolation step noted by Kwan as the missing pixel values are now provided by Zhu’s switchable pixels, and further to dynamically modify the sensitivity of the image sensor, such that the image sensor has improved sensitivity to light input in dark environments (see [0035], Zhu).
Regarding Claim 2, Kwan in view of Zhu teaches the demosaicing method of claim 1, wherein each of the switchable pixels comprises an electrochromic layer, wherein the electrochromic layer is transparent when each of the switchable pixels is under the clear state, wherein the electrochromic layer is colored when each of the switchable pixels is under the color state wherein each of the switchable pixels comprises an electrochromic layer, wherein the electrochromic layer is transparent when each of the switchable pixels is under the clear state, wherein the electrochromic layer is colored when each of the switchable pixels is under the color state ([0030-0033], Fig. 1, Zhu discloses the addition of an electrochromic region on a color filter layer, which based on a voltage can adjust from a filter state (i.e., color state) and a fully light-transmissive state (i.e., clear state).).
Regarding Claim 3, Kwan in view of Zhu teaches the demosaicing method of claim 2, wherein the electrochromic layer is unbiased when each of the switchable pixels is under the clear state, wherein the electrochromic layer is biased when each of the switchable pixels is under the color state ([0078], Zhu discloses applying a control voltage to the electrochromic region to switch between a fully light-transmissive state and a filter state. Specifically, [0070] discloses that a condition of no driving voltage (i.e., unbiased) leads to a fully light-transmissive pixel and [0071] notes that a driving device is turned on and driven such that there is a change in voltage (i.e., biased state) leading the pixel to be in a filter/color state.).
Regarding Claim 4, Kwan in view of Zhu teaches the demosaicing method of claim 1, wherein the step of performing the weighted average calculation and the signal subtraction to generate the plurality of the first resolution chromic images comprises: generating a subtracted chromic image from the chromic image and the panchromatic image, wherein a resolution of the subtracted chromic image is lower than the resolution of the first resolution chromic images (Fig. 2, Kwan discloses subtracting a reduced resolution color image (i.e., chromic image) from a reduced resolution luminance image (i.e., panchromatic image) to generate a reduced resolution chrominance-luminance image. Note that the reduced resolution chrominance-luminance image has a lower resolution than the full resolution chrominance-luminance image (i.e., first resolution chromic images).).
Regarding Claim 5, Kwan in view of Zhu teaches the demosaicing method of claim 4, wherein the step of generating the subtracted chromic image comprises: performing the weighted average calculation on the chromic image and the panchromatic image to generate an averaged chromic image and an averaged panchromatic image (Fig. 2, Zhu discloses performing averaging on a color (i.e., chromic) image to generate a reduced resolution color image, and also performing averaging on a panchromatic image to generate a reduced resolution average luminance (i.e., panchromatic) image.); and performing the signal subtraction by subtracting a signal of the averaged panchromatic image from a signal of the averaged chromic image to generate the subtracted chromic image (Fig. 2, Zhu discloses subtracting the averaged luminance and averaged chromic image to generate a subtracted chrominance-luminance image.).
[AltContent: textbox (Annotated Fig. 2 from Kwan, wherein the marked red square defines the claimed “pixel set”)] Regarding Claim 9, Kwan in view of Zhu teaches the demosaicing method of claim 1, wherein a pixel set of the pixel array comprises two of the switchable pixels and two of the clear pixels arranged in an array of two rows by two columns (The Examiner notes Fig. 2 and the annotated image, wherein the red box drawn on the annotated message comprises the claimed “pixel set”. Furthermore, note the combination made in Claim 1, wherein Kwan is modified by replacing the normal colored pixels with Zhu’s switchable pixels via an electrochromic region.).
Regarding Claim 10, Kwan in view of Zhu teaches the demosaicing method of claim 9, wherein the step of performing the weighted average calculation on the panchromatic image comprises: averaging signals of the switchable pixels under the clear state and the clear pixels under the clear state to generate an averaged signal of the pixel set (Fig. 2, Kwan discloses averaging the pixels of the luminance (i.e., panchromatic) image. The Examiner further notes the combination in claim 1 and the annotated image shown in claim 1 regarding the averaging step involves both the clear pixels as well as the switchable pixels in a clear state.).
Regarding Claim 11, Kwan in view of Zhu teaches the demosaicing method of claim 9, wherein the step of performing the weighted average calculation on the chromic image comprises: averaging signals of the switchable pixels under the color state to generate an averaged signal of the pixel set (Fig. 2, Kwan discloses averaging the pixels of the color component pixels (i.e., chromic) image. The Examiner further notes the combination in claim 1 and the annotated image shown in claim 1 regarding the averaging step involves the switchable pixels in a color state.).
[AltContent: textbox (Annotated Fig. 2 from Kwan presenting the 4 pixels sets: 2 green pixel sets, 1 red pixel set, and 1 blue pixel set.)]
PNG
media_image3.png
158
135
media_image3.png
Greyscale
[AltContent: rect][AltContent: rect][AltContent: rect][AltContent: rect] Regarding Claim 14, Kwan in view of Zhu teaches the demosaicing method of claim 1, wherein the pixel array comprises two first pixel sets, one second pixel set, and one third pixel set, wherein each of the first pixel sets, the second pixel set, and the third pixel set comprises two of the switchable pixels and two of the clear pixels, wherein the first pixel sets, the second pixel set, and the third pixel set are arranged in an array of two rows by two columns (The Examiner notes the annotated image of Kwan’s Fig. 2 to the right, wherein the marked green box is analogous to the claimed “two first pixel sets”, the marked red box is analogous to the claimed “one second pixel set”, and the marked blue box is analogous to the claimed “one third pixel set”.).
Regarding Claim 15, Kwan in view of Zhu teaches the demosaicing method of claim 14, wherein the switchable pixels of the first pixel sets under the color state are green pixels, the switchable pixels of the second pixel set under the color state are red pixels, and the switchable pixels of the third pixel set under the color state are blue pixels (The Examiner notes the combination and annotated image presented for claim 1, wherein Zhu’s disclosure of the switchable pixels replaces each of the color pixels presented by Kwan. Furthermore, [0060] of Zhu discloses that the electrochromic region of the switchable pixel can be tuned for outputting a red spectrum, green spectrum, and blue spectrum.).
Regarding Claim 16, Kwan in view of Zhu teaches the demosaicing method of claim 1, wherein the clear pixels and the switchable pixels are alternately arranged (The Examiner notes the combination and annotated image presented for claim 1, wherein the combination of Zhu’s switchable pixels and the Kwan’s clear pixels are alternately arranged.).
Regarding Claim 17, Kwan discloses a demosaicing device for an image sensor, comprising (Note Kwan’s Fig. 2 regarding using an RGBW color filter array that is used with an image sensor, and furthermore that the computation and processing disclosed by Kwan is performed on an electronic device (further note Table 3 from Kwan providing the computer machine used for the image processing).): a pixel array capturing a panchromatic image and a chromic image (see Fig. 2, Kwan obtains an image of panchromatic pixels and chromic/color pixels using a RGBW color filter array.), comprising: color pixel and comprising a second photodiode (Fig. 2, Kwan discloses a clear panchromatic pixel adjacent to a color pixel. The Examiner asserts that the RGBW color filter array disclosed by Kwan consists of photodiodes/photosensors as the photodiodes/photosensors are utilized to absorb light from the environment (note the Wikipedia article about Bayer filters (wherein the RGBW color filter array is a modification of the Bayer filter) which disclose that a Bayer filter consists of RGB color filters laid on top of a grid of photosensors).), (The Examiner notes that the clear/panchromatic pixel disclosed by Kwan does not contain an electrochromic layer, and therefore is unaffected by voltage/bias and remains clear.); a calculating circuit performing a weighted average calculation and a signal subtraction on the chromic image and the panchromatic image to generate a plurality of first resolution chromic images (Section II. New Pansharpening Approach to Demosaicing of RGBW CFAs, Fig. 2, Kwan discloses a demosaicing process applied to RGBW pattern color filter arrays, wherein an average process is applied to both a panchromatic image and an RBG chromic image such that both images are at the same reduced resolution (noted that the chromic image undergoes a “weighted” average as it averages the red, blue, and green pixels). Consequently, the reduced resolution images are subtracted to produce a chromic image (i.e., the “Chrominance-Luminance image” from Fig. 2). The Examiner asserts that the computation disclosed by Kwan is performed on some electronic device consisting of a circuit.); and a correction circuit fusing the plurality of the first resolution chromic images with the panchromatic image to generate a plurality of demosaiced chromic images (Fig. 2, Kwan discloses a final fusing process between the panchromatic luminance image and the chrominance-luminance image to generate a plurality of demosaiced image. The Examiner asserts that the computation disclosed by Kwan is performed on some electronic device consisting of a circuit.).
Kwan does not disclose a switchable pixel, comprising a first photodiode and an electrochromic layer above the first photodiode, and wherein the switchable pixel is under a clear state when the electrochromic layer is unbiased, and wherein the switchable pixel is under a color state when the electrochromic layer is biased.
Zhu discloses a switchable pixel, comprising a first photodiode and an electrochromic layer above the first photodiode, and wherein the switchable pixel is under a clear state when the electrochromic layer is unbiased, and wherein the switchable pixel is under a color state when the electrochromic layer is biased (Fig. 1, [0030], [0070-0078], Zhu discloses a camera module consisting of a plurality of color filter regions and electrochromic regions, wherein the electrochromic regions can alternate between a filter state and a light-transmissive state (i.e., switchable pixels) based on a change in voltage/bias. Specifically, [0070] discloses that a condition of no driving voltage (i.e., unbiased) leads to a fully light-transmissive pixel and [0071] notes that a driving device is turned on and driven such that there is a change in voltage (i.e., biased state) leading the pixel to be in a filter/color state.).).
Kwan and Zhu are considered to be analogous to the claimed invention as they are in the same field of utilizing RGBW color filter arrays for image processing. Therefore, it would have been obvious to one having ordinary skill in the art before the effective filling date of the claimed invention to modify the demosaicing device of Kwan such that the RGBW pattern used to generate panchromatic and chromatic images for image demosaicing is modified using the disclosure provided by Zhu, such that the color pixels present in the base RGBW pattern disclosed by Kwan are modified to be the switchable pixels disclosed by Zhu (wherein the switchable pixels can change between a color state and a clear state based on a change in voltage) (Also see the annotated image presented with Claim 1 regarding the orientation of clear pixels and switchable pixels as taught by Kwan in view of Zhu). The motivation for this combination being the ability to remove the interpolation step noted by Kwan as the missing pixel values are now provided by Zhu’s switchable pixels, and further to dynamically modify the sensitivity of the image sensor, such that the image sensor has improved sensitivity to light input in dark environments (see [0035], Zhu).
Claims 18 is rejected as being unpatentable over Kwan in view of Zhu in view of Liu et al. (CN 105681771; hereinafter “Liu”).
Regarding Claim 18, Kwan in view of Zhu teaches the demosaicing device of claim 17.
Kwan in view of Zhu does not teach the first photodiode and the electrochromic layer are individually controlled by different transistors.
Liu teaches the first photodiode and the electrochromic layer are individually controlled by different transistors (Fig. 6-7, Liu discloses two circuit structures, wherein Fig. 6 discloses a circuit structure (consisting of transistors 242, 244, and 246) which controls the voltage signal applied to an electrochromatic sheet. Fig. 7 discloses another circuit structure (consisting of transistors 222, 224, and 226) which controls the electrical signal obtained from a photodiode).
Kwan in view of Zhu and Liu are considered to be analogous to the claimed invention as they are in the same field of utilizing photodiodes and electrochromic materials for image processing. Therefore, it would have been obvious to one having ordinary skill in the art before the effective filling date of the claimed invention to modify the demosaicing device of Kwan in view of Zhu such that the first photodiode and electrochromic layer in the switchable pixel was controlled by two independent circuits (each with individually controlled transistors) as disclosed by Liu. The motivation for this combination being the ability to have specific control over each component (i.e., the photodiode and electrochromic layer) such that they are able to operate individually/independently of each other.
Claims 19 is rejected as being unpatentable over Kwan in view of Zhu in view of Qiao (CN 212807296; hereinafter “Qiao”).
Regarding Claim 19, Kwan in view of Zhu teaches the demosaicing device of claim 17, wherein the switchable pixel further comprises:
and a micro lens array (See Fig. 1, Zhu discloses a micro lens layer 5 that is present above the electrochromic and photosensitive layer.).
Kwan in view of Zhu does not explicitly teach a bottom electrode between the first photodiode and the electrochromic layer;
an electrolyte layer on the electrochromic layer;
an ion storage layer on the electrolyte layer;
a top electrode on the ion storage layer;
and a micro lens array (italicized for context) on the top electrode (The Examiner notes Fig. 1, Fig. 7, and [0078-0088] from Zhu, which disclose the electrochromic region (i.e, switchable pixel) containing an electrolyte layer, electrochromic layer, ion storage layer, and two electrodes.).
Qiao discloses a bottom electrode between the first photodiode and the electrochromic layer (see Figs. 2-3 where a light emitting device/diode (3) is emitting light towards an electrochromic cell (1) that is placed in front/above the photodiode (2). Note that the electrochromic cell contains an electrochromic layer (12) and bottom electrode (11/111).);
an electrolyte layer on the electrochromic layer (see Fig. 3, where electrolyte layer (13) is placed on the electrochromic layer (12));
an ion storage layer on the electrolyte layer (see Fig. 3, where ion storage layer (14) is placed on the electrolyte layer (13).);
a top electrode on the ion storage layer;
and a micro lens array (italicized for context) on the top electrode (see Fig. 3, which places an electrode (15/151) on top of the ion storage layer (14)).
Kwan in view of Zhu and Qiao are considered to be analogous to the claimed invention as they are in the same field of utilizing electrochromic elements. Therefore, it would have been obvious to one having ordinary skill in the art before the effective filling date of the claimed invention to modify the demosaicing device of Kwan in view of Zhu such that the electrochromic layer of the switchable pixel, taught by Kwan in view of Zhu, follows the structure of the electrochromic layer disclosed by Qiao in order for the micro lens array to be placed above the modified electrochromic layer. The motivation for this combination being the ability to horizontally orient the electrodes applying the voltage/bias to the electrochromic layer, minimizing the space required to store all the elements of the electrochromic layer.
Allowable Subject Matter
Claims 6-8, 13, and 20 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Zhang (WO2023179527A1)
Wang et al. (“High-quality image deblurring with panchromatic pixels”)
Any inquiry concerning this communication or earlier communications from the examiner should be directed to PROMOTTO TAJRIAN ISLAM whose telephone number is (703)756-5584. The examiner can normally be reached Monday - Friday 8:30 am - 5:00 pm EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Chan Park can be reached at (571) 272-7409. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/PROMOTTO TAJRIAN ISLAM/Examiner, Art Unit 2669 /CHAN S PARK/Supervisory Patent Examiner, Art Unit 2669