DETAILED ACTION
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after allowance or after an Office action under Ex Parte Quayle, 25 USPQ 74, 453 O.G. 213 (Comm'r Pat. 1935). Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, prosecution in this application has been reopened pursuant to 37 CFR 1.114. Applicant's submission filed on 05/22/2025 has been entered.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The text of those sections of Title 35, U.S. Code not included in this action can be found in a prior Office action.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1 – 5, 7, 9 – 17, 19 and 21 are rejected under 35 U.S.C. 103 as being unpatentable over Nakata (US 20230042435 A1; hereafter referred to as Nakata) in view of Kurita et al (US 20210281786 A1; hereafter referred to as Kurita) further in view of Hashimoto et al. (See Machine Translation for JP 2013106882 A; hereafter referred to as Hashimoto).
Regarding Claim 1, Nakata teaches:
An image data processing device that processes image data captured by an imaging device including an optical system which spectrally separates incident light into a plurality of wavelengths, polarizes light with the spectrally separated wavelengths in a specific direction, and emits the polarized light and an imaging element including a plurality of pixel sets each of which includes different types of polarizers (Nakata, [0014] “the present disclosure, there is provided an electronic apparatus including: [0015] a display unit that is capable of emitting light at a plurality of different emission wavelengths; [0016] an imaging unit that is disposed on an opposite side to a display surface of the display unit; and [0017] an abnormality detection unit that detects an abnormality on the display surface on the basis of a plurality of images captured by the imaging unit in a state in which at least a part of the display surface is caused to emit light at each of the plurality of emission wavelengths”), the image data processing device comprising:
a processor (Nakata, [0096] “an application processor 22”) configured to perform:
a process of acquiring the image data from the imaging device(Nakata, “[0032] an image communication unit that transmits the image captured by the imaging unit”);
a process of detecting an abnormal pixel from the acquired image data (Nakata, [0025] An abnormality determination unit that determines a type of the abnormality”);
a process of generating images of the spectrally separated wavelengths from the image data in which the pixel value of the abnormal pixel has been corrected in a case in which the abnormal pixel is detected (Nakata, [0031] a correction determination unit that determines whether or not correction by the correction processing unit is effective; [0032] an image communication unit that transmits the image captured by the imaging unit and information regarding the abnormality to an information processing apparatus and receives the image corrected by the information processing apparatus …; and [0033] an output unit that outputs the image corrected by the information processing apparatus”).
While Nakata teaches detecting abnormality and correcting the abnormality, it fails to explicitly teach:
a process of detecting a pixel whose pixel value is out of a predetermined range as an abnormal pixel from the acquired image data;
a process of correcting a pixel value of the abnormal pixel on the basis of pixel values of peripheral pixels having polarizers whose types are different from a type of a polarizer of the abnormal pixel in a case in which the abnormal pixel is detected;
In the same filed of endeavor, Kurita teaches:
a process of detecting a pixel whose pixel value is out of a predetermined range as an abnormal pixel from the acquired image data (Kurita, [0010] “a difference between the pixel value of the target polarized pixel and the pixel value estimated is out of the predetermined first allowable range and in a case where the difference is within the predetermined first allowable range and a difference between the pixel value of the target polarized pixel and the pixel values of the peripheral pixels in the identical polarization direction is out of the predetermined second allowable range, the defect detecting section may determine that the target polarized pixel is the defective pixel”; Kurita, [0068] “On the basis of the pixel value of the target polarized pixel and the estimated pixel value (hereinafter referred to as “estimated pixel value”), the defect detecting section 35 determines whether the target polarized pixel is the defective pixel…in a case where the difference between the pixel value of the target polarized pixel and the estimated pixel value is out of a predetermined allowable range, the defect detecting section 35 determines that the target polarized pixel is the defective pixel”);
a process of correcting a pixel value of the abnormal pixel on the basis of pixel values of peripheral pixels having polarizers whose types are different from a type of a polarizer of the abnormal pixel in a case in which the abnormal pixel is detected (Kurita, [0062] “the image processing apparatus may correct the defective pixel using the pixel values of the peripheral pixels in an identical polarization direction to the polarization direction of the defective pixel”; Kurita, [0068] “The defect detecting section 35 estimates the pixel value of the target polarized pixel on the basis of the polarization model formula after fitting that indicates polarization characteristics corresponding to the pixel values of peripheral pixels in different polarization directions.”; Kurita, [0069] “the defect correcting section 36 corrects the defective pixel of the polarized image indicated by the defect information using the peripheral pixels positioned in the periphery of the defective pixel”; Kurita, [0079] “the corrected pixel value of the defective pixel, the defect correcting section 36 specifies the estimated pixel value estimated on the basis of the polarization characteristics corresponding to the pixel values generated in the peripheral pixels in the polarization directions different from the polarization direction of the defective pixel”);
Nakata and Kurita are considered analogous art as they are reasonably pertinent to the same field of endeavor of image processing. Therefore, it would have been obvious to one of the ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Nakata with the process of detecting abnormalities as taught by Kurita to make the invention that processes the acquired image and detects abnormal pixels whole pixel value is out of a predetermined range as an abnormal pixel from the acquired image data; corrects a pixel value of the abnormal pixel on the basis of pixel values of peripheral pixels having polarizers whose types are different from a type of a polarizer of the abnormal pixel; and generates images of the spectrally separated wavelengths from the image data in which the pixel value of the abnormal pixel has been corrected; doing so can efficiently detect defects in a polarizing imaging device (Kurita, [0006]); thus, one of the ordinary skill in the art would have been motivated to combine the references.
However, Nakata in view of Kurita does not explicitly teach:
wherein the processor performs a process of generating the images of the spectrally separated wavelengths from the image data excluding the abnormal pixel, without correcting the pixel value of the abnormal pixel, in a case in which the pixel value is out of the predetermined range due to a failure.
In the same field of endeavor Hashimoto teaches:
wherein the processor performs a process of generating the images of the spectrally separated wavelengths from the image data excluding the abnormal pixel, without correcting the pixel value of the abnormal pixel, in a case in which the pixel value is out of the predetermined range due to a failure (Hashimoto, page 8, para 5 – 7, Fig. 5, “The abnormal pixel value determination unit 40 determines whether each pixel value is a normal pixel value or an abnormal pixel value for each pixel …The abnormal pixel value removing unit 41 removes the abnormal pixel value determined by the abnormal pixel value determining unit 40… Specifically, as shown in FIG. 6, the abnormal pixel value determination unit 40 normalizes the pixel values Ik (x, y) included in a predetermined range ... The pixel value is determined, and the other pixel values are determined as abnormal pixel values. In the case of FIG. 6, since the pixel value I2 (x, y) is out of the predetermined range, the abnormal pixel value determination unit 40 determines that the pixel value is an abnormal pixel value, and the abnormal pixel removal unit 41 removes it”; Hashimoto, page 10, para 2, “correction method generates a phase differential image without performing correction in the state of the image data Dk, and then generates a pixel value (abnormal phase differential value) corresponding to the abnormal pixel value on the image data Dk”).
Nakata, Kurita and Hashimoto are considered analogous art as they are reasonably pertinent to the same field of endeavor of image processing. Therefore, it would have been obvious to one of the ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Nakata in view of Kurita with the process of removing abnormal pixels as taught by Hashimoto to make the invention that processes the acquired image and detects abnormal pixels whole pixel value is out of a predetermined range as an abnormal pixel from the acquired image data; corrects a pixel value of the abnormal pixel on the basis of pixel values of peripheral pixels having polarizers whose types are different from a type of a polarizer of the abnormal pixel; generates images of the spectrally separated wavelengths from the image data in which the pixel value of the abnormal pixel has been corrected; and excludes the abnormal pixel, without correcting the pixel value of the abnormal pixel doing so can improve the quality of the generated image (Hashimoto, page 3, par 1); thus, one of the ordinary skill in the art would have been motivated to combine the references.
Regarding Claim 2, Nakata in view of Kurita further in view of Hashimoto teaches the image data processing device according to claim 1, wherein, in the process of correcting the pixel value of the abnormal pixel, the pixel value of the abnormal pixel is corrected based on pixel values of other pixels in a pixel set in which the abnormal pixel has been detected (Kurita, [0079] “the corrected pixel value of the defective pixel, the defect correcting section 36 specifies the estimated pixel value estimated on the basis of the polarization characteristics corresponding to the pixel values generated in the peripheral pixels in the polarization directions different from the polarization direction of the defective pixel, for example. In a case where the target polarized pixel C3(x, y) has been determined to be the defective pixel, the defect correcting section 36 specifies the estimated pixel value I3est, which has been calculated, as the corrected pixel value I3c(x, y) of the target polarized pixel C3(x, y)”).
Regarding Claim 3, Nakata in view of Kurita further in view of Hashimoto teaches the image data processing device according to claim 1, wherein, in the process of correcting the pixel value of the abnormal pixel, the pixel value of the abnormal pixel is corrected based on a relational expression of pixel values established between pixels constituting the pixel sets (Kurita, [0110] “the defect correcting section 36 compares the horizontal direction texture determination value dh with the horizontal direction texture determination value dh. As indicated by formulas (15) to (17), the defect correcting section 36 calculates the corrected pixel value of the pixel C3(x, y) using the pixel values of the peripheral pixels determined to be in the identical polarization direction to the defective pixel and to have the identical texture to the defective pixel”; see also [0144]).
Regarding Claim 4, Nakata in view of Kurita further in view of Hashimoto teaches the image data processing device according to claim 3, wherein the relational expression is obtained based on orientations of the transmission axes of respective pixels constituting the pixel sets (Kurita, [0109] “the defect correcting section 36 performs the texture detection using a Laplacian filter, for example. In a case where the target polarized pixel is the pixel C3(x, y) depicted in FIG. 9, the defect correcting section 36 calculates a horizontal direction texture determination value dh on the basis of a formula (13) and a vertical direction texture determination value dv on the basis of a formula (14)”).
Regarding Claim 5, Nakata in view of Kurita further in view of Hashimoto teaches the image data processing device according to claim 1, wherein the processor performs a process of correcting the pixel value of the abnormal pixel on the basis of the pixel values of the peripheral pixels in a case in which the pixel value is out of the predetermined range due to saturation (Kurita, [0086] “the defect detecting section 35 calculates the estimated pixel value I3est on the basis of the formula (3) and determines whether the pixel value I3(x, y) of the target polarized pixel C3(x, y) exceeds the allowable range Lp based on the estimated pixel value I3est”; [0087] “in a case where a ratio of the combinations with which the pixel value I3(x, y) of the target polarized pixel C3(x, y) has been determined to be out of the allowable range Lp is greater than a predetermined ratio, the defect detecting section 35 determines that the target polarized pixel C3(x, y) is the defective pixel…. [0088] In a case where the target polarized pixel C3(x, y) has been determined to be the defective pixel, the defect correcting section 36 calculates the corrected pixel value I3c(x, y) of the target polarized pixel C3(x, y) on the basis of the estimated pixel value I3est calculated with each combination”).
Regarding Claim 7, Nakata in view of Kurita further in view of Hashimoto teaches the image data processing device according to claim 1, wherein the process of detecting the abnormal pixel includes:
a process of detecting a pixel set including the abnormal pixel (Kurita, Fig. 14, [0123] “ in a case where the pixel C3 has not been determined as the defective pixel in the n−2th frame, but has been determined as the defective pixel successively in the n−1th frame to the n+1th frame, the defect detecting section 35 generates additional defect information indicating the pixel position of the pixel C3 that has been determined as the defective pixel in the predetermined number of successive frames (three frames in the figure). The defect detecting section 35 then outputs the additional defect information to the defect information storage section 34”); and
a process of specifying the abnormal pixel from the detected pixel set (Kurita, [0123] “in a case where the pixel C3 has not been determined as the defective pixel in the n−2th frame, but has been determined as the defective pixel successively in the n−1th frame to the n+1th frame, the defect detecting section 35 generates additional defect information indicating the pixel position of the pixel C3 that has been determined as the defective pixel in the predetermined number of successive frames (three frames in the figure). The defect detecting section 35 then outputs the additional defect information to the defect information storage section 34”).
Regarding Claim 9, Nakata in view of Kurita further in view of Hashimoto teaches the image data processing device according to claim 7, wherein, in the process of detecting the pixel set including the abnormal pixel, the pixel set including the abnormal pixel is detected on the basis of pixel values of pixels constituting the pixel set (Kurita, [0085] “The defect detecting section 35 calculates the estimated pixel value using the pixel values of the peripheral pixels in the 3×3 pixel region (peripheral pixel region) around the target polarized pixel C3(x, y). In a case where the polarization direction of the target polarized pixel C3 is 90°, the polarization directions of the peripheral pixels C1(x−1, y−1), C1(x−1, y+1), C1(x+1, y+1), and C1(x+1, y−1) are 0°, the polarization directions of the peripheral pixels C2(x−1, y) and C2(x+1, y) are 45°, and the polarization directions of the peripheral pixels C4(x, y−1) and C4(x, y+1) are 135°, the formula (3) holds in the ideal state as described above. In the second embodiment, each of the pixel values I1, I2, and I4 in the formula (3) is selected and used from the plurality of pixel values in the corresponding identical polarization direction within the peripheral pixel region. Here, the peripheral pixel region includes four pixels C1, two pixels C2, and two pixels C4. Therefore, the number of combinations of the peripheral pixels is sixteen (4×2×2). The defect detecting section 35 calculates the estimated pixel value for each of the sixteen combinations”).
Regarding Claim 10, Nakata in view of Kurita further in view of Hashimoto teaches the image data processing device according to claim 9, wherein, in the process of detecting the pixel set including the abnormal pixel, a sum of the pixel values of the pixels constituting the pixel set or a sum of values obtained by multiplying the pixel values of the pixels constituting the pixel set by a specific coefficient is calculated, and a pixel set in which the calculated sum is equal to or greater than a first threshold value is detected as the pixel set including the abnormal pixel (Kurita, [0095] “The defect detecting section 35 performs the defect detection using the peripheral information. In a case where the difference between the average value of the peripheral pixels in the identical polarization direction and the pixel value of the target polarized pixel is greater than a threshold value, the defect detecting section 35 determines that the target polarized pixel is the defective pixel candidate and proceeds to step ST13”; Kurita, [0144] “Further, the defect detecting section 35 calculates the pixel values I2 on the basis of the pixel values of the four peripheral pixels C2 that are in the different polarization directions from the polarization direction of the target polarized pixel C1. For example, the defect detecting section 35 specifies the average of the pixel values of the eight peripheral pixels C1 as the pixel value I1 and the average value of the pixel values of the four peripheral pixels C2 as the pixel value I2. In addition, the defect detecting section 35 calculates the pixel values I3 and I4 using the pixel values I1 and I2 calculated from the pixel values of the peripheral pixels to calculate the estimated pixel value of the target polarized pixel C1 on the basis of a formula”).
Regarding Claim 11, Nakata in view of Kurita further in view of Hashimoto teaches the image data processing device according to claim 9, wherein, in the process of specifying the abnormal pixel, a pixel whose pixel value is equal to or less than a second threshold value and/or a pixel whose pixel value is a saturation value is extracted from the pixel set including the abnormal pixel, and the abnormal pixel is specified (Kurita, [0093] “In the defect detection using the peripheral information, in a case where the difference between the average value of the peripheral pixels in the identical polarization direction and the pixel value of the target polarized pixel is out of a second allowable range, for example, in a case where a condition of a formula (10) is satisfied, it is determined that the target polarized pixel is a defective pixel candidate”).
Regarding Claim 12, Nakata in view of Kurita further in view of Hashimoto teaches the image data processing device according to claim 9, wherein, in the process of specifying the abnormal pixel, the abnormal pixel is specified on the basis of pixel values of pixels around the pixel set including the abnormal pixel (Kurita, [0081] “the defect detecting section 35 is not limited to calculating the estimated pixel value using the pixel values of the peripheral pixels in the 3×3 pixel region (peripheral pixel region) around the target polarized pixel C3(x, y)”; [0086] “the defect detecting section 35 calculates the estimated pixel value I3est on the basis of the formula (3) and determines whether the pixel value I3(x, y) of the target polarized pixel C3(x, y) exceeds the allowable range Lp based on the estimated pixel value I3est. (a) of FIG. 8 depicts the peripheral pixels in the 3×3 pixel region (peripheral pixel region) around the target polarized pixel C3(x, y)”).
Regarding Claim 13, Nakata in view of Kurita further in view of Hashimoto teaches the image data processing device according to claim 12, wherein the process of specifying the abnormal pixel includes:
a process of detecting the pixel set including the abnormal pixel from pixel sets around the pixel set including the abnormal pixel (Kurita, [0081] “the defect detecting section 35 is not limited to calculating the estimated pixel value using the pixel values of the peripheral pixels in the 3×3 pixel region (peripheral pixel region) around the target polarized pixel C3(x, y)”); and
a process of specifying the abnormal pixel from the pixel set including the abnormal pixel on the basis of a detection result of the pixel set including the abnormal pixel (Kurita, [0081] “the defect detecting section 35 may perform fitting to the polarization model formula using pixel values of other three pixels in the different polarization directions within a 2×2 pixel region including the target polarized pixel and calculate the estimated pixel value of the target polarized pixel. By performing the defect detection using the pixel values in the 2×2 pixel region in this manner, the estimated pixel value can be calculated when the pixel values of two pixels from the beginning of the second line are obtained. This enables prompt start of the pixel defect detection and defect correction”).
Regarding Claim 14, Nakata in view of Kurita further in view of Hashimoto teaches the image data processing device according to claim 13, wherein a range in which the pixel set including the abnormal pixel is detected is switched according to a resolution of the optical system (Nakata, [0078] The normal image generation section 36 generates a normal image having, as each pixel value, three elements of the normal vector obtained for each pixel. The image can basically have the same resolution as that of the captured image. On the other hand, depending on the resolution required for the normal vectors and a distance image in a succeeding step, the normal image may be generated at a lower resolution than the captured image”).
Regarding Claim 15, Nakata in view of Kurita further in view of Hashimoto teaches the image data processing device according to claim 12, wherein the process of specifying the abnormal pixel includes:
a process of estimating a pixel value of a pixel from the pixel values of the peripheral pixels (Kurita, [0075] “The defect detecting section 35 calculates the estimated pixel value of the target polarized pixel using the pixel values of the peripheral pixels in the periphery of the target polarized pixel”); and
a process of specifying a pixel in which a difference from the estimated pixel value is equal to or greater than a third threshold value as the abnormal pixel (Kurita, [0073] “he defect detecting section determines that the target polarized pixel is the defective pixel. Since the target polarized pixel is the known defective pixel or the absolute difference value with the estimated pixel value is greater than the threshold value, the defect detecting section 35 determines that the target polarized pixel is the defective pixel”).
Regarding Claim 16, Nakata in view of Kurita further in view of Hashimoto teaches the image data processing device according to claim 15, wherein, in the process of estimating the pixel value of the pixel from the pixel values of the peripheral pixels, the pixel value is estimated from pixel values of peripheral pixels including polarizers of the same type (Kurita, [0062] “ the image processing apparatus may correct the defective pixel using the pixel values of the peripheral pixels in an identical polarization direction to the polarization direction of the defective pixel”; Kurita, [0104] “the defect correcting section calculates the corrected pixel value of the target polarized pixel using the pixel values of the peripheral pixels in the identical polarization direction”).
Regarding Claim 17, Nakata teaches:
An image data processing method that processes image data captured by an imaging device including an optical system which spectrally separates incident light into a plurality of wavelengths, polarizes light with the spectrally separated wavelengths in a specific direction, and emits the polarized light and an imaging element including a plurality of pixel sets each of which includes different types of polarizers (Nakata, [0014] “the present disclosure, there is provided an electronic apparatus including: [0015] a display unit that is capable of emitting light at a plurality of different emission wavelengths; [0016] an imaging unit that is disposed on an opposite side to a display surface of the display unit; and [0017] an abnormality detection unit that detects an abnormality on the display surface on the basis of a plurality of images captured by the imaging unit in a state in which at least a part of the display surface is caused to emit light at each of the plurality of emission wavelengths”), the image data processing device comprising:
a process of acquiring the image data captured by the imaging device (Nakata, “[0032] an image communication unit that transmits the image captured by the imaging unit”);
a process of detecting an abnormal pixel from the acquired image data (Nakata, [0025] An abnormality determination unit that determines a type of the abnormality”);
a process of generating images of the spectrally separated wavelengths from the image data in which the pixel value of the abnormal pixel has been corrected in a case in which the abnormal pixel is detected (Nakata, [0031] a correction determination unit that determines whether or not correction by the correction processing unit is effective; [0032] an image communication unit that transmits the image captured by the imaging unit and information regarding the abnormality to an information processing apparatus and receives the image corrected by the information processing apparatus …; and [0033] an output unit that outputs the image corrected by the information processing apparatus”).
While Nakata teaches detecting abnormality and correcting the abnormality, it fails to explicitly teach:
a process of detecting a pixel whose pixel value is out of a predetermined range as an abnormal pixel from the acquired image data;
a process of correcting a pixel value of the abnormal pixel on the basis of pixel values of peripheral pixels having polarizers whose types are different from a type of a polarizer of the abnormal pixel in a case in which the abnormal pixel is detected;
In the same filed of endeavor, Kurita teaches:
a process of detecting a pixel whose pixel value is out of a predetermined range as an abnormal pixel from the acquired image data (Kurita, [0010] “a difference between the pixel value of the target polarized pixel and the pixel value estimated is out of the predetermined first allowable range and in a case where the difference is within the predetermined first allowable range and a difference between the pixel value of the target polarized pixel and the pixel values of the peripheral pixels in the identical polarization direction is out of the predetermined second allowable range, the defect detecting section may determine that the target polarized pixel is the defective pixel”; Kurita, [0068] “On the basis of the pixel value of the target polarized pixel and the estimated pixel value (hereinafter referred to as “estimated pixel value”), the defect detecting section 35 determines whether the target polarized pixel is the defective pixel…in a case where the difference between the pixel value of the target polarized pixel and the estimated pixel value is out of a predetermined allowable range, the defect detecting section 35 determines that the target polarized pixel is the defective pixel”);
a process of correcting a pixel value of the abnormal pixel on the basis of pixel values of peripheral pixels having polarizers whose types are different from a type of a polarizer of the abnormal pixel in a case in which the abnormal pixel is detected (Kurita, [0062] “the image processing apparatus may correct the defective pixel using the pixel values of the peripheral pixels in an identical polarization direction to the polarization direction of the defective pixel”; Kurita, [0068] “The defect detecting section 35 estimates the pixel value of the target polarized pixel on the basis of the polarization model formula after fitting that indicates polarization characteristics corresponding to the pixel values of peripheral pixels in different polarization directions.”; Kurita, [0069] “the defect correcting section 36 corrects the defective pixel of the polarized image indicated by the defect information using the peripheral pixels positioned in the periphery of the defective pixel”; Kurita, [0079] “the corrected pixel value of the defective pixel, the defect correcting section 36 specifies the estimated pixel value estimated on the basis of the polarization characteristics corresponding to the pixel values generated in the peripheral pixels in the polarization directions different from the polarization direction of the defective pixel”);
Nakata and Kurita are considered analogous art as they are reasonably pertinent to the same field of endeavor of image processing. Therefore, it would have been obvious to one of the ordinary skill in the are before the effective filing date of the claimed invention to modify the invention of Nakata with the process of detecting abnormalities as taught by Kurita to make the invention that processes the acquired image and detects abnormal pixels whole pixel value is out of a predetermined range as an abnormal pixel from the acquired image data; corrects a pixel value of the abnormal pixel on the basis of pixel values of peripheral pixels having polarizers whose types are different from a type of a polarizer of the abnormal pixel; and generates images of the spectrally separated wavelengths from the image data in which the pixel value of the abnormal pixel has been corrected; doing so can efficiently detect defects in a polarizing imaging device (Kurita, [0006]); thus, one of the ordinary skill in the art would have been motivated to combine the references.
However, Nakata in view of Kurita does not explicitly teach:
wherein a process of generating the images of the spectrally separated wavelengths from the image data excluding the abnormal pixel, without correcting the pixel value of the abnormal pixel, in a case in which the pixel value is out of the predetermined range due to a failure.
In the same field of endeavor Hashimoto teaches:
wherein a process of generating the images of the spectrally separated wavelengths from the image data excluding the abnormal pixel, without correcting the pixel value of the abnormal pixel, in a case in which the pixel value is out of the predetermined range due to a failure (Hashimoto, page 8, para 5 – 7, Fig. 5, “The abnormal pixel value determination unit 40 determines whether each pixel value is a normal pixel value or an abnormal pixel value for each pixel …The abnormal pixel value removing unit 41 removes the abnormal pixel value determined by the abnormal pixel value determining unit 40… Specifically, as shown in FIG. 6, the abnormal pixel value determination unit 40 normalizes the pixel values Ik (x, y) included in a predetermined range ... The pixel value is determined, and the other pixel values are determined as abnormal pixel values. In the case of FIG. 6, since the pixel value I2 (x, y) is out of the predetermined range, the abnormal pixel value determination unit 40 determines that the pixel value is an abnormal pixel value, and the abnormal pixel removal unit 41 removes it”; Hashimoto, page 10, para 2, “correction method generates a phase differential image without performing correction in the state of the image data Dk, and then generates a pixel value (abnormal phase differential value) corresponding to the abnormal pixel value on the image data Dk”).
Nakata, Kurita and Hashimoto are considered analogous art as they are reasonably pertinent to the same field of endeavor of image processing. Therefore, it would have been obvious to one of the ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Nakata in view of Kurita with the process of removing abnormal pixels as taught by Hashimoto to make the invention that processes the acquired image and detects abnormal pixels whole pixel value is out of a predetermined range as an abnormal pixel from the acquired image data; corrects a pixel value of the abnormal pixel on the basis of pixel values of peripheral pixels having polarizers whose types are different from a type of a polarizer of the abnormal pixel; generates images of the spectrally separated wavelengths from the image data in which the pixel value of the abnormal pixel has been corrected; and excludes the abnormal pixel, without correcting the pixel value of the abnormal pixel doing so can improve the quality of the generated image (Hashimoto, page 3, par 1); thus, one of the ordinary skill in the art would have been motivated to combine the references.
Regarding Claim 19, Nakata in view of Kurita further in view of Hashimoto teaches a non-transitory, computer-readable tangible recording medium that records thereon a program for causing, when read by a computer, the computer to execute the image data processing method according to claim 17 (Nakata, [0176] “the CPU 56 controls imaging by the camera 52 and data accumulation operation in the memory 53, and controls data transmission of the wireless transmitter 55 from the memory 53 to a data reception device”; Kurita, [0168] “The storage section 7690 may include a read only memory (ROM) that stores various kinds of programs executed by the microcomputer and a random access memory (RAM) that stores various kinds of parameters”).
Regarding Claim 21, Nakata in view of Kurita further in view of Hashimoto teaches as imaging system comprising:
an imaging device that includes an optical system which spectrally separates incident light into a plurality of wavelengths, polarizes light with the spectrally separated wavelengths in a specific direction, and emits the polarized light and an imaging element including a plurality of pixel sets each of which includes different types of polarizers (Nakata, [0040] The imaging unit may include: a plurality of photoelectric conversion units that photoelectrically converts light incident through the display unit; and [0042] a plurality of polarization elements that is disposed on a light incident side of at least one of the plurality of photoelectric conversion units”; Nakata, [0044] “The plurality of polarization elements may include a plurality of types of polarization elements that detects different polarization states”; Nakata, [0125] In a case where the user starts the camera module 3, as illustrated in FIG. 7A, the partial display region 20, which is within the angle-of-view range of the camera module 3, is caused to emit light at a first emission wavelength (Step 313)”; Nakata, [0126] While the light is emitted in Step S13, the imaging unit 4 performs exposure (Step S14), and the signal processing unit 32 performs various signal processing to generate captured image data (Step S15)”); and
the image data processing device according to claim 1 that processes image data captured by the imaging device (see rejection for claim 1 above).
Claims 8, 18, 20 and 22 are rejected under 35 U.S.C. 103 as being unpatentable over Nakata (US 20230042435 A1; hereafter referred to as Nakata) in view of Kurita et al (US 20210281786 A1; hereafter referred to as Kurita).
Regarding Claim 8 Nakata teaches:
An image data processing device that processes image data captured by an imaging device including an optical system which spectrally separates incident light into a plurality of wavelengths, polarizes light with the spectrally separated wavelengths in a specific direction, and emits the polarized light and an imaging element including a plurality of pixel sets each of which includes different types of polarizers (Nakata, [0014] “the present disclosure, there is provided an electronic apparatus including: [0015] a display unit that is capable of emitting light at a plurality of different emission wavelengths; [0016] an imaging unit that is disposed on an opposite side to a display surface of the display unit; and [0017] an abnormality detection unit that detects an abnormality on the display surface on the basis of a plurality of images captured by the imaging unit in a state in which at least a part of the display surface is caused to emit light at each of the plurality of emission wavelengths”), the image data processing device comprising:
a processor (Nakata, [0096] “an application processor 22”) configured to perform:
a process of acquiring the image data (Nakata, “[0032] an image communication unit that transmits the image captured by the imaging unit”);
a process of detecting an abnormal pixel from the acquired image data (Nakata, [0025] An abnormality determination unit that determines a type of the abnormality”);
a process of generating images of the spectrally separated wavelengths from the image data in which the pixel value of the abnormal pixel has been corrected in a case in which the abnormal pixel is detected (Nakata, [0031] a correction determination unit that determines whether or not correction by the correction processing unit is effective; [0032] an image communication unit that transmits the image captured by the imaging unit and information regarding the abnormality to an information processing apparatus and receives the image corrected by the information processing apparatus …; and [0033] an output unit that outputs the image corrected by the information processing apparatus”);
wherein the pixel set comprises pixels respectively from each of the different types of polarizers (Nakata, [0049] “pixels 124a and 124b indicated by thick frames are provided with polarizers. Thick oblique lines in the pixels 124a and 124b represent wires included in the polarizers. Specifically, the pixels 124a and 124b include polarizers with different main axis angles. FIG. 3 illustrates two types of polarizers with main axis angles orthogonal to each other, and further different pixels are used to provide four types of polarizers”; Nakata, [0200] A plurality of types of polarizers with a plurality of types of main axis angles may be provided in a plurality of image capturing sections (or in pixel units within a single image capturing section). According to this modified example, a polarization image (or a plurality of types of polarization images corresponding to a plurality of directions) can be obtained”);
While Nakata teaches detecting abnormality and correcting the abnormality, it fails to explicitly teach:
a process of detecting a pixel whose pixel value is out of a predetermined range as an abnormal pixel from the acquired image data;
a process of correcting a pixel value of the abnormal pixel on the basis of pixel values of peripheral pixels having polarizers whose types are different from a type of a polarizer of the abnormal pixel in a case in which the abnormal pixel is detected;
a process of detecting a pixel set including the abnormal pixel; and
a process of specifying the abnormal pixel from the detected pixel set.
In the same filed of endeavor, Kurita teaches:
a process of detecting a pixel whose pixel value is out of a predetermined range as an abnormal pixel from the acquired image data (Kurita, [0010] “a difference between the pixel value of the target polarized pixel and the pixel value estimated is out of the predetermined first allowable range and in a case where the difference is within the predetermined first allowable range and a difference between the pixel value of the target polarized pixel and the pixel values of the peripheral pixels in the identical polarization direction is out of the predetermined second allowable range, the defect detecting section may determine that the target polarized pixel is the defective pixel”; Kurita, [0068] “On the basis of the pixel value of the target polarized pixel and the estimated pixel value (hereinafter referred to as “estimated pixel value”), the defect detecting section 35 determines whether the target polarized pixel is the defective pixel…in a case where the difference between the pixel value of the target polarized pixel and the estimated pixel value is out of a predetermined allowable range, the defect detecting section 35 determines that the target polarized pixel is the defective pixel”);
a process of correcting a pixel value of the abnormal pixel on the basis of pixel values of peripheral pixels having polarizers whose types are different from a type of a polarizer of the abnormal pixel in a case in which the abnormal pixel is detected (Kurita, [0062] “the image processing apparatus may correct the defective pixel using the pixel values of the peripheral pixels in an identical polarization direction to the polarization direction of the defective pixel”; Kurita, [0068] “The defect detecting section 35 estimates the pixel value of the target polarized pixel on the basis of the polarization model formula after fitting that indicates polarization characteristics corresponding to the pixel values of peripheral pixels in different polarization directions.”; Kurita, [0069] “the defect correcting section 36 corrects the defective pixel of the polarized image indicated by the defect information using the peripheral pixels positioned in the periphery of the defective pixel”; Kurita, [0079] “the corrected pixel value of the defective pixel, the defect correcting section 36 specifies the estimated pixel value estimated on the basis of the polarization characteristics corresponding to the pixel values generated in the peripheral pixels in the polarization directions different from the polarization direction of the defective pixel”);
a process of detecting a pixel set including the abnormal pixel (Kurita, Fig. 14, [0123] “ in a case where the pixel C3 has not been determined as the defective pixel in the n−2th frame, but has been determined as the defective pixel successively in the n−1th frame to the n+1th frame, the defect detecting section 35 generates additional defect information indicating the pixel position of the pixel C3 that has been determined as the defective pixel in the predetermined number of successive frames (three frames in the figure). The defect detecting section 35 then outputs the additional defect information to the defect information storage section 34”); and
a process of specifying the abnormal pixel from the detected pixel set (Kurita, [0123] “in a case where the pixel C3 has not been determined as the defective pixel in the n−2th frame, but has been determined as the defective pixel successively in the n−1th frame to the n+1th frame, the defect detecting section 35 generates additional defect information indicating the pixel position of the pixel C3 that has been determined as the defective pixel in the predetermined number of successive frames (three frames in the figure). The defect detecting section 35 then outputs the additional defect information to the defect information storage section 34”).
Nakata and Kurita are considered analogous art as they are reasonably pertinent to the same field of endeavor of image processing. Therefore, it would have been obvious to one of the ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Nakata with the process of detecting abnormalities as taught by Kurita to make the invention that processes the acquired image and detects abnormal pixels whole pixel value is out of a predetermined range as an abnormal pixel from the acquired image data; corrects a pixel value of the abnormal pixel on the basis of pixel values of peripheral pixels having polarizers whose types are different from a type of a polarizer of the abnormal pixel; and generates images of the spectrally separated wavelengths from the image data in which the pixel value of the abnormal pixel has been corrected; doing so can efficiently detect defects in a polarizing imaging device (Kurita, [0006]); thus, one of the ordinary skill in the art would have been motivated to combine the references.
Regarding Claim 18 Nakata teaches:
An image data processing method that processes image data captured by an imaging device including an optical system which spectrally separates incident light into a plurality of wavelengths, polarizes light with the spectrally separated wavelengths in a specific direction, and emits the polarized light and an imaging element including a plurality of pixel sets each of which includes different types of polarizers (Nakata, [0014] “the present disclosure, there is provided an electronic apparatus including: [0015] a display unit that is capable of emitting light at a plurality of different emission wavelengths; [0016] an imaging unit that is disposed on an opposite side to a display surface of the display unit; and [0017] an abnormality detection unit that detects an abnormality on the display surface on the basis of a plurality of images captured by the imaging unit in a state in which at least a part of the display surface is caused to emit light at each of the plurality of emission wavelengths”), the image data processing device comprising:
a process of acquiring the image data (Nakata, “[0032] an image communication unit that transmits the image captured by the imaging unit”);
a process of detecting an abnormal pixel from the acquired image data (Nakata, [0025] An abnormality determination unit that determines a type of the abnormality”);
a process of generating images of the spectrally separated wavelengths from the image data in which the pixel value of the abnormal pixel has been corrected in a case in which the abnormal pixel is detected (Nakata, [0031] a correction determination unit that determines whether or not correction by the correction processing unit is effective; [0032] an image communication unit that transmits the image captured by the imaging unit and information regarding the abnormality to an information processing apparatus and receives the image corrected by the information processing apparatus …; and [0033] an output unit that outputs the image corrected by the information processing apparatus”);
wherein the pixel set comprises pixels respectively from each of the different types of polarizers (Nakata, [0049] “pixels 124a and 124b indicated by thick frames are provided with polarizers. Thick oblique lines in the pixels 124a and 124b represent wires included in the polarizers. Specifically, the pixels 124a and 124b include polarizers with different main axis angles. FIG. 3 illustrates two types of polarizers with main axis angles orthogonal to each other, and further different pixels are used to provide four types of polarizers”; Nakata, [0200] A plurality of types of polarizers with a plurality of types of main axis angles may be provided in a plurality of image capturing sections (or in pixel units within a single image capturing section). According to this modified example, a polarization image (or a plurality of types of polarization images corresponding to a plurality of directions) can be obtained”);
While Nakata teaches detecting abnormality and correcting the abnormality, it fails to explicitly teach:
a process of detecting a pixel whose pixel value is out of a predetermined range as an abnormal pixel from the acquired image data;
a process of correcting a pixel value of the abnormal pixel on the basis of pixel values of peripheral pixels having polarizers whose types are different from a type of a polarizer of the abnormal pixel in a case in which the abnormal pixel is detected;
a process of detecting a pixel set including the abnormal pixel; and
a process of specifying the abnormal pixel from the detected pixel set.
In the same filed of endeavor, Kurita teaches:
a process of detecting a pixel whose pixel value is out of a predetermined range as an abnormal pixel from the acquired image data (Kurita, [0010] “a difference between the pixel value of the target polarized pixel and the pixel value estimated is out of the predetermined first allowable range and in a case where the difference is within the predetermined first allowable range and a difference between the pixel value of the target polarized pixel and the pixel values of the peripheral pixels in the identical polarization direction is out of the predetermined second allowable range, the defect detecting section may determine that the target polarized pixel is the defective pixel”; Kurita, [0068] “On the basis of the pixel value of the target polarized pixel and the estimated pixel value (hereinafter referred to as “estimated pixel value”), the defect detecting section 35 determines whether the target polarized pixel is the defective pixel…in a case where the difference between the pixel value of the target polarized pixel and the estimated pixel value is out of a predetermined allowable range, the defect detecting section 35 determines that the target polarized pixel is the defective pixel”);
a process of correcting a pixel value of the abnormal pixel on the basis of pixel values of peripheral pixels having polarizers whose types are different from a type of a polarizer of the abnormal pixel in a case in which the abnormal pixel is detected (Kurita, [0062] “the image processing apparatus may correct the defective pixel using the pixel values of the peripheral pixels in an identical polarization direction to the polarization direction of the defective pixel”; Kurita, [0068] “The defect detecting section 35 estimates the pixel value of the target polarized pixel on the basis of the polarization model formula after fitting that indicates polarization characteristics corresponding to the pixel values of peripheral pixels in different polarization directions.”; Kurita, [0069] “the defect correcting section 36 corrects the defective pixel of the polarized image indicated by the defect information using the peripheral pixels positioned in the periphery of the defective pixel”; Kurita, [0079] “the corrected pixel value of the defective pixel, the defect correcting section 36 specifies the estimated pixel value estimated on the basis of the polarization characteristics corresponding to the pixel values generated in the peripheral pixels in the polarization directions different from the polarization direction of the defective pixel”);
a process of detecting a pixel set including the abnormal pixel (Kurita, Fig. 14, [0123] “ in a case where the pixel C3 has not been determined as the defective pixel in the n−2th frame, but has been determined as the defective pixel successively in the n−1th frame to the n+1th frame, the defect detecting section 35 generates additional defect information indicating the pixel position of the pixel C3 that has been determined as the defective pixel in the predetermined number of successive frames (three frames in the figure). The defect detecting section 35 then outputs the additional defect information to the defect information storage section 34”); and
a process of specifying the abnormal pixel from the detected pixel set (Kurita, [0123] “in a case where the pixel C3 has not been determined as the defective pixel in the n−2th frame, but has been determined as the defective pixel successively in the n−1th frame to the n+1th frame, the defect detecting section 35 generates additional defect information indicating the pixel position of the pixel C3 that has been determined as the defective pixel in the predetermined number of successive frames (three frames in the figure). The defect detecting section 35 then outputs the additional defect information to the defect information storage section 34”).
Nakata and Kurita are considered analogous art as they are reasonably pertinent to the same field of endeavor of image processing. Therefore, it would have been obvious to one of the ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Nakata with the process of detecting abnormalities as taught by Kurita to make the invention that processes the acquired image and detects abnormal pixels whole pixel value is out of a predetermined range as an abnormal pixel from the acquired image data; corrects a pixel value of the abnormal pixel on the basis of pixel values of peripheral pixels having polarizers whose types are different from a type of a polarizer of the abnormal pixel; and generates images of the spectrally separated wavelengths from the image data in which the pixel value of the abnormal pixel has been corrected; doing so can efficiently detect defects in a polarizing imaging device (Kurita, [0006]); thus, one of the ordinary skill in the art would have been motivated to combine the references.
Regarding Claim 20, Nakata in view of Kurita teaches a non-transitory, computer-readable tangible recording medium that records thereon a program for causing, when read by a computer, the computer to execute the image data processing method according to claim 18 (Nakata, [0176] “the CPU 56 controls imaging by the camera 52 and data accumulation operation in the memory 53, and controls data transmission of the wireless transmitter 55 from the memory 53 to a data reception device”; Kurita, [0168] “The storage section 7690 may include a read only memory (ROM) that stores various kinds of programs executed by the microcomputer and a random access memory (RAM) that stores various kinds of parameters”).
Regarding Claim 22, Nakata in view of Kurita teaches as imaging system comprising:
an imaging device that includes an optical system which spectrally separates incident light into a plurality of wavelengths, polarizes light with the spectrally separated wavelengths in a specific direction, and emits the polarized light and an imaging element including a plurality of pixel sets each of which includes different types of polarizers (Nakata, [0040] The imaging unit may include: a plurality of photoelectric conversion units that photoelectrically converts light incident through the display unit; and [0042] a plurality of polarization elements that is disposed on a light incident side of at least one of the plurality of photoelectric conversion units”; Nakata, [0044] “The plurality of polarization elements may include a plurality of types of polarization elements that detects different polarization states”; Nakata, [0125] In a case where the user starts the camera module 3, as illustrated in FIG. 7A, the partial display region 20, which is within the angle-of-view range of the camera module 3, is caused to emit light at a first emission wavelength (Step 313)”; Nakata, [0126] While the light is emitted in Step S13, the imaging unit 4 performs exposure (Step S14), and the signal processing unit 32 performs various signal processing to generate captured image data (Step S15)”); and
the image data processing device according to claim 8 that processes image data captured by the imaging device (see rejection for claim 8 above).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
WO 2020250774 A1 IMAGING DEVICE The present invention provides an imaging device for capturing a good-quality multispectral image. An imaging device (1) is provided with: an imaging optical system (10) having a pupil region divided into a plurality of regions including a first pupil region and a second pupil region different from the first pupil region, and provided with a polarization filter that polarizes light passing through the first pupil region and the second pupil region in directions different from each other; an imaging element (100) including a first pixel that receives the light passing through the first pupil region, and a second pixel that receives the light passing through the second pupil region; and a signal processing unit (200) that processes a signal outputted from the imaging element (100), and outputs at least first image data comprising an output signal of the first pixel, and second image data comprising an output signal of the second pixel. In the imaging optical system (10), the wavelengths of light passing through the first pupil region and the second pupil region are different from each other, and the aberration properties of regions corresponding to the first pupil region and the second pupil region are different from each other.
Contact Information
Any inquiry concerning this communication or earlier communications from the examiner should be directed to VAISALI RAO KOPPOLU whose telephone number is (571)270-0273. The examiner can normally be reached Monday - Friday 8:30 - 5.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Jennifer Mehmood can be reached at (571) 272-2976. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format.
For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
VAISALI RAO. KOPPOLU
Examiner
Art Unit 2664
/VAISALI RAO KOPPOLU/Examiner of Art Unit 2664
Prosecution Insights