DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Receipt is acknowledged that application is a National Stage application of PCT/JP2023/009946 dated 03/14/2023. Receipt is acknowledged that application claims priority to foreign application with application number JP2022-073791 dated 04/27/2022. Copies of certified papers required by 37 CFR 1.55 have been received. Priority is acknowledged under 35 USC 119(e) and 37 CFR 1.78. Claims 1-12 have been afforded the benefit of filing date 04/27/2022.
Drawings
The drawings are objected to as failing to comply with 37 CFR 1.84(p)(5) because they include the following reference character(s) not mentioned in the description: “S505” in FIG. 21. Corrected drawing sheets in compliance with 37 CFR 1.121(d), or amendment to the specification to add the reference character(s) in the description in compliance with 37 CFR 1.121(b) are required in reply to the Office action to avoid abandonment of the application. Any amended replacement drawing sheet should include all of the figures appearing on the immediate prior version of the sheet, even if only one figure is being amended. Each drawing sheet submitted after the filing date of an application must be labeled in the top margin as either “Replacement Sheet” or “New Sheet” pursuant to 37 CFR 1.121(d). If the changes are not accepted by the examiner, the applicant will be notified and informed of any required corrective action in the next Office action. The objection to the drawings will not be held in abeyance.
Claim Objections
Claims 1, 6, and 12 are objected to because of the following informalities:
Claim 1 should read: “…a narrowband signal generator that uses a coefficient matrix to generate, from the plurality of wavelength signals output from the multispectral sensor section, a plurality of (M: M>N)) narrowband signals that are narrower in bandwidth than the plurality of wavelength signals; and a controller that calculates an evaluation value by performing weighted average processing on the plurality of wavelength signals or a plurality of detection values generated using the plurality of wavelength signals, and performs exposure control of the multispectral sensor section on a basis of the calculated evaluation value.”
Claim 6 should read: “…on a basis of elements included in the coefficient matrix.”
Claim 12 should read: “…generating, from pixel signals that are a plurality of (N) wavelength signals output from a multispectral sensor, a plurality of (M: M>N) narrowband signals narrower in bandwidth than the plurality of wavelength signals using a coefficient matrix; and calculating an evaluation value by performing weighted average processing on the plurality of wavelength signals or a plurality of detection values generated using the plurality of wavelength signals, and performing exposure control of the multispectral sensor on a basis of the calculated evaluation value.”
Appropriate correction is required.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 5, and 9-12 are rejected under 35 U.S.C. 103 as being unpatentable over Rinehart et al. (U.S. Patent No. 2022/0191404 A1), hereinafter Rinehart, in view of Yi et al. (Yi, C., Zhao, Y.-q., Chan, J. C.-W., & Kong, S. G. (2020). Joint Spatial-spectral Resolution Enhancement of Multispectral Images with Spectral Matrix Factorization and Spatial Sparsity Constraints. Remote Sensing, 12(6), 993. https://doi.org/10.3390/rs12060993), hereinafter Yi.
Regarding claim 1, Rinehart teaches an imaging device (Rinehart, para 73: “image capture device”) comprising: a sensor section that outputs a plurality of (N) wavelength signals as pixel signals (Rinehart, para 73: “method 900 includes capturing, by an image capture device, a first image comprising a plurality of pixels, wherein each pixel comprises a plurality of channels”); and a controller that calculates an evaluation value (Rinehart, para 76: “gain”) by performing weighted average processing on the plurality of wavelength signals or a plurality of detection values generated using the plurality of wavelength signals (Rinehart, para 80: “determining the average pixel intensities (block 904), determining the weighted average (block 906), and setting the gain (block 908) are performed responsive to determining that at least two of the plurality of channels are overexposed”; see step 908), and performs exposure control of the sensor section on a basis of the calculated evaluation value (Rinehart, para 77: “The second exposure parameters are based on the gain”; see steps outlined in Figure 9).
However, Rinehart fails to explicitly teach wherein the sensor section is a multispectral sensor section; and a narrowband signal generator that uses a coefficient matrix to generate, from the plurality of wavelength signals output from the multispectral sensor section, a plurality of (M: M>N)) narrowband signals that are narrower in bandwidth than the plurality of wavelength signals.
Yi teaches an imaging device comprising a multispectral sensor section (Yi, sensor that generates input MSI images, first sentence in last para on pg. 3: “The proposed scheme recovers a high spatial resolution HSI from an input low spatial resolution MSI based on high spatial-spectral correlation in an input low resolution MSI and the desired high quality HSI”; see multispectral ALI sensor in section 4.3 on pg. 16); and a narrowband signal generator that uses a coefficient matrix to generate, from the plurality of wavelength signals output from the multispectral sensor section, a plurality of (M: M>N)) narrowband signals that are narrower in bandwidth than the plurality of wavelength signals (Yi, abstract: “Reconstructed hyperspectral images (HSIs) from an input multispectral image represent the same scene in higher spatial resolution, with more spectral bands of narrower wavelength width than the input multispectral image.”; coefficient matrices used to generate the HSI spectral bands are described in section 3.3, pg. 8-9). Multispectral image sensors capture information about the environment that is not captured by RGB camera sensors (i.e., infrared spectra). Furthermore, generating an increased number of narrowband signals from the multispectral data further increases the precision of data captured.
Accordingly, Rinehart discloses an imaging device, but does not disclose a multispectral sensor and subsequently generated narrowband signals. Yi discloses a multispectral sensor and subsequently generated narrowband signals. Thus, Rinehart and Yi each disclose an imaging device. A person of ordinary skill in the art, before the effective filing date of the claimed invention, would have recognized that the image sensor and output image signals taught by Rinehart could have been substituted for the sensor and generated image signals taught by Yi because both serve the purpose of capturing pixel signal data. Furthermore, a person of ordinary skill in the art would have been able to carry out the substitution. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to substitute the image sensor and output image signals of Rinehart for the multispectral sensor and subsequently generated narrowband signals, taught by Yi, according to known methods to yield the predictable result of increasing the spectral resolution and coverage of the image signals used to inform the exposure control. Doing so allows for increased control over which bands contribute to image over-/under-exposure in the weighted average calculation.
Regarding claim 5 (dependent on claim 1), Rinehart in view of Yi teaches further comprising a weighting coefficient determination section that determines a plurality of weighting coefficients, each of which is assigned to a corresponding one of the wavelength signals or a corresponding one of the detection values, wherein the controller performs the weighted average processing on a basis of the plurality of weighting coefficients determined by the weighting coefficient determination section (Rinehart, x, y, and 0 multiplied by corresponding channels, para 63: “In the example scenario of FIG. 7, gain be D/(x*R+y*G+0*B), where D is the desired average pixel intensity, R is the average intensity of the red channel, G is the average intensity of the green channel, and B is the average intensity of the blue channel. B is multiplied by 0 in order to remove the blue channel 406 from the weighted average”).
Regarding claim 9 (dependent on claim 5), Rinehart in view of Yi teaches wherein the weighting coefficient determination section determines the plurality of weighting coefficients for each of divided regions obtained by dividing a pixel region of the multispectral sensor section (Rinehart, system identifies object regions, dividing the image into object region (i.e., license plate) and non-object region in para 50, and determines the weighting coefficients for these regions, see results in Figure 8A-8D; para 71: “FIGS. 8A-8D show an example scenario in which lighting conditions cause auto exposure operations of a first mode of an image capture device (e.g., a non-weighted average mode) to overexpose one or more channels in an output image…For example, the second mode can use a channel-weighted average or selective channel-weighted average to determine a gain that decreases the likelihood of overexposing one or more channels”).
Regarding claim 10 (dependent on claim 1), Rinehart in view of Yi teaches wherein the controller calculates the evaluation value by classifying the wavelength signals according to divided regions obtained by dividing a pixel region of the multispectral sensor section (Rinehart, system identifies object regions, dividing the image into object region (i.e., license plate) and non-object region, para 50: “the object 408 is a license plate, though other objects might be of interest in a given scene, such as vehicles, street signs, or other objects that can be detected using image processing techniques”), and performing the weighted average processing on the plurality of detection values generated for each of the divided regions (Rinehart, weighting coefficients are determined for the image and applied to the image, and therefore applied to each of the regions, see results in Figure 8A-8D; para 71: “FIGS. 8A-8D show an example scenario in which lighting conditions cause auto exposure operations of a first mode of an image capture device (e.g., a non-weighted average mode) to overexpose one or more channels in an output image…For example, the second mode can use a channel-weighted average or selective channel-weighted average to determine a gain that decreases the likelihood of overexposing one or more channels”).
Regarding claim 11 (dependent on claim 1), Rinehart in view of Yi teaches wherein the controller performs the exposure control on a basis of the evaluation value (Rinehart, exposure set on the basis of the gain value, para 77: “The second exposure parameters are based on the gain”) and information obtained from a device that performs processing using the plurality of narrowband signals (Exposure is based on the wavelength signals – through the RGB values in the gain of Rinehart, see para 63; Narrowband signals taught in combination with Yi, see claim 1 rejection.).
Regarding claim 12, Rinehart teaches a signal processing method (Rinehart, Figure 9) comprising: pixel signals that are a plurality of (N) wavelength signals output from a sensor (Rinehart, para 73: “method 900 includes capturing, by an image capture device, a first image comprising a plurality of pixels, wherein each pixel comprises a plurality of channels”); and calculating an evaluation value (Rinehart, para 76: “gain”) by performing weighted average processing on the plurality of wavelength signals or a plurality of detection values generated using the plurality of wavelength signals (Rinehart, para 80: “determining the average pixel intensities (block 904), determining the weighted average (block 906), and setting the gain (block 908) are performed responsive to determining that at least two of the plurality of channels are overexposed”; see step 908), and performing exposure control of the sensor on a basis of the calculated evaluation value (Rinehart, para 77: “The second exposure parameters are based on the gain”; see steps outlined in Figure 9).
However, Rinehart fails to explicitly teach wherein the sensor is a multispectral sensor; and generating, from pixel signals that are a plurality of (N) wavelength signals output from a multispectral sensor, a plurality of (M: M>N) narrowband signals narrower in bandwidth than the plurality of wavelength signals using a coefficient matrix.
Yi teaches an signal processing method comprising a multispectral sensor section (Yi, Input MSI images, first sentence in last para on pg. 3: “The proposed scheme recovers a high spatial resolution HSI from an input low spatial resolution MSI based on high spatial-spectral correlation in an input low resolution MSI and the desired high quality HSI” ; see multispectral ALI sensor in section 4.3 on pg. 16); and a narrowband signal generator that uses a coefficient matrix to generate, from the plurality of wavelength signals output from the multispectral sensor section, a plurality of (M: M>N)) narrowband signals that are narrower in bandwidth than the plurality of wavelength signals (Yi, abstract: “Reconstructed hyperspectral images (HSIs) from an input multispectral image represent the same scene in higher spatial resolution, with more spectral bands of narrower wavelength width than the input multispectral image.”; coefficient matrices used to generate the HSI spectral bands are described in section 3.3, pg. 8-9). Multispectral image sensors capture information about the environment that is not captured by RGB camera sensors (i.e., infrared spectra). Furthermore, generating an increased number of narrowband signals from the multispectral data further increases the precision of data captured.
Accordingly, Rinehart discloses an imaging device and signal processing method, but does not disclose a multispectral sensor and subsequently generated narrowband signals. Yi discloses a multispectral sensor and subsequently generated narrowband signals. Thus, Rinehart and Yi each disclose an imaging device used for signal processing. A person of ordinary skill in the art, before the effective filing date of the claimed invention, would have recognized that the image sensor and output image signals taught by Rinehart could have been substituted for the sensor and generated image signals taught by Yi because both serve the purpose of capturing pixel signal data. Furthermore, a person of ordinary skill in the art would have been able to carry out the substitution. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to substitute the image sensor and output image signals of Rinehart for the multispectral sensor and subsequently generated narrowband signals, taught by Yi, according to known methods to yield the predictable result of increasing the spectral resolution and coverage of the image signals used to inform the exposure control. Doing so allows for increased control over which bands contribute to image over-/under-exposure in the weighted average calculation.
Claims 3 and 7 are rejected under 35 U.S.C. 103 as being unpatentable over Rinehart in view of Yi, in further view of Sohaib et al. (Sohaib, A., & Robles-Kelly, A. (2015). Exposure time calculation for spectral cameras. Journal of Electronic Imaging, 24(5), 053025-053025.), hereinafter Sohaib.
Regarding claim 3 (dependent on claim 1), Rinehart in view of Yi fails to explicitly teach wherein the controller performs the weighted average processing on a basis of spectral sensitivity characteristics of the multispectral sensor section (emphasis added).
However, Sohaib teaches a system for performing exposure control of a multispectral sensor (Sohaib, abstract: “This paper presents a method for automatic exposure time adjustment for multispectral and hyperspectral cameras”) wherein a controller performs processing to control the exposure on a basis of spectral sensitivity characteristics of the multispectral sensor (Sohaib, abstract: “Here, we use the photopic response function due to its widespread usage in photography and psychophysics. Note that, however, the method presented here is quite general in nature and can employ a number of spectral sensitivity functions for the computation of the spectral power image. Making use of this spectral power image, the exposure time is then computed”; see section 3 on pg. 3-4).
It would have been obvious to a person having ordinary skill in the art, before the effective filing date of the claimed invention, to have combined the exposure control using spectral sensitivity, taught by Sohaib, with the imaging device of Rinehart in view of Yi in order to improve the resulting output image by automatically setting the sensor exposure based on characteristics of each spectral band at each pixel (Sohaib, pg. 3, section 3.1: “we employ a spectral power image, which is computed making use of the CIE photopic function so as to weight the contribution of each spectral band to the power at each pixel”; pg. 9, 3rd para: “Also, note that the images delivered at the output depict a much improved contrast as compared to those taken with the initial exposure times”).
Regarding claim 7 (dependent on claim 5), Rinehart in view of Yi fails to explicitly teach wherein the weighting coefficient determination section determines the plurality of weighting coefficients on a basis of spectral sensitivity characteristics of the multispectral sensor section (emphasis added).
However, Sohaib teaches a system for performing exposure control of a multispectral sensor (Sohaib, abstract: “This paper presents a method for automatic exposure time adjustment for multispectral and hyperspectral cameras”) determines control of the exposure on a basis of spectral sensitivity characteristics of the multispectral sensor (Sohaib, abstract: “Here, we use the photopic response function due to its widespread usage in photography and psychophysics. Note that, however, the method presented here is quite general in nature and can employ a number of spectral sensitivity functions for the computation of the spectral power image. Making use of this spectral power image, the exposure time is then computed”; see section 3 on pg. 3-4).
It would have been obvious to a person having ordinary skill in the art, before the effective filing date of the claimed invention, to have combined the exposure control using spectral sensitivity, taught by Sohaib, with the imaging device of Rinehart in view of Yi in order to improve the resulting output image by automatically setting the sensor exposure based on characteristics of each spectral band at each pixel (Sohaib, pg. 3, section 3.1: “we employ a spectral power image, which is computed making use of the CIE photopic function so as to weight the contribution of each spectral band to the power at each pixel”; pg. 9, 3rd para: “Also, note that the images delivered at the output depict a much improved contrast as compared to those taken with the initial exposure times”).
Allowable Subject Matter
Claims 2, 4, 6, and 8 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims. The following is a statement of reasons for the indication of allowable subject matter:
Regarding claims 2, 4, 6, and 8, the prior art fails to teach, alone or in reasonable combination, wherein the controller performs exposure control by calculating the evaluation value by performing weighted average processing on a basis of the coefficient matrix used to generate the narrowband signals in claim 1.
Therefore, the prior art fails to teach as a whole wherein “the controller performs the weighted average processing on a basis of each element in the coefficient matrix” and “wherein the weighting coefficient determination section determines the plurality of weighting coefficients on a basis elements included in the coefficient matrix.” In view of the foregoing, the prior art references alone or in reasonable combination are insufficient to teach the invention as a whole, as claimed in claims 2, 4, 6, and 8.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure:
U.S. Patent No. 2007/0263097 A1
U.S. Patent No. 2021/0010862 A1
Any inquiry concerning this communication or earlier communications from the examiner should be directed to EMMA E DRYDEN whose telephone number is (571)272-1179. The examiner can normally be reached M-F 9-5 EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, ANDREW BEE can be reached at (571) 270-5183. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/EMMA E DRYDEN/Examiner, Art Unit 2677
/ANDREW W BEE/Supervisory Patent Examiner, Art Unit 2677
Prosecution Insights