DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Specification
The abstract of the disclosure is objected to because it contains an apparent typographical error. The Abstract recites “generating a mode from the fitted mathematical function for each of the one or more wavelengths,” where “generating a model” is clearly intended, as evidenced by the consistent use of “model” throughout the specification, see paragraphs [0008–0010] and in all the pending claims.
A corrected abstract of the disclosure is required and must be presented on a separate sheet, apart from any other text. See MPEP § 608.01(b).
Claim Objections
Claims 1, 6, 7, 9, 10, and 11 are objected to because of the following minor informalities: These claims recite “a calibrated spectra.” The word “spectra” is the standard plural form of “spectrum.” The use of the singular indefinite article “a” with the plural noun “spectra” is grammatically inconsistent and introduces ambiguity as to whether the output is a single calibrated spectral dataset or multiple calibrated spectral datasets. Where a single calibrated spectral dataset is intended, Applicant should amend the claims to recite “a calibrated spectrum.”
Appropriate correction is required.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claims 1, 2, 5, 6, 7, 12, 13, 16, 17 and 18 are rejected under 35 U.S.C. § 102(a)(1) as being anticipated by Ma et al., “Modeling of Diurnal Changing Patterns in Airborne Crop Remote Sensing Images,” Remote Sens. 2021, 13(9), 1719 (hereinafter referred to as Ma).
Regarding independent Claim 1, Ma teaches a method of calibrating hyperspectral images from a field of one or more plants to account for diurnal changes at different times (Ma, Abstract, last line: “It can also be used to calibrate/compensate the remote sensing result against the time effect.”; Section 3.3.3, second full paragraph: “the diurnal changing model can also be used to easily calibrate the diurnal variances”), the method comprising:
receiving a plurality of hyperspectral images over a plurality of days from a field having planted thereon one or more plants (Ma, Sections 2.1–2.2., Ma discloses collecting “a total of 8631 hyperspectral images” using a “VNIR push-broom hyperspectral camera…scanning a field of “two genotypes of corn plants, including genotype B73 × Mo17 and P1105AM” over “31 days,” with imaging from “8:00 a.m.” to “7:30 p.m.” at a frequency of “every 2.5 min.”);
deriving a plurality of spectra from the received plurality of hyperspectral images (Ma, Section 2.3., Ma discloses: “The raw hyperspectral images were firstly calibrated with the real-time white reference” using Equation (2), then “The average reflectance spectrum from each plot was calculated. In total, 51,786 spectra (8631 images × 6 plots/image) were calculated.”);
decomposing the derived plurality of spectra into a trend component representing a trend associated with the plurality of days (Ma, Section 2.6, Equation (5), Ma discloses: “we decomposed the changing signal of each feature into two major parts: day-to-day trend (Tt) and diurnal pattern (Dt). Tt is calculated with LOESS (locally estimated scatterplot smoothing) method…the day-to-day changing trend can be clearly extracted from the raw signal. This trend is majorly reflecting the changes of plant growth stage and general weather conditions over the 31 days of imaging.”);
subtracting the trend component from the derived plurality of spectra to thereby generate diurnal spectra for one or more wavelengths (Ma, Section 2.6, Ma discloses: “The diurnal component (Dt) was calculated by subtracting the day-to-day trend (Tt) from the raw signal. Dt is also called the detrended data.”; Ma, Section 3.4, Equations (10)–(11)), Ma further discloses that the decomposition and modeling were performed for individual spectral bands including Red (670 nm) and NIR (760 nm)), This satisfies the “for one or more wavelengths” limitation.);
fitting one or more mathematical functions associating spectrum to time of day to the generated diurnal spectra for each of the one or more wavelengths (Ma, Sections 2.7, 3.3.2, 3.4, Ma discloses fitting piecewise polynomial regression models to the diurnal patterns: a piecewise 1st-order (linear) model for NDVI (Ma, Equations (6), (8)) and piecewise 2nd-order polynomial models for individual spectral bands Red and NIR (Ma, Equations (7), (10), (11)). Each model is a function of time offset from solar noon (i.e., time of day).);
generating a model from the fitted mathematical functions for each of the one or more wavelengths (Ma, Sections 3.3.2, 3.4., Ma discloses generating calibration models as Equations (8)–(11), including per-band models for Red (Equation (10)) and NIR (Equation (11)), with model performance reported in Table 8.) and
applying the model to the derived spectra at a first time to generate a calibrated spectra at a second time (Ma, Section 3.3.3, Ma discloses: “the NDVI measured 2 h before the solar noon time should be decreased by 0.024 (0.012 × 2), so the calibrated NDVI is as if it is taken exactly at the solar noon time. It’ll be important to run this calibration process, which will help to remove the significant diurnal variance and enable phenotyping researchers to do ‘apple-to-apple’ comparison between the NDVIs of the different plots.”; Ma, Section 5, the conclusion states: “The reported diurnal models can also be used to calibrate the remote sensing result so to remove the diurnal impacts.” It teaches applying the model to measurements taken at a first time (e.g., 2 hours before solar noon) to generate a calibrated result representative of a second time (e.g., solar noon).
Regarding Claim 2, Ma teaches the method of claim 1 and teaches wherein the one or more mathematical functions include a polynomial (Ma, Equations (6)–(11), Sections 2.7, 3.3.2, 3.4 ,specifically, piecewise 1st-order (linear) and 2nd-order polynomial regression functions).
Regarding Claim 5, Ma teaches the method of claim 1 and teaches wherein the one or more plants includes one or more of corn, wheat or soybean (Ma, Section 2.2, teaches corn plants: specifically, genotypes B73 × Mo17 and P1105AM).
Regarding Claim 6, Ma teaches the method of claim 1 and teaches wherein the plurality of spectra are derived based on pre-processing the received plurality of hyperspectral images by applying a reference calibration to account for variations in illumination conditions (Ma, Section 2.3, Equation (2), Ma discloses a reference calibration preprocessing step wherein spectral reflectance data is derived by dividing a difference between a raw value and a dark reference by a difference between a white reference value and the dark reference. Section 2.1, Ma further discloses “[a] white reference panel was installed 0.5 m underneath the hyperspectral camera”) and atmospheric effects affecting spectral signatures of the one or more plant (Ma, Section 2.1, Ma discloses…weatherproof VNIR push-broom hyperspectral camera (MSV-101-W, Middleton Spectral Vision, Middleton, WI, USA, Table 1) was carried by the 7 m high gantry platform to scan a 50 m by 5m strip field under a wide range of weather conditions.).
Regarding Claim 7, Ma teaches the method of claim 6 and teaches wherein the pre-processing further includes segmenting a region of interest (ROI) based on a distinction between background field and foreground plants followed by an averaging function to generate an average spectrum of an entire plant region (Ma, Section 2.3, Figure 1h, Ma discloses that deriving spectra comprises segmenting the received hyperspectral images and averaging spectra: “processed using a segmentation procedure with convolution methodology…the plant tissue was successfully segmented from the background.” Then: “The average reflectance spectrum from each plot was calculated.”).
Claim 12 is rejected for the same reasons as claim 1.
Claim 13 is rejected for the same reasons as claim 2.
Claim 16 is rejected for the same reasons as claim 5.
Claim 17 is rejected for the same reasons as claim 6.
Claim 18 is rejected for the same reasons as claim 7.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 3, 4, 8, 9, 10, 11, 14, 15, 19, 20, 21 and 22 are rejected under 35 U.S.C. 103 as being unpatentable over Ma et al., “Modeling of Diurnal Changing Patterns in Airborne Crop Remote Sensing Images,” Remote Sens. 2021, 13(9), 1719 (hereinafter referred to as Ma) in view of Garcia Torres Luis et al. (WO 2014102416 A1) (hereinafter referred to as Garcia).
Regarding claim 3, Ma teaches the method of claim 1 as set forth above but does not explicitly teach wherein the generated model is a transformational matrix representing the one or more mathematical functions.
Ma discloses generating piecewise polynomial regression models for individual features and spectral bands (Ma, Sections 3.3.2 – 3.4, Equations (8)–(11), Table 8) but does not explicitly characterize the collection of per-wavelength model coefficients as a “transformational matrix.”
Reference Garcia teaches an automatic method for radiometric normalization of multitemporal remote sensing images of agricultural scenes, explicitly including “multispectral and hyperspectral” images taken “at different times of a day.” (García, Detailed Description Section, page 5, paragraphs 5 - 8; Claim 1(a), García discloses computing per-band correction factors (CFs) for each image in a series of multitemporal images, where each CF is defined as:
CFxi=xRxi
where xR is the reference value (average band value across the original image series) and xi is the value of band x in image i.
García organizes these per-band, per-image correction factors in a structured table (García, pages 7-9, Table 1 — “ARIN Results”), constituting a matrix of transformation parameters indexed by spectral band (columns: Blue, Green, Red, NIR) and image (rows: V1–V6). García applies this correction factor matrix to transform each image’s spectral bands through “linear transformation of each band of each image by applying the previously calculated CF.” (García, Claim 1(g)).
These arts are analogous since they are both related to imaging devices. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention (AIA ) to organize Ma’s per-wavelength polynomial regression coefficients into a matrix data structure such as a transformational matrix as suggested by García’s established practice of organizing per-band correction factors in a structured table/matrix for systematic application to multitemporal agricultural image normalization.
The motivation to combine is as follows: Both Ma and García address the same general problem of radiometric normalization of multitemporal remote sensing images of agricultural scenes taken at different times and both compute per-band transformation parameters that are applied multiplicatively to spectral data. García demonstrates that organizing such per-band normalization parameters in a matrix format is the standard, practical approach for efficient automated processing of multitemporal image series as seen in García Table 1, ARIN software. One of ordinary skill in the art would have seen the benefit of extending Ma’s per-band diurnal models to additional wavelengths as expressly suggested by Ma’s teaching that the method generalizes to other features (Ma, Section 4.2) would have recognized that a matrix is the natural data structure for storing and applying per-wavelength model coefficients, as taught by García’s analogous practice.
One of ordinary skill in the art would have had a reasonable expectation of success because organizing known numerical coefficients (polynomial regression coefficients) into a two-dimensional matrix data structure (wavelengths × coefficients) is a routine computational task requiring no experimentation. García confirms that matrix-organized per-band transformation parameters are standard in the multitemporal agricultural image normalization art. See KSR Int’l Co. v. Teleflex Inc., 550 U.S. 398, 417 (2007) (applying a known technique to a known method to yield predictable results is obvious).
Claim 14 is rejected for the same reasons as claim 3.
Regarding claim 4, Ma teaches the method of claim 1 as set forth above but does not explicitly teach wherein the generated model is a 3-dimensional graph representing the one or more mathematical functions.
Ma discloses generating per-wavelength, per-time-of-day diurnal pattern models and presents results as 2D plots (Ma, Figures 8, 10) but does not explicitly present the model as a 3-dimensional graph.
Reference Garcia teaches computing per-band correction parameters across a series of multitemporal images and presenting the results in a structured tabular format (García, Tables 1–4), Garcia discloses where spectral band values and correction factors are displayed across multiple images (representing different acquisition times). (García, Tables 2–3.), García further discloses computing vegetation indices (NDVI, B/G) from the spectral bands for each image, demonstrating multi-dimensional visualization of spectral and temporal parameters.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention (AIA ) to visualize Ma’s per-wavelength, per-time-of-day diurnal pattern models as a three-dimensional surface plot (wavelength × time of day × reflectance), as suggested by García’s practice of presenting multi-dimensional spectral-temporal normalization data. A three-dimensional graph is a routine data visualization technique well within the ordinary skill of a one of ordinary skill in the art in the remote sensing and data analysis arts. One of ordinary skill in the art seeking to comprehensively display the relationship between wavelength, time of day, and spectral reflectance, the three variables in Ma’s per-band diurnal models, would have found it obvious to generate a 3D surface plot, which is a standard tool in scientific computing environments such as MATLAB, Python (matplotlib), and R.
One of ordinary skill in the art would have had a reasonable expectation of success because generating 3D surface plots from multi-variable regression models is a standard capability of scientific visualization software.
Claim 15 is rejected for the same reasons as claim 4.
Regarding claim 8, Ma teaches the method of claim 7 as set forth above but does not explicitly teach wherein the segmentation includes obtaining Normalized Difference Vegetation Index (NDVI) to establish a heatmap and a threshold for segmentation.
Ma discloses image segmentation using a red-edge convolution method (Ma, Section 2.3) but does not use NDVI thresholding specifically for segmentation.
Reference Garcia teaches the use of NDVI for identifying and characterizing vegetative land uses in remote sensing images. Specifically, García discloses selecting plots that “exhibit a high reflectance or index ‘of greenery’, for example of NDVI [(NIR-R)/(NIR+R)], in each of the multitemporal images.” (García, Detailed Description Section, page 5, paragraph 3 – page 6, paragraph 1, step (c); Claim 1(c)) García further discloses that “[a] high photosynthetic activity, that is to say healthy and vigorous vegetation, implies a high NDVI value.” (García, Sector and object of the invention section, page 1, paragraph 3 make it clear that this is common in the State of the Art.) García uses NDVI values to distinguish vegetation from non-vegetation in the processed images, with positive NDVI values representing vegetation coverage and near-zero values representing rock or bare soil. (García, Sector and object of the invention section page 1, paragraph 3 make it clear that this is common in State of the Art.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention (AIA ) to substitute NDVI-based thresholding for Ma’s red-edge convolution segmentation method, as suggested by García’s teaching that NDVI is effective for distinguishing vegetation from non-vegetation in remote sensing images of agricultural scenes. Both Ma and García operate in the same field of agricultural remote sensing, and both compute NDVI from their spectral data. NDVI-based vegetation segmentation is one of the most widely established techniques in the remote sensing art. Applicant’s own specification acknowledges NDVI as “a classic spectral index” see paragraph [0026]. Substituting one known vegetation segmentation method (NDVI thresholding, as taught by García) for another (red-edge convolution, as taught by Ma) to achieve the same predictable result of separating plant tissue from background in a remote sensing image would have been an obvious design choice. See KSR Int’l Co. v. Teleflex Inc., 550 U.S. 398, 416 (2007).
Claim 19 is rejected for the same reasons as claim 8.
Regarding claim 9, Ma teaches the method of claim 7 as set forth above but does not explicitly teach wherein the pre-processing further includes discrete wavelet transformation, Savitzky-Golay smoothing, and moving average smoothing transforming resolution of spectral features into multilevel components, representing both high- and low-frequency information, and provide an averaging thereof. Ma does not disclose applying these specific smoothing techniques.
Reference Garcia teaches preprocessing multispectral and hyperspectral remote images for radiometric normalization, including the use of specialized image processing software (ENVI) with various signal processing capabilities. (García, Detailed Description, page 6, paragraphs 1- page 9, State of the Art; ARIN software.) García processes spectral band data to extract statistical parameters and compute correction factors. (García, Claim 1(e)–(f)).
Discrete wavelet transformation, Savitzky-Golay filtering, and moving average smoothing are well-known, standard spectral preprocessing techniques that have been routinely applied to hyperspectral vegetation reflectance data for noise reduction and signal quality improvement for decades. These techniques are described in standard remote sensing and signal processing textbooks and are standard tools in spectral analysis software (including ENVI, as referenced by García, and Python libraries, as referenced by Ma’s Table 2 (Python polyfit implementation)).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention (AIA ) to apply one or more of these conventional noise reduction techniques to Ma’s hyperspectral canopy spectra prior to diurnal pattern modeling, in order to improve spectral data quality and reduce noise. The selection of any particular smoothing method from among well-known alternatives represents routine skill, not patentable innovation.
Claim 20 is rejected for the same reasons as claim 9.
Regarding claim 10, the combination of Ma in view of Garcia teaches the method of claim 9 as set forth above but does not explicitly teach wherein the pre-processing further includes partial spectral removal to thereby remove several wavelengths from beginning and end of the spectra having a plurality of wavelengths to mitigate impact of noise and spectra artifacts brought about by the instrumentation to give rise to the one or more wavelengths.
Ma does not explicitly describe removing edge wavelengths as a preprocessing step.
Reference Garcia teaches processing multispectral images with specific spectral bands: Blue (450–510 nm), Green (510–580 nm), Red (655–690 nm), and NIR (780–920 nm). (García, Description, page 7, Examples of Embodiment.) García processes only the defined spectral bands relevant to the analysis, excluding spectral regions outside the bands of interest.
It is well known in the hyperspectral imaging art that spectral channels near the extreme edges of a sensor’s detection range exhibit higher noise levels and lower signal-to-noise ratios due to reduced sensor sensitivity at the boundaries of its operational range. One of ordinary skill in the art processing data from the MSV-101-W sensor (376–1044 nm), as taught by Ma, would have found it obvious to remove noisy edge wavelengths as a routine quality improvement measure to focus analysis on the reliable portion of the spectral range and discard the noisy beginning and end portions. This practice is standard in the hyperspectral data processing art.
Claim 21 is rejected for the same reasons as claim 10.
Regarding Claim 11, the combination of Ma in view of Garcia teaches the method of claim 10 and teaches wherein the processing further includes a spectra quality control based on an interquartile range from a single wavelength to generate the plurality of spectra at variant times by removing datapoints outside of the interquartile range (Ma, Section 2.4, Ma discloses IQR-based quality control: “the measurements between the upper inner fence (Q3 + 1.5IQR) and lower inner fence (Q1 − 1.5IQR) were kept. IQR is the interquartile range, being equal to the difference between 75th (Q3) and 25th (Q1) percentiles. This quality filtering removed the blunders/gross errors in the image acquisition setup/process.”).
Claim 22 is rejected for the same reasons as claim 11.
Related Art
The following cited reference defines the general state of the art but are not considered to teach the Applicant's claimed invention.
Cong (CN 116597157) demonstrates that spectral analysis of plants across different time periods is known. The reference is relevant to the general field of plant spectral analysis but does not address the specific problem (diurnal variation) or solution (time-series decomposition + polynomial fitting + ratio-based temporal calibration) claimed by Jin.
Gou (CN 115829854 A) addresses the same general problem domain — correcting plant spectral imagery for changing illumination. The technical approach is fundamentally different.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SUPERVISORY PATENT EXAMINER TWYLER HASKINS whose telephone number is (571)272-7406. The SUPERVISORY PATENT EXAMINER TWYLER HASKINS can normally be reached Mon- Thursday: 7:30 am-4:30 pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http:/Awwww.uspto.gov/interviewpractice.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https:/Awww.uspto.gov/patents/apply/patent-center for more information about Patent Center and https:/Awww.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/TWYLER L HASKINS/ Supervisory Patent Examiner, Art Unit 2639
Prosecution Insights