DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement(s) (IDS) received on November 25th 2024, August 26th 2024, February 18th 2024, and July 10th 2026, is/are in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement(s) is/are being considered by the examiner.
Drawings
The drawings are objected to because :
In figure 2, label 204 is used to label two distinct elements.
Corrected drawing sheets in compliance with 37 CFR 1.121(d) are required in reply to the Office action to avoid abandonment of the application. Any amended replacement drawing sheet should include all of the figures appearing on the immediate prior version of the sheet, even if only one figure is being amended. The figure or figure number of an amended drawing should not be labeled as “amended.” If a drawing figure is to be canceled, the appropriate figure must be removed from the replacement sheet, and where necessary, the remaining figures must be renumbered and appropriate changes made to the brief description of the several views of the drawings for consistency. Additional replacement sheets may be necessary to show the renumbering of the remaining figures. Each drawing sheet submitted after the filing date of an application must be labeled in the top margin as either “Replacement Sheet” or “New Sheet” pursuant to 37 CFR 1.121(d). If the changes are not accepted by the examiner, the applicant will be notified and informed of any required corrective action in the next Office action. The objection to the drawings will not be held in abeyance.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claim(s) 1, 7, 9, 13, 20-21, 24, 32, and 34 is/are rejected under 35 U.S.C. 102(a)(2) as being unpatentable over Beckett et al. (US 20180172824 A1) , hereinafter Beckett.
Regarding claim 1,
A synthetic aperture radar (SAR) antenna system comprising:
a phased array antenna that transmits radar signals toward a target area that reflects the radar signals (Beckett [0031] “In one mode of operation, system 100 acquires SAR and optical images of one or more targets of interest with essentially the same viewing angle and at essentially the same time”, further, Beckett [0082] “The various embodiments described above can be combined to provide further embodiments. U.S. Provisional Patent Application Ser. No. 62/180,421, filed Jun. 16, 2015 and entitled “EFFICIENT PLANAR PHASED ARRAY ANTENNA ASSEMBLY” … are each incorporated herein by reference, in their entirety.” Examiner notes that phased array antennas are implicitly incorporated into the invention of Beckett by reference.);
a signal processing unit that performs one or more SAR algorithms to convert the reflected radar signals into one or more of images, video, and measurements of the target area (Beckett [0029] “The SAR sensor acquires single-band or multi-band, and single-polarization or multi-polarization SAR data (for example, dual-polarization or quad-polarization).”, further, Beckett [0034] ” Data processing subsystem 125 comprises one or more SAR image processors 210, one or more optical image processors 220, a post-processing raw training data gathering set of instructions also interchangeably referred to as a post-processing raw training data gathering module 230”); and
at least one data integrator configured executable by a processor to:
receive one or more of ground-based visual data, aerial data, and/or historical or architectural data (Beckett [0035] “Optical image processor 220 generates optical imagery from raw or processed optical data received from optical satellites 144-1, 144-2, through 144-N of FIG. 1.” Here examiner notes that the limitation “aerial data” would include satellite data in a broadest reasonable interpretation.);
integrate one or more of ground-based visual data, aerial data, and/or the historical data, or architectural data with the one or more images, the video, and the measurements produced by the SAR antenna system to produce a cohesive an integrated visualization (Beckett [0041] “When an optical image is available, it can be used to update the application-specific SAR enhancer parameters, and then the process described above can be applied to generate an enhanced image product.”); and
send the integrated visualization over a communication network to a remote system or device for storage, display and/or interaction therewith (Beckett [0034] “Data processing subsystem 125 can be installed on one or more servers, and the elements of data processing 125 can be communicatively coupled directly or via a network.”)
Regarding claim 7,
The system of claim 3, wherein the at least one data integrator spatially aligns the at least one of the ground-based visual data, the aerial data, the historical data, and the architectural data with the at least one the images, the video, and the measurements using spatial coordinates associated therewith (Beckett [0048] “In one example implementation, data processing subsystem 125 may identify SAR and optical data as being coincident if the SAR and optical data cover a coincident geographic area, for example if the imagery data is geographically coextensive or at least partially geographically overlapping in areas represented by the SAR and optical data.”).
Regarding claim 9,
The system of claim 1, wherein the at least one data integrator is further executable to integrate at least one of the ground-based visual data, the aerial data, the historical data, the architectural data with at least one of the images, the video, and the measurements using a data fusion process (Bennett [0006] “SAR data and optical data can contain complementary information. Various data fusion methods have been proposed for combining SAR data and optical data to generate enhanced image products.”, further, Bennett [0009] “An image processing system may generate an enhanced synthetic aperture radar (SAR) image by generating an information model determined from SAR and optical image data, and applying the information model to an input SAR image.”).
Regarding claim 13,
The system of claim 1, wherein the at least one data integrator includes a drone data integrator that receives the aerial data from one or more drones (Beckett [0081] “[0081] While the foregoing description refers, for the most part, to satellite platforms for SAR and optical sensors, remotely sensed imagery can be acquired using airborne sensors including, but not limited to, aircraft and drones.”).
Regarding claim 20,
The system of claim 1, wherein the integrated visualization includes one or more of radar images, 3D models, layered maps, and analytical results (Beckett [0077] “In one example implementation of the processing of an enhanced SAR image described above, the enhanced SAR image uses an RGB application. The result can be displayed or printed as a SAR image with RGB colors reflecting the SAR classifications, and the correlations between the SAR classifications and in the trained data retrieved from a data set, such as data set (e.g., database) 450a of FIG. 4A.”).
Regarding claim 21,
The system of claim 1, wherein the remote system or device includes one or more of a user device, a centralized server, a distributed server, or a decentralized server (Beckett [0034] “Data processing subsystem 125 can be installed on one or more servers, and the elements of data processing 125 can be communicatively coupled directly or via a network.”).
Regarding claim 24,
A method of synthetic aperture radar (SAR) antenna operation, the method comprising:
transmitting radar signals toward a target area via a phased array antenna of a synthetic aperture radar (SAR) antenna system (Beckett [0031] “In one mode of operation, system 100 acquires SAR and optical images of one or more targets of interest with essentially the same viewing angle and at essentially the same time”, further, Beckett [0082] “The various embodiments described above can be combined to provide further embodiments. U.S. Provisional Patent Application Ser. No. 62/180,421, filed Jun. 16, 2015 and entitled “EFFICIENT PLANAR PHASED ARRAY ANTENNA ASSEMBLY” … are each incorporated herein by reference, in their entirety.” Examiner notes that phased array antennas are implicitly incorporated into the invention of Beckett by reference.);
receiving reflected radar signals from the target area (Beckett [0035] “SAR image processor 210 generates SAR imagery from raw or processed SAR data received from SAR satellites 142-1, 142-2, through 142-N of FIG. 1 via interface 130.”);
performing, via a signal processor, one or more SAR algorithms to convert the reflected radar signals into one or more of images, video, and measurements of the target area (Beckett [0029] “The SAR sensor acquires single-band or multi-band, and single-polarization or multi-polarization SAR data (for example, dual-polarization or quad-polarization).”, further, Beckett [0034] ” Data processing subsystem 125 comprises one or more SAR image processors 210, one or more optical image processors 220, a post-processing raw training data gathering set of instructions also interchangeably referred to as a post-processing raw training data gathering module 230”);
integrating one or more of ground-based visual data, aerial data, historical data, and architectural data with one or more of the images, video, and measurements to produce an integrated visualization; and (Beckett [0041] “When an optical image is available, it can be used to update the application-specific SAR enhancer parameters, and then the process described above can be applied to generate an enhanced image product.”)
sending the integrated visualization to a remote system over a communication network (Beckett [0034] “Data processing subsystem 125 can be installed on one or more servers, and the elements of data processing 125 can be communicatively coupled directly or via a network.”).
Regarding claim 32,
The method of claim 24, wherein integrating one or more of the ground-based visual data, the aerial data, the historical data, and the architectural data with one or more of the images, the video, and the measurements includes using a data fusion process. (Bennett [0006] “SAR data and optical data can contain complementary information. Various data fusion methods have been proposed for combining SAR data and optical data to generate enhanced image products.”, further, Bennett [0009] “An image processing system may generate an enhanced synthetic aperture radar (SAR) image by generating an information model determined from SAR and optical image data, and applying the information model to an input SAR image.”).
Regarding claim 34,
A non-transitory, computer-readable storage medium, having embodied thereon a program executable by a processor to perform a method of synthetic aperture radar (SAR) antenna operation, the method comprising (Beckett [0014] “The SAR image processing system may further include a set of display instructions which, when executed by the at least one processor, cause the at least one processor to manifest the enhanced SAR image product”):
transmitting radar signals toward a target area via a phased array antenna of a synthetic aperture radar (SAR) antenna system (Beckett [0031] “In one mode of operation, system 100 acquires SAR and optical images of one or more targets of interest with essentially the same viewing angle and at essentially the same time”, further, Beckett [0082] “The various embodiments described above can be combined to provide further embodiments. U.S. Provisional Patent Application Ser. No. 62/180,421, filed Jun. 16, 2015 and entitled “EFFICIENT PLANAR PHASED ARRAY ANTENNA ASSEMBLY” … are each incorporated herein by reference, in their entirety.” Examiner notes that phased array antennas are implicitly incorporated into the invention of Beckett by reference.);
receiving reflected radar signals from the target area (Beckett [0035] “SAR image processor 210 generates SAR imagery from raw or processed SAR data received from SAR satellites 142-1, 142-2, through 142-N of FIG. 1 via interface 130.”);
performing, via a signal processor, one or more SAR algorithms to convert the reflected radar signals into one or more of images, video, and measurements of the target area; integrating one or more of ground-based visual data, aerial data, historical data, and architectural data with one or more of the images, video, and measurements to produce an integrated visualization; and (Beckett [0029] “The SAR sensor acquires single-band or multi-band, and single-polarization or multi-polarization SAR data (for example, dual-polarization or quad-polarization).”, further, Beckett [0034] ” Data processing subsystem 125 comprises one or more SAR image processors 210, one or more optical image processors 220, a post-processing raw training data gathering set of instructions also interchangeably referred to as a post-processing raw training data gathering module 230”);
sending the integrated visualization to a remote system over a communication network (Beckett [0034] “Data processing subsystem 125 can be installed on one or more servers, and the elements of data processing 125 can be communicatively coupled directly or via a network.”).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 2, 17, 25, and 33 is/are rejected under 35 U.S.C. 103 as being unpatentable over Beckett et al. (US 20180172824 A1), hereinafter Beckett, in view of Huang et al. (Huang, Bo, et al. "Cloud removal from optical satellite imagery with SAR imagery using sparse representation." IEEE Geoscience and Remote Sensing Letters 12.5 (2015): 1046-1050. ), hereinafter Huang.
Regarding claim 2, Beckett teaches the system of claim 1. Beckett fails to disclose the limitations below. Huang teaches:
The system of claim 1, further comprising: a generative artificial intelligence (AI) model trained to fill at least one data gap associated with one or more of the images, the video, the measurements, the ground-based visual data, the aerial data, the historical data, or the architectural data (Huang Pg. 1047 “In this letter, we use a set of prior-sample optical and SAR images to train two offline overcomplete dictionary pairs, with one pair generated from HRI and LRI patches and the other generated from HRI and SAR image patches, and then to reconstruct the missing data via sparse coding using the valid data in the region to be repaired. ” Here it is understood that the training algorithm described in Huang is a generative AI model.).
It would have been obvious to one of ordinary skill in the art prior to the effective filing date of the claimed invention to incorporate the features as disclosed by Huang into the invention of Beckett. Both Beckett and Huang are considered analogous arts to the claimed invention as they both disclose SAR radar methods for combining optical and SAR radar datasets to produce a fused and enhanced image. It would have been obvious to one of ordinary skill in the art prior to the effective filing date of the claimed invention to modify the system as disclosed by Beckett to utilize generative AI methods to fill in gaps in optical data as taught by Huang. One of ordinary skill in the art prior to the effective filing date of the claimed invention would have been motivated to implement an AI-based interpolation method in order to fill in spaces in the measurement that were unavailable due to optical obstructions using the SAR data (See Huang Pg. 1047 column 1, “In view of these methods and their limitations, we are inspired to adopt SAR imagery as another auxiliary image for reconstructing missing data in an HR optical satellite image (HRI).”).
Regarding claim 17, Beckett teaches the system of claim 1. Beckett fails to clearly teach the limitation below. Huang teaches:
The system of claim 1, wherein the signal processing unit is further configured to enhance the one or more images using at least one of noise reduction, contrast enhancement, edge sharpening, artifact or distortion removal, Kalman filtering, and Independent Component Analysis (ICA) (Huang Pg. 1048 “Before the coregistration of two images, SAR data must be terrain corrected and speckle reduced. Gamma Map(3×3) filter and multi look were applied to reduce speckle noise in this experiment.”)
It would have been obvious to one of ordinary skill in the art prior to the effective filing date of the claimed invention to incorporate the features as disclosed by Huang into the invention of Beckett. Both Beckett and Huang are considered analogous arts to the claimed invention as they both disclose SAR radar methods for combining optical and SAR radar datasets to produce a fused and enhanced image. It would have been obvious to one of ordinary skill in the art prior to the effective filing date of the claimed invention to modify the system as disclosed by Beckett to use a noise reduction correction as taught by Huang. Noise correction methods are well-known within the art for “speckle reduction”, which as a fundamental limitation of radar-based methods as opposed to optical methods. One of ordinary skill in the art prior to the effective filing date of the claimed invention would have been motivated to utilize an image enhancement method such as the noise correction method of Huang in order to improve the image quality of measured SAR images before training a dataset, in order to improve the quality of the training data (See Huang Pg. 1048).
Regarding claim 25, Beckett teaches the method of claim 24. Beckett fails to clearly disclose the below limitations. Huang teaches:
The method of claim 24, further comprising using a generative AI model trained to fill at least one data gap associated with one or more of the images, the video, the measurements, the ground-based visual data, the aerial data, the historical data, and the architectural data (Huang Pg. 1047 “In this letter, we use a set of prior-sample optical and SAR images to train two offline overcomplete dictionary pairs, with one pair generated from HRI and LRI patches and the other generated from HRI and SAR image patches, and then to reconstruct the missing data via sparse coding using the valid data in the region to be repaired. ” Here it is understood that the training algorithm described in Huang is a generative AI model.).
It would have been obvious to one of ordinary skill in the art prior to the effective filing date of the claimed invention to incorporate the features as disclosed by Huang into the invention of Beckett. Both Beckett and Huang are considered analogous arts to the claimed invention as they both disclose SAR radar methods for combining optical and SAR radar datasets to produce a fused and enhanced image. It would have been obvious to one of ordinary skill in the art prior to the effective filing date of the claimed invention to modify the system as disclosed by Beckett to utilize generative AI methods to fill in gaps in optical data as taught by Huang. One of ordinary skill in the art prior to the effective filing date of the claimed invention would have been motivated to implement an AI-based interpolation method in order to fill in spaces in the measurement that were unavailable due to optical obstructions using the SAR data (See Huang Pg. 1047 column 1, “In view of these methods and their limitations, we are inspired to adopt SAR imagery as another auxiliary image for reconstructing missing data in an HR optical satellite image (HRI).”).
Regarding claim 33,
The method of claim 24, further comprising normalizing one or more of the ground-based visual data, the aerial data, the historical data, the architectural data, the images, the video, and the measurements to account for differences in scale, resolution, or perspective (Huang Pg. 1048 “Before the coregistration of two images, SAR data must be terrain corrected and speckle reduced. Gamma Map(3×3) filter and multi look were applied to reduce speckle noise in this experiment. The SAR images were then resampled to 30 m to be compatible with the optical images; both types of images were well coregistered with an error of less than four pixels.” Examiner notes here that “coregistered within an error of less than four pixels” implies that coalignment of images or pixels so that they may be overlayed later.).
It would have been obvious to one of ordinary skill in the art prior to the effective filing date of the claimed invention to incorporate the features as disclosed by Huang into the invention of Beckett. Both Beckett and Huang are considered analogous arts to the claimed invention as they both disclose SAR radar methods for combining optical and SAR radar datasets to produce a fused and enhanced image. It would have been obvious to one of ordinary skill in the art prior to the effective filing date of the claimed invention to modify the system as disclosed by Beckett to normalize the aerial data to account for differences in scale, resolution, or perspective as taught by Huang. One of ordinary skill in the art prior to the effective filing date of the claimed invention would have been motivated to normalize the aerial data to account for differences in scale, resolution, or perspective in order enable operations between the SAR and optical datasets such as overlaying, subtracting, and other mathematical operations required for training datasets. These operations are required to produce a “dictionary pair” as described in Huang (See Huang Pg. 1048).
Claim(s) 3, 4, 10, 26-27, and 30 is/are rejected under 35 U.S.C. 103 as being unpatentable over Beckett et al. (US 20180172824 A1), hereinafter Beckett, in view of in view of Yokoyama et al. (US 20210104154 A1), hereinafter Yokoyama.
Regarding claim 3, Beckett teaches the system of claim 1. Beckett fails to teach the limitations below. Yokoyama teaches:
wherein the at least one data integrator is further executable to temporally or spatially align at least one of the ground-based visual data, the aerial data, the historical data, and the architectural data with at least one of the images, the video, and the measurements (Yokoyama [0050] ”Data synchronizer 42 synchronizes the image data and the millimeter-wave data. For example, data synchronizer 42 adjusts a timing at which the image data is generated and a timing at which the millimeter-wave data is generated, based on time information included in the image data and time information included in the millimeter-wave data.”.)
It would have been obvious to one of ordinary skill in the art prior to the effective filing date of the claimed invention to incorporate the features as disclosed by Yokoyama into the invention of Beckett. Both Beckett and Yokoyama are considered analogous arts to the claimed invention as they both disclose the fusion of visual and radar measurements. It would have been obvious to one of ordinary skill in the art prior to the effective filing date of the claimed invention to modify the system as disclosed by Beckett to temporally or spatially align the optical and radar data as taught by Yokoyama. One of ordinary skill in the art prior to the effective filing date of the claimed invention would have been motivated to align the optical and radar datasets in order to synchronize the measurements in order to provide directly comparable images such that the information extracted from within said images is directly comparable, such as the detection of moving objects like cars, objects, etc. (See Yokoyama [0007-0008], [0050-0054] ).
Regarding claim 4, Beckett in view of Yokoyama teaches the system of claim 3. Beckett fails to teach the limitations below. Yokoyama teaches:
wherein the at least one data integrator temporally aligns the at least one of the ground-based visual data, the aerial data, the historical data, and the architectural data with the at least one of the images, the video, and the measurements based on one or more timestamps (Yokoyama [0050] “Data synchronizer 42 synchronizes the image data and the millimeter-wave data. For example, data synchronizer 42 adjusts a timing at which the image data is generated and a timing at which the millimeter-wave data is generated, based on time information included in the image data and time information included in the millimeter-wave data.”).
It would have been obvious to one of ordinary skill in the art prior to the effective filing date of the claimed invention to incorporate the features as disclosed by Yokoyama into the invention of Beckett. Both Beckett and Yokoyama are considered analogous arts to the claimed invention as they both disclose the fusion of visual and radar measurements. It would have been obvious to one of ordinary skill in the art prior to the effective filing date of the claimed invention to modify the system as disclosed by Beckett to temporally align the optical and radar data as taught by Yokoyama. One of ordinary skill in the art prior to the effective filing date of the claimed invention would have been motivated to align the optical and radar datasets in order to synchronize the measurements in order to provide directly comparable images such that the information extracted from within said images is synchronized, such as moving objects like cars, objects, etc. (See Yokoyama [0007-0008], [0050-0054] ).
Regarding claim 10, Beckett teaches the system of claim 1. Beckett fails to teach the limitations below. Yokoyama teaches:
wherein the at least one data integrator is further executable to normalize one or more of the ground-based visual data, the aerial data, the historical data, the architectural data, the images, the video, and the measurements to account for differences in scale, resolution, or perspective (Yokoyama [0058] “Further, screen generator 44 performs conversion such that millimeter-wave data corresponding to a scanning area corresponds to a monitoring area. For example, screen generator 44 may perform coordinate conversion processing in which millimeter-wave data information defined by the millimeter-wave radar coordinate system is converted into that of the camera coordinate system. Screen generator 44 superimposes millimeter-wave data information after the coordinate conversion on an image indicated by image data.”).
It would have been obvious to one of ordinary skill in the art prior to the effective filing date of the claimed invention to incorporate the features as disclosed by Yokoyama into the invention of Beckett. Both Beckett and Yokoyama are considered analogous arts to the claimed invention as they both disclose the fusion of visual and radar measurements. It would have been obvious to one of ordinary skill in the art prior to the effective filing date of the claimed invention to modify the system as disclosed by Beckett to normalize the optical data to account for differences in scale, resolution, or perspective as taught by Yokoyama. One of ordinary skill in the art prior to the effective filing date of the claimed invention would have been motivated to rescale optical measurement in order to directly overlay the datasets, superimposing them to create a single fused image (See Yokoyama [0058, 0081, 0090] ).
Regarding claim 26, Beckett teaches the method of claim 24. Beckett fails to teach the limitations below. Yokoyama teaches:
The method of claim 24, further comprising temporally or spatially aligning at least one of the ground-based visual data, the aerial data, the historical data, and the architectural data with one or more of the images, the video, and the measurements (Yokoyama [0050] ”Data synchronizer 42 synchronizes the image data and the millimeter-wave data. For example, data synchronizer 42 adjusts a timing at which the image data is generated and a timing at which the millimeter-wave data is generated, based on time information included in the image data and time information included in the millimeter-wave data.”.).
It would have been obvious to one of ordinary skill in the art prior to the effective filing date of the claimed invention to incorporate the features as disclosed by Yokoyama into the invention of Beckett. Both Beckett and Yokoyama are considered analogous arts to the claimed invention as they both disclose the fusion of visual and radar measurements. It would have been obvious to one of ordinary skill in the art prior to the effective filing date of the claimed invention to modify the system as disclosed by Beckett to temporally align the optical and radar data as taught by Yokoyama. One of ordinary skill in the art prior to the effective filing date of the claimed invention would have been motivated to align the optical and radar datasets in order to synchronize the measurements in order to provide directly comparable images such that the information extracted from within said images is synchronized, such as moving objects like cars, objects, etc. (See Yokoyama [0007-0008], [0050-0054] ).
Regarding claim 27, Beckett in view of Yokoyama teaches the method of claim 26. Beckett fails to teach the limitation below. Yokoyama teaches:
wherein temporally aligning the at least one of the ground-based visual data, the aerial data, the historical data, and the architectural data with the at least one of the images, the video, and the measurements based on one or more timestamps (Yokoyama [0050] “Data synchronizer 42 synchronizes the image data and the millimeter-wave data. For example, data synchronizer 42 adjusts a timing at which the image data is generated and a timing at which the millimeter-wave data is generated, based on time information included in the image data and time information included in the millimeter-wave data.”).
It would have been obvious to one of ordinary skill in the art prior to the effective filing date of the claimed invention to incorporate the features as disclosed by Yokoyama into the invention of Beckett. Both Beckett and Yokoyama are considered analogous arts to the claimed invention as they both disclose the fusion of visual and radar measurements. It would have been obvious to one of ordinary skill in the art prior to the effective filing date of the claimed invention to modify the system as disclosed by Beckett to temporally align the optical and radar data as taught by Yokoyama. One of ordinary skill in the art prior to the effective filing date of the claimed invention would have been motivated to align the optical and radar datasets in order to synchronize the measurements in order to provide directly comparable images such that the information extracted from within said images is synchronized, such as moving objects like cars, objects, etc. (See Yokoyama [0007-0008], [0050-0054] ).
Regarding claim 30, Beckett in view of Yokoyama teaches the method of claim 26. Beckett further teaches:
The method of claim 26, wherein spatially aligning the ground-based visual data, the aerial data, and/or the historical or architectural data with one or more of the images, the video, and the measurements using spatial coordinates associated therewith (Beckett [0048] “In one example implementation, data processing subsystem 125 may identify SAR and optical data as being coincident if the SAR and optical data cover a coincident geographic area, for example if the imagery data is geographically coextensive or at least partially geographically overlapping in areas represented by the SAR and optical data.”).
Claim(s) 5 and 28 is/are rejected under 35 U.S.C. 103 as being unpatentable over Beckett et al. (US 20180172824 A1), hereinafter Beckett, in view of Yokoyama et al. (US 20210104154 A1), hereinafter Yokoyama, and further in view of Ikefuji et al. (US 20210223389 A1), hereinafter Ikefuji.
Regarding claim 5, Beckett in view of Yokoyama discloses the system of claim 3. Beckett in view of Yokoyama fails to disclose the limitations below. Ikefuji discloses:
wherein the at least one data integrator temporally aligns the at least one of the ground-based visual data, the aerial data, the historical data, and the architectural data with the at least one of the images, the video, and the measurements using temporal interpolation (Ikefuji [0012] “The interpolation processing unit 920 has a function of computing the displacement rates of all the pixels in each observing direction on the basis of the displacement rate of the stable reflection point obtained by the displacement analysis. The interpolation processing unit 920 computes the displacement rates of all the pixels by interpolating the displacement rates of the stable reflection points, for example.”).
It would have been obvious to one of ordinary skill in the art prior to the effective filing date of the claimed invention to incorporate the features as disclosed by Ikefuji into the invention of Beckett in view of Yokoyama. The set of Beckett, Yokoyama, and Ikefuji are considered analogous arts to the claimed invention as they both disclose the fusion of radar measurements with optical image data. It would have been obvious to one of ordinary skill in the art prior to the effective filing date of the claimed invention to modify the system as disclosed by Beckett in view of Yokoyama to temporally align the optical data with the SAR measurements using an interpolation method as taught by Ikefuji. One of ordinary skill in the art prior to the effective filing date of the claimed invention would have been motivated to utilize temporal interpolation in order to generate a set of optical and radar measurements in a scenario when two precisely synchronous measurements do not exist, which would allow for the coregistration of said images. (See Yokoyama [0007-0008] and Ikefuji [0015-0017]).
Regarding claim 28, Beckett in view of Yokoyama discloses the system of claim 3. Beckett in view of Yokoyama fails to disclose the limitations below. Ikefuji discloses:
The method of claim 27, wherein temporally aligning the at least one of the ground-based visual data, the aerial data, the historical data, and the architectural data with the at least one of the images, the video, and the measurements using temporal interpolation (Ikefuji [0012] “The interpolation processing unit 920 has a function of computing the displacement rates of all the pixels in each observing direction on the basis of the displacement rate of the stable reflection point obtained by the displacement analysis. The interpolation processing unit 920 computes the displacement rates of all the pixels by interpolating the displacement rates of the stable reflection points, for example.”).
It would have been obvious to one of ordinary skill in the art prior to the effective filing date of the claimed invention to incorporate the features as disclosed by Ikefuji into the invention of Beckett in view of Yokoyama. The set of Beckett, Yokoyama, and Ikefuji are considered analogous arts to the claimed invention as they both disclose the fusion of radar measurements with optical image data. It would have been obvious to one of ordinary skill in the art prior to the effective filing date of the claimed invention to modify the system as disclosed by Beckett in view of Yokoyama to temporally align the optical data with the SAR measurements using an interpolation method as taught by Ikefuji. One of ordinary skill in the art prior to the effective filing date of the claimed invention would have been motivated to utilize temporal interpolation in order to generate a set of optical and radar measurements in a scenario when two precisely synchronous measurements do not exist, which would allow for the coregistration of said images. (See Yokoyama [0007-0008] and Ikefuji [0015-0017]).
Claim(s) 6 and 29 is/are rejected under 35 U.S.C. 103 as being unpatentable over Beckett et al. (US 20180172824 A1), hereinafter Beckett, in view of in view of Yokoyama et al. (US 20210104154 A1), hereinafter Yokoyama, and further in view of Huang et al. (Huang, Bo, et al. "Cloud removal from optical satellite imagery with SAR imagery using sparse representation." IEEE Geoscience and Remote Sensing Letters 12.5 (2015): 1046-1050. ), hereinafter Huang.
Regarding claim 6, Beckett in view of Yokoyama discloses the system of claim 3. Beckett in view of Yokoyama fails to disclose the limitations below. Huang discloses:
wherein the at least one data integrator spatially aligns the at least one of the ground-based visual data, the aerial data, the historical data, and the architectural data with the at least one the images, the video, and the measurements using a registration process (Huang 1048 “Before the coregistration of two images, SAR data must be terrain corrected and speckle reduced”).
It would have been obvious to one of ordinary skill in the art prior to the effective filing date of the claimed invention to incorporate the features as disclosed by Huang into the invention of Beckett in view of Yokoyama. The set of Beckett, Yokoyama, and Huang are considered analogous arts to the claimed invention as they both disclose the fusion of radar measurements with optical image data. It would have been obvious to one of ordinary skill in the art prior to the effective filing date of the claimed invention to modify the system as disclosed by Beckett in view of Yokoyama to spatially align the optical data with the SAR measurements using a registration process as taught by Huang. Yokoyama teaches spatially aligning measurements, but not specifically by a registration method. One of ordinary skill in the art prior to the effective filing date of the claimed invention would have been motivated to utilize a registration method to spatially coalign a set of optical and radar measurements in order to create dictionary pairs of matching optical and radar images for the purposes of classification training of an AI system, or so that matching images may be overlayed on top of each other. (See Huang Pg. 1048).
Regarding claim 29,
The method of claim 26, wherein spatially aligning the at least one of the ground-based visual data, the aerial data, the historical data, and the architectural data with the at least one of the images, the video, and the measurements using a registration process (Huang 1048 “Before the coregistration of two images, SAR data must be terrain corrected and speckle reduced”).
It would have been obvious to one of ordinary skill in the art prior to the effective filing date of the claimed invention to incorporate the features as disclosed by Huang into the invention of Beckett in view of Yokoyama. The set of Beckett, Yokoyama, and Huang are considered analogous arts to the claimed invention as they both disclose the fusion of radar measurements with optical image data. It would have been obvious to one of ordinary skill in the art prior to the effective filing date of the claimed invention to modify the system as disclosed by Beckett in view of Yokoyama to spatially align the optical data with the SAR measurements using a registration process as taught by Huang. Yokoyama teaches spatially aligning measurements, but not specifically by a registration method. One of ordinary skill in the art prior to the effective filing date of the claimed invention would have been motivated to utilize a registration method to spatially coalign a set of optical and radar measurements in order to create dictionary pairs of matching optical and radar images for the purposes of classification training of an AI system, or so that matching images may be overlayed on top of each other. (See Huang Pg. 1048).
Claim(s) 8 is/are rejected under 35 U.S.C. 103 as being unpatentable over Beckett et al. (US 20180172824 A1), hereinafter Beckett, in view of Sandberg, Monica. "Land cover mapping with multi-temporal SAR and optical satellite data." (2016), hereinafter Sandberg.
Regarding claim 8, Beckett teaches the system of claim 7. Beckett fails to teach the limitations below. Sandberg teaches:
The system of claim 7, wherein the spatial coordinates include global positioning system (GPS) coordinates (Sandberg Tables 3 and 4 specify optical and SAR measurements by GPS coordinates.).
It would have been obvious to one of ordinary skill in the art prior to the effective filing date of the claimed invention to incorporate the features as disclosed by Sandberg into the invention of Beckett. Both Beckett and Sandberg are considered analogous arts to the claimed invention as they both disclose methods of combining SAR data with aerial optical data. It would have been obvious to one of ordinary skill in the art prior to the effective filing date of the claimed invention to modify the system as disclosed by Beckett to utilize GPS coordinates as taught by Sandberg. GPS coordinates are a standard method of labelling location data in maps as known to those within the art, and is explicitly taught by Sandberg. One of ordinary skill in the art prior to the effective filing date of the claimed invention would have been motivated to utilize GPS coordinates in order to clearly classify images based on their location, which is required in order to that the paired images are of the same target of interest (See Beckett [0026-0028]).
Claim(s) 11 is/are rejected under 35 U.S.C. 103 as being unpatentable over Beckett et al. (US 20180172824 A1), hereinafter Beckett, in view of in view of Yokoyama et al. (US 20210104154 A1), hereinafter Yokoyama, and further in view of Pourshasmi et al. (Pourshamsi, Maryam, et al. "A machine-learning approach to PolInSAR and LiDAR data fusion for improved tropical forest canopy height estimation using NASA AfriSAR Campaign data." IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing 11.10 (2018): 3453-3463.), hereinafter Pourshasmi.
Regarding claim 11,
The system of claim 10, wherein the at least one data integrator is further executable to rescale, reproject, or match resolutions of one or more of the ground-based visual data, the aerial data, the historical data, the architectural data, the images, the video, and the measurements (Pourshamsi Pg. 3455 “UAVSAR was registered to the DEM using an accurate geometry-based colocation based on airborne look vectors and Doppler information commonly used in interferometry. The DEM then was up sampled to match the full resolution of UAVSAR images.”).
It would have been obvious to one of ordinary skill in the art prior to the effective filing date of the claimed invention to incorporate the features as disclosed by Pourshasmi into the invention of Beckett in view of Yokoyama. The set of Pourshasmi, Beckett, and Yokoyama are considered analogous arts to the claimed invention as they disclose methods for the matching of aerial visual data and SAR measurements. It would have been obvious to one of ordinary skill in the art prior to the effective filing date of the claimed invention to modify the system as disclosed by Beckett in view of Yokoyama match resolutions of the radar and optical measurements as taught by Pourshasmi. One of ordinary skill in the art prior to the effective filing date of the claimed invention would have been motivated to modify the system of Beckett in view of Yokoyama in order to directly compare the two measurements, using operations like subtraction and pixel-by-pixel correlation for training data (See Beckett [0014], [0068], [0071]).
Claim(s) 12 is/are rejected under 35 U.S.C. 103 as being unpatentable over Beckett et al. (US 20180172824 A1), hereinafter Beckett, in view of Broggi et al. (US 20210142055 A1), hereinafter Broggi.
Regarding claim 12,
The system of claim 1, wherein the at least one data integrator includes a sensor data integrator that receives the ground-based visual data from data sources including one or more street cameras or traffic light cameras (Broggi [0135] “Fixed structures 420a-420d are shown. In the example scenario 400, the fixed structure 420a may be a set of traffic lights” .. “Generally, the fixed structures 420a-420n may comprise objects that may be stationary (or not moved often, such as a street sign that may be replaced occasionally) and/or may provide a vantage point for mounting the camera system 100.”).
Claim(s) 14 and 18 is/are rejected under 35 U.S.C. 103 as being unpatentable over Beckett et al. (US 20180172824 A1), hereinafter Beckett, in view of Ikefuji et al. (US 20210223389 A1), hereinafter Ikefuji.
Regarding claim 14,
The system of claim 1, wherein the at least one data integrator includes a historical data integrator that receives the historical data or the architectural data including one or more of architectural plans, blueprints, construction plans, historical photographs, and zoning maps (Ikefuji [0035] “an association unit which associates the generated cluster corresponding to the predetermined observing direction with a structure indicated by map data corresponding to the region to be observed; ”).
Regarding claim 18,
The system of claim 1, wherein the at least one data integrator operates in a cloud environment (Ikefuji [0205] “For example, the information processing device or circuit may be implemented as a client and server system or a cloud computing system, each component of which is connected via a communication network.”).
Claim(s) 15 is/are rejected under 35 U.S.C. 103 as being unpatentable over Beckett et al. (US 20180172824 A1), hereinafter Beckett, in view of Ikefuji et al. (US 20210223389 A1), hereinafter Ikefuji, and further in view of Broggi et al (US 20210142055 A1) , hereinafter Broggi.
Regarding claim 15,
The system of claim 14, wherein the historical data integrator is executable to perform Optical Character Recognition (OCR) to produce text data from one or more of the images and the video (Broggi [0028] “In some embodiments, optical character recognition (OCR) may be implemented. The image quality may be sufficient to enable the license plate characters to be human-readable (e.g., to enable a person to double-check results). ”).
Claim(s) 16 is/are rejected under 35 U.S.C. 103 as being unpatentable over Beckett et al. (US 20180172824 A1), hereinafter Beckett, in view of Reynolds et al. (US 20210405182 A1), hereinafter Reynolds.
Regarding claim 16,
The system of claim 1, wherein the one or more SAR algorithms include one or more of a Range-Doppler Algorithm, a Chirp Scaling Algorithm, a Polar Format Algorithm, and a Backprojection Algorithm (Reynolds [01114], “In some examples, post-processing may be used to up-sample the measurement data before image formation if using algorithms such as a Range-Doppler algorithm, Ωk algorithm, SPECAN, or Chirp-Scaling algorithm.” Examiner notes that these algorithms are well known to those of ordinary skill in the art in the generation of images from SAR signals, with Reynolds being a nonlimiting example that states this explicitly.).
It would have been obvious to one of ordinary skill in the art prior to the effective filing date of the claimed invention to incorporate the features as disclosed by Reynolds into the invention of Beckett. Both Beckett and Reynolds are considered analogous arts to the claimed invention as they both disclose SAR image processing. It would have been obvious to one of ordinary skill in the art prior to the effective filing date of the claimed invention to modify the system as disclosed by Beckett process SAR images using a Range-doppler algorithm as taught by Reynolds. The production of images from SAR data inherently implies the use of an algorithm, and while one is not specifically disclosed in Beckett it is known to those of ordinary skill in the art that common algorithms for this purpose include those listed in the limitation, and Reynolds provides a specific example of this. One of ordinary skill in the art prior to the effective filing date of the claimed invention would have been motivated to utilize a Range-Doppler algorithm in order to process the raw SAR data, improving the quality of the SAR image by reducing the inherent noise of the SAR image from effects like speckling, or by removing the effects of Doppler aliasing as known to those in the art (See Reynolds [0114]).
Claim(s) 19 is/are rejected under 35 U.S.C. 103 as being unpatentable over Beckett et al. (US 20180172824 A1), hereinafter Beckett, in view of Sajwaj et al. (US 20190025423 A1), hereinafter Sajwaj.
Regarding claim 19,
The system of claim 1, wherein one or more of the ground-based visual data, the aerial data, the historical data, and the architectural data are obtained from one or more third party sources (Sajwaj [0103] “Alternatively, the detector module 207 may request the SAR and optical data from the external database 217, where the external database has been provided with the SAR and optical data from the SAR detector 203 and optical detector 205, respectively.”).
It would have been obvious to one of ordinary skill in the art prior to the effective filing date of the claimed invention to incorporate the features as disclosed by Sajwaj into the invention of Beckett. Both Beckett and Sajwaj are considered analogous arts to the claimed invention as they both disclose systems for the combination of optical and SAR measurements. It would have been obvious to one of ordinary skill in the art prior to the effective filing date of the claimed invention to modify the system as disclosed by Beckett to obtain the optical measurements from an external source as taught by Sajwaj. One of ordinary skill in the art prior to the effective filing date of the claimed invention would have been motivated to modify the system of Beckett in order to combine the SAR measurements with those from publicly available satellite surveys or other publicly available data, eliminating the need for both an SAR measurement and an optical measurement (See Sajwaj [0096]).
Claim(s) 22-23 is/are rejected under 35 U.S.C. 103 as being unpatentable over Beckett et al. (US 20180172824 A1), hereinafter Beckett, in view of Kim et al. (US 20210215818 A1), hereinafter Kim .
Regarding claim 22,
The system of claim 21, wherein the distributed or decentralized server includes a peer-to-peer server (Kim [0057] “In an example, the computing machine 500 may act as a peer machine in peer-to-peer (P2P) (or other distributed) network environment. In this document, the phrases P2P, device-to-device (D2D) and sidelink may be used interchangeably. ”).
It would have been obvious to one of ordinary skill in the art prior to the effective filing date of the claimed invention to incorporate the features as disclosed by Kim into the invention of Beckett. Both Beckett and Kim are considered analogous arts to the claimed invention as they both disclose analysis methods for SAR image data. It would have been obvious to one of ordinary skill in the art prior to the effective filing date of the claimed invention to as disclosed by Beckett to utilize a peer-to-peer server as taught by Kim. A peer-to-peer server is simply a network communication protocol, which is well known within the art. It is obvious that the result of the image generation process must be output somehow, and a peer-to-peer network is one method to achieve this. One of ordinary skill in the art prior to the effective filing date of the claimed invention would have been motivated to modify the system of Beckett in order to output the visualization to a peer-to-peer network.
Regarding claim 23,
The system of claim 1, further comprising artificial intelligence (AI) and machine learning (ML) models trained to detect people, objects, or events in the integrated visualization (Kim [0043] “The machine learning program may be implemented at one or more computing machines. Block 302 illustrates a training set, which includes multiple classes 304. Each class 304 includes multiple images 306 associated with the class. Each class 304 may correspond to a type of object in the image 306 (e.g., a digit 0-9, a man or a woman, a cat or a dog, etc.).”).
It would have been obvious to one of ordinary skill in the art prior to the effective filing date of the claimed invention to incorporate the features as disclosed by Kim into the invention of Beckett. Both Beckett and Kim are considered analogous arts to the claimed invention as they both disclose analysis methods for SAR image data. It would have been obvious to one of ordinary skill in the art prior to the effective filing date of the claimed invention to as disclosed by Beckett to use a machine learning model to detect people or objects taught by Kim. This would be advantageous in situations where the system is being used to track a localized area, such as a traffic intersection or a worksite. One of ordinary skill in the art prior to the effective filing date of the claimed invention would have been motivated to modify the system of Beckett in order to classify characteristics within the generated images (See Kim [0043-0045]).
Claim(s) 31 is/are rejected under 35 U.S.C. 103 as being unpatentable over Beckett et al. (US 20180172824 A1), hereinafter Beckett, in view of in view of Yokoyama et al. (US 20210104154 A1), hereinafter Yokoyama, and further in view of Sandberg, Monica. "Land cover mapping with multi-temporal SAR and optical satellite data." (2016), hereinafter Sandberg.
Regarding claim 31,
The method of claim 30, wherein the spatial coordinates include global positioning system (GPS) coordinates (Sandberg Tables 3 and 4 specify optical and SAR measurements by GPS coordinates.).
It would have been obvious to one of ordinary skill in the art prior to the effective filing date of the claimed invention to incorporate the features as disclosed by Sandberg into the invention of Beckett in view of Yokoyama. The set of Beckett, Yokoyama, and Sandberg are considered analogous arts to the claimed invention as they disclose methods of combining SAR data with aerial optical data. It would have been obvious to one of ordinary skill in the art prior to the effective filing date of the claimed invention to modify the system as disclosed by Beckett in view of Yokoyama to utilize GPS coordinates as taught by Sandberg. GPS coordinates are a standard method of labelling location data in maps as known to those within the art, and is explicitly taught by Sandberg. One of ordinary skill in the art prior to the effective filing date of the claimed invention would have been motivated to utilize GPS coordinates in order to clearly classify images based on their location, which is required in order to that the paired images are of the same target of interest (See Beckett [0026-0028]).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to THOMAS JAMES HALLORAN whose telephone number is (571)272-8643. The examiner can normally be reached Mon-Fri. 7:30am-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Resha H. Desai can be reached at (571) 270-7792. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/T.J.H./Examiner, Art Unit 3648
/PETER M BYTHROW/Primary Examiner, Art Unit 3648
Prosecution Insights