DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1-20 are rejected under 35 U.S.C. 102(a)(2) as being anticipated by Liu et al. (U.S. Publication No. 2020/0026720; hereinafter Liu).
Regarding claim 1, Liu teaches a system for monitoring volumetric spaces using time-series analysis, the system comprising: a multi-modal sensor module including at least one imaging modality operable in a non-visual wavelength spectrum (Liu: Par. 73; i.e., the UAV may be able to detect elevation data relating to objects in a nearby airspace; Par. 94; i.e., the additional information can include a timestamp for when a sensor of the UAV generated data that led to an estimate of a surface height at the 2D coordinate; Par. 68; i.e., Exemplary sensors suitable for use with the embodiments disclosed herein include … vision sensors (e.g., imaging devices capable of detecting visible, infrared, or ultraviolet light, such as cameras)),
the multi-modal sensor module configured to generate data of a volumetric space (Liu: Par. 68; i.e., the UAVs described herein can include one or more sensors configured to collect relevant data, such as information relating to … the surrounding environment);
a mobility platform module configured to carry, integrate, and/or position the multi-modal sensor module, the mobility platform module including a navigable mission plan configured to guide the mobility platform through the volumetric space (Liu: Par. 62; i.e., the UAV can be controlled to move … along a specified movement path; Par. 68; i.e., the UAV may create the map or a portion of the map with aid of one or more sensors on-board the UAV);
a data processing module configured to: cause the multi-modal sensor module to perform one or more inspection instances by generating data of the volumetric space in accordance with the navigable mission plan for each inspection instance (Liu: Par. 95; i.e., In step 1310, the UAV receives data from a sensor. The UAV identifies a set of 2D coordinates… For each of the identified 2D coordinate, in steps 1320 and 1330, the UAV computes a surface height);
generate one or more scalar field arrays based on a time series analysis of the data, the one or more scalar field arrays being representative of one or more orthographic projections of spatially distributed physical properties of the volumetric space (Liu: Par. 82; i.e., The sensor data can be processed to build an elevation map, which indicates surface height information for each point in a 2D coordinate system… the surface height can also be determined using one or more sensors, such as a GPS receiver generating a 3D coordinate that is to be projected onto the reference level; Par. 91; i.e., FIG. 10 illustrates how a UAV builds an elevation map from sensor data);
and align, compare, and analyze the one or more scalar field arrays to detect a change, a trend, and/or an anomaly within the volumetric space (Liu: Par. 106; i.e., in step 1530, the central server examines the local elevation maps and resolves any inconsistencies or conflicts between the local elevation maps. For example, for each 2D coordinate of the overlapping regions, the central server can identify differences with respect to the data (e.g., estimated surface heights, confidence indicators or categories associated with the estimated surface heights) associated with the 2D coordinate between the local elevation maps).
Regarding claim 2, Liu teaches the system according to claim 1. Liu further teaches wherein the imaging modality includes: one or more radiometric infrared cameras; one or more near infrared cameras; one or more ultraviolet cameras; one or more optical gas imaging cameras; one or more acoustic imaging cameras; and/or one or more multi-spectral imaging cameras (Liu: Par. 68; i.e., Exemplary sensors suitable for use with the embodiments disclosed herein include … vision sensors (e.g., imaging devices capable of detecting visible, infrared, or ultraviolet light, such as cameras)).
Regarding claim 3, Liu teaches the system according to claim 1. Liu further teaches wherein: the multi-modal sensor module is configured to perform spatial sensing and generate spatial sensing data, wherein the spatial sensing is performed by: one or more visual spectrum imaging cameras configured to provide an image-based three- dimensional (3D) reconstruction via photogrammetry, neural radiance fields (NeRF), 3D gaussian splatting (3DGS), stereo vision, monocular depth estimation, and/or structured light projection; and/or one or more active ranging sensors configured to perform direct spatial measurements supported by localization via a LiDAR camera and/or a time-of-flight camera; the data processing module is configured to derive spatial structure or identify spatial features of the volumetric space from the spatial sensing data (Liu: Par. 68; i.e., other sensors may generate relative measurement data that is provided in terms of a local coordinate system… relative distance information provided by an ultrasonic sensor, lidar, or time-of-flight camera)).
Regarding claim 4, Liu teaches the system according to claim 1. Liu further teaches wherein: the data processing module is configured to associate spatial positioning and/or temporal metadata via data obtained from a localization sensor modality of the multi-modal sensor module, wherein the localization sensor modality includes: one or more global positioning system (GPS) modules; one or more real time kinetics (RTK) modules; one or more post processing kinetics (PPK) modules; one or more inertial measurement units (IMU) modules; one or more visual odometry (VO) modules; and/or one or more simultaneous location and movement (SLAM) modules (Liu: Par. 68; i.e., some sensors may generate absolute measurement data that is provided in terms of a global coordinate system (e.g., position data provided by a GPS sensor)).
Regarding claim 5, Liu teaches the system according to claim 1. Liu further teaches wherein: the mobility platform is configured to carry, integrate, and/or position the multimodal sensor module in or on an autonomous or operator guided device; the autonomous or operator guided device includes: one or more unmanned aerial vehicles (UAVs); one or more autonomous ground vehicles (AGVs); one or more autonomous underwater vehicles (AUVs); one or more handheld devices; one or more wearable devices; and/or one or more mobile phones comprising a sensor augmentation attachment for non-visual spectrum modality (Liu: Par. 113; i.e., The UAV may be an example of a movable object as described herein; Par. 115; i.e., the load includes a payload… any suitable sensor can be incorporated into the payload).
Regarding claim 6, Liu teaches the system according to claim 1. Liu further teaches wherein: the navigable mission plan includes one or more defined data collection actions along a defined path or coverage pattern through the volumetric space; the one or more data collection actions include: one or more discrete data collection actions at one or more sets of waypoints; and/or one or more continuous data acquisition actions along one or more splines (Liu: Par. 62; i.e., the UAV can be controlled to move … along a specified movement path; Par. 95; i.e., in steps 1350 and 1360, the UAV returns to the process of generating elevation data for each of the identified 2D coordinates until the end in step 1370, where the end could be the end of flight).
Regarding claim 7, Liu teaches the system according to claim 6. Liu further teaches wherein: the defined path or coverage pattern is based on a two-dimensional analysis, a pixel-based analysis workflow, a three-dimensional analysis, and/or a voxel-based analysis workflow (Liu: Par. 95; i.e., The UAV identifies a set of 2D coordinates corresponding to the detection range of the sensor with respect to a reference level);
one or more sensor modality parameters for the defined path or coverage pattern are optimized for comparative analysis (Liu: Par. 94; i.e., a timestamp for when a sensor of the UAV generated data that led to an estimate of a surface height at the 2D coordinate, as data freshness may play a role in prioritizing different data for the same 2D coordinate).
Regarding claim 8, Liu teaches the system according to claim 1. Liu further teaches wherein: the system is configured to execute the navigable mission plan performed autonomously, semi- autonomously, and/or by guided direction of a system operator (Liu: Par. 62; i.e., The UAVs described herein can be operated completely autonomously (e.g., by a suitable computing system such as an onboard controller), semi-autonomously, or manually (e.g., by a human user));
each execution is an instance of inspection; each instance of inspection repeats the navigable mission plan with consistent sensor modality parameters, consistent mobility platform orientation, and consistent spatial coverage of the volumetric space (Liu: Par. 95; i.e., The UAV can repeat this process based on a required frequency of data analysis).
Regarding claim 9, Liu teaches the system according to claim 8. Liu further teaches wherein: the system is configured to perform a series of inspection instances by repeated execution of the navigable mission plan across plural distinct timepoints; the data processing module is configured to generate comparative, trend-based, and/or time-series analyses of the volumetric space (Liu: Par. 106; i.e., the central server may receive, at a first time T1, a first local elevation map from a first UAV, and store the first local elevation map as a portion of the global elevation map. After a period of time the central server may receive, at a second time T2, a second local elevation map… The central server can be configured to process data … of the two local elevation map to resolve any conflicts or inconsistencies to update the global elevation map).
Regarding claim 10, Liu teaches the system according to claim 1. Liu further teaches wherein: the data processing module is configured to generate one or more data analysis pipelines for analyzing scalar field arrays from a non-visual spectrum imaging device collected from a series of inspection instances, wherein the one or more data analysis pipelines includes:(i) a two-dimensional analysis pipeline based on comparison of aligned orthographic projections or arrays of scalar field data; and/or (ii) a three-dimensional analysis pipeline based on comparison of scalar field data mapped to a meshed, point cloud, splat, and/or voxel representations (Liu: Par. 106; i.e., in step 1530, the central server examines the local elevation maps and resolves any inconsistencies or conflicts between the local elevation maps. For example, for each 2D coordinate of the overlapping regions, the central server can identify differences with respect to the data (e.g., estimated surface heights, confidence indicators or categories associated with the estimated surface heights) associated with the 2D coordinate between the local elevation maps).
Regarding claim 11, Liu teaches the system according to claim 10. Liu further teaches wherein: the data processing module is configured to convert one or more raw arrays of scalar field data and associated metadata into one or more spatially normalized and temporally sequential data structures for comparative analysis (Liu: Par. 106; i.e., the central server can be configured to process data (e.g., estimated surface heights, confidence indicators or categories associated with the estimated surface heights) of the two local elevation map to resolve any conflicts or inconsistencies to update the global elevation map).
Regarding claim 12, Liu teaches the system according to claim 11. Liu further teaches wherein: the data processing module is configured to perform the two-dimensional analysis pipeline by: a) using localization metadata to identify corresponding scalar field arrays from the at least one imaging modality operable in a non-visual wavelength spectrum across the series of inspection instances; and b) applying spatial sensing data to align the corresponding scalar field arrays at the pixel level using one or more registration techniques, the one or more registration techniques including a photogrammetry target technique, a computer vision target technique, a structure-from-motion (SfM) technique, a scale-invariant feature transform (SIFT) technique, a COLMAP technique, a feature-based matching technique, a mutual information technique, and/or a photogrammetric adjustment technique (Liu: Par. 106; i.e., the local elevation maps can correspond to distinct or overlapping regions… for each 2D coordinate of the overlapping regions, the central server can identify differences with respect to the data (e.g., estimated surface heights, confidence indicators or categories associated with the estimated surface heights) associated with the 2D coordinate between the local elevation maps; the elevation maps are aligned based on matching coordinates from each map).
Regarding claim 13, Liu teaches the system according to claim 11. Liu further teaches wherein: the data processing module is configured to perform the three-dimensional analysis by: a) processing spatial sensing data into a volumetric model; b) using localization metadata to assign scalar field values from the at least one imaging modality operable in a non-visual wavelength spectrum to specific spatial locations, thereby compiling all non-visual spectrum data from a specific instance of inspection into a 3D representation that is directly comparable to other non-visual spectrum data across the series of inspection instances through a common coordinate system; and c) using a photogrammetry target technique and/or a computer vision target technique for registration (Liu: Par. 73; i.e., a UAV collects data during its flight and builds two-dimensional (2D) or three-dimensional (3D) maps based on the collected data; Par. 106; i.e., the local elevation maps can correspond to distinct or overlapping regions… for each 2D coordinate of the overlapping regions, the central server can identify differences with respect to the data (e.g., estimated surface heights, confidence indicators or categories associated with the estimated surface heights) associated with the 2D coordinate between the local elevation maps; the elevation maps are aligned based on matching coordinates from each map either in 2D or 3D).
Regarding claim 14, Liu teaches the system according to claim 11. Liu further teaches a segmentation routine configured to define virtual sensors as individual pixels, pixel groupings, voxels, or voxel groupings, said virtual sensors being spatially registered for comparative analysis across time (Liu: Par. 101; i.e., for a single 2D coordinate, there may be more than one corresponding sensor data (pixel), such as a vertical edge of a building).
Regarding claim 15, Liu teaches the system according to claim 10. Liu further teaches wherein: the data processing module is configured to summarize or transform virtual sensor data using a statistical summarization technique, a rule-based inference technique, a physics-based model technique, a finite element model technique, a computational fluid dynamics technique, and/or a reduced order model technique (Liu: Par. 101; i.e., for a single 2D coordinate, there may be more than one corresponding sensor data (pixel), such as a vertical edge of a building. Therefore, there are two approaches in calculating the surface height: … (2) averaging surface heights for the 2D coordinate or a plurality of 2D coordinates within a unit area (e.g., a 1 m×1 m square) including the 2D coordinate).
Regarding claim 16, Liu teaches the system according to claim 1. Liu further teaches wherein: the data processing module includes a time-series analysis engine configured to analyze virtual sensor data across a series of inspection instances to identify a trend, forecast a condition, and/or detect an anomaly within the volumetric space (Liu: Par. 106; i.e., the central server may receive, at a first time T1, a first local elevation map from a first UAV, and store the first local elevation map as a portion of the global elevation map. After a period of time the central server may receive, at a second time T2, a second local elevation map… The central server can be configured to process data … of the two local elevation map to resolve any conflicts or inconsistencies to update the global elevation map).
Claims 17 and 18-20 are rejected under the same rationale as provided in the rejection of claims 1 and 6-8, respectively.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Additional prior art deemed pertinent in the art of volumetric space monitoring using scalar field arrays based on time series analysis includes Abrahams et al. (U.S. Publication No. 2017/0124843), Rosario et al. (U.S. Publication No. 2015/0023553), Montantes (U.S. Publication No. 2019/0332931), and Liu et al. (U.S. Publication No. 2018/0074200).
Any inquiry concerning this communication or earlier communications from the examiner should be directed to BRANDON Z WILLIS whose telephone number is (571)272-5427. The examiner can normally be reached Weekdays 8:00-5:30.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Erin D. Bishop can be reached at (571) 270-3713. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/BRANDON Z WILLIS/Examiner, Art Unit 3665
Prosecution Insights