Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-20 are rejected under 35 U.S.C. 103 as being unpatentable over DAS (US 20220083774) in view of Bhate et al. (US 20220309708).
Regarding claims 1, 8 and 15 DAS disclose a system/method and a non-transitory computer readable medium (DAS Figs. 1-2 and 3A-3B, paragraph 0004-0006 and also note: claims 1, 6 and 11)
a processor; and a memory in communication with the processor, the memory having instructions that, when executed by the processor, cause the processor to (DAS Figs. 1-2 and 3A-3B):
generate a point cloud of a location having one or more assets using a pre-trained monocular depth estimation network that receives an image of the location as an input (DAS Figs. 1 and 3A, DAS paragraph 0029 discloses referring now to the steps of the method 300, at step 306, the one or more hardware processors 104 generate, via a 3D point cloud generator, a 3D point cloud based on the overlapped images identified from the plurality of input images of asset being inspected. Here, the plurality of input images of the rooftop, the chimneys, the pipelines and the gas holder are captured using the unmanned aerial vehicles which are considered as one or more object of interests in overlapped with a predefined percentage value such as 60-80% from all the views of the monitoring environment. Further, the 3D point cloud for the plurality of input image frames serve as input to the 3D reconstruction algorithm associated with the 3D point cloud generator generated based on the overlapped images. This obviously corresponds to generate a point cloud of a location having one or more assets using a pre-trained monocular depth estimation network that receives an image of the location as an input), and
generate asset information at the location using an output head that receives the point cloud and generates the asset information (DAS Figs. 1 and 3B [BLOCKS 312-316 , note: CONVERTING 3D PPONT CLOUD INTO 2D IMAGE FRAMES] paragraph DAS disclose 0032 Referring now to the steps of the method 300, at step 312, the one or more hardware processors 104 detect one or more object of interests from the plurality 2D image frames using a mask convolutional network (RCNN). Further, from the plurality of 2D image frames one or more object of interests are detected using a single shot detector (SSD) mobile net deep neural network. The object detection module 110 of the system 100 detects one or more object of interests present in each 2D image frame from the plurality of 2D image frames. The single shot detector (SSD) mobile net deep neural network detects one or more object of interests at faster rate. The one or more object of interests are marked and fed to the single shot detector (SSD) mobile net deep neural network which learns the features to predict real-time/real-world scenarios. The trained convolutional network (RCNN) model with images marked with objects is defects. The masked convolutional network (RCNN) is an instance segmentation to localize the defects for different objects and the model is trained to identify defects corresponding the object and paragraph 0034 DAS disclose Referring now to the steps of the method 300, at step 316, the one or more hardware processors 104 detect, via a change detection technique, one or more defects observed with the one or more objects of interests based on the change observed in each 2D image frame from the plurality 2D image frames in a specific view with an identical objects of interest of the 2D image frame representing the same asset based on varying time stamps and a EXIF data based closest possible pairing. The geographical distance of the 2D image frames is calculated in a specific view with the identical objects of interest of the 2D image frame representing the same asset in varying time stamps. Further, the EXIF data is identified based on closest possible pairing based on the calculated geographical distance. This obviously corresponds to generate asset information at the location using an output head that receives the point cloud and generates the asset information).
In the same field of endeavor Bhate disclose generate a point cloud of a location having one or more assets using a pre-trained monocular depth estimation network that receives an image of the location as an input (Bhate Fig. 3, ABSTRACT, paragraph 0009 and 0058 Bhate disclose one or more embodiments, the 3D model generator may be configured to receive tracked binary masks 224 across plurality of images 204 and may be configured to generate a 3D model of the physical asset based on the position data 206 and orientation data 208 associated with each of the binary masks 224 and create a dense 3D point cloud model. The received binary masks 224 for the physical asset from each of the plurality of images 204 may be imported into a 3D space in the form of flat surface along the virtual vertical central axis located at the origin of the coordinate system [0,0,0]. The orientation data 208 may comprise the pitch angles, yaw angles and roll angles which together define the orientation of the camera/UAV at the time of image capture. From this set of values, the corresponding yaw angle may be used to rotate the binary mask by an angle equal to the yaw angle. The rotation angle may further be corrected using the FOV (Field of View) which is a camera parameter. The FOV (in degrees) is the extent of the observable world captured by the image capturing device within an image. There may be multiple assets at different locations within the same image, each asset having its own binary mask).
generate asset information at the location using an output head that receives the point cloud and generates the asset information (Bhate 0058 disclose the 3D model generator may be configured to receive tracked binary masks 224 across plurality of images 204 and may be configured to generate a 3D model of the physical asset based on the position data 206 and orientation data 208 associated with each of the binary masks 224 and create a dense 3D point cloud model. The received binary masks 224 for the physical asset from each of the plurality of images 204 may be imported into a 3D space in the form of flat surface along the virtual vertical central axis located at the origin of the coordinate system and paragraph 0059 Bhate disclose Once the mask has been rotated and repositioned, an inverse-projection from the 2D binary mask may be performed to generate a 3D volume towards and beyond the vertical central axis. The 3D volume may be represented by a mesh, which is a minimal set of vertices describing 3D shape of the projected volume. The inverse-projection process may be repeated for each mask associated with the physical asset in each of the plurality of images 204. In case of multiple physical assets i.e. multiple binary masks 224 in each image, the afore-mentioned process of inverse-projection may be performed for each of the multiple physical assets. In one or more embodiments, the position data 206 associated each of the plurality of images 204 may be used to estimate the geographical location of the physical asset. This can be used to integrate the estimated orientation information with Geographic Information System (GIS) tools and for visualizing the geographical locations of the physical assets at the site).
Therefore it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to generate a point cloud of a location having one or more assets using a pre-trained monocular depth estimation network that receives an image of the location as an input and generate asset information at the location using an output head that receives the point cloud and generates the asset information as shown by combination of DAS and Bhate because such a system provides automated system for structural asset inspection and tacking over time.
Regarding claim 2, 9 and 16 DAS disclose the output head is trained separately from the pre-trained monocular depth estimation network (DAS Fig. 2, In the system of it would be obvious to trained output head disclose in paragraph 0032-0034 of detecting defects and pre-trained monocular depth estimation network disclosed in paragraph 0029 to train separately).
Regarding claim 3, 10 and 17 DAS disclose the asset information includes at least one of: identifiers of the one or more assets, locations of the one or more assets, types of the one or more assets, a number of the one or more assets, distances of the one or more assets to a camera that generated the image, and distances between the one or more assets (DAS Figs. 3A-3B paragraph 0034 disclose Referring now to the steps of the method 300, at step 316, the one or more hardware processors 104 detect, via a change detection technique, one or more defects observed with the one or more objects of interests based on the change observed in each 2D image frame from the plurality 2D image frames in a specific view with an identical objects of interest of the 2D image frame representing the same asset based on varying time stamps and a EXIF data based closest possible pairing. The geographical distance of the 2D image frames is calculated in a specific view with the identical objects of interest of the 2D image frame representing the same asset in varying time stamps. Further, the EXIF data is identified based on closest possible pairing based on the calculated geographical distance and paragraph 0035 n one embodiment, based on the EXIF data available in each of the captured input images. This EXIF data provides different geographic locations data which helps to locate neighboring input images in different sessions. The EXIF data includes latitude, longitude, altitude of drone position, camera pan/tilt/yaw and therefore to find closest matched image pairs between the sessions of flight for any arbitrary target asset. Further, a correspondence has been established through the aforementioned method and defect magnitude is compared. In the system of DAS it would be obvious to detect the asset information includes at least one of: identifiers of the one or more assets, locations of the one or more assets, types of the one or more assets, a number of the one or more assets, distances of the one or more assets to a camera that generated the image, and distances between the one or more assets).
Regarding claim 4, 11 and 18 DAS disclose label points of the point cloud by the output head with the asset information (DAS Figs. 3A-3B paragraph 0030-0031 disclose Referring now to the steps of the method 300, at step 308, the one or more hardware processors 104 convert the 3D point cloud into a plurality of 2D image frames based on the angle of projection from the top view of the asset being inspected. A distance map is created from the above generated 3D point cloud using a fixed reference point. Further, a 3D grid is generated for the given output image size using the distance map and intensity values of the corresponding points of the 3D point cloud. Further, the angle of projections is calculated dynamically by determining the flight pattern using GPS metadata such as latitude, longitude, and altitude and Referring now to the steps of the method 300, at step 310, the one or more hardware processors 104 generate a birds eye view of the asset based on the plurality 2D image frames, wherein one or more empty patches of each 2D image frame from the plurality 2D image frames are filled based on range-domain filtering. The redundant part of each 2D image frame is cropped by detecting the biggest closed contour. The empty patches filled with range-domain filtering prevents the asset edges and outer boundary of each 2D image frame from the plurality of 2D image frames using the dynamic kernels and paragraph 0032 disclose Referring now to the steps of the method 300, at step 312, the one or more hardware processors 104 detect one or more object of interests from the plurality 2D image frames using a mask convolutional network (RCNN). Further, from the plurality of 2D image frames one or more object of interests are detected using a single shot detector (SSD) mobile net deep neural network. The object detection module 110 of the system 100 detects one or more object of interests present in each 2D image frame from the plurality of 2D image frames. The single shot detector (SSD) mobile net deep neural network detects one or more object of interests at faster rate. The one or more object of interests are marked and fed to the single shot detector (SSD) mobile net deep neural network which learns the features to predict real-time/real-world scenarios. The trained convolutional network (RCNN) model with images marked with objects is defects. The masked convolutional network (RCNN) is an instance segmentation to localize the defects for different objects and the model is trained to identify defects corresponding the object. In the system of DAS it would obvious to label 2D image frames with 3D point cloud from which 2D image frame are converted and the features).
Regarding claims 5, 12 and 19 DAS disclose store a plurality of point clouds generated by the pre-trained monocular depth estimation network of images captured at different times; and determine one or more temporal characteristics of the one or more assets over time by comparing at least two of the plurality points clouds (DAS Fig 3B note: 3D point cloud is converted into 2D image frames and paragraph 0034 disclose Referring now to the steps of the method 300, at step 316, the one or more hardware processors 104 detect, via a change detection technique, one or more defects observed with the one or more objects of interests based on the change observed in each 2D image frame from the plurality 2D image frames in a specific view with an identical objects of interest of the 2D image frame representing the same asset based on varying time stamps and a EXIF data based closest possible pairing. The geographical distance of the 2D image frames is calculated in a specific view with the identical objects of interest of the 2D image frame representing the same asset in varying time stamps. Further, the EXIF data is identified based on closest possible pairing based on the calculated geographical distance. This obviously corresponds to store a plurality of point clouds generated by the pre-trained monocular depth estimation network of images captured at different times; and determine one or more temporal characteristics of the one or more assets over time by comparing at least two of the plurality points clouds )
Regarding claim 6, 13 and 20 DAS the one or more temporal characteristics include one or more of: three-dimensional location of the one or more assets, changes in a shape of the one or more assets, changes in a volume of the one or more assets, velocity of the one or more assets (DAS paragraph 0033 disclose Referring now to the steps of the method 300, at step 314, the one or more hardware processors 104 detect, via an anomaly detection technique, anomalies present in the one or more objects of interests associated with the plurality 2D image frames. The anomalies are detected based on a training data serving as ground truth associated with the trained convolutional autoencoder. The training data comprises samples of the one or more objects of interest for determining deviation observed. Based on the one or more object of interests detected, the anomaly detection module 112 of the system 100 detects anomalies such as foreign objects which includes ropes, wires and thereof are detected as unseen and unknown definition. The unknown objects are considered as anomalies present on the one or more object of interests. The anomaly detection technique extracts a plurality of features of the one or more objects of interest present in each 2D image frame to detect anomalies are extracted by the trained convolutional autoencoder and further tries to regenerate the same 2D image in the encoder block based on the encoded features. The plurality of features of the one or more objects of interest present in each 2D image frame are matched with the training dataset. The convolutional autoencoder has several clean one or more object of interests and is capable of generating clean images even if anomalies detected 2D image frame are fed to the autoencoder. The anomalies present in the one or more objects of interest observed in each 2D image frame are detected based on the maximum matching difference obtained with the one or more objects of interest present in each 2D image frame with the training dataset. The anomalies are null hypothesis of the pattern statistics with the clean objects. Here, difference between the 2D image frame with the anomalies present and the regenerated clean object image is computed based on pixel wise tolerance. A clustering algorithm is implemented by the system 100 of the present disclosure, wherein the clustering algorithm groups the nearby highlighted points and localizes it as one object of interest and paragraph 0034 DAS disclose Referring now to the steps of the method 300, at step 316, the one or more hardware processors 104 detect, via a change detection technique, one or more defects observed with the one or more objects of interests based on the change observed in each 2D image frame from the plurality 2D image frames in a specific view with an identical objects of interest of the 2D image frame representing the same asset based on varying time stamps and a EXIF data based closest possible pairing. The geographical distance of the 2D image frames is calculated in a specific view with the identical objects of interest of the 2D image frame representing the same asset in varying time stamps. It is obvious in the system of DAS to one or more temporal characteristics include one or more of: three-dimensional location of the one or more assets, changes in a shape of the one or more assets, changes in a volume of the one or more assets, velocity of the one or more assets)
Regarding claim 7 and 14 DAS capture the image of the location using at least one camera mounted on one or more of a movable entity and a fixed location (DAS ABSTRACT Fig. 2, paragraph 0017, 0023 and 0029 disclose capture images using aerial vehicle using at-least one camera of the location area and furthermore it would be obvious to deploy one fixed conventional camera on the ground to capture images for inspection of the location).
Communication
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ISHRAT I SHERALI whose telephone number is (571)272-7398. The examiner can normally be reached Monday-Friday 8:00AM -5:00 PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Matthew Bella can be reached on 571-272-7778. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ISHRAT I SHERALI/Primary Examiner, Art Unit 2667
ISHRAT I. SHERALI
Examiner
Art Unit 2667
Prosecution Insights