DETAILED ACTION
This action is in response to the application filed on October 17th, 2024. Claims 1-15 are pending and have been examined.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submitted on October 17th, 2024 is being considered by the examiner.
Claim Objections
Claim 3 is objected to because of the following informality: “enhancing a resolution of the tagged image before the execution of the second decision criterion is performed using a deep neural network”, it is not immediately clear in this claim if the deep neural network is being used to enhance the resolution of the tagged image or if it used to perform execution of the second decision criterion. This claim could instead read “enhancing a resolution of the tagged image before the execution of the second decision criterion, wherein the enhancing the resolution is performed using a deep neural network”. Appropriate correction is required.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1, 5, 7-10, and 14 are rejected under 35 U.S.C. 103 as being unpatentable over “UAS: IoT on-line sensors for power line inspection” (herein after referred to by its primary author, Medrano) in view of “A New Scene Analysis Using Genetic Algorithm Based Fuzzy ID3 Method” (herein after referred to by its primary author, Chang).
In regards to claim 1, Medrano teaches a computer implemented method for selecting candidate images for a power line inspection process, the method comprising performing by one or more processors the following steps: receiving an image of an electricity distribution voltage infrastructure acquired by a camera included in a flying object (Medrano Section I B “For this paper we propose the use of a VTOL UAS, equipped with a low-power, low-weight, low-cost Autonomous Data Acquisition System (ADAS), for the capture of high quality visual data. A VTOL is the ideal platform for power line inspection because of its capabilities for long- range flights.”); tagging the received image with Global Navigation Satellite System (GNSS) metadata including a Global Positioning System (GPS) location of the camera when the image is acquired and a pose of the camera based on Inertial Measurement Unit (IMU) values of the flying object (Medrano Section II “The IoT architecture is designed in a way that data acquisition tasks do not interfere with flight tasks. The controller (FC), runs the autopilot software, which controls the flight of the UAS and allows for the communication of attitude, bearing, and positioning data with the ADAS.”); executing a first-decision criterion that determines whether a GPS location of a power tower is comprised within a 2D projection captured in the image using at least one of: a field- of-view (FOV), a pose, and a GPS location of the camera, and a list of possible GPS power tower locations (Medrano Section IV “For a scenario we define as Relevant (RTT) those Towers (TT) we want to capture, and Spurious (STT) those which do not. We define Action Radio (AcRa) of the ADAS as the maximum estimated distance that, rightly orientated, it can identify a TT in an image. The algorithm needs to know the position of the RTTs and their closest STTs as an input, once, in global coordinates. STT knowledge is needed to distinguish them from RTTs, as they may either be physically similar and confound the detection algorithm, close enough to mask RTTs, or both. Then, in each iteration, the FCU feeds the ADAS the current position, speed, and bearing of the VTOL. In all cases, these data are approximate, as the actual maps, and the flight sensors, are usually unprecise. Nevertheless, we can estimate if there could be RTT inside the AcRa.” Examiner note: This section teaches that the current position, speed, and bearing of the flying object can be used in conjunction with the positions of the relevant towers to determine if a relevant tower is within the maximum sight distance of the flying object.); executing a second-decision criterion that determines whether a power tower is included in the tagged image by analyzing the tagged image (Medrano Section IV “If there are, the gimbal is currently orientated towards the closest one, and the processing by the ADAS of Camera 1 video stream should start inferring TT. The detections, if any, are compared and paired against the input TT map to discard STT and select the RTT.”; Section II “While the OC executes code and acts as an interface with the rest of the system, it delegates the CNN inferences on the TPU, an application-specific integrated circuit for NN.” Examiner note: Section IV teaches that if a relevant tower is determined to be within the range of the camera, inference of the tower within the image is performed. This is performed by the CNN, as taught by section II); and selecting or discarding the received image as a candidate image of containing a power tower by executing a ruled based Medrano Section IV “The detections, if any, are compared and paired against the input TT map to discard STT and select the RTT. The ADAS corrects gimbal orientation, displacing the bounding box to the center of the captured stream. The process repeats until the RTT is considered as captured or, capturing time ends.” Examiner note: This section teaches that the flying object must be within a maximum distance of the relevant tower (first decision), and if that is true, then an image is passed to the CNN for inferring the location of the relevant tower within the image (second decision). Then, based on these decision, the power infrastructure can be detected within the image)
Medrano does not teach a rule based fuzzy decision system that combines a result of the first-decision criterion and of the second-decision criterion.
However, Chang teaches teach a rule based fuzzy decision system that combines a result of the first-decision criterion and of the second-decision criterion (Chang Section III “We humans recognize the objects appearing in scenes according to many features like colors, intensity, boundaries, and their positions. For example, the sky often has a bright blue color and is always in the topmost space. Trees are mostly in green color, grow on the ground, stretch the branches, and leaves toward the sky. The ground is in dark gray and always locates at the lowest space. Man-made objects can have various colors, but they often have regular shapes like rectangle or circle. To model scene objects, we adopt the intuition of human beings, i.e., the features of color, height, and shapes; these are the most useful ones to describe objects. It is well known that colors can be described by many color spaces. Among them, RGB is mostly accepted color space when figures are to be displayed on the computer screen. The object is, however, more adherent to another color space, that is, the HIS system. According to humans' visual experiences, the three components of Hue, Intensity, and Saturation are not in the same efficiency for classification processes. We choose only the hue and intensity components as our features. As described above, we also use the vertical position as the third feature to describe the objects.” Examiner note: This section teaches that, when identifying objects in images, the color, height, shape , and position of objects in the scene can be used. These attributes are analogous to the first and second decision criteria, as the first criterion identifies if the objects position is correct in the camera image, and the second criterion uses the shape and RGB data of the image to identify the power tower within the image. ).
Chang is considered to be analogous to the claimed invention because they are both in the same field of fuzzy rule based image classification. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have modified the system of Medrano to include the teachings of Chang, to provide the advantage of flexible representation when classifying objects in images (Chang Section I “Image analysis by defining image regions semantically through fuzzy induction rule inference has recently become a newer and favored research for image analysis task [19] because it can avoid making assumption about the data that, when violated, will cause parametric technique to fail. Moreover, fuzzy set has flexible representation ability in describing uncertainty, vagueness, and imprecision, which are easily found in the image itself or in every phase of image processing schemes whether in low-level or high-level. Hence, fuzzy notion can also be a suitable fit tool for image analysis.”)
In regards to claim 5, Medrano in view of Chang teaches the method of claim 1, wherein the second-decision criterion comprises applying a deep neural network (Medrano Section III “CNN-BASED TOWER IDENTIFICATION” ).
In regards to claim 7, Medrano in view of Chang teaches the method of claim 1, wherein the flying object comprises an unmanned aerial vehicle, a helicopter, a zeppelin, or an airplane (Medrano Section I “A current alternative is the use of BVLOS (Beyond Visual Line Of Sight) drone flights, which reduces human risks, and UAS (Unmanned Aerial Systems) based long endurance flights significantly reduce operational costs.”).
In regards to claim 8, Medrano in view of Chang renders obvious the claim limitations as in the consideration of claim 1.
In regards to claim 9, Medrano in view of Chang renders obvious the claim limitations as in the consideration of claim 7.
In regards to claim 10, Medrano in view of Chang renders obvious the claim limitations as in the consideration of claim 1.
In regards to claim 14, Medrano in view of Chang renders obvious the claim limitations as in the consideration of claim 5.
Claims 2 and 11 are rejected under 35 U.S.C. 103 as being unpatentable over Medrano in view of Chang as applied to the claims above, and further in view of “UAV-Based Oblique Photogrammetry for Outdoor Data Acquisition and Offsite Visual Inspection of Transmission Line” (herein after referred to by its primary author, Jiang) and US20160232792 (herein after referred to by its primary author, Van Cruyningen).
In regards to claim 2, Medrano in view of Chang teaches the method of claim 1, wherein the first-decision criterion comprises checking whether the FOV comprises the GPS location of the power tower based on the computed angle (Medrano Section IV “If there are, the gimbal is currently orientated towards the closest one, and the processing by the ADAS of Camera 1 video stream should start inferring TT. The detections, if any, are compared and paired against the input TT map to discard STT and select the RTT. The ADAS corrects gimbal orientation, displacing the bounding box to the center of the captured stream. The process repeats until the RTT is considered as captured or, capturing time ends.”).
Medrano in view of Chang does not teach projecting a triangle according to the FOV of the camera comprising a GPS location of the power tower; delimiting a trapezoid of the GPS location of the power tower by considering a minimum and a maximum distance; computing an angle at which the power tower is with respect to a center of a lens of the camera using the GPS location of the power tower and the delimited trapezoid; and checking whether the delimited trapezoid comprises the GPS location of the power tower based on the computed angle.
However, Jiang teaches projecting a triangle according to the FOV of the camera comprising a GPS location of the power tower (Jiang Figure 2); delimiting a trapezoid of the GPS location of the power tower by considering a minimum Jiang Figure 2; Section 2.1 “Figure 2 illustrates imaging geometry of the UAV oblique photogrammetry system for transmission line in both along-corridor and across-corridor directions, where h and H stand for heights of power pylon and flight trajectory, respectively; and w is the distance between the ground projection of flight trajectory and the base center of power pylon, which will be restricted by the minimum operation distance. Under determination of camera’s mounting angles α , β and FOV (Field Of View) θ, ground coverage width will be given by W, where α and β represent roll and pitch.”); computing an angle at which the power tower is with respect to a center of a lens of the camera using the GPS location of the power tower and the delimited trapezoid (Jiang Figure 2; Section 2.1 “Figure 2 illustrates imaging geometry of the UAV oblique photogrammetry system for transmission line in both along-corridor and across-corridor directions, where h and H stand for heights of power pylon and flight trajectory, respectively; and w is the distance between the ground projection of flight trajectory and the base center of power pylon, which will be restricted by the minimum operation distance. Under determination of camera’s mounting angles α , β and FOV (Field Of View) θ, ground coverage width will be given by W, where α and β represent roll and pitch.”); and checking whether the delimited trapezoid comprises the GPS location of the power tower based on the computed angle (Jiang Figure 2).
Jiang is considered to be analogous to the claimed invention because they are both in the same field of image based power tower inspection. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have modified the system of Medrano in view of Chang to include the teachings of Jiang, to provide the advantage of acquiring all information required for 3D analysis of power towers (Jiang Section 5.1 “the oblique photogrammetric system in this research has some advantages. Firstly, it facilitates obtaining sufficient images in along-track and across-track directions, which provides necessary information for 3D reconstruction of power pylon.”).
Furthermore, Van Cruyningen teaches considering a maximum distance (Van Cruyningen Paragraph [0026] “The resolution in aerial photography is estimated by the ground sample distance (GSD). The GSD is the separation between camera pixels as projected on the ground. The GSD equals the object distance (flying height above ground here) times the pixel separation divided by the lens focal length. So for 5 micron pixels and a lens focal length of 28 mm, at a flying height of 400 feet (˜120 m), the GSD˜2.1 cm. The sample distance requirement sets the maximum distance from camera 16 to the object of interest.”);
Van Cruyningen is considered to be analogous to the claimed invention because they are both in the same field of image based power tower inspection. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have modified the system of Medrano in view of Chang and Jiang to include the teachings of Van Cruyningen, to provide the advantage of ensuring the power tower is sufficiently within frame for inspection in one flight (Van Cruyningen Paragraph [0025] “Often it is desirable to capture a major part of the object of interest in one flight”)
In regards to claim 11, Medrano in view of Chang, Jiang, and Van Cruyningen renders obvious the claim limitations as in the consideration of claim 2.
Claims 3 and 12 are rejected under 35 U.S.C. 103 as being unpatentable over Medrano in view of Chang as applied to the claims above, and further in view of “Joint-SRVDNet: Joint Super Resolution and Vehicle Detection Network” (herein after referred to by its primary author, Mostofa).
In regards to claim 3, Medrano in view of Chang teaches the method of claim 1, but fails to teach enhancing a resolution of the tagged image before the execution of the second- decision criterion is performed using a deep neural network, the enhanced image being the input of the second-decision criterion.
However, Mostofa teaches enhancing a resolution of the tagged image before the execution of the second- decision criterion is performed using a deep neural network, the enhanced image being the input of the second-decision criterion (Mostofa Figure 4).
Mostofa is considered to be analogous to the claimed invention because they are both in the same field of object classification in images. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have modified the system of Medrano in view of Chang to include the teachings of Mostofa, to provide the advantage of improved super resolution reconstruction (Mostofa Section I “In this paper, we propose an end-to-end jointly trainable deep neural network what we named Joint-SRVDNet, which offers a multi-tasking paradigm by handling both super-resolution and vehicle detection for aerial and satellite imagery. To the best of our knowledge, our proposed Joint-SRVDNet is the first multi-task model that leverages complementary information of the two tasks to jointly learn Super-Resolution (SR) and vehicle detection in aerial images. Such a novel framework allows for improved super-resolution reconstructions and more accurate vehicle detection in aerial imagery.”)
In regards to claim 12, Medrano in view of Chang and Mostofa renders obvious the claim limitations as in the consideration of claim 3.
Claims 4 and 13 are rejected under 35 U.S.C. 103 as being unpatentable over Medrano in view of Chang and Mostofa as applied to the claims above, and further in view of “Super-Resolution Reconstruction Algorithm Based on Improved ESRGAN” (herein after referred to by its primary author, Yin).
In regards to claim 4, Medrano in view of Chang and Mostofa teaches the method of claim 3, but fails to teach wherein the deep neural network comprises an ESRGAN algorithm.
However, Yin teaches wherein the deep neural network comprises an ESRGAN algorithm (Yin Section II A).
It would have been obvious for one of ordinary skill in the art, before the effective filing date of the claimed invention, to implement the ESRGAN teaching of Yin into the system of Medrano in view of Chang and Mostofa since both teach a super resolution network that takes as input a low resolution image and outputs a high resolution image. One of ordinary skill in the art would have recognized that the ESRGAN offers advantages in reducing PSNR of reconstructed images (Yin Table III). The teachings of Yin can be incorporated into Medrano in view of Chang and Mostofa in that the SRGAN of Mostofa could be replaced with the ESRGAN of Sukesh. Furthermore, one of ordinary skill in the art could have combined the elements as claimed by known methods and, in combination, each component functions the same as it does separately. One of ordinary skill in the art would have recognized that the results of the combination would be predictable.
In regards to claim 13, Medrano in view of Chang, Mostofa, and Yin renders obvious the claim limitations of claim 4.
Claims 6 and 15 are rejected under 35 U.S.C. 103 as being unpatentable over Medrano in view of Chang as applied to the claims above, and further in view of “Autonomous Vision-Based Primary Distribution Systems Porcelain Insulators Inspection Using UAVs” (herein after referred to by its primary author, Rahman).
In regards to claim 6, Medrano in view of Chang teaches the method of claim 5, but fails to teach wherein the deep neural network comprises a Yolo algorithm.
However, Rahman teaches wherein the deep neural network comprises a Yolo algorithm (Rahman Section 2.7 “YoloV4”).
Rahman is considered to be analogous to the claimed invention because they are both in the same field of image based powerline inspection. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have modified the system of Medrano in view of Chang to include the teachings of Rahman, to provide the advantage of classification and prediction being performed at the same time (Rahman Section 2.7 “The YoloV4 object detection model is benchmarked on the MS COCO dataset [32], achieving 65 fps inference speed with an accuracy of 43.5% AP (65.7% AP50) on Tesla V100 [33]. Object detectors compress features of an input image down through a convolutional neural network backbone. The mixing and holding up of the feature layers from the convolutional backbone happens in the neck part of the object detector. The detection of a specific object in the image happens in the head part of the detector. As YoloV4 is a single-stage object detector, the classification and prediction of object localization are done at the same time.”)
In regards to claim 15, Medrano in view of Chang and Rahman renders obvious the claim limitations as in the consideration of claim 6.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to CALEB LOGAN ESQUINO whose telephone number is (703)756-1462. The examiner can normally be reached M-Fr 8:00AM-4:00PM EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Andrew Bee can be reached at (571) 270-5183. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/CALEB L ESQUINO/ Examiner, Art Unit 2677
/ANDREW W BEE/ Supervisory Patent Examiner, Art Unit 2677
Prosecution Insights