Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1-3, 10, and 23-25 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Harvey et al. (US PG Pub 20210374111, hereinafter “Harvey”).
Regarding claim 1, Harvey describes a method of drone-based inspection of an overhead asset of an electrical grid (para. 0003, wherein the disclosure of Harvey is directed to a method of inspection of utility assets, such as power poles), the method comprising: flying a drone toward the overhead asset (paras. 0044-0046, disclosing the use of an unmanned aerial vehicle (UAV) to gather the images used for construct analysis, wherein the construct is the aforementioned utility asset, and the UAV may vary in its type of operation (either fully autonomous, partially autonomous, or fully teleoperated)); capturing, via the drone, a plurality of digital images of the overhead asset (paras. 0047-0049, disclosing the capture of multiple images, either in the form of static images or a video stream, and associated information about the respective images); identifying, using computer vision with respect to the digital images, the overhead asset (para. 0051, containing the explicit disclosure of the determination of the construct and its location within images); and updating a geographic information system (GIS) database in response to identifying the overhead asset (paras. 0096-0097, disclosing the updating of a GIS with information on the construct upon recognition of the construct, its location, and particular information regarding its state).
Regarding claim 2, Harvey discloses all limitations of claim 1. Harvey further discloses wherein the digital images comprise a first digital image of the overhead asset taken at a first shot angle; and a second digital image of the overhead asset taken at a second shot angle that is different from the first shot angle (para. 0047, wherein the capture of multiple digital images from multiple different angles using gimballed cameras and alternate UAV positions is disclosed, wherein a first and a second image could arbitrarily be determined from the image stream as any images captured from different angles).
Regarding claim 3, Harvey discloses all limitations of claim 2. Harvey further discloses identifying the first and second shot angles (paras. 0047 and 0173, wherein para. 0047 discloses UAV image captures at multiple different angles and para. 0173 discloses wherein all images captured as part of the collected construct data pre-identification include associated capture angle data).
Regarding claim 10, Harvey discloses all limitations of claim 1. Harvey further discloses wherein identifying the overhead asset comprises classifying the overhead asset (para. 0051, containing the explicit disclosure of the determination of the construct and its location within images: “The machine learning model may use object recognition and/or the like (e.g., computer vision) on each of the one or more images of the construct 205 to determine/identify an image take from directly above the top of the construct 205 (e.g., a pole top, etc.).”), and wherein the method further comprises identifying, using computer vision with respect to the digital images, an attribute of the overhead asset.
Regarding claim 23, Harvey discloses all limitations of claim 1. Harvey further discloses wherein the digital images comprise respective still images that are captured by one or more cameras of the drone (para. 0047, wherein the capture of multiple digital images from multiple different angles using gimballed cameras and alternate UAV positions is disclosed, wherein the captured images may be static images captured at different points in time).
Regarding claim 24, Harvey discloses all limitations of claim 1. Harvey further discloses wherein the digital images comprise respective frames of a digital video that is captured by one or more cameras of the drone (para. 0047, wherein the capture of multiple digital images from multiple different angles using gimballed cameras and alternate UAV positions is disclosed, wherein the captured images may be adjacent or alternate frames of a video/image stream).
Regarding claim 25, Harvey discloses all limitations of claim 1. Harvey further discloses wherein the GIS database comprises GIS data about a distribution network of the electrical grid (para. 0097, wherein the GIS contains additional information and facilitated access).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 4-5, 7-9, and 21-22 are rejected under 35 U.S.C. 103 as being unpatentable over Harvey in view of Nguyen et al. (“Intelligent Monitoring and Inspection of Power Line Components Powered by UAVs and Deep Learning”, hereinafter “Nguyen”).
Regarding claim 4, Harvey discloses all limitations of claim 1. Harvey does not explicitly disclose herein the overhead asset comprises a distribution line device.
However, Nguyen explicitly discloses wherein the overhead asset comprises a distribution line device (pg. 15, para. 1, “The selected component classes include…a class for transformers, and 43 insulator classes”, wherein transformers and insulators are distribution line devices). Specifically, Nguyen discloses a method and system of UAV inspection of power line components, including convolutional neural net-based image object detection architecture and fault detection based on post-classification analysis. Therefore, both Harvey and Nguyen disclose methods and systems of power grid component detection, identification, and image analysis based on UAV capture of images. As the method and system of Nguyen explicitly discloses detection and classification of transformers and insulators, it would have been obvious to one having ordinary skill in the art prior to the effective filing date of the claimed invention to have utilized the detection and classification method of Nguyen of detecting and classifying power line distribution devices within the method and system of Harvey as the use of a known technique to improve a similar device; specifically, enabling detection of crucial powerline components, leading to a more robust assessment of potential issues within the grid framework.
Regarding claim 5, Harvey and Nguyen disclose all limitations of claim 4. Harvey does not disclose wherein the distribution line device comprises a transformer.
However, Nguyen discloses wherein the distribution line device comprises a transformer (pg. 15, para. 1, “The selected component classes include…a class for transformers”). Thus, it would have been obvious to one having ordinary skill in the art prior to the effective filing date of the claimed invention to have utilized the detection and classification method of Nguyen of detecting and classifying a transformer within the method and system of Harvey using the rationale of claim 4. (Note Fig. 4 of Harvey appears to show a transformer as well, but does not discuss it further.)
Regarding claim 7, Harvey discloses all limitations of claim 1. Harvey does not explicitly disclose wherein identifying the overhead asset comprises determining a likelihood that the overhead asset is a particular type of overhead asset. However, Nguyen discloses wherein identifying the overhead asset comprises determining a likelihood that the overhead asset is a particular type of overhead asset (pg. 14, section 4, para. 1, “YOLO(You Only Look Once) is a real-time object detection framework that directly predicts bounding boxes and class probabilities”; and pg. 16, section C and Algorithm 1, wherein the classification and defect detection are both probability-based determinations of class). Thus, it would have been obvious to one having ordinary skill in the art prior to the effective filing date of the claimed invention to have utilized the likelihood classification method of Nguyen within the method and system of Harvey as modified by Nguyen according to the rationale of claim 4.
Regarding claim 8, Harvey discloses all limitations of claim 1.Harvey further discloses the overhead asset comprises a first overhead asset of the electrical grid (para. 0051, containing the explicit disclosure of the determination of the construct and its location within images); and updating the GIS database in response to identifying an overhead asset (paras. 0096-0097, disclosing the updating of a GIS with information on the construct upon recognition of the construct, its location, and particular information regarding its state). Harvey does not explicitly disclose identifying, using computer vision with respect to the digital images, a second overhead asset of the electrical grid. However, Nguyen discloses identifying, using computer vision with respect to the digital images, a second overhead asset of the electrical grid (pg. 14 section 4 para. 1, 16, section C, and Algorithm 1, wherein the classification and defect detection are both probability-based determinations of class, and pg. 19 fig. 7 for the visualization of the algorithm, wherein the algorithm’s use of a YOLO-mediated bounding box method allows for multiple object detection and classification, as disclosed within the algorithm and visualized via fig. 7). Thus, it would have been obvious to one having ordinary skill in the art prior to the effective filing date of the claimed invention to have utilized the multi-asset detection and inspection method of Nguyen within the method and system of Harvey as modified by Nguyen according to the rationale of claim 4.
Regarding claim 9, Harvey and Nguyen disclose all limitations of claim 8. Nguyen further discloses wherein the first and second overhead assets are different types of overhead assets (pg. 14 section 4 para. 1, 16, section C, and Algorithm 1, wherein the classification and defect detection are both probability-based determinations of class, and pg. 19 fig. 7 for the visualization of the algorithm, wherein the algorithm’s use of a YOLO-mediated bounding box method allows for multiple object detection and classification, and wherein the detection of objects of different classes is evident from the experiment of fig. 7). Thus, it would have been obvious to one having ordinary skill in the art prior to the effective filing date of the claimed invention to have utilized the multi-asset detection and classification method with different asset classes of Nguyen within the method and system of Harvey as modified by Nguyen according to the rationale of claim 4.
Regarding claim 21, Harvey discloses all limitations of claim 1. Harvey does not disclose classifying, for a first of the digital images, a background object and a foreground object, wherein the overhead asset comprises the foreground object and is on a utility pole, and wherein the background object is not on the utility pole.
However, Nguyen discloses classifying, for a first of the digital images, a background object and a foreground object, wherein the overhead asset comprises the foreground object and is on a utility pole, and wherein the background object is not on the utility pole (pg. 14 section 4 para. 1, 16, section C, and Algorithm 1, wherein the classification and defect detection are both probability-based determinations of class, and pg. 19 fig. 7 for the visualization of the algorithm, wherein the algorithm’s use of a YOLO-mediated bounding box method allows for multiple object detection and classification, and wherein the detection of objects of different classes is evident from the experiment of fig. 7; and further containing a plurality of both foreground objects and background objects (foreground being interpreted by the Examiner to mean objects nearest to the observer’s perspective), each of which is classified using YOLO, and wherein a utility pole in the background is classified separately from insulators and crossarms in the foreground) . Thus, it would have been obvious to one having ordinary skill in the art prior to the effective filing date of the claimed invention to have utilized the multi-asset detection and classification method in both the foreground and background of an image of Nguyen within the method and system of Harvey as modified by Nguyen according to the rationale of claim 4.
Regarding claim 22, Harvey and Nguyen disclose all limitations of claim 21. Harvey does not disclose wherein the background object is on another utility pole. However, Nguyen discloses wherein the background object is on another utility pole (pg. 14 section 4 para. 1, 16, section C, and Algorithm 1, wherein the classification and defect detection are both probability-based determinations of class, and pg. 19 fig. 7 for the visualization of the algorithm, wherein the objects classified within the background of the image not on the first utility pole are present on another utility pole). Thus, it would have been obvious to one having ordinary skill in the art prior to the effective filing date of the claimed invention to have utilized the multi-asset detection and classification method in both the foreground and background of an image of Nguyen within the method and system of Harvey as modified by Nguyen according to the rationale of claim 4.
Claims 6 and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Harvey in view of Wong (WIPO PG Pub 2022101882).
Regarding claim 6, Harvey discloses all limitations of claim 1. Harvey does not disclose wherein the overhead asset comprises a transmission line or a distribution line. However, Wong discloses wherein the overhead asset comprises a transmission line or a distribution line (paras. 0002 and 0030, “Powerline inspection can be a very effective method for finding defects on powerline assets such as transmission and distribution lines”; and “the system can employ one or more drones that conduct aerial inspections to detect emerging faults on transmission and distribution lines using radio frequency (RF) data collection devices, global positioning system (GPS) antennas, wireless and cellular communication systems, and high definition cameras with monocular or stereo vision”). Specifically, Wong discloses a multi-sensor method and system of drone-mediated powerline inspection through RF data measurement and image capture. Therefore, both Harvey and Wong disclose powerline inspection methods and system using at least image sensor data for defect detection and robust measurement. Thus, it would have been obvious to one having ordinary skill in the art prior to the effective filing date of the claimed invention to have applied the teachings of Wong with respect to transmission and distribution lines within the method of Harvey as a teaching or suggestion in the prior art which would have led one having ordinary skill to have modified the disclosure of Harvey to yield the predictable result of accurate inspection and defect detection on crucial powerline infrastructure.
Regarding claim 18, Harvey discloses all limitations of claim 10. Harvey does not disclose wherein classifying comprises determining that the overhead asset comprises a distribution line, and wherein identifying the attribute comprises determining one or more of the following: conductor wire code, conductor size, conductor material, and whether the distribution line is insulated or stranded. However, Wong discloses wherein classifying comprises determining that the overhead asset comprises a distribution line (paras. 0002 and 0030, “Powerline inspection can be a very effective method for finding defects on powerline assets such as transmission and distribution lines”; and “the system can employ one or more drones that conduct aerial inspections to detect emerging faults on transmission and distribution lines using radio frequency (RF) data collection devices, global positioning system (GPS) antennas, wireless and cellular communication systems, and high definition cameras with monocular or stereo vision”), and wherein identifying the attribute comprises determining whether the distribution line is insulated or stranded (para. 0031, “Examples of RF events can include punctures in the insulation of wires, conductors with broken strands”). Thus, it would have been obvious to one having ordinary skill in the art prior to the effective filing date of the claimed invention to have applied the teachings of Wong with respect to distribution lines, and the punctured insulation leading to stranded wire within the method of Harvey as a teaching or suggestion in the prior art which would have led one having ordinary skill to have modified the disclosure of Harvey to yield the predictable result of accurate inspection and defect detection on crucial powerline infrastructure.
Claims 11 and 13 are rejected under 35 U.S.C. 103 as being unpatentable over Harvey in view of Nguyen and in further view of Galli et al. (“For the Grid and Through the Grid: The Role of Power Line Communications in the Smart Grid”, hereinafter “Galli”).
Regarding claim 11, Harvey discloses all limitations of claim 10. As mentioned above, Harvey does not explicitly disclose wherein the detected and classified components of the power line are distribution line devices. However, Nguyen discloses a fine-grained classification mechanism, allowing for more granular detection and identification of powerline components of different types (pg. 14, section III subsection C, “The detected components are then classified into more fine-grained power components classes using our component classification models”). Thus, it would have been obvious to one having ordinary skill in the art prior to the effective filing date of the claimed invention to have utilized the classification model of Nguyen within the method and system of Harvey according to the rationale of claim 4.
The combination of Harvey and Nguyen fails, however, to disclose wherein the classifying comprises determining that the overhead asset comprises a switch, and wherein identifying the attribute comprises determining one or more of the following: switch normal status, switch size, switch operating mechanism, whether switch is gang-operated.
However, Galli discloses the presence, utility, and operating mechanism of switches within a power grid network (pg. 14 section B para. 1, “In the case of fault location, fault isolation and service restoration, substation IEDs must communicate with external IEDs such as switches, reclosers, or sectionalizers”; and pg. 21 section B para. 7, “points of connection are normally open but allow various configurations by the operating utility by closing and opening switches. Operation of these switches may be by remote control from a control center or by a lineman”.). Specifically, Galli analyzes the effectiveness of power line communications as a method of interconnection within smart utility grids, specifically power distribution networks. To accomplish this, Galli describes identification of a variety of power line components crucial to the functioning of a power distribution system including their electrical and topological positions within a sample powerline framework. One having ordinary skill in the art would have found it obvious that the components such as switches (and their respective topological and electrical information) could easily be integrated as classes for object detection within the method of Nguyen as integrated within the system of Harvey. Thus, it would have been obvious to one having ordinary skill in the art prior to the effective filing date of the claimed invention to have utilized the disclosure of Galli with respect to switches of a power line network within the method and system of Harvey as modified by Nguyen as a teaching in the prior art which would led one of ordinary skill in the art to have augmented Nguyen’s classification model (within the method of Harvey as modified by Nguyen) to include an additional class for switches and an associated operating mechanism attribute.
Regarding claim 13, Harvey discloses all limitations of claim 10. As mentioned above, Harvey does not explicitly disclose wherein the detected and classified components of the power line are distribution line devices. However, Nguyen discloses a fine-grained classification mechanism, allowing for more granular detection and identification of powerline components of different types (pg. 14, section III subsection C, “The detected components are then classified into more fine-grained power components classes using our component classification models”). Thus, it would have been obvious to one having ordinary skill in the art prior to the effective filing date of the claimed invention to have utilized the classification model of Nguyen within the method and system of Harvey according to the rationale of claim 4.
The combination of Harvey and Nguyen fails, however, to disclose wherein classifying comprises determining that the overhead asset comprises a capacitor, and wherein identifying the attribute comprises determining one or more of the following: capacitor switch type, capacitor SCADA type, capacitor control type, capacitor sensing phase and sensor location.
However, Galli discloses wherein an overhead asset comprises a capacitor, and wherein identifiable attribute of the capacitor is capacitor SCADA type, capacitor control type (pg. 9, section IV A, paras. 1-2, “A system conforming to the SCADA model usually comprises the following components… sets of Intelligent Electronic Devices (IEDs), and the supporting communications infrastructure that furnishes the communications between the supervisory Master and the RTUs and between the RTUs and IEDs. The IEDs usually include various types of microprocessor-based controllers of power system equipment, such as circuit breakers, transformers, and capacitor banks”). Thus, it would have been obvious to one having ordinary skill in the art prior to the effective filing date of the claimed invention to have utilized the disclosure of Galli with respect to capacitor identification and SCADA type determination within the method and system of Harvey as modified by Nguyen according to the rationale of claim 11.
Regarding claim 17, Harvey discloses all limitations of claim 10. Harvey does not explicitly disclose wherein classifying comprises determining that the overhead asset comprises a transformer. However, Nguyen discloses wherein classifying a captured overhead asset comprises determining that the overhead asset comprises a transformer (pg. 15, para. 1, “The selected component classes include… a class for transformers”). Thus, it would have been obvious to one having ordinary skill in the art prior to the effective filing date of the claimed invention to have integrated Nguyen’s disclosure of the determination of a transformer into the method and system of Harvey as modified by Nguyen according to the rationale of claim 4. The combination of Harvey and Nguyen fails to disclose wherein identifying the attribute comprises determining one of more of the following: transformer size, transformer KVA, and transformer secondary voltage. However, Galli discloses wherein the identifiable attribute of a transformer includes transformer secondary voltage (pg. 14, section B, para. 4, “voltage measurement on the secondary winding”, which would be observable and determinable). Thus, it would have been obvious to one having ordinary skill in the art prior to the effective filing date of the claimed invention to have utilized the disclosure of Galli with respect to transformer secondary voltage within the method and system of Harvey as modified by Nguyen according to the rationale of claim 11.
Claim(s) 12, 14-16 and 19-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Harvey as applied to Claim 10, above, in view of “Distribution System” (IEEEE Long Island, 2009, downloaded from https://www.ieee.li/pdf/viewgraphs/automating_power_distribution_system.pdf). Harvey describes the elements of Claim 10 as outlined above. Harvey describes identifying electrical grid infrastructure elements but does not describe certain specific components of the electrical grid including fuses (claim 12) reclosers (Claim 14) regulators (Claim 15) or sectionalizers (Claim 16). Harvey does describe (para. 69-72) that the system takes images of components of the constructs to identify and analyze the components including adverse conditions, and that this includes labeling on the components. From [0069], “Signs/tags of the components may be included with the image data/information. “. “Distribution System” describes that fuses (page 5) sectionalizers (page 11) regulators (page 12) and reclosers (page 5 and 11) are known components of the electrical grid. Harvey also describes at [0069] that “The UAV 302, or another data gathering system, may capture image data/information (e.g., static images, video, etc.) of any/all components of the one or more constructs from all angles.”
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the application, to use the system of Harvey to gather information about all of the typical components of an electrical grid shown in “Distribution Systems” on a construct including their identity, operating status, and adverse conditions affecting their performance in order to “capture image data/information (e.g., static images, video, etc.) of any/all components of the one or more constructs from all angles” as described in Harvey. With respect to Claims 19 and 20, Harvey captures data about all of the components on a construct. When Harvey identifies elements, the list of elements itself is a total number of assets. Claims 19 and 20 do not require a specific format for the number or processing to occur based on the number.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ROHAN TEJAS MUKUNDHAN whose telephone number is (571)272-2368. The examiner can normally be reached Monday - Friday 9AM - 6PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Gregory Morse can be reached at 5712723838. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ROHAN TEJAS MUKUNDHAN/Examiner, Art Unit 2663
/GREGORY A MORSE/Supervisory Patent Examiner, Art Unit 2698