DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Objections
Claim(s) 1 objected to because of the following informalities:
Claim 1 limitations: “IV) an object to digital twin matching step is performed on the server, mapping detected objects, in particular the insulators and/or insulator groups of the captured 2d images and camera geolocation and pose with in a database of the service provider previously stored digital twins of overhead power lines and pylons”, should be corrected to “IV) an object to digital twin matching step is performed on the server, mapping detected objects, in particular the insulators and/or insulator groups of the captured 2d images and camera geolocation and pose with digital twins of overhead power lines and pylons previously stored in a database of the service provider”.
Claim 1 is objected to on the basis of the terms “before” and “before a final” recited in the limitation: “III) AI-based object detection on the captured 2d images based on at least one neural network on a server resulting in detected objects, before IV) an object to digital twin matching step is performed on the server, mapping detected objects, in particular the insulators and/or insulator groups of the captured 2d images and camera geolocation and pose with in a database of the service provider previously stored digital twins of overhead power lines and pylons, before a final”. Wherein the terms inclusion within the claim limitation, while not rendering the claim indefinite, are awkwardly worded within the claim limitation. For clarity, examiner recommends amending the claim to something along the lines of : “III) AI-based object detection on the captured 2d images based on at least one neural network on a server resulting in detected objects, IV) an object to digital twin matching step is performed on the server, mapping detected objects, in particular the insulators and/or insulator groups of the captured 2d images and camera geolocation and pose with in a database of the service provider previously stored digital twins of overhead power lines and pylons, , wherein step III is performed prior to step IV, and step IV is performed prior to step V.”.
Appropriate correction is required.
Claim Rejections - 35 USC § 112
The following is a quotation of the first paragraph of 35 U.S.C. 112(a):
(a) IN GENERAL.—The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor or joint inventor of carrying out the invention.
The following is a quotation of the first paragraph of pre-AIA 35 U.S.C. 112:
The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor of carrying out his invention.
Claim 5 is rejected under 35 U.S.C. 112(a) or 35 U.S.C. 112 (pre-AIA ), first paragraph, as failing to comply with the written description requirement. The claim(s) contains subject matter which was not described in the specification in such a way as to reasonably convey to one skilled in the relevant art that the inventor or a joint inventor, or for applications subject to pre-AIA 35 U.S.C. 112, the inventor(s), at the time the application was filed, had possession of the claimed invention.
Regarding claim 5, the claim limitation “wherein the matching comprises at least: estimation of 3D suspension points on the 2D image plane, calculation of profit matrix” is asserted as failing to comply with the written description requirement because the “calculation of price matrix” would not have been clear to one of ordinary skill in the art, since the term “price matrix” is not well known or conventional to one of ordinary skill in the art, and the specification fails to disclose how the “calculation of profit matrix” is performed. For the purposes of examination, the limitation is interpreted as “wherein the matching comprises at least: estimation of 3D suspension points on the 2D image plane, calculation of a matrix”.
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1-13 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Regarding claim 1, the phrase "in particular" renders the claim indefinite because it is unclear whether the limitation(s) following the phrase are part of the claimed invention. See MPEP § 2173.05(d). For the purposes of examination, the phrase is interpreted as “including”.
Regarding claims 2-13, they are rejected under 112b for inheriting and failing to cure the deficiencies of the parent claim 1.
Claim 2 recites the limitation "wherein step (III) AI-based object detection based on at least one neural network on a server comprises at least one additional grouping process". There is insufficient antecedent basis for this limitation in the claim. For the purposes of examination, the limitation is interpreted as “wherein step (III) AI-based object detection based on at least one neural network on a server comprises at least one additional grouping process, wherein the AI-based object detection is a grouping process”.
Regarding claims 3 and 7-8, they are rejected under 112b for inheriting and failing to cure the deficiencies of the parent claim 2.
Claim 3 recites the limitation " wherein the grouping process after object detection in step (III) comprises…". There is insufficient antecedent basis for this limitation in the claim. For the purposes of examination, the limitation is interpreted as “wherein the grouping process in step (III) comprises…”.
Claim 6 recites the limitation "wherein the required points comprise the insulator end points and insulator suspension points of each installed insulator or insulator group". There is insufficient antecedent basis for this limitation in the claim. For the purposes of examination, the limitation is interpreted as "wherein the required points comprise insulator end points and insulator suspension points of each installed insulator or insulator group".
Claim 9 recites the limitation "The method for identification and subsequent inspection of overhead power lines and pylons according to claim 1, wherein the additional grouping process in the object detection is performed as follows…". There is insufficient antecedent basis for this limitation in the claim. For the purposes of examination, the claim is interpreted as depending on claim 2.
Regarding claim 10-11, they are rejected under 112b for inheriting and failing to cure the deficiencies of the parent claim 9.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claim(s) 1, 4, and 12 is/are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Xiong et al. (US-20220058591-A1) hereinafter referenced as Xiong.
Regarding claim 1, Xiong discloses: A method for identification and subsequent inspection of overhead power lines and pylons, with conductor ropes and insulators at the pylon (Xiong: Figure 3C; 0005: “The present disclosure generally relates to the detection of damage in structural assets. Non limiting examples of structural assets include crops, irrigation systems, solar panels, windmills, cell towers, and utility assets (e.g., utility towers). To provide a better understanding of the disclosed system and method, the disclosure provides non-limiting examples related to utility asset management (UAM). UAM refers to the process of identifying and maintaining various remote assets held by a utility industry, company, or other such service provider. For example, electrical utility assets typically include utility-size transformers, insulators, arrestors, bushing, switches, capacitors, fuses, poles, manholes, vaults, etc. ”), the method comprising:
I) image acquisition with drones or helicopters, capturing 2d images of the overhead power lines and pylons, in particular of the insulators and/or insulator groups, including camera geolocation and pose (Claim limitation is interpreted according to the rejection of claim 1 under 35 U.S.C. 112b disclosed above) (The limitation’s use of “and/or” is interpreted as disjunctive “or”, thus indicating that only one limitation is required.) (Xiong: 0023-0024: “the imagery obtained and processed by the system will be collected by aerial vehicles (AVs). For purposes of this disclosure, AVs refer to any kind of plane, helicopter, drone, or other flying vehicles…the AV will include one or more sensors configured to gather data associated with a mission. The sensor(s) may include a variety of types of sensors that may be categorized as sight sensors, sound sensors, touch sensors, smell sensors, position sensors, external communication sensors, and other (e.g., miscellaneous sensors). The sight sensors may include sensors for ascertaining light intensity, color, distance (e.g., by infrared (IR), measuring angle of light bounce), video capture, rotation (e.g., optical encoders), and/or light signal read (e.g., infrared codes), and may include LIDAR, digital cameras, infrared cameras, and filters/preprocessing devices…The position sensors may include sensors (e.g., accelerometer, digital compass, gyroscope) for ascertaining location (e.g., based on global positioning system (GPS), proximity to a beacon, etc.), and/or tilt.”)
II) forwarding captured 2d images and camera geolocation and pose to a server (Xiong: Figure 2; 0030: “ In a first phase 210 , the system environment manages the pipeline across multiple use cases. For each use case (e.g., a gas, electricity, or other utility provider) the system engages in a data collection about an entity's assets related to respective use cases. The information can be collected by, for example, by UAVs via various sensors, such as LIDAR, digital image devices, and infrared cameras. The collected data is then filtered and preprocessed during a second phase 220 . During a third stage 230 , a ground station unit or preprocessing server performs its analysis on the potential faults.”),
III) AI-based object detection on the captured 2d images based on at least one neural network on a server resulting in detected objects (Xiong: 0028: “During third stage 130 , the AI system receives the pre-processed data and runs a quality check on the selected images and data. If any data is deemed deficient (i.e., unable to use in analysis), the AI model will reject and remove this data. Furthermore, the AI model includes an asset detection system that implements deep learning models for identifying specific asset component types, e.g., transformers, insulators, and bushing at 150 .”;
0031: “the artificial intelligence system 232 is configured to identify the asset type(s) within images of the composite structure, as well as run a classification model to identify the type(s) of damage. In other words, the AI system first classifies the images into different assets, and then, for each asset type detected…the classification process is configured to, among other thing, execute one or more image classification models built using Deep Learning Open Source Frameworks such as Python callable libraries and other Machine Learning Applications, including Convolution Neural Networks (CNNs) using TensorFlow”), before
IV) an object to digital twin matching step is performed on the server, mapping detected objects, in particular the insulators and/or insulator groups of the captured 2d images and camera geolocation and pose with in a database of the service provider previously stored digital twins of overhead power lines and pylons (Claim limitation is interpreted according to the rejection of claim 1 under 35 U.S.C. 112b disclosed above) (The limitation’s use of “and/or” is interpreted as disjunctive “or”, thus indicating that only one limitation is required.) (Xiong: Figure 5A; 0044-0045: “metadata features 522 are concatenated with the features extracted from the images (i.e., the output from feature extraction module 520 ) through a fully connected layer 524 and then the same is processed through a final Softmax layer that represents the total feature set that is used to train the model to identify specific asset types and whether or not the specific asset types in each image are damaged…
Then, the model can be used to identify specific asset types and whether or not the specific asset types in each image are damaged. In other words, the identifying features extracted by the feature extraction model may be concatenated together with the metadata features through a fully connected layer and finally processed through a Softmax layer to identify the structural assets shown in each image…
in order to ensure a single asset is accurately tracked across multiple images of the asset, an asset matching model 560 can also be incorporated by the system. Asset matching model 560 is configured to receive the concatenated data layer and track the asset across the set of images taken from different angles. For example, an asset similarity, or match score, may be computed for a set of K images l1 , I2 , . . . , IK and a set of K assets A1 , A2 , . . . , AK which are present in all of the images. To compute this score each pair of images within the set of images I1 , I2 , . . . , Ik may compared with each other to determine the similarity for each asset appearing in each of the pair of images. When a pair of images containing assets have a high similarity (e.g., at or above a threshold of 98%), the assets having the similarity are considered to be the same asset. For example, if the similarity score for an asset shown in I1 has a similarity of 99% with an asset shown in I2 , it may be determined that images I1 and I2 show the same asset A1 . The output from asset matching model 560 includes assets and the images showing the assets. Using the previous example, the output could identify that AI appears in both images I1 and I2 .”; Wherein asset matching is performed after asset detection), before a final
V) defect detection of overhead power lines and pylons on the captured 2d images and mapping of detected defects to the digital twin in the database of digital twins is performed either by a person and/or software-supported, and the detected defects are stored in the database of the service provider and/or are transmitted to the service provider and/or network operator for repair (The limitation’s use of “and/or” is interpreted as disjunctive “or”, thus indicating that only one limitation is required.) (Xiong: 0006: “the proposed systems and methods describe an autonomous asset detection system that leverages artificial intelligence (AI) models for three-dimensional asset identification and damage detection, asset damage classification, automatic in-field asset tag readings, and real-time asset management…
The system then detects whether the asset is damaged and, if so, determine the type of damage, and further captures and stores asset tag information for the target asset. The collected and processed data is then provided to end-users via a comprehensive user interface platform for managing the assets in real-time. The platform can also be configured to generate insights on damage and repair, facilitating an expedited and thoughtful response to asset maintenance. This approach can be used to remotely conduct asset reviews and address asset issues before catastrophic failure and streamline the inspection process.”;
0046: “The output from asset matching model 560 is passed thru to an asset damage status assignment model 570 , which produces a final asset determination and a binary damage detection decision (e.g., damaged/not damaged, good/bad, etc.)…Damage status assignment model 570 analyzes the output from asset matching model 560 to determine whether any assets are damaged or not and then applies the damage status (e.g., damaged/not damaged, good/bad, etc.) to the other images containing the same assets, as determined by asset matching model 560.”).
Regarding claim 4, Xiong discloses: The method for identification and subsequent inspection of overhead power lines and pylons according to claim 1, wherein the image acquisition captures one or multiple images at one or multiple locations and/or camera settings, wherein the locations and camera settings are calculated from previously captured 2d images, geolocation and pose, detected objects and/or database of the stored digital twins (The limitation’s use of “and/or” is interpreted as disjunctive “or”, thus indicating that only one limitation is required.)
(Xiong: 0027: “During the first stage 110 , the AV(s) captures data around towers and power line paths, among other assets. The AV will be able to expeditiously track and collect information for a large volume of assets, and link each image to its geographic location. In some embodiments, the AV will arrive near or around a target asset and capture multiple images at a variety of positions having different angles and/or distances with respect to the target asset. Images taken at a variety of positions can help provide views of assets from different perspectives, such that identifying features or damage hidden in one view will appear in another view.”; Wherein the capturing of multiple images around a target at a variety of positions, distances, and angles, constitutes the calculation of locations and camera settings based on previously captured images.).
Regarding claim 12, Xiong discloses: The method for identification and subsequent inspection of overhead power lines and pylons according to claim 1, wherein the AI-based object detection on the captured 2d images based on at least one neural network is a convolutional neural network architecture for object detection and/or object segmentation, which is trained on a dataset of training images to detect and/or segment overhead power lines and pylons with conductor ropes, insulators and/or
insulator groups (The limitation’s use of “and/or” is interpreted as disjunctive “or”, thus indicating that only one limitation is required.)
(Xiong: 0031: “the AI system first classifies the images into different assets…the AI system is developed, trained, tuned, and evaluated using customized models making use of one or more data science tools for model integration and simulation. In some embodiments, the classification process is configured to, among other thing, execute one or more image classification models built using Deep Learning Open Source Frameworks such as Python callable libraries and other Machine Learning Applications, including Convolution Neural Networks (CNNs) using TensorFlow,”;
0038: “in the third example of FIG. 3C, the system has implemented an insulator and arrestor detection and damage model to examine the utility tower 302.”).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 2-3 and 7-8 is/are rejected under 35 U.S.C. 103 as being unpatentable over Xiong, and further in view of Zheng et al. (Insulator-Defect Detection Algorithm Based on Improved YOLOv7) hereinafter referenced as Zheng.
Regarding claim 2, Xiong discloses: The method for identification and subsequent inspection of overhead power lines and pylons according to claim 1.
Xiong does not disclose expressly: wherein step (III) AI-based object detection based on at least one neural network on a server comprises at least one additional grouping process (Claim limitation is interpreted according to the rejection of claim 2 under 35 U.S.C. 112b disclosed above).
Zheng discloses: A method for detecting insulator defects within an image based on target box clustering. Wherein the anchor boxes are adjusted based on a Kmeans++ algorithm (Zheng: 1. Introduction: “this paper proposes an improved insulator-defect detection method based on the YOLOv7 algorithm. Firstly, to improve the accuracy and efficiency of detection, this paper uses the anchor box size obtained from the K-means++ clustering insulator dataset to replace the default anchor box size of YOLOv7…The SIoU regression loss function and focal loss classification function are introduced to improve the network convergence speed and detection efficiency and solve the dataset’s sample imbalance problem. Finally, SIoU-NMS is used to implement a new non-maximum suppression process to reduce the problem of false detection of insulators and insulator defects.”), wherein the detected objects in the anchor boxes, which are grouped by size, are classified as either pollution, damage, or insulator (Zheng: 2.1.3. Image Database and Label Database: “At the same time, the Labelme tool is used to label the ground-truth box of the image, and the label category is divided into pollution flashover, damage, and insulator… In Figure 3c, the abscissa width is the ratio of the label width to the image width, and the ordinate height is the ratio of the label height to the image height. The dataset contains data of various sizes, mainly small and medium target data, which is more suitable for the actual situation.”).
Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to substitute the asset classification model disclosed by Xiong with the proposed YOLOv7 model taught by Zheng, in order to group detected insulators by size. The suggestion/motivation for doing so would have been “the CoordAtt attention mechanism and HorBlock module are integrated into the original backbone network to enhance the network’s ability to extract image features and increase the network’s detection accuracy for small insulator defect targets. The SIoU regression loss function and focal loss classification function are introduced to improve the network convergence speed and detection efficiency and solve the dataset’s sample imbalance problem. Finally, SIoU-NMS is used to implement a new non-maximum suppression process to reduce the problem of false detection of insulators and insulator defects. Experimental results show that the improved network has a better detection effect on insulator defects in complex environments.” (Zheng: 1. Introduction). Further, one skilled in the art could have substituted the elements as described above by known methods with no change in their respective functions, and the substitution would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Xiong with Zheng to obtain the invention as specified in claim 2.
Regarding claim 3, Xiong in view of Zheng discloses: The method for identification and subsequent inspection of overhead power lines and pylons according to claim 2, wherein the grouping process after object detection in step (III) comprises a multiplicity of different processing steps for obtaining the group of imaged insulators from the captured 2d images (Claim limitation is interpreted according to the rejection of claim 3 under 35 U.S.C. 112b disclosed above) (Zheng: Abstract: “Firstly, the target boxes of the insulator dataset are clustered based on K-means++ to generate more suitable anchor boxes for detecting insulator-defect targets. Secondly, the Coordinate Attention (CoordAtt) module and HorBlock module are added to the network. Then, in the channel and spatial domains, the network can enhance the effective features of the feature-extraction process and weaken the ineffective features. Finally, the SCYLLA-IoU (SIoU) and focal loss functions are used to accelerate the convergence of the model and solve the imbalance of positive and negative samples.”).
Regarding claim 7, Xiong in view of Zheng discloses: The method for identification and subsequent inspection of overhead power lines and pylons according to claim 2, wherein the additional grouping process of found insulators is done with a multi-layer approach with at least three layers, namely a small layer, a medium layer and a large layer, wherein each found insulator is assigned to at least one of the layers, based on the bounding box size of the found insulator (Zheng: 2.3. Anchor-Box Optimization: “YOLOv7 uses the K-means algorithm to cluster the anchor boxes obtained from the COCO dataset by default and uses the genetic algorithm to adjust the anchor boxes during the training process…this paper uses the K-means++ algorithm [27] to alleviate this problem and improve the accuracy and efficiency of detection…
(2) Calculate the probability 𝑂(𝑥) that each insulator sample box is selected as the next cluster center and use the roulette method to select the next cluster center…
(4) Calculate the distance from each sample in the dataset to the cluster centers, divide the sample into the class corresponding to the cluster center with the smallest distance, and recalculate the cluster center of each category 𝑐𝑖. Update the classification and cluster center repeatedly until the anchor box size remains unchanged…
Since the detection model in this paper contains three detection feature maps, and each feature map corresponds to three anchor boxes…
The small-sized anchor box corresponds to the final output of the 80 × 80 feature map, which is responsible for detecting small-sized objects. The medium-sized anchor box corresponds to a feature map with a size of 40 × 40, which is responsible for detecting medium-sized objects. The large-sized anchor box corresponds to a feature map with a size of 20 × 20, which is responsible for detecting large objects in the image.”).
Regarding claim 8, Xiong in view of Zheng discloses: The method for identification and subsequent inspection of overhead power lines and pylons according to claim 2, wherein the additional grouping process comprises three steps: a layer-independent preprocessing (Zheng: 2.3. Anchor-Box Optimization: “YOLOv7 uses the K-means algorithm to cluster the anchor boxes obtained from the COCO dataset by default and uses the genetic algorithm to adjust the anchor boxes during the training process…this paper uses the K-means++ algorithm [27] to alleviate this problem and improve the accuracy and efficiency of detection…
(4) Calculate the distance from each sample in the dataset to the cluster centers, divide the sample into the class corresponding to the cluster center with the smallest distance, and recalculate the cluster center of each category 𝑐𝑖. Update the classification and cluster center repeatedly until the anchor box size remains unchanged”), a per-layer processing (Zheng: 2.4. Backbone Network: “After the PAFPN network, the network’s output is three layers of feature maps of different sizes. Finally, the network outputs the prediction results through the RepC and Conv modules in the head part.”) and a layer-independent postprocessing, wherein each step comprises one or multiple computational steps (Zheng: 2.6. Non-Maximum Suppression: “The improvement of this paper is to introduce target scale and distance into the consideration of IOU, and use SIoU to calculate the IOU values of the candidate box with the highest confidence and all other boxes to determine which box to delete. It can solve the problem that of the insulator shield being too close to the insulator. The improved non-maximum suppression algorithm of SIoU-NMS is used to filter the preliminary prediction box of the image output to be recognized, and the final prediction box is obtained by the following steps”).
Claim(s) 5 is/are rejected under 35 U.S.C. 103 as being unpatentable over Xiong, and further in view of Huang et al. (A Model-Driven Method for Pylon Reconstruction from Oblique UAV Images) hereinafter referenced as Huang.
Regarding claim 5, Xiong discloses: The method for identification and subsequent inspection of overhead power lines and pylons according to claim 1, wherein the matching comprises at least: performing an assignment algorithm to identify insulators and insulator groups and therewith digital twins (Xiong: 0045: “in order to ensure a single asset is accurately tracked across multiple images of the asset, an asset matching model 560 can also be incorporated by the system. Asset matching model 560 is configured to receive the concatenated data layer and track the asset across the set of images taken from different angles. For example, an asset similarity, or match score, may be computed for a set of K images l1 , I2 , . . . , IK and a set of K assets A1 , A2 , . . . , AK which are present in all of the images. To compute this score each pair of images within the set of images I1 , I2 , . . . , Ik may compared with each other to determine the similarity for each asset appearing in each of the pair of images. When a pair of images containing assets have a high similarity (e.g., at or above a threshold of 98%), the assets having the similarity are considered to be the same asset.”).
Xiong does not disclose expressly:, wherein the matching comprises at least: estimation of 3D suspension points on the 2D image plane and calculation of profit matrix (Claim limitation is interpreted according to the rejection of claim 5 under 35 U.S.C. 112a disclosed above).
Huang discloses: wherein the matching comprises at least: estimation of 3D suspension points on the 2D image plane (Huang: 4. Efficient Pylon Detection: “In this algorithm, the 2D line segments are firstly extracted using the LSD method. In addition, gradient symmetry of the line segment is employed to filter the amount of line segments from the natural background. After that, the rest of the line segments are used to calculate the valid intersection points, which can then be clustered to obtain the proposed regions. Then, the DPM method is used to detect pylons in the proposed regions.”), calculation of a matrix (Huang: 5.2.1. Pylon Body Matching Based on a Priori Shape and Coplanar Constraint: “The IDSC method can identify the pylon type and meanwhile the matched sampling points on shape contours can be used to find the correlation of pylon 3D legs and extracted 2D line segments from different visible images. The pylon body line segment matching algorithm includes two steps. First, the four principle legs are matched using the correlation of pylon legs and the prior information regarding pylon body structures. Second, the parameters of the four side planes of the pylon body are fitted by the four matched legs and applied to match the rest of the line segments on the pylon body…
As the projection matrix 𝑃𝑖 is known, the line segments from different images associated with a same 3D line segment of pylon legs in 𝐿𝑀𝑇 can be triangulated to compute the 3D line segment coordinates of pylon legs using an epipolar-guided line matching algorithm similar to that used in [14,30].”).
Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to incorporate the procedures for pylon detection and pylon reconstruction taught by Huang in order to perform the asset matching disclosed by Xiong. The suggestion/motivation for doing so would have been “The DPM method is applied to detect the pylon only in these proposed regions, instead of the whole image region, which improves the efficiency of pylon detection and saves memory. Once the shape contours of the pylons in the images are extracted, the IDSC method is employed to identify the pylon type, and meanwhile, for the pylon legs, the correlation of the extracted 2D line segments in the different images and the 3D line segments of the pylon model is confirmed in order to recover the 3D line coordinates” (Huang: 8. Conclusions). Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Xiong with Huang to obtain the invention as specified in claim 5.
Claim(s) 6 is/are rejected under 35 U.S.C. 103 as being unpatentable over Xiong, and further in view of Snower et al. (15 Keypoints Is All You Need) hereinafter referenced as Snower.
Regarding claim 6, Xiong discloses: The method for identification and subsequent inspection of overhead power lines and pylons according to claim 1, wherein points per digital twin of each pylon are required for the matching step (Xiong: 0045: “in order to ensure a single asset is accurately tracked across multiple images of the asset, an asset matching model 560 can also be incorporated by the system. Asset matching model 560 is configured to receive the concatenated data layer and track the asset across the set of images taken from different angles. For example, an asset similarity, or match score, may be computed for a set of K images l1 , I2 , . . . , IK and a set of K assets A1 , A2 , . . . , AK which are present in all of the images. To compute this score each pair of images within the set of images I1 , I2 , . . . , Ik may compared with each other to determine the similarity for each asset appearing in each of the pair of images. When a pair of images containing assets have a high similarity (e.g., at or above a threshold of 98%), the assets having the similarity are considered to be the same asset. For example, if the similarity score for an asset shown in I1 has a similarity of 99% with an asset shown in I2 , it may be determined that images I1 and I2 show the same asset A1 . The output from asset matching model 560 includes assets and the images showing the assets. Using the previous example, the output could identify that A1 appears in both images I1 and I2.”), wherein the required points comprise the insulator end points and insulator suspension points of each installed insulator or insulator group (Claim limitation is interpreted according to the rejection of claim 1 under 35 U.S.C. 112b disclosed above) (Xiong: 0031: “the artificial intelligence system 232 is configured to identify the asset type(s) within images of the composite structure, as well as run a classification model to identify the type(s) of damage. In other words, the AI system first classifies the images into different assets, and then, for each asset type detected, the AI system classifies images of the assets by type of damage. Some non-limiting examples of labels that may be assigned to an asset include “INSULATOR_OK”, “LOOSE_KEY”, “FLASHED_INSULATOR””; Wherein the asset identification includes insulators, which comprise the insulator endpoints and insulator suspension points.).
Xiong does not disclose expressly: wherein less than 20 points per digital twin of each pylon are required for the matching step.
Snower discloses: A method for tracking and matching a person and their pose across video frames, based on keypoint extraction and tracking (Snower: Abstract). Wherein less than 20 points per person are required for the matching step (Snower: Section: 3.2. Pose Entailment: “we seek to classify whether a pose in a timestep pt-δ, i.e. the premise, and a pose in timestep pt, i.e. the hypothesis, are the same person. To solve this problem, instead of using visual feature based similarity that incurs large computational cost, we use the set of human key points, K, detected by our pose estimator. It is computationally efficient to use these as there are a limited number of them (in our case |K|= 15), and they are not affected by unexpected visual variations such as lighting changes in the tracking step.”).
Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to incorporate the algorithms for assignment and extraction of the unique keypoints disclosed by Snower in order to perform the pylon asset matching disclosed by Xiong. The suggestion/motivation for doing so would have been “It is computationally efficient to use these as there are a limited number of them (in our case |K|= 15), and they are not affected by unexpected visual variations such as lighting changes in the tracking step” (Snower: Section: 3.2. Pose Entailment). Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Xiong with Snower to obtain the invention as specified in claim 6.
Claim(s) 13 is/are rejected under 35 U.S.C. 103 as being unpatentable over Xiong, and further in view of Alahyari et al. (Segmentation and Defect Classification of the Power Line Insulators: A Deep Learning-based Approach) hereinafter referenced as Alahyari.
Regarding claim 13, Xiong discloses: The method for identification and subsequent inspection of overhead power lines and pylons according to claim 1, wherein the defect detection of overhead power lines and pylons on the captured 2d images is based on at least one neural network (Xiong: 0031: “the AI system is developed, trained, tuned, and evaluated using customized models making use of one or more data science tools for model integration and simulation. In some embodiments, the classification process is configured to, among other thing, execute one or more image classification models built using Deep Learning Open Source Frameworks such as Python callable libraries and other Machine Learning Applications, including Convolution Neural Networks (CNNs) using TensorFlow”;
0046: “The output from asset matching model 560 is passed thru to an asset damage status assignment model 570 , which produces a final asset determination and a binary damage detection decision (e.g., damaged/not damaged, good/bad, etc.). ”).
Xiong does not disclose expressly: wherein the defect detection of overhead power lines and pylons on the captured 2d images is based on at least one neural network, wherein at least one of the neural networks is trained on a multitude of 2d images of objects without defects and therefore detects anomalies.
Alahyari discloses: wherein at least one of the neural networks is trained on a multitude of 2d images of objects without defects and therefore detects anomalies (Alahyari: Abstract: “Thus, in this study, we introduce a two-stage model that segments insulators from their background to then classify their states based on four different categories, namely: healthy, broken, burned/corroded and missing cap.”;
III. DATASET PREPARATION: “In order to mimic the real-world behavior of how data is actually captured for analysis through the UAVs, we utilized directly the high-quality videos given by UAVs. These videos are taken by several companies throughout the world and are publicly accessible [8], [18]. This gives an advantage to our model. When training is completed, the network can recognize several types of insulators utilized in different locations and companies all over the world…We have three types of defects: missing cap, broken cap, and burned cap samples of which are depicted in Fig. 2. We also have separate types of augmentation applied for these images to have a sufficient number of images for this task. The number original images and augmented are specified in Table I”).
Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to train the asset damage status assignment model disclosed by Xiong with the training dataset as taught by Alahyari. The suggestion/motivation for doing so would have been “In order to mimic the real-world behavior of how data is actually captured for analysis through the UAVs, we utilized directly the high-quality videos given by UAVs…This gives an advantage to our model. When training is completed, the network can recognize several types of insulators utilized in different locations and companies all over the world” (Alahyari: III. DATASET PREPARATION). Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Xiong with Alahyari to obtain the invention as specified in claim 13.
Allowable Subject Matter
The following is a statement of reasons for the indication of allowable subject matter:
Claims 9-11 would be allowable if rewritten to overcome the rejection(s) under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), 2nd paragraph, set forth in this Office action and to include all of the limitations of the base claim and any intervening claims.
With respect to claim 9, in addition to other limitations in the claims the Prior Art of Record fails to teach, disclose or render obvious the applicant’ s invention as claimed, in particular the limitations:
“wherein the additional grouping process in the object detection is performed as follows:
A) layer-independent preprocessing…generate binary mask for each found insulator,
B) per-layer processing: apply at least one morphological operation on each layer independently, discover individual regions and individually enclosed by its convex hull on each layer, check for misclassifications between adjacent layers and eliminate found misclassifications by assigning misclassified insulators and insulator groups to a different layer
C) layer-independent post-processing: Check for misclassification in the border regions of the 2d images and eliminate found misclassifications by assigning misclassified insulators and insulator groups to a different layer,”
Xiong in view of Zheng discloses: The method for identification and subsequent inspection of overhead power lines and pylons according to claim 1, wherein the additional grouping process in the object detection is performed as follows:
A) layer-independent preprocessing: assign cluster bounding boxes into different layers using the k-means clustering algorithm (Zheng: 2.3. Anchor-Box Optimization: “YOLOv7 uses the K-means algorithm to cluster the anchor boxes obtained from the COCO dataset by default and uses the genetic algorithm to adjust the anchor boxes during the training process…this paper uses the K-means++ algorithm [27] to alleviate this problem and improve the accuracy and efficiency of detection…
Since the detection model in this paper contains three detection feature maps, and each feature map corresponds to three anchor boxes…
The small-sized anchor box corresponds to the final output of the 80 × 80 feature map, which is responsible for detecting small-sized objects. The medium-sized anchor box corresponds to a feature map with a size of 40 × 40, which is responsible for detecting medium-sized objects. The large-sized anchor box corresponds to a feature map with a size of 20 × 20, which is responsible for detecting large objects in the image.”),
C) layer-independent post-processing: Check for misclassification in the border regions of the 2d images and eliminate found misclassifications, remove insulator groups with a single insulator when their confidence score is below a threshold, calculate bounding box and center point of each insulator group (Zheng: 2.5. Loss Function: “As shown in Figure 8a, 𝐶𝑤 and 𝐶ℎ are the width and height of the rectangle constructed diagonally by connecting the center points of the two boxes, 𝐶𝑤1 and 𝐶ℎ1 are the width and height of the minimum bounding rectangle of the two boxes, 𝑤𝑔𝑡 and ℎ𝑔𝑡 are the width and height of the ground truth box, and w and h are the width and height of the predicted box. 𝛼 is the included angle between the line connecting the center point of the two boxes and the x-axis, and 𝛽 is the included angle between the diagonal of the two boxes’ center points and the y-axis. 𝜃 is an adjustable variable, which indicates how much weight the network gives to the shape loss. The schematic diagram of IoU calculation is shown in Figure 8b, which calculates the ratio of the intersection and union of the ground-truth box and the predicted box.”;
2.6. Non-Maximum Suppression: “The improved non-maximum suppression algorithm of SIoU-NMS is used to filter the preliminary prediction box of the image output to be recognized, and the final prediction box is obtained by the following steps:
(1) Set the confidence threshold and SIoU threshold;
(2) Calculate the confidence level of all preliminary prediction boxes output by the network model, put the preliminary prediction boxes whose confidence level is higher than the confidence threshold in the candidate list, and sort the preliminary prediction boxes in descending order of confidence from high to low in the candidate list;
(3) Take the initial prediction box with the highest confidence from the candidate list, save it to the output list, and delete the initial prediction box from the candidate list;
(4) Calculate the cross-merger loss of the initial prediction box with the highest confidence obtained in the previous step and all the other preliminary prediction boxes in the candidate list, and delete the initial prediction box whose cross-merger loss is higher than the set SIoU threshold from the candidate list;
(5) Repeat Steps 3 and 4 until the candidate list is empty;
(6) Use the preliminary prediction box in the output list as the final prediction box.”; Wherein bounding boxes and center points of each insulator are calculated for SIoU-NMS determination.).
Xiong in view of Zheng fails to disclose: wherein the additional grouping process in the object detection is performed as follows:
A) layer-independent preprocessing: generate binary mask for each found insulator,
B) per-layer processing: apply at least one morphological operation on each layer independently, discover individual regions and individually enclosed by its convex hull on each layer, check for misclassifications between adjacent layers and eliminate found misclassifications by assigning misclassified insulators and insulator groups to a different layer
C) layer-independent post-processing: Check for misclassification in the border regions of the 2d images and eliminate found misclassifications by assigning misclassified insulators and insulator groups to a different layer.
Therefore, claim 9 is allowable subject matter.
Regarding claims 10-11, the claims are allowed due to their dependence on parent claim 9.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ANTHONY J RODRIGUEZ whose telephone number is (703)756-5821. The examiner can normally be reached Monday-Friday 10am-7pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Sumati Lefkowitz can be reached at (571) 272-3638. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ANTHONY J RODRIGUEZ/Examiner, Art Unit 2672
/SUMATI LEFKOWITZ/Supervisory Patent Examiner, Art Unit 2672
Prosecution Insights