DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
This is a Final Office Action on the merits. Claims 1-20 are currently pending and are addressed below.
Response to Amendment
The drawings were objected to due to minor informalities. Applicant amended the drawings accordingly; therefore, the drawings objection is withdrawn.
Claims 4 and 13 were objected to due to minor informalities. Applicant amended the claims accordingly; therefore, the objection is withdrawn.
Claims 4 and 10-19 were rejected under 35 U.S.C. 112 as being indefinite. Applicant amended claims 4, 13, and 19 accordingly; therefore, the rejection is withdrawn.
Response to Arguments
Applicant’s arguments on pgs. 13-16 of the response with respect to the rejection of claims 10-18 under 35 U.S.C. 112 have been fully considered and are persuasive; therefore, the rejection is withdrawn.
Applicant’s arguments on pgs. 17-18 of the response, with respect to the rejection(s) of claim(s) 1-5, 7-14, and 16-20 under 35 U.S.C. 102 and have been fully considered and are persuasive. Therefore, the rejection has been withdrawn. However, upon further consideration, a new ground(s) of rejection is made in view of Koyama.
Applicant’s arguments on pages 18-20 of the response, with respect to the rejection(s) of claim(s) 12 under 35 U.S.C. 102 have been fully considered but they are not persuasive. Applicant argues that “Li’s Sub-step S75 does not teach, either expressly or inherently, the claimed sub-area-based deletion operation” because “Li’s sub-step S72 is performed after all inspection data has been collected” and “the method described by claim 3 is performed before adding newly detected inspection targets”. However, the arguments are not directed to the claim as written because neither claim 12 nor its parent claim recites deleting a second inspection target and a geographical position of the second inspection target “before adding newly detected inspection targets for the current sub-area”.
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action.
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because the claim limitation(s) uses a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier. Such claim limitation(s) is/are:
the inspection device in claims 1-20
the target detection module in claims 10-18
the target display module in claims 10-18
Because this/these claim limitation(s) is/are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, it/they is/are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof.
If applicant does not intend to have this/these limitation(s) interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation(s) to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation(s) recite(s) sufficient structure to perform the claimed function so as to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-5, 7-14, and 16-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Li of CN 113286129 A, published 07/23/2021, hereinafter “Li”, in view of Koyama of US 20240149909 A1, filed 05/16/2022, hereinafter “Koyama”.
Regarding claim 1, Li teaches:
A method for detecting and positioning an inspection target, comprising: detecting a first inspection target and a first geographical position of the first inspection target in a target inspection sub-area of an inspection device, (See at least [n0031-n0035]: “Step S4: Perform defect detection on the infrared image to obtain the defect type and the location bounding box of the defect on the infrared image, and obtain the center coordinates of the location bounding box…Step S6: Transform the center position coordinates mapped onto the wide-angle image to the world coordinate system to obtain the defect geographic coordinates.” See also [0059-0062] regarding obtaining the defect geographic coordinates.)
the target inspection sub-area being a sub-area inspected by the inspection device during a process of performing an inspection task; and (See at least [n0020]: “…During the inspection process, the inspection area is divided into multiple sub-areas…”)
displaying the first inspection target at the first geographical position in an inspection map (See at least [0063]: “The reporting module is used to project the geographic coordinates of each defect after weight reduction processing onto the website image, and generate an inspection report based on the defect type.”)
However, Li does not explicitly teach displaying the inspection target during the process of performing the inspection task.
Koyama teaches a monitoring map that displays a monitoring area divided into a grid of cells via a display device and decreasing a monitoring time interval for each cell as time passes, such that “continuous monitoring is automatically executed” (See at least Figs. 2-3, [0045], [0067] & [0080]).
One having ordinary skill in the art, before the effective filing date of the claimed invention, would have found it obvious to combine Li’s method with Koyama’s technique of displaying the monitoring area during the process of performing the inspection task. Doing so would be obvious since “As a result, only an area that is actually monitored can be reliably managed as monitored, so that it is possible to prevent a situation where the user intends to perform monitoring but is not actually able to, thereby improving a monitoring accuracy” (See [0093] of Koyama).
Regarding claim 2, Li and Koyama in combination teach all the limitations of claim 1 as discussed above.
Li additionally teaches:
wherein the displaying the first inspection target at the first geographical position in an inspection map comprises: adding the first inspection target and the first geographical position to an inspection target set, the inspection target set comprising all inspection targets and geographical positions of all the inspection targets that have been detected by the inspection device during the process of performing an inspection task; and (See at least [n0054-n0055]: “The above calculations can be used to statistically identify the type, location, and number of defects, and generate an inspection report. At this point, the inspection report mainly reflects the results of the drone inspection, specifically including the types of defects, locations and numbers of defects, etc., as mentioned above. It may also include the proportion of defect types, inspection mileage, and geographical coordinates of defects…”. See also [n0056] regarding the inspection report.)
displaying all the inspection targets in the inspection target set at corresponding geographical positions of all the inspection targets in the inspection map. (See at least [n0052-n0053]: “Step S8: Project the geographic coordinates of each defect after weight reduction onto the site image, and generate an inspection report based on the defect type. Optionally, projecting the geographic coordinates of each defect after deduplication onto the website image includes: obtaining the TFW file used during website construction. The TFW file contains the geographic coordinates of the top left and bottom right corners of the website image in the world coordinate system, as well as the pixel lengths of the website image in both the horizontal and vertical directions…”)
Regarding claim 3, Li and Koyama in combination teach all the limitations of claim 2 as discussed above.
Li additionally teaches:
wherein before the adding the first inspection target and the first geographical position to an inspection target set, the method further comprises: obtaining the target inspection sub-area; and (See at least [n0026]: “…step S2 specifically includes: dividing the inspection range into multiple regional subarrays based on the actual terrain of the photovoltaic power station and the difficulty of drone inspection…”. See also [n0052], in which step s2 occurs prior to step s8 of generating the inspection report.)
Koyama additionally teaches:
deleting a second inspection target and a geographical position of the second inspection target from the inspection target set, the second inspection target being an inspection target in the inspection target set and located in the target inspection sub-area. (See at least Figs. 2A-2B & [0042]: “…For example, anon-monitoring target area 101 within the monitoring area 100 is an area to be excluded from the monitoring target. In such a case, a corresponding area is also excluded from the monitoring target in the monitoring map 200 (a shaded area 201)…” & [0076]: “For example, the monitored area identification unit 13 may further determine, in addition to the positional information, whether a monitoring target area is shown in an image on the basis of the image acquired in step S18, and exclude cells for which a predetermined number or more of images that do not contain information necessary for monitoring have been photographed among cells related to the actual movement route from the monitored area. For example, when an image where the monitoring target cannot be photographed because an obstacle blocks a field of view of the camera, or the monitoring target cannot be photographed because it is covered by a structure, and the like is photographed a certain percentage or more while a vehicle moves in a certain area, the monitored area identification unit 13 excludes a cell corresponding to that area from the monitored area…”)
Regarding claim 4, Li and Koyama in combination teach all the limitations of claim 3 as discussed above.
Li additionally teaches:
wherein before the deleting a second inspection target and a geographical position of the second inspection target from the inspection target set, the method further comprises: determining whether a second geographical position in the target inspection sub-area exists in the inspection target set; and (See at least [n0048]: “Sub-step S72: Construct an adjacency matrix between all pairs of defective geographic coordinates in the same set, and set a threshold for the measurement between two defective geographic coordinates (equivalent to setting a threshold for the values in the adjacency matrix)” & [0128]: “Sub-step S73: Determine whether the metric between the two defect geographic coordinates is less than the threshold…”)
determining, in response to the second geographical position existing in the inspection target set, that the second inspection target exists in the inspection target set; or determining, in response to no second geographical position existing in the inspection target set, that no second inspection target exists in the inspection target set. (See at least [0129]: “When the metric is less than the threshold, determine that the geographic coordinates of the two defects correspond to the same defect on the infrared image.”)
Regarding claim 5, Li and Koyama in combination teach all the limitations of claim 3 as discussed above.
Li additionally teaches:
wherein the obtaining the target inspection sub-area comprises: obtaining an inspection image obtained by the inspection device; and (See at least [n0020]: “…During the inspection process, the inspection area is divided into multiple sub-areas. A drone performs inspections according to a pre-planned route, acquiring infrared and wide-angle images…”)
determining an inspection map area corresponding to the inspection image as the target inspection sub-area. (See at least [n0026]: “…step S2 specifically includes: dividing the inspection range into multiple regional subarrays based on the actual terrain of the photovoltaic power station and the difficulty of drone inspection; exporting a kml file, which includes the world geographic coordinate range of the photovoltaic power station and each regional subarray; generating a tfw file that marks the range of each regional subarray based on the kml file, and determining the interval and unit pixel value size defined for each regional subarray.”)
Regarding claim 7, Li and Koyama in combination teach all the limitations of claim 1 as discussed above.
Li additionally teaches:
wherein the detecting a first inspection target and a first geographical position of the first inspection target in a target inspection sub-area of an inspection device comprises: performing inspection target identification on the inspection image obtained by the inspection device to obtain the first inspection target and first image coordinates of the first inspection target in the inspection image; and (See at least [n0031-n0032]: “Step S4: Perform defect detection on the infrared image to obtain the defect type and the location bounding box of the defect on the infrared image, and obtain the center coordinates of the location bounding box. Optionally, the infrared image is input into a neural network deep learning model to perform defect detection, obtain the defect type and the location bounding box of the defect on the infrared image, and obtain the center coordinates of the location bounding box. For example, the center point of the dual-light camera (the drone hovering position) is marked as a dot on the infrared image, and the location range of the defect on the infrared image is represented by a rectangle. In infrared images, different defect types can be indicated by different colors and/or shapes.”)
performing coordinate transformation on the first image coordinates to obtain first geographical position coordinates of the first inspection target, the first geographical position coordinates being coordinates in a world coordinate system. (See at least [n0035]: “Step S6: Transform the center position coordinates mapped onto the wide-angle image to the world coordinate system to obtain the defect geographic coordinates” & [n0038]: “Sub-step S62: Combining the intrinsic parameters, external attitude angles, and hovering position information of the dual-light camera, the center position coordinates mapped to the wide-angle image are transformed to the world coordinate system to obtain the geographic coordinates of the defect.”)
Regarding claim 8, Li and Koyama in combination teach all the limitations of claim 7 as discussed above. Li additionally teaches:
wherein the performing inspection target identification on the inspection image obtained by the inspection device to obtain the first inspection target and first image coordinates of the first inspection target in the inspection image comprises: performing, by means of a target identification model, inspection target identification on the inspection image obtained by the inspection device to obtain the first inspection target and the first image coordinates of the first inspection target in the inspection image. (See at least [n0032]: “…the infrared image is input into a neural network deep learning model to perform defect detection, obtain the defect type and the location bounding box of the defect on the infrared image, and obtain the center coordinates of the location bounding box…”)
Regarding claim 9, Li and Koyama in combination teach all the limitations of claim 7 as discussed above.
Li additionally teaches:
wherein the performing coordinate transformation on the first image coordinates to obtain first geographical position coordinates of the first inspection target comprises: obtaining a first device posture and first device position coordinates of the inspection device, the first device posture being a device posture when the inspection device obtains the inspection image, the first device position coordinates being geographical position coordinates when the inspection device obtains the inspection image, and the first device position coordinates being coordinates in the world coordinate system; (See at least [0022]: “Aerial triangulation is performed on the wide-angle image to obtain the external attitude angle of the UAV relative to the world coordinate system when the UAV acquires the wide-angle image” & [0026-0027]: “The process of combining the intrinsic parameters of the dual-light camera, the external attitude angle, and the hovering position information to transform the center position coordinates mapped onto the wide-angle image to the world coordinate system, thereby obtaining the defect geographic coordinates, includes: Acquire the longitude, latitude, and absolute elevation of the UAV hovering position when the wide-angle image is acquired…”. See also [0148] regarding transforming center position coordinates to the world coordinate system to obtain the defect geographic coordinates.)
determining a target transformation matrix based on the first device posture and the first device position coordinates, the target transformation matrix being used to denote a transformation relationship between a pixel coordinate system and the world coordinate system when the inspection device obtains the inspection image; and (See at least [n0008]: “…mapping the infrared image to the wide-angle image includes: selecting four marked positions from the infrared image; mapping the four marked positions to the wide-angle image to determine the parameters of the perspective transformation matrix…” & [0025-0029]: “…The coordinates, elevation information X, Y, Z, and the external attitude angles include the rotation angles around the three spatial coordinate axes X, Y, Z; The process of combining the intrinsic parameters of the dual-light camera, the external attitude angle, and the hovering position information to transform the center position coordinates mapped onto the wide-angle image to the world coordinate system, thereby obtaining the defect geographic coordinates, includes…Calculate the rotation matrix according to the order of rotation along the Y, Z, and X axes;”)
performing, based on the target transformation matrix, coordinate transformation on the first image coordinates to obtain the first geographical position coordinates. (See at least [0030]: “The geographical coordinates of the defect are calculated based on the rotation matrix and the geographical coordinates of the center point of the wide-angle image in the world coordinate system.”)
Regarding claim 10, Li teaches:
An apparatus for detecting and positioning an inspection target, comprising: a target detection module, configured to detect a first inspection target and a first geographical position of the first inspection target in a target inspection sub-area of an inspection device, (See at least [n0031-n0035]: “Step S4: Perform defect detection on the infrared image to obtain the defect type and the location bounding box of the defect on the infrared image, and obtain the center coordinates of the location bounding box…Step S6: Transform the center position coordinates mapped onto the wide-angle image to the world coordinate system to obtain the defect geographic coordinates.” See also [0059-0062] regarding obtaining the defect geographic coordinates.)
the target inspection sub-area being a sub-area inspected by the inspection device during a process of performing an inspection task; and (See at least [n0020]: “…During the inspection process, the inspection area is divided into multiple sub-areas…”)
a target display module, configured to display the first inspection target at the first geographical position in an inspection map (See at least [0063]: “The reporting module is used to project the geographic coordinates of each defect after weight reduction processing onto the website image, and generate an inspection report based on the defect type.”)
However, Li does not explicitly teach displaying the inspection target during the process of performing the inspection task.
Koyama teaches a monitoring map that displays a monitoring area divided into a grid of cells via a display device and decreasing a monitoring time interval for each cell as time passes, such that “continuous monitoring is automatically executed” (See at least Figs. 2-3, [0045], [0067] & [0080]).
One having ordinary skill in the art, before the effective filing date of the claimed invention, would have found it obvious to combine Li’s method with Koyama’s technique of displaying the monitoring area during the process of performing the inspection task. Doing so would be obvious since “As a result, only an area that is actually monitored can be reliably managed as monitored, so that it is possible to prevent a situation where the user intends to perform monitoring but is not actually able to, thereby improving a monitoring accuracy” (See [0093] of Koyama).
Regarding claim 11, Li and Koyama in combination teach all the limitations of claim 10 as discussed above.
Li additionally teaches:
wherein the display the first inspection target at the first geographical position in an inspection map comprises: add the first inspection target and the first geographical position to an inspection target set, the inspection target set comprising all inspection targets and geographical positions of all the inspection targets that have been detected by the inspection device during the process of performing an inspection task; and (See at least [n0054-n0055]: “The above calculations can be used to statistically identify the type, location, and number of defects, and generate an inspection report. At this point, the inspection report mainly reflects the results of the drone inspection, specifically including the types of defects, locations and numbers of defects, etc., as mentioned above. It may also include the proportion of defect types, inspection mileage, and geographical coordinates of defects…”. See also [n0056] regarding the inspection report.)
display all the inspection targets in the inspection target set at corresponding geographical positions of all the inspection targets in the inspection map. (See at least [n0052-n0053]: “Step S8: Project the geographic coordinates of each defect after weight reduction onto the site image, and generate an inspection report based on the defect type. Optionally, projecting the geographic coordinates of each defect after deduplication onto the website image includes: obtaining the TFW file used during website construction. The TFW file contains the geographic coordinates of the top left and bottom right corners of the website image in the world coordinate system, as well as the pixel lengths of the website image in both the horizontal and vertical directions…”)
Regarding claim 12, Li and Koyama in combination teach all the limitations of claim 10 as discussed above.
Li additionally teaches:
further comprises: obtain the target inspection sub-area; and (See at least [n0026]: “…step S2 specifically includes: dividing the inspection range into multiple regional subarrays based on the actual terrain of the photovoltaic power station and the difficulty of drone inspection…”)
delete a second inspection target and a geographical position of the second inspection target from the inspection target set, the second inspection target being an inspection target in the inspection target set and located in the target inspection sub-area. (See at least [n0049]: “In sub-step S75, duplicate defect geographic coordinates can be removed by calculating the distance from the location of the two defect geographic coordinates to the corresponding infrared image center point. Sub-step S75 retains the defect geographic coordinates corresponding to the location on the infrared image that is closer to the infrared image center point…”)
Regarding claim 13, Li and Koyama in combination teach all the limitations of claim 12 as discussed above.
Li additionally teaches:
further comprises: determine whether a second geographical position in the target inspection sub-area exists in the inspection target set; and (See at least [n0048]: “Sub-step S72: Construct an adjacency matrix between all pairs of defective geographic coordinates in the same set, and set a threshold for the measurement between two defective geographic coordinates (equivalent to setting a threshold for the values in the adjacency matrix)” & [0128]: “Sub-step S73: Determine whether the metric between the two defect geographic coordinates is less than the threshold…”)
determine, in response to the second geographical position existing in the inspection target set, that the second inspection target exists in the inspection target set; or determine, in response to no second geographical position existing in the inspection target set, that no second inspection target exists in the inspection target set. (See at least [0129]: “When the metric is less than the threshold, determine that the geographic coordinates of the two defects correspond to the same defect on the infrared image.”)
Regarding claim 14, Li and Koyama in combination teach all the limitations of claim 12 as discussed above.
Li additionally teaches:
wherein the obtain the target inspection sub-area comprises: obtain an inspection image obtained by the inspection device; and (See at least [n0020]: “…During the inspection process, the inspection area is divided into multiple sub-areas. A drone performs inspections according to a pre-planned route, acquiring infrared and wide-angle images…”)
determine an inspection map area corresponding to the inspection image as the target inspection sub-area. (See at least [n0026]: “…step S2 specifically includes: dividing the inspection range into multiple regional subarrays based on the actual terrain of the photovoltaic power station and the difficulty of drone inspection; exporting a kml file, which includes the world geographic coordinate range of the photovoltaic power station and each regional subarray; generating a tfw file that marks the range of each regional subarray based on the kml file, and determining the interval and unit pixel value size defined for each regional subarray.”)
Regarding claim 16, Li and Koyama in combination teach all the limitations of claim 10 as discussed above.
Li additionally teaches:
further comprises: perform inspection target identification on the inspection image obtained by the inspection device to obtain the first inspection target and first image coordinates of the first inspection target in the inspection image; and (See at least [n0031-n0032]: “Step S4: Perform defect detection on the infrared image to obtain the defect type and the location bounding box of the defect on the infrared image, and obtain the center coordinates of the location bounding box. Optionally, the infrared image is input into a neural network deep learning model to perform defect detection, obtain the defect type and the location bounding box of the defect on the infrared image, and obtain the center coordinates of the location bounding box. For example, the center point of the dual-light camera (the drone hovering position) is marked as a dot on the infrared image, and the location range of the defect on the infrared image is represented by a rectangle. In infrared images, different defect types can be indicated by different colors and/or shapes.”)
perform coordinate transformation on the first image coordinates to obtain first geographical position coordinates of the first inspection target, the first geographical position coordinates being coordinates in a world coordinate system. (See at least [n0035]: “Step S6: Transform the center position coordinates mapped onto the wide-angle image to the world coordinate system to obtain the defect geographic coordinates” & [n0038]: “Sub-step S62: Combining the intrinsic parameters, external attitude angles, and hovering position information of the dual-light camera, the center position coordinates mapped to the wide-angle image are transformed to the world coordinate system to obtain the geographic coordinates of the defect.”)
Regarding claim 17, Li and Koyama in combination teach all the limitations of claim 16 as discussed above. Li additionally teaches:
further comprising: perform, by means of a target identification model, inspection target identification on the inspection image obtained by the inspection device to obtain the first inspection target and the first image coordinates of the first inspection target in the inspection image. (See at least [n0032]: “…the infrared image is input into a neural network deep learning model to perform defect detection, obtain the defect type and the location bounding box of the defect on the infrared image, and obtain the center coordinates of the location bounding box…”)
Regarding claim 18, Li and Koyama in combination teach all the limitations of claim 16 as discussed above. Li additionally teaches:
further comprises: obtain a first device posture and first device position coordinates of the inspection device, the first device posture being a device posture when the inspection device obtains the inspection image, the first device position coordinates being geographical position coordinates when the inspection device obtains the inspection image, and the first device position coordinates being coordinates in the world coordinate system; (See at least [0022]: “Aerial triangulation is performed on the wide-angle image to obtain the external attitude angle of the UAV relative to the world coordinate system when the UAV acquires the wide-angle image” & [0026-0027]: “The process of combining the intrinsic parameters of the dual-light camera, the external attitude angle, and the hovering position information to transform the center position coordinates mapped onto the wide-angle image to the world coordinate system, thereby obtaining the defect geographic coordinates, includes: Acquire the longitude, latitude, and absolute elevation of the UAV hovering position when the wide-angle image is acquired…”. See also [0148] regarding transforming center position coordinates to the world coordinate system to obtain the defect geographic coordinates.)
determine a target transformation matrix based on the first device posture and the first device position coordinates, the target transformation matrix being used to denote a transformation relationship between a pixel coordinate system and the world coordinate system when the inspection device obtains the inspection image; and (See at least [n0008]: “…mapping the infrared image to the wide-angle image includes: selecting four marked positions from the infrared image; mapping the four marked positions to the wide-angle image to determine the parameters of the perspective transformation matrix…” & [0025-0029]: “…The coordinates, elevation information X, Y, Z, and the external attitude angles include the rotation angles around the three spatial coordinate axes X, Y, Z; The process of combining the intrinsic parameters of the dual-light camera, the external attitude angle, and the hovering position information to transform the center position coordinates mapped onto the wide-angle image to the world coordinate system, thereby obtaining the defect geographic coordinates, includes…Calculate the rotation matrix according to the order of rotation along the Y, Z, and X axes;”)
perform, based on the target transformation matrix, coordinate transformation on the first image coordinates to obtain the first geographical position coordinates. (See at least [0030]: “The geographical coordinates of the defect are calculated based on the rotation matrix and the geographical coordinates of the center point of the wide-angle image in the world coordinate system.”)
Regarding claim 19, Li teaches:
A non-transitory computer readable storage medium, wherein the computer-readable storage medium stores a computer program, the computer program comprising program instructions, and when the program instructions are executed by a processor, the processor is enabled to perform a method for detecting and positioning an inspection target, comprising: detecting a first inspection target and a first geographical position of the first inspection target in a target inspection sub-area of an inspection device, (See at least [n0031-n0035]: “Step S4: Perform defect detection on the infrared image to obtain the defect type and the location bounding box of the defect on the infrared image, and obtain the center coordinates of the location bounding box…Step S6: Transform the center position coordinates mapped onto the wide-angle image to the world coordinate system to obtain the defect geographic coordinates.” See also [0059-0062] regarding obtaining the defect geographic coordinates and [n0064] regarding a computer readable storage medium.)
the target inspection sub-area being a sub-area inspected by the inspection device during a process of performing an inspection task; and (See at least [n0020]: “…During the inspection process, the inspection area is divided into multiple sub-areas…”)
displaying the first inspection target at the first geographical position in an inspection map (See at least [0063]: “The reporting module is used to project the geographic coordinates of each defect after weight reduction processing onto the website image, and generate an inspection report based on the defect type.”)
However, Li does not explicitly teach displaying the inspection target during the process of performing the inspection task.
Koyama teaches a monitoring map that displays a monitoring area divided into a grid of cells via a display device and decreasing a monitoring time interval for each cell as time passes, such that “continuous monitoring is automatically executed” (See at least Figs. 2-3, [0045], [0067] & [0080]).
One having ordinary skill in the art, before the effective filing date of the claimed invention, would have found it obvious to combine Li’s method with Koyama’s technique of displaying the monitoring area during the process of performing the inspection task. Doing so would be obvious since “As a result, only an area that is actually monitored can be reliably managed as monitored, so that it is possible to prevent a situation where the user intends to perform monitoring but is not actually able to, thereby improving a monitoring accuracy” (See [0093] of Koyama).
Regarding claim 20, Li and Koyama in combination teach all the limitations of claim 1 as discussed above. Li additionally teaches:
wherein the displaying the first inspection target at the first geographical position in an inspection map comprises: adding the first inspection target and the first geographical position to an inspection target set, the inspection target set comprising all inspection targets and geographical positions of all the inspection targets that have been detected by the inspection device during the process of performing an inspection task; and (See at least [n0054-n0055]: “The above calculations can be used to statistically identify the type, location, and number of defects, and generate an inspection report. At this point, the inspection report mainly reflects the results of the drone inspection, specifically including the types of defects, locations and numbers of defects, etc., as mentioned above. It may also include the proportion of defect types, inspection mileage, and geographical coordinates of defects…”. See also [n0056] regarding the inspection report.)
displaying all the inspection targets in the inspection target set at corresponding geographical positions of all the inspection targets in the inspection map. (See at least [n0052-n0053]: “Step S8: Project the geographic coordinates of each defect after weight reduction onto the site image, and generate an inspection report based on the defect type. Optionally, projecting the geographic coordinates of each defect after deduplication onto the website image includes: obtaining the TFW file used during website construction. The TFW file contains the geographic coordinates of the top left and bottom right corners of the website image in the world coordinate system, as well as the pixel lengths of the website image in both the horizontal and vertical directions…”)
Claim(s) 6 and 15 is/are rejected under 35 U.S.C. 103 as being unpatentable over Li in view of Koyama and further in view of Zhang of CN111667429A, filed 06/06/2020, hereinafter “Zhang”.
Regarding claim 6, Li and Koyama in combination teach all the limitations of claim 5 as discussed above.
Li and Koyama in combination do not explicitly teach:
wherein before the determining an inspection map area corresponding to the inspection image as the target inspection sub-area, the method comprises: obtaining vertex image coordinates of the inspection image, the vertex image coordinates being coordinates in a pixel coordinate system;
performing coordinate transformation on the vertex image coordinates to obtain inspection position coordinates corresponding to the inspection image, the inspection position coordinates being coordinates in a world coordinate system; and
determining, in the inspection map, a sub-area formed by enclosing the inspection position coordinates as the inspection map area corresponding to the inspection image.
Zhang teaches:
wherein before the determining an inspection map area corresponding to the inspection image as the target inspection sub-area, the method comprises: obtaining vertex image coordinates of the inspection image, the vertex image coordinates being coordinates in a pixel coordinate system; (See at least [0013-0015]: “…obtaining the image of the area of the device to be inspected by processing the image containing the marker and the device to be inspected includes: Obtain the image outline of the marker; Extract the coordinates of each square corner point in the image coordinate system from the image outline of the marker;”)
performing coordinate transformation on the vertex image coordinates to obtain inspection position coordinates corresponding to the inspection image, the inspection position coordinates being coordinates in a world coordinate system; and (See at least [0023]: “By measuring the relative positional relationship between the device under inspection and the marker in the world coordinate system, the position coordinates of the device under inspection in the world coordinate system can be obtained.” See also [0065-0068] regarding the homography matrix and obtaining the position coordinates of the device under inspection in the world coordinate system.)
determining, in the inspection map, a sub-area formed by enclosing the inspection position coordinates as the inspection map area corresponding to the inspection image. (See at least Fig. 2 & [0024]: “Obtain the position coordinates of the device under inspection in the image containing the marker and the device under inspection; crop to obtain the image of the area of the device under inspection.”)
One having ordinary skill in the art, before the effective filing date of the claimed invention, would have found it obvious to combine Li and Koyama’s method with Zhang’s technique of determining an inspection map area corresponding to the inspection image as the target inspection sub-area. Doing so would be obvious to “improve the target positioning accuracy of the inspection robot, which has good economy and practicality” (See [0031] of Zhang).
Regarding claim 15, Li and Koyama in combination teach all the limitations of claim 14 as discussed above.
Li and Koyama in combination do not explicitly teach:
further comprises: obtain vertex image coordinates of the inspection image, the vertex image coordinates being coordinates in a pixel coordinate system;
perform coordinate transformation on the vertex image coordinates to obtain inspection position coordinates corresponding to the inspection image, the inspection position coordinates being coordinates in a world coordinate system; and
determine, in the inspection map, a sub-area formed by enclosing the inspection position coordinates as the inspection map area corresponding to the inspection image.
Zhang teaches:
further comprises: obtain vertex image coordinates of the inspection image, the vertex image coordinates being coordinates in a pixel coordinate system; (See at least [0013-0015]: “…obtaining the image of the area of the device to be inspected by processing the image containing the marker and the device to be inspected includes: Obtain the image outline of the marker; Extract the coordinates of each square corner point in the image coordinate system from the image outline of the marker;”)
perform coordinate transformation on the vertex image coordinates to obtain inspection position coordinates corresponding to the inspection image, the inspection position coordinates being coordinates in a world coordinate system; and (See at least [0023]: “By measuring the relative positional relationship between the device under inspection and the marker in the world coordinate system, the position coordinates of the device under inspection in the world coordinate system can be obtained.” See also [0065-0068] regarding the homography matrix and obtaining the position coordinates of the device under inspection in the world coordinate system.)
determine, in the inspection map, a sub-area formed by enclosing the inspection position coordinates as the inspection map area corresponding to the inspection image. (See at least Fig. 2 & [0024]: “Obtain the position coordinates of the device under inspection in the image containing the marker and the device under inspection; crop to obtain the image of the area of the device under inspection.”)
One having ordinary skill in the art, before the effective filing date of the claimed invention, would have found it obvious to combine Li and Koyama’s method with Zhang’s technique of determining an inspection map area corresponding to the inspection image as the target inspection sub-area. Doing so would be obvious to “improve the target positioning accuracy of the inspection robot, which has good economy and practicality” (See [0031] of Zhang).
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to NIKKI MARIE M MOLINA whose telephone number is (571)272-5180. The examiner can normally be reached M-F, 9am-6pm PT.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Aniss Chad can be reached at 571-270-3832. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/NIKKI MARIE M MOLINA/Examiner, Art Unit 3662
/ANISS CHAD/Supervisory Patent Examiner, Art Unit 3662
Prosecution Insights