DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
This is in response to the applicant’s reply filed January 9, 2026. In the applicant’s reply; no claims were amended, cancelled, or added. Claims 1-20 are pending in this application.
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
Examiner’s Responses to Applicant’s Remark
Applicants' amendments filed on January 9, 2026 have been fully considered. The amendments overcome the following rejections set forth in the office action mailed on November 5, 2026.
Applicant’s amendments overcome the objection to the title of the specification and the objection is hereby withdrawn.
Applicant’s arguments are persuasive and overcome the provisional non-statutory double patenting rejections of Claim 1-20 as being unpatentable over claims 1-20 of copending Application No. 18/128,773 (reference application) and the rejection is hereby withdrawn.
Applicants' arguments filed on January 9, 2026 have been fully considered but they are not persuasive.The Examiner has thoroughly reviewed Applicants' arguments but firmly believes that the cited reference to reasonably and properly meet the claimed limitation.
Applicant argues that the claims are directed towards statutory subject matter, as the “Specification describes – and the claims embody – a specific technical solution that materially improves the process of auditing annotated image datasets“.
Examiner respectfully disagrees. The rejection of the claims for being directed towards non-statutory subject matter still holds for the reasons cited below. Applicant’s arguments and allegation that the specification describes “a specific technical solution” is not conveyed or interpreted from the claimed limitations in accordance with the plain language of the claims. Applicant is encouraged to incorporate in such features from the specification in order to be patent eligible in accordance with (Step 2B), which In the context of the flowchart in MPEP 2106, subsection III, Step 2B determines whether: Does the claim recited additional elements that amount to significantly more than the judicial exception, and to advance prosecution on the merits with respect to this matter.
Applicant argues that Vanska or Yang do not anticipate or obviate the recited features in independent claims 1 and 8 for “determining, based on a predetermined confidence interval, a predetermined target error ratio, and a predetermined interval width, a minimum number of inspections of the annotated image set”.
Examiner respectfully disagrees. With respect to Vanska, Applicants are reminded that the Examiner is entitled to give the broadest reasonable interpretation to the language of the claims. Applicant argues that
PNG
media_image1.png
182
797
media_image1.png
Greyscale
However, the manner in which the claim is recited appears to be a series of limitations that are only dependent upon the confidence interval; “determining, based on a predetermined confidence interval, a predetermined target error ratio, and a predetermined interval width, a minimum number of inspections of the annotated image set”. The target error ratio, the interval width and the minimum number of inspections are all determined with respect to the confidence interval; the features argued by applicant for “determining any minimum number of inspections of annotated images” based on “a predetermined confidence interval, a predetermined target error ratio, and a predetermined interval width” are NOT recited in accordance with claimed language and consequently were not interpreted in that manner. For this claim limitation, [0044] and Figure 6 of Vanska was cited; Vanska’s first and second accuracy measures in steps 602 and 604 act as a predetermined confidence interval, while the process in steps 610 and 612 for generating a first and second set of tile sub-images is analogous in scope to a predetermined interval width, while the process in step 614 compare the zoomed tiled sub-images using a predefined threshold would be analogous in scope to a predetermined target error ratio. These steps in combination are then later used in step 616, for determining the annotation of a tiled sub-image and its coordinate and step 618 to generate an inspection report. So the Examiner considers these features of Vanska to be Applicants' “determining, based on a predetermined confidence interval, a predetermined target error ratio, and a predetermined interval width, a minimum number of inspections of the annotated image set” within the broad meaning of the term. The Examiner is not limited to Applicants' definition which is not specifically set forth in the claims. In re Tanaka et al., 193 USPQ 139, (CCPA) 1977.
Vanska: [0044] The user terminal may generate an inspection report that comprises location specific annotations and electronically (e.g. a Global Positioning System) acquired time stamped routes when a user (e.g. an inspector of a construction site) completes the inspection process. In an embodiment, a motion sensing circuit that is communicatively coupled to the user terminal is used to verify whether the inspection is carried out by a user or not. In another embodiment, the user terminal communicates the inspection report to the server when the internet connection is available. [0061] FIGS. 6A-6C are flow diagrams that illustrate a method for facilitating an inspection process in accordance with an embodiment of the present disclosure. At step 602, a vector graphic version of a construction drawing is converted to a first set of raster images using a first accuracy. At step 604, the vector graphic version of the construction drawing is converted to a second set of raster images using a second accuracy. The second accuracy is higher than the first accuracy. At step 606, the first set of raster images is divided to a first set of tiled sub-images. At step 608, the second set of raster images is divided to a second set of tiled sub-images. At step 610, the first set of tiled sub-images and the second set of tiled sub-images are received by a user terminal. At step 612, at least one tiled sub-image of the first set of tiled sub-images is rendered on the user terminal. At step 614, at least one tiled sub-image of the second set of tiled sub-images corresponding to the at least one tiled sub-image of the first set of tiled sub-images is rendered when a request to zoom the at least one tiled sub-image by a zoom rate exceeding a predefined threshold. At step 616, the annotation of a tiled sub-image and its coordinate are saved when a request to annotate a given coordinate of the tiled sub-image is received. At step 618, an inspection report that comprises the annotation and its coordinate for the tiled sub-images associated the vector graphic version of the construction drawings is generated.
Additionally, With respect to Yang, Applicants are again reminded that the Examiner is entitled to give the broadest reasonable interpretation to the language of the claims, and the claim language was interpreted in the manner presented above. The manner in which the claim is recited in itself appears to be a series of limitations requiring features that are analogous in scope to the following elements: “determining, based on a predetermined confidence interval, a predetermined target error ratio, and a predetermined interval width, a minimum number of inspections of the annotated image set”. Yang teaches the use of a confidence measure in [0036] “ensure the quality of the quality check” and in [0047] “probability of error risk” and “a set review proportion”, and “the number of pieces of data annotated” which corresponds to the elements listed. These features are further described in Figure 6, and paragraphs [0092]-[0097], which were also cited in the previously presented rejection of record.. So the Examiner considers Yang’s teachings as cited to be Applicants' “determining, based on a predetermined confidence interval, a predetermined target error ratio, and a predetermined interval width, a minimum number of inspections of the annotated image set” within the broad meaning of the term. The Examiner is not limited to Applicants' definition which is not specifically set forth in the claims. In re Tanaka et al., 193 USPQ 139, (CCPA) 1977.
Yang: [0036] Further, in order to ensure the quality of the quality check, a quality check person can perform a secondary quality check if the quality check result obtained by using the algorithm quality check has low confidence. [0047] The to-be-reviewed person is the annotation person selected from the candidate annotation persons and that needs to be paid close attention to. That is, compared with other persons among the candidate annotation persons, the to-be-reviewed person has a higher probability of error risk in annotated data. The number of to-be-reviewed persons may be one or more; in order to ensure the reasonableness of the number of pieces of to-be-reviewed data for subsequent review, the number of to-be-reviewed persons is further determined by a set review proportion, the number of pieces of data annotated by each candidate annotation person this time, the annotation person information of each candidate annotation person and the like. [0065] In S302, a quality check is performed on the to-be-reviewed data. [0092]-[0098] FIG. 6)
Yang: [0051] -[0052] In S202, data annotated by the to-be-reviewed person is selected from the set of annotated data as the to-be-reviewed data. [0053] -[0054], [0092]-[0097], FIG. 6 is a flowchart of another data processing method according to an embodiment of the present disclosure. This embodiment of the present disclosure provides an example on the basis of the preceding embodiment. As shown in FIG. 6, the method includes steps described below. [0093] In S601, cleaning processing is performed on a set of to-be-annotated data. [0094] In S602, an annotation rule of the set of to-be-annotated data subjected to the cleaning processing is determined according to an annotation material type and an annotation scenario. [0095] In S603, in a process of annotating the set of to-be-annotated data, a quality check is performed on annotated data according to the annotation rule to obtain a set of annotated data. [0096] In S604, to-be-reviewed data is selected from the set of annotated data according to at least one of data annotation information or annotation person information. [0097] In S605, a quality check is performed on the to-be-reviewed data.)
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to non-statutory subject matter. The claim(s) does/do not fall within at least one of the four categories of patent eligible subject matter because the claimed invention is directed to a judicial exception (i.e., abstract idea – an idea to itself/mathematical concept) without significantly more.
(1) Are the claims directed to a process, machine, manufacture or composition of matter;
(2A) Prong One: Are the claims directed to a judicially recognized exception, i.e., a law of nature, a natural phenomenon, or an abstract idea;
Prong Two: If the claims are directed to a judicial exception under Prong One, then is the judicial exception integrated into a practical application;
(2B) If the claims are directed to a judicial exception and do not integrate the judicial exception, do the claims provide an inventive concept.
(Step 1) In the context of the flowchart in MPEP § 2106, subsection III, Step (1): Are the claims directed to a process, machine, manufacture or composition of matter? YES: the instant claims recite a method, a system and a non-transitory computer readable medium.
(Step 2A) In the context of the flowchart in MPEP § 2106, subsection III, Step 2A Prong Two determines whether: Is the claim directed to a law of nature, a natural phenomenon, or an abstract idea? YES.
When viewed under the broadest most reasonable interpretation, the instant claims are directed to a Judicial Exception – an abstract idea belonging to the group of mental process or mathematical concept.As a whole, the claimed features can be interpreted as an overall mathematical process, as they are a natural language representation of an overall mathematical algorithm, which qualifies as a judicial exception.
Claim 1 is exemplary of the independent claims 1, 8 and 15, and is presented below. Specifically, the claim recites:
-a; A method for auditing the inspection of image annotation quality in an annotated image dataset, the method comprising:
-b; obtaining an annotated image dataset;
-c; determining, based on a predetermined confidence interval, a predetermined target error ratio, and a predetermined interval width, a minimum number of inspections of the annotated image set;
-d; selecting a plurality of frames of the annotated image dataset for inspection based on the minimum number of inspections;
-e; and outputting the selected plurality of frames of the annotated image dataset for inspection.
When viewed under the broadest most reasonable interpretation, the instant claims are directed to a Judicial Exception – an abstract idea belonging to the group of mental process and/or a mathematical concept.As a whole, the claimed features can be interpreted as an overall mental process with some mathematical elements, as they are a natural language representation of an overall mathematical algorithm, which qualifies as a judicial exception. The step (a) is merely the preamble and is directed to the intended use of the claims. The step (b) of “obtaining an annotated image dataset;” is considered to be extra-solution activity and can be done manually. The step (c) of for “determining, based on a predetermined confidence interval, a predetermined target error ratio, and a predetermined interval width, a minimum number of inspections of the annotated image set” is considered to be judicially recited mathematical concept/algorithm. The step (c) of “obtaining at least one prediction residual…” is also considered to be a mathematical concept. The step of (d) “selecting a plurality of frames of the annotated image dataset for inspection based on the minimum number of inspections” is also considered to be a mathematical concept that can also be done mentally or manually. The step (e) of “outputting the selected plurality of frames of the annotated image dataset for inspection” is merely an extra-solution step of outputting data. There is nothing in the claim that requires more than an operation that a human, armed with a general computer or an appropriate apparatus executing a mathematical algorithm can perform.
These features, in combination, are applied to an overall image dataset, and the nature of the term "image dataset" can be applied to any input that can be computationally processed. As the overall features of the claims can be interpred to be a mathematical algorithm, it does meet the requirements of Step2A for a judicial exception.
Since the claim as a whole does not integrate the exception into a practical application, in which case the claim is directed to the judicial exception (Step 2A: YES), it requires further analysis under Step 2B (where it may still be eligible if it amounts to an inventive concept).
(Step 2B) In the context of the flowchart in MPEP 2106, subsection III, Step 2B determines whether: Does the claim recited additional elements that amount to significantly more than the judicial exception? NO.
The instant claims do not apply, rely on, or use the judicial exception in a manner that imposes a meaningful limit on the judicial exception of “provide/receive first/second fractional flow reserve value” and therefore does not integrate the judicial exception into a practical application.In accordance with the MPEP § 2106.04(d) Integration of a Judicial Exception Into A Practical Application [R-07.2022],
Similarly, in a growing body of decisions, the Federal Circuit has distinguished between claims that are ‘‘directed to’’ a judicial exception (which require further analysis to determine their eligibility) and those that are not (which are therefore patent eligible), e.g., claims that improve the functioning of a computer or other technology or technological field. See Diamond v. Diehr, 450 U.S. 175, 209 USPQ 1 (1981); Gottschalk v. Benson, 409 U.S. 63, 175 USPQ 673 (1972). See, e.g., MPEP § 2106.06(b) (summarizing Enfish, LLC v. Microsoft Corp., 822 F.3d 1327, 118 USPQ2d 1684 (Fed. Cir. 2016), McRO, Inc. v. Bandai Namco Games Am. Inc., 837 F.3d 1299, 120 USPQ2d 1091 (Fed. Cir. 2016), and other cases that were eligible as improvements to technology or computer functionality instead of being directed to abstract ideas)”
“Accordingly, after determining that a claim recites a judicial exception in Step 2A Prong One, examiners should evaluate whether the claim as a whole integrates the recited judicial exception into a practical application of the exception in Step 2A Prong Two. A claim that integrates a judicial exception into a practical application will apply, rely on, or use the judicial exception in a manner that imposes a meaningful limit on the judicial exception, such that the claim is more than a drafting effort designed to monopolize the judicial exception.” See MPEP § 2106.05 for discussion of Step 2B.
With respect to the claimed limitations, the following features are directed to a mathematical process that do not integrate the judicial exception into a practical application, as there is no “meaningful limit”.
Claim 1 is exemplary of the independent claims 1, 8 and 15, and is presented below. In particular, the claim includes additional elements to perform the following:
-a; A method for auditing the inspection of image annotation quality in an annotated image dataset, the method comprising:
-b; obtaining an annotated image dataset;
-c; determining, based on a predetermined confidence interval, a predetermined target error ratio, and a predetermined interval width, a minimum number of inspections of the annotated image set;
-d; selecting a plurality of frames of the annotated image dataset for inspection based on the minimum number of inspections;
-e; and outputting the selected plurality of frames of the annotated image dataset for inspection.
The steps a. and b. use an apparatus to audit/inspect/obtain “annotated image data” at a high level of generality such that said “annotated image data” can be used in the operation of the recited judicial exception (of being a mental step or manual step of ”auditing”/“obtaining”). Supplying “annotated image dta” does not provide for “integration” of the abstract idea into a practical application, as said “data”/”model” do not change the way in which said apparatus operates. In fact, there are no limits on the apparatus, which is recited at a high level of generality and thus said apparatus does nothing more than perform generic computing functions of “auditing”/”obtaining” in the claim.
Step c and d are mathematical steps that can also be done mentally; there are no specifics about what they are doing in “determining” or “selecting”, and thus one can merely ‘look’ at the data and make a ‘determination’. This is considered to be either a mental/manual process or a mathematical calculation step. One can manually select the ‘frames’ for inspection and or use a mathematical formula for the determination. Step e. is simply a “outputting” and is a routine step in the image analysis and is considered extra-solution activity. The step of “displaying” does not make the claim as a whole patent eligible because the claim as a whole of judicial exception does not integrate into a practical application.
With regard to (2B), the pending claims do not show what is more than a routine in the art presented in the claims, i.e., the additional elements are nothing more than routine and well-known steps. There is no improvement to technology here. There is only a “providing”/“receiving”, “correction” (mental process), “determining”, “displaying” (extra-solution step), and it has not been shown that the mental process allows the “technology” (whether it is computer technology or any other technology) to do something that it previously was not able to do. DIAMOND VS. DIEHR
The claimed features, at best would invoke the analysis of the features under MPEP § 2106.05(a) as to whether the features qualify as improvements to the functioning of a computer or to any other technology or technical field. Even when considering the relevant consideration for evaluating whether the additional elements amount to an inventive concept, the limitations do not recite any elements that can be considered to qualify as “significantly more” when recited in a claim with judicial exception. At best, the claimed limitations only recite features that “apply” the judicial exception, and add “insignificant extra-solution activity to the judicial exception”,
Dependent claims 2-7, 9-14 and 16-20 are rejected for the same reasons; claims are directed to a judicial exception and do not integrate the judicial exception.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1, 8 and 15 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Vanska et al. (US PGPub US20180039715 A1, hereby referred to as “Vanska”).
Consider Claims 1, 8 and 15.
Vanska teaches ……
1. A method for auditing the inspection of image annotation quality in an annotated image dataset, the method comprising: / 8. An apparatus for auditing the inspection of image annotation quality in an annotated image dataset, the apparatus comprising: / 15. A non-transitory computer-readable recording medium having recorded thereon instructions executable by at least one processor to cause the processor to perform a method comprising: (Vanska: abstract, A method for facilitating an inspection process that involves comparing a construction drawing of a given site and said site. The method includes steps of: converting a vector graphic version of the construction drawing to a first set of raster images using a first accuracy; converting the vector graphic version of the construction drawing to a second set of raster images using a second accuracy, which second accuracy is higher than the first accuracy; dividing the first set of raster images to a first set of tiled sub-images; dividing the second set of raster images to a second set of tiled sub-images; receiving the first set of tiled sub-images and the second set of tiled sub-images by a user terminal; rendering at least one tiled sub-image, of the first set of tiled sub-images, on the user terminal; upon request to zoom the at least one tiled sub-image by a zoom rate exceeding a pre-defined threshold, rendering at least one tiled sub-image of the second set of tiled sub-images, corresponding to said at least one tiled sub-image of the first set of tiled sub-images, on the user terminal; upon request to annotate a given coordinate of a tiled sub-image, saving the annotation and its coordinate at the user terminal; and generating an inspection report including the annotation and its coordinate.)
8. at least one memory storing computer-executable instructions; and at least one processor configured to execute the computer-executable instructions to: (Vanska: [0012], [0022], [0050] The medium can be an electronic, magnetic, optical, electromagnetic, infrared or semiconductor system (or apparatus or device) or a propagation medium. Examples of a computer-readable medium comprise a semiconductor or solid-state memory, magnetic tape, a removable computer diskette, a random access memory (RAM), a read-only memory (ROM), a rigid magnetic disk and an optical disk. Current examples of optical disks comprise a compact disk-read only memory (CD-ROM), a compact disk-read/write (CD-R/W) and a digital versatile disc (DVD). [0051] A data processing system suitable for storing and/or executing a program code may comprise at least one processor coupled directly or indirectly to memory elements through a system bus. The memory elements can include local memory employed during actual execution of the program code, bulk storage and cache memories which provide temporary storage of at least some program code in order to reduce the number of times the code must be retrieved from the bulk storage during execution.)
1. obtaining an annotated image dataset; / 8. obtain an annotated image dataset; / 15. obtaining an annotated image dataset; (Vanska: [0036] In an embodiment, the user terminal comprises a tiled sub-image receiving module, a tiled sub-image rendering module, and an annotation module to generate an inspection report for a construction drawing. The tiled sub-image receiving module may receive the first set of tiled sub-images and the second set of tiled sub-images. The tiled sub-image rendering module may render at least one tiled sub-image of the first set of tiled sub-images. The tiled sub-image rendering module may render the at least one tiled sub-image of the second set of tiled sub-images corresponding to the at least one tiled sub-image of the first set of tiled sub-images when a request to zoom the at least one tiled sub-image by a zoom rate exceeding a pre-defined threshold. [0037] The annotation module may save the annotation of a tiled sub-image and its coordinate when a request to annotate a given coordinate of the tiled sub-image is received. The user terminal may comprise a terminal database that stores the annotation and its coordinate of the tiled sub-images. [0038], [0054] The server 104 that comprises communication means is configured to convert the vector graphic version of the construction drawing to a first set of raster images using a first accuracy. The server 104 converts the vector graphic version of the construction drawing to a second set of raster images using a second accuracy. The server 104 divides the first set of raster images to a first set of tiled sub-images, and the second set of raster images to a second set of tiled sub-images. The user terminal 102 is configured to receive the first set of tiled sub-images and the second set of tiled sub-images through the communication means. The user terminal 102 renders at least one tiled sub-image of the first set of tiled sub-images. The user terminal 102 renders at least one tiled sub-image of the second set of tiled sub-images corresponding to the at least one tiled sub-image of the first set of tiled sub-images when a request to zoom the at least one tiled sub-image by a zoom rate exceeding a pre-defined threshold. The user terminal 102 saves the annotation of a tiled sub-image and its coordinate when a request to annotate a given coordinate of the tiled sub-image is received. The user terminal 102, and/or the server 104 may generate an inspection report that comprises the annotation and its coordinate of the tiled sub-image. [0055], Figure 2)
1. determining, based on a predetermined confidence interval, a predetermined target error ratio, and a predetermined interval width, a minimum number of inspections of the annotated image set; / 8. determine, based on a predetermined confidence interval, a predetermined target error ratio, and a predetermined interval width, a minimum number of inspections of the annotated image set; / 15. determining, based on a predetermined confidence interval, a predetermined target error ratio, and a predetermined interval width, a minimum number of inspections of the annotated image set; (Vanska: [0044] The user terminal may generate an inspection report that comprises location specific annotations and electronically (e.g. a Global Positioning System) acquired time stamped routes when a user (e.g. an inspector of a construction site) completes the inspection process. In an embodiment, a motion sensing circuit that is communicatively coupled to the user terminal is used to verify whether the inspection is carried out by a user or not. In another embodiment, the user terminal communicates the inspection report to the server when the internet connection is available. [0061] FIGS. 6A-6C are flow diagrams that illustrate a method for facilitating an inspection process in accordance with an embodiment of the present disclosure. At step 602, a vector graphic version of a construction drawing is converted to a first set of raster images using a first accuracy. At step 604, the vector graphic version of the construction drawing is converted to a second set of raster images using a second accuracy. The second accuracy is higher than the first accuracy. At step 606, the first set of raster images is divided to a first set of tiled sub-images. At step 608, the second set of raster images is divided to a second set of tiled sub-images. At step 610, the first set of tiled sub-images and the second set of tiled sub-images are received by a user terminal. At step 612, at least one tiled sub-image of the first set of tiled sub-images is rendered on the user terminal. At step 614, at least one tiled sub-image of the second set of tiled sub-images corresponding to the at least one tiled sub-image of the first set of tiled sub-images is rendered when a request to zoom the at least one tiled sub-image by a zoom rate exceeding a predefined threshold. At step 616, the annotation of a tiled sub-image and its coordinate are saved when a request to annotate a given coordinate of the tiled sub-image is received. At step 618, an inspection report that comprises the annotation and its coordinate for the tiled sub-images associated the vector graphic version of the construction drawings is generated.
1. selecting a plurality of frames of the annotated image dataset for inspection based on the minimum number of inspections; / 8. select a plurality of frames of the annotated image dataset for inspection based on the minimum number of inspections; / 15. selecting a plurality of frames of the annotated image dataset for inspection based on the minimum number of inspections;
1. and outputting the selected plurality of frames of the annotated image dataset for inspection. / 8. and output the selected plurality of frames of the annotated image dataset for inspection. / 15. and outputting the selected plurality of frames of the annotated image dataset for inspection. (Vanska: [0060] Referring to FIG. 5, illustrated is an exemplary view of visualization of rendering of sub-images at a user terminal in accordance with an embodiment of the present disclosure. The exemplary view comprises a display frame 502 that represents a display of a user terminal, and tiled sub-images (e.g. twelve tiled sub-images as shown in FIG. 5) on a zoom level (e.g. a zero zoom level, or a first zoom level, etc.). The display frame 502 displays active tiled sub-images (e.g. 4 tiled sub-images within the display frame 502 as shown in FIG. 5). The user terminal may store the active tiled sub-images at a display memory for faster rendering of tiled sub-images. [0061] FIGS. 6A-6C are flow diagrams that illustrate a method for facilitating an inspection process in accordance with an embodiment of the present disclosure. At step 602, a vector graphic version of a construction drawing is converted to a first set of raster images using a first accuracy. At step 604, the vector graphic version of the construction drawing is converted to a second set of raster images using a second accuracy. The second accuracy is higher than the first accuracy. At step 606, the first set of raster images is divided to a first set of tiled sub-images. At step 608, the second set of raster images is divided to a second set of tiled sub-images. At step 610, the first set of tiled sub-images and the second set of tiled sub-images are received by a user terminal. At step 612, at least one tiled sub-image of the first set of tiled sub-images is rendered on the user terminal. At step 614, at least one tiled sub-image of the second set of tiled sub-images corresponding to the at least one tiled sub-image of the first set of tiled sub-images is rendered when a request to zoom the at least one tiled sub-image by a zoom rate exceeding a predefined threshold. At step 616, the annotation of a tiled sub-image and its coordinate are saved when a request to annotate a given coordinate of the tiled sub-image is received. At step 618, an inspection report that comprises the annotation and its coordinate for the tiled sub-images associated the vector graphic version of the construction drawings is generated.)
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent may not be obtained though the invention is not identically disclosed or described as set forth in section 102 of this title, if the differences between the subject matter sought to be patented and the prior art are such that the subject matter as a whole would have been obvious at the time the invention was made to a person having ordinary skill in the art to which said subject matter pertains. Patentability shall not be negatived by the manner in which the invention was made.
Claims 1, 8 and 15 are further rejected under 35 U.S.C. 103 as being unpatentable over Vanska et al. (US PGPub US 20180039715 A1, hereby referred to as “Vanska”), in view of Yang et al. (US PGPub US20220027854A1, hereby referred to as “Yang”).
Consider Claims 1, 8 and 15.
Vanska teaches:
1. A method for auditing the inspection of image annotation quality in an annotated image dataset, the method comprising: / 8. An apparatus for auditing the inspection of image annotation quality in an annotated image dataset, the apparatus comprising: / 15. A non-transitory computer-readable recording medium having recorded thereon instructions executable by at least one processor to cause the processor to perform a method comprising: (Vanska: abstract, A method for facilitating an inspection process that involves comparing a construction drawing of a given site and said site. The method includes steps of: converting a vector graphic version of the construction drawing to a first set of raster images using a first accuracy; converting the vector graphic version of the construction drawing to a second set of raster images using a second accuracy, which second accuracy is higher than the first accuracy; dividing the first set of raster images to a first set of tiled sub-images; dividing the second set of raster images to a second set of tiled sub-images; receiving the first set of tiled sub-images and the second set of tiled sub-images by a user terminal; rendering at least one tiled sub-image, of the first set of tiled sub-images, on the user terminal; upon request to zoom the at least one tiled sub-image by a zoom rate exceeding a pre-defined threshold, rendering at least one tiled sub-image of the second set of tiled sub-images, corresponding to said at least one tiled sub-image of the first set of tiled sub-images, on the user terminal; upon request to annotate a given coordinate of a tiled sub-image, saving the annotation and its coordinate at the user terminal; and generating an inspection report including the annotation and its coordinate.)
8. at least one memory storing computer-executable instructions; and at least one processor configured to execute the computer-executable instructions to: (Vanska: [0012], [0022], [0050] The medium can be an electronic, magnetic, optical, electromagnetic, infrared or semiconductor system (or apparatus or device) or a propagation medium. Examples of a computer-readable medium comprise a semiconductor or solid-state memory, magnetic tape, a removable computer diskette, a random access memory (RAM), a read-only memory (ROM), a rigid magnetic disk and an optical disk. Current examples of optical disks comprise a compact disk-read only memory (CD-ROM), a compact disk-read/write (CD-R/W) and a digital versatile disc (DVD). [0051] A data processing system suitable for storing and/or executing a program code may comprise at least one processor coupled directly or indirectly to memory elements through a system bus. The memory elements can include local memory employed during actual execution of the program code, bulk storage and cache memories which provide temporary storage of at least some program code in order to reduce the number of times the code must be retrieved from the bulk storage during execution.)
1. obtaining an annotated image dataset; / 8. obtain an annotated image dataset; / 15. obtaining an annotated image dataset; (Vanska: [0036] In an embodiment, the user terminal comprises a tiled sub-image receiving module, a tiled sub-image rendering module, and an annotation module to generate an inspection report for a construction drawing. The tiled sub-image receiving module may receive the first set of tiled sub-images and the second set of tiled sub-images. The tiled sub-image rendering module may render at least one tiled sub-image of the first set of tiled sub-images. The tiled sub-image rendering module may render the at least one tiled sub-image of the second set of tiled sub-images corresponding to the at least one tiled sub-image of the first set of tiled sub-images when a request to zoom the at least one tiled sub-image by a zoom rate exceeding a pre-defined threshold. [0037] The annotation module may save the annotation of a tiled sub-image and its coordinate when a request to annotate a given coordinate of the tiled sub-image is received. The user terminal may comprise a terminal database that stores the annotation and its coordinate of the tiled sub-images. [0038], [0054] The server 104 that comprises communication means is configured to convert the vector graphic version of the construction drawing to a first set of raster images using a first accuracy. The server 104 converts the vector graphic version of the construction drawing to a second set of raster images using a second accuracy. The server 104 divides the first set of raster images to a first set of tiled sub-images, and the second set of raster images to a second set of tiled sub-images. The user terminal 102 is configured to receive the first set of tiled sub-images and the second set of tiled sub-images through the communication means. The user terminal 102 renders at least one tiled sub-image of the first set of tiled sub-images. The user terminal 102 renders at least one tiled sub-image of the second set of tiled sub-images corresponding to the at least one tiled sub-image of the first set of tiled sub-images when a request to zoom the at least one tiled sub-image by a zoom rate exceeding a pre-defined threshold. The user terminal 102 saves the annotation of a tiled sub-image and its coordinate when a request to annotate a given coordinate of the tiled sub-image is received. The user terminal 102, and/or the server 104 may generate an inspection report that comprises the annotation and its coordinate of the tiled sub-image. [0055], Figure 2)
1. determining, based on a predetermined confidence interval, a predetermined target error ratio, and a predetermined interval width, a minimum number of inspections of the annotated image set; / 8. determine, based on a predetermined confidence interval, a predetermined target error ratio, and a predetermined interval width, a minimum number of inspections of the annotated image set; / 15. determining, based on a predetermined confidence interval, a predetermined target error ratio, and a predetermined interval width, a minimum number of inspections of the annotated image set; (Vanska: [0044] The user terminal may generate an inspection report that comprises location specific annotations and electronically (e.g. a Global Positioning System) acquired time stamped routes when a user (e.g. an inspector of a construction site) completes the inspection process. In an embodiment, a motion sensing circuit that is communicatively coupled to the user terminal is used to verify whether the inspection is carried out by a user or not. In another embodiment, the user terminal communicates the inspection report to the server when the internet connection is available. [0061] FIGS. 6A-6C are flow diagrams that illustrate a method for facilitating an inspection process in accordance with an embodiment of the present disclosure. At step 602, a vector graphic version of a construction drawing is converted to a first set of raster images using a first accuracy. At step 604, the vector graphic version of the construction drawing is converted to a second set of raster images using a second accuracy. The second accuracy is higher than the first accuracy. At step 606, the first set of raster images is divided to a first set of tiled sub-images. At step 608, the second set of raster images is divided to a second set of tiled sub-images. At step 610, the first set of tiled sub-images and the second set of tiled sub-images are received by a user terminal. At step 612, at least one tiled sub-image of the first set of tiled sub-images is rendered on the user terminal. At step 614, at least one tiled sub-image of the second set of tiled sub-images corresponding to the at least one tiled sub-image of the first set of tiled sub-images is rendered when a request to zoom the at least one tiled sub-image by a zoom rate exceeding a predefined threshold. At step 616, the annotation of a tiled sub-image and its coordinate are saved when a request to annotate a given coordinate of the tiled sub-image is received. At step 618, an inspection report that comprises the annotation and its coordinate for the tiled sub-images associated the vector graphic version of the construction drawings is generated.
1. selecting a plurality of frames of the annotated image dataset for inspection based on the minimum number of inspections; / 8. select a plurality of frames of the annotated image dataset for inspection based on the minimum number of inspections; / 15. selecting a plurality of frames of the annotated image dataset for inspection based on the minimum number of inspections;
1. and outputting the selected plurality of frames of the annotated image dataset for inspection. / 8. and output the selected plurality of frames of the annotated image dataset for inspection. / 15. and outputting the selected plurality of frames of the annotated image dataset for inspection. (Vanska: [0060] Referring to FIG. 5, illustrated is an exemplary view of visualization of rendering of sub-images at a user terminal in accordance with an embodiment of the present disclosure. The exemplary view comprises a display frame 502 that represents a display of a user terminal, and tiled sub-images (e.g. twelve tiled sub-images as shown in FIG. 5) on a zoom level (e.g. a zero zoom level, or a first zoom level, etc.). The display frame 502 displays active tiled sub-images (e.g. 4 tiled sub-images within the display frame 502 as shown in FIG. 5). The user terminal may store the active tiled sub-images at a display memory for faster rendering of tiled sub-images. [0061] FIGS. 6A-6C are flow diagrams that illustrate a method for facilitating an inspection process in accordance with an embodiment of the present disclosure. At step 602, a vector graphic version of a construction drawing is converted to a first set of raster images using a first accuracy. At step 604, the vector graphic version of the construction drawing is converted to a second set of raster images using a second accuracy. The second accuracy is higher than the first accuracy. At step 606, the first set of raster images is divided to a first set of tiled sub-images. At step 608, the second set of raster images is divided to a second set of tiled sub-images. At step 610, the first set of tiled sub-images and the second set of tiled sub-images are received by a user terminal. At step 612, at least one tiled sub-image of the first set of tiled sub-images is rendered on the user terminal. At step 614, at least one tiled sub-image of the second set of tiled sub-images corresponding to the at least one tiled sub-image of the first set of tiled sub-images is rendered when a request to zoom the at least one tiled sub-image by a zoom rate exceeding a predefined threshold. At step 616, the annotation of a tiled sub-image and its coordinate are saved when a request to annotate a given coordinate of the tiled sub-image is received. At step 618, an inspection report that comprises the annotation and its coordinate for the tiled sub-images associated the vector graphic version of the construction drawings is generated.)
Even if Vanska does not teach: determining, based on a predetermined confidence interval, a predetermined target error ratio, and a predetermined interval width, a minimum number of inspections of the annotated image set
Yang teaches
1. A method for auditing the inspection of image annotation quality in an annotated image dataset, the method comprising: / 8. An apparatus for auditing the inspection of image annotation quality in an annotated image dataset, the apparatus comprising: / 15. A non-transitory computer-readable recording medium having recorded thereon instructions executable by at least one processor to cause the processor to perform a method comprising: (Yang: abstract, Provided are a data processing method and apparatus, an electronic device and a storage medium. The implementation solution is selecting to-be-reviewed data from a set of annotated data according to at least one of data annotation information or annotation person information and performing a quality check on the to-be-reviewed data. [0025] FIG. 1 is a flowchart of a data processing method according to an embodiment of the present disclosure. The embodiments of the present disclosure are suitable for the case of how to process data and especially suitable for the case of how to perform a quality check on annotated data in scenarios in which data annotation is required, such as obstacle recognition scenarios, target (such as a vehicle) tracking scenarios, human key point (such as face) recognition scenarios and named entity recognition scenarios such that the efficiency of the data quality check is improved while data quality is ensured. This embodiment may be performed by a data processing apparatus. The apparatus may be implemented by software and/or hardware and may be integrated into an electronic device carrying data processing functions, such as a server device. As shown in FIG. 1, the method includes steps described below.)
8. at least one memory storing computer-executable instructions; and at least one processor configured to execute the computer-executable instructions to: (Yang: Figure 8, [0122]-[0130], [0125] The memory 802 as a non-transitory computer-readable storage medium is configured to store non-transitory software programs, non-transitory computer-executable programs, and modules, for example, program instructions/modules (for example, the data selection module 701 and the quality check module 702 shown in FIG. 7) corresponding to the data processing method according to the embodiments of the present disclosure. The processor 801 executes non-transitory software programs, instructions and modules stored in the memory 802 to execute the various function applications and data processing of a server, that is, implement the data processing method provided in the preceding method embodiments. [0126] The memory 802 may include a program storage region and a data storage region. The program storage region may store an operating system and an application required by at least one function. The data storage region may store data created based on the use of the electronic device for performing the data processing method. Additionally, the memory 802 may include a high-speed random-access memory and a non-transient memory, for example, at least one disk memory, a flash memory or another non-transient solid-state memory. In some embodiments, the memory 802 optionally includes memories disposed remote from the processor 801, and these remote memories may be connected, through a network, to the electronic device for performing the data processing method. Examples of the preceding network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network and a combination thereof. [0127] The electronic device for performing the data processing method may further include an input device 803 and an output device 804. The processor 801, the memory 802, the input device 803 and the output device 804 may be connected by a bus or in other manners. FIG. 8 uses connection by a bus as an example.)
1. obtaining an annotated image dataset; / 8. obtain an annotated image dataset; / 15. obtaining an annotated image dataset; (Yang: [0036] If the annotation requirement of the user is objective, the manner in which the to-be-reviewed data is reviewed may be an algorithm quality check such as an optical character recognition (OCR) algorithm quality check. In an embodiment, an existing OCR algorithm may be called for automatically performing the quality check on the to-be-reviewed data. Further, in order to ensure the quality of the quality check, a quality check person can perform a secondary quality check if the quality check result obtained by using the algorithm quality check has low confidence. Figure 2, [0044]-[0054], [0055] In S203, a quality check is performed on the to-be-reviewed data. [0056] According to the technical solution of this embodiment of the present disclosure, the time of engagement in annotation and the historical annotation accuracy rate are introduced and the person that needs to be paid close attention to, that is, the to-be-reviewed person, is selected from the candidate annotation persons so that the to-be-reviewed data can be selected from the set of annotated data with the to-be-reviewed person as a bridge. This technical solution provides an idea for selecting the to-be-reviewed data from the set of annotated data. Meanwhile, the quality check is performed on the to-be-reviewed data so that the efficiency of a data quality check is improved and the labor cost is reduced while the data quality is ensured.)
1. determining, based on a predetermined confidence interval, a predetermined target error ratio, and a predetermined interval width, a minimum number of inspections of the annotated image set; / 8. determine, based on a predetermined confidence interval, a predetermined target error ratio, and a predetermined interval width, a minimum number of inspections of the annotated image set; / 15. determining, based on a predetermined confidence interval, a predetermined target error ratio, and a predetermined interval width, a minimum number of inspections of the annotated image set; (Yang: [0036] Further, in order to ensure the quality of the quality check, a quality check person can perform a secondary quality check if the quality check result obtained by using the algorithm quality check has low confidence. [0047] The to-be-reviewed person is the annotation person selected from the candidate annotation persons and that needs to be paid close attention to. That is, compared with other persons among the candidate annotation persons, the to-be-reviewed person has a higher probability of error risk in annotated data. The number of to-be-reviewed persons may be one or more; in order to ensure the reasonableness of the number of pieces of to-be-reviewed data for subsequent review, the number of to-be-reviewed persons is further determined by a set review proportion, the number of pieces of data annotated by each candidate annotation person this time, the annotation person information of each candidate annotation person and the like. [0065] In S302, a quality check is performed on the to-be-reviewed data. [0092]-[0098] FIG. 6)
1. selecting a plurality of frames of the annotated image dataset for inspection based on the minimum number of inspections; / 8. select a plurality of frames of the annotated image dataset for inspection based on the minimum number of inspections; / 15. selecting a plurality of frames of the annotated image dataset for inspection based on the minimum number of inspections; (Yang: [0051] -[0052] In S202, data annotated by the to-be-reviewed person is selected from the set of annotated data as the to-be-reviewed data. [0053] -[0054], [0092] FIG. 6 is a flowchart of another data processing method according to an embodiment of the present disclosure. This embodiment of the present disclosure provides an example on the basis of the preceding embodiment. As shown in FIG. 6, the method includes steps described below. [0093] In S601, cleaning processing is performed on a set of to-be-annotated data. [0094] In S602, an annotation rule of the set of to-be-annotated data subjected to the cleaning processing is determined according to an annotation material type and an annotation scenario. [0095] In S603, in a process of annotating the set of to-be-annotated data, a quality check is performed on annotated data according to the annotation rule to obtain a set of annotated data. [0096] In S604, to-be-reviewed data is selected from the set of annotated data according to at least one of data annotation information or annotation person information. [0097] In S605, a quality check is performed on the to-be-reviewed data.)
1. and outputting the selected plurality of frames of the annotated image dataset for inspection. / 8. and output the selected plurality of frames of the annotated image dataset for inspection. / 15. and outputting the selected plurality of frames of the annotated image dataset for inspection. (Yang: [0097] In S605, a quality check is performed on the to-be-reviewed data. [0098] It is to be noted that in this embodiment, the whole process from the automatic data cleaning, the data annotation stage, the selection of to-be-reviewed data to the final quality check is essentially a complete set of whole-process automatic quality check process of data annotation. Further, the whole process involves three quality checks: firstly, automatic data cleaning is essentially also a quality check means; then the automatic quality check is performed based on the annotation rule in the data annotation stage, which greatly improves the data quality; and finally, in the quality check stage, the quality check can be performed on the extracted to-be-reviewed data through a combination of manual quality check and automatic quality check. The quality of finally output data can be ensured through three stages of quality checks, and a whole-process automatic quality check manner is provided for acquiring high-quality data.)
It would have been obvious before the effective filing date of the claimed invention to one of ordinary skill in the art to modify Vanska’s method and system for image analysis and inspection with Yang’s method and system for quality check of data processing. The determination of obviousness is predicated upon the following findings: One skilled in the art would have been motivated to modify the inspection method and system for comparative analysis of annotated image data as proposed by Vanska in order to leverage standardized metrics for qualitative analysis of annotated data as disclosed by Yang. Furthermore, the prior art collectively includes each element claimed (though not all in the same reference), and one of ordinary skill in the art could have combined the elements in the manner explained above using known engineering design, interface and programming techniques, without changing a “fundamental” operating principle of Vanska, while the teaching of Yang continues to perform the same function as originally taught prior to being combined, in order to produce the repeatable and predictable result of including the automation of quality checks and increase efficiency and quality. It is for at least the aforementioned reasons that the examiner has reached a conclusion of obviousness with respect to the claim in question.
Claims 2-7, 9-14, and 16-20 are rejected under 35 U.S.C. 103 as being unpatentable over Vanska et al. (US PGPub US20180039715A1), hereby referred to as “Vanska”, in view of Yang et al. (US PGPub US20220027854A1, hereby referred to as “Yang”, further in view of Cote et al. (US PGPub US20180248905A1), hereby referred to as “Cote”.
Consider Claims 2, 9 and 16.
The combination of Vanska and Yang teaches:
2. The method according to claim 1, wherein the outputting the selected plurality of frames for inspection comprises: comparing a current number of inspections with the minimum number of inspections; and based on determining that the current number of inspections is less than the minimum number of inspections: outputting an indication of insufficient inspections. / 9. The apparatus according to claim 8, wherein the at least one processor is further configured to execute the computer-executable instructions to output the selected plurality of frames for inspection by: comparing a current number of inspections with the minimum number of inspections; and based on determining that the current number of inspections is less than the minimum number of inspections: outputting an indication of insufficient inspections. / 16. The non-transitory computer-readable recording medium according to claim 15, wherein the outputting the selected plurality of frames for inspection comprises: comparing a current number of inspections with the minimum number of inspections; and based on determining that the current number of inspections is less than the minimum number of inspections: outputting an indication of insufficient inspections. (Vanska: [0056] Referring to FIG. 3, illustrated is a functional block diagram of a user terminal 302 in accordance with an embodiment of the present disclosure. The functional block diagram of the user terminal 302 comprises a terminal database 304, a tiled sub-image receiving module 306, a tiled sub-image rendering module 308, and an annotation module 310. These modules function as has been described above. [0065] A user terminal provides options to a user (e.g. an inspector of a construction site) to mark a position for annotation (e.g. a text, a voice, a photo, graphics overlaid on top of the drawing etc.) on raster images associated with a vector graphic version of a construction drawing. A coordinate of the marked position may be mapped to a real position on the vector graphic version of the construction drawing by calculating from known real centre coordinate of a display of the user terminal, zoom ratio (z), and a display related to distance vector from the touch coordinate to display centre. / Yang: [0049] Alternatively, a candidate annotation person whose historical annotation accuracy rate is less than a set accuracy rate may be regarded as a to-be-reviewed person; the candidate annotation persons may also be sorted in ascending order according to the historical annotation accuracy rate, and then the first one or more candidate annotation persons may be taken as the to-be-reviewed person; in order to ensure the reasonableness of the number of pieces of to-be-reviewed data for subsequent review, the to-be-reviewed person may be further selected from the candidate annotation persons according to the sorting result, the set review proportion, the number of pieces of data annotated by each candidate annotation person this time and the like. [0050] Alternatively, a candidate annotation person whose time of engagement in annotation is less than a set time value and whose historical annotation accuracy rate is less than a set accuracy rate is regarded as a to-be-reviewed person. The candidate annotation persons may also be sorted according to data of two dimensions, that is, the time of engagement in annotation and the historical annotation accuracy rate; and then the to-be-reviewed person may be selected from the candidate annotation persons according to the sorting result, the set review proportion, the number of pieces of data annotated by each candidate annotation person this time and the like. Figure 6, [0092]-[0098])
The combination of Vanska and Yang does not teach: based on determining that the current number of inspections is less than the minimum number of inspections: outputting an indication of insufficient inspections; and continuing the outputting of the selected plurality of frames until the number of frames inspected is equal to or greater than the minimum number of inspections.
Cote teaches:
2. The method according to claim 1, wherein the outputting the selected plurality of frames for inspection comprises: comparing a current number of inspections with the minimum number of inspections; and based on determining that the current number of inspections is less than the minimum number of inspections: outputting an indication of insufficient inspections; and continuing the outputting of the selected plurality of frames until the number of frames inspected is equal to or greater than the minimum number of inspections. / 9. The apparatus according to claim 8, wherein the at least one processor is further configured to execute the computer-executable instructions to output the selected plurality of frames for inspection by: comparing a current number of inspections with the minimum number of inspections; and based on determining that the current number of inspections is less than the minimum number of inspections: outputting an indication of insufficient inspections; and continuing the outputting of the selected plurality of frames until the number of frames inspected is equal to or greater than the minimum number of inspections. / 16. The non-transitory computer-readable recording medium according to claim 15, wherein the outputting the selected plurality of frames for inspection comprises: comparing a current number of inspections with the minimum number of inspections; and based on determining that the current number of inspections is less than the minimum number of inspections: outputting an indication of insufficient inspections; and continuing the outputting of the selected plurality of frames until the number of frames inspected is equal to or greater than the minimum number of inspections. (Cote: abstract, Using Testing Datasets to Validate the Performance of Trained Classifiers
[0069] After the ML training using labeled datasets, the development of the classifier techniques is completed. Before deploying in production, it is possible to measure the performance of the classifier 218 in diverse conditions with independent testing datasets. Through this procedure, the classifier 218 performance can be characterized by a number of standard metrics described below. [0070] The accuracy of the classifiers 218, thereby their ability to correctly predict anomalies in the network 120 is evaluated using 10× cross-validation. This validation approach is widely used in machine-learning and includes:
1. Training a classifier using 90% of the labeled dataset randomly selected,
2. Evaluating classifier using the remaining 10%, and
3. Repeating 10 times and report consolidated results.
[0074] As anomalies are rare events and the dataset is unbalanced, reporting the overall accuracy of the classifiers is not sufficient. The following Key Performance Indicators (KPIs) are provided: Overall accuracy; Precision (probability that a predicted anomaly is a real anomaly); Recall (probability that an anomaly is detected when it occurs); and F1-score (harmonic mean of precision and recall), [0079]-[0088] Deployment, Unlabeled Training Data [0089] Again, the anomaly detection software aims to detect abnormal behaviors in telecommunications network elements with a software application connected to a network data acquisition system. The data acquisition system has access to multiple Performance Monitoring (PM) metrics that characterize the network's behavior in a comprehensive way. The anomaly detection software performs statistical comparisons of given network snapshots with respect to one or several reference data sample(s). In another embodiment, the anomaly detection software can operate with unlabeled training data. The previous approach required a human expert to classify the inputs 202, 204 into normal and anomaly datasets 206, 208. This unlabeled approach omits this step. The approach is remarkable for two main reasons: 1) it works with “unlabeled” reference data samples that only require minimal preparation and 2) it exploits information from multiple metrics and reduces it to a single statistically sound probability (a.k.a. p-value). Here, the anomaly detection software can be seen as a specialized anomaly detection application for telecommunications networks based on unsupervised Machine Learning (ML).)
It would have been obvious before the effective filing date of the claimed invention to one of ordinary skill in the art to modify the combination of Vanska and Yang for a method and system for automated image analysis and inspection of annotated image data with Cote’s method and system for statistical analysis using machine learning for performance monitoring and statistical metrics. The determination of obviousness is predicated upon the following findings: One skilled in the art would have been motivated to modify the inspection method and system for qualitative analysis of annotated image data as proposed by the combination of Vanska and Yang in order to leverage Cote’s established and standardized statistical metrics and algorithms for ensuring accuracy in the assessment of the ML algorithms for annotated image data. Furthermore, the prior art collectively includes each element claimed (though not all in the same reference), and one of ordinary skill in the art could have combined the elements in the manner explained above using known engineering design, interface and programming techniques, without changing a “fundamental” operating principle of the combination of Vanska and Yang, while the teaching of Cote continues to perform the same function as originally taught prior to being combined, in order to produce the repeatable and predictable result of leveraging established statistical metrics in order to assess the quality and efficiency of the automation of quality checks. It is for at least the aforementioned reasons that the examiner has reached a conclusion of obviousness with respect to the claim in question.
Consider Claims 3, 10 and 17. They are rejected for the same reason as Claims 2, 9 and 16 above.
The combination of Vanska and Yang teaches:
3. The method according to claim 1, further comprising: determining, based on the inspection, a number of errors; and calculating, based on the number of errors, a probability / 10. The apparatus according to claim 8, wherein the at least one processor is further configured to execute the computer-executable instructions to: determine, based on the inspection, a number of errors; and calculate, based on the number of errors, a probability / 17. The non-transitory computer-readable recording medium according to claim 15, wherein the method further comprises: determining, based on the inspection, a number of errors; and calculating, based on the number of errors, a probability (Yang: [0047] The to-be-reviewed person is the annotation person selected from the candidate annotation persons and that needs to be paid close attention to. That is, compared with other persons among the candidate annotation persons, the to-be-reviewed person has a higher probability of error risk in annotated data. The number of to-be-reviewed persons may be one or more; in order to ensure the reasonableness of the number of pieces of to-be-reviewed data for subsequent review, the number of to-be-reviewed persons is further determined by a set review proportion, the number of pieces of data annotated by each candidate annotation person this time, the annotation person information of each candidate annotation person and the like. Figure 4, [0068] In S401, first to-be-reviewed data is selected from a set of annotated data according to at least one of data annotation information or annotation person information. [0069] In an embodiment, according to a first set review proportion, part of the annotated data may be selected from the set of annotated data as the first to-be-reviewed data according to at least one of the data annotation information and the annotation person information. [0070] In S402, second to-be-reviewed data is selected from the set of annotated data according to a data attention degree of a user.)
The combination of Vanska and Yang does not teach: and calculating, based on the number of errors, a sample error ratio
Cote teaches:
3. The method according to claim 1, further comprising: determining, based on the inspection, a number of errors; and calculating, based on the number of errors, a sample error ratio. / 10. The apparatus according to claim 8, wherein the at least one processor is further configured to execute the computer-executable instructions to: determine, based on the inspection, a number of errors; and calculate, based on the number of errors, a sample error ratio. / 17. The non-transitory computer-readable recording medium according to claim 15, wherein the method further comprises: determining, based on the inspection, a number of errors; and calculating, based on the number of errors, a sample error ratio.(Cote: [0049] While optimized, classifiers 218 cannot perfectly detect anomalies in a network for a variety of reasons, i.e., it is usually not possible to achieve 100% precision and 100% sensitivity: there is a tradeoff between precision and sensitivity, resulting in false negatives and false positives. Network operators have different requirements in terms of precision/sensitivity. To accommodate those various needs, regression techniques are employed to produce the actual output of the machine-learning algorithm, that is, the algorithm outputs a floating number between 0 (normal behavior) and 1 (abnormal behavior). FIG. 3 is a graph of a regression output from the Random Forest algorithm on a sample dataset for massError==0 (normal behavior) and massError=1 (abnormal behavior) in a test sample. [0050] The final binary classifier is obtained by thresholding the regression output. The choice of the threshold greatly impacts the final precision and sensitivity of the algorithm. Standard Receiver Operating Characteristic (ROC) curves (FIG. 4 is a graph of ROC curves of several ML algorithms on a sample dataset) represent the sensitivity as a function of the false positive rate (i.e., 1-precision) and can be used to visualize the performance of the classifier 218 when the threshold varies between 0 and 1. A major benefit of analyzing the regression output of the algorithm using its ROC curve is that it empowers operators to tune the discriminating threshold to apply to the regression output depending on their precision/sensitivity requirements, thereby optimizing the costs to operate the network 120. This is a major feature of the systems and methods. [0051] The classifier 218 can optionally be trained to recognize the type of anomaly, which can be useful for root cause analysis. This is achieved by training multiple intermediate classifiers 218 as described above, each specialized to recognize one type of anomaly. The final classifier is obtained by combining the output of the intermediate classifiers 218. Alternatively, it is also possible to train a multi-class classifier 218. The anomaly detection software can leverage both approaches to detect multiple types of anomalies. At the end of this process, the trained ML model(s) are persisted to a storage unit so they can be used by external programs out-of-the-box, without needing to re-train.)
It would have been obvious before the effective filing date of the claimed invention to one of ordinary skill in the art to modify the combination of Vanska and Yang for a method and system for automated image analysis and inspection of annotated image data with Cote’s method and system for statistical analysis using machine learning for performance monitoring and statistical metrics. The determination of obviousness is predicated upon the following findings: One skilled in the art would have been motivated to modify the inspection method and system for qualitative analysis of annotated image data as proposed by the combination of Vanska and Yang in order to leverage Cote’s established and standardized statistical metrics and algorithms for ensuring accuracy in the assessment of the ML algorithms for annotated image data. Furthermore, the prior art collectively includes each element claimed (though not all in the same reference), and one of ordinary skill in the art could have combined the elements in the manner explained above using known engineering design, interface and programming techniques, without changing a “fundamental” operating principle of the combination of Vanska and Yang, while the teaching of Cote continues to perform the same function as originally taught prior to being combined, in order to produce the repeatable and predictable result of leveraging established statistical metrics in order to assess the quality and efficiency of the automation of quality checks. It is for at least the aforementioned reasons that the examiner has reached a conclusion of obviousness with respect to the claim in question.
Consider Claims 4, 11 and 18. They are rejected for the same reasons as Claims 3, 10 and 17 above.
The combination of Vanska, Yang and Cote teaches:
4. The method according to claim 3, further comprising: determining, based on the sample error ratio, whether the sample error ratio exceeds the target error ratio; and based on a determination that the sample error ratio exceeds the target error ratio, outputting an indication of bad quality. / 11. The apparatus according to claim 10, wherein the at least one processor is further configured to execute the computer-executable instructions to: determine, based on the sample error ratio, whether the sample error ratio exceeds the target error ratio; and based on a determination that the sample error ratio exceeds the target error ratio, output an indication of bad quality. / 18. The non-transitory computer-readable recording medium according to claim 17, further comprising: determining, based on the sample error ratio, whether the sample error ratio exceeds the target error ratio; and based on a determination that the sample error ratio exceeds the target error ratio, outputting an indication of bad quality. (Cote: [0049] While optimized, classifiers 218 cannot perfectly detect anomalies in a network for a variety of reasons, i.e., it is usually not possible to achieve 100% precision and 100% sensitivity: there is a tradeoff between precision and sensitivity, resulting in false negatives and false positives. Network operators have different requirements in terms of precision/sensitivity. To accommodate those various needs, regression techniques are employed to produce the actual output of the machine-learning algorithm, that is, the algorithm outputs a floating number between 0 (normal behavior) and 1 (abnormal behavior). FIG. 3 is a graph of a regression output from the Random Forest algorithm on a sample dataset for massError==0 (normal behavior) and massError=1 (abnormal behavior) in a test sample. [0050] The final binary classifier is obtained by thresholding the regression output. The choice of the threshold greatly impacts the final precision and sensitivity of the algorithm. Standard Receiver Operating Characteristic (ROC) curves (FIG. 4 is a graph of ROC curves of several ML algorithms on a sample dataset) represent the sensitivity as a function of the false positive rate (i.e., 1-precision) and can be used to visualize the performance of the classifier 218 when the threshold varies between 0 and 1. A major benefit of analyzing the regression output of the algorithm using its ROC curve is that it empowers operators to tune the discriminating threshold to apply to the regression output depending on their precision/sensitivity requirements, thereby optimizing the costs to operate the network 120. This is a major feature of the systems and methods. [0051] The classifier 218 can optionally be trained to recognize the type of anomaly, which can be useful for root cause analysis. This is achieved by training multiple intermediate classifiers 218 as described above, each specialized to recognize one type of anomaly. The final classifier is obtained by combining the output of the intermediate classifiers 218. Alternatively, it is also possible to train a multi-class classifier 218. The anomaly detection software can leverage both approaches to detect multiple types of anomalies. At the end of this process, the trained ML model(s) are persisted to a storage unit so they can be used by external programs out-of-the-box, without needing to re-train.)
Consider Claims 5, 12 and 19. They are rejected for the same reasons as Claims 4, 11 and 18 above.
The combination of Vanska, Yang and Cote teaches:
5. The method according to claim 4, wherein the outputting the indication of bad quality further comprises: providing a message to an operator that the quality needs to be improved. / 12. The apparatus according to claim 11, wherein the at least one processor is further configured to execute the computer-executable instructions to output the indication of bad quality by: providing a message to an operator that the quality needs to be improved. / 19. The non-transitory computer-readable recording medium according to claim 18, wherein the outputting the indication of bad quality further comprises: providing a message to an operator that the quality needs to be improved. (Cote: [0049] While optimized, classifiers 218 cannot perfectly detect anomalies in a network for a variety of reasons, i.e., it is usually not possible to achieve 100% precision and 100% sensitivity: there is a tradeoff between precision and sensitivity, resulting in false negatives and false positives. Network operators have different requirements in terms of precision/sensitivity. To accommodate those various needs, regression techniques are employed to produce the actual output of the machine-learning algorithm, that is, the algorithm outputs a floating number between 0 (normal behavior) and 1 (abnormal behavior). FIG. 3 is a graph of a regression output from the Random Forest algorithm on a sample dataset for massError==0 (normal behavior) and massError=1 (abnormal behavior) in a test sample. [0050] The final binary classifier is obtained by thresholding the regression output. The choice of the threshold greatly impacts the final precision and sensitivity of the algorithm. Standard Receiver Operating Characteristic (ROC) curves (FIG. 4 is a graph of ROC curves of several ML algorithms on a sample dataset) represent the sensitivity as a function of the false positive rate (i.e., 1-precision) and can be used to visualize the performance of the classifier 218 when the threshold varies between 0 and 1. A major benefit of analyzing the regression output of the algorithm using its ROC curve is that it empowers operators to tune the discriminating threshold to apply to the regression output depending on their precision/sensitivity requirements, thereby optimizing the costs to operate the network 120. This is a major feature of the systems and methods. [0051] The classifier 218 can optionally be trained to recognize the type of anomaly, which can be useful for root cause analysis. This is achieved by training multiple intermediate classifiers 218 as described above, each specialized to recognize one type of anomaly. The final classifier is obtained by combining the output of the intermediate classifiers 218. Alternatively, it is also possible to train a multi-class classifier 218. The anomaly detection software can leverage both approaches to detect multiple types of anomalies. At the end of this process, the trained ML model(s) are persisted to a storage unit so they can be used by external programs out-of-the-box, without needing to re-train.)
Consider Claims 6, 13 and 20. They are rejected for the same reason as Claims 2, 9 and 16 above.
The combination of Vanska and Yang teaches: The method according to Claim 1, The apparatus of Claim 8, and The non-transitory computer readable medium of Claim 15.
Cote teaches:
6. The method according to claim 1, wherein the predetermined confidence interval is one of 99%, 95%, or 90%. / 13. The apparatus according to claim 8, wherein the predetermined confidence interval is one of 99%, 95%, or 90%./ 20. The non-transitory computer-readable recording medium according to claim 15, wherein the predetermined confidence interval is one of 99%, 95%, or 90%.(Cote: [0046] The goal of training ML algorithms is to construct a classification function—also known as the classifier 218—that can recognize normal/abnormal behaviors by formally encoding human expert knowledge. Human-expert knowledge is communicated from the inputs 202, 204 in which relevant PM data is labeled as “normal” or “abnormal” by a human expert (described in additional detail herein). It is expected that anomalies in a real network are rare events. In order to be effective, the distribution of the anomalies in training set should thus be representative of the live PMs that are collected from the network 120 and used to detect the anomalies. This distribution yields an unbalanced dataset, for example where 95% of the instances describe a normal behavior, and 5% describe anomalies. [0049] While optimized, classifiers 218 cannot perfectly detect anomalies in a network for a variety of reasons, i.e., it is usually not possible to achieve 100% precision and 100% sensitivity: there is a tradeoff between precision and sensitivity, resulting in false negatives and false positives. Network operators have different requirements in terms of precision/sensitivity. To accommodate those various needs, regression techniques are employed to produce the actual output of the machine-learning algorithm, that is, the algorithm outputs a floating number between 0 (normal behavior) and 1 (abnormal behavior). FIG. 3 is a graph of a regression output from the Random Forest algorithm on a sample dataset for massError==0 (normal behavior) and massError=1 (abnormal behavior) in a test sample.)
It would have been obvious before the effective filing date of the claimed invention to one of ordinary skill in the art to modify the combination of Vanska and Yang for a method and system for automated image analysis and inspection of annotated image data with Cote’s method and system for statistical analysis using machine learning for performance monitoring and statistical metrics. The determination of obviousness is predicated upon the following findings: One skilled in the art would have been motivated to modify the inspection method and system for qualitative analysis of annotated image data as proposed by the combination of Vanska and Yang in order to leverage Cote’s established and standardized statistical metrics and algorithms for ensuring accuracy in the assessment of the ML algorithms for annotated image data. Furthermore, the prior art collectively includes each element claimed (though not all in the same reference), and one of ordinary skill in the art could have combined the elements in the manner explained above using known engineering design, interface and programming techniques, without changing a “fundamental” operating principle of the combination of Vanska and Yang, while the teaching of Cote continues to perform the same function as originally taught prior to being combined, in order to produce the repeatable and predictable result of leveraging established statistical metrics in order to assess the quality and efficiency of the automation of quality checks. It is for at least the aforementioned reasons that the examiner has reached a conclusion of obviousness with respect to the claim in question.
Consider Claims 7 and 14. They are rejected for the same reason as Claims 2, 9 and 16 above.
The combination of Vanska and Yang teaches: The method according to claim 1 and The apparatus according to Claim 8.
Cote teaches:
7. The method according to claim 1, wherein the predetermined confidence interval is based on one of a F-distribution or a T-distribution. / 14. The apparatus according to claim 8, wherein the predetermined confidence interval is based on one of a F-distribution or a T-distribution. (Cote: Using Testing Datasets to Validate the Performance of Trained Classifiers [0069] After the ML training using labeled datasets, the development of the classifier techniques is completed. Before deploying in production, it is possible to measure the performance of the classifier 218 in diverse conditions with independent testing datasets. Through this procedure, the classifier 218 performance can be characterized by a number of standard metrics described below.[0070] The accuracy of the classifiers 218, thereby their ability to correctly predict anomalies in the network 120 is evaluated using 10× cross-validation. This validation approach is widely used in machine-learning and includes:
1. Training a classifier using 90% of the labeled dataset randomly selected,
2. Evaluating classifier using the remaining 10%, and
3. Repeating 10 times and report consolidated results.
[0074] As anomalies are rare events and the dataset is unbalanced, reporting the overall accuracy of the classifiers is not sufficient. The following Key Performance Indicators (KPIs) are provided: Overall accuracy; Precision (probability that a predicted anomaly is a real anomaly); Recall (probability that an anomaly is detected when it occurs); and F1-score (harmonic mean of precision and recall),)
It would have been obvious before the effective filing date of the claimed invention to one of ordinary skill in the art to modify the combination of Vanska and Yang for a method and system for automated image analysis and inspection of annotated image data with Cote’s method and system for statistical analysis using machine learning for performance monitoring and statistical metrics. The determination of obviousness is predicated upon the following findings: One skilled in the art would have been motivated to modify the inspection method and system for qualitative analysis of annotated image data as proposed by the combination of Vanska and Yang in order to leverage Cote’s established and standardized statistical metrics and algorithms for ensuring accuracy in the assessment of the ML algorithms for annotated image data. Furthermore, the prior art collectively includes each element claimed (though not all in the same reference), and one of ordinary skill in the art could have combined the elements in the manner explained above using known engineering design, interface and programming techniques, without changing a “fundamental” operating principle of the combination of Vanska and Yang, while the teaching of Cote continues to perform the same function as originally taught prior to being combined, in order to produce the repeatable and predictable result of leveraging established statistical metrics in order to assess the quality and efficiency of the automation of quality checks. It is for at least the aforementioned reasons that the examiner has reached a conclusion of obviousness with respect to the claim in question.
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to TAHMINA N ANSARI whose telephone number is (571)270-3379. The examiner can normally be reached on IFP Flex - Monday through Friday 9 to 5.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, O' NEAL MISTRY can be reached on 313-446-4912. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
TAHMINA N. ANSARI
Examiner
Art Unit 2672
2672
March 29, 2026
/TAHMINA N ANSARI/Primary Examiner, Art Unit 2674
Prosecution Insights