DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Examiner’s Note
The instant application has a lengthy prosecution history and the examiner encourages the applicant to have an interview (telephonic or personal) with the examiner prior to filing a response to the instant office action. Also, prior to the interview the examiner encourages the applicant to present multiple possible claim amendments, so as to enable the examiner to identify claim amendments that will advance prosecution in a meaningful manner.
Continued Examination Under 37 CFR 1.114
The present application is being examined under the pre-AIA first to invent provisions. A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 03/30/2026 has been entered.
Response to Arguments/Amendments
Presented arguments have been fully considered, but are rendered moot in view of the new ground(s) of rejection necessitated by amendment(s) initiated by the applicant(s).
Claim Interpretation
Amended claim to exemplary claim 1, “initiating an alert autonomously in response to the at least one feature meeting a defined condition or a threshold associated with a failure mode determined by calculating a severity level based on multiple detected features across multiple images of a same object captured from different angles, wherein the severity level is consolidated using clustering analysis” seems to introduce some 112(b) issues. For the purpose of the new prior reference rejection, exemplary claim 1 is being interpreted, in light of the original disclosure, as follow:
The “multiple detected features across multiple images of a same object captured from different angles” seems to require “receiving images autonomously captured via multiple image capturing devices” (see first limitation of claim 1), wherein the number of “image capturing devices” = the number of “different angles” (i.e. the first limitation will be interpreted as “receiving multiple images of a same object autonomously captured via multiple image capturing devices from different angles”);
The “multiple detected features across multiple images” correspond to the number of feature identified in the second limitation (i.e. the second limitations will be interpreted as “identifying/detecting multiple features of the at least one object autonomously based on the set of inference data”);
The third limitation will be interpreted as “identifying multiple features of the same object autonomously based on the set of inference data”
The fourth and fifth limitations will be interpreted together as “initiating an alert autonomously in response to:
at least one of the multiple features meeting a defined condition; or
a severity level, computed based on at least one of the multiple features, meeting a severity threshold, wherein the severity level is consolidated using clustering analysis and wherein a failure mode is associated with said severity level”.
The above interpretation is consistent with ¶0022-23, ¶0038 and ¶0119 of the original disclosure. It is to be noted that the last limitation seems to consider either (a) “a defined condition” or (b) “a threshold associated with a failure mode determined by calculating a severity level based on multiple detected features across multiple images of a same object captured from different angles, wherein the severity level is consolidated using clustering analysis”. Therefore, Saeed Bakhshmand et al. [US 20240160194 A1: already of record] still anticipates claims 1, 5, 12, 14, 19 and 20 considering (a) alone (see Office Action mailed on 12/31/2025).
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1-20 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor, or for pre-AIA the applicant regards as the invention.
Regarding claims 1, 19 and 20: A broad range or limitation together with a narrow range or limitation that falls within the broad range or limitation (in the same claim) may be considered indefinite if the resulting claim does not clearly set forth the metes and bounds of the patent protection desired. See MPEP § 2173.05(c). In the present instance, exemplary claim 1 recites the broad recitation “at least one image capturing device”, “at least one feature of the at least one object”, and the claim also recites “multiple detected features” and “multiple images of a same object captured from different angles”(necessarily suggesting as many image capturing devices as angles) which is the narrower statement of the range/limitation. The claim(s) are considered indefinite because there is a question or doubt as to whether the feature introduced by such narrower language is (a) merely exemplary of the remainder of the claim, and therefore not required, or (b) a required feature of the claims.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries set forth in Graham v. John Deere Co., 383 U.S. 1, 148 USPQ 459 (1966), that are applied for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1, 5, 12, 14, 19 and 20 are rejected under AIA 35 U.S.C. 103(a) as being unpatentable over by Saeed Bakhshmand et al. [US 20240160194 A1: already of record] in view of Joo Yeon Chung et al. [US 20220253641 A1].
Regarding claim 1, Saeed teaches:
1. A method of autonomous visual inspection (i.e. A system and method for automated visual inspection is provided herein- Abstract), comprising:
receiving images autonomously captured via at least one image capturing device (i.e. The system 300 includes a camera 304. The camera 304 captures image data of a target article 306. The image data may include a single image or a plurality of images. The plurality of images (frames) may be captured by the camera 304 as a video. To image an area of the target article 306, the camera 304 and the target article 306 may move relative to one another. For example, the target article 306 may be rotated and a plurality of images captured by the camera 304 at different positions on the target article 306 to provide adequate inspection from multiple angles. The camera 304 may be configured to capture a plurality of frames, wherein each frame is taken at a respective target article position (e.g. if the target article 306 is rotating relative to the camera 304). The camera 304 may be a USB 3.0 camera or an internet protocol (“IP”) camera- ¶0086… The pipeline 500 includes an inspection image 502. The inspection image 502 comprises image data of a target article, such as target article 306 of FIG. 3, captured by a camera device (e.g. camera device 304 of FIG. 3). The inspection image 502 may be a single frame captured by the camera device 304. Generally, the pipeline 500 may be configured to receive a plurality of inspection images 502 for a given target article 306. In some embodiments, however, a single inspection image 502 may be sufficient- ¶0148);
identifying at least one object from the images autonomously based on a set of inference data (i.e. feature map 508- fig. 5) from a machine learning (ML) module (i.e. The inspection image 502 is provided to an object detection module 504. The object detection module 504 includes an object detection model for performing object detection tasks. The object detection model performs computer vision and image processing for detecting instances of semantic objects of a certain class in a digital image (inspection image). Generally, the object detection module 504 receives the inspection image as input, passes the inspection image through the object detection model (e.g. CNN), and generates object detection (OD) output data. The OD output data may be in the form of an annotated inspection image including metadata describing objects detected in the inspection image 502- ¶0150);
identifying at least one feature (i.e. The defect data may include a list of detected defects- ¶0156) of the at least one object autonomously based on the set of inference data (i.e. The object detection model is trained and configured to find defects in the inspection image 502. The object detection model may also act as a classifier having one class for all defect types. Images 502 including one or more defects may be labelled as “NG”. Images with no detected defects may be labelled “OK”- ¶0151… Inspection images 502 labelled as NG include one or more defects that have been identified by the object detection model and are passed to a defect classifier (e.g. defect classification module 520). The OD output data 518 for an NG inspection image may include the inspection image data and defect data. The defect data may be stored as annotation data or metadata of the inspection image 502.- ¶0153-0155); and
initiating an alert autonomously in response to the at least one feature meeting a defined condition or a threshold (i.e. a probability threshold which must be met in order for a given class assignment to be made- ¶0069) associated with a failure mode (i.e. Each output node in the output layer corresponds to a defect class associated with a defect type- ¶0168-170) determined by calculating a severity level (i.e. The defect classification module 520 is configured to assign a class label to a particular defect image based on the output of the activation function- ¶00169) based on multiple detected features across multiple images of a same object captured from different angles (i.e. The system 300 includes a camera 304. The camera 304 captures image data of a target article 306. The image data may include a single image or a plurality of images. The plurality of images (frames) may be captured by the camera 304 as a video. To image an area of the target article 306, the camera 304 and the target article 306 may move relative to one another. For example, the target article 306 may be rotated and a plurality of images captured by the camera 304 at different positions on the target article 306 to provide adequate inspection from multiple angles. The camera 304 may be configured to capture a plurality of frames, wherein each frame is taken at a respective target article position (e.g. if the target article 306 is rotating relative to the camera 304). The camera 304 may be a USB 3.0 camera or an internet protocol (“IP”) camera- ¶0086… i.e. In some cases, the defect classification module 520 may set a probability threshold which must be met in order for a given class assignment to be made. In some cases, the defect classification module 520 may assign a defect class label and include a confidence/probability level- ¶0169…Object tracking requires that unique defects be seen over multiple frames before the object tracker alerts the visual inspection system- ¶0376… The anomaly detection module 2240 is configured to compare the OD output data 2218 to the classification output data 2256 to determine whether an object (defect) detected by the object detection module 2204 corresponds to an anomalous defect determined by the golden sample module 2226. This may include the anomaly detection module 2240 determining whether the location data of the object detection defect and the location data of the anomalous defect meet a predetermined condition… Where the predetermined condition is met, the anomaly detection module 2240 may generate and assign a “confirmed defect” status or label to the defect- ¶0392).
However, Saeed does not teach explicitly:
wherein the severity level is consolidated using clustering analysis.
In the same field of endeavor, Joo teaches:
wherein the severity level (i.e. reference score) is consolidated using clustering analysis (i.e. A method performed by a computing device for clustering an image according to an embodiment of the present disclosure includes performing a first clustering on feature vectors of the plurality of images, and performing a second clustering for feature vectors belonging to some clusters that do not satisfy a reference score among clusters formed as a result of the first clustering, wherein a clustering parameter of the second clustering and a clustering parameter of the first clustering are different from each other- Abstract… wherein the reference score is a value determined based on an attribute of a feature vector included in a cluster- ¶0029… According to another aspect of the present disclosure, a method performed by a computing device for inspecting an abnormal circuit patter comprising, performing clustering on a plurality of images, in which a circuit pattern is photographed, while repeatedly performing clustering based on a changed clustering parameter for an image included in a cluster that does not satisfy a reference score until all clusters formed as a result of preforming clustering satisfy the reference score and detecting a defect in a circuit pattern corresponding to the cluster based on a difference between a sample image included in a cluster formed based on the step of repeatedly performing the clustering and a predetermined standard image- ¶0031).
It would have been obvious to one with ordinary skill in the art before the effective filing date of the claimed invention, to modify the teachings of Saeed with the teachings of Joo to improve the classification speed for circuit pattern images (Joo- ¶0009).
Regarding claim 5, Saeed and Joo teach all the limitations of claim 1 and Saeed further teaches:
further comprising: monitoring the at least one object or the at least one feature for a period of time (i.e. The object tracking process includes tracking objects that are detected using a computer-implemented object detection process across multiple images or frames of a target article. For example, the target article may be imaged a plurality of times (e.g. a video comprising a plurality of frames) as the target article is moved relative to the camera. In an embodiment, the tracked objects are defects in the target article detected by an object detection component of the visual inspection system (e.g. object detection component 314 of FIG. 3)- ¶0309).
Regarding claim 12, Saeed and Joo teach all the limitations of claim 1 and Saeed further teaches:
wherein the identifying the at least one object from the images autonomously based on the set of inference data further comprises: classifying the at least one object as an object of interest from a plurality of objects in the images based on the set of inference data (i.e. The worker node device 310 includes an object detection component 314. The object detection component 314 is configured to perform an object detection process on the image data to detect defects (objects) in the image data. Generally, the object detection process determines whether a defect (or defects) is present in the image data or not- ¶0114… The inspection image 502 is provided to an object detection module 504. The object detection module 504 includes an object detection model for performing object detection tasks. The object detection model performs computer vision and image processing for detecting instances of semantic objects of a certain class in a digital image (inspection image). Generally, the object detection module 504 receives the inspection image as input, passes the inspection image through the object detection model (e.g. CNN), and generates object detection (OD) output data. The OD output data may be in the form of an annotated inspection image including metadata describing objects detected in the inspection image 502.- ¶0150).
Regarding claim 14, Saeed and Joo teach all the limitations of claim 1 and Saeed further teaches:
further comprising:
receiving sensing information or surrounding information associated with the at least one object from at least one (i.e. camera 304- ¶0148) sensor(i.e. The system 300 includes a camera 304. The camera 304 captures image data of a target article 306. The image data may include a single image or a plurality of images. The plurality of images (frames) may be captured by the camera 304 as a video. To image an area of the target article 306, the camera 304 and the target article 306 may move relative to one another. For example, the target article 306 may be rotated and a plurality of images captured by the camera 304 at different positions on the target article 306 to provide adequate inspection from multiple angles. The camera 304 may be configured to capture a plurality of frames, wherein each frame is taken at a respective target article position (e.g. if the target article 306 is rotating relative to the camera 304). The camera 304 may be a USB 3.0 camera or an internet protocol (“IP”) camera- ¶0086… The pipeline 500 includes an inspection image 502. The inspection image 502 comprises image data of a target article, such as target article 306 of FIG. 3, captured by a camera device (e.g. camera device 304 of FIG. 3). The inspection image 502 may be a single frame captured by the camera device 304. Generally, the pipeline 500 may be configured to receive a plurality of inspection images 502 for a given target article 306. In some embodiments, however, a single inspection image 502 may be sufficient- ¶0148);
wherein the identifying of the at least one object, the identifying of the at least one feature on the at least one object, or determining whether the at least one feature meets the defined condition or the threshold is further based on the sensing information or the surrounding information (i.e. In some cases, the defect classification module 520 may set a probability threshold which must be met in order for a given class assignment to be made- ¶0169).
Regarding claim 16, Saeed and Joo teach all the limitations of claim 1 and Saeed further teaches:
further comprising:
recording a condition of the at least one object in a database(i.e. The server node device 334 is also communicatively connected to an analytics database 342 via a communication link 344. The analytics database 342 stores analytics data as well as visual inspection output data from the worker node device 310 (e.g. defect data)- ¶0136);
tracking the condition of the at least one object over a period of time (i.e. The AI visual inspection device 12 may be a purpose-built machine designed specifically for performing object (e.g. defect) detection tasks, object (e.g. defect) classification tasks, golden sample analysis tasks, object (e.g. defect) tracking tasks, and other related data processing tasks using an inspection image captured by the camera device 14.- ¶0073); and updating the database based on the tracking (i.e. The data structure (or a subset of the data contained therein) may be used for analytics purposes, such as by providing the data structure to the server node 334 for storage in the analytics database 342.- ¶0394).
Regarding claim 19, computer-readable medium storing instructions claim 19 corresponds to the same method as claimed in claim 1, and therefore is also rejected for the same reasons of obviousness as listed above.
Regarding claim 20, apparatus claim 20 is drawn to the apparatus using/performing the same method as claimed in claim 1. Therefore, apparatus claim 20 corresponds to method claim 1 and is rejected for the same reasons of obviousness as used above.
Claim 2 and 8 are rejected under 35 U.S.C. 103 as being unpatentable over Saeed Bakhshmand et al. [US 20240160194 A1: already of record] in view of Joo Yeon Chung et al. [US 20220253641 A1] and further in view of William R. Buras et al. [US 20180225993 A1: already of record].
Regarding claim 2, Saeed and Joo teach all the limitations of claim 1.
However, Saeed and Joo do not teach explicitly:
further comprising:
deduplicating one or more images from the images captured via the at least one image capturing device.
In the same field of endeavor, William teaches:
further comprising:
deduplicating one or more images from the images captured via the at least one image capturing device(i.e. images that are down to exist both in the training set and the validation set are likewise removed (i.e., the image evaluation module computes SSI values for each image in the training set against each image in the validation set, and removes duplicate or near-duplicate images from one of the training and validation sets)- ¶0097).
It would have been obvious to one with ordinary skill in the art before the effective filing date of the claimed invention, to modify the teachings of Saeed and Joo with the teachings of William to allow the neural network to more accurately classify images in the validation set, since the chance of overfitting the neural network during training to a majority of identical images is reduced or eliminated (William- ¶0097).
Regarding claim 8, Saeed and Joo teach all the limitations of claim 1.
However, Saeed and Joo do not teach explicitly:
further comprising:
applying a de-duplication process to the images to remove one or more images with a same image context.
In the same field of endeavor, William teaches:
further comprising:
applying a de-duplication process to the images to remove one or more images with a same image context (i.e. images that are down to exist both in the training set and the validation set are likewise removed (i.e., the image evaluation module computes SSI values for each image in the training set against each image in the validation set, and removes duplicate or near-duplicate images from one of the training and validation sets)- ¶0097).
It would have been obvious to one with ordinary skill in the art before the effective filing date of the claimed invention, to modify the teachings of Saeed and Joo with the teachings of William to allow the neural network to more accurately classify images in the validation set, since the chance of overfitting the neural network during training to a majority of identical images is reduced or eliminated (William- ¶0097).
Claim 3 is rejected under 35 U.S.C. 103 as being unpatentable over Saeed Bakhshmand et al. [US 20240160194 A1: already of record] in view of Joo Yeon Chung et al. [US 20220253641 A1] and further in view of Alvise Memo et al. [US 20180322623 A1: already of record].
Regarding claim 3, Saeed and Joo teach all the limitations of claim 1.
However, Saeed and Joo do not teach explicitly:
wherein the at least one feature includes a severity condition, a failure mode, a remaining useful life, or a combination thereof.
In the same field of endeavor, Alvise teaches:
wherein the at least one feature includes a severity condition, a failure mode, a remaining useful life, or a combination thereof (i.e. As another example, in the quality inspection process for car seats in a production line, multiple possible defect classes may be defined, including: wrinkles at panels or at seams; puckers at seams; knuckles or waves at the zipper sew; bumps on side panels; bagginess in trims; bad seam alignment; misaligned panels; and gaps on zippers or between adjoining parts. In addition, defects may exist in the fabric material itself or on its installation, such as visible needle holes, hanging threads, loop threads, frayed threads, back tacks, bearding, and misaligned perforations. Some of these defects types can be quantified, and the measured quantities may be used to determine whether a car seat is acceptable, requires fixing, or must be discarded. For example, one acceptance criterion could be that any given panel should have no more than two wrinkles of up to 40 mm in length and no more than 5 wrinkles up to 25 mm in length. Other criteria may involve the maximum gap at a zipper or the maximum depth of a seam. The ability to quantify specific characteristics of a “defect” enables qualification of its severity. For example, based on displayed information about a detected and quantified defect, a quality assurance (QA) professional could mark a certain car seat as “moderately defective,” deferring the final decision about acceptance of this seat to a later time- ¶0068).
It would have been obvious to one with ordinary skill in the art before the effective filing date of the claimed invention, to modify the teachings of Saeed and Joo with the teachings of Alvise to automatically detect defects in objects and automatically classify and/or quantify the defects (Alvise- ¶0069).
Claim 15 is rejected under 35 U.S.C. 103 as being unpatentable over Saeed Bakhshmand et al. [US 20240160194 A1: already of record] in view of Joo Yeon Chung et al. [US 20220253641 A1] and further in view of Alvise Memo et al. [US 20180322623 A1: already of record] and even further in view of Jun Fujimoto [US 7258612 B2].
Regarding claim 15, Saeed and Joo teach all the limitations of claim 1.
However, Saeed and Joo do not teach explicitly:
further comprising: calculating a severity level automatically for the at least one feature using at least one of a physics model based algorithm, a rule based model algorithm, or a deep learning model based algorithrn, wherein the at least one feature meets the defined condition or the threshold including a failure mode when the severity level exceeds a severity threshold.
In the same field of endeavor, Alvise teaches:
further comprising: calculating a severity level automatically for the at least one feature using at least one of a physics model based algorithm, a rule based model algorithm, or a deep learning model based algorithrn (i.e. FIG. 7 is a block diagram of a convolutional neural network 310 according to one embodiment of the present invention. According to some embodiments of the present invention, a convolutional neural network (CNN) is used to process the synthesized 2-D views 16 to generate the defect classification of the object. Generally, a deep CNN processes an image by passing the input image data (e.g., a synthesized 2-D view) through a cascade of layers. These layers can be grouped into multiple stages. The deep convolutional neural network shown in FIG. 7 includes two stages, a first stage CNN.sub.1 made up of N layers (or sub-processes) and a second stage CNN.sub.2 made up of M layers. In one embodiment, each of the N layers of the first stage CNN1 includes a bank of linear convolution layers, followed by a point non-linearity layer and a non-linear data reduction layer. In contrast, each of the M layers of the second stage CNN2 is a fully connected layer. The output p of the second stage is a class-assignment probability distribution. For example, if the CNN is trained to assign input images to one of k different classes, then the output of the second stage CNN.sub.2 is an output vector p that includes k different values, each value representing the probability (or “confidence”) that the input image should be assigned the corresponding defect class (e.g., containing a tear, a wrinkle, discoloration or marring of fabric, missing component, etc.)- ¶0149), wherein the at least one feature meets the defined condition or the threshold including a failure mode when the severity level exceeds a severity threshold (i.e. The defect detector may be configured to compute at least one of the one or more defect classifications of the target object by: counting or measuring the shapes of the one or more features of the 3-D model to generate at least one count or at least one measurement; comparing the at least one count or at least one measurement to a tolerance threshold; and determining the at least one of the one or more defect classifications as being present in the target object in response to determining that the at least one count or at least one measurement is outside the tolerance threshold- ¶0014 &¶0028… As another example, in the quality inspection process for car seats in a production line, multiple possible defect classes may be defined, including: wrinkles at panels or at seams; puckers at seams; knuckles or waves at the zipper sew; bumps on side panels; bagginess in trims; bad seam alignment; misaligned panels; and gaps on zippers or between adjoining parts. In addition, defects may exist in the fabric material itself or on its installation, such as visible needle holes, hanging threads, loop threads, frayed threads, back tacks, bearding, and misaligned perforations. Some of these defects types can be quantified, and the measured quantities may be used to determine whether a car seat is acceptable, requires fixing, or must be discarded. For example, one acceptance criterion could be that any given panel should have no more than two wrinkles of up to 40 mm in length and no more than 5 wrinkles up to 25 mm in length. Other criteria may involve the maximum gap at a zipper or the maximum depth of a seam. The ability to quantify specific characteristics of a “defect” enables qualification of its severity. For example, based on displayed information about a detected and quantified defect, a quality assurance (QA) professional could mark a certain car seat as “moderately defective,” deferring the final decision about acceptance of this seat to a later time- ¶0068, ¶0177, ¶0186).
It would have been obvious to one with ordinary skill in the art before the effective filing date of the claimed invention, to modify the teachings of Saeed and Joo with the teachings of Alvise to automatically detect defects in objects and automatically classify and/or quantify the defects (Alvise- ¶0069).
However, Saeed, Joo and Alvise do not teach explicitly:
triggering the at least one image capturing device to capture additional images of the at least one feature for further investigation in response to the severity level exceeding the severity threshold.
In the same field of endeavor, Jun teaches:
triggering the at least one image capturing device to capture additional images of the at least one feature for further investigation in response to the severity level exceeding the severity threshold (i.e. when the threshold anomaly level has been exceeded, a captured image of the person, wherein the computer includes a processor programmed so that, in response to assigning the first of the anomaly levels to a disagreement, to permit completion of the associated non-fraudulent transaction, in response to assigning the second of the anomaly levels to a disagreement, to trigger the camera to capture an image of the person attempting the associated potentially fraudulent transaction and to record transaction information for later review, and in response to assigning the third of the anomaly levels to a disagreement, to trigger the camera to capture an image of the person attempting the associated fraudulent transaction, to record transaction information, and to trigger an alarm for immediate response- claim 1).
It would have been obvious to one with ordinary skill in the art before the effective filing date of the claimed invention, to modify the teachings of Saeed, Joo and Alvise with the teachings of Jun to provide greatly improved monitoring against wrong or fraudulent acts (Jun- Abstract).
Claim 4 is rejected under 35 U.S.C. 103 as being unpatentable over Saeed Bakhshmand et al. [US 20240160194 A1: already of record] in view of Joo Yeon Chung et al. [US 20220253641 A1] and further in view of Florian Hillen [US 20190313963 A1: already of record].
Regarding claim 4, Saeed and Joo teach all the limitations of claim 1 and Saeed further teaches:
further comprising:
annotating the at least one object or the at least one feature identified on the images via the ML module(i.e. the output data may include an annotated inspection image including artifact data. The artifact data may include location information (e.g. coordinates, bounding box) and label information such that artifacts (e.g. defects, anomalies) in the inspection image that were identified by the worker node device 310 can be identified visually in a displayed image- ¶0128).
However, Saeed and Joo do not teach explicitly:
reviewing and reannotating one or more images from the images based on a set of inferenced images; and retraining the ML module based on the one or more images.
In the same field of endeavor, Florian teaches:
reviewing and reannotating one or more images from the images based on a set of inferenced images; and retraining the ML module based on the one or more images (i.e. In some arrangements feedback data 522, which can come from various sources e.g. detected features from a previous machine learning model which output has been clinically validated or validated by another annotator, dental clinician or user, can also be provided to the machine learning trainer to further improve training- ¶0031, figs. 4-5).
It would have been obvious to one with ordinary skill in the art before the effective filing date of the claimed invention, to modify the teachings of Saeed and Joo with the teachings of Florian to further improve training (Florian- ¶0031).
Regarding claim 13, Saeed and Joo teache all the limitations of claim 1 and Saeed further teaches:
further comprising:
a wherein the identifying the at least one feature of the at least one object autonomously further comprises: generating an annotation for the at least one object for the at least one on the images autonomously based on the set of inference data (i.e. the output data may include an annotated inspection image including artifact data. The artifact data may include location information (e.g. coordinates, bounding box) and label information such that artifacts (e.g. defects, anomalies) in the inspection image that were identified by the worker node device 310 can be identified visually in a displayed image- ¶0128).
However, Saeed and Joo do not teach explicitly:
identifying whether the annotation for the at least one object is erroneous; and updating a set of training data based on the annotation for the at least one object being identified as erroneous for retraining the ML module.
In the same field of endeavor, Florian teaches:
identifying whether the annotation for the at least one object is erroneous; and updating a set of training data based on the annotation for the at least one object being identified as erroneous for retraining the ML module (i.e. In some arrangements feedback data 522, which can come from various sources e.g. detected features from a previous machine learning model which output has been clinically validated or validated by another annotator, dental clinician or user, can also be provided to the machine learning trainer to further improve training- ¶0031, figs. 4-5).
It would have been obvious to one with ordinary skill in the art before the effective filing date of the claimed invention, to modify the teachings of Saeed and Joo with the teachings of Florian to further improve training (Florian- ¶0031).
Claims 6-7 are rejected under 35 U.S.C. 103 as being unpatentable over Saeed Bakhshmand et al. [US 20240160194 A1: already of record] in view of Brent Ronald Frei et al. [US 20200020093 A1: already of record].
Regarding claim 6, Saeed and Joo teach all the limitations of claim 1.
However, Saeed and Joo do not teach explicitly:
wherein the at least one image capturing device is allocated on at least one drone that is configured to identify at least a location of the at least one object based on a geographic information system (GIS).
In the same field of endeavor, Brent teaches:
wherein the at least one image capturing device is allocated on at least one drone that is configured to identify at least a location of the at least one object based on a geographic information system (GIS) (i.e. Accordingly, a GPS captured location 140b of the drone 105 may be different from that actual image location 144b, which in this illustration is off to the right of the drone location 140b. As described in more detail herein, avionic telemetry information of the drone and the drone's location 140b at a time of image capture can be utilized to determine the actual image location 144b of the image 170. Once the location 144b of the image is determined, a position of object 150 in the image 170 can be translated to an actual position of the object 150 within the target geographical area- ¶0120).
It would have been obvious to one with ordinary skill in the art before the effective filing date of the claimed invention, to modify the teachings of Saeed and Joo with the teachings of Brent to improve accuracy for areas of interest, accommodate for different resolutions, utilize different sensors that may operate more efficiently at different heights, or other factor (Brent- ¶0181).
Regarding claim 7, Saeed, Joo and Brent teach all the limitations of claim 6 and Saeed further teaches:
wherein the identifying of the at least one object or the identifying of the at least one feature on the at least one object is based on the location of the at least one object (i.e. the anomaly detection 540 may be configured to accept and confirm items having approximately the same location, which may be determined by an acceptable pixel amount variance. In other embodiments, the items may need to have identical locations in order for the anomaly detection module 540 to accept or confirm the defects- ¶0260).
Claim 9 is rejected under 35 U.S.C. 103 as being unpatentable over Saeed Bakhshmand et al. [US 20240160194 A1: already of record] in view of Joo Yeon Chung et al. [US 20220253641 A1] and further in view of Shriram Ganesh et al. [US 20160068264 A1: already of record].
Regarding claim 9, Saeed and Joo teach all the limitations of claim 1.
However, Saeed and Joo do not teach explicitly:
wherein the at least one image capturing device is allocated on at least one drone that is configured to use a landmark-based anchor to determine a surveillance approach and route for capturing the images of the at least one object.
In the same field of endeavor, Shriram teaches:
wherein the at least one image capturing device is allocated on at least one drone that is configured to use a landmark-based anchor to determine a surveillance approach and route for capturing the images of the at least one object (i.e. The drone may use location information associated with the source of the alternate signals together with additional information (e.g., dead-reckoning in combination with last trusted GNSS location, dead-reckoning in combination with location of the drone truck or take-off zone, etc.) for positioning and navigation. Thus, in some embodiments the drone may navigate to the delivery destination using a combination of navigation techniques, including dead-reckoning, camera-based recognition of the land features below the drone (e.g. recognizing a road, landmarks, highway signage, etc), etc. that may be used instead of or in combination with GNSS location determination and triangulation off known locations of detected wireless access points. In some embodiments, a portable landing pad with optical features, a radio beacon, or other landing aids, may be set up at the landing site by the purchaser, or the person receiving the package, to further guide the drone to its destination and landing- ¶0031 & ¶0102).
It would have been obvious to one with ordinary skill in the art before the effective filing date of the claimed invention, to modify the teachings of Saeed and Joo with the teachings of Shriram in case GNSS signals may become “jammed,” “hacked,” or otherwise interfered with(Shriram- ¶0031 &0102).
Claim 10 is rejected under 35 U.S.C. 103 as being unpatentable over Saeed Bakhshmand et al. [US 20240160194 A1: already of record] in view of Joo Yeon Chung et al. [US 20220253641 A1] and further in view of Andrea Britto Mattos Lima et al. [US 20190258883 A1: already of record].
Regarding claim 10, Saeed and Joo teach all the limitations of claim 1.
However, Saeed and Joo do not teach explicitly:
wherein the at least one image capturing device is configured to verify a quality of the images captured for the at least one object and take additional images of the at least one object if the quality does not meet a quality threshold or a set of defined criteria.
In the same field of endeavor, Andrea teaches:
wherein the at least one image capturing device is configured to verify a quality of the images captured for the at least one object and take additional images of the at least one object if the quality does not meet a quality threshold or a set of defined criteria (i.e. The hardware processor controls the drone such that the drone captures the next image while the region is not fully mapped or there is sufficient battery capacity. If the area is fully mapped but there is still remaining battery time, the hardware processor may compute an optimized route for the drone to revisit the regions with lower confidence scores and re-capture images with higher resolution, e.g., capture images again at a lower altitude- ¶0037).
It would have been obvious to one with ordinary skill in the art before the effective filing date of the claimed invention, to modify the teachings of Saeed and Joo with the teachings of Andrea to optimizing a process of recapturing pictures of regions with lower confidence (Andrea- ¶0018).
Claim 11 is rejected under 35 U.S.C. 103 as being unpatentable over Saeed Bakhshmand et al. [US 20240160194 A1: already of record] in view of Joo Yeon Chung et al. [US 20220253641 A1] and further in view of Michael B. Wallace et al. [US 20200000362 A1: already of record].
Regarding claim 11, Saeed and Joo teach all the limitations of claim 1.
However, Saeed and Joo do not teach explicitly:
further comprising: training the ML module based on a previous classification or detection ML models using a transfer learning based approach, and based on a set of annotated images being approved to create the set of inference data.
In the same field of endeavor, Michael teaches:
further comprising: training the ML module based on a previous classification or detection ML models using a transfer learning based approach, and based on a set of annotated images being approved to create the set of inference data (i.e. In some embodiments, the mechanisms described herein can be used to generate inflated pre-trained neural networks that can be trained (e.g., using transfer learning techniques) to classify images from a relatively small training dataset (e.g., on the order of dozens to hundreds of labeled examples). For example, the mechanisms described herein can be used to generate an inflated version of an Inception-based convolution neural network (e.g., based on Inception V3 described in Szegedy et al., “Rethinking inception architecture for computer vision,” Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2818-2826 (2016), which is hereby incorporated by reference herein in its entirety) that has been pre-trained (e.g., based on ImageNet images) as a general image classification CNN. As another example, the mechanisms described herein can be used to generate an inflated version of a DenseNet-based convolution neural network (e.g., based on DenseNet121 described in Huang et al., “Densely connected convolutional neural networks,” Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4700-4708 (2017), which is hereby incorporated by reference herein in its entirety) that has been pre-trained (e.g., based on ImageNet images) as a general image classification CNN- ¶0124, ¶0130 & ¶0142).
It would have been obvious to one with ordinary skill in the art before the effective filing date of the claimed invention, to modify the teachings of Saeed and Joo with the teachings of Michael to support transfer learning, as the lowest-level layers can generate useful features across for many different types of images, while higher-level layers can become more specialized to the specific training data (Michael- ¶0123).
Claim 17 is rejected under 35 U.S.C. 103 as being unpatentable over Saeed Bakhshmand et al. [US 20240160194 A1: already of record] in view of Joo Yeon Chung et al. [US 20220253641 A1] and further in view of Ron Samuel Dembo [US 20130069949 A1: already of record].
Regarding claim 17, Saeed and Joo teach all the limitations of claim 1.
However, Saeed and Joo do not teach explicitly:
wherein if the at least one object is associated with a meter, the method further comprises:
capturing meter images via the at least one image capturing device; identifying a reading of the meter based on the meter images; recording the reading of the meter in a database; periodically tracking readings associated with the meter over a period of time; and updating the database based on the tracking.
In the same field of endeavor, Ron teaches:
wherein if the at least one object is associated with a meter, the method further comprises:
capturing meter images via the at least one image capturing device; identifying a reading of the meter based on the meter images(i.e. Input system 206 is part of a class of components for receiving data input by server 104. Input system 206 may a keyboard, mouse device, trackball, touchpad, touch screen, microphone, accelerometer, lightpen or other device. Input system 206 may also include a component that receives data over network 102 or via application programming interfaces (APIs). Input system 206 may also include a component configured to receive electronic data from a monitoring device, such as a utility meter, smart meter, water meter, volt meter, amperage meter, light meter, wind meter, hydrometer, etc. Input system 206 may be an indirect data capturing system. For example, system 206 may be camera that records images of a meter, capturing changes in its readings. Processes may then be provided to analyse the images and extract the data readings from them- ¶0054); recording the reading of the meter in a database (i.e. ] Next at process 306, the data is processed and stored. This may involve tagging the data with a timestamp and source details and may involve storing the data in a certain section of database 110- ¶0063); periodically tracking readings associated with the meter over a period of time; and updating the database based on the tracking (i.e. The data may reflect instantaneous usage data, data gathered over a time period, change data, rate of change data, projection data of usage or other values. For instance, the data may indicate a rate of water usage every hour, on the hour, for a particular household, the data being a negative or positive value representing the difference in rate for the present point in time, as compared to an earlier point in time. Additional data that may be provided includes identifying information about the data source, e.g. serial number of device, street address for house for meter, size information for house, weather data for the day, utility prices for the day, etc.- ¶0062… In particular, an embodiment provides tracking of consumption data and generation of graphs and charts where a large data set is provided. For example, in real-time environments, consumption data (and/or other data) can be gathered and stored for a given site (or collection of sites). An embodiment provides graphical tools to show historic, current and projected levels of consumption. FIGS. 5-7 and their related descriptions provide a snapshot of a current level of consumption/use of the tracked item. The snapshot can be at a certain point in time, such as a current time, a specific previous time, a projected time or an average over a determined period of time. An embodiment provides additional data, in graphical form, where a range data of consumption levels over a period of time tracked in a dataset- ¶0105).
It would have been obvious to one with ordinary skill in the art before the effective filing date of the claimed invention, to modify the teachings of Saeed and Joo with the teachings of Ron such that readings for a site spanning a number of days/weeks/seasons can be normalized to unitized values to allow them to be more easily compared (Ron- ¶0072).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to CLIFFORD HILAIRE whose telephone number is (571)272-8397. The examiner can normally be reached 5:30-1400.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, SATH V PERUNGAVOOR can be reached at (571)272-7455. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
CLIFFORD HILAIRE
Primary Examiner
Art Unit 2488
/CLIFFORD HILAIRE/Primary Examiner, Art Unit 2488
