DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 11/21/2024 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Election/Restriction
I. Claim 23, drawn to training an object detection system classified in G06N 20
II. Claim 1-21, drawn to a generic image frame object detection system, classified in G06V 20
III. Claim 22 drawn to a method for detecting an object in a real time image stream, classified in H04N 21/2187
The inventions are independent or distinct, each from the other because:
Inventions I, II and III are related as combination and subcombinations. Inventions in this relationship are distinct if it can be shown that (1) the combination as claimed does not require the particulars of the subcombination as claimed for patentability, and (2) that the subcombination has utility by itself or in other combinations (MPEP § 806.05(c)). In the instant case, the combination as claimed does not require the particulars of the subcombinations as claimed because, in regards to Group I vs Group II, the training method of the combination Group I (Claim 23) is directed to training and retraining a network model using training data, user provided ground truth, loss function determination and retraining operations and does not require the particular object detection system recited in Group II (claim 1-21). The subcombination in Group II (claim 1-21) has separate utility such as detecting a targeted object within image data using Gaussian Mixture Model regardless of how the model was trained.
In regards to Group I vs. Group III, The combination as claimed in Group I (Claim 23) does not require the particulars of the subcombination as claimed in Group III (Claim 22) because the training method may be performed using training datasets, prerecorded imagery, previously annotated images and offline learning environments which does not require receiving real time image data from an image capturing device. The subcombination as claimed in Group III (claim 22) has separate utility such as real time surveillance, live monitoring, continuous image stream analytics and low latency object recognition/detection.
In regards to Group II vs Group III the subcombination as claimed in Group II (Claims 1-21) does not require the particulars of the subcombination as claimed in Group III (Claim 22) because the object detection system is broadly directed to detecting targeted objects from image data and does not require processing of real time image data. The subcombination as claimed in Group III (claim 22) has separate utility such as performing live surveillance, continuous image stream analytics and real time object detection from streamed data.
Restriction for examination purposes as indicated is proper because all the inventions listed in this action are independent or distinct for the reasons given above and there would be a serious search and/or examination burden if restriction were not required because one or more of the following reasons apply:
Group I (Claim 23) requires a search directed to, Machine learning training, active learning, ground truth generation and insertion, loss function optimization and retraining operations.
Group II (claims 1-21) requires search directed to, object detection systems , image analysis systems, Gaussian Mixture object detection architecture.
Group III (Claim 22) requires a search directed to real time image acquisition, streaming video analysis, live surveillance systems, reduced latency object detection, and continuous image stream processing and analytics.
Separate searches, separate classifications and separate search strategies as well as differing prior art would be needed for the examination of the three groups resulting in a search and examination burden.
The examiner has required restriction between combination and subcombination inventions. Where applicant elects a subcombination, and claims thereto are subsequently found allowable, any claim(s) depending from or otherwise requiring all the limitations of the allowable subcombination will be examined for patentability in accordance with 37 CFR 1.104. See MPEP § 821.04(a). Applicant is advised that if any claim presented in a divisional application is anticipated by, or includes all the limitations of, a claim that is allowable in the present application, such claim may be subject to provisional statutory and/or nonstatutory double patenting rejections over the claims of the instant application.
During a telephone conversation with attorney Spencer Carter on 5/27/2026 a provisional election was made without traverse to prosecute the invention of Group II , claims 1-21. Affirmation of this election must be made by applicant in replying to this Office action. Claims 22 and 23 are withdrawn from further consideration by the examiner, 37 CFR 1.142(b), as being drawn to a non-elected invention.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claims 1-4 and 17 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Wang et al (Wang hereinafter “Flying Small Target Detection for Anti-UAV Based on a Gaussian Mixture Model in a Compressive Sensing Domain”).
As per claim 1
Wang teaches object detection system, comprising: an image capture system configured to obtain image data comprising at least one image frame (Abstract: “images captured by stationary visual sensors are broken into patches and the candidate patches which perhaps contain targets are identified by using a Gaussian mixture background model in a compressive sensing domain.”), a memory storing instructions that, when executed by one or more processors (Section 4 Experimental Results and Analysis: “All the experiments are carried out on a general computer with 8-GB memory and 2.2-GHz Intel Core i5-5200U processor.”) cause the one or more processors to execute a network model configured to implement a Gaussian Mixture model (GMM) (Section 2. Gaussian Mixture Modeling in a compressive sensing domain “To tackle this problem, the Gaussian mixture model (GMM) is proposed as a statistical approach by training multiple models of probability density functions over each pixel…When a new image is captured at time t during model training, a series of linear projections with Gaussian random matrix is operated over the image vector “ Section 3. Approach to Flying Small Target Detection by Local Image Separation: “…candidate patches identification based on GMM_CS is depicted in Figure 1.”) to detect one or more targeted objects from a plurality of potential objects in the at least one image frame (Introduction :“candidate patches which perhaps contain targets that are identified by using background models of each patch. “ Section 3 Approach to Flying Small Target Detection by Local Image Separation: “the obtained background model based on GMM_CS can be employed to identify the candidate patches which perhaps maintain a flying small target. Thereinto, the identification of an image patch maintains a flying small target”) the one or more targeted objects comprising less than 1/100th pixels of a total number of pixels in the at least one image frame. (Figure 4, Introduction “the size of a target is often very small and with only a few pixels because of a long imaging distance” Section 4 Experiment Setting and Analysis “The image sequence for the first experiment has a 189-frame visible image with resolution 640×480 pixels…The image sequence for the second experiment has a 337-frame infrared image with resolution 640×480 pixels acquired “ A person of ordinary skill in the art is well aware that the targeted objects must be less than 1/100th of the 307,200 pixels being represented. Wang expresses that Micro-UAVs “occupies a small portion of the field of view from a far distance “ and that “ the size of a target is often very small and with only a few pixels because of a long imaging distance, “ Only a few pixels in an image of 307,200 is well below 3072 pixels.).
As per claim 2
Wang teaches all claim limitations previously rejected in claim 1’s 102 rejection. See claim 1’s 102 rejection.
Wang teaches wherein the GMM is configured to generate a Gaussian Receptive Fields (GRFs) (Figure 1) to dynamically adapt to diverse shapes and sizes of the one or more targeted objects in the at least one image frame (Introduction: “not only occupies a small portion of the field of view from a far distance and flies against complex backgrounds under various weather and lighting conditions, but also appears changing size and diverse shapes in fully 3D dimensional environment.” Section 4 Experimental Results and Analysis “Due to the changing flight attitude, near or far distance and variable lighting conditions, the shape and size of the drone is in flux and is challenging for detection…These results show that the proposed method can effectively detect the actual flying small target and generate less false alarm targets relative to the baseline methods.”)
As per claim 3
Wang teaches all claim limitations previously rejected in claim 2’s 102 rejection. See claim 2’s 102 rejection.
Wang teaches wherein execution of the instructions causes the one or more processors to divide the at least one image frame into a plurality of image tiles, (Figure 1) wherein the network model is configured to apply one or more of the GRFs to each of the plurality of image tiles independently to capture local details and variations in the one or more targeted objects in the at least one image frame. (Figure 1 and figure 3 , Section 3 Approach to Flying Small Target Detection by Local Image Separation “break the whole image into overlapping or non-overlapping patches by patch transform [36]. Assume a new image is represented by a matrix D the non-overlapping patch transform will convert the matrix into several block matrices with r rows and c columns…the background modeling and target detection procedures will be applied over image patches…establish a Gaussian mixture model for each linear measurement of image patches generated from a group of training images…first step is the comparison between each linear measurement of input image patch with the obtained background model by Equation (7); the second step is to compute the statistical ratio of matched linear measurements to the total linear measurements,)
As per claim 4
Wang teaches all claim limitations previously rejected in claim 4’s 102 rejection. See claim 1’s 102 rejection.
Wang teaches wherein execution of the instructions causes the one or more processors to divide the at least one image frame into a plurality of image tiles, to detect the one or more targeted objects in each of the plurality of image tiles (Figure 1 and Figure 3) the one or more targeted objects comprising less than a factor of each of the image tiles (Figure 4) , to aggregate the plurality of images tiles for the detection of the one or more targeted objects in the at least one image frame. (Figure 1 and figure 3).
In regards to “the factor being 1/100th multiplied by a number of image tiles of the plurality of tiles” This mathematical operation does not introduce a new detection operation. This simply mathematically redefines the already disclosed less than 1/100th of frame pixels introduced in claim 1 now after subdivision into tiles. Once Wang performs tiled patch detection on targets already constrained to less than 1/1100th of frame size, the claimed scaling factor necessarily results from the disclosed subdivision operation. Wang’s target detection is done after the tiles of the original image are made, the relative target size threshold scales proportionally with the number of tiles. The claimed factor is an inherent mathematical effect of the tile image processing and small target detection operation. 1/100 multiplied by number of image tiles is not a new detection architecture, a new training operation or a processing stage. 1/100 multiplied by number of image tiles is a proportional normalization relationship after tiling.
As per claim 17
Wang teaches all claim limitations previously rejected in claim 1’s 102 rejection.
Wang teaches wherein each of the one or more targeted objects occupy less than 1/500th pixels than the total number of pixels in the at least one image frame. (Figure 4, Introduction “the size of a target is often very small and with only a few pixels because of a long imaging distance” Section 4 Experiment Setting and Analysis “The image sequence for the first experiment has a 189-frame visible image with resolution 640×480 pixels…The image sequence for the second experiment has a 337-frame infrared image with resolution 640×480 pixels acquired “ A person of ordinary skill in the art is well aware that the targeted objects must be less than 1/500th of the 307,200 pixels being represented. Wang expresses that Micro-UAVs “occupies a small portion of the field of view from a far distance “ and that “ the size of a target is often very small and with only a few pixels because of a long imaging distance, “ “Only a few pixels” in an image of 307,200 can be seen as below 614.4 pixels.)
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim 5 and 10 are rejected under 35 U.S.C. 103 as being unpatentable over Wang et al (Wang hereinafter “Flying Small Target Detection for Anti-UAV Based on a Gaussian Mixture Model in a Compressive Sensing Domain”) in view of Lee et al (Lee hereinafter KR 20240117181 A “Small Infiltration Drone Identification System In Short-wave Infrared Images”)
As per claim 5
Wang teaches all claim limitations previously rejected in claim 4’s 102 rejection. See claim 4’s 102 rejection.
Wang does not teach wherein execution of the instruction cause the one or more processors to alert a user responsive to the one or more targeted objects being detected in an image tile of the plurality of image tiles.
Lee teaches wherein execution of the instruction cause the one or more processors to alert a user responsive to the one or more targeted objects being detected in an image tile of the plurality of image tiles. (Detailed description: “In addition, the learning data generation function of the image analysis unit 200 performs the task of analyzing where normal drones and dangerous drones are located in the image divided by frame…Specifically, if the image analysis unit 200 analyzes the image and determines that the image contains a dangerous drone, the control unit 300 sends a warning message warning the user to continuously track the threatening drone while the user is monitoring, A control function is provided to output a warning sound, etc. to the monitoring unit 400.)
Accordingly, a person of ordinary skill in the art at the time this invention was effectively filed would have found it obvious to modify Wang’s UAV detection pipeline with Lee’s concept of notifying the user once a target object is found in the image’s frame/tile. A person of ordinary skill would be motivated to do this because Wang states “…the rapid spread of micro-UAV, many new security threats and social problems, including disturbing aviation safety of commercial aircraft, breaching no-fly security in sensitive areas and invading public privacy, frequently occur and appear to be continually increasing “ whilst Lee states “the scope of use of drones gradually expands, concerns that drones will be misused for various crimes, terrorism, and invasion of privacy are also increasing.” A person of ordinary skill in the art is aware that a rapid response and or human evaluation is needed in response to plausible to object detection and must be done by subsequent notification to a user.
As per claim 10
Wang teaches all claim limitations previously rejected in claim 1’s 102 rejection. See claim 1’s 102 rejection.
Lee teaches wherein the network model comprises a modified You Only Look Once (YOLO) architecture (“The learning function of the image analysis unit 200 learns images based on the YOLO (You Only Look Once) model”) wherein the modified YOLO architecture comprises a feature pyramid network (“the neck collects the feature maps extracted from the backbone and creates a feature pyramid, and the head consists of the output layer with the final detection.”) configured to upsample the image data by at least four times to capture fine-grained details of the one or more targeted objects in the at least one image frame. (Enabling YOLO to upsample four times is a neural architectural routine optimization and known scaling tradeoff that a person of ordinary skill in the art can do through extending the feature pyramid.)
Claim 6, 7 and 8 is rejected under 35 U.S.C. 103 as being unpatentable over Wang et al (Wang hereinafter “Flying Small Target Detection for Anti-UAV Based on a Gaussian Mixture Model in a Compressive Sensing Domain”) in view of Ekaterina et al (Ekaterina hereinafter “Learning Generative Models of Object Parts from A Few Positive Examples”)
As per claim 6
Wang teaches all claim limitations previously rejected in claim 1’s 102 rejection. See claim 1’s 102 rejection.
Wang teaches network model is configured to determine an optimal number of Gaussian components for the GMM for accurate identification of the one or more targeted objects in the at least one image frame, (“Making use of the advantages of compressive sensing which can efficiently compress a higher-dimensional data to lower-dimensional one and perfectly reconstruct from a small number of linear measurements with high probability,…the greater incoherence of the measurement/sparsity pair (Φ,Ψ), the smaller number of the linear measurements needed. In a Gaussian random matrix, the elements are independent and identically distributed (i.i.d.) random variables from a zero-mean, 1/𝑛-variance Gaussian density. Then, a Gaussian random matrix is incoherent with most of orthogonal sparse basis matrix, and is chosen as measurement matrix of compressive sensing to collect information of random projections… When a new image is captured at time t during model training, a series of linear projections with Gaussian random matrix is operated over the image vector by Equation … If there exists a matched Gaussian distribution, the corresponding mean and variance should be updated by utilizing current CS measurement…” GMM with compressive sensing models signals as a collection of several simple gaussian distributions. The algorithm determines which specific Gaussian mixture component the signal belongs to and uses that for reconstruction. ) the Gaussian components are associated with at least one structure in the one or more targeted objects. (“the flying target with salient movement is the foreground, while the background scene changes are relatively slight. Thereupon, a Gaussian distribution will catch the pattern of follow-up compressive sensing measurements.”)
Wang does not teach Gaussian components are associated with at least one structure in the one or more targeted objects.
Ekaterina teaches Gaussian components are associated with at least one structure in the one or more targeted objects. (Section 4 Probabilistic Detection of Object a Parts: “Since the representation of each part is now encoded into B random Gaussian mixture models in the aligned object space, we need an efficient and effective procedures to combine the likelihoods of each” “Parts” corresponds to structure. The object parts are encoded into the gaussian mixture models, the gaussian components are associated with respective object parts)
Accordingly, a person of ordinary skill in the art at the time this invention was filed would have found it obvious to incorporate Ekaterina’s concept of Gaussian component association with structure of the targeted object. A person of ordinary skill in the art would be motivated to do this because associating Gaussian mixture component with object structures improves the fidelity of identification when it comes to plausible structural variation that can be caused when, for example, things are for example “changing flight attitude, near or far distance and variable lighting conditions, the shape and size of the drone is in flux and is challenging for detection” as Wang states.
As per claim 7
Wang and Ekaterina teach the claim limitations rejected in claim 6’s 103 rejection. See claim 6’s 103 rejection.
Ekaterina teaches execution of the instructions causes the one or more processors to identify the one or more targeted objects based on comparing each the Gaussian components to a plurality of anchors, the plurality of anchors comprising at least one structure in the plurality of potential objects. ( Figure 5, Section B Randomised Gaussian Mixture Model PDF : “We allow each landmark of each object to have its unique frequency and orientation representation… The procedure is repeated T times for each landmark of each object …The chosen Gabor filters are then applied to each pixel of the image and transformed into likelihood maps with the corresponding GMM.” Ekaterina extracts features with Gabor filter while the GMM statistically classify these features. This is done for each feature within an object. Likelihood maps that represent those Gaussians are then compared when detecting a “anchor” within the detected object as seen in figure 5. The plurality of “anchors” can be seen in Figure 7 illustrating the same concept but looking at multiple “anchors” within a single detected image.)
As per claim 8
Wang and Ekaterina teach all claim limitations previously rejected in claim 7’s 103 rejection. See claim 7’s 103 rejection.
Ekaterina teaches wherein execution of the instructions causes the one or more processors to identify the one or more targeted objects based on a maximum likelihood estimation for each of the plurality of anchors. (Figure 5, Figure 7)
Claim 9 is rejected under 35 U.S.C. 103 as being unpatentable over Wang et al (Wang hereinafter “Flying Small Target Detection for Anti-UAV Based on a Gaussian Mixture Model in a Compressive Sensing Domain”) in view of Ekaterina et al (Ekaterina hereinafter “Learning Generative Models of Object Parts from A Few Positive Examples”) in further view of Liang et al (Liang hereinafter “GMMSeg: Gaussian Mixture based Generative Semantic Segmentation Models”)
As per claim 9
Wang and Ekaterina teach the claim limitations rejected in claim 6’s 103 rejection. See claim 6’s 103 rejection.
Wang nor Ekaterina teach wherein the network model comprises a neural network model configured to process the at least one image frame to determine the optimal number of the Gaussian components for each of the one or more targeted objects in the at least one image frame, the optimal number of the Gaussian components comprising a minimum number of the Gaussian components.
Liang teaches wherein the network model comprises a neural network model (Experimental Protocol we adopt ResNet101-DeepLabV3+ architecture. For completeness, we also report the results of our GMMSeg based on ResNet101-FCNandMiTB5-SegFormer.”) configured to process the at least one image frame (Figure 3 and 4) to determine the optimal number of the Gaussian components for each of the one or more targeted objects in the at least one image frame ( Abstract: “For each class, GMMSeg builds Gaussian Mixture Models (GMMs) via Expectation-Maximization (EM), so as to capture class-conditional densities.” Figure 3, Table 5 ) the optimal number of the Gaussian components comprising a minimum number of the Gaussian components. (Section 3.2 GMMSeg: Dense GMM Generative Classification: “GMMSeg achieves the merits of both generative and discriminative learning. The online EM based generative optimization enables the GMM to best fit the data distribution even on the evolving feature space.” Table 5 )
Accordingly, a person of ordinary skill in the art, at the time this invention was effectively filed would have found it obvious to incorporate Liang’s GMMseg neural network based Gaussian component optimization techniques into the previously modified Wang/Ekaterina workflow in order to determine an optimal minimum number of Gaussian components for targeted object detection. A person of ordinary skill in the art would be motivated to do this because GMMSeg teaches optimizing Gaussian component count for accurate identification performance which improves detection accuracy while reducing unnecessary, and or redundant computational complexity that can cause overfitting and false positives in the detection pipeline. Ekaterina states “localising object parts remains challenging due to the variant appearance of object parts as well as the changes of illumination and viewing angles.” , that “…the discriminative information of object parts, which can play an important role (verified in fine-grained visual recognition [2]), are missing due to failing to localise object parts explicitly.”) and that “The overfitting occurs with any classifier and the standard procedure is to limit the size of the Gabor bank (number of frequencies/orientations).” A person of ordinary skill understands that using Liang’s neural network which optimizes and best fits gaussian components to a “sweet spot” (shown in Table 5) helps counter overfitting which is in turn is crucial in the Wang/Ekaterina UAV/small object determination to accurately discriminate what the object actually is.
Claim 11 is rejected under 35 U.S.C. 103 as being unpatentable over Wang et al (Wang hereinafter “Flying Small Target Detection for Anti-UAV Based on a Gaussian Mixture Model in a Compressive Sensing Domain”) in view of Lee et al (Lee hereinafter KR 20240117181 “A Small Infiltration Drone Identification System In Short-wave Infrared Images”) in further view Paik et al (Paik hereinafter KR 20230137007 A “Efficient Object Detection Method And Apparatus For Drone Environment”)
As per claim 11
Wang and Lee teach all claim limitations previously rejected in claim 10’s 103 rejection
Wang nor Lee teach wherein the modified YOLO architecture is configured to perform compound scaling of the image data.
Paik teaches wherein the modified YOLO architecture (Detailed description; “in one embodiment of the present invention, it is assumed that an object detection model is constructed based on the YOLOv4-s model.” The basic configuration is similar to the YOLOv4-s model (hereinafter referred to as the baseline model). However, the baseline model has three head layers that detect objects using multiple feature maps generated from the backbone network.) is configured to perform compound scaling of the image data. (Detailed description: “Accordingly, compound scaling was experimented by combining the third and fourth models with depth scaling….The result of applying the head layer removal method is Compound Scaling + Head Layer Elimination in Figure 3.)
Accordingly, a person of ordinary skill in the art at the time this invention was effectively filed would have been motivated to incorporate Paik’s concept of YOLO based compound scaling withing the Wang/Lee pipeline. A person of ordinary skill in the art would have been motivated to do this because this improves the accuracy of detecting small objects across a variety of object scales and image resolutions. Compound scaling is known to scale network depth, width and resolution in a predictable manner to improve feature extraction and object detection performance while at the same time maintaining computational efficiency in real time.
Claim 12 are rejected under 35 U.S.C. 103 as being unpatentable over Wang et al (Wang hereinafter “Flying Small Target Detection for Anti-UAV Based on a Gaussian Mixture Model in a Compressive Sensing Domain”) in view of Tanaka et al (Tanaka hereinafter US 20230079528 A1)
As per claim 12
Wang teaches all claim limitations previously rejected in claim 1’s 102 rejection. See claim 1’s 102 rejection.
Wang does not teach wherein execution of the instructions causes the one or more processors to determine a confidence score for the one or more targeted objects.
Tanaka teaches wherein execution of the instructions causes the one or more processors to determine a confidence score for the one or more targeted objects. (Figure 7, Paragraph [0044] “Each of the plurality of images 152 included in the discrimination image 153 displays the confidence score that the target object exists.”)
Accordingly, a person of ordinary skill in the art at the time this invention was effectively filed would have found it obvious to modify Wang’s small object detection workflow with Tanaka’s use of confidence scores. A person of ordinary skill in the art would have been motivated to do this because adding a confidence score to an image within a small object detection pipeline helps filter out background noise, prevents false positives and establishes a metric for the overalls system reliability.
Claim 13, 18 and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Wang et al (Wang hereinafter “Flying Small Target Detection for Anti-UAV Based on a Gaussian Mixture Model in a Compressive Sensing Domain”) in view of Tanaka et al (Tanaka hereinafter US 20230079528 A1) in further view of Savvides et al (Savvides hereinafter US 20180096457 A1)
As per claim 13
Wang and Tanaka teach all claim limitations previously rejected in claim 12’s 103 rejection. See claim 12’s 103 rejection.
Tanaka teaches a display configured to display the at least one image frame, the one or more targeted objects, (Figure 7) wherein execution of the instructions causes the one or more processors to determine…the confidence score for the one or more targeted objects. (Figure 7)
Tanaka nor Wang teach display configured to display features corresponding to each of the one or more targeted objects, and wherein execution of the instructions causes the one or more processors to determine the features comprising a bounding box surrounding the one or more targeted objects
Savvides teaches display configured to display features corresponding to each of the one or more targeted objects (Paragraph [0060] “such as displaying a message to a user on a visual display, issue a flag, and/or display the image annotated with the bounding box to the user, among other things.” paragraph [0065] Each bounding box, here just bounding box 340, suspected of containing an occurrence of the desired class, is then projected back to each of feature maps 316(1) to 316(3) on convolution layers 304(3)s3 to 304(5)s3. This back-projecting is used to isolate a corresponding ROI 348(1) to 348(3) in each of feature maps 316(1) to 316(3). Based on bounding box 340, each of these ROIs 348(1) to 348(3) is suspected to contain an occurrence of an object of the desired class”)
wherein execution of the instructions causes the one or more processors to determine the features comprising a bounding box surrounding the one or more targeted objects (Paragraph [0064] “Highlighted feature vector 336 is then processed by fully connected layers (not shown) to determine one or more bounding boxes (only one bounding box 340 shown), each of which is suspected to contain an occurrence of an object agnostic to class (e.g., human face, chair, vehicle, weapon, etc.). The fully connected layers also determine an objectness score 344 for each bounding box 340 they have identified…Each bounding box, here just bounding box 340, suspected of containing an occurrence of the desired class, is then projected back to each of feature maps…this back-projecting is used to isolate a corresponding ROI… in each of feature maps 316(1) to 316(3). Based on bounding box 340, each of these ROIs 348(1) to 348(3) is suspected to contain an occurrence of an object of the desired class… dimensionally reduced ROI 368 is then processed by fully connected layers 372 (also indicated by “fc”) to determine a confidence score 376 for the suspected occurrence of the object in the dimensionally reduced ROI.” Here Savvides puts a bounding box around a suspected class then back projects the bounding box to the feature map. The ROI is then normalized and concatenated. The dimensionally reduced ROI is then given a confidence score for the suspected occurrence. This pipeline effectively uses a bounding box projected on a feature map to determine through confidence the feature/class of the object)
Accordingly, a person of ordinary skill in the art at the time this invention was effectively filed would have found it obvious to further modify the Wang/Tanaka pipeline with Savvides concept of displaying features corresponding to each of the one or more targeted objects as well as using a bounding box to determine the one or more features within said bounding box. A person of ordinary skill in the art would have been motivated to do this because they would be aware that using a display to display features of small objects is a simple and effective way to illustrate targets. A person of ordinary skill in the art is also aware that using a bounding box to determine features of small objects, such as objects in the Wang/Tanaka pipeline, provides a standardized way to isolate locations in space. Small objects lack rich distinguishable textures and bounding boxes reduce background noise and provide a uniform shape for downstream processes such as machine learning models that need to learn geometric features within precise locations, A person of ordinary skill in the art is also aware that creating bounding boxes simplifies mathematical processes and convolution. Instead of analyzing an entire image for object recognition, the bounding box localizes an area and saves processing power.
As per claim 18
Wang teaches all claim limitations previously rejected in claim 1’s 102 rejection. See claim 1’s 102 rejection.
Savvides teaches wherein the network model is configured to determine at least one loss function associated with the detection of the one or more targeted objects. (Paragraph [0033] network also adopts a multitask loss, i.e., a classification loss and a bounding-box regression loss.)
As per claim 19
Wang, Tanaka and Savvides teach all claim limitations previously rejected in claim 18’s 103 rejection. See claim 18’s 103 rejection.
Savvides teaches wherein execution of the instructions causes the one or more processors to determine a confidence score for the detection of the one or more targeted objects, the confidence score being associated with an image quality associated with the image data (Paragraph [0065] “…dimensionally reduced ROI 368 is then processed by fully connected layers 372 (also indicated by “fc”) to determine a confidence score 376 for the suspected occurrence of the object in the dimensionally reduced ROI.” The confidence score is associated with a dimensionally reduced ROI. ) wherein execution of the instructions causes the one or processors to train the network model based on the at least one loss function (Paragraph [0033] “The network also adopts a multitask loss, i.e., a classification loss and a bounding-box regression loss. Based on the two improvements, the framework is trained end-to-end. The processing time for each image significantly reduced to 0.3s.”)
As per claim 14
Claim 14 is rejected under 35 U.S.C. 103 as being unpatentable over Wang et al (Wang hereinafter “Flying Small Target Detection for Anti-UAV Based on a Gaussian Mixture Model in a Compressive Sensing Domain”) in view of Tanaka et al (Tanaka hereinafter US 20230079528 A1) in further view of Savvides et al (Savvides hereinafter US 20180096457 A1) in further view of Viss et all (Viss hereinafter US 20210264153 A1)
Neither Wang, Tanaka or Savvides teach the object detection system of claim 13, wherein the one or more targeted objects change at least one of an appearance or location from a first image frame in the at least one image frame to a second image frame in the at least one image frame, and wherein the display is configured to display the at least one image frame, the one or more targeted objects in each respective image frame, and the features corresponding to each of the one or more targeted objects in a sequential order, the at least one frame comprising the first image frame and the second image frame.
Viss teaches The object detection system of claim 13, wherein the one or more targeted objects change at least one of an appearance or location from a first image frame in the at least one image frame to a second image frame in the at least one image frame, and wherein the display is configured to display the at least one image frame, the one or more targeted objects in each respective image frame, and the features corresponding to each of the one or more targeted objects in a sequential order, the at least one frame comprising the first image frame and the second image frame (Figure 2)
Accordingly, a person of ordinary art at the time this invention was effectively filed would have found it obvious to further modify the Wang/Tanaka/Savvides system to incorporate Viss’ concept of displaying multiple frames of tracking where each sequential frame shows a new orientation of the tracked object in order to simulate a temporal spatial tracking analysis if some modality of live real time data isn’t directly available or becomes offline. A person of ordinary skill in the art understands that certain technologies for example, UAVs, may be able to jam livestreams by using targeted radio frequencies. By having an auxiliary modality to simulate tracking through sequential image frames this can be directly countered.
Claim 15 is rejected under 35 U.S.C. 103 as being unpatentable over Wang et al (Wang hereinafter “Flying Small Target Detection for Anti-UAV Based on a Gaussian Mixture Model in a Compressive Sensing Domain”) in view of Tanaka et al (Tanaka hereinafter US 20230079528 A1) in further view of Savvides et al (Savvides hereinafter US 20180096457 A1) in further view of Barisic et al (Barisic hereinafter Vision-based system for a real-time detection and following of UAV)
As per claim 15
Wang, Tanaka and Savvides teach all claim limitations previously rejected in claim 13’s 103 rejection. See claim 13’s 103 rejection.
Neither Wang, Tanaka or Savvides teach wherein the one or more processors are further caused to detect the one or more targeted objects and the corresponding features in 500 milliseconds or less
Barisic teaches wherein the one or more processors are further caused to detect the one or more targeted objects and the corresponding features in 500 milliseconds or less. (Section V experimental results: “ As already mentioned, detection of UAV is performed on Intel Neural Compute Stick 2 (NCS2) using OpenVINO Toolkit specialised for convolutional neural networks like YOLO…Finally, validation of the complete vision-based system for UAV tracking has been done. The target UAV was executing a 3D trajectory in a shape of number eight. Object detection was performed at a speed of 20 FPS. To further confirm the performance of UAV detection, the proposed algorithm was compared to the ground truth at 387 frames taken from the experiment.”)
Accordingly, a person of ordinary art at the time this invention was effectively filed would have found it obvious to further modify the Wang/Tanaka/Savvides system to incorporate Barisic’s concept of having the processor detect the one or more targeted objects and the corresponding features in 500 milliseconds or less. A person of ordinary skill in the art would be motivated to do this to increase the speed and efficiency of aerial object detection. A person of ordinary skill in the art is aware that small aerial objects such as UAVs are usually high in agility and speed and that fast millisecond detection is crucial for detection. Lower latency processing/detection is critical for targeting, analyzing, classifying and tracking plausible threats.
Claim 16 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Wang et al (Wang hereinafter “Flying Small Target Detection for Anti-UAV Based on a Gaussian Mixture Model in a Compressive Sensing Domain”) in view of Coluccia et al (Coluccia hereinafter “Drone vs. Bird Detection: Deep Learning Algorithms and Results from a Grand Challenge”)
As per claim 16
Wang teaches all claim limitations previously rejected in claim 1’s 102 rejection. See claim 1’s 102 rejection.
Wang does not teach wherein the plurality of potential objects comprises animals, unmanned vehicles, and manned vehicles, and wherein the one or more targeted objects comprise the unmanned vehicles.
Coluccia teaches wherein the plurality of potential objects comprises animals, unmanned vehicles (Figure 4) , and manned vehicles (the approach consists of a first module used to detect the most probable regions containing the target, followed by a second module that classifies the detected target in either drone or other flying entity… , and wherein the one or more targeted objects comprise the unmanned vehicles (Figure 4).
Accordingly, a person of ordinary skill in the art at the time this invention was effectively filed would have found it obvious to incorporate Coluccias’ classification and discrimination of various flying objects including animals, manned and unmanned vehicles when trying to target an unmanned vehicle within Wang’s Gaussian Mixture based UAV detection pipeline. A person of ordinary skill in the art would have been motivated to do this because distinguishing unmanned vehicles from other potential aerial objects improves detection robustness and reduces false detections in aerial surveillance. A person of ordinary skill in the art understands that discernment between flying objects is crucial if some sort of response is needed in regards to the nature of the flying object.
As per claim 20
Wang teaches all claim limitations previously rejected in claim 1’s 102 rejection. See claim 1’s 102 rejection.
Coluccia teaches wherein execution of the instructions causes the one or more processors to train the network model based on training data comprising the plurality of potential objects in various scenarios (Section 2.1 Dataset “The training set for the challenge consists of 77 different videos. The videos comprise 1384 frames on average, while each frame contains on average 1.12 annotated drones… the videos have been recorded at two locations with different geo characteristics.) wherein the various scenarios comprises at least one of the plurality of potential objects in at least one of a different orientation or distance. (Section 2.1 Dataset “Moreover, the distance of the drones from the camera varies strongly across and within the videos, yielding large variations in…)
Claim 17 is rejected under 35 U.S.C. 103 as being unpatentable over Wang et al (Wang hereinafter “Flying Small Target Detection for Anti-UAV Based on a Gaussian Mixture Model in a Compressive Sensing Domain”) in view of Svanström et al (Svanström hereinafter “Drone Detection and Tracking in Real-Time by Fusion of Different Sensing Modalities”)
Wang teaches all claim limitations previously rejected in claim 1’s 102 rejection. See claim 1’s 102 rejection.
Wang does not teach wherein the plurality of potential objects comprises animals, unmanned vehicles, and manned vehicles, and wherein the one or more targeted objects comprise the unmanned vehicles.
Svanström teaches wherein the plurality of potential objects comprises animals, unmanned vehicles, and manned vehicles, and wherein the one or more targeted objects comprise the unmanned vehicles. (Table 4, Table 6 Figure A1 and A2, Table 1)
Accordingly, a person of ordinary skill in the art at the time this invention was effectively filed would have found it obvious to include the plurality of objects detected by the object detection system to be animals, unmanned vehicles and manned vehicle as well as one or more targeted objects to be an unmanned vehicle. A person of ordinary skill in the art is aware that differentiation between flying objects is essential in downstream decisions in regards to UAV detection. Svanström explains that “To achieve effective detection of drones, in building the classes, we have considered including other flying objects that are likely to be mistaken for a drone” and that “to detect drones with high efficiency, the system must also recognize and track other flying objects that are likely to be mistaken for drones.” A person of ordinary skill in the art would see that this concept of classification is essential in small flying object detection and would be inclined to modify Wang’s pipeline Svanström ‘s with classification of flying objects to boost efficiency of detection and labeling.
Claim 21 is rejected under 35 U.S.C. 103 as being unpatentable over (Wang hereinafter “Flying Small Target Detection for Anti-UAV Based on a Gaussian Mixture Model in a Compressive Sensing Domain”) in view of Sommer et al (Sommer hereinafter “DEEP LEARNING BASED UAV PAYLOAD RECOGNITION”)
As per claim 21
Wang teaches all claim limitations previously rejected in claim 1’s 102 rejection. See claim 1’s 102 rejection
Wang does not teach wherein execution of the instructions causes the one or more processors to: detect that the one or more targeted objects is carrying a payload, and alert a user on a location of the one or more targeted objects carrying the payload.
Sommer teaches wherein execution of the instructions causes the one or more processors to: detect that the one or more targeted objects is carrying a payload, (Figure 1) and alert a user on a location of the one or more targeted objects carrying the payload. (Figure 1, Abstract “In this work, we examine the potential of UAV payload classification in EO imagery, which facilitates direct interpretability by human operators.” )
In a combined teaching, Wang identifies/localizes the UAV target in the frame; Sommer provides image based payload classification that is presented to the human operator.
Accordingly, a person of ordinary skill in the art at the time this invention was effectively filed would have found it obvious to modify Wang’s image based UAV detection system with Sommer’s EO-image based UAV payload recognition to identify whether a detected UAV is carrying a payload and alert the operator to the detected UAV location. A person of ordinary skill in the art would have been motivated to make the combination to improve counter UAV threat assessment by distinguishing ordinary UAVs from payload carrying UAVs using visual imagery.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SHANE WRENSFORD CODRINGTON whose telephone number is (571)272-8130. The examiner can normally be reached 8:00am-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Matthew Bella can be reached at (571) 272-7778. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/SHANE WRENSFORD CODRINGTON/Examiner, Art Unit 2667
/MATTHEW C BELLA/Supervisory Patent Examiner, Art Unit 2667
Prosecution Insights