Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Remarks
Claim Rejections – 35 U.S.C. 101
Applicant’s amendments have been fully considered but they are not persuasive.
Applicant argues (pg. 11-13) that the limitations cannot be performed in the human mind because they are machine-executed operations over large datasets and model parameters, not mental steps.
Examiner respectfully disagrees. Limitations, such as determining an importance score for each annotated object sample, defining a set of importance score thresholds, counting a number of annotated object samples that fulfills the threshold, selecting annotated object samples are all mental steps that can be performed by a human. That these are actually operated on a computer does not change this fact. The question is not whether or not they are mental steps that can be performed by a human but rather if it’s practical. Applicant claims that it’s done over large datasets and model parameters, which does not automatically preclude a human from practically performing it. Examiner suggests pointing to the Specification that suggests why it is not practical for a human to perform such operations.
Furthermore, training the neural network is merely a high-level recitation of training the machine learning engine using the selected annotated object samples (i.e. adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea - see MPEP 2106.05(f)). See rejection below for more details. Examiner suggests elaborating on the mechanics/details of the training in the claims to advance it from its current black-box depiction.
Applicant claims (pg. 13-14) that the amended claims specify a particular dataset-selection and training regiment to improve model performance in object detection/classification.
Examiner respectfully disagrees. This is not a direct improvement to a computer. Instead, it generally argues that there is some improvement in general artificial intelligence (object detection and classification is a broad category). Furthermore, it seems like the improvement is not in the mechanics of the training itself but rather in the data-gathering and data-filtering prior to training.
Applicant claims (pg. 14-15) that the non-final Office Action did not support WURC with facts in the context of Step 2B.
Examiner respectfully disagrees. That the Office Action did not support WURC with facts is because there were no further elements past the steps prior to Step 2B. Step 2B and WURC only needs to be support with facts if the Examiner alleges that a limitation is indeed WURC, which there was none in the non-final office action.
The foregoing applies to all independent claims and their dependent claims.
Claim Rejections – 35 U.S.C. 103
Applicant’s prior art arguments have been fully considered and they are partially persuasive.
Applicant argues (pgs. 16-18) that the statistical methods in Zhang are not across classes or datasets as in the claim. Applicant argues that Zhang only does per-object standard deviation which is used solely to set a local IoU cutoff for that object’s candidate anchors. Applicant argues that critically, Zhang’s statistical measures are only used for a single ground-truth object.
Examiner respectfully disagrees. Zhang is indeed valid because it calculates the standard deviation across multiple objects too and attempts to maintain fairness between objects based on these statistical measures. See [Page 9764, Column 1, Paragraph 1]: “Maintaining fairness between different objects. According to the statistical theory4, about 16% of samples are in the confidence interval [mg +vg, 1] in theory. Although the IoU of candidates is not a standard normal distribution, the statistical results show that each object has about 0.2 _ kL positive samples, which is invariant to its scale, aspect ratio and location.”
Applicant argues (pgs. 18-19) that Asthana and Zhang do not teach the newly amended limitations that further clarify compute, for each annotated object sample, an importance score that weights quality and dataset relevance, set of importance-score thresholds, count per-class, per-threshold eligible samples, select a combination that minimizes the inter-class standard deviation/variance of sample counts; or train a detector only on the curated, class-balanced subset.
Examiner agrees. Accordingly, a new reference, Liu (“A comprehensive active learning method for multiclass imbalanced data streams with concept drift”), has been added to the rejection, as further detailed below.
The foregoing applies to all independent claims and their dependent claims.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(B) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1, 13, 14, 15 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor, or for pre-AIA the applicant regards as the invention.
Claims 1, 13, 14, 15 recites the limitation “quality” in line 7 which is a relative term that renders the claim indefinite. The phrase “quality” is not defined by the claim. It is mentioned in the specification (page 3 line 5) but the specification does not provide a standard for ascertaining the requisite degree of quality, and one of ordinary skill in the art would not be reasonably apprised of the scope of the invention. For examination purposes, Examiner treats quality as something important or relevant enough to be weighted more.
Claim Rejections – 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-15 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1: Claims 1-12 are method claims. Claims 13-15 are machine/system/product claims. Therefore, claims 1-15 are directed to either a process, machine, manufacture or composition of matter.
With respect to claim 1:
Step 2A – Prong 1:
A … method for selecting object samples for training of a neural network for object detection and classification from more than one dataset comprising annotated object samples of at least two object classes, the method comprising: determining a respective importance score for each annotated object sample of at least a portion of the annotated object samples, wherein the respective importance score reflects a weight between a quality and a relevance of each annotated object sample; (mental process – a person can manually determine a respective importance score for each annotated object sample of at least a portion of the annotated object samples, wherein the respective importance score reflects a weight between a quality and a relevance of each annotated object sample with the assistance of a pen/paper.)
defining a set of importance score thresholds, wherein an annotated object sample that fulfills an importance score threshold of the set of importance score thresholds is of sufficiently high quality and relevance to be used for training of the neural network; (mental process – a person can manually define a set of importance score thresholds with the assistance of a pen/paper.)
counting, for each object class and for each importance score threshold of the set of importance score thresholds, a number of annotated object samples in the object class that fulfill the importance score threshold; (mental process – a person can manually count, for each object class and for each importance score threshold of the set of importance score thresholds, a number of annotated object samples in the object class that fulfill the importance score threshold with the assistance of a pen/paper.)
selecting a number of annotated object samples from each object class that fulfill a respective importance score threshold of the defined set of importance score thresholds, so that the standard deviation or variance of the number of annotated object samples from each object class as counted is as small as possible, the selected annotated object samples to be used for training of the neural network for object detection and classification; (mental process – a person can manually select a number of annotated object samples from each object class that fulfill a respective importance score threshold with the assistance of a pen/paper.)
Step 2A – Prong 2: The claim does not include additional elements considered individually and in combination that integrate the judicial exception into a practical application.
… computer implemented …
…
…
…
and training the neural network using the selected annotated object samples. (Adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea - see MPEP 2106.05(f) – Examiner’s note: High level recitation of training the machine learning engine using the selected annotated object samples.);
Step 2B: The claim does not include additional elements considered individually and in combination that are sufficient to amount to significantly more than the judicial exception.
With respect to claim 2:
Step 2A – Prong 1:
The method according to claim 1, comprising: ignoring object samples excluded in the selection of object samples for training of the neural network as positive samples and as negative samples. (mental process – a person can manually ignore object samples excluded in the selection of object samples for training of the neural network with the assistance of a pen/paper.)
Step 2A – Prong 2: The claim does not include additional elements considered individually and in combination that integrate the judicial exception into a practical application.
Step 2B: The claim does not include additional elements considered individually and in combination that are sufficient to amount to significantly more than the judicial exception.
With respect to claim 3:
Step 2A – Prong 1:
The method according to claim 1, wherein fulfilling the respective importance score threshold is to exceed or be equal to the respective importance score threshold, the step of selecting further comprises: for a specified object class, selecting only object samples having an importance score that exceeds or is equal to a minimum importance score threshold that exceeds at least one of the importance score thresholds of the defined set of importance score thresholds. (mental process – a person can manually select only object samples having an importance score that exceeds or is equal to a minimum importance score threshold that exceeds at least one of the importance score thresholds of the defined set of importance score thresholds, with the assistance of a pen/paper.)
Step 2A – Prong 2: The claim does not include additional elements considered individually and in combination that integrate the judicial exception into a practical application.
Step 2B: The claim does not include additional elements considered individually and in combination that are sufficient to amount to significantly more than the judicial exception.
With respect to claim 4:
Step 2A – Prong 1:
The method according to claim 2, wherein fulfilling the respective importance score threshold is to exceed or be equal to the respective importance score threshold, the step of selecting further comprises: for a specified object class, selecting only object samples having an importance score that exceeds or is equal to a minimum importance score threshold that exceeds at least one of the importance score thresholds of the defined set of importance score thresholds wherein a step of ignoring further comprises: ignoring object samples in the specified object class having an importance score below the minimum importance score threshold in the selection of object samples for training of the neural network. (mental process – a person can manually, for a specified object class, selecting only object samples having an importance score that exceeds or is equal to a minimum importance score threshold that exceeds at least one of the importance score thresholds of the defined set of importance score thresholds wherein a step of ignoring further comprises: ignoring object samples in the specified object class having an importance score below the minimum importance score threshold in the selection of object samples for training of the neural network, with the assistance of a pen/paper.)
Step 2A – Prong 2: The claim does not include additional elements considered individually and in combination that integrate the judicial exception into a practical application.
Step 2B: The claim does not include additional elements considered individually and in combination that are sufficient to amount to significantly more than the judicial exception.
With respect to claim 5:
Step 2A – Prong 1:
The method according to claim 1, wherein the step of defining further comprises: defining more than one set of importance score thresholds, where a first set of importance score thresholds for a first object class is different from a second set of importance score thresholds for a second object class. (mental process – a person can manually define more than one set of importance score thresholds, where a first set of importance score thresholds for a first object class is different from a second set of importance score thresholds for a second object class with the assistance of a pen/paper.)
Step 2A – Prong 2: The claim does not include additional elements considered individually and in combination that integrate the judicial exception into a practical application.
Step 2B: The claim does not include additional elements considered individually and in combination that are sufficient to amount to significantly more than the judicial exception.
With respect to claim 6:
Step 2A – Prong 1:
The method according to claim 1, comprising: calculating a standard deviation for all possible combinations of the number of counted object samples and wherein the step of selecting further comprises selecting a combination of object samples from each object class based on the minimum standard deviation among all possible combinations. (mental process – a person can manually calculating a standard deviation for all possible combinations of the number of counted object samples and wherein the step of selecting further comprises selecting a combination of object samples from each object class based on the minimum standard deviation among all possible combinations with the assistance of a pen/paper.)
Step 2A – Prong 2: The claim does not include additional elements considered individually and in combination that integrate the judicial exception into a practical application.
Step 2B: The claim does not include additional elements considered individually and in combination that are sufficient to amount to significantly more than the judicial exception.
With respect to claim 7:
Step 2A – Prong 1:
The method according to claim 1, wherein calculating the standard deviation comprises calculating the standard deviation of the number of object samples in each group. (mental process – a person can recognize that calculating the standard deviation comprises calculating the standard deviation of the number of object samples in each group.)
Step 2A – Prong 2: The claim does not include additional elements considered individually and in combination that integrate the judicial exception into a practical application.
Step 2B: The claim does not include additional elements considered individually and in combination that are sufficient to amount to significantly more than the judicial exception.
With respect to claim 8:
Step 2A – Prong 1:
The method according to claim 1, wherein the determining comprises: for each of the annotated object samples, calculating the importance score based on an object sample confidence value and a relevance value, where the object sample confidence value is larger for manually annotated samples than for automatically annotated samples, and the relevance value is higher for a dataset considered more relevant for the use case the neural network is trained for than for datasets more remote from the use case. (mental process – a person can manually, for each of the annotated object samples, calculate the importance score based on an object sample confidence value and a relevance value, where the object sample confidence value is larger for manually annotated samples than for automatically annotated samples, and the relevance value is higher for a dataset considered more relevant for the use case the neural network is trained for than for datasets more remote from the use case, with the assistance of a pen/paper.)
Step 2A – Prong 2: The claim does not include additional elements considered individually and in combination that integrate the judicial exception into a practical application.
Step 2B: The claim does not include additional elements considered individually and in combination that are sufficient to amount to significantly more than the judicial exception.
With respect to claim 9:
Step 2A – Prong 1:
The method according to claim 8, wherein the confidence value of automatically annotated samples is a confidence value obtained from a model or algorithm used for annotating the object samples. (mental process – a person can recognize that the confidence value of automatically annotated samples is a confidence value obtained from a model or algorithm used for annotating the object samples.)
Step 2A – Prong 2: The claim does not include additional elements considered individually and in combination that integrate the judicial exception into a practical application.
Step 2B: The claim does not include additional elements considered individually and in combination that are sufficient to amount to significantly more than the judicial exception.
With respect to claim 10:
Step 2A – Prong 1:
The method according to claim 8, wherein calculating the importance score includes adjusting a tuning factor for adjusting the relative importance of the object sample confidence value and a relevance value when calculating the importance score. (mental process – a person can recognize that wherein calculating the importance score includes adjusting a tuning factor for adjusting the relative importance of the object sample confidence value and a relevance value when calculating the importance score.)
Step 2A – Prong 2: The claim does not include additional elements considered individually and in combination that integrate the judicial exception into a practical application.
Step 2B: The claim does not include additional elements considered individually and in combination that are sufficient to amount to significantly more than the judicial exception.
With respect to claim 11:
Step 2A – Prong 1:
The method according to claim 1, wherein the set of importance score thresholds comprise at least 3, or at least 5, or at least 8 importance score thresholds. (mental process – a person can recognize that wherein the set of importance score thresholds comprise at least 3, or at least 5, or at least 8 importance score thresholds.)
Step 2A – Prong 2: The claim does not include additional elements considered individually and in combination that integrate the judicial exception into a practical application.
Step 2B: The claim does not include additional elements considered individually and in combination that are sufficient to amount to significantly more than the judicial exception.
Claim 12 is substantially similar to claim 1, but has the following additional elements:
With respect to claim 12:
Step 2A – Prong 1:
The method according to claim 1, wherein the neural network is a Convolutional Neural Network. (mental process – a person can recognize that the neural network is a Convolutional Neural Network.)
Step 2A – Prong 2: The claim does not include additional elements considered individually and in combination that integrate the judicial exception into a practical application.
Step 2B: The claim does not include additional elements considered individually and in combination that are sufficient to amount to significantly more than the judicial exception.
Claim 13 is substantially similar to claim 1, but has the following additional elements:
With respect to claim 13:
Step 2A – Prong 2: This judicial exception is not integrated into a practical application.
A control unit for selecting object samples for training of a neural network for object detection and classification from more than one dataset comprising annotated object samples of at least two object classes, the control unit being configured to: (mere instructions to apply the exception using a generic computer component – control unit applies exception.)
Step 2B: The claim does not include additional elements considered individually and in combination that are sufficient to amount to significantly more than the judicial exception.
Claim 14 is substantially similar to claim 1, but has the following additional elements:
With respect to claim 14:
Step 2A – Prong 2: This judicial exception is not integrated into a practical application.
A system comprising an image capturing device for capturing images of a scene including objects, and a control unit configured to operate a neural network model for detecting and classifying objects in the scene, (mere instructions to apply the exception using a generic computer component – image capturing device and control unit apply exception.)
the neural network model having been trained on object samples selected according to a method for selecting object samples for training of a neural network from more than one dataset comprising annotated object samples of at least two object classes, the method comprising: (Adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea - see MPEP 2106.05(f) – Examiner’s note: High level recitation of training the machine learning engine on selected object samples.)
Step 2B: The claim does not include additional elements considered individually and in combination that are sufficient to amount to significantly more than the judicial exception.
Claim 15 is substantially similar to claim 1, but has the following additional elements:
With respect to claim 15:
Step 2A – Prong 2: This judicial exception is not integrated into a practical application.
A non-transitory computer-readable storage medium having stored thereon a computer program comprising instructions which, when the program is executed by a computer, cause the computer to carry out a method for selecting object samples for training of a neural network for object detection and classification from more than one dataset comprising annotated object samples of at least two object classes, the method comprising: (mere instructions to apply the exception using a generic computer component – non-transitory computer-readable storage medium applies exception.)
Step 2B: The claim does not include additional elements considered individually and in combination that are sufficient to amount to significantly more than the judicial exception.
Claim Rejections – 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-15 are rejected under 35 U.S.C. 103 as being unpatentable over Liu et al. (“A comprehensive active learning method for multiclass imbalanced data streams with concept drift”) hereinafter known as Liu in view of Zhang et al. (“Bridging the Gap Between Anchor-based and Anchor-free Detection via Adaptive Training Sample Selection”) hereinafter known as Zhang.
Regarding independent claim 1, Liu teaches:
A method for selecting object samples for training of a neural network from more than one dataset comprising annotated object samples of at least two object classes, the method comprising: determining an importance score for at least a portion of the annotated object samples; (Liu [Page 4, Column 1, Paragraph 5]: “we propose an asymmetric margin threshold matrix (because the margin threshold between class A and class B may not be identical to that of class B and class A). Under dynamic adjustment, the margin threshold matrix becomes an asymmetric matrix.” Liu teaches a threshold matrix that factors in the relationship between classes. This is essentially an importance score as the category of importance is whether a given class can be a majority to a subset of classes while being minority to others. As this index is important and is used in the calculation of the threshold, it is an importance score.)
defining a set of importance score thresholds, wherein an annotated object sample that fulfills an importance score threshold of the set of importance score thresholds is of sufficiently high quality and relevance to be used for training of the neural network; (Liu [Page 4, Column 1, Paragraph 5]: “we propose an asymmetric margin threshold matrix (because the margin threshold between class A and class B may not be identical to that of class B and class A). Under dynamic adjustment, the margin threshold matrix becomes an asymmetric matrix.” Liu teaches a threshold matrix that factors in the relationship between classes.)
counting, for each object class and for each importance score threshold of the set of importance score thresholds, a number of annotated object samples in the object class that fulfill the importance score threshold; (Liu [Page 5, Column 2, Paragraph 3]: “We propose a novel hybrid labelling strategy that consists of the random strategy, an uncertainty strategy based on the asymmetric margin threshold matrix and a selective sampling strategy” Liu teaches a sampling/labeling method based on the threshold matrix. Since this is a probability distribution, the probability of selecting the sample is, over a large number of samples, equal to the count.)
Liu does not explicitly teach:
selecting the annotated object samples from each object class that fulfill a respective importance score threshold of the defined set of importance score thresholds, so that the standard deviation or variance of the number of annotated object samples from each object class as counted is as small as possible, the selected annotated object samples, to be used for training of the neural network for object detection and classification; and training the neural network using the selected annotated object samples
However, Zhang teaches:
selecting the annotated object samples from each object class that fulfill a respective importance score threshold of the defined set of importance score thresholds, so that the standard deviation or variance of the number of annotated object samples from each object class as counted is as small as possible, the selected annotated object samples, to be used for training of the neural network for object detection and classification; and training the neural network using the selected annotated object samples. (Zhang [Page 9763, Column 2, Algorithm 1]: “for each ground-truth g … compute IoU between C_g and g … compute standard deviation of D_g … compute IoU threshold for ground-truth …” Zhang teaches that the groups are chosen from the classes of samples using a loss called IoU. There is a threshold that needs to be met for the IoU. This is based on the standard deviation of the group. As the IoU needs to be greater than the threshold, that means that the standard deviation, or the variation, has to be small. The smallest variation best provides the groupings.)
Liu and Zhang are in the same field of endeavor as the present invention, as the references are directed to selecting datasets for training neural networks. It would have been obvious, before the effective filing date of the claimed invention, to a person of ordinary skill in the art, to combine selecting data samples based on a set of importance score thresholds as taught in Liu with using standard deviation to select this subset as taught in Zhang. Zhang provides this additional functionality. As such, it would have been obvious to one of ordinary skill in the art to modify the teachings of Liu to include teachings of Zhang because the combination would allow for the variance/standard deviation between the samples to be specified This has the potential benefit of creating a training data set that more accurately represents the data.
Regarding dependent claim 2, Lin and Zhang teach:
The method according to claim 1, comprising:
Zhang teaches:
ignoring object samples excluded in the selection of object samples for training of the neural network as positive samples and as negative samples. (Zhang [Page 9763, Column 2, Algorithm 1, Line 17]: “N = A – P” Zhang teaches that the set of negative samples is the set of all anchor boxes, excluding the elements that are in the set of positive samples. This shows that it negative samples have no overlap with the positive samples, and must have been ignored if the positive were chosen.)
The reasons to combine are substantially similar to those of claim 1.
Regarding dependent claim 3, Lin and Zhang teach:
The method according to claim 1,
Zhang teaches:
wherein fulfilling the respective importance score threshold is to exceed or be equal to the respective importance score threshold, the step of selecting further comprises: for a specified object class, selecting only object samples having an importance score that exceeds or is equal to a minimum importance score threshold that exceeds at least one of the importance score thresholds of the defined set of importance score thresholds. (Zhang [Page 9763, Column 2, Algorithm 1]: “for each ground-truth g … compute IoU between C_g and g … compute standard deviation of D_g … compute IoU threshold for ground-truth …” Zhang teaches that the groups are chosen from the classes of samples using a loss called IoU. There is a threshold that needs to be met for the IoU. This is based on the standard deviation of the group. As the IoU needs to be greater than the threshold, that means that the standard deviation, or the variation, has to be small. The smallest variation best provides the groupings.)
The reasons to combine are substantially similar to those of claim 1.
Regarding dependent claim 4, Lin and Zhang teach:
The method according to claim 2,
Zhang teaches:
wherein fulfilling the respective importance score threshold is to exceed or be equal to the respective importance score threshold, the step of selecting further comprises: for a specified object class, selecting only object samples having an importance score that exceeds or is equal to a minimum importance score threshold that exceeds at least one of the importance score thresholds of the defined set of importance score thresholds wherein a step of ignoring further comprises: ignoring object samples in the specified object class having an importance score below the minimum importance score threshold in the selection of object samples for training of the neural network. (Zhang [Page 9763, Column 2, Algorithm 1]: “for each ground-truth g … compute IoU between C_g and g … compute standard deviation of D_g … compute IoU threshold for ground-truth …” Zhang teaches that the groups are chosen from the classes of samples using a loss called IoU. There is a threshold that needs to be met for the IoU. This is based on the standard deviation of the group. As the IoU needs to be greater than the threshold, that means that the standard deviation, or the variation, has to be small. The smallest variation best provides the groupings. Zhang [Page 9763, Column 2, Algorithm 1, Line 17]: “N = A – P” Zhang teaches that the set of negative samples is the set of all anchor boxes, excluding the elements that are in the set of positive samples. This shows that it negative samples have no overlap with the positive samples, and must have been ignored if the positive were chosen.)
The reasons to combine are substantially similar to those of claim 1.
Regarding dependent claim 5, Lin and Zhang teach:
The method according to claim 1,
Zhang teaches:
wherein the step of defining further comprises: defining more than one set of importance score thresholds, where a first set of importance score thresholds for a first object class is different from a second set of importance score thresholds for a second object class. (Zhang [Page 9763, Column 2, Algorithm 1]: “for each ground-truth g … compute IoU between C_g and g … compute standard deviation of D_g … compute IoU threshold for ground-truth …” Zhang teaches that the groups are chosen from the classes of samples using a loss called IoU. There is a threshold that needs to be met for the IoU. This is based on the standard deviation of the group. Note that in each iteration, a set/group is created. For each group, the IoU and the IoU threshold is calculated, which shows that for each group, there is a different threshold.)
The reasons to combine are substantially similar to those of claim 1.
Regarding dependent claim 6, Lin and Zhang teach:
The method according to claim 1, comprising:
Zhang teaches:
and calculating a standard deviation for all possible combinations of the number of counted object samples and wherein the step of selecting further comprises selecting a combination of object samples from each object class based on the minimum standard deviation among all possible combinations. (Zhang [Page 9763, Column 2, Algorithm 1]: “for each ground-truth g … compute IoU between C_g and g … compute standard deviation of D_g … compute IoU threshold for ground-truth …” Zhang teaches that the groups are chosen from the classes of samples using a loss called IoU. There is a threshold that needs to be met for the IoU. This is based on the standard deviation of the group. As the IoU needs to be greater than the threshold, that means that the standard deviation, or the variation, has to be small. The smallest variation best provides the groupings, which is the minimum standard deviation. The combination of object samples is necessarily from amongst all possible combinations.)
The reasons to combine are substantially similar to those of claim 1.
Regarding dependent claim 7, Lin and Zhang teach:
The method according to claim 1,
Zhang teaches:
wherein calculating the standard deviation comprises calculating the standard deviation of the number of object samples in each group. (Zhang [Page 9763, Column 2, Algorithm 1]: “for each ground-truth g … compute IoU between C_g and g … compute standard deviation of D_g … compute IoU threshold for ground-truth …” Zhang teaches that the groups are chosen from the classes of samples using a loss called IoU. There is a threshold that needs to be met for the IoU. This is based on the standard deviation of the group. As the IoU needs to be greater than the threshold, that means that the standard deviation, or the variation, has to be small. The smallest variation best provides the groupings, which is the minimum standard deviation. The combination of object samples is necessarily from amongst all possible combinations.)
The reasons to combine are substantially similar to those of claim 1.
Regarding dependent claim 8, Lin and Zhang teach:
The method according to claim 1,
Zhang teaches:
wherein the determining comprises: for each of the annotated object samples, calculating the importance score based on an object sample confidence value and a relevance value, where the objects ample confidence value is larger for manually annotated samples than for automatically annotated samples, and the relevance value is higher for a dataset considered more relevant for the use case the neural network is trained for than for datasets more remote from the use case. (Zhang [Page 9761, Column 2, Paragraph 1]: “Finally, the Non-Maximum Suppression (NMS) is applied with the IoU threshold 0.6 per class to generate final top 100 confident detections per image. Zhang teaches a confidence value of the detection samples. Zhang chooses the top 100 detections with the highest confidence values, which shows that the groupings based on the importance scores are based on this confidence value.)
The reasons to combine are substantially similar to those of claim 1.
Regarding dependent claim 9, Lin and Zhang teach:
The method according to claim 8,
Zhang teaches:
wherein the confidence value of automatically annotated samples is a confidence value obtained from a model or algorithm used for annotating the object samples. (Zhang [Page 9761, Column 2, Paragraph 1]: “Finally, the Non-Maximum Suppression (NMS) is applied with the IoU threshold 0.6 per class to generate final top 100 confident detections per image. Zhang teaches a confidence value of the detection samples. Zhang chooses the top 100 detections with the highest confidence values, which shows that the groupings based on the importance scores are based on this confidence value.)
The reasons to combine are substantially similar to those of claim 1.
Regarding dependent claim 10, Lin and Zhang teach:
The method according to claim 8,
Zhang teaches:
wherein calculating the importance score includes adjusting a tuning factor for adjusting the relative importance of the object sample confidence value and a relevance value when calculating the importance score. (Zhang [Page 9763, Column 1, Paragraph 3]: “we select k anchor boxes whose center are closest to the center of g based on L2 distance. Supposing there are L feature pyramid levels, the ground-truth box g will have k x L candidate positive samples.” Zhang teaches a hyperparameter k which is responsible for the number of k anchors or groups that are chosen. This indubitably affects the sample confidence and relevance value that are mentioned above.)
The reasons to combine are substantially similar to those of claim 1.
Regarding dependent claim 11, Lin and Zhang teach:
The method according to claim 1,
Zhang teaches:
wherein the set of importance score thresholds comprise at least 3, or at least 5, or at least 8 importance score thresholds. (Zhang [Page 9763, Column 2, Algorithm 1]: “for each ground-truth g … compute IoU between C_g and g … compute standard deviation of D_g … compute IoU threshold for ground-truth …” Zhang teaches that the groups are chosen from the classes of samples using a loss called IoU. There is a threshold that needs to be met for the IoU. This is based on the standard deviation of the group. Note that in each iteration, a set/group is created. For each group, the IoU and the IoU threshold is calculated, which shows that for each group, there is a different threshold. Therefore, if the number of groups is greater than 3, there are more than 3 thresholds.)
The reasons to combine are substantially similar to those of claim 1.
Regarding dependent claim 12, Lin and Zhang teach:
The method according to claim 1,
Zhang teaches:
wherein the neural network is a Convolutional Neural Network. (Zhang [Page 9765, Column 2, Paragraph 1]: “We further use the Deformable Convolutional Networks (DCN) to the ResNet and ResNeXt backbones as well as the last layer of detector towers.” Zhang teaches that the algorithm is applied on a convolutional neural network.)
The reasons to combine are substantially similar to those of claim 1.
Claim 13 is substantially similar to claim 1, but has the following additional limitations:
Regarding independent claim 13, Zhang teaches:
A control unit for selecting object samples for training of a neural network for object detection and classification from more than one dataset comprising annotated object samples of at least two object classes, the control unit being configured to: (Zhang [Page 9759, Column 1, Paragraph 2]: “Object detection is a long-standing topic in the field of computer vision, … Accurate object detection would have far reaching impact on various applications including image recognition and video surveillance. In recent years, with the development of convolutional neural network (CNN), object detection has been dominated by anchor-based detectors” Zhang teaches that the training for the neural network is in the domain of computer vision, which requires a computer and thus a processing control unit.)
The reasons to combine are substantially similar to those of claim 1.
Claim 14 is substantially similar to claim 1, but has the following additional limitations:
Regarding independent claim 14, Zhang teaches:
A system comprising an image capturing device for capturing images of a scene including objects, and a control unit configured to operate a neural network model for detecting and classifying objects in the scene, the neural network model having been trained on object samples selected according to a method for selecting object samples for training of a neural network from more than one dataset comprising annotated object samples of at least two object classes, the method comprising: (Zhang [Page 9763, Column 1, Paragraph 3]: “Algorithm 1 describes how the proposed method works for an input image. For each ground-truth box g on the image, we first find out its candidate positive samples.” Zhang teaches that the input data is in the form of images, which must be captured by an image capturing device. This process is part of the system of collecting and analyzing data.)
The reasons to combine are substantially similar to those of claim 1.
Claim 15 is substantially similar to claim 1, but has the following additional limitations:
Regarding independent claim 15, Liu teaches:
A non-transitory computer-readable storage medium having stored thereon a computer program comprising instructions which, when the program is executed by a computer, cause the computer to carry out a method for selecting object samples for training of a neural network for object detection and classification from more than one dataset comprising annotated object samples of at least two object classes, the method comprising: (Liu [Page 8, Column 1, Paragraph 4]: “The experiments were executed on a machine equipped with an eight-core Intel i7-6700 CPU, a 3.4 GHz processor and 32 GB of RAM” Liu teaches that the datasets and the current model version are stored in memory.)
The reasons to combine are substantially similar to those of claim 1.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to KYU HYUNG HAN whose telephone number is (703) 756-5529. The examiner can normally be reached on MF 9-5.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Alexey Shmatov can be reached on (571) 270-3428. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Kyu Hyung Han/
Examiner
Art Unit 2123
/ALEXEY SHMATOV/Supervisory Patent Examiner, Art Unit 2123