DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
This action is made non-final.
Claims 1-20 are pending. Claims 1, 10 and 19 are independent claims.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Regarding claim 1:
Step 1: This part of the eligibility analysis evaluates whether the claim falls within any statutory category. See MPEP 2106.03. Claim 1 recites: A computer-implemented method for optimizing models for open-vocabulary detection, comprising… Claim 1 is directed to a method (Step 1: YES).
Step 2A prong 1: Does the claim recite a judicial exception? Claim 1 recites: obtaining region proposals (a human can propose regions of interest within visual data)… obtaining object feature predictions (a human can mentally determine object feature predictions from visual data) filtering object feature predictions above a proposal threshold to obtain pseudo labels (a human can mentally compare numbers or probabilities to a threshold and determine labels for items, or guess if an item is unknown)… These steps can be performed mentally or are mathematical calculations (Step 2A prong 1: YES).
Step 2A prong 2: Does the claim recite additional elements? Do those additional elements, considered individually and in combination, integrate the judicial exception into a practical application? Claim 1 recites: by employing a pre-trained vision-language model and a pre-trained region proposal network… by employing a trained teacher neural network with the region proposals… training a student neural network with a split-and-fusion detection head by utilizing the region proposals, ground truth class labels, ground truth bounding boxes, and the pseudo labels; optimizing the pseudo labels by reducing noise from the pseudo labels by employing the trained split-and-fusion detection head of the trained student neural network to obtain optimized object detections; and performing an action relative to a scene layout with the optimized object detections. Employing pretrained models, training a model using certain data, and optimizing a model by reducing pseudo label noise by employing a “split-and-fusion detection head” of the trained model (i.e., optimize model output by using the trained model), is an attempt to use the neural network models by merely applying the abstract idea (i.e., perform the mental processes/math) without placing any limits on how the models operate, or limits placing any specific limits on model structure. Further, the claim omits any details as to how the models solve a technical problem and instead recites only the idea of a solution or outcome. See MPEP 2106.05(f). Thus, the limitation represents no more than mere instructions to implement the abstract idea which is equivalent to adding the words “apply it” to the recited judicial exception. Performing an action relative to the scene layout using the optimized detection is more instructions to “apply” the judicial exception (Step 2A prong 2: NO).
Step 2B: These elements are recited at such a high level of generality that they fail to integrate the abstract idea into a practical application, since they provide nothing more than mere instructions to implement an abstract idea on a generic computer (MPEP 2106.05(f)). These limitations, taken either alone or in combination, fail to provide an inventive concept (Step 2B: NO). Thus, the claim is not patent eligible.
Regarding claims 2-9, they recite limitations which further narrow the abstract idea by specifying more details of the mental and mathematical process that occurs (Claim 2, updating parameters of one neural network with parameters from another neural network is a mental process; Claim 3, splitting a detection head into a closed branch and open branch to obtain optimized pseudo labels still omits any details as to how the splitting solves a technical problem and instead recites only the idea of a solution or outcome – closed and open branch can be interpreted broadly, and are not associated with any specific data structure; Claim 4, training the closed branch with certain data is insignificant extra-solution activity of selecting types of data to be manipulated (see MPEP 2106.05(g)); Claim 5, employing cross-entropy loss is a mathematical calculation; Claim 6, employing box regression loss is a mathematical calculation; Claim 7, training the open branch with certain data is insignificant extra-solution activity of selecting types of data to be manipulated (see MPEP 2106.05(g)); Claim 8, employing cross-entropy loss is a mathematical calculation; Claim 9, fusing scores by using a geometric mean is a mathematical calculation).
Regarding claim 10, it is a system (i.e., an apparatus) that implements a method similar to the method of claim 1 and is rejected on the same grounds – see above.
Regarding claims 11-18, they recite similar limitations to claims 2-9 respectively and are rejected on the same grounds – see above.
Regarding claim 19,
Step 1: This part of the eligibility analysis evaluates whether the claim falls within any statutory category. See MPEP 2106.03. Claim 19 recites: A non-transitory computer program product comprising a computer-readable storage medium including program code for optimizing models for open-vocabulary detection wherein the program code when executed on a computer causes the computer to perform… Claim 19 is directed to an apparatus (Step 1: YES).
Step 2A prong 1: Does the claim recite a judicial exception? Claim 19 recites: obtaining region proposals (a human can propose regions of interest within visual data)… obtaining object feature predictions (a human can mentally determine object feature predictions from visual data)… filtering object feature predictions above a proposal threshold to obtain pseudo labels (a human can mentally compare numbers or probabilities to a threshold and determine labels for items, or guess if an item is unknown)… by fusing prediction scores from a closed-branch and an open-branch by computing a geometric mean of the prediction scores at inference time (computing a geometric mean between scores is a mathematical calculation)… These steps can be performed mentally or are mathematical calculations (Step 2A prong 1: YES).
Step 2A prong 2: Does the claim recite additional elements? Do those additional elements, considered individually and in combination, integrate the judicial exception into a practical application? Claim 19 recites: by employing a pre-trained vision-language model and a pre-trained region proposal network;… by employing a trained teacher neural network with the region proposals… training a student neural network with a split-and-fusion detection head by utilizing the region proposals, base ground truth class labels, ground truth bounding boxes, and the pseudo labels; optimizing the pseudo labels by reducing noise from the pseudo labels by employing the trained split-and-fusion detection head of the trained student neural network to obtain optimized object detections… and performing an action relative to a scene layout with the optimized object detections. Employing pretrained models, training a model using certain data, and optimizing a model by reducing pseudo label noise by employing a “split-and-fusion detection head” of the trained model (i.e., optimize model output by using the trained model), is an attempt to use the neural network models by merely applying the abstract idea (i.e., perform the mental processes/math) without placing any limits on how the models operate, or limits placing any specific limits on model structure. Further, the claim omits any details as to how the models solve a technical problem and instead recites only the idea of a solution or outcome. See MPEP 2106.05(f). Thus, the limitation represents no more than mere instructions to implement the abstract idea which is equivalent to adding the words “apply it” to the recited judicial exception. Performing an action relative to the scene layout using the optimized detection is more instructions to “apply” the judicial exception (Step 2A prong 2: NO).
Step 2B: These elements are recited at such a high level of generality that they fail to integrate the abstract idea into a practical application, since they provide nothing more than mere instructions to implement an abstract idea on a generic computer (MPEP 2106.05(f)). These limitations, taken either alone or in combination, fail to provide an inventive concept (Step 2B: NO). Thus, the claim is not patent eligible.
Regarding claim 20, it recites similar limitations to claim 2 and is rejected on the same grounds – see above.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-5, 7, 8, 10-14 and 16-17 is/are rejected under 35 U.S.C. 103 as being unpatentable over Chang et al. (US 20200302230 A1), herein Chang, in view of Long et al. (P3OVD: Fine-grained Visual-Text Prompt-Driven Self-Training for Open-Vocabulary Object Detection), and Hu et al. (US 20230342606 A1), herein Hu.
Regarding claim 1, Chang teaches: A computer-implemented method for optimizing models (¶70, In one embodiment, the present invention is directed to a machine-readable or computer-readable medium containing a machine-executable or computer-executable code that when read by a machine or computer causes the machine or computer to perform a method for incremental learning object detection)… obtaining object feature predictions by employing a trained teacher neural network with the region proposals (Fig. 1, 112 – ¶27, For each input image, the teacher network 110 yields a teacher region proposal (RPN(T)) 111 used to extract a feature vector 112 and a teacher object prediction 113 identifying one of the previously-learned object classes as corresponding to an object in each region of the teacher RPN)… obtain pseudo labels (¶27, For each input image, the teacher network 110 yields a teacher region proposal (RPN(T)) 111 used to extract a feature vector 112 and a teacher object prediction); training a student neural network with a split-and-fusion detection head by utilizing the region proposals, ground truth class labels, ground truth bounding boxes, and the pseudo labels (¶26, A current version of the two-stage neural network object detector named “Student Network (S)” is trained to detect objects pertaining to a new object class. The following description refers to a single new object class, but plural object classes may be added successively or simultaneously – and – ¶28-29, A region proposal network distillation loss 131 quantifying a decrease of student network's ability to locate objects of the previously-learned object classes is based on comparing the teacher region proposal 111 with the student region proposal 121. An object identification distillation loss 132 quantifying a decrease of student network's ability to identify the objects of the previously-learned object classes is based on comparing the teacher object prediction 112 with the student object prediction 122. Further, comparing the student network's outputs with ground-truth labels associated with the training image enables estimating student network's region proposal loss… and student network's object identification loss 142 … “ground truth” refers to the training set's classification label or simpler said to the objects in the training images real location and identity – the “split” can be seen in how there is a both a distillation loss and a “normal” loss for the student, for both the region prediction and object identification processes – see fig. 1 – the “fusion” is discussed in ¶30, In one embodiment, a total loss is defined by combining the region proposal network distillation loss, the object identification distillation loss, the region proposal network loss and the object identification loss. The student network may then be updated so as to minimize the total loss); and performing an action relative to a scene layout with the optimized object detections (¶61, The detection results are then output to the user: shown (e.g., on screen 408)).
Chang fails to teach: for open-vocabulary detection, comprising: obtaining region proposals by employing a pre-trained vision-language model and a pre-trained region proposal network… filtering object feature predictions above a proposal threshold to…
However, in the same field of endeavor, Long teaches: for open-vocabulary detection, comprising: obtaining region proposals by employing a pre-trained vision-language model and a pre-trained region proposal network (pg. 3, fig. 3, in step (a), a pretrained vision-language model (VLM) is used, and in (b) a pretrained region proposal network (RPN) is used for open-vocabulary object detection (OVD))… filtering object feature predictions above a proposal threshold to (pg. 5, IV, B, For the pseudo labeling stage, the RPN is trained only on the base classes in the 2X schedule. The objectness score threshold for RPN is set to 0.98 – objectness can be used to measure proposal confidence: pg. 7, objectness threshold for RPN to filter the low-confident proposals)…
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to use a pretrained VLM and RPN, as well as feature prediction thresholds as disclosed by Long in the method disclosed by Chang to take advantage of VLM classification capabilities and RPN localization capabilities, while filtering out low confidence predictions (pg. 1, Benefiting from the massive scale of pre-trained datasets collected from the web [35], [39], recent vision-language pre-training models (VLM), e.g., CLIP [22], have shown a surprising zero-shot classification capability – and – pg. 3, fig. 3 caption, to obtain non-base classes’ pseudo labels by utilizing the dense classification ability of prompt-enhanced VLM and the localization ability from a pre-trained RPN – and – pg. 7, objectness threshold for RPN to filter the low-confident proposals).
Chang in view of Long fails to explicitly teach: optimizing the pseudo labels by reducing noise from the pseudo labels by employing the trained split-and-fusion detection head of the trained student neural network to obtain optimized object detections (Chang does employ the trained split-and-fusion detection head of the trained network to obtain optimized object detections)…
However, in the same field of endeavor, Hu teaches: optimizing the pseudo labels by reducing noise from the pseudo labels by employing(¶27, Then, a student model is trained by using an expanded labeled data set… and the teacher model is updated by using the trained student model. As such, iterations are repeated until the student model converges – while Hu does not discuss a trained split and fusion detection head, or object detection specifically, Hu does discuss making improvements to the teacher model based on student model parameters – this should reduce erroneous pseudo labels (i.e., noise from the pseudo labels) as long as the student network is being trained successfully, such as with the method of Chang).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to update teacher parameters periodically with student parameters as disclosed by Hu in the method disclosed by Chang in view of Long to overcome data shortages (¶27, As described above, the specification provides a solution that can break through a limitation of labeled data shortage during GNN training. The solution includes a way of self-training that solves, among others, the problem of scarcity of labeled data by making full use of abundant unlabeled data).
Regarding claim 2, Chang in view of Long fails to teach: The computer-implemented method of claim 1, further comprising periodically updating the teacher neural network with parameters of the student neural network to self-train the teacher neural network and the student neural network.
However, in the same field of endeavor, Hu teaches: further comprising periodically updating the teacher neural network with parameters of the student neural network to self-train the teacher neural network and the student neural network (¶27, Then, a student model is trained by using an expanded labeled data set… and the teacher model is updated by using the trained student model. As such, iterations are repeated until the student model converges).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to update teacher parameters periodically with student parameters as disclosed by Hu in the method disclosed by Chang in view of Long to overcome data shortages (¶27, As described above, the specification provides a solution that can break through a limitation of labeled data shortage during GNN training. The solution includes a way of self-training that solves, among others, the problem of scarcity of labeled data by making full use of abundant unlabeled data).
Regarding claim 3, Chang further teaches: The computer-implemented method of claim 1, wherein optimizing the pseudo labels further comprises splitting a detection head into a closed-branch and an open-branch (¶8, The object detector is updated using a total loss that enables balancing an ability to detect new class objects with an ability to detect previously-learned classes objects without requiring access to the images previously used to train the object detector. The trained and updated object detector retains the ability to detect objects pertaining to previously-learned object classes, while extending its detection ability to objects pertaining to the new object classes – the distillation losses in fig. 1 are associated with the open branch, and the non-distillation losses are associated with the closed branch).
Chang fails to explicitly teach: to obtain optimized pseudo labels.
However, in the same field of endeavor, Hu teaches: to obtain optimized pseudo labels (¶27, Then, a student model is trained by using an expanded labeled data set… and the teacher model is updated by using the trained student model. As such, iterations are repeated until the student model converges).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to obtain optimized pseudo labels by updating the teacher model as disclosed by Hu in the method disclosed by Chang in view of Long to overcome data shortages (¶27, As described above, the specification provides a solution that can break through a limitation of labeled data shortage during GNN training. The solution includes a way of self-training that solves, among others, the problem of scarcity of labeled data by making full use of abundant unlabeled data).
Regarding claim 4, Chang further teaches: The computer-implemented method of claim 3, wherein the closed-branch is trained with ground truth bounding boxes and ground truth class labels (¶32, Training images have associated ground-truth labels indicating locations (e.g. regions which may be boxes) and object classes of the objects therein, respectively – the RPN and RCNN loss are shown in fig,.1 – RCNN is the recognition model, and RPN is the region proposal network).
Regarding claim 5, Chang in view of Hu fails to teach: The computer-implemented method of claim 4, wherein the trained closed-branch employs cross entropy loss to obtain class loss for classification.
However, in the same field of endeavor, Long teaches: wherein the trained closed-branch employs cross entropy loss to obtain class loss for classification (pg. 5, The objective of self-training stage… is calculated as …Where the LCLS and LREG denote the cross-entropy loss for classification head and L1-loss for regression head).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to utilize a cross-entropy loss for classification as disclosed by Long in the method disclosed by Chang in view of Hu to calculate multiple class classification loss for use in a training pipeline (pg. 5, we are able to train a final open-vocabulary detector Φd to fulfill the self-training pipeline).
Regarding claim 7, Chang further teaches: The computer-implemented method of claim 3, wherein the open-branch is trained with the pseudo labels and ground truth class labels (fig. 1, the open branch, associated with distillation-based losses 131 and 132, directly involve labels produced by the teacher network, i.e., pseudo labels, and also the ground truth labels, as the RCNN loss contributes to student network model weights. Because the ground truth class labels influence the student network parameters, they also impact the distillation loss).
Regarding claim 8, Chang in view of Hu fails to teach: The computer-implemented method of claim 7, wherein the open-branch employs cross entropy loss to obtain class loss for classification.
However, in the same field of endeavor, Long teaches: wherein the open-branch employs cross entropy loss to obtain class loss for classification (pg. 5, The objective of self-training stage… is calculated as …Where the LCLS and LREG denote the cross-entropy loss for classification head and L1-loss for regression head).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to utilize a cross-entropy loss for classification as disclosed by Long in the method disclosed by Chang in view of Hu to calculate multiple class classification loss for use in a training pipeline (pg. 5, we are able to train a final open-vocabulary detector Φd to fulfill the self-training pipeline).
Regarding claim 10, it is an apparatus that recites similar limitations to claim 1 and is rejected on the same grounds – see above.
Regarding claims 11-14 and 16-17, they recite limitations similar to claims 2-5 and 7-8 respectively and are rejected on the same grounds – see above.
Claim(s) 6 and 15 is/are rejected under 35 U.S.C. 103 as being unpatentable over Chang in view of Long and Hu as applied to claims 4 and 13 above, and further in view of Choi et al. (US 20180268292 A1), herein Choi.
Regarding claim 6, Chang in view of Long and Hu fails to explicitly teach: The computer-implemented method of claim 4, wherein the trained closed-branch employs box regression loss to obtain box loss for localization.
However, in the same field of endeavor, Choi teaches: wherein the trained closed-branch employs box regression loss to obtain box loss for localization (¶70, An L2 boundary loss layer is further employed to achieve distillation for bounding box regression).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to utilize a box regression loss as disclosed by Choi in the method disclosed by Chang in view of Long and Hu to help transfer localization knowledge (¶71, bounding box regression that allows transferring knowledge of better localization accuracy).
Regarding claim 15, it recites limitations similar to claim 6 and is rejected on the same grounds – see above.
Claim(s) 9 and 18 is/are rejected under 35 U.S.C. 103 as being unpatentable over Chang in view of Long and Hu as applied to claims 1 and 10 above, and further in view of Niyaz et al. (“AUGMENTING KNOWLEDGE DISTILLATION WITH PEER-TO-PEER MUTUAL LEARNING FOR MODEL COMPRESSION”, 2021), herein Niyaz, and Mroueh et al. (US 20200342361 A1), herein Mroueh.
Regarding claim 9, Chang in view of Long and Hu fails to teach: The computer-implemented method of claim 1, wherein optimizing the pseudo labels further comprises fusing prediction scores of a closed-branch and an open-branch of the detection head…
However, in the same field of endeavor, Niyaz teaches: wherein optimizing the pseudo labels further comprises fusing prediction scores of a closed-branch and an open-branch of the detection head (see pg. 2, fig 1, which displays split student networks that learn via traditional distillation as well as from other student networks – the classification predictions from students are eventually ensembled: pg. 3, The ensemble class probabilities… of the multi-student network for the classification task is defined as ŝ = max{sk}Kk=1 and final label prediction ŷ = argmaxc∈C ŝ – the open branch will be interpreted under BRI to be a larger model, and the closed branch will be interpreted as a smaller model – Niyaz uses an experimental setup of a large teacher and two students, the two students being different sized models: see pg. 3, Table 1)…
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to utilize a multiple student model approach with fused predictions as disclosed by Niyaz in the method disclosed by Chang in view of Long and Hu to improve knowledge transfer (pg. 2, passing the predictions from the teacher to each of the students and simultaneously sharing of logits information between the students at each iteration – and – Abstract, establishing the benefit of augmenting knowledge transfer from teacher to students with peer-to-peer learning between students).
Chang in view of Long, Hu and Niyaz fails to teach: by computing a geometric mean of the prediction scores at inference time to obtain optimized pseudo labels.
However, in the same field of endeavor, Mroueh teaches: by computing a geometric mean of the prediction scores at inference time to obtain optimized pseudo labels (¶31, Model ensembling consists in combining many models into a stronger, robust and more accurate model. Ensembling is ubiquitous in machine learning and yields to improved accuracies across multiple prediction tasks such as mutli-class or multi-label classification. For instance in deep learning, output layers of DNNs (Deep Neural Networks) such as softmaxes or sigmoids are usually combined using a simple arithmetic or geometric mean).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to use a geometric mean to combine prediction scores as disclosed by Mroueh in the method disclosed by Chang in view of Long, Hu and Niyaz to capture model consensus (¶31, The arithmetic mean rewards confidence of the models and the geometric means seeks the consensus across models).
Regarding claim 18, it recites similar limitations to claim 9 and is rejected on the same grounds – see above.
Claim(s) 19 and 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Chang in view of Long, Hu, Niyaz and Mroueh.
Regarding claim 19, Chang teaches: A non-transitory computer program product comprising a computer-readable storage medium including program code for optimizing models (¶70, In one embodiment, the present invention is directed to a machine-readable or computer-readable medium containing a machine-executable or computer-executable code that when read by a machine or computer causes the machine or computer to perform a method for incremental learning object detection)… obtaining object feature predictions by employing a trained teacher neural network with the region proposals (Fig. 1, 112 – ¶27, For each input image, the teacher network 110 yields a teacher region proposal (RPN(T)) 111 used to extract a feature vector 112 and a teacher object prediction 113 identifying one of the previously-learned object classes as corresponding to an object in each region of the teacher RPN); training a student neural network with a split-and-fusion detection head by utilizing the region proposals, base ground truth class labels, ground truth bounding boxes, and the pseudo labels (¶26, A current version of the two-stage neural network object detector named “Student Network (S)” is trained to detect objects pertaining to a new object class. The following description refers to a single new object class, but plural object classes may be added successively or simultaneously – and – ¶28-29, A region proposal network distillation loss 131 quantifying a decrease of student network's ability to locate objects of the previously-learned object classes is based on comparing the teacher region proposal 111 with the student region proposal 121. An object identification distillation loss 132 quantifying a decrease of student network's ability to identify the objects of the previously-learned object classes is based on comparing the teacher object prediction 112 with the student object prediction 122. Further, comparing the student network's outputs with ground-truth labels associated with the training image enables estimating student network's region proposal loss… and student network's object identification loss… “ground truth” refers to the training set's classification label or simpler said to the objects in the training images real location and identity (i.e. true label and true bounding box) – the “split” can be seen in how there is a both a distillation loss and a “normal” loss for the student, for both the region prediction and object identification processes – see fig. 1 – the “fusion” is discussed in ¶30, In one embodiment, a total loss is defined by combining the region proposal network distillation loss, the object identification distillation loss, the region proposal network loss and the object identification loss. The student network may then be updated so as to minimize the total loss)… and performing an action relative to a scene layout with the optimized object detections (¶61, If object detection is performed locally… then the object detector is run on the local computing device… The detection results are then output to the user: shown (e.g., on screen 408)).
Chang fails to teach: for open-vocabulary detection wherein the program code when executed on a computer causes the computer to perform: obtaining region proposals by employing a pre-trained vision-language model and a pre-trained region proposal network… filtering object feature predictions above a proposal threshold to obtain pseudo labels;
However, in the same field of endeavor, Long teaches: for open-vocabulary detection wherein the program code when executed on a computer causes the computer to perform: obtaining region proposals by employing a pre-trained vision-language model and a pre-trained region proposal network (pg. 3, fig. 3, in step (a), a pretrained vision-language model (VLM) is used, and in (b) a pretrained region proposal network (RPN) is used for open-vocabulary object detection (OVD)) … filtering object feature predictions above a proposal threshold to obtain pseudo labels (pg. 5, IV, B, For the pseudo labeling stage, the RPN is trained only on the base classes in the 2X schedule. The objectness score threshold for RPN is set to 0.98)…
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to use a pretrained VLM and RPN, as well as feature prediction thresholds as disclosed by Long in the method disclosed by Chang to take advantage of VLM classification capabilities and RPN localization capabilities, while filtering out low confidence predictions (pg. 1, Benefiting from the massive scale of pre-trained datasets collected from the web [35], [39], recent vision-language pre-training models (VLM), e.g., CLIP [22], have shown a surprising zero-shot classification capability – and – pg. 3, fig. 3 caption, to obtain non-base classes’ pseudo labels by utilizing the dense classification ability of prompt-enhanced VLM and the localization ability from a pre-trained RPN – and – pg. 7, objectness threshold for RPN to filter the low-confident proposals).
Chang in view of Long fails to teach: optimizing the pseudo labels by reducing noise from the pseudo labels by employing the trained split-and-fusion detection head of the trained student neural network to obtain optimized object detections (Chang does employ the trained split-and-fusion detection head of the trained network to obtain optimized object detections)…
However, in the same field of endeavor, Hu teaches: optimizing the pseudo labels by reducing noise from the pseudo labels by employing the trained split-and-fusion detection head of the trained student neural network to obtain optimized object detections (¶27, Then, a student model is trained by using an expanded labeled data set… and the teacher model is updated by using the trained student model. As such, iterations are repeated until the student model converges – while Hu does not discuss a trained split and fusion detection head or object detection specifically, Hu does discuss making improvements to the teacher model based on student model parameters – this should reduce erroneous pseudo labels (i.e., noise from the pseudo labels) as long as the student network is being trained successfully, such as with the method of Chang).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to update teacher parameters periodically with student parameters as disclosed by Hu in the method disclosed by Chang in view of Long to overcome data shortages (¶27, As described above, the specification provides a solution that can break through a limitation of labeled data shortage during GNN training. The solution includes a way of self-training that solves, among others, the problem of scarcity of labeled data by making full use of abundant unlabeled data).
Chang in view of Long and Hu fails to teach: by fusing prediction scores from a closed-branch and an open-branch…
However, in the same field of endeavor, Niyaz teaches: by fusing prediction scores from a closed-branch and an open-branch (see pg. 2, fig 1, which displays split student networks that learn via traditional distillation as well as from other student networks – the classification predictions from students are eventually ensembled: pg. 3, The ensemble class probabilities… of the multi-student network for the classification task is defined as ŝ = max{sk}Kk=1 and final label prediction ŷ = argmaxc∈C ŝ – the open branch will be interpreted under BRI to be a larger model, and the closed branch will be interpreted as a smaller model – Niyaz uses an experimental setup of a large teacher and two students, the two students being different sized models: see pg. 3, Table 1)…
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to utilize a multiple student model approach with fused predictions as disclosed by Niyaz in the method disclosed by Chang in view of Long and Hu to improve knowledge transfer (pg. 2, passing the predictions from the teacher to each of the students and simultaneously sharing of logits information between the students at each iteration – and – Abstract, establishing the benefit of augmenting knowledge transfer from teacher to students with peer-to-peer learning between students).
Chang in view of Long, Hu and Niyaz fails to teach: by computing a geometric mean of the prediction scores at inference time…
However, in the same field of endeavor, Mroueh teaches: by computing a geometric mean of the prediction scores at inference time (¶31, Model ensembling consists in combining many models into a stronger, robust and more accurate model. Ensembling is ubiquitous in machine learning and yields to improved accuracies across multiple prediction tasks such as mutli-class or multi-label classification. For instance in deep learning, output layers of DNNs (Deep Neural Networks) such as softmaxes or sigmoids are usually combined using a simple arithmetic or geometric mean)…
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to use a geometric mean to combine prediction scores as disclosed by Mroueh in the method disclosed by Chang in view of Long, Hu, and Niyaz to capture model consensus (¶31, The arithmetic mean rewards confidence of the models and the geometric means seeks the consensus across models).
Regarding claim 20, Chang in view of Long, Niyaz and Mroueh fails to teach: The non-transitory computer program product of claim 19, further comprising periodically updating the teacher neural network with parameters of the student neural network to self-train the teacher neural network and the student neural network.
However, in the same field of endeavor, Hu teaches: further comprising periodically updating the teacher neural network with parameters of the student neural network to self-train the teacher neural network and the student neural network (¶27, Then, a student model is trained by using an expanded labeled data set… and the teacher model is updated by using the trained student model. As such, iterations are repeated until the student model converges).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to update teacher parameters periodically with student parameters as disclosed by Hu in the method disclosed by Chang in view of Long, Niyaz and Mroueh to overcome data shortages (¶27, As described above, the specification provides a solution that can break through a limitation of labeled data shortage during GNN training. The solution includes a way of self-training that solves, among others, the problem of scarcity of labeled data by making full use of abundant unlabeled data).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Chen et al. (“Learning Efficient Object Detection Models with Knowledge Distillation”, 2017), applies knowledge distillation to object detection, including cross entropy loss and bounding box regression loss.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to HARRISON CHAN YOUNG KIM whose telephone number is (571)272-0713. The examiner can normally be reached Monday - Friday 10:00 am - 6:00 pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, CESAR PAULA can be reached at (571) 272-4128. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/HARRISON C KIM/ Examiner, Art Unit 2145
/CESAR B PAULA/ Supervisory Patent Examiner, Art Unit 2145
Prosecution Insights