Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1 and 11 are rejected under 35 U.S.C. 102 (a)(1) as being anticipated by Li et al. (non-patent literature titled “Symmetric Joint Detection and Re-Identification for UAV-based Multiple Object Tracking”; hereinafter “Li”).
In re to claim 1, Li teaches wherein: a method performed by at least one computing device, the method comprising: generating, based on a first frame and via a backbone of a detection model (model; Fig. 1 shows a model used to perform multi-object tracking, understood as the detection model), a first feature (Fig. 1 and section 3 para. 1-section 3.1 disclose the extraction of features and subsequent generation of feature maps from the backbone. It is understood that a given feature comprising a feature map is the first feature. See also Fig. 1 which shows the processing of an input image, understood as a first frame);
generating, based on the first feature (section 3.2 para. 1 discloses use of the feature maps generated by the backbone in for object detection) and via a first neck of the detection model, first detection information (section 3.2.1 para. 1 discloses the generation of bounding boxes with position and size information to denote a detected object, this is understood to be first detection information) indicating a detection result for a first object, wherein the first neck is configured for object detection (detection neck; Fig. 1 displays a detection branch that comprises a neck used for object detection, as described in section 3.2 para. 1 (see exemplary Fig. 1 below, the right side annotation highlights the detection branch));
Exemplary Figure 1: Li Fig. 1 with annotation of Re-ID branch and detection branch
generating, based on the first feature (Fig. 1 indicates use of the first feature, correspondent to the claims, as shown in exemplary Fig. 1’s left side annotation that highlights the Re-ID branch taking the backbone output (being feature maps, per section 3.1) as input. This is further corroborated by section 3.3 para. 1, which denotes use of output from the backbone layers) and via a second neck of the detection model (section 3.3 and Fig. 1 indicate the use of a branch that performs re-identification (Re-ID). Per section 3.3 para. 1, this branch is similar to the detection branch in design. Further, as shown in Fig. 1, the Re-ID branch produces output via a head based on input from the backbone. Thus, it is understood to comprise a neck that performs re-identification, being understood as the second neck), a first feature vector for a visual feature of the first object (section 3.3 para. 1 lines 9-15 discloses the generation of feature vectors for a predicted target (which is understood as a first object)), wherein the second neck is configured for object re-identification (exemplary Fig. 1’s left side annotation highlights the Re-ID branch, which is further noted to be for object re-identification per section 3.3); and
performing, based on the first detection information and the first feature vector, tracking of at least one object comprising the first object (section 4.1 lines 12-15 discloses evaluation of the system for multi-object tracking using the detection results and Re-ID feature outputs. Thus, disclosing the use of the first detection information and first feature vector for tracking at least one object).
As to claim 11, by virtue of being a neural network based system and thus using a processor and memory (as suggested by Fig. 1), it is the apparatus that performs the method of claim 1. As such it recites similar limitations and is rejected for the same reasons as provided above.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 2-5 and 12-15 are rejected under 35 U.S.C. 103 as being unpatentable over Li in further view of Peng et al. (non-patent literature titled “AN EFFICIENT PERSON REID METHED BASED ON KNOWLEDGE DISTILLATION”; hereinafter “Peng”).
In re to claim 2 [dependent on claim 1], Li teaches wherein: further comprising: training (section 3.4 para. 1 discloses training of the model and its respective first and second branches, and thus its first and second necks (respectively)) a second model, wherein the second model comprises the backbone, the first neck, and the second neck (model when undergoing training; section 3.4 para. 1 discloses training of the model shown in Fig. 1 by virtue of training its branches, said model including a backbone as shown in Fig. 1. It is understood that during the training process, the model used is the second model).
Li does not explicitly teach wherein: training is based on a first model pre-trained on the object re-identification
However, in a similar field of endeavor, Peng teaches wherein: training is based on a first model pre-trained on the object re-identification (section 3 para. 1 and section 3.1 para. 1 discloses use of a teacher network (MGN) to teach a student network regarding Re-ID tasks. Further, the teacher network is a trained network per section 4 subsection titled “result” para. 1 lines 2-4, which states “To train the student network, we used the trained MGN model on the DukeMTMC dataset as a teacher”).
Peng, like Li, teaches a system that trains a system for object re-identification.
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Li, in view of Peng, to train using knowledge distillation. The motivation for the proposed modification would have been to aid in the training of a smaller student network to improve its performance, as is a noted benefit of Peng detailed in the abstract and section 2 para. 1.
In re to claim 3 [dependent on claim 2], Li teaches wherein: the training of the second model comprises: training the second neck in a state where parameters of the backbone and the first neck are fixed (section 3.4 para. 1 discloses that during training of the module, the input image has a fixed size of 640x640. As the model takes input via the backbone per Fig. 1, it is understood that the backbone, and subsequently the first neck due to using the backbone as its input, have a fixed parameter. Additionally, per section 3.4 lines 1-4, the first neck has its parameters solidified to train the second neck (indicating additional fixed parameters)).
In re to claim 4 [dependent on claim 2], Li, in view of Peng, teaches wherein: the training of the second model comprises: generating, based on a first input image and via the first model, a second feature vector (Peng section 3.1 para. 2 and Peng Fig. 2 shows the output of a feature vector (understood as a second feature vector) from the first model (correspondent to the claims) based on an input image to said first model (understood as a first input image));
generating, based on a second input image and via the backbone and the second neck, a third feature vector (Peng section 3.1 para. 2 and Peng Fig. 2 shows the generation of a feature vector in relation to the training process (understood as a third feature vector) resultant from processing an input image using the backbone of the student re-identification neck (and as such does so using what is understood as the second neck). Additionally, it is understood that, when input to the second neck’s backbone, the input image is the second input image);
generating, based on the second feature vector and the third feature vector, a first loss (Peng Fig. 2 shows the use of a loss function using the second and third feature vectors (correspondent to the claims, each respectively). Further, it is understood that the resultant loss is the first loss); and
training, based on the first loss, the second neck (Peng Fig. 2 indicates the training, based on the first loss (correspondent to the claims), is of the second neck (correspondent to the claims)).
The reasons for combination are the same as provided above.
In re to claim 5 [dependent on claim 4], Li, in view of Peng, teaches wherein: the training of the second model comprises: generating, based on the third feature vector and ground truth (GT), a second loss, and wherein the training of the second neck comprises training, based on the first loss and the second loss, the second neck (Peng section 3.2 discloses the use of ground truth label data when performing knowledge distillation. Additionally, Peng section 3.2 details the use of loss value that uses the GT according to Peng equation 4. Thus, as the training uses the first loss (correspondent to the claims) and the loss of Peng equation 4 (understood as a second loss based on a third feature vector and the GT per Peng Fig. 2 and Peng section 3.2), the second neck is trained based on the GT as well as the first and second loss).
The reasons for combination are the same as provided above.
As to claims 12-15, they are the apparatus that performs the method of claims 2-15. As such they recite similar limitations and are rejected for the same reasons as provided above.
Claim 7 and 17 are rejected under 35 U.S.C. 103 as being unpatentable over Li, in view of Peng, in further view of Vivona et al. (US patent 12039012 B2; hereinafter “Vivona”).
In re to claim 7 [dependent on claim 5], Li, in view of Peng, does not explicitly teach wherein: the GT comprises hard labeled data.
However, in a similar field of endeavor, Vivona teaches wherein: the GT comprises hard labeled data (col. 11 lines 47-48 discloses the use of hard labeled ground truth labels in relation to training operations).
Vivona, like Li, teaches the training of models for the sake of addressing a model that processes data using a backbone based methodology that can be utilized with respect to image data.
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Li, in view of Peng, to use hard labeled GT data, as taught by Vivona. The motivation for the proposed modification would have been to train the system based on particular labels rather than a soft label GT, thereby more strongly directing the training process to a particular outcome.
As to claim 17, it is the apparatus that performs the method of claim 7. As such it recites similar limitations and is rejected for the same reasons as provided above.
Claims 10 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Li, in view of Peng, in further view of Ryan (US publication 20230040513 A1; hereinafter “Ryan”).
In re to claim 10 [dependent on claim 1], Li, in view of Peng teaches wherein:
the performing of the tracking comprises: generating an association obtained by combining the first detection information and the first feature vector (section 3.5 discloses the performance of association between detection and Re-ID feature);
generating, based on a second frame and via the backbone, a second feature, wherein the second frame is a next frame of the first frame (section 4.5 discloses the tracking of objects across a video sequence. Thus, as the system uses the backbone to generate a first feature, it is understood that a second feature (being a feature data generated by the system following a first frame input) is generated by a frame following the first when input into the model via the backbone (understood as a second frame, which is a next frame by virtue of tracking an object through a video indicating a passage of time));
generating, pieces of second detection information about a plurality of second objects (section 4.5 discloses the tracking of objects across a video sequence. Thus, as the system uses the backbone to generate a detection information, it is understood that second detection information is generated. Further, see Fig. 4, which shows the tracking of a plurality of objects (the objects being tracked that are not the first object being understood as a plurality of second objects));
generating, based on the second feature and via the second neck, fourth feature vectors for visual features (Fig. 1 shows extraction of visual features being used by the second neck, see also section ) of the plurality of second objects (Fig. 1 and section 3.3 para. 1 lines 9-15 discloses the generation of feature vectors for predicted targets, thus, the second objects (correspondent to the claims) are understood to have feature vectors generated (understood as fourth feature vectors). See also section 3 para. 1-section 3.1, which discloses the extraction of features and subsequent generation of feature maps from the backbone and its input image data (thus teaching feature vectors for visual features of the input image));
Li, in view of Peng does not explicitly teach wherein: second detection information based on the second feature nor generating a first association vector obtained by combining the first detection information and the first feature vector nor generating second association vectors, wherein the second association vectors are obtained by respectively combining the pieces of second detection information and the fourth feature vectors; and
performing the tracking by associating the first object with an object having an association vector, among the second association vectors, that is closest to the first association vector.
However, in a similar field of endeavor, Ryan teaches wherein: second detection information is based on the second feature ([0142] discloses generation of detection information (correspondent to the claims) for entities in images outside the first, including for the identified entity in additional images (the second detection information thus uses feature data based on a second feature)) as well as generating a first association vector obtained by combining the first detection information (initial embedding vector of entity; [0142] discloses identification of an entity based on determination of an embedding vector. See also [0165], which discloses an entity being identified by its embedding vector according to characteristics of an identified object (being detection information, with first detection information being characteristics of an initial entity) and further generates an embedding vector. The system links a given entity (identified by the embedding vector) with a characteristic feature vector, per [0141]. It is understood that an embedding vector linked to a characteristic feature vector (understood as a feature vector) is an association vector, the first association vector being one with respect to an initial entity being tracked) and the first feature vector ([0141] discloses linking a given entity with a characteristic feature vector, the characteristic feature vector of an initial entity being a first feature vector) as well as generating second association vectors, wherein the second association vectors are obtained by respectively combining the pieces of second detection information and the fourth feature vectors ([0141]-[0142] and [0165] discloses generation of embedding vectors and respective characteristic feature vectors associated with them.); and
performing the tracking by associating the first object with an object having an association vector, among the second association vectors, that is closest to the first association vector ([0142] discloses the searching of an embedding vector based on distance (and as such closeness) to the initial entity embedding vector (and as such their respective association vectors) in order to re-identify an entity as a part of the tracking process, as suggested by [0140]).
Ryan, like Li, teaches the tracking of detected objects within image data and the re-identification of said objects.
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Li, in view of Peng, to track objects based on association vectors. The motivation for the proposed modification would have been to enable the system to process image data of frames from a plurality of camera views, as is performed in Ryan by virtue of tracking the objects based on their unique identifiers between images with respect to Euclidean distances).
As to claim 20, it is the apparatus that performs the method of claim 10. As such it recites similar limitations and is rejected for the same reasons as provided above.
Allowable Subject Matter
Claims 6, 8, 9, 16, 18, and 19 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
The following is an examiner’s statement of reasons for allowance for claim 6 (and due to reciting similar limitations, claim 16). The claimed features of claim 6 is not anticipated nor obvious in view of prior art of record.
Li teaches a system that performs the detection of an object, as well as re-identification of said object, for the purpose of tracking. It additionally teaches the generation of feature vectors and detection information for the sake of further tracking operations as a result of training operations (see section 3.4). However, it does not explicitly teach the use of a first pre-trained model in order to train a second model that comprises the backbone, first neck, and second neck of the detection model. Nor does it further perform loss calculations in the manner required by the claims as a result.
Peng discloses the training of an object reidentification model with respect to knowledge distillation. Further using a pre-trained model to perform training, as is indicated by section 3 para. 1 and section 3.1 para. 1. It additionally shows the use of loss function in Fig. 2 and section 3.2 as a part of the training process. However, it does not indicate that the second loss is the result of a classification vector being compared to the GT while also indicating that this classification vector is based on the third feature vector and generated by a classification network. Further, while section 3.2 shows the use of a second neck to generate data used to calculate loss values via equations 1-5, these equations do not indicate use of a classification function that fits the requirements of the claims. Multiple vectors are used in the equations via the softmax functions, with some being the result of the teaching model’s output (being a feature vector). However, these vectors are not shown to be generated by a classification network. Additionally, while equation 5 shows comparison of a vector to the ground truth, this vector is not based on the third feature vector.
Additionally, the other known prior art or record do not address all the limitations of the independent claims without the use of impermissible hindsight bias. As such, claim 6 is neither anticipated nor rendered obvious in view of prior art of record.
The following is an examiner’s statement of reasons for allowance for claim 8 (and due to reciting similar limitations, claim 18). The claimed features of claim 8 is not anticipated nor obvious in view of prior art of record.
Li teaches a system that performs the detection of an object, as well as re-identification of said object, for the purpose of tracking. It additionally teaches the generation of feature vectors and detection information for the sake of further tracking operations as a result of training operations (see section 3.4). However, it does not explicitly teach the use of a first pre-trained model in order to train a second model that comprises the backbone, first neck, and second neck of the detection model. Nor does it further perform loss calculations in the manner required by the claims as a result. Nor does it explicitly state that the resultant second feature vector is soft labeled data.
Peng discloses the training of an object reidentification model with respect to knowledge distillation. Further using a pre-trained model to perform training, as is indicated by section 3 para. 1 and section 3.1 para. 1. It additionally shows the use of loss function in Fig. 2 and section 3.2 as a part of the training process. However, it does not indicate that the second feature vector is explicitly soft labeled data. It only mentions that the data generated by the first model is softened in section 3.2. As this indicates that the feature vector is changed to be softened, it does not indicate that the second feature vector itself comprises soft labeled data.
Vivona discloses the use of hard and soft labeled data in the course of training a model. Col. 11 lines 47-48 discloses the use of soft labeled ground truth labels in relation to training operations. However, it does not explicitly stat the some second feature vector data is generated by a first model. Nor does it further indicate said information is soft labeled data.
Additionally, the other known prior art or record do not address all the limitations of the independent claims without the use of impermissible hindsight bias. As such, claim 6 is neither anticipated nor rendered obvious in view of prior art of record.
The following is an examiner’s statement of reasons for allowance for claim 9 (and due to reciting similar limitations, claim 19). The claimed features of claim 9 are not anticipated nor obvious in view of prior art of record.
Li teaches a system that performs the detection of an object, as well as re-identification of said object, for the purpose of tracking. It additionally teaches the generation of feature vectors and detection information for the sake of further tracking operations as a result of training operations (see section 3.4). However, it does not explicitly teach the use of a first pre-trained model in order to train a second model that comprises the backbone, first neck, and second neck of the detection model.
Peng discloses the training of an object reidentification model with respect to knowledge distillation. Further using a pre-trained model to perform training, as is indicated by section 3 para. 1 and section 3.1 para. 1. It additionally shows the use of loss function in Fig. 2 and section 3.2 as a part of the training process. However, it does not teach that the input image for the teacher model is the result of an object area extraction of the second input image (used for the input to the second neck, per claim 4).
Additionally, the other known prior art or record do not address all the limitations of the independent claims without the use of impermissible hindsight bias. As such, claim 6 is neither anticipated nor rendered obvious in view of prior art of record.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to KEVIN M COOMBER whose telephone number is (571)270-0950. The examiner can normally be reached Monday - Friday 8:00am-5:00pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Gregory Morse can be reached at (571) 272-3838. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/KEVIN M COOMBER/Examiner, Art Unit 2663
/GREGORY A MORSE/Supervisory Patent Examiner, Art Unit 2698
Prosecution Insights