DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
Applicant’s arguments with respect to the rejection of the claims under 35 U.S.C. 112(b) as being indefinite have been fully considered and are persuasive. Accordingly, the rejection is withdrawn. Applicant’s arguments with respect to the rejections of independent claims 1, 11 and 20 under 35 U.S.C. 102(a)(2) as being anticipated by Kang have been fully considered, but are not persuasive. Nevertheless, a new ground of rejection necessitated by the amendments to the claims has been made.
In arguing differences between the claimed invention and the Kang reference, Applicant argues that Kang fails to show certain features of the invention. However, the features upon which applicant relies (i.e., pages 16-18 of Applicant’s Remarks/Arguments provide details of limitations of claim 1 that Applicant refers to as Op1-Op5 with reference to Tables 1-3) are not recited in claim 1. Although the claims are interpreted in light of the specification, limitations from the specification are not read into the claims. See In re Van Geuns, 988 F.2d 1181, 26 USPQ2d 1057 (Fed. Cir. 1993).
The main thrust of Applicant’s arguments is that the operations recited in claim 1 that Applicant refers to as Op1-Op5 are corrective, post-processing operations whereas the operations of Kang that the examiner cited in the nonfinal Office Action as corresponding to the Op1-Op5 operations are allegedly additive data association operations. However, the mere use of the phrase “post-processing” in the preamble of the independent claims without any further definition in the claim of what the term means is insufficient to distinguish the operations that follow the preamble from prior art that teaches the same operations. Furthermore, the present specification does not define what is meant by the term “post processing” except to indicate that it “reduces identification-switch”. (paras. [007]-[0072] of the present specification).
When reading the preamble in the context of the entire claim, the recitation of “post-processing” in the preamble is not limiting because the body of the claim describes a complete invention and the language recited solely in the preamble does not provide any distinct definition of any of the claimed invention’s limitations. Thus, the preamble of the claim is not considered a limitation and is of no significance to claim construction. See Pitney Bowes, Inc. v. Hewlett-Packard Co., 182 F.3d 1298, 1305, 51 USPQ2d 1161, 1165 (Fed. Cir. 1999). See MPEP § 2111.02.
In Applicant’s arguments, Applicant identified operations Op1-Op5 as corresponding to limitations recited in the independent claims. Applicant argues that Op1 of claim 1 is different from step 101 of Kang because “[s]tep S101 determines the target re-identification feature of each object in the target frame image (cf Step S102 of Kang, the determined target re-identification feature in step S10I of Kang has not been associated with any target ID). That is, the operation object of step SI0l in Kang is the ‘target frame image’, not the ‘tracklet’ Trkn-1[i] of the previously generated time period p'=[0,t'] (corresponding to image frames 1 to n-1).”
First of all, Op1 of claim 1 does not recite “tracklet Trkn-1[i] of the previously generated time period p'=[0,t'] (corresponding to image frames 1 to n-1).” Second, Applicant’s argument assumes that S101 of Kang is performed only on a target frame and not on the sequence of frames following the target frame, which would not make sense because the process of “re-identification” is used to tracking objects across sequences of image frames, as discussed in para. [0028] of Kang:
[p]erson re-identification (Re-ID) is a technology for determining whether there is a specific person in an image or video sequence using a computer vision technology. The present disclosure is not limited to person re-identification, but may include the identification of other target objects. That is to say, the target object re-identification in the present disclosure includes determining whether there is a specific object in an image or video sequence using the computer vision technology.” Therefore, the continuation of the process represented by S101 and S102 of Kang after it is performed for a current target image frame and the adjacent-in-time image frame constitutes post-processing.
Para. [0054] of Kang also describes Op1.
The Op2 limitation of claim 1 determines whether a “candidate identification switch image patch is present” based on similarities between re-identification features of image patches of the image patch sequence. The present specification describes this operation with reference to Figs. 1-2e as determining similarities between features sets of image patches in a sequence of image frames corresponding to two people walking near one another such that the corresponding bounding boxes “have been overlaid”. Due to their bounding boxes being overlaid and their features being sufficiently similar, a determination is made that an ID switch is likely to have occurred. If so, it is determined that a “candidate identification switch image patch is present”.
These operations are taught in paras. [0031]-[0032] of Kang: “[s]ince A and B have similar appearances, and the re-identification feature is the appearance feature using the existing technology, an incorrect ID switch may occur during data association. That is because a re-identification feature of the target object A is likely to match a historical re-identification feature of a tracker corresponding to the ID of (i.e., a corresponding vector distance is less than the predetermined threshold), and a re-identification feature of the target object B is likely to match a historical re-identification feature of a tracker corresponding to the ID of 23.”
Applicant argues that “Kang does not need to determine their appearance similarity (see claims 1-7 of Kang), because the determined ‘target re-identification features’ contain the position information of the target object, which can reduce ‘ID switching’. Paragraph [0032] of Kang only explains the advantage of the target tracking solution of Kang.”
The examiner disagrees. Kang does determine appearance similarity. See para. [0043]: “[s]pecifically, the first re-identification feature may include the visual feature (appearance feature) and/or the motion feature. That is to say, the first re-identification feature may include only one of the visual feature and the motion feature, or may include both of the visual feature and the motion feature.”
Kang teaches using a Hungarian algorithm as part of a core Deep Sort algorithm to perform the feature similarity determination (paras. [0060]-[0061]). Applicant argues that the Deep Sort algorithm is mainly concerned with data association, whereas the present invention is directed to post-processing, corrective operations. As indicated above, the present specification does not define what is meant by the term “post processing” except to indicate that it “reduces identification-switch”. (paras. [007]-[0072] of the present specification). Kang is directed to performing operations for “reducing the occurrence of incorrect ID switch in the use of the original FairMOT-based object tracking model.” (Kang, para. [0079]). This quoted language from the present specification and from Kang means the same thing.
Applicant also argues that Kang does not disclose splitting the tracklet into two tracklets based on the candidate identification switch image patch, updating the tracklet to each of the two tracklets respectively, and continuing to make attempts to split a current tracklet. The examiner agrees. However, as indicated above, a new ground of rejection is set forth below.
Claim Interpretation
The claims in this application are given their broadest reasonable interpretation (BRI) using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The BRI of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification.
BRIs for some of the claim terms in light of the specification are provided below. These definitions are used for purposes of searching for prior art, but cannot be incorporated into the claims.
Should Applicant believe that different interpretations are warranted, Applicant should point to the portions of the specification that clearly support a different interpretation.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1-4, 11-14 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over U.S. Publ. Appl. No. 2022/0301183 A1 to Kang et al. (hereinafter referred to as “Kang”) in view of an article entitled “Split and Connect: A Universal Tracklet Booster for Multi-Object Tracking”, by Wang et al., published in arXiv:2105.02426 in May 2021 (hereinafter referred to as “Wang”).
Regarding claim 1, Kang discloses a computer-implemented method (Fig. 3) for post-processing in multi-target tracking (paras. [0026]), characterized by comprising making attempts to split a tracklet indicative of a trajectory of a single target (para. [0034], Fig. 1, due to objects A and B having similar appearances, this could result in an incorrect ID switch leading to an incorrect attempt to split a tracklet indicative of a trajectory of object A) by performing operations of:
determining a re-identification feature set of an image patch sequence by determining a re-identification feature of each image patch in the image patch sequence of the tracklet (para. [0024], Fig. 1, step S101; also para. [0054]: “determining candidate box information and the first re-identification feature of each target object based on a pre-trained object detection network model, the candidate box information including candidate box position information”);
determining whether a candidate identification switch image patch is present in the tracklet based on feature similarities of a plurality of re-identification feature pairs in the re-identification feature set (paras. [0031]-[0032] of Kang: “[s]ince A and B have similar appearances, and the re-identification feature is the appearance feature using the existing technology, an incorrect ID switch may occur during data association. That is because a re-identification feature of the target object A is likely to match a historical re-identification feature of a tracker corresponding to the ID of (i.e., a corresponding vector distance is less than the predetermined threshold), and a re-identification feature of the target object B is likely to match a historical re-identification feature of a tracker corresponding to the ID of 23”; see also paras. [0060]-[0061] discussing the Hungarian algorithm of the Deep Sort algorithm);
in a case where a determination result is "yes", verifying whether it is credible that identification-switch has occurred at the candidate identification switch image patch (para. [0061], after candidate switch image patches are identified based on feature similarities, the Kalman filtering algorithm of the Deep SORT model verifies, based on position features encoded into, or fused with, the re-identification features of the candidate switch image patches, whether the candidate identification switch image patches correspond to the same or different objects, which means that it verifies that an identification switch has occurred or is likely to have occurred; the position determination corresponds to the “based on at least one predetermined condition” of claim 1);
in a case where a verification result is "credible", splitting the tracklet into two tracklets based on the candidate identification switch image patch;
wherein the operations further include:
updating the tracklet to each of the two tracklets respectively, and continuing to making attempts to split a current tracklet.
Kang does not explicitly disclose the claim 1 limitations of:
in a case where a verification result is "credible", splitting the tracklet into two tracklets based on the candidate identification switch image patch, wherein the operations further include updating the tracklet to each of the two tracklets respectively, and continuing to making attempts to split a current tracklet.
Wang, in the same field of endeavor, discloses a post-processing, corrective algorithm referred to as a tracker booster algorithm that “can be efficiently plugged into any existing trackers” and targets errors of: “1) different objects are associated to the same tracklet, 2) tracklets from the same object are assigned to different IDs.” The booster tracker algorithm includes a Splitter algorithm that “aims at finding potential ID-switch positions of a tracklet and split the tracklet into small pieces at the detected ID-switch positions.” The booster tracker algorithm is designed “to directly address the tem-poral association errors that exist in almost all trackers in the MOT field. Besides, the proposed tracklet booster can be integrated with any existing trackers to significantly improve their tracking performance” (See I. Introduction). In Wang, the Splitter algorithm is designed to “split tracklets into small pieces on the potential ID-switch positions to ensure split tracklets have purer IDs as much as possible.” (Section III. Method, Fig. 2).
Therefore, Wang discloses: in a case where a verification result is "credible" (Splitter algorithm that “aims at finding potential ID-switch positions of a tracklet), splitting the tracklet into two tracklets based on the candidate identification switch image patch (“split tracklets into small pieces on the potential ID-switch positions to ensure split tracklets have purer IDs as much as possible”), wherein the operations further include updating the tracklet to each of the two tracklets respectively (the tracklet is updated by assigning “the purer IDs” to the split tracklets), and continuing to making attempts to split a current tracklet (the Splitter algorithm continues attempting to split current tracklets received at the input of the Splitter shown in Fig. 2 as potential ID-switch positions are detected).
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the present disclosure, to modify the multi-target tracking (MTT) system and method of Kang to include the Splitter algorithm of Wang to perform splitting of tracklets into two tracklets based on verification by the system of Kang that a credible ID switch has occurred as taught by Wang. One of ordinary skill in the art would have been motivated to make the modification in order to correct for ID switch errors as taught by Wang. The modification could have been made by one of ordinary skill in the art before the effective filing date of the present disclosure with a reasonable expectation of success because making the modification merely involves combining prior art elements according to known methods to yield predictable results (Wang indicates that the tracklet booster can be “efficiently plugged into any existing trackers”, which would include the tracker of Kang).
Regarding claim 2, Kang discloses that determining whether a candidate identification switch image patch is present in the tracklet based on feature similarities of a plurality of re-identification feature pairs in the re-identification feature set comprises:
determining feature similarities of re-identification feature pairs of a plurality of adjoining image patch pairs in the image patch sequence and determining whether a candidate identification switch image patch is present in the tracklet according to whether a special feature similarity less than a predetermined similarity threshold is present in the feature similarities (Kang discloses determining feature similarities between the re-identification feature pairs, para. [0032], to determine whether a candidate identification switch patch is present and discloses that the determination of whether a candidate identification switch image patch is present can be according to whether the feature similarity is less than a threshold, para. [0030]).
Regarding claim 3, Kang discloses that in the case where it is determined that a special feature similarity less than the predetermined similarity threshold is present in the feature similarities, it is determined that a candidate identification switch image patch is present in the tracklet and an image patch associated with the special feature similarity is designated as the candidate identification switch image patch (in Kang, if a determination is made that the similarity between the re-identification features of the image patches is below the threshold and that the position feature does not indicate that the objects are the same, the image patch associated with the target object is designated as the candidate identification switch patch because the trajectory of the target object has to be updated to reflect the determinations, paras. [0030] and [0061]).
Regarding claim 4, Kang discloses that the determining of whether a candidate identification switch image patch is present in the tracklet based on feature similarities of a plurality of re-identification feature pairs in the re-identification feature set comprises generating a global similarity matrix representing similarities between respective image patches in the image patch sequence based on feature similarities of the plurality of re-identification feature pairs in the re-identification feature set and determining whether the candidate identification switch image patch is present in the tracklet based on the global similarity matrix (the BRI for “global similarity matrix” is a set of similarity values obtained by determining the similarities between a plurality of re-identification features of a plurality of image patches; Kang discloses obtaining a set of similarity values by comparing the similarities between a plurality of re-identification features of a plurality of image patches over a sequence of image frames, paras. [0104], which constitutes generating a set of global similarity values comprising a matrix; the global similarity values are used in combination with the position information encoded into the re-identification features to determine whether a candidate identification switch image patch is present in the tracklet, para. [0104]).
Regarding claim 11, to the extent that claim 11 recites the same limitations that are recited in claim 1, the rejection of claim 1 applies mutatis mutandis to claim 11. The only limitations that are recited in claim 11 that are not also recited in claim 1 are the memory having instructions stored thereon and at least one processor for executing the instructions to cause it to perform the operations recited in claims 1 and 11. Kang discloses memory having instructions stored thereon and at least one processor for executing the instructions to cause it to perform the operations recited in claims 1 and 11 (Fig. 3, computing unit 301 and memories 302 and 303, paras. [0109]-[0110]).
Regarding claims 12, 13 and 14, the rejections of claims 2, 3 and 4 apply mutatis mutandis to claims 12, 13 and 14, respectively.
Regarding claim 20, to the extent that claim 20 recites the same limitations that are recited in claim 1, the rejection of claim 1 applies mutatis mutandis to claim 20. The only limitations that are recited in claim 20 that are not also recited in claim 1 are the non-transitory computer-readable medium. Kang discloses a non-transitory computer-readable medium (Fig. 3, memories 302 and 303, paras. [0109]-[0110]).
Claims 10 and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Kang in view of Wang and further in view of U.S. Publ. Appl. No. 2019/0340431 A1 to Wang et al. (hereinafter referred to as “Wang 2”).
Regarding claim 10, the combined teachings of Kang and Wang do not explicitly teach that verifying whether it is credible that identification-switch has occurred at the candidate identification switch image patch is based at least on the first, second and third conditions recited in claim 10. The BRI for occlusion rate, based on para. [0053] of the present specification, is that it means the degree of overlap between the bounding boxes of the patches that are being processed to determine their similarity; the BRI for the largest occlusion rate is the occlusion rate corresponding to the largest overlap of the bounding boxes of the image patches.
Wang 2, in the same field of endeavor, discloses verifying whether a credible identification-switch has occurred based at least in part on the first condition, namely, the occlusion rate of a largest occlusion rate of a bounding box of the candidate identification switch image patch being greater than a predetermined occlusion rate threshold. Specifically, Wang 2 discloses using an overlap ratio of the overlap of the bounding box of the object region of the previous track and the bounding box of the object region of the subsequent track to determine motion similarity between the tracks, or trajectories, paras. [0087]-[0093]. Wang 2 teaches that these motion similarity determinations can be used to reduce incorrect ID switches, paras. [0121] and [0193]. Wang 2 discloses that if the overlap ratio, including the largest overlap ratio, is greater than a predetermined threshold, the trajectories are sufficiently similar that they are deemed to be a match, para. [0093].
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the present disclosure, to modify the multi-target tracking (MTT) system and method of Kang as modified by the teachings of Wang further based on the teachings of Wang 2 to use the overlap ratio technique of Wang 2 to verify whether it is credible that an identification switch has occurred by using the overlap ratio technique as an additional condition over and above the similarity determinations performed in Kang for verifying that an identification switch has occurred. One of ordinary skill in the art would have been motivated to make the modification to improve the accuracy of identification switch determinations by using the additional overlap ratio similarity determination to verify that an identification switch has occurred. The modification could have been made by one of ordinary skill in the art before the effective filing date of the present disclosure with a reasonable expectation of success because making the modification merely involves combining prior art elements according to known methods (i.e., further optimizing the Deep SORT model executed by the computing device 301 of Kang to also compute similarity based on overlap ratio as discussed in para. [0061] of Kang) to yield predictable results.
Regarding claim 19, the rejection of claim 10 applies mutatis mutandis to claim 19.
Allowable Subject Matter
Claims 5, 6, 8, 9, 15, 16 and 18 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims and to overcome the rejection under 35 U.S.C. 112(b).
Regarding claims 5 and 15, these claims recite that the determination of whether the candidate identification switch image patch is present in the tracklet based on the global similarity matrix comprises:
determining a Gaussian checkerboard kernel based on a common checkerboard kernel and a two-dimensional Gaussian function;
determining a checkerboard kernel transformation value of each image patch in the image patch sequence by summing the products of elements of a local similarity matrix of each image patch in the image patch sequence and corresponding elements in the Gaussian checkerboard kernel; and
in a case where a transformation value curve representing changes in the checkerboard kernel transformation value of each image patch has at least one peak, determining an image patch corresponding to a highest peak among the at least one peak as the candidate identification switch image patch.
None of the art of record teaches or suggests this combination of limitations.
Claims 6 and 16 recite allowable subject matter due their dependencies from claims 5 and 15, respectively.
Regarding claims 8 and 18, none of the prior art teaches or suggests, in combination with the other claimed limitations, that when the number of times of splitting exceeds a predetermined number threshold of times, making attempts to split the current tracklet is stopped, and that the number of times of splitting refers to the number of times of splitting an initial tracklet to obtain the current tracklet.
Regarding claim 9, none of the prior art teaches or suggests, in combination with the other claimed limitations, that when the number of times of splitting exceeds a predetermined number threshold of times equal to 4, making attempts to split the current tracklet is stopped, where the number of times of splitting refers to the number of times of splitting an initial tracklet to obtain the current tracklet.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to 2 whose telephone number is (571)272-2867. The examiner can normally be reached M-F 9-5.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Matt Bella can be reached at (571)272-7778. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/DANIEL J. SANTOS/Examiner, Art Unit 2667
/MATTHEW C BELLA/Supervisory Patent Examiner, Art Unit 2667