Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Status
Claims 1, 11, and 12 have been amended. Claim 2 has been canceled. Claims 1, and 2-12 are pending.
Response to Arguments
In the remarks filed 09/04/24, on page 7, applicant asserts:
“Applicant respectfully submits that Shibata, Bewley, Zhang and Lan, either individually or in combination, fail to disclose or suggest at least the aforementioned features recited in amended independent claims 1, 11 and 12, and fail to provide the technical advantage offered by the claimed subject matter.
Shibata, in paragraph [0046], discloses the following: "The kinetic model selection means 203 selects an appropriate kinetic model from a plurality of kinetic models stored in the kinetic model dictionary based on the (object) identification result obtained by the object identification means 202 and the tracking result obtained by the hypothesis selection means 2044." (Emphasis added.)
As taught above, Shibata selects an appropriate kinetic model based on the object identification result and the tracking result.
Shibata, however, does not select an appropriate kinetic model based on a location where the object is present. The "object identification result" refers to the type and attribute information of the object (see Shibata, [0038]-[0040]), and the "tracking result" refers to the trajectory of the object (see Shibata, [0073]). Therefore, the "object identification result" and the "tracking result" do not include the "location where the object is present." Accordingly, Applicant respectfully submits that Shibata fails to disclose or suggest at least the aforementioned features recited in amended independent claims 1, 11 and 12.
The examiner respectfully disagrees, Shibata does select an appropriate kinetic model based on the objects location. Shibata selects an appropriate kinetic model based on the object identification result and the tracking result. The tracking result is further described in Fig. 6 and paragraphs [0072-0073]:
[0072: "The figure on the left side of FIG. 6 shows how the hypothesis generation means 2042 selects detection targets in the predicted region and generates a hypothesis (that is, a trajectory candidate) from these detection targets
[0073]: "As shown in the figure on the right side of FIG. 6, the hypothesis selection means 2044 selects the hypothesis having the highest reliability among the reliabilities of the hypotheses (trajectory candidates) calculated by the reliability calculation means 2043 as the tracking target trajectory. As a result, the trajectory is established even in the t-th frame. This established trajectory is used as the tracking result”.
As seen in the right figure of Fig. 6, the trajectory (shown with the arrows) is going from the dots (black and white) in each frame to the next, those are the detected object positions ([0053]: “The figure on the left side of FIG. 3 shows a state in which the position of the object candidate in the t-th frame is detected by the image feature calculation means 201, and the figure on the right side of FIG. 3 shows a state in which the movement of the t-th frame is predicted based on the detection position of the (t-1)-th frame and the kinetic model by the object position prediction means 2041”). These object positions are derived from image features and have coordinates ([0036]: “The image feature calculation means 201 detects the positions where a tracking target object is likely to currently exist from these image features, and outputs the position coordinates thereof”). Due to these objects having detected positions, and their positions having coordinates, their location is known. This location (detected position) is part of the trajectory (the trajectory would have to be from one location to another as shown in the figure). Since the model is selected based off the trajectory, and the trajectory includes location information of the object, then the model is selected based of the location information of the object. Therefore Shibata teaches the claimed limitation.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claims 1 and 11-12 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by SHIBATA et al. (US 20230107372 A1 Hereinafter “SHIBATA”).
Regarding claim 1, SHIBATA teaches a non-transitory computer-readable storage medium storing a program for causing a computer ([0089]: “Due to the above-mentioned configuration, an image processing apparatus, an image processing system, an image processing method, and a non-transitory computer-readable medium storing an image processing program therein, capable of tracking an object in a video with high accuracy can be provided”) to execute processing comprising:
acquiring image data at a first time (Fig. 1, [0030]: “The time-series image input/output apparatus 10 of the image processing system 10S acquires a time-series image (moving image) and outputs the acquired time-series image to the image processing apparatus 20”);
detecting a position and a kind of an object from the image data at the first time, thereby generating an object detection result at the first time in which the detected position and the detected kind of the object are correlated ([0044]: “The object identification means 202 integrates the object information obtained from these image features and the object information obtained from the tracking result, and acquires object information”. The image features include positions of the object “The image feature calculation means 201 detects the positions where a tracking target object is likely to currently exist from these image features, and outputs the position coordinates thereof” [0036], and the object identification means determines the type of object “The identification of an object as used herein may refer to identifying a type of object such as a person, a car, a dog, or a bicycle”[0038]. These are correlated together by the object identification means, which groups the types an attribute information from the features as object information “Hereinafter, the types and attribute information of these objects will be referred to as object information” [0040]. This is done for every time frame but for mapping purposes frame t in the figures will be this first time representing the current time);
selecting a prediction model for predicting a position of the object at the first time, based on an object tracking result at a second time that precedes the first time and a model database, a position and a kind of the object determined at the second time being correlated in the object tracking result at the second time, the kind of the object, a location where the object is present, and the prediction model are correlated in the database ([0046]: “The kinetic model selection means 203 selects an appropriate kinetic model from a plurality of kinetic models stored in the kinetic model dictionary based on the (object) identification result obtained by the object identification means 202 and the tracking result obtained by the hypothesis selection means 2044”. This model is selected in the previous timeframe (t-1) relative to the current timeframe (t), where the previous timeframe acts as a second time “As shown in the figure on the right side of FIG. 3, the object position prediction means 2041 predicts the position of the t-th object using the kinetic model selected in the (t-1)-th frame” (Fig. 3, [0056]). The model is selected based off the correlated information for the second time since it is selected based off the result of the object identification result obtained by the object identification means, due to the result of the object identification result obtained by the object identification means being correlated information about the object to its relevant timeframe as seen in the mapping for the first timeframe. The dictionary acts as the database and the model, location, and type of object are correlated since the model is chosen based off the location and object type);
predicting a position of the object at the first time, based on the selected prediction model and a time-series continuous tracking result including the object tracking result at the second time, thereby generating a position prediction result at the first time in which the predicted position and the kind of the object are correlated (Fig. 3, [0056]: “As shown in the figure on the right side of FIG. 3, the object position prediction means 2041 predicts the position of the t-th object using the kinetic model selected in the (t-1)-th frame”. Both the kinetic model and the time-series continuous tracking result are used for determining the predicted point);
generating a correlation result by executing correlation between the object detection result at the first time and the position prediction result at the first time ([0066]: “The reliability calculation means 2043 calculates the reliability of the object identification result as the object reliability, and calculates the reliability corresponding to the distance between the position of the object predicted by the object position prediction means 2041 and the detection position of the object in the target frame as the movement reliability. The reliability calculation means 2043 calculates the reliability of each hypothesis by integrating the object reliability and the movement reliability”); and
generating an object tracking result at the first time, based on the correlation result (Fig. 6, [0073]: “As shown in the figure on the right side of FIG. 6, the hypothesis selection means 2044 selects the hypothesis having the highest reliability among the reliabilities of the hypotheses (trajectory candidates) calculated by the reliability calculation means 2043 as the tracking target trajectory. As a result, the trajectory is established even in the t-th frame. This established trajectory is used as the tracking result”).
Regarding claim 11, the content of claim 11 is similar to the content of claim 1, with the additional teachings of processing circuitry. SHIBATA also discloses this information ([0089]: “Due to the above-mentioned configuration, an image processing apparatus, an image processing system, an image processing method, and a non-transitory computer-readable medium storing an image processing program therein, capable of tracking an object in a video with high accuracy can be provided”). Therefore, claim 11 is rejected for the same reasons of anticipation as claim 1, along with the additional teachings above.
Regarding claim 12, the content of claim 12 is similar to the content of claim 1, therefore it is rejected for the same reasons of anticipation as claim 1.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 3 and 5-8 are rejected under 35 U.S.C. 103 as being unpatentable over SHIBATA et al. (US 20230107372 A1 Hereinafter “SHIBATA”) in view of Bewley et al. (“SIMPLE ONLINE AND REALTIME TRACKING” Hereinafter “Bewley”).
Regarding claim 3, SHIBATA teaches the non-transitory computer-readable storage medium according to claim 1, wherein the selected prediction model is a prediction ([0046]: “The kinetic model selection means 203 selects an appropriate kinetic model from a plurality of kinetic models stored in the kinetic model dictionary based on the (object) identification result obtained by the object identification means 202 and the tracking result obtained by the hypothesis selection means 2044”).
SHIBATA does not expressly disclose the selected prediction model being a Kalman filter.
However, Bewley teaches using a Kalman filter as a predictive model (Page 3466, section 3.2: “When a detection is associated to a target, the detected bounding box is used to update the target state where the velocity components are solved optimally via a Kalman filter framework [14]”).
At the time the invention was made, it would have been obvious to one of ordinary skill in the art to modify SHIBATA’s predictive model to include Bewley’s Kalman filter as a predictive model because such a modification is taught, suggested, or motivated by the art. More specifically, the motivation to modify SHIBATA to include Bewley is expressly provided by Bewley, stating that using a Kalman filter for predictive model allows for competitive accuracy alongside faster processing time (Abstract: “Despite only using a rudimentary combination of familiar techniques such as the Kalman Filter and Hungarian algorithm for the tracking components, this approach achieves an accuracy comparable to state-of-the-art online trackers. Furthermore, due to the simplicity of our tracking method, the tracker updates at a rate of 260Hz which is over 20x faster than other state-of-the-art trackers”). Therefore, it would have been obvious to one of ordinary skill in the art at the time of the invention to modify SHIBATA’s predictive model to include Bewley’s Kalman filter as a predictive model with the motivation of improving prediction efficiency. The person of ordinary skill in the art would have recognized the benefit of improved prediction efficiency.
Regarding claim 5, SHIBATA teaches the non-transitory computer-readable storage medium according to claim 1, ([0066]: “The reliability calculation means 2043 calculates the reliability of the object identification result as the object reliability, and calculates the reliability corresponding to the distance between the position of the object predicted by the object position prediction means 2041 and the detection position of the object in the target frame as the movement reliability. The reliability calculation means 2043 calculates the reliability of each hypothesis by integrating the object reliability and the movement reliability”).
SHIBATA does not expressly disclose calculating an evaluation value by evaluating an overlap in all combinations between the object detection results at the first time and the position prediction results at the first time, and generating the correlation result by executing allocation between the object detection results and the position prediction results by using the evaluation value.
However, Bewley teaches calculating an evaluation value by evaluating an overlap in all combinations between the object detection results at the first time and the position prediction results at the first time (Page 3466, section 3.3: “In assigning detections to existing targets, each target’s bounding box geometry is estimated by predicting its new location in the current frame. The assignment cost matrix is then computed as the intersection-over-union (IOU) distance between each detection and all predicted bounding boxes from the existing targets. The assignment is solved optimally using the Hungarian algorithm”), and generating the correlation result by executing allocation between the object detection results and the position prediction results by using the evaluation value (Page 3466, section 3.3: “Additionally, a minimum IOU is imposed to reject assignments where the detection to target overlap is less than IOUmin”. The assignment act as the correlation result, which are determined by executing allocation between the object detection results and position predictions using the evaluation value (cost) of the intersection over union).
At the time the invention was made, it would have been obvious to one of ordinary skill in the art to modify SHIBATA’s correlative process to include Bewley’s use of bounding boxes for points and Intersection over union as an evaluation value to determine the correlation result because such a modification is taught, suggested, or motivated by the art. More specifically, the motivation to modify SHIBATA to include Bewley is expressly provided by Bewley, stating that their method allows for competitive accuracy alongside faster processing time (Abstract: “Despite only using a rudimentary combination of familiar techniques such as the Kalman Filter and Hungarian algorithm for the tracking components, this approach achieves an accuracy comparable to state-of-the-art online trackers. Furthermore, due to the simplicity of our tracking method, the tracker updates at a rate of 260Hz which is over 20x faster than other state-of-the-art trackers”. The Hungarian algorithm is used to optimize the intersection over union so if the Hungarian algorithm is mentioned then it presumes the teaching of the intersection over union being included in their improved method). Therefore, it would have been obvious to one of ordinary skill in the art at the time of the invention to modify SHIBATA’s predictive model to include Bewley’s use of bounding boxes for points and Intersection over union as an evaluation value to determine the correlation result with the motivation of improving prediction efficiency. The person of ordinary skill in the art would have recognized the benefit of improved prediction efficiency.
Regarding claim 6, the combination of SHIBATA and Bewley teaches the non-transitory computer-readable storage medium according to claim 5, in addition, Bewley further teaches further comprising calculating the evaluation value by using at least one of an IoU (Intersection over Union) and a similarity of image features (Page 3466, section 3.3: “In assigning detections to existing targets, each target’s bounding box geometry is estimated by predicting its new location in the current frame. The assignment cost matrix is then computed as the intersection-over-union (IOU) distance between each detection and all predicted bounding boxes from the existing targets. The assignment is solved optimally using the Hungarian algorithm”. By using the intersection over union, the evaluation value (cost) is calculated by using at least one of an Intersection over union, and a similarity of image features (by analyzing how much the bounding boxes overlap for detected object, the similarity of their position and shape is used which are both features of the object in the image).
The rationale for this combination is similar to the rationale mentioned in the claim 5 combination due to similar method of combination (using intersection over union for evaluation calculation) and benefits (improved prediction efficiency).
Regarding claim 7, the combination of SHIBATA and Bewley teaches the non-transitory computer-readable storage medium according to claim 5, in addition, Bewley further teaches further comprising generating the correlation result by executing the allocation by using a Hungarian algorithm (Page 3466, section 3.3: “The assignment cost matrix is then computed as the intersection-over-union (IOU) distance between each detection and all predicted bounding boxes from the existing targets. The assignment is solved optimally using the Hungarian algorithm”).
The rationale for this combination is similar to the rationale mentioned in the claim 5 combination due to similar method of combination (using intersection over union for evaluation calculation presumes optimization using the Hungarian algorithm for it to work as intended in Bewley) and benefits (improved prediction efficiency).
Regarding claim 8, SHIBATA teaches the non-transitory computer-readable storage medium according to claim 1, wherein the correlation result includes information of a combination of correlated positions([0066]: “The reliability calculation means 2043 calculates the reliability of the object identification result as the object reliability, and calculates the reliability corresponding to the distance between the position of the object predicted by the object position prediction means 2041 and the detection position of the object in the target frame as the movement reliability. The reliability calculation means 2043 calculates the reliability of each hypothesis by integrating the object reliability and the movement reliability”).
SHIBATA does not expressly disclose the correlation result including information about uncorrelated positions.
However, Bewley teaches the correlation result including information of uncorrelated positions (Page 3466, section 3.3 : “Additionally, a minimum IOU is imposed to reject assignments where the detection to target overlap is less than IOUmin. We found that the IOU distance of the bounding boxes implicitly handles short term occlusion caused by passing targets. Specifically, when a target is covered by an occluding object, only the occluder is detected, since the IOU distance appropriately favours detections with similar scale. This allows both the occluder target to be corrected with the detection while the covered target is unaffected as no assignment is made”. This system allows for uncorrelated objects to be identified (objects with no assignment) in order to handle occlusion).
At the time the invention was made, it would have been obvious to one of ordinary skill in the art to modify SHIBATA’s correlative process to include Bewley’s identification of uncorrelated objects in the correlation results because such a modification is taught, suggested, or motivated by the art. More specifically, the motivation to modify SHIBATA to include Bewley is expressly provided by Bewley, stating that their method of identifying uncorrelated objects (objects with no assignment) help handle occlusions (Page 3466, section 3.3 : “Additionally, a minimum IOU is imposed to reject assignments where the detection to target overlap is less than IOUmin. We found that the IOU distance of the bounding boxes implicitly handles short term occlusion caused by passing targets. Specifically, when a target is covered by an occluding object, only the occluder is detected, since the IOU distance appropriately favours detections with similar scale. This allows both the occluder target to be corrected with the detection while the covered target is unaffected as no assignment is made”). Therefore, it would have been obvious to one of ordinary skill in the art at the time of the invention to modify SHIBATA’s predictive model to include Bewley’s identification of uncorrelated objects in the correlation results with the motivation of improving occlusion handling. The person of ordinary skill in the art would have recognized the benefit of improved occlusion handling.
Claim 4 is rejected under 35 U.S.C. 103 as being unpatentable over SHIBATA et al. (US 20230107372 A1 Hereinafter “SHIBATA”) in view of Zhang et al. (“ByteTrack: Multi-Object Tracking by Associating Every Detection Box” Hereinafter “Zhang”).
Regarding claim 4, SHIBATA teaches the non-transitory computer-readable storage medium according to claim 1, further comprising
SHIBATA does not expressly disclose determining the object tracking result at the second time as the position prediction result at the first time, in a case where the selected prediction model indicates no prediction.
However, Zhang teaches determining the object tracking result at the second time as the position prediction result at the first time, in a case where the selected prediction model indicates no prediction (Page 5, implementation details: “For the lost tracklets, we keep it for 30 frames in case it appears again”. When a tracklet is lost, they keep it for 30 frames, so the location of the object for the 30 frames is the location that was last detected. So the object tracking result at the second time (prior to current frame that has no prediction) is determined as the position prediction in the first frame (current frame), since the model has no prediction for the current frame).
At the time the invention was made, it would have been obvious to one of ordinary skill in the art to modify SHIBATA’s object tracking to include Zhang’s method for handling no predictions because such a modification is the result of applying a known technique to a known device ready for improvement to yield predictable results. More specifically, Zhang’s method for handling no predictions permits method for handing lost tracks in tracking without having to terminate them immediately and losing data. This known benefit in Zhang is applicable to SHIBATA as they both share characteristics and capabilities, namely, they are directed to multiple object tracking. Therefore, it would have been recognized that modifying SHIBATA’s object tracking to include Zhang’s method for handling no predictions would have yielded predictable results because (i) the level of ordinary skill in the art demonstrated by the references applied shows the ability to incorporate Zhang’s method for handling no predictions in multiple object tracking and (ii) the benefits of such a combination would have been recognized by those of ordinary skill in the art.
Claim 9 is rejected under 35 U.S.C. 103 as being unpatentable over SHIBATA et al. (US 20230107372 A1 Hereinafter “SHIBATA”) in view of Lan et al. (US 10882535 B2 Hereinafter “Lan”).
Regarding claim 9, SHIBATA teaches the non-transitory computer-readable storage medium according to claim 1, further comprising selecting the prediction model, based on the object tracking result at the second time and a model database in which the kind of the object, a location where the object is present, ([0046]: “The kinetic model selection means 203 selects an appropriate kinetic model from a plurality of kinetic models stored in the kinetic model dictionary based on the (object) identification result obtained by the object identification means 202 and the tracking result obtained by the hypothesis selection means 2044”. The dictionary acts as the database and the model, location, and type of object are correlated since the model is chosen based off the location and object type (see claim 1 citation regarding selection of predictive model)).
SHIBATA does not expressly disclose selecting a prediction model based off a nearby object, with the nearby object being correlated with the prediction model.
However, Lan teaches selecting a prediction model based off a nearby object, with the nearby object being correlated with the prediction model (Col. 6, lines 55-65: “For example, the vehicle computing system 102 can select the trajectories for a first object 202 that are not in conflict with the predicted interaction trajectories of the one or more second objects (e.g., would not cause the first object 202 to collide with the second object(s)) as the predicted interaction trajectories 220A-B of the first object 202 that may occur as a result of the interaction”. The trajectory acts as the prediction model, and it is selected based on a nearby object (the trajectory of a nearby object). The new trajectory is correlated with the nearby object in this way because it was selected based on the nearby object’s trajectory. If combined with SHIBATA’s system the correlated object would be present in the model database of SHIBATA, hence the combination of SHIBATA and Lin teaches the claimed limitation).
At the time the invention was made, it would have been obvious to one of ordinary skill in the art to modify SHIBATA’s object tracking to include Lan’s method for prediction model selection based on objects around the tracked object because such a modification is taught, suggested, or motivated by the art. More specifically, the motivation to modify SHIBATA to include Lan is implicitly provided by Lan, stating that performing selection of prediction models in their way improves prediction of object trajectories (Col. 9, lines 30-35: “For instance, the present disclosure provides systems and methods for improved predictions of object trajectories”). Therefore, it would have been obvious to one of ordinary skill in the art at the time of the invention to modify SHIBATA’s object tracking to include Lan’s method for prediction model selection based on objects around the tracked object with the motivation of improving object location prediction. The person of ordinary skill in the art would have recognized the benefit of improved object location prediction.
Allowable Subject Matter
Claim 10 is objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure:
Du et al. (US 20210276572 A1) teaches selection of motion models based on object prediction status
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to STEFANO A DARDANO whose telephone number is (703)756-4543. The examiner can normally be reached Monday - Friday 11:00 - 7:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Greg Morse can be reached at (571) 272-3838. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/STEFANO ANTHONY DARDANO/ Examiner, Art Unit 2663
/GREGORY A MORSE/Supervisory Patent Examiner, Art Unit 2698
