DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claim(s) 1-7, 10-22 is/are rejected under 35 U.S.C. 102(a)(2) as being anticipated by US 2019/0191098 by Ishii et al.
Regarding claim 1, an information processing apparatus comprising at least one processor configured to function as:
an acquisition unit configured to acquire images captured in time series and distance information in a depth direction on a plurality of areas in each of the images (fig. 8A, 18, paragraph 0114 teaches “In FIG. 8A, (a) illustrates a frame (image) 1102 at time point tin video captured by one camera. When a first head 701A and a second head 701C in the frame 1102 are objects being tracked, the object tracking apparatus 1 predicts the three-dimensional positions of the first head 701A and the second head 701C at time point t+Δt (step S4). Thereafter, the object tracking apparatus 1 performs the priority-camera information update process (step S5).”, paragraph 0203 teaches “In FIG. 18, the movement destination of the object 7A is denoted by a solid-line arrow. That is, the direction of movement of the object 7A is generally parallel to a depth direction in the frame captured by the second camera 2B(cam2) and a depth direction in the frame captured by the fourth camera 2D(cam4). In contrast, the direction of the movement of the object 7A is generally orthogonal to a depth direction in the frame captured by the first camera 2A(cam1) and a depth direction in the frame captured by the third camera 2C(cam3). Accordingly, the amounts of movement of the object 7A in the frame captured by the first camera 2A(cam1) and in the frame captured by the third camera 2C(cam3) become larger than the amounts of movement of the object 7A in the frame captured by the second camera 2B(cam2) and in the frame captured by the fourth camera 2D(cam4).”, paragraph 0191 teaches “The three-dimensional position of the movement destination of the object m, the movement destination being predicted by the movement-destination predicting unit 130, is a predicted value and thus may differ from the three-dimensional position of the object at the image-capture time point of a next frame. Also, there are tendencies that as the distance (the amount of movement of the object m) from the position of the object m detected in a frame to the predicted position of the object m increases, the amount of influence of an error in the predicted position increases, and a difference between the predicted position of the object m in a next frame and the actual position of the object m increases.”);
a detection unit configured to detect a candidate area of an object to be a tracking target from each of the images based on an image feature of each of the images (in addition to discussion above, paragraph 0116 teaches “The two-dimensional positions of the first head 701A and the second head 701C in a next frame in the video including the frame 1102 in (a) in FIG. 8A are assumed to be a first position PP1 and a second position PP2, respectively, denoted by dotted-line circles in the frame 1102. The first position PP1 in the frame 1102 includes a first portion 1203A and a second portion 1203B, which are included in an object (background) 1203 not to be tracked and are separated by a boundary line B3 extending in a horizontal direction. Thus, when a case in which the objects to be tracked are human heads, and only the movement destinations of the human heads are considered, backside information at the movement destination of the first head 701A is information including features of the surroundings of a region that overlaps the first position PP1 on the object 1203.”);
an estimation unit configured to estimate an occlusion state indicating whether the object to be the tracking target is occluded by another object different from the tracking target for the candidate area detected from each of the images based on time-series data of the distance information (in addition to discussion above, paragraph 0159 teaches “The priority-camera determination unit 140 in the object tracking apparatus 1 in the present embodiment updates the priority camera information 195, based on a prediction result obtained by the movement-destination predicting unit 130. The priority-camera determination unit 140 according to the present embodiment calculates the detection difficulty degrees D.sub.CP, based on backside influence degrees D.sub.BG, occlusion degrees D.sub.FG, and predicted influence degrees D.sub.MV. As described above in the first embodiment, each backside influence degree D.sub.BG is a numerical value indicating an influence that another object that overlaps an object to be tracked at the back side of the object to be tracked has on the detection accuracy of the object to be tracked. Each occlusion degree D.sub.FG is a numerical value indicating how much an object to be tracked is occluded by another object that overlaps the object to be tracked at the front side of the object to be tracked. Each predicted influence degree D.sub.MV is a numerical value indicating an influence that a prediction error in the two-dimensional position of the movement destination of an object to be tracked in a next frame, the movement destination being predicted by the movement-destination predicting unit 130, has on the detection accuracy of the object.”); and
a determination unit configured to determine the candidate area of the object to be the tracking target from the candidate area detected from each of the images based on an estimation result of the occlusion state (in addition to discussion above, paragraph 0117-0118 teaches “However, when the objects to be tracked are human heads, and the second head 701C is assumed to be at the second position PP2, an accompanying portion 702C that accompanies the second head 701C moves to below the second position PP2. Thus, the first position PP1 in the frame 1102 overlaps an accompanying portion 702C′ of the second head 701C. In this case, it is assumed that the prediction result of the three-dimensional positions of the first head 701A and the second head 701C indicates that the second head 701C is farther (more backward) than the first head 701A when viewed from a camera that captures the video including the frame 1102. In this case, in the next frame, the accompanying portion 702C of the second head 701C exists at the back side of the first head 701A and between the first head 701A and the object (background) 1203, as in a frame 1102 in (b) in FIG. 8A. Thus, the object tracking apparatus 1 estimates, for example, a feature of the first position PP1 in a state in which the second head 701C and the accompanying portion 702C are superimposed on the background, as in the frame 1102 in (c) in FIG. 8B. Based on the similarity between the feature of the first position PP1 and the feature of the first head 701A, the object tracking apparatus 1 calculates a backside influence degree D.sub.BG for detecting the first head 701A from the next frame.”).
Regarding claim 2, the information processing apparatus wherein the estimation unit estimates, based on the time-series data of the distance information for each of the candidate areas associated with a first object being the object to be the tracking target and for each of the candidate areas associated with a second object being different from the first object, a front-back relationship between the first object and the second object in the depth direction, and wherein the estimation unit determines the occlusion state of the first object based on an estimation result of the front-back relationship (in addition to discussion above, paragraph 0159, 0163-0170 teaches “Based on a prediction result of the movement destination of an object to be tracked, the superimposition-information obtaining unit 142d obtains superimposition information indicating a feature of a region that overlaps, in a next frame, the object to be tracked and surroundings of the region. The superimposition-information obtaining unit 142d obtains backside information at the movement destination of the object and obtains information indicating another object that overlaps that object at the front side of that object. Based on the similarity between a feature of the object to be tracked and the backside information obtained by the backside-information obtaining unit 142a, the backside-influence-degree calculating unit 142b calculates a backside influence degree D.sub.BG. For example, after calculating a degree of similarity between the feature of the object to be tracked and the background information, the backside-influence-degree calculating unit 142b converts the degree of similarity into a backside influence degree D.sub.BG, based on the correspondence relationship between the degree of similarity and the detection difficulty degree.”).
Regarding claim 3, the information processing apparatus wherein the acquisition unit sequentially acquires frames of a moving image and acquires the distance information for each of the frames, wherein the detection unit detects the candidate area for each of the frames, and wherein the estimation unit estimates the occlusion state for each of the frames (in addition to discussion above, paragraph 0198 teaches “In the frame captured by the first camera 2A(cam1), another object (including the objects 7B and 7C) that overlaps the object 7A at the back side of the object 7A and influences detection of the object 7A does not exist. Similarly, in the frame captured by the fourth camera 2D(cam4), another object (including the objects 7B and 7C) that overlaps the object 7A at the back side of the object 7A and that influences detection of the object 7A does not exist. In contrast, in the video (frame) captured by the second camera 2B(cam2), the object 7D that is located at the back side of the object 7A overlaps the object 7A. Thus, when the similarity between a feature of the object 7A in the video captured by the second camera 2B and a feature of the object 7D is high, the backside influence degree D.sub.BG(2) becomes larger than the backside influence degrees D.sub.BG(1) and D.sub.BG(4).”, paragraph 0191).
Regarding claim 4, the information processing wherein, in a case where the occlusion state in an immediately previous frame indicates that the object to be the tracking target is occluded, the estimation unit determines the occlusion state in a current frame based on the estimation result of the front-back relationship (in addition to discussion above, paragraph 0038 teaches “Based on a change in the three-dimensional position of the object with time, the three-dimensional position being calculated from the set of frames, the movement-destination predicting unit 130 predicts a movement destination of the object. Based on the object three-dimensional position stored in the object position information 194, the movement-destination predicting unit 130 calculates a three-dimensional position of the object at the image-capture time point of a set of next frames. For example, based on an object position calculated from a set of frames at time point t and an object position calculated from a set of frames (for example, a set of frames at time point t−Δt) prior to the set of frames at time point t, the movement-destination predicting unit 130 calculates a position of the object at the image-capture time point of a set of next frames.”, paragraph 0042-0043).
Regarding claim 5, the information processing apparatus wherein the estimation unit determines the occlusion state in a current frame based on the estimation result of the front-back relationship in a case where at least a portion of the candidate area in an immediately previous frame or a current frame overlaps another candidate area in the frame, and wherein, in a case where the candidate area in the immediately previous frame or the current frame does not overlap the other candidate area in the frame, the estimation unit determines the candidate area to be associated with the first object from the candidate area in the current frame by performing matching using the image features of the candidate areas respectively associated with the first object and the second object in the immediately previous frame (in addition to discussion above, paragraph 0069 teaches “The priority-camera determination process in step S5 that the object tracking apparatus 1 performs after the process in steps S4 is performed by the priority-camera determination unit 140 in the object tracking apparatus 1. Based on a prediction result of the three-dimensional position of the object in the next frames, the priority-camera determination unit 140 calculates a movement destination of the object in each piece of video (a frame) captured by each camera 2 and obtains backside information at the movement destination. The backside information is information indicating, at the predicted movement destination of the object in the corresponding next frame, a feature of a background and another object that overlap that object at the back side of that object. Thereafter, based on the similarity between the feature of the object being tracked and the obtained backside information, the priority-camera determination unit 140 calculates the detection difficulty degrees D.sub.CP of the object in each of the next frame and updates the detection difficulty degrees D.sub.CP in the priority camera information 195.”).
Regarding claim 6, the information processing apparatus wherein the estimation unit estimates the front-back relationship based on a difference between pieces of the distance information respectively corresponding to the candidate area associated with the first object and the candidate area associated with the second object in each of a plurality of frames including an immediately previous frame arranged in time series (in addition to discussion above, paragraph 0043 teaches “Based on a feature of one object to be tracked and a feature of the surroundings of the object which is extracted based on the obtained backside information, the backside-influence-degree calculating unit 142b calculates a backside influence degree D.sub.BG that the state of the back side of the object has on the detection accuracy for detecting the object from a next frame. The backside-influence-degree calculating unit 142b calculates the backside influence degree D.sub.BG, for example, by using a conversion equation with which the backside influence degree D.sub.BG increases as the degree of similarity between a feature of one object being tracked and a feature of the surroundings of the object increases.”, paragraph 0099, 0110-0111).
Regarding claim 7, the information processing apparatus wherein the estimation unit estimates the front-back relationship in a case where differences between the pieces of the distance information have a same sign consecutively in a predetermined number of frames (in addition to discussion above, paragraph 0034 teaches “The tracking unit 120 detects, in the pieces of video, an object to be tracked and tracks the object. The tracking unit 120 performs a process for detecting, for each set of frames captured at a same time point in the respective pieces of video, an object from the set of frames and tracking the object in a time series. By using priority camera information 195 stored in the storage unit 190, the tracking unit 120 in the object tracking apparatus 1 in the present embodiment detects an object to be tracked and tracks the object. The priority camera information 195 includes information indicating the camera 2 that captures a frame that is included in the frames included in the set of frames to be subjected to a tracking process and to which priority is to be given in a process in which the tracking unit 120 tracks an object being tracked. For example, the tracking unit 120 uses the priority camera information 195 to narrow down one set of frames to a frame from which an object being tracked is to be detected during detection and tracking of the object from the set of frames. For example, for detecting one object being tracked from a set of frames, the tracking unit 120 sets, as a frame from which the object is to be detected, only a frame that is included in the frames included in the set of frames and that was captured by the camera(s) 2 with which the detection difficulty degree(s) in the priority camera information 195 is smaller than or equal to a threshold.”).
Regarding claim 10, the information processing apparatus wherein the estimation unit calculates a moving average of the differences between the pieces of the distance information (in addition to discussion above, paragraph 0143-0145 teaches “AD.sub.CP in equation (1) is the average value of detection difficulty degrees of all cameras for an object to be detected. R0 in equation (1) is a reference diameter (a constant).”).
Regarding claim 11, the information processing apparatus wherein the estimation unit extracts a plurality of candidate areas overlapping the candidate area associated with the first object, and wherein the estimation unit determines the occlusion state of the first object by estimating the front-back relationship for each combination of the candidate area associated with the first object and the extracted plurality of candidate areas (in addition to discussion above, paragraph 0069 teaches “Based on a prediction result of the three-dimensional position of the object in the next frames, the priority-camera determination unit 140 calculates a movement destination of the object in each piece of video (a frame) captured by each camera 2 and obtains backside information at the movement destination. The backside information is information indicating, at the predicted movement destination of the object in the corresponding next frame, a feature of a background and another object that overlap that object at the back side of that object. Thereafter, based on the similarity between the feature of the object being tracked and the obtained backside information, the priority-camera determination unit 140 calculates the detection difficulty degrees D.sub.CP of the object in each of the next frame and updates the detection difficulty degrees D.sub.CP in the priority camera information 195.”, paragraph 0090 teaches “In the present embodiment, each detection difficulty degree D.sub.CP represents the degree of influence (a backside influence degree D.sub.BG) that a second object that overlaps, in a frame, a first object to be tracked at the back side of the first object has on the detection accuracy for detecting the first object. The detection difficulty degrees D.sub.CP are updated in the priority-camera information update process (step S5).”, paragraph 0116-0117).
Regarding claim 12, the information processing apparatus wherein the estimation unit determines the occlusion state of the first object by performing matching of the candidate area in a current frame using the image feature of the candidate area associated with the first object in the immediately previous frame (in addition to discussion above, paragraph 0159, 0163-0170 teaches “Based on a prediction result of the movement destination of an object to be tracked, the superimposition-information obtaining unit 142d obtains superimposition information indicating a feature of a region that overlaps, in a next frame, the object to be tracked and surroundings of the region. The superimposition-information obtaining unit 142d obtains backside information at the movement destination of the object and obtains information indicating another object that overlaps that object at the front side of that object. Based on the similarity between a feature of the object to be tracked and the backside information obtained by the backside-information obtaining unit 142a, the backside-influence-degree calculating unit 142b calculates a backside influence degree D.sub.BG. For example, after calculating a degree of similarity between the feature of the object to be tracked and the background information, the backside-influence-degree calculating unit 142b converts the degree of similarity into a backside influence degree D.sub.BG, based on the correspondence relationship between the degree of similarity and the detection difficulty degree.”).
Regarding claim 13, the information processing apparatus wherein the estimation unit determines that the first object is not occluded in a case where a matching cost obtained as a result of the matching is a threshold value or less (in addition to discussion above, paragraph 0117-0118 teaches “The backside influence degree D.sub.BG when an accompanying portion of another object overlaps at the back side of an object to be tracked in a next frame may increase owing to the overlapping of the accompanying portion or may decrease owing to the overlapping of the accompanying portion.”).
Regarding claim 14, the information processing apparatus wherein the estimation unit determines that the first object is occluded in a case where a matching cost obtained as a result of the matching is larger than a threshold value and another candidate area is present near the candidate area associated with the first object (in addition to discussion above, fig. 10, paragraph 0125-0127).
Regarding claim 15, the information processing apparatus wherein the estimation unit determines the occlusion state of the first object based on the estimation result of the front-back relationship in a case where the estimation unit has been able to estimate the front-back relationship based on the time-series data of the distance information, and wherein the estimation unit determines the occlusion state of the first object by performing matching of the candidate area in the current frame using the image feature of the candidate area associated with the first object in the immediately previous frame in a case where the estimation unit has not been able to estimate the front-back relationship based on the time-series data of the distance information (in addition to discussion above, paragraph 0126 teaches “In this case, when the position of the person 7C in the next frame is assumed to be at the back side of the position of the person 7A, the positional relationship between the two people 7C and 7A in the field of view 13 at the image-capture time point of the next frame is a relationship as illustrated in (b) in FIG. 10. Accordingly, when a detection difficulty degree D.sub.CP for detecting the first head 701A from the next frame is calculated, the feature of the accompanying portion 702C of the person 7C which exists at the back side of the first head 701A is considered. The person 7C in FIG. 10 wears a white shirt and trousers with high-brightness color other than white. That is, the feature of the accompanying portion 702C in the frame including the person 7C include information indicating that a portion above a boundary line B6 extending in the horizontal direction at a height position that is generally center in the up-and-down direction of the accompanying portion 702C is white, and a portion below the boundary line B6 has high-brightness color other than white. Thus, when the accompanying portion 702C of the person 7C overlaps at the back side of the first head 701A, as in (b) in FIG. 10, the degree of similarity between the feature of the first head 701A and the feature of the surroundings of the first head 701A decreases. Hence, in the example illustrated in FIG. 10, since the accompanying portion 702C overlaps at the back side of the first head 701A, the backside influence degree D.sub.BG decreases, compared with a case in which the accompanying portion 702C does not overlap.”).
Regarding claim 16, the information processing apparatus wherein the estimation unit corrects the time-series data of the distance information based on an operation state of an image capturing apparatus that has captured the images (in addition to discussion above, paragraph 0190-0191 teaches “After step S523, the detection-difficulty-degree estimating unit 142 calculates the amount of movement of the object m in the frame captured by the camera n (step S524). The process in steps S524 is performed by the amount-of-movement calculating unit 142f included in the difficulty-degree calculating unit 142c. The amount-of-movement calculating unit 142f calculates the amount of movement of the object m, based on the detection position of the object m in the frame captured by the camera n and the position of the movement destination of the object m. Next, the detection-difficulty-degree estimating unit 142 calculates a predicted influence degree D.sub.MV(n), based on the amount of movement of the object (step S525). The process in steps S525 is performed by the predicted-influence-degree calculating unit 142g included in the detection-difficulty-degree estimating unit 142. The three-dimensional position of the movement destination of the object m, the movement destination being predicted by the movement-destination predicting unit 130, is a predicted value and thus may differ from the three-dimensional position of the object at the image-capture time point of a next frame. Also, there are tendencies that as the distance (the amount of movement of the object m) from the position of the object m detected in a frame to the predicted position of the object m increases, the amount of influence of an error in the predicted position increases, and a difference between the predicted position of the object m in a next frame and the actual position of the object m increases.”).
Regarding claim 17, the information processing apparatus wherein the estimation unit acquires a lens driving amount from the image capturing apparatus and corrects the time-series data of the distance information based on the lens driving amount (in addition to discussion above, paragraph 0034-0040 teaches “Based on the prediction result of the movement destination of the object being tracked, the two-dimensional-position calculating unit 141 calculates (predicts), for each frame included in one set of frames, the two-dimensional position of the object in a frame plane at the image-capture time point of the next frame. The frame plane is a plane that is set in real space based on the field of view and the focal distance of each camera and that includes a projection plane (projection region) of the object in an image-capture range.”).
Regarding claim 18, the information processing apparatus further comprising a control unit configured to control lens driving to focus on the candidate area determined by the determination unit, wherein the control unit controls the lens driving not to focus on the candidate area in a case where the occlusion state indicates that the object to be the tracking target is occluded (in addition to discussion above, paragraph 0042 teaches “The backside-information obtaining unit 142a obtains, for each frame, a feature of another object that exists at the back side of an object being tracked, the back side being located at an in-frame movement destination of the object being tracked, and in the surroundings of the object being tracked. The backside-information obtaining unit 142a obtains a feature of the other object from the background information 192 and the object feature information 193. The background information 192 includes features in video captured in a state in which the object to be tracked does not exist in the field of view of each camera 2 (for example, features of a floor surface and a building). When the object to be tracked is a portion of a moving object, the backside-information obtaining unit 142a obtains, as a feature of another object, a feature of an accompanying portion that moves in conjunction with the object to be tracked.”).
Regarding claim 19, the information processing apparatus wherein the acquisition unit acquires a defocus amount detected from each focus detection area on an imaging plane as the distance information (in addition to discussion above, paragraph 0042-0043 teaches “Based on a feature of one object to be tracked and a feature of the surroundings of the object which is extracted based on the obtained backside information, the backside-influence-degree calculating unit 142b calculates a backside influence degree D.sub.BG that the state of the back side of the object has on the detection accuracy for detecting the object from a next frame. The backside-influence-degree calculating unit 142b calculates the backside influence degree D.sub.BG, for example, by using a conversion equation with which the backside influence degree D.sub.BG increases as the degree of similarity between a feature of one object being tracked and a feature of the surroundings of the object increases.”).
Claim 20 is rejected for the same reason as discussed in the corresponding claim 1 above.
Claim 21 is rejected for the same reason as discussed in the corresponding claim 1 above.
Claim 22 is rejected for the same reason as discussed in the corresponding claim 1 above.
Allowable Subject Matter
Claims 8-9 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to NIGAR CHOWDHURY whose telephone number is (571)272-8890. The examiner can normally be reached Monday-Friday 9AM-5PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Thai Tran can be reached at 571-272-7382. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/NIGAR CHOWDHURY/Primary Examiner, Art Unit 2484
Prosecution Insights