DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Status
Claims 1-7 were pending for examination in the Application No. 18/101,593 filed January 26th, 2023. In the remarks and amendments received on June 23rd, 2025, claims 1-3 and 5-7 are amended. Accordingly, claims 1-7 are currently pending for examination in the application.
Response to Amendment
Applicant’s amendments filed June 23rd, 2025, to the Claims have overcome each and every objection and 35 U.S.C. 112 (b) rejections previously set forth in the Non-Final Office Action mailed April 4th, 2025. Accordingly, the objections and 35 U.S.C. 112 (b) rejections are withdrawn in response to the remarks and amendments filed. Additionally, the 35 U.S.C. 112(f) interpretations previously set forth in the Non-Final Office Action are withdrawn in response to the remarks and amendments received from Applicant. Examiner warmly thanks Applicant for considering the objections and the suggested amendments to be made to the disclosure.
Response to Arguments
Applicant’s arguments filed June 23rd, 2025, regarding the rejection(s) of claim(s) 1-7 have been fully considered but are not persuasive.
The examiner respectfully disagrees with Applicant’s assertion that Hasejima does not teach or suggest the features of claim 1 related to “the object tracker tracks the object in a future image frame”. As detailed in the current rejection below, the object tracker of Hasejima (e.g., “tracking unit 55b”) discloses tracking the object (e.g., a “moving body”) in a future image frame as predicting the “position corresponding to coordinates of the [tracked] moving body in the front image in a subsequent frame”, where a “subsequent frame” is a future image frame (see the rejection of claim limitation “…and the object tracker tracks…” in claim 1 below).
Therefore, independent claim 1 (and similar independent claims 6 and 7) and dependent claim 2 remain anticipated by Hasejima, for similarly recited features therein, and dependent claims 3-5, not being rejected by the same combination, are rendered obvious by Hashimoto, Zhang, and Xu in the combination(s) set forth in the current rejection below (see analysis in “Claim Rejections - 35 USC § 103” below).
Claim Objections
Claim 5 is objected to because of the following informalities fail to comply with 37 CFR 1.71(a) for "full, clear, concise, and exact terms" (see MPEP § 608.01(m)). The examiner respectfully suggests the following claim amendments to comply with 37 CFR 1.71(a) by maintaining consistency in claim language and improving the clarity of claim language:
In line 3 of claim 5, “compared to the size of the image when” should be “compared to the size of the image area when”.
Appropriate correction is required.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1, 2, 6, and 7 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Hasejima et al. (Hasejima; US 2018/0342068 A1).
Regarding claim 1, Hasejima discloses an object tracking device comprising:
a memory that stores a program (para. [0036], recite(s)
[0036] “The image processing ECU 18 includes a communication unit 18 a, a CPU, a ROM, and a RAM and loads a program saved in the ROM in the RAM to execute the program…”
, where “ROM” or “RAM” is memory); and
a processor that, upon execution of the program (para. [0036]—see preceding citation above—, where a “CPU” is a processor), is configured to operate as:
an image acquirer configured to acquire image data including a plurality of image frames captured in time series by an imager mounted on a mobile object (paras. [0033] and [0188], recite(s)
[0033] “FIG. 1 is a diagram illustrating a configuration of a vehicle 10 mounted with an image processing ECU 18. The vehicle 10 includes… a camera 12 that photographs the surroundings of the vehicle 10…”
[0188] “(1) The tracking device, that is, the image processing ECU 18 includes an imaging unit, that is, an image input unit to which an image photographed by the camera 12 is input, that is, the communication unit 18 a, a first moving body detection unit, that is, the movement vector calculation unit 51 that calculates the optical flow using the plurality of images input to the image input unit…”
, where the “imaging unit” or “camera 12” is an image acquirer, the “plurality of images” used for optical flow are a plurality of image frames acquired in time series, and the “vehicle 10” is a mobile object);
a recognizer configured to recognize an object from the image data acquired by the image acquirer (paras. [0041-0042], recite(s)
[0041] “The movement vector calculation unit 51 receives an input of a photographed image from the camera 12, calculates an optical flow using a latest photographed image and an immediately preceding photographed image, and detects a region (hereinafter a first candidate region) which is likely to correspond to a moving body. Then, coordinates and a direction of movement of the detected first candidate region are output to the first detection result RES1. The first detection result RES1 has the same number of records as that of the first candidate region detected by the movement vector calculation unit 51. For example, the first detection result RES1 is as follows.”
[0042] “FIG. 3 is a diagram illustrating an example of the first detection result RES1. The first detection result RES1 includes one or more records, and coordinates of the first candidate region and a direction of movement of the first candidate region are recorded in each record. The first candidate region is defined as a rectangle, and coordinates thereof are identified by top-left coordinates and bottom-right coordinates. However, the first candidate region is not limited to the rectangle, and it is possible to adopt a region having an elliptical or arbitrary shape. In addition, the direction of movement of the first candidate region may include not only a component of a left-right direction but also a component of a depth direction, that is, a direction approaching/away from the camera 12, and the direction of movement may be expressed by an angle of 0 to 360 degrees. The description will be continued by returning to FIG. 2.”
, where the “movement vector calculation unit 51” is at least a recognizer configured to recognize an object as a position of the object (i.e., the “coordinates” of the “moving body”));
an area setter configured to set an image area including the object recognized by the recognizer (paras. [0041-0042]—see citation above—, where the “movement vector calculation unit 51” is at least an area setter configured to set an image area (i.e., the “first candidate region”) including the object (i.e., the “moving body”)); and
an object tracker configured to track the object on a basis of an amount of time-series change in the image area set by the area setter (paras. [0047-0048], recite(s)
[0047] “The tracking processing unit 55 includes an estimation unit 55a and a tracking unit 55b. The estimation unit 55a predicts a position of a moving body in a process described below, and the tracking unit 55b performs another process. The tracking processing unit 55 reads the integrated object MOF, the first detection result RES1, and the first detection result RES1 output by the association unit 53 and reads the tracked moving body DMO from the memory 26. Information about a moving body which is being tracked is stored as one record for each moving body in the tracked moving body DMO…”
[0048] “FIG. 6 is a diagram illustrating an example of the tracked moving body DMO. The tracked moving body DMO includes one or more records, and a predicted position corresponding to coordinates of the moving body in the front image in a subsequent frame, a speed of the moving body, a direction of movement of the moving body, a tracking certainty factor, and an updated position corresponding to coordinates of the moving body in the front image in a current frame, that is, an identified position of the moving body are recorded in each record. The integrated candidate region is defined as a rectangle, and coordinates thereof are identified by top-left coordinates and bottom-right coordinates. However, the integrated candidate region is not limited to the rectangle, and it is possible to adopt a region having an elliptical or arbitrary shape. Here, the speed of the moving body is recorded on the assumption that the moving body moves in uniform motion. In addition, the direction of movement of the moving body may include not only a component of a left-right direction but also a component of a depth direction, that is, a direction approaching/away from the camera 12. The description will be continued by returning to FIG. 2.”
, where the “tracking unit 55b” is an object tracker configured to track an object on a basis of an amount of time-series change (e.g., “position” between frames such as between a “current frame” and a “subsequent frame”)), wherein
the area setter sets a position and a size of the image area (paras. [0041-0042]—see citation in limitation “a recognizer…” above—, where setting the image area (i.e., the “first candidate region”) includes setting a position and a size of the image area (e.g., “coordinates thereof… identified by top-left coordinates and bottom-right coordinates”) on the basis of the amount of time-series change in the image area including the object in a past image frame (i.e., an “immediately preceding photographed image”); where para. [0056] further recites including behavior information of the mobile object:
[0056] “In step S302, an optical flow expected to be generated in response to movement of the vehicle 10 during a time Δt from photographing of an immediately preceding image to the present is calculated using outputs of the wheel speed sensor 20 and the steering angle sensor 22 output from the sensor interface 24. Subsequently, the operation proceeds to step S303.”
, where the “movement of the vehicle 10” including “wheel speed” and “steering angle” outputs are behavior information of the mobile object (i.e., “vehicle 10”)) and the object tracker tracks the object in a future image frame on the basis of the amount of time-series change in the image area including the object in a past image frame and behavior information of the mobile object (para. [0048]—see citation in claim limitation “an object tracker…” above—, where predicting the “position corresponding to coordinates of the [tracked] moving body in the front image in a subsequent frame” is tracking the object in a future image frame (e.g., “subsequent frame”) based on the amount of time series change (e.g., “movement of the moving body”) in the image area (e.g., “candidate region”); wherein para. [0056]—see citation in claim limitation “the area setter…” above— further recite(s) the amount of time series change (e.g., “movement”) includes the object (e.g., “vehicle”) in a past image frame (e.g., a “preceding image”) and behavior information of the mobile object (e.g., vehicle “wheel speed” or “steering angle”)).
Regarding claim 2, Hasejima discloses the object tracking device according to claim 1,
wherein the area setter estimates a position and a speed of the object after a time point of recognition on a basis of an amount of change in a position of the object in the past image frame captured at a time point prior to the time point of recognition of the object by the recognizer (para. [0048]—see citation in claim 1 limitation “an object tracker…” above---, where the “predicted position” and “speed of the moving object” are an estimated position and speed of the object after a time point of recognition on a basis of an amount of change in a position of the object in the past image frame (e.g., the “preceding photographed image”) as recited in para. [0041]—see citation in claim 1 limitation “a recognizer…” above), and sets the position and the size of the image area and the object tracker tracks the object in the future image frame on a basis of the estimated position and speed, and the behavior information of the mobile object in the past image frame captured at a time point prior to the time point of recognition (para. [0048]—see citation in claim 1 limitation “an object tracker…” above---, where the “predicted position” is the set position and size of the image area on a basis of at least the estimated position and speed (e.g., the “predicted position” and “speed of the moving object”); where para. [0056] further recites on the basis of the behavior information of the mobile object (e.g., “movement of the vehicle 10”)—see citation in claim 1 limitation “wherein the area setter sets…” and the rejection of similar claim limitation “…and the object tracker tracks” in claim 1 above).
Regarding claim 6, the claim is the method performed by the device of claim 1. Therefore, claim 6 recites similar limitations to claim 1 and is rejected for similar rationale and reasoning (see the analysis for claim 1 above).
Regarding claim 7, the claim differs from claim 1 in that the claim is in the form of a computer-readable non-transitory storage medium that has stored therein a program causing a computer to execute the method performed by the device of claim 1. Therefore, claim 7 recites similar limitations to claim 1 and is rejected for similar rationale and reasoning (see the analysis for claim 1 above).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim 3 is rejected under 35 U.S.C. 103 as being unpatentable over Hasejima as applied to claim 1 above, and further in view of Hashimoto et al. (Hashimoto; US 2021/0312199 A1).
Regarding claim 3, Hasejima discloses the object tracking device according to claim 1,
wherein, when the object is recognized by the recognizer, the area setter projects and converts the image captured by the imager into a bird's-eye view image (para. [0043], recite(s)
[0043] “The foot position calculation unit 52 creates overhead images from the latest photographed image and the immediately preceding photographed image, respectively, and detects a region (hereinafter a second candidate region) which is likely to correspond to the moving body from a difference thereof. Then, coordinates of a center of gravity G of the detected second candidate region on the overhead image and coordinates of a closest point of approach Ha with respect to the vehicle 10 on the overhead image are output to the second detection result RES2. The second detection result RES2 has the same number of records as that of the second candidate region detected by the foot position calculation unit 52. Further, the foot position calculation unit 52 calculates a closest point of approach Hb corresponding to the closest point of approach Ha on a plane conversion image in a reverse procedure to that of creation of the overhead images, and similarly outputs the calculated closest point of approach Hb to the second detection result RES2. For example, the second detection result RES2 has the following configuration.”
, where the “overhead image” is a bird’s-eye view image), acquires a position and a size of the object in the bird's-eye view image (para. [0044], recite(s)
[0044] “FIG. 4 is a diagram illustrating an example of the second detection result RES2. The second detection result RES2 includes one or more records, and information about the second candidate region in an overhead image and information about the second candidate region in a front image are recorded in each record. The information about the second candidate region in the overhead image refers to coordinates of a center of gravity G of the second candidate region and coordinates of a point closet to the vehicle 10 in the second candidate region, that is, the closest point of approach Ha. The information about the second candidate region in the front image refers to coordinates corresponding to the closest point of approach Ha in the overhead image and is obtained by coordinate-transforming the coordinates of the closest point of approach Ha in the overhead image. The description will be continued by returning to FIG. 2.”
, where the “second candidate region” is an acquired position (e.g., “coordinates of a center of gravity G”) and size of the object (e.g., coordinates designating the area of the region similar to the “first candidate region” such as recited in para. [0042]—see citation in claim 1 limitation “a recognizer…” above)), estimates a future position of the object (paras. [0049-0050], recite(s)
[0049] “The tracked moving body confirmation unit 54 reads the integrated object MOF, the first detection result RES1, and the second detection result RES2 output by the tracking processing unit 55 and reads the moving body candidate CPE from the memory 26. Information related to a moving body candidate region corresponding to a candidate for a tracking target is stored as one record for each candidate in the moving body candidate CPE…”
[0050] “FIG. 7 is a diagram illustrating an example of the moving body candidate CPE. The moving body candidate CPE includes one or more records, and a plurality of predicted positions in a subsequent frame of the moving body candidate region, a speed of the moving body candidate region, a direction of movement of the moving body candidate region, and a moving body certainty factor are recorded in each record. An expected position of the moving body candidate region is defined as a rectangle, and coordinates thereof are identified by top-left coordinates and bottom-right coordinates. However, the expected position is not limited to the rectangle, and it is possible to adopt a region having an elliptical or arbitrary shape. In addition, the direction of movement of the moving body candidate region may include not only a component of a left-right direction but also a component of a depth direction, that is, a direction approaching/away from the camera 12.”
, where the “predicted positions in a subsequent frame of the moving body candidate region” is estimating a future position of the object (i.e., the “moving body”) on a basis of at least the acquired position and size in the bird’s eye view image (i.e., the “second candidate region” recited previously in para. [0044] above, which is part of the “second detection result RES2” recited in para. [0049] above)), and sets the position and size of the image area and the object tracker tracks the object in a next image frame by associating the estimated future position with the captured image (paras. [0049-0050], recite(s)
[0049] “The tracked moving body confirmation unit 54 reads the integrated object MOF, the first detection result RES1, and the second detection result RES2 output by the tracking processing unit 55 and reads the moving body candidate CPE from the memory 26. Information related to a moving body candidate region corresponding to a candidate for a tracking target is stored as one record for each candidate in the moving body candidate CPE…”
[0050] “FIG. 7 is a diagram illustrating an example of the moving body candidate CPE. The moving body candidate CPE includes one or more records, and a plurality of predicted positions in a subsequent frame of the moving body candidate region, a speed of the moving body candidate region, a direction of movement of the moving body candidate region, and a moving body certainty factor are recorded in each record. An expected position of the moving body candidate region is defined as a rectangle, and coordinates thereof are identified by top-left coordinates and bottom-right coordinates. However, the expected position is not limited to the rectangle, and it is possible to adopt a region having an elliptical or arbitrary shape. In addition, the direction of movement of the moving body candidate region may include not only a component of a left-right direction but also a component of a depth direction, that is, a direction approaching/away from the camera 12.”
, where the “predicted positions in a subsequent frame of the moving body candidate region” is estimating a future position of the object (i.e., the “moving body”) on a basis of at least the acquired position and size of the bird’s eye-view image (e.g., the “second candidate region” is part of “second detection result RES2” recited in previously in para. [0044] in the current claim above); where para. [0056] further recites including behavior information of the mobile object:
[0073] “In step S603, a movement amount and a direction of movement of the vehicle 10 during time Δt from photographing of an immediately preceding image to the present are calculated using vehicle information, that is, outputs of the wheel speed sensor 20 and the steering angle sensor 22. Further, the overhead image created in step S602, an influence of movement of the vehicle 10 is corrected based on the movement amount and the direction of movement. Subsequently, the operation proceeds to step S604.”
, where the “movement of the vehicle 10” including “wheel speed” and “steering angle” outputs are behavior information of the mobile object (i.e., “vehicle 10”);
the examiner additionally notes that the phrase “for tracking the object in a next image frame” in this claim recites intended use/result, and thus is not interpreted as a functional or structural requirement of the claim—see “Additional Claim Interpretations” B under section “Claim Interpretations” above).
Where Hasejima does not specifically disclose
…estimates a future position of the object in the bird's-eye view image…;
Hashimoto teaches in the same field of endeavor of object tracking
…estimates a future position of the object in the bird's-eye view image… (para. [0081], recites
[0081] “The driving planning unit 35 refers to the detected-object list to generate one or more trajectories to be traveled of the vehicle 10 so that the vehicle 10 will not collide with an object near the vehicle 10 . Each trajectory to be traveled is represented as, for example, a set of target locations of the vehicle 10 at points in time from the current time to a predetermined time ahead thereof. For example, the driving planning unit 35 refers to the detected-object list to perform viewpoint transformation, using information such as the position at which the camera 2 is mounted on the vehicle 10 , thereby transforming the image coordinates of the objects in the detected-object list into coordinates in an aerial image (“aerial-image coordinates”). The driving planning unit 35 then performs a tracking process on sequential aerial-image coordinates, using a Kalman filter, a particle filter, or another filter, to track the objects entered in the detected-object list, and uses the trajectories obtained from the tracking results to determine predicted trajectories of the respective objects to a predetermined time ahead. The driving planning unit 35 uses the results of identification of the states of the detection targets to determine the predicted trajectories.”
, where determining for “sequential aerial-image coordinates” of tracked objects “predicted trajectories of the respective [tracked] objects to a predetermined time ahead” is estimating a future position of at least an object (e.g., one of the “objects in the detected-object list”) in the bird’s-eye view image (e.g., an “aerial image”)).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the presently filed invention to modify the system of Hasejima to incorporate estimating the future position of the object in the bird’s-eye view image to prevent collision of the mobile object with the object in a generated driving plan of the mobile object as taught by Hashimoto (para. [0081]—see citation in the current claim above).
Claim 4 is rejected under 35 U.S.C. 103 as being unpatentable over Hasejima as applied to claim 1 above, and further in view of Zhang et al. (Zhang; US 2023/0085024 A1).
Regarding claim 4, Hasejima discloses the object tracking device according to claim 1,
wherein the object tracker uses a (para. [0202], recite(s)
[0202] “(Modification 1) In the above-described embodiment, the tracking processing unit 55 saves the speed of the moving body in the tracked moving body DMO on the assumption that the moving body moves in uniform motion. However, the tracking processing unit 55 may calculate movement of the moving body using the Kalman filter, and save a parameter thereof in the tracked moving body DMO.”
, where the Kalman filter is a filter for tracking the object).
Where Hasejima does not specifically disclose
wherein the object tracker uses a kernelized correlation filter (KCF) for the tracking of the object;
Zhang teaches in the same field of endeavor of object tracking
wherein the object tracker uses a kernelized correlation filter (KCF) for the tracking of the object (paras. [0026] and [0028], recite(s)
[0026] “Referring now to FIG. 8, due to significant size variation of target objects from far field to near field (i.e., the distance of the object from the vehicle), the performance of conventional object detection frameworks typically degenerate in near field. To solve this issue, implementations herein include a tracking-by-detection strategy that is applied to further track the valid inter-hypotheses which are determined valid in previous frames of captured image data. These KCF-tracked objects are combined with current valid inter-hypothesis objects to generate final outputs for the object detection framework. The KCF-hypothesis module includes a KCF tracker submodule and a Hypotheses evaluation submodule.”
[0028] “The KCF-hypothesis processing module (FIG. 8) receives the valid objects from the inter-hypotheses processing module (determined from previous frames). The module further tracks the objects in the following frames of image data using a tracking-by-detection algorithm (e.g. Kernelized Correlation Filters (KCF)). These tracked objects will be then combined with newly generated objects from the inter-hypotheses processing module to generate the final outputs. With this additional step, the system further improves the stability and consistency of object detection. Notably, the system improves the near-field object detection performance due to significant size variation of target objects from far to near field.”
, where the “Kernelized Correlation Filters (KCF)” is an object tracking algorithm (e.g., “tracking-by-detection-algorithm”)).
Since Zhang also discloses using a Kalman filter for tracking objects (para. [0025], recite(s)
[0025] “Referring now to FIG. 7, to reduce jumpy or inconsistent detection and improve the stability and consistency of object detection, the inter-hypotheses processing module further filters and smooths valid objects generated by the intra-hypotheses processing module (FIG. 6) and correspond to the same real objects. …With this module, the current valid hypotheses (i.e., from the current frame of captured image data) are merged with a tracked hypotheses which the system determined was valid in previous frames of captured image data. That is, intra-hypotheses that represent the same detected object are merged into a single inter-hypothesis, and thus the quantity of the inter-hypotheses is less than or equal to the quantity of the intra-hypotheses. After the merging, the assigned hypotheses are filtered using, for example, a Kalman filter as their predictions. These predicted results are further evaluated as valid inter-hypotheses outputs. The inter-hypotheses merging, the inter-hypotheses prediction, and the inter-hypotheses evaluation modules may operate in a manner similar to the modules of FIG. 6, only instead using valid intra-hypothesis as input instead of the filtered hypotheses from the hypotheses filtering module.”
), it would have been obvious to one of ordinary skill in the art before the effective filing date of the presently filed invention to modify the system of Hasejima to incorporate further incorporate tracking the object using a kernelized correlation filter (KCF) to improve object tracking of near field objects as taught by Zhang (para. [0026]—see citation in the current claim above).
Claim 5 is rejected under 35 U.S.C. 103 as being unpatentable over Hasejima as applied to claim 1 above, and further in view of Xu et al. (Xu; US 2019/0359205 A1).
Regarding claim 5, Hasejima discloses the object tracking device according to claim 1,
wherein Xu teaches in the same field of endeavor of object tracking the area setter increases the size of the image area when the mobile object travels to avoid contact with the object, compared to the size of the image area when the mobile object does not travel to avoid the contact (para. [0021] and [0062], recite(s)
[0021] “According to some embodiments, a new method for determining a path for an autonomous driving vehicle (ADV) is utilized. Various moving obstacles/objects may be located in the same geographical area where an ADV is travelling/located. These moving obstacles may move unpredictably. For example, although a pedestrian may be moving along a path, the pedestrian may suddenly change direction (e.g., turn left) or may accelerate/decelerate. This may cause problems when attempting to predict the path of the moving object in order to avoid colliding with, hitting, or striking the moving object. Thus, it may be useful to determine (e.g., calculate, compute, obtain, etc.) an area that includes possible locations where a moving obstacle may move to.”
[0062] “In one embodiment, the area component 433 may determining the predicted area based on a plurality of surrounding areas determined by the variation component 432 . The variation component 432 may determine a surrounding area for each of the predicted points determined (e.g., obtained, calculated, generated, etc.) by the path component 431 , as discussed in more detail below. For example, the variation component 432 may determine a surrounding area for each of the predicted points generated by the path component 431 . Each surrounding area may be determined based on a threshold standard deviation from a respective predicted point. For example, the size (e.g., the length, width, etc.) of a surrounding area may be equal to three standard deviations from a respective predicted point. The surrounding area may be referred to as an X-sigma (X-σ) area, where X is the number of standard deviations from the respective predicted point. For example, if the size (e.g., the length, width, etc.) of the surrounding area is equal to three standard deviations from a respective predicted point, then the surrounding area may be referred to as a 3-sigma (3-σ) area. A 3-σ area may be an area where there is a 99.7% chance (e.g., probability) that the moving obstacle will not move outside the boundaries of the 3-σ area. In some embodiments, the size of the surrounding area may be different if a different number of standard deviations are used. For example, a surrounding area with a size that is equal to two standard deviations from a respective predicted point (which may be referred to as a 2-sigma (2-σ) area) may be smaller than a surrounding area with a size that is equal to three standard deviations from a respective predicted point (e.g., a 3-σ area). A 2-σ area may be an area where there is a 95% chance (e.g., probability) that the moving obstacle will not move outside the boundaries of the 2-σ area.”
, where changing the size of a “surrounding area” of a “predicted point” (i.e., predicted position) of a tracked object when determining motion plans for a vehicle to avoid collision with moving objects comprises of at least increasing the size of the image area (e.g., a changing from a “2-σ area” to a “3-σ area”) when the mobile object travels to avoid contact with the object (e.g., moving obstacles/objects avoidance) to increase the chance of the mobile object from not hitting the object (e.g., a “3-σ area” has a 99.7% chance that the mobile object will not hit the object compared to 95% for a “2-σ area”)).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the presently filed invention to modify the system of Hasejima to incorporate increasing the size of the image area when the mobile object travels to avoid contact with the object compared to the size of the image area when the mobile object does not travel to avoid the contact to include all possible positions the object may move to in order to increase the chances of preventing the mobile object from colliding with the object when planning a path for the mobile object as taught by Xu (para. [0083], recite(s)
[0083] “In some embodiments, the predicted area 635 may include or encompass likely locations where the pedestrian 630 may move to. Because the pedestrian 630 may move unpredictably while moving/travelling through the environment 600A it may be useful to determine an area that includes the possible locations where the moving obstacle may move. If 3-sigma (3-σ) surrounding areas are used to determine the predicted area 635 , then the predicted area 635 may encompasses 99.7% of the possible locations that the pedestrian 630 may move to. Thus, if the ADV avoids the predicted area 635, the ADV may have a 99.7% change of avoiding (e.g., not hitting) the pedestrian 630 . This may also allow the ADV to operate more safely by increasing the chances that the ADV will collide with, hit, strike, etc., the moving obstacle.”
).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Liu et al. (US 2019/0103026 A1) discloses in the abstract and paras. [0042] and [0044]:
[abstract] “A collision warning system determines probabilities of potential collisions between a vehicle and other objects such as other vehicles. In an embodiment, sensors of a client device capture sensor data including motion data and image frames from a forward-facing view of the vehicle. An orientation of the client device relative to the vehicle may be determined using the motion data. The collision warning system determines cropped portions of the image frames and detects an object captured the image frames by processing the cropped portions. The collision warning system determines a probability of a potential collision between the vehicle and the object by tracking motion of the object. Responsive to determining that the probability is greater than a threshold value, the collision warning system may provide a notification of the potential collision to a driver of the vehicle.”
[0042] “The tracker 330 tracks objects in image frames detected by the object detector 320. The tracker 330 may receive image parameters from the cropping engine 310 as input to perform tracking. The tracker 330 determines trajectory information (also referred to as features or object data), which may include a distance between a vehicle 140 and another object, or a rate of change of the distance (e.g., indicating that the vehicle 140 is quickly approaching the other object). To reduce the compute resources for object tracking, the tracker 330 processes cropped images of image frames determined by the cropping engine 310. Since cropped images include a lower amount of data (e.g., less pixels) to process, the required compute time decreases. In some embodiments, the tracker 330 uses optical flow to find pixel correspondence of cropped portions of image frames from the cropping engine 310. For instance, the tracker 330 detects an amount of pixel shift in a certain one or more directions (e.g., up, down, left, right) between image frames to determine motion or trajectory of detected objects.”
[0044] “FIGS. 6B-C show diagrams illustrating bounding boxes for tracking objects according to various embodiments. To track a detected object, the tracker 330 may predict motion of the detected object. For example, as shown in FIG. 6B, given an old position 610 of the detected object, the tracker 330 uses optical flow (or another suitable algorithm) to determine a candidate position for a new position 620. The tracker 330 performs a local search around an initial predicted position using a bounding box (e.g., a rectangular search window) having fixed dimensions to find the new position 620. As shown in FIG. 6C, the tracker 330 may also perform a search around candidate positions by scaling the bounding box 630 to one or more different dimensions (e.g., expanding, shrinking, or skewing). The tracker 330 may scale the bounding box 630 based on a vanishing point 640 or projected lines intersecting the vanishing point 640 in the 2D image frame. The tracker 330 may predict how the bounding box 630 may change as the detected object moves closer toward a provider's vehicle 140, e.g., based on changes in a vertical direction of the image frames. For instance, a first bounding box tracking a first vehicle in a different lane on the road will have greater change in the vertical direction than a second bounding box tracking a second vehicle in a same lane as the provider's vehicle 140.”
Sakai et al. (US 2022/0254164 A1) discloses in the abstract and paras. [0039] and [0056]:
[abstract] “An external recognition device that detects and tracks a moving object includes an object position detection unit that detects a position of the moving object as an observation value, based on an image; a region determination unit that determines a region to which the moving object belongs in the image, based on the observation value; an observation error setting unit that calculates an error to an observation value, based on a determination result; a state prediction unit that predicts a state of the moving object at a current time, based on the observation value up to a previous time that is a time earlier than the current time and the error; an association unit that associates the state of the moving object at the current time with the observation value; and a state update unit that updates the state of the moving object, based on a result of the association.”
[0039] “The state prediction unit 15 predicts, based on the state of the detection moving object obtained by updating in the state update unit 16, the state of the moving object at a current time, by utilizing a result detected by the object position detection unit 11 up to a time (previous time) earlier than the current time. Here, it is assumed that a state has at least a position and a velocity. A prediction result by the state prediction unit 15 is utilized in the association unit 14.”
[0056] “Moreover, a criterion (boundary) of division of separating into the four regions A1, A2, A3, and A4 is set so that a part higher than a vanishing point (see FIG. 6) in the image 20 is the region A1, a part up to the region A1 from a spot where a detection position of a distance of about 0.8 m on the mobile body coordinate system changes due to an error of one pixel is the region A2, a part up to the region A2 from a spot where a detection position of a distance of about 0.2 m on the mobile body coordinate system changes from the region A2 due to an error of one pixel is the region A3, and a part lower than the region A3 is the region A4.”
Taghavi et al. (US 2022/0012466 A1) discloses in the abstract and paras. [0026] and [0052]:
[abstract] “A system and method for generating a bounding box for an object in proximity to a vehicle are disclosed. The method includes: receiving a three-dimensional (3D) point cloud representative of an environment; receiving a two-dimensional (2D) image of the environment; processing the 3D point cloud to identify an object cluster of 3D data points for a 3D object in the 3D point cloud; processing the 2D image to detect a 2D object in the 2D image and generate information regarding the 2D object from the 2D image; and when the 3D object and the 2D object correspond to the same object in the environment: generating a bird's eye view (BEV) bounding box for the object based on the object cluster of 3D data points and the information from the 2D image.”
[0073] “In some embodiments, the tracked heading h.sub.track of the object is computed using a multi-target tracking method, which may use Kalman filtering to predict the position of a target in a given (e.g. a most recent) image or 3D point cloud based on information from a set of 2D images or 3D point clouds immediately preceding the given image or 3D point cloud. The perception module 176 may use data representing dynamics of one or more objects from 3D point cloud 210 as determined by the point cloud processing method 215, or data representing dynamics of one or more objects in the 2D images 220 as determined by the 2D image processing method 225, in order to determine a variety of information regarding each of the one or more objects present in the 2D images 220. For example, the 2D image processing method 225 may be implemented to use a set of information regarding the objects, including for example an object ID, a location, as well as their respective 2D minimum bounding box across the set of 2D images immediately preceding the given (e.g., most recent) image, to determine the tracked heading h.sub.track 230 of the object. Optionally, the point cloud processing method 215 may also be implemented to use a set of information regarding the objects, each represented by an object ID, as well as their respective 3D bounding box, across the set of 3D point clouds immediately preceding the given 3D point cloud, to determine the tracked heading h.sub.track of the object. The set of 2D images and the set of 3D point clouds may be captured in the same time period, and the same object may be associated with the same object ID in both the 2D image and the 3D point cloud. For a given object identified in a 2D image or 3D point cloud taken at a current time t, the perception module 176 needs historical information from 2D images or 3D point clouds taken up to time t, in order to predict the estimations for the object in the current time t. The perception module 176 may execute the multi-target tracking method to compute, in real time or near real time, various estimations regarding the object, including for example velocity, position, acceleration, class, heading, and an estimated uncertainty of the heading, of the object.”
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JULIA Z YAO whose telephone number is (571)272-2870. The examiner can normally be reached Monday - Friday (8:30AM - 5PM).
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Emily Terrell can be reached on (571)270-3717. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/J.Z.Y./Examiner, Art Unit 2666
/EMILY C TERRELL/Supervisory Patent Examiner, Art Unit 2666
Prosecution Insights