DETAILED ACTION
Response to Amendment
Applicants’ response to the last Office Action, filed on 11/21/2025 has been entered and made of record.
In view of the Applicant’s amendments, the rejection under 35 U.S.C. 112 of claim 9 is expressly withdrawn.
Applicant’s amendment has necessitated new grounds of rejection. Thus, new grounds of rejection are presented in this Office Action.
Response to Arguments
Applicant’s arguments with respect to the claims have been considered, however, the arguments are indicated towards the newly added limitation of detecting a road user in the scene in each of the received digital image and the Lidar point cloud. Thus, Examiner has brought in reference Lo et al. (US 2020/0272155) to address the added limitation to the claims. Examiner also suggests considering the additional pertinent prior art made of record and not relied upon disclosed in the Conclusion section below.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1, 10, 11, and 14-17 are rejected under 35 U.S.C. 103 as being unpatentable over Mao et al. (US 2020/0125112) in view of Lo et al. (US 2020/0272155).
With regards to claim 1, Mao et al. discloses a method of determining information related to a road user in an environment of a vehicle, the method comprising:
receiving, from vehicle sensors, a digital image and a Lidar point cloud, both representing a scene in the environment of the vehicle (Para. 0044 lines 7-15, 0045 lines 1-7, "images" "LIDAR");
detecting the road user in the scene in the received digital image and the received Lidar point cloud (Para. 0047 lines 1-7, 0049 lines 1-15, "combination of LIDAR and camera images" “bounding box is correct in LIDAR data as well as in camera image” "action label" "pedestrians");
generating a combined digital representation of the detected road user by combining corresponding digital image data and Lidar data associated with the detected road user (Para. 0046 lines 1-15, "fused"); and
determining information related to the detected road user by processing the combined digital representation of the detected road user (Para. 0047 lines 1-7, 0048 lines 1-6, 0050 lines 1-7, "bounding box" "descriptors" "action").
Mao et al. does not explicitly teach detecting the road user in the scene in each of the received digital image and the received Lidar point cloud.
However, Lo et al. discloses the concept of receiving a digital image and Lidar data, detecting a pedestrian in each of the received digital image and the received Lidar data, and generating a combined digital representation of the detected road user to perform further processing on in order to detect all potential objects in the scene since different types of images of the same scene are used (Para. 0064 lines 1-14, 0065 lines 1-8, 0086 lines 1-9, 0087 lines 1-10, 0089 lines 1-19, 0090 lines 1-5, “pedestrians” “image data” “LiDAR data”).
It would have been obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to include the concept of receiving a digital image and Lidar data and detecting a pedestrian in each of the received digital image and the received Lidar data as taught by Lo et al. into the method of Mao et al. The motivation for this would be to detect all potential objects in the scene.
With regards to claim 10, the combination of Mao et al. and Lo et al. discloses the method of claim 1 wherein the combined digital representation of the detected road user includes: a collection of points; and for each point, a combination of corresponding RGB data and Lidar data (Mao et al.: Para. 0062 lines 1-25, "LIDAR data" "RGB color channel data" "intensity").
With regards to claim 11, the combination of Mao et al. and Lo et al. discloses the method of claim 1 wherein the combined digital representation of the detected road user includes :a collection of points; and for each point, a combination of corresponding RGB data, Lidar depth data, and Lidar intensity data (Mao et al.: Para. 0062 lines 1-25, "LIDAR data" "RGB color channel data" "depth information" "intensity").
With regards to claim 14, the combination of Mao et al. and Lo et al. discloses the method of claim 1 further comprising: predicting a trajectory of the road user based on the determined information related to the road user (Mao et al.: Para. 0073 lines 1-13, "trajectory for the pedestrian"); and controlling a function of the vehicle based on the predicted trajectory (Mao et al.: Para. 0073 lines 1-16, "generate trajectories" "respond").
With regards to claim 15, Mao et al. discloses a computer system for determining information related to a road user in an environment of a vehicle, the computer system comprising a memory and at least one processor configured to execute instructions (Para. 0020 lines 7-10, 0021 lines 1-4, “processor” “memory”) including:
receiving, from vehicle sensors, a digital image and a Lidar point cloud, both representing a scene in the environment of the vehicle (Para. 0044 lines 7-15, 0045 lines 1-7, "images" "LIDAR");
detecting the road user in the scene in the received digital image and the received Lidar point cloud (Para. 0047 lines 1-7, 0049 lines 1-15, "combination of LIDAR and camera images" “bounding box is correct in LIDAR data as well as in camera image” "action label" "pedestrians");
generating a combined digital representation of the detected road user by combining corresponding digital image data and Lidar data associated with the detected road user (Para. 0046 lines 1-15, "fused"); and
determining information related to the detected road user by processing the combined digital representation of the detected road user (Para. 0047 lines 1-7, 0048 lines 1-6, 0050 lines 1-7, "bounding box" "descriptors" "action").
Mao et al. does not explicitly teach detecting the road user in the scene in each of the received digital image and the received Lidar point cloud.
However, Lo et al. discloses the concept of receiving a digital image and Lidar data, detecting a pedestrian in each of the received digital image and the received Lidar data, and generating a combined digital representation of the detected road user to perform further processing on in order to detect all potential objects in the scene since different types of images of the same scene are used (Para. 0064 lines 1-14, 0065 lines 1-8, 0086 lines 1-9, 0087 lines 1-10, 0089 lines 1-19, 0090 lines 1-5, “pedestrians” “image data” “LiDAR data”).
It would have been obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to include the concept of receiving a digital image and Lidar data and detecting a pedestrian in each of the received digital image and the received Lidar data as taught by Lo et al. into the computer system of Mao et al. The motivation for this would be to detect all potential objects in the scene.
With regards to claim 16, the combination of Mao et al. and Lo et al. discloses a vehicle comprising the computer system of claim 15 (Mao et al.: Para. 0020 lines 7-10, 0021 lines 1-4, “vehicle”, see also claim 15 rejection above).
With regards to claim 17, it recites the apparatus of claim 15 as a non-transitory computer-readable medium comprising instructions to perform the functions. Mao et al. discloses the non-transitory computer-readable medium (Para. 0020 lines 7-10, 0021 lines 1-4, 0022 lines 1-3, “processor” “memory”). Thus, the analysis in rejecting claim 15 is equally applicable to claim 17.
Claim(s) 2-5 and 8 are rejected under 35 U.S.C. 103 as being unpatentable over Mao et al. (US 2020/0125112) in view of Lo et al. (US 2020/0272155) and further in view of Goel et al. (US 12,100,224).
With regards to claim 2, the combination of Mao et al. and Lo et al. discloses the method of claim 1, wherein determining information related to the detected road user comprises determining a bounding box of the pedestrian and predicting action labels of the road user to use in predicting a trajectory of the road user (Para. 0047 lines 1-7, 0048 lines 1-6, 0050 lines 1-7, 0073 lines 1-13, "descriptors" "action").
The combination of Mao et al. and Lo et al. does not explicitly teach wherein determining information related to the detected road user includes determining key points of the detected road user.
However, Goel et al. discloses determining key points of the detected road user to use in predicting a trajectory of the road user (Col. 18 lines 47-51, Col. 19 lines 39-42 and 51-58, Col. 20 lines 22-27 and 37-50, the combination of Mao et al. and Lo et al. discloses determining a bounding box of the road user to use in predicting action labels and the trajectory of the road user, Goel et al. teaches determining key points of the detected road user for the same purpose of determining actions and trajectory of the road user. In both cases, information of the road user is obtained to use in determining the trajectory of the road user.
It would have been obvious for one of ordinary skill in the art before the effective filing date to modify the combination of Mao et al. and Lo et al. to replace the technique of determining information related to the road user to use in predicting the trajectory of the road user with more specifically determining key points of the detected road user to use in predicting a trajectory of the road user as taught by Goel et al. since one of ordinary skill in the art would have been able to carry out such a substitution and the results from the substitution would be predictable to obtain information related to the detected road user to use in predicting a trajectory of the road user.
With regards to claim 3, the combination of Mao et al., Lo et al., and Goel et al. discloses the method of claim 2 further comprising determining 3D key points in a 3D space from the determined key points of the detected road user, based on the Lidar data (Goel et al.: Col. 18 lines 47-51 and 59, "key points" "3D space" "lidar data").
With regards to claim 4, the combination of Mao et al., Lo et al., and Goel et al. discloses the method of claim 3 further comprising predicting a trajectory of the detected road user based on the determined information related to the detected road user (Mao et al.: Para. 0073 lines 1-13, "trajectory for the pedestrian").
With regards to claim 5, the combination of Mao et al., Lo et al., and Goel et al. discloses the method of claim 4 further comprising controlling a function of the vehicle based on the predicted trajectory (Mao et al.: Para. 0073 lines 1-16, "generate trajectories" "respond").
With regards to claim 8, the combination of Mao et al., Lo et al., and Goel et al. discloses the method of claim 2 wherein: the detected road user is a pedestrian (Mao et al.: 0049 lines 1-15, "pedestrians"), and detecting key points includes detecting body key points of the pedestrian (Goel et al.: Col. 18 lines 47-51, Fig. 7A, Fig. 7B, "key points").
Claim(s) 6 and 7 are rejected under 35 U.S.C. 103 as being unpatentable over Mao et al. (US 2020/0125112) in view of Lo et al. (US 2020/0272155) and Goel et al. (US 12,100,224) and further in view of Li et al. (US 2022/0171065).
With regards to claim 6, the combination of Mao et al., Lo et al., and Goel et al. discloses the method of claim 4.
The combination of Mao et al., Lo et al., and Goel et al. does not explicitly teach wherein: predicting the trajectory of the detected road user includes predicting a plurality of trajectories of the detected road user with respective probability values; and the method further comprises, for each predicted trajectory: assigning a score to the predicted trajectory, based on the determined 3D key points, and updating the probability value of the predicted trajectory based on the assigned score.
However, Li et al. discloses where predicting a trajectory of the pedestrian includes predicting a plurality of trajectories of the road user with respective probability values and for each predicted trajectory, assigning a score to the predicted trajectory, and updating the probability value of the predicted trajectory based on the assigned score (Para. 0031 lines 1-6 and 12-14, 0040 lines 1-6, 0049 lines 1-8, 0053 lines 1-17, 0054 lines 1-11, "score" "probability"). The combination of Mao et al., Lo et al., and Goel et al. discloses predicting a trajectory of the road user, and the technique as taught by Li et al. is just one of a finite number of ways to predict a trajectory of a road user, by predicting a plurality of trajectories of the road user with respective probability values and for each predicted trajectory, assigning a score to the predicted trajectory, and updating the probability value of the predicted trajectory based on the assigned score. Thus, the combination of Mao et al., Lo et al., and Goel et al. would be modified to, in predicting the trajectory of the road user based on the 3D key points, predict a plurality of trajectories with probability values for each predicted trajectory, assigning a score to each predicted trajectory (determined based on the 3D key points), and update the probability value of the predicted trajectory based on the assigned score.
It would have been obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to try and include the technique of predicting a trajectory of the pedestrian includes predicting a plurality of trajectories of the road user with respective probability values and for each predicted trajectory, assigning a score to the predicted trajectory, and updating the probability value of the predicted trajectory based on the assigned score as taught by Li et al. into the method of the combination of Mao et al., Lo et al., and Goel et al. since one of ordinary skill in the art could have pursued the technique with a reasonable expectation of success of predicting a trajectory of the road user.
With regards to claim 7, the combination of Mao et al., Lo et al., and Goel et al. discloses the method of claim 4, further comprising determining, based on the determined 3D key points, information on a direction that the predicted trajectory should be; and providing the information on the direction as input for the predicting of the trajectory (Goel et al.: Col. 19 lines 39-42 and 51-58, Col. 20 lines 22-27 and 37-50, "key points" "bounding box" "direction of travel").
The combination of Mao et al., Lo et al., and Goel et al. does not explicitly teach a range of directions that the predicted trajectory should be.
However, Li et al. discloses determining information on a range of directions that the predicted trajectory should be and providing the information as input for the prediction of trajectory (Para. 0031 lines 1-6 and 12-14, 0040 lines 1-6, 0049 lines 1-8, 0053 lines 1-22, 0054 lines 1-11, Fig. 5, "candidate trajectories" "predicted movement trajectory") in order to determine more accurate predictions of pedestrian movement trajectories (Para. 0054 lines 1-11, 0060 lines 15-17, "highest probability").
It would have been obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to include the technique of determining information on a range of directions that the predicted trajectory should be and providing the information as input for the prediction of trajectory as taught by Li et al. into the method of the combination of Mao et al., Lo et al., and Goel et al. The motivation for this would be to determine more accurate predictions of pedestrian movement trajectories by using the trajectory of highest probability.
Claim(s) 9 is rejected under 35 U.S.C. 103 as being unpatentable over Mao et al. (US 2020/0125112) in view of Lo et al. (US 2020/0272155) and Goel et al. (US 12,100,224) and further in view of Mao et al. (US 2023/0059370, herein referred to as Mao-2).
With regards to claim 9, the combination of Mao et al., Lo et al., and Goel et al. discloses the method of claim 2 wherein determining information related to the detected road user includes: determining, based on the determined key points, at least one of an orientation or a pose of the detected road user (Goel et al.: Col. 20 lines 37-50, "key points" "pose of the pedestrian").
The combination of Mao et al., Lo et al., and Goel et al. does not explicitly teach estimating, based on the at least one of the determined orientation or the determined pose of the detected road user, an awareness state of the detected road user selected among a plurality of predefined awareness states indicative of how the detected road user is aware of the vehicle.
However, Mao-2 discloses the concept of estimating, based on a determined pose of the road user, an awareness state of the road user among a plurality of predefined awareness states indicative of how the user is aware of the vehicle in order to better plan and predict whether it is safe to drive close to the road user (Para. 0013 lines 1-11, 0083 lines 1-4, 0090 lines 1-14, "awareness signal" "whether the agent is aware" "agent pose" "whether it is safe").
It would have been obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to include the concept of estimating, based on a determined pose of the road user, an awareness state of the road user among a plurality of predefined awareness states indicative of how the user is aware of the vehicle as taught by Mao-2 into the method of the combination of Mao et al., Lo et al., and Goel et al. The motivation for this would be to predict whether it is safe for a vehicle to drive close to the road user.
Claim(s) 12 and 13 are rejected under 35 U.S.C. 103 as being unpatentable over Mao et al. (US 2020/0125112) in view of Lo et al. (US 2020/0272155) and further in view of Liu et al. (Pedestrian Detection with Lidar Point Clouds Based on Single Template Matching).
With regards to claim 12, the combination of Mao et al. and Lo et al. discloses the method of claim 1.
The combination of Mao et al. and Lo et al. does not explicitly teach further comprising increasing a Lidar point density of the detected road user by performing a morphological image processing operation, before generating the combined digital representation.
However, Liu et al. discloses the concept of increasing a Lidar point density of a detected pedestrian by performing a morphological image processing operation (2.3.1 Project-Image Generation: Para. 3 lines 1-4, Fig. 3(c) and (d), "morphological expansion and hole filling") in order to obtain a clear pedestrian outline to use (2.3.2. Feature Extraction: Para. 1 lines 1-3, "clear pedestrian outline"). Thus, the combination of Mao et al. and Lo et al. would be modified to, in detecting the road user based on the received Lidar point cloud, increase a Lidar point density of the detected pedestrian by performing a morphological image processing operation, and then generating the combined digital representation of the detected road user.
It would have been obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to include the technique of increasing a Lidar point density of a detected pedestrian by performing a morphological image processing operation as taught by Liu et al. into the method of the combination of Mao et al. and Lo et al. The motivation for this would be to obtain a clear pedestrian outline to use.
With regards to claim 13, the combination of Mao et al., Lo et al., and Liu et al. discloses the method of claim 12 wherein the morphological image processing operation includes a morphological closing operation for filling gaps in the detected road user (Liu et al.: 2.3.1 Project-Image Generation: Para. 3 lines 1-4, Fig. 3(c) and (d), "morphological expansion and hole filling").
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Reference Wu et al. (LiDAR/Camera Sensor Fusion Technology for Pedestrian Detection) discloses the concept of receiving a digital image and Lidar data, detecting objects in each of the received digital image and the received Lidar data, and matching the detected objects to confirm a same object in the images.
Reference Lv et al. (CN112487905) discloses the concept of receiving a digital image and Lidar data, detecting a pedestrian in each of the received digital image and the received Lidar data, and determining information on the pedestrian detected.
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Contact Information
Any inquiry concerning this communication or earlier communications from the examiner should be directed to CAROL W CHAN whose telephone number is (571)272-5766. The examiner can normally be reached 9:30-3:30 M-F.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Sumati Lefkowitz can be reached at (571) 272-3638. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/CAROL W CHAN/Primary Examiner, Art Unit 2672