DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Status of Claims
Claims 1-20 are pending.
Response to Arguments
Applicant’s arguments, see p.11-12, filed 05/15/2026, with respect to the rejections of Claims 1-3, 6-14, and 17-20 under 35 U.S.C. 101 have been fully considered and are persuasive. Therefore, the rejections of Claims 1-3, 6-14, and 17-20 under this section of the Rules has been withdrawn.
Applicant’s arguments and amendments, see p. 12, filed 05/15/2026, with respect to the rejections of Claims 1-12 and 18 under 35 U.S.C. 112(b) have been fully considered and are persuasive. Therefore, the rejections of Claims 1-12 and 18 under this section of the Rules has been withdrawn.
Applicant’s arguments, see p.12-13, filed 05/15/2026, with respect to the rejections of Claims 1-20 under 35 U.S.C. 103 have been fully considered but are moot because Applicant’s amendments of the independent claims has altered the scope of the claims, and therefore, necessitated new grounds of rejection which are presented below. Accordingly, THIS ACTION IS MADE FINAL.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1, 11-13, and 18-19 are rejected under 35 U.S.C. 103 as being unpatentable over Yi et al. (CN 111652900 A) in view of Vallespi-Gonzalez (US 20170359561 A1), Kakegawa et al. (“Road surface segmentation based on vertically local disparity histogram for stereo camera”), Stein et al. (US 20190325595 A1), and Pei et al. (CN 112906449 A).
Regarding Claim 1, Yi teaches "A system comprising at least one processor to: determine a disparity image over a plurality of frames by performing, using a neural network, disparity estimation using at least a pair of images, the plurality of frames including a current frame and at least one previous frame"; (Yi, Claims 1, 2, 4, and 18, teaches a system comprising a processor for acquiring a binocular stereo image of a previous frame and a current frame and calculating the binocular stereo image of the previous frame and the current frame to obtain a disparity map of the previous frame and of the current frame wherein the calculations of the binocular stereo images of the current and previous frame are through an SFNet network model to obtain the disparity map of the previous frame and of the current frame, i.e., determine a disparity image over a plurality of frames by performing disparity estimation using a pair of images including current and previous frames using a neural network).
However, Yi does not explicitly teach "determine track points of an ego machine; determine first track point heights of the track points for each frame of the plurality of frames of the disparity image in disparity space; the first track point heights including previous track point heights of the track points for each previous frame of the at least one previous frame of the plurality of frames of the disparity image and current track point heights of the track points for the current frame of the plurality of frames of the disparity image; determine second track point heights by temporally fusing the current track point heights for the current frame and the previous track point heights for each previous frame of the at least one previous frame; adjust, based at least on the second track point heights, at least one of a suspension system or a speed of the ego machine".
In an analogous field of endeavor, Vallespi-Gonzalez teaches "determine track points of an ego machine"; (Vallespi-Gonzalez, Paras. 37 and 45-46, teaches the disparity mapper finds a set of points in one image which can be identified as the same points in the other image in order to create a disparity map by matching points or features in one image with the corresponding points or features in the other image wherein mapping resource data and sub-maps contain surface data for a given region, i.e., determine track points of the vehicle being the points identified in the scene of the vehicle).
It would have been obvious to one having ordinary skill in the art before the effective filing date to modify the invention of Yi by including the determination of track points of an ego machine taught by Vallespi-Gonzalez. One of ordinary skill in the art would be motivated to combine the references since it improves disparity map calculations (Vallespi-Gonzalez, Para. 10, teaches the motivation of combination to be to improve disparity map calculations).
However, Vallespi-Gonzales does not explicitly teach "determine first track point heights of the track points for each frame of the plurality of frames of the disparity image in disparity space; the first track point heights including previous track point heights of the track points for each previous frame of the at least one previous frame of the plurality of frames of the disparity image and current track point heights of the track points for the current frame of the plurality of frames of the disparity image; determine second track point heights by temporally fusing the current track point heights for the current frame and the previous track point heights for each previous frame of the at least one previous frame; adjust, based at least on the second track point heights, at least one of a suspension system or a speed of the ego machine".
In an analogous field of endeavor, Kakegawa teaches "determine first track point heights of the track points for each frame of the plurality of frames of the disparity image in disparity space"; (Kakegawa, Figures 4 and 5 and Section 5.1, teaches the road height profile was estimated by fitting a line using the least square method to the disparities of the segmented road surface in V-disparity space wherein the scene length is 100 frames and the evaluation is performed for each frame, i.e., determine track point heights being the road height profile for each frame of the plurality of frames of the disparity image in disparity space).
It would have been obvious to one having ordinary skill in the art before the effective filing date to modify the invention of Yi and Vallespi-Gonzalez by including the determination of track point heights for each frame in disparity space taught by Kakegawa. One of ordinary skill in the art would be motivated to combine the references since it improves the accuracy of the surface segmentation and height profile (Kakegawa, Abstract, teaches the motivation of combination to be to improve the accuracy of the road surface segmentation and height profile).
However, the combination of references of Yi in view of Vallespi-Gonzalez and Kakegawa does not explicitly teach “the first track point heights including previous track point heights of the track points for each previous frame of the at least one previous frame of the plurality of frames of the disparity image and current track point heights of the track points for the current frame of the plurality of frames of the disparity image; determine second track point heights by temporally fusing the current track point heights for the current frame and the previous track point heights for each previous frame of the at least one previous frame; adjust, based at least on the second track point heights, at least one of a suspension system or a speed of the ego machine".
In an analogous field of endeavor, Stein teaches "the first track point heights including previous track point heights of the track points for each previous frame of the at least one previous frame of the plurality of frames of the disparity image and current track point heights of the track points for the current frame of the plurality of frames of the disparity image"; (Stein, Abstract and Paras. 45 and 55, teaches obtaining a time-ordered sequence of images representative of a road surface wherein a sensor capturing an image determines height of a point above a plane representing the road surface and wherein modeling the road surface includes warping a previous image to the current image using the gamma value which is a ratio of height of a point by a distance from a sensor which enables a feature to be matched between images based on its distance and height above the road surface, i.e., determine track point heights for the plurality of frames including track point heights for the previous frames and track point heights of the current frames).
It would have been obvious to one having ordinary skill in the art before the effective filing date to modify the invention of Yi, Vallespi-Gonzalez, and Kakegawa wherein the frames are of the disparity image by including the determination of track point heights of previous frames and current frames taught by Stein. One of ordinary skill in the art would be motivated to combine the references since it creates accurate warping between images (Stein, Para. 55, teaches the motivation of combination to be to create accurate warping between images).
However, the combination of references of Yi, Vallespi-Gonzalez, Kakegawa, and Stein does not explicitly teach “determine second track point heights by temporally fusing the current track point heights for the current frame and the previous track point heights for each previous frame of the at least one previous frame; adjust, based at least on the second track point heights, at least one of a suspension system or a speed of the ego machine".
In an analogous field of endeavor, Pei teaches "determine second track point heights by temporally fusing the current track point heights for the current frame and the previous track point heights for each previous frame of the at least one previous frame"; (Pei, Abstract and Claim 7, teaches calculating the area point cloud information based on the image information and the parallax information of the detection area, modelling the road plane, and calculating the height information of each discrete point in the area point cloud to the plane of the road surface wherein a height map is generated according to the area point cloud information and the height information and detection result is filtered and corrected by fusing the height map of the multi-frame continuous image to obtain the fusion result and wherein the fusing comprises acquiring time information of the multi-frame continuous image and calculating moving distance between two adject frames and updating the detection result of the two adjacent frames, i.e., determine second track point heights being the fusion result by fusing the track point heights of the current and previous frames being the fusion of the height map of the multi-frame continuous image comprising two adjacent frames);
"adjust, based at least on the second track point heights, at least one of a suspension system or a speed of the ego machine"; (Pei, Abstract and Background, teaches detecting and confirming the road fluctuation condition of the front detection area and the area with the pit characteristic according to the fusion result wherein the vehicle may reduce the influence of the road to the vehicle through adjusting the suspension setting if the pothole condition in the road ahead is detected in real time, i.e., adjust suspension of the ego machine based on track point heights).
It would have been obvious to one having ordinary skill in the art before the effective filing date to modify the invention of Yi, Vallespi-Gonzalez, Kakegawa, and Stein by including the temporal fusion of current and previous track point heights and adjusting suspension based on the determined heights taught by Pei. One of ordinary skill in the art would be motivated to combine the references since it detects surface road pits (Pei, Abstract, teaches the motivation of combination to be to detect road surface pits and improve driving comfort and safety).
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date.
Regarding Claim 11, the combination of references of Yi in view of Vallespi-Gonzalez, Kakegawa, Stein, and Pei teaches "The system of claim 1, the at least one processor further to temporally fuse the current track point heights and the previous track point heights by: determining the track points of the ego machine for the current frame"; (Pei, Abstract, teaches calculating the area point cloud information based on the image information and the parallax information of the detection area, modelling the road plane, and calculating the height information of each discrete point in the area cloud point of the plane of the road surface for a multi-frame continuous image, i.e., track points of the ego vehicle are determined for a current frame);
"transforming the track points of the at least one previous frame into a coordinate system of the current frame"; (Pei, Claim 7, teaches obtaining the multi-frame continuous image of the same road scene through the vehicle-mounted binocular stereo vision system, recording the collecting time information and speed information of the multi-frame continuous image, calculating the moving distance between two adjacent frames for the detection result of two adjacent frames, updating the position of the detection result in the previous state according to the moving distance, and then adding the detection result in the next state as the new detection data, i.e., transform the points of the previous frame to coordinate system of the current frame by calculating moving distance between the two frames and updating the positions according to the result);
"fitting the first track point heights to a same plane"; (Pei, Claim 5, teaches modelling the road plane through the region point cloud comprises fitting the road model based on the rebuilding information of the 3D point cloud, i.e., fitting track points including their heights to a same plane being the road surface);
"determining the second track point heights using the fitted first track point heights"; (Pei, Abstract and Claim 5, teaches modeling the road plane and calculating the height information of each discrete point in the area cloud point to the plane of the road surface equation, generating a height map according to the area point cloud information and the height information, and filtering and correcting the detection result by fusing the height map of the multi-frame continuous image to obtain the fusion result wherein modeling the road plane comprises fitting the road model based on the rebuilding information of the 3D point cloud, i.e., the second or fused track point heights are determined using the fitted point heights);
"and measuring the second track point heights against a virtual plane"; (Pei, Abstract and Claims 5 and 6, teaches modeling the road plane and calculating the height information of each discrete point in the area cloud point to the plane of the road surface equation, generating a height map according to the area point cloud information and the height information, filtering and correcting the detection result by fusing the height map of the multi-frame continuous image to obtain the fusion result, and detecting and confirming road fluctuation according to the fusion result wherein modeling the road plane comprises fitting the road model based on the rebuilding information of the 3D point cloud in which the road surface model plane follows an equation wherein the distance from each discrete point in the point cloud to the plane where the road surface equation is located is obtained by a formula in which the height is evaluated based on the discrete three-dimensional points of the distance road surface, i.e., measuring the track point heights against a virtual plane being the road surface plane evaluated to determine road fluctuations).
The proposed combination as well as the motivation for combining the Yi, Vallespi-Gonzalez, Kakegawa, Stein, and Pei references presented in the rejection of Claim 1, applies to claim 11. Thus, the system recited in claim 11 is met by Yi in view of Vallespi-Gonzalez, Kakegawa, Stein, and Pei.
Regarding Claim 12, the combination of references of Yi in view of Vallespi-Gonzalez, Kakegawa, Stein, and Pei teaches "The system of claim 1, wherein the system is comprised in at least one of: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system implemented using a robot; an aerial system; a medical system; a boating system; a smart area monitoring system; a system for performing one or more deep learning operations; a system for performing one or more simulation operations; a system for generating or presenting virtual reality (VR) content, augmented reality (AR) content, or mixed reality (MR) content; a system for performing one or more digital twin operations; a system implemented using an edge device; a system incorporating one or more virtual machines (VMs); a system for generating synthetic data; a system implemented at least partially in a data center; a system for performing one or more conversational artificial intelligence (AI) operations; a system for performing one or more generative AI operations; a system implementing one or more language models; a system implementing one or more large language models (LLMs); a system for hosting one or more real-time streaming applications; a system for performing light transport simulation; a system for performing collaborative content creation for 3D assets; or a system implemented at least partially using cloud computing resources"; (Vallespi-Gonzalez, Para. 5 and FIG. 1, teaches an example control system for operating an autonomous vehicle, i.e., a control system for an autonomous machine).
The proposed combination as well as the motivation for combining the Yi, Vallespi-Gonzalez, Kakegawa, Stein, and Pei references presented in the rejection of Claim 1, applies to claim 12. Thus, the system recited in claim 12 is met by Yi in view of Vallespi-Gonzalez, Kakegawa, Stein, and Pei.
Claim 13 recites a system with elements corresponding to the steps recited in Claim 1. Therefore, the recited elements of this claim are mapped to the proposed combination in the same manner as the corresponding steps in its corresponding method claim. Additionally, the rationale and motivation to combine the Yi in view of Vallespi-Gonzalez, Kakegawa, Stein, and Pei references, presented in rejection of Claim 1, apply to this claim. Finally, the combination of the Yi in view of Vallespi-Gonzalez, Kakegawa, Stein, and Pei references discloses an ego vehicle comprising a stereo camera for capturing pairs of images (for example, see Vallespi-Gonzalez, Claim 6 and Abstract).
Claim 18 recites a system with elements corresponding to the steps recited in Claim 11. Therefore, the recited elements of this claim are mapped to the proposed combination in the same manner as the corresponding steps in its corresponding method claim. Additionally, the rationale and motivation to combine the Yi in view of Vallespi-Gonzalez, Kakegawa, Stein, and Pei references, presented in rejection of Claim 1, apply to this claim. Finally, the combination of the Yi in view of Vallespi-Gonzalez, Kakegawa, Stein, and Pei references discloses an ego vehicle comprising a stereo camera for capturing pairs of images (for example, see Vallespi-Gonzalez, Claim 6 and Abstract).
Claim 19 recites a system with elements corresponding to the steps recited in Claim 12. Therefore, the recited elements of this claim are mapped to the proposed combination in the same manner as the corresponding steps in its corresponding method claim. Additionally, the rationale and motivation to combine the Yi in view of Vallespi-Gonzalez, Kakegawa, Stein, and Pei references, presented in rejection of Claim 1, apply to this claim. Finally, the combination of the Yi in view of Vallespi-Gonzalez, Kakegawa, Stein, and Pei references discloses an ego vehicle comprising a stereo camera for capturing pairs of images (for example, see Vallespi-Gonzalez, Claim 6 and Abstract).
Claim 2 is rejected under 35 U.S.C. 103 as being unpatentable over Yi in view of Vallespi-Gonzalez, Kakegawa, Stein, Pei, and Matzner (US 20200167938 A1).
Regarding Claim 2, the combination of references of Yi in view of Vallespi-Gonzalez, Kakegawa, Stein, and Pei does not explicitly teach "The system of claim 1, wherein the pair of images includes a rectified stereo pair comprising a left rectified image and a right rectified image; the left rectified image is generated by rectifying a stereo raw left frame; the right rectified image is generated by rectifying a stereo raw right frame; and the output generated using a stereo camera that generates the pair of images comprises the stereo raw left frame and the stereo raw right frame".
In an analogous field of endeavor, Matzner teaches "The system of claim 1, wherein the pair of images includes a rectified stereo pair comprising a left rectified image and a right rectified image; the left rectified image is generated by rectifying a stereo raw left frame; the right rectified image is generated by rectifying a stereo raw right frame; and the output generated using a stereo camera that generates the pair of images comprises the stereo raw left frame and the stereo raw right frame"; (Matzner, Para. 37, teaches 2D tracks are extracted from raw video from each thermal camera in the stereo pair in which 2D track data is rectified from the left and right thermal cameras wherein the two rectified images are matched and a depth map also called a disparity map is determined in order to generated 3D points from the matched stereo tracks, i.e., output form a stereo camera comprises raw left and right frames in which the left and right frames are rectified so that the pair of images includes a rectified stereo pair).
It would have been obvious to one having ordinary skill in the art before the effective filing date to modify the invention of Yi, Vallespi-Gonzalez, Kakegawa, Stein, and Pei by including the rectification of the stereo raw left and right frames taught by Matzner. One of ordinary skill in the art would be motivated to combine the references since it produces a detailed characterization of motion track (Matzner, Para. 37, teaches the motivation of combination to be to produce a detailed characterization of an observed flying object's motion track).
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date.
Claims 3-4 and 14-15 are rejected under 35 U.S.C. 103 as being unpatentable over Yi in view of Vallespi-Gonzalez, Kakegawa, Stein, Pei, and Jadhav et al. (US 20240371035 A1).
Regarding Claim 3, the combination of references of Yi in view of Vallespi-Gonzalez, Kakegawa, Stein, and Pei does not explicitly teach "The system of claim 1, wherein the disparity estimation is performed using a neural network; and the at least one processor is further to update the neural network using a disparity map determined using depth information from a plurality of spins of a light detection and ranging (LiDAR) sensor”.
In an analogous field of endeavor, Jadhav teaches "The system of claim 1, wherein the disparity estimation is performed using a neural network"; (Jadhav, Para. 156, teaches a subject frame of camera image data may be provided to a machine learning model such as a neural network model which processes the image data and generated an estimated depth output such as a disparity map, i.e., disparity estimation is performed using a neural network);
"and the at least one processor is further to update the neural network using a disparity map determined using depth information from a plurality of spins of a light detection and ranging (LiDAR) sensor"; (Jadhav, FIG. 11 and Paras. 149, 156, 165, and 171, teaches a LiDAR sensor performs a rotating 360 degree scan around the ego vehicle in which a training architecture for partial supervision in self-supervised depth estimation includes processing image data and generating a depth output such as a disparity map wherein using this architecture makes the model learn to generate improved and more accurate depth estimations, i.e., update the neural network using a disparity map determined using depth information from a plurality of spins of a LiDAR sensor being the training architecture of the model which uses a depth output and disparity map to learn).
It would have been obvious to one having ordinary skill in the art before the effective filing date to modify the invention of Yi, Vallespi-Gonzalez, Kakegawa, Stein, and Pei by including the use of a neural network for disparity estimation and updating the network using depth information from a spinning LiDAR taught by Jadhav. One of ordinary skill in the art would be motivated to combine the references since it improves depth estimation (Jadhav, Para. 171, teaches the motivation of combination to be to improve and generate more accurate depth estimations).
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date.
Regarding Claim 4, the combination of references of Yi in view of Vallespi-Gonzalez, Kakegawa, Stein, Pei, and Jadhav teaches "The system of claim 3, wherein the updating the neural network comprises: accumulating the depth information from the plurality of spins of the LiDAR sensor"; (Jadhav, Paras. 6 and 149, teaches a typical LiDAR sensor performs a rotating 360 degree scan around the ego vehicle wherein an edge detected point cloud frame based on depth is projected onto a 2D camera image and wherein the final calibration matrix utilizes both camera and LiDAR sensor outputs together such as depth estimation, i.e., accumulating depth information from plurality of spins of the LiDAR sensor);
"determining a depth map using the depth information from the plurality of spins of the LiDAR sensor"; (Jadhav, Paras. 149 and 156, teaches a typical LiDAR sensor performs a rotating 360 degree scan around the ego vehicle wherein an edge detected point cloud frame based on depth is projected onto a 2D camera image wherein a subject frame of camera image data is provided to the model to generate an estimated depth output such as a depth map, i.e., determining a depth map using the depth information from the LiDAR sensor);
"determining the disparity map using the depth map; and updating the neural network using the disparity map"; (Jadhav, FIG. 11 and Paras. 149, 156, 165, and 171, teaches a training architecture for partial supervision in self-supervised depth estimation includes processing image data and generating a depth output such as a depth map or a disparity map wherein depth and disparity are related and can be proportionally derived from each other and wherein using this architecture makes the model learn to generate improved and more accurate depth estimations, i.e., update the neural network using a disparity map determined using depth information from a plurality of spins of a LiDAR sensor being the training architecture of the model which uses a depth output and disparity map to learn).
The proposed combination as well as the motivation for combining the Yi, Vallespi-Gonzalez, Kakegawa, Stein, Pei, and Jadhav references presented in the rejection of Claim 3, applies to claim 4. Thus, the system recited in claim 4 is met by Yi, Vallespi-Gonzalez, Kakegawa, Stein, Pei, and Jadhav.
Claim 14 recites a system with elements corresponding to the steps recited in Claim 3. Therefore, the recited elements of this claim are mapped to the proposed combination in the same manner as the corresponding steps in its corresponding method claim. Additionally, the rationale and motivation to combine the Yi in view of Vallespi-Gonzalez, Kakegawa, Stein, Pei, and Jadhav references, presented in rejection of Claim 3, apply to this claim. Finally, the combination of the Yi in view of Vallespi-Gonzalez, Kakegawa, Stein, Pei, and Jadhav references discloses an ego vehicle comprising a stereo camera for capturing pairs of images (for example, see Vallespi-Gonzalez, Claim 6 and Abstract).
Claim 15 recites a system with elements corresponding to the steps recited in Claim 4. Therefore, the recited elements of this claim are mapped to the proposed combination in the same manner as the corresponding steps in its corresponding method claim. Additionally, the rationale and motivation to combine the Yi in view of Vallespi-Gonzalez, Kakegawa, Stein, Pei, and Jadhav references, presented in rejection of Claim 3, apply to this claim. Finally, the combination of the Yi in view of Vallespi-Gonzalez, Kakegawa, Stein, Pei, and Jadhav references discloses an ego vehicle comprising a stereo camera for capturing pairs of images (for example, see Vallespi-Gonzalez, Claim 6 and Abstract).
Claims 5 and 16 are rejected under 35 U.S.C. 103 as being unpatentable over Yi in view of Vallespi-Gonzalez, Kakegawa, Stein, Pei, Jadhav, and Wu et al. (US 20230281843 A1).
Regarding Claim 5, the combination of references of Yi in view of Vallespi-Gonzalez, Kakegawa, Stein, Pei, and Jadhav teaches "
"wherein the second pair of images are obtained using output from a stereo camera of a second ego machine"; (Vallespi-Gonzalez, Claim 6 and Abstract teaches an autonomous vehicle includes a stereoscopic camera which acquires a first image and a second image wherein the 3D environment data comprises sensor data compiled from a fleet of autonomous vehicles, i.e., second pair of images are obtained from a stereo camera of a second ego vehicle).
The proposed combination as well as the motivation for combining the Yi, Vallespi-Gonzalez, Kakegawa, Stein, and Pei references presented in the rejection of Claim 1, applies to claim 5.
However, the combination of references of Yi in view of Vallespi-Gonzalez, Kakegawa, Stein, Pei, and Jadhav does not explicitly teach “The system of claim 4, the at least one processor further to: determine second disparity image for a frame by performing, using the neural network, the disparity estimation using a second pair of images; determine a loss of the second disparity image based on the disparity map; and update one or more parameters of the neural network using the loss”.
In an analogous field of endeavor, Wu teaches "The system of claim 4, the at least one processor further to: determine second disparity image for a frame by performing, using the neural network, the disparity estimation using a second pair of images"; (Wu, Paras. 6 and 100, teaches the system generating two disparity maps using the machine learning model generating a swapped image pair and generating a second disparity for the swapped image pair wherein the machine learning models are neural networks, i.e., second disparity image using a neural network by disparity estimation of a second pair of images);
"determine a loss of the second disparity image based on the disparity map"; (Wu, Abstract and Para. 100, teaches generating a predicted disparity map and generating a total loss using the predicted disparity map and wherein occlusion loss is determined from the two disparity maps, i.e., determine a loss of the second disparity image based on the disparity map);
"and update one or more parameters of the neural network using the loss"; (Wu, Abstract, teaches updating the plurality of parameters of the machine learning model by minimizing the total losses, i.e., update parameters of the neural network using the loss).
It would have been obvious to one having ordinary skill in the art before the effective filing date to modify the invention of Yi in view of Vallespi-Gonzalez, Kakegawa, Stein, Pei, and Jadhav by including the determination of disparity using a second pair of images and determining a loss of the second image and updating the neural network using the loss taught by Wu. One of ordinary skill in the art would be motivated to combine the references since it improves depth map accuracy (Wu, Para. 18, teaches the motivation of combination to be to improve the accuracy of depth map generation).
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date.
Claim 16 recites a system with elements corresponding to the steps recited in Claim 5. Therefore, the recited elements of this claim are mapped to the proposed combination in the same manner as the corresponding steps in its corresponding method claim. Additionally, the rationale and motivation to combine the Yi in view of Vallespi-Gonzalez, Kakegawa, Stein, Pei, Jadhav, and Wu references, presented in rejection of Claim 5, apply to this claim. Finally, the combination of the Yi in view of Vallespi-Gonzalez, Kakegawa, Stein, Pei, Jadhav, and Wu references discloses an ego vehicle comprising a stereo camera for capturing pairs of images (for example, see Vallespi-Gonzalez, Claim 6 and Abstract).
Claim 6 is rejected under 35 U.S.C. 103 as being unpatentable over Yi in view of Vallespi-Gonzalez, Kakegawa, Stein, Pei, and Gao (US 20230256970 A1).
Regarding Claim 6, the combination of references of Yi in view of Vallespi-Gonzalez, Kakegawa, Stein, and Pei does not explicitly teach "The system of claim 1, the at least one processor further to generate the track points using at least one of a wheel angle of the ego machine or a tire angle of the ego machine";
In an analogous field of endeavor, Gao teaches "The system of claim 1, the at least one processor further to generate the track points using at least one of a wheel angle of the ego machine or a tire angle of the ego machine"; (Gao, Para. 148, teaches a total quantity of track points on the lane change track wherein a change amount between a steering wheel rotation angle corresponding to a track point on the lane change track and a steering wheel rotation angle corresponding to a given numbered track point, i.e., track points generated using a wheel/tire angle of the ego vehicle).
It would have been obvious to one having ordinary skill in the art before the effective filing date to modify the invention of Yi, Vallespi-Gonzalez, Kakegawa, Stein, and Pei by including the generation of track points based on a wheel angle of the vehicle taught by Gao. One of ordinary skill in the art would be motivated to combine the references since it improves the lane change planning (Gao, Para. 6, teaches the motivation of combination to be to improve properness of lane change track planning).
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date.
Claims 7, 9-10, and 17 are rejected under 35 U.S.C. 103 as being unpatentable over Yi in view of Vallespi-Gonzalez, Kakegawa, Stein, Pei, and Oh et al. (US 20150199585 A1).
Regarding Claim 7, the combination of references of Yi in view of Vallespi-Gonzalez, Kakegawa, Stein, and Pei teaches "
"
"
"determining a plane normal and a plane offset of the ego machine with respect to a ground plane"; (Stein, Para. 198, teaches the ground plane engine indicating the plane normal vector of the road and the distance to the plane based on pitch and roll data provided by the ego-motion engine and vehicle suspension information, i.e., determining a plane normal and a plane offset of the ego vehicle with respect to the ground plane);
"and determining the first track point heights using the homography matrix"; (Stein, Abstract and Paras. 35-36, 45, and 55, teaches obtaining a time-ordered sequence of images representative of a road surface wherein a sensor capturing an image determines height of a point above a plane representing the road surface and wherein modeling the road surface includes warping a previous image to the current image using the gamma value which is a ratio of height of a point by a distance from a sensor which enables a feature to be matched between images based on its distance and height above the road surface wherein warping an earlier image to largely match a later image is completed via homography and measuring remaining pixel motion may be used to model the environment, i.e., determine track point heights using homography).
The proposed combination as well as the motivation for combining the Yi, Vallespi-Gonzalez, Kakegawa, Stein, and Pei references presented in the rejection of Claim 1, applies to claim 7.
However, the combination of references of Yi in view of Vallespi-Gonzalez, Kakegawa, Stein, and Pei does not explicitly teach “The system of claim 1, the at least one processor further to determine the first track point heights by: constructing a region of interest (ROI) for each frame of the plurality of frames of the disparity image; sampling a plurality of points within the ROI; determining a homography matrix for the plurality of points”.
In an analogous field of endeavor, Oh teaches "The system of claim 1, the at least one processor further to determine the first track point heights by: constructing a region of interest (ROI) for each frame of the plurality of frames of the disparity image"; (Oh, Abstract and Para. 54, teaches a region divides configured to divide a first image and second image into regions, i.e., constructing a region of interest for each of the plurality of frames);
"sampling a plurality of points within the ROI"; (Oh, Abstract and FIG. 4A-K, teaches sampling a number of feature points for an image region corresponding to the number of regions, i.e., sampling a plurality of points within the ROI);
"determining a homography matrix for the plurality of points"; (Oh, Abstract and Para. 39, teaches estimating a homography on the basis of the extracted sampled feature points wherein the homography is a matrix indicating a correspondence relationship between the feature points of the first and second images, i.e., determining a homography matrix for the plurality of points).
It would have been obvious to one having ordinary skill in the art before the effective filing date to modify the invention of Yi, Vallespi-Gonzalez, Kakegawa, Stein, and Pei wherein frames are of the disparity image by including the use of a region for each of the frames of the image, sampling points within the region, and determining a homography matrix for the points taught by Oh. One of ordinary skill in the art would be motivated to combine the references since it improves image quality (Oh, Para. 41, teaches the motivation of combination to be to improve image quality on the first and second images).
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date.
Regarding Claim 9, the combination of references of Yi in view of Vallespi-Gonzalez, Kakegawa, Stein, Pei, and Oh teaches "The system of claim 7, wherein the plurality of points are sampled randomly within the ROI"; (Oh, Paras. 7 and 39, teaches estimating a homography by randomly sampling feature points in a specific region of a first and second image).
The proposed combination as well as the motivation for combining the Yi, Vallespi-Gonzalez, Kakegawa, Stein, Pei, and Oh references presented in the rejection of Claim 7, applies to claim 9. Thus, the system recited in claim 9 is met by Yi, Vallespi-Gonzalez, Kakegawa, Stein, Pei, and Oh.
Regarding Claim 10, the combination of references of Yi in view of Vallespi-Gonzalez, Kakegawa, Stein, Pei, and Oh teaches "The system of claim 7, wherein the first track point heights are determined using a residual flow, the plane offset, and the disparity image"; (Stein, Abstract and Paras. 35-36, 45, 55, 198, and 204 teaches obtaining a time-ordered sequence of images representative of a road surface wherein a sensor capturing an image determines height of a point above a plane representing the road surface and wherein modeling the road surface includes warping a previous image to the current image using the gamma value which is a ratio of height of a point by a distance from a sensor which enables a feature to be matched between images based on its distance and height above the road surface wherein warping an earlier image to largely match a later image is completed via homography and measuring remaining pixel motion or residual motion may be used to model the environment wherein ground plane information computes the plane normal vector of the road and distance to the plane and wherein a height map is produced representing vertical height form the plane of the road using the ground plane information, i.e., determine track point heights by modeling the road surface which includes residual flow, ground plane information comprising the plane offset, and the sequence of images).
The proposed combination as well as the motivation for combining the Yi, Vallespi-Gonzalez, Kakegawa, Stein, Pei, and Oh references presented in the rejection of Claim 7, applies to claim 10. Thus, the system recited in claim 10 is met by Yi in view of Vallespi-Gonzalez, Kakegawa, Stein, Pei, and Oh.
Claim 17 recites a system with elements corresponding to the steps recited in Claim 7. Therefore, the recited elements of this claim are mapped to the proposed combination in the same manner as the corresponding steps in its corresponding method claim. Additionally, the rationale and motivation to combine the Yi, Vallespi-Gonzalez, Kakegawa, Stein, Pei, and Oh references, presented in rejection of Claim 7, apply to this claim. Finally, the combination of the Yi, Vallespi-Gonzalez, Kakegawa, Stein, Pei, and Oh references discloses an ego vehicle comprising a stereo camera for capturing pairs of images (for example, see Vallespi-Gonzalez, Claim 6 and Abstract).
Claim 8 is rejected under 35 U.S.C. 103 as being unpatentable over Yi in view of Vallespi-Gonzalez, Kakegawa, Stein, Pei, Oh, and Gao.
Regarding Claim 8, the combination of references of Yi in view of Vallespi-Gonzalez, Kakegawa, Stein, Pei, Oh, and Gao teaches "The system of claim 7, wherein the ROI is generated at least one track defined using the track points"; (Gao, Para. 86, teaches performing sampling in a sampling interval of the lane change control parameter to obtain a plurality of groups of sampling points of the lane change control parameter, and planning the lane change track of the vehicle based on the plurality of groups of the sampling points, i.e., ROI being the groups of sampling points representing the lane change track).
The proposed combination as well as the motivation for combining the Yi, Vallespi-Gonzalez, Kakegawa, Stein, Pei, and Gao references presented in the rejection of Claim 6, applies to claim 8. Thus, the system recited in claim 8 is met by Yi in view of Vallespi-Gonzalez, Kakegawa, Stein, Pei, Oh, and Gao.
Claim 20 is rejected under 35 U.S.C. 103 as being unpatentable over Pei in view of Kakegawa and Stein.
Regarding Claim 20, the combination of references of Pei in view of Kakegawa and Stein teaches "A method, comprising: determining first track point heights of a ground surface for each frame of a plurality of frames of a disparity image based on a plane parallax algorithm"; (Pei, Abstract, teaches obtaining the left and right views of the same road scene, calculating the dense parallax image of the road scene based on the left and right views, intercepting the detecting area based on the obtained dense view disparity image, and calculating the height information of each discrete point in the area cloud point to the plane of the road surface equation, i.e., determine track point heights of a ground surface for a plurality of frames of a disparity image based on a plane parallax algorithm);
"determining first track point heights of a ground surface for each frame of a plurality of frames of a disparity image in disparity space"; (Kakegawa, Figures 4 and 5 and Section 5.1, teaches the road height profile was estimated by fitting a line using the least square method to the disparities of the segmented road surface in V-disparity space wherein the scene length is 100 frames and the evaluation is performed for each frame, i.e., determine track point heights of a ground surface being the road height profile for each frame of the plurality of frames of the disparity image in disparity space);
"the first track point heights including previous track point heights of the ground surface for each previous frame of the at least one previous frame of the plurality of frames of the disparity image and current track point heights of the ground surface for the current frame of the plurality of frames of the disparity image"; (Stein, Abstract and Paras. 45 and 55, teaches obtaining a time-ordered sequence of images representative of a road surface wherein a sensor capturing an image determines height of a point above a plane representing the road surface and wherein modeling the road surface includes warping a previous image to the current image using the gamma value which is a ratio of height of a point by a distance from a sensor which enables a feature to be matched between images based on its distance and height above the road surface, i.e., determine track point heights for the plurality of frames including track point heights for the previous frames and track point heights of the current frames);
"determining second track point heights by temporally fusing the current track point heights for the current frame and the previous track point heights for each previous frame of the at least one previous frame"; (Pei, Abstract and Claim 7, teaches calculating the area point cloud information based on the image information and the parallax information of the detection area, modelling the road plane, and calculating the height information of each discrete point in the area point cloud to the plane of the road surface wherein a height map is generated according to the area point cloud information and the height information and detection result is filtered and corrected by fusing the height map of the multi-frame continuous image to obtain the fusion result and wherein the fusing comprises acquiring time information of the multi-frame continuous image and calculating moving distance between two adject frames and updating the detection result of the two adjacent frames, i.e., determine second track point heights being the fusion result by fusing the track point heights of the current and previous frames being the fusion of the height map of the multi-frame continuous image comprising two adjacent frames);
"and adjust, based at least on the second track point heights, at least one of a suspension system or a speed of the ego machine"; (Pei, Abstract and Background, teaches detecting and confirming the road fluctuation condition of the front detection area and the area with the pit characteristic according to the fusion result wherein the vehicle may reduce the influence of the road to the vehicle through adjusting the suspension setting if the pothole condition in the road ahead is detected in real time, i.e., adjust suspension of the ego machine based on track point heights).
The proposed combination as well as the motivation for combining the Yi, Vallespi-Gonzalez, Kakegawa, Stein, and Pei references presented in the rejection of Claim 1, applies to claim 20. Thus, the method recited in claim 20 is met by Pei in view of Kakegawa and Stein.
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ANDREW STEVEN BUDISALICH whose telephone number is (703)756-5568. The examiner can normally be reached Monday - Friday 8:30am-5:00pm EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Amandeep Saini can be reached on (571) 272-3382. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx
for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ANDREW S BUDISALICH/Examiner, Art Unit 2662
/AMANDEEP SAINI/Supervisory Patent Examiner, Art Unit 2662
Prosecution Insights