DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 1, 6, 7, 11, 16, 17 and 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Popov (U.S. PG-PUB NO. 2025/0045952) in view of Lev (U.S. PG-PUB NO. 2024/0412412).
-Regarding claim 1, Popov discloses a method of performing feature matching to map an environment of a vehicle (generating three-dimensional (3D) map data using neural networks, [0022]), the method comprising: generating image-domain data based on image data received from a plurality of cameras mounted on the vehicle, wherein the image-domain data includes data corresponding to a plurality of images (image data 108 can be retrieved from sensors coupled with vehicles, [0034]; image data 108 can include a plurality of images from multiple poses, such as from multiple sensors coupled at different locations to a vehicle, [0035]; image data 108 includes camera image frames from a front camera (e.g., 120 degree field of view), side left, side right, rear left, and rear right (e.g., 200 degree field of view) cameras, [0036]); performing feature extraction on a first image of the plurality of images to identify and extract features from the first image (perform operations including identifying features (e.g., representations of objects or other structures in the environment) from image data 108, [0038]; featurizer 112 can process the image data 108 to determine a plurality of tokens representative of features of the image data 108, [0039]); providing the merged image to a vision transformer (models 104 can include one or more transformers, detection transformers (DETRs), [0033]); and using the vision transformer to (i) generate semantic descriptors for the merged image (identify similarities/correspondences amongst vectors of the latent representation, such as to associate features across vectors from different sensor images, [0059]).
Popov is silent to teaching that performing segmentation on the first image to identify contours around the extracted features; embedding one or more patches in the first image based on the identified contours to obtain a merged image; and (ii) perform feature matching to match, based on the semantic descriptors, the first image to a second image of the plurality of images. However, the claimed limitation is well known in the art as evidenced by Lev.
In the same field of endeavor, Lev teaches performing segmentation on the first image to identify contours around the extracted features (segmentation methods may detect edges to delineate boundaries and extract multiple patches accordingly, [0094]); embedding one or more patches in the first image based on the identified contours to obtain a merged image (generating a first plurality of patches by applying segmentation on a first image from the plurality of images, [0089]); and (ii) perform feature matching to match, based on the semantic descriptors, the first image to a second image of the plurality of images (generating a plurality of sets, each set comprising at least two patch from at least two different groups of patches by applying a geometric matching between the parameters characterizing each patch, [0107]).
Therefore, it would have been obvious to one of ordinary skills in the art before the effective filing date of the claimed invention to combine the teaching of Popov with the teaching of Lev in order to improve matching accuracy and reliability in low-texture, dynamic, or reflective environments.
-Regarding claim 6, the combination further discloses using a feature-detection machine learning model to perform semantic segmentation on the image data to extract the features from the first image (Lev, generating a first plurality of patches by applying segmentation on a first image from the plurality of images, [0089]; Fully Convolutional Network (FCN), [0096]).
-Regarding claim 7, the combination further discloses the performing feature extraction includes using at least one of scale invariant feature transform (SIFT) techniques and semantic segmentation techniques (Lev, SIFT, SURF, [0104]).
-Regarding claim 11, Popov discloses a system for performing feature matching to map an environment of a vehicle (FIG. 1, [0022]), the system comprising: a plurality of image sensors configured to be mounted to the vehicle and to generate image data (the image data 108 can include a plurality of images from multiple poses, such as from multiple sensors coupled at different locations to a vehicle, [0035]); one or more processors (processor, [0030]); and memory coupled to the one or more processors (memory, [0030]), the memory storing instructions that, when executed by the one or more processors, cause the one or more processors to: generate image-domain data based on image data received from the plurality of image sensors, wherein the image-domain data includes data corresponding to a plurality of images (image data 108 can be retrieved from sensors coupled with vehicles, [0034]; image data 108 can include a plurality of images from multiple poses, such as from multiple sensors coupled at different locations to a vehicle, [0035]; image data 108 includes camera image frames from a front camera (e.g., 120 degree field of view), side left, side right, rear left, and rear right (e.g., 200 degree field of view) cameras, [0036]); perform feature extraction on a first image of the plurality of images to identify and extract features from the first image (perform operations including identifying features (e.g., representations of objects or other structures in the environment) from image data 108, [0038]; featurizer 112 can process the image data 108 to determine a plurality of tokens representative of features of the image data 108, [0039]); provide the merged image to a vision transformer (models 104 can include one or more transformers, detection transformers (DETRs), [0033]); and use the vision transformer to (i) generate semantic descriptors for the merged image (identify similarities/correspondences amongst vectors of the latent representation, such as to associate features across vectors from different sensor images, [0059]).
Popov is silent to teaching that performing segmentation on the first image to identify contours around the extracted features; embed one or more patches in the first image based on the identified contours to obtain a merged image; and (ii) perform feature matching to match, based on the semantic descriptors, the first image to a second image of the plurality of images. However, the claimed limitation is well known in the art as evidenced by Lev.
In the same field of endeavor, Lev teaches performing segmentation on the first image to identify contours around the extracted features (segmentation methods may detect edges to delineate boundaries and extract multiple patches accordingly, [0094]); embed one or more patches in the first image based on the identified contours to obtain a merged image (generating a first plurality of patches by applying segmentation on a first image from the plurality of images, [0089]); and (ii) perform feature matching to match, based on the semantic descriptors, the first image to a second image of the plurality of images (generating a plurality of sets, each set comprising at least two patch from at least two different groups of patches by applying a geometric matching between the parameters characterizing each patch, [0107]).
Therefore, it would have been obvious to one of ordinary skills in the art before the effective filing date of the claimed invention to combine the teaching of Popov with the teaching of Lev in order to improve matching accuracy and reliability in low-texture, dynamic, or reflective environments.
-Regarding claim 16, the combination further discloses the instructions further cause the one or more processors to use a feature-detection machine learning model to perform semantic segmentationon the image data to extract the features from the first image (Lev, generating a first plurality of patches by applying segmentation on a first image from the plurality of images, [0089]; Fully Convolutional Network (FCN), [0096]).
-Regarding claim 17, the combination further discloses the perform feature extraction further includes using at least one of scale invariant feature transform (SIFT) techniques and semantic segmentation techniques (Lev, SIFT, SURF, [0104]).
-Regarding claim 20, Popov discloses a non-transitory computer-readable storage medium storing one or more programs (memory, [0030]), the one or more programs comprising instructions which, when executed by one or more processors of an electronic device (processor, [0030]), cause the electronic device to perform feature matching to map an environment of a vehicle (generating three-dimensional (3D) map data using neural networks, [0022]) by: generating image-domain data based on image data received from a plurality of cameras mounted on the vehicle, wherein the image-domain data includes data corresponding to a plurality of images (image data 108 can be retrieved from sensors coupled with vehicles, [0034]; image data 108 can include a plurality of images from multiple poses, such as from multiple sensors coupled at different locations to a vehicle, [0035]; image data 108 includes camera image frames from a front camera (e.g., 120 degree field of view), side left, side right, rear left, and rear right (e.g., 200 degree field of view) cameras, [0036]); performing feature extraction on a first image of the plurality of images to identify and extract features from the first image (perform operations including identifying features (e.g., representations of objects or other structures in the environment) from image data 108, [0038]; featurizer 112 can process the image data 108 to determine a plurality of tokens representative of features of the image data 108, [0039]); providing the merged image to a vision transformer (models 104 can include one or more transformers, detection transformers (DETRs), [0033]); and using the vision transformer to (i) generate semantic descriptors for the merged image (identify similarities/correspondences amongst vectors of the latent representation, such as to associate features across vectors from different sensor images, [0059]).
Popov is silent to teaching that performing segmentation on the first image to identify contours around the extracted features; embedding one or more patches in the first image based on the identified contours to obtain a merged image; and (ii) perform feature matching to match, based on the semantic descriptors, the first image to a second image of the plurality of images. However, the claimed limitation is well known in the art as evidenced by Lev.
In the same field of endeavor, Lev teaches performing segmentation on the first image to identify contours around the extracted features (segmentation methods may detect edges to delineate boundaries and extract multiple patches accordingly, [0094]); embedding one or more patches in the first image based on the identified contours to obtain a merged image (generating a first plurality of patches by applying segmentation on a first image from the plurality of images, [0089]); and (ii) perform feature matching to match, based on the semantic descriptors, the first image to a second image of the plurality of images (generating a plurality of sets, each set comprising at least two patch from at least two different groups of patches by applying a geometric matching between the parameters characterizing each patch, [0107]).
Therefore, it would have been obvious to one of ordinary skills in the art before the effective filing date of the claimed invention to combine the teaching of Popov with the teaching of Lev in order to improve matching accuracy and reliability in low-texture, dynamic, or reflective environments.
Claim(s) 2-5 and 12-15 is/are rejected under 35 U.S.C. 103 as being unpatentable over Popov (U.S. PG-PUB NO. 2025/0045952) in view of Lev (U.S. PG-PUB NO. 2024/0412412) and further in view of Wen (U.S. PG-PUB NO. 2025/0218117).
-Regarding claim 2, the combination is silent to teaching that image data is associated with a parking zone outside the vehicle, and wherein the image-domain data is generated by a feature- detection machine learning model. However, the claimed limitation is well known in the art as evidenced by Wen.
In the same field of endeavor, Wen teaches image data is associated with a parking zone outside the vehicle, and wherein the image-domain data is generated by a feature-detection machine learning model (after the vehicle enters the garage, a 3D point cloud map of the garage that the vehicle passes through is constructed using the sequence of image frames captured by the camera during the vehicle's movement, [0030]; feature point recognition model is a neural network model, [0040]).
Therefore, it would have been obvious to one of ordinary skills in the art before the effective filing date of the claimed invention to combine the teaching of the combination with the teaching of Wen in order to apply the feature-matching pipeline to a common vehicle parking use-case (obvious application of a known technique [KSR]).
-Regarding claim 3, the combination further discloses generating a bird's-eye-view (BEV) image based on results of the feature matching, wherein the BEV image is a projected image of a parking zone (Popov, output 160 can have a top-down/BEV perspective, extending in X and Y directions, [0054]; Wen, the vehicle's position in the 3D point cloud map is determined based on the locations of the map points corresponding to the matched feature points, [0063]).
-Regarding claim 4, the combination further discloses generating BEV-domain data associated with the BEV image, wherein the BEV-domain data includes data associated with parking landmarks in the parking zone; localizing the vehicle within the parking zone based on the BEV-domain data and the image-domain data to generate localization data; and mapping the parking zone based on the BEV-domain data, the image-domain data, and the localization data (Popov, output 160 can have a top-down/BEV perspective, extending in X and Y directions, [0054]; Wen, the vehicle's position in the 3D point cloud map is determined based on the locations of the map points corresponding to the matched feature points, [0063]; A pose determination unit, configured to determine the first pose of the vehicle corresponding to the first image frame and the second pose of the vehicle corresponding to the second image frame; and a second map creation unit, configured to create a 3D point cloud map based on the first pose, the second pose, and a set of matched feature points, [0068]).
-Regarding claim 5, the combination further discloses the localizing and the mapping are performed by a simultaneous localization and mapping (SLAM) system (Lev, SLAM, [0046]).
-Regarding claim 12, the combination is silent to teaching that the image data is associated with a parking zone outside the vehicle, and wherein the image-domain data is generated by a feature- detection machine learning model. However, the claimed limitation is well known in the art as evidenced by Wen.
In the same field of endeavor, Wen teaches the image data is associated with a parking zone outside the vehicle, and wherein the image-domain data is generated by a feature- detection machine learning model (after the vehicle enters the garage, a 3D point cloud map of the garage that the vehicle passes through is constructed using the sequence of image frames captured by the camera during the vehicle's movement, [0030]; feature point recognition model is a neural network model, [0040]).
Therefore, it would have been obvious to one of ordinary skills in the art before the effective filing date of the claimed invention to combine the teaching of the combination with the teaching of Wen in order to apply the feature-matching pipeline to a common vehicle parking use-case (obvious application of a known technique [KSR]).
-Regarding claim 13, the combination further discloses the instructions further cause the one or more processors to generate a bird's-eye-view (BEV) image based on results of the feature matching, wherein the BEV image is a projected image of a parking zone (Popov, output 160 can have a top-down/BEV perspective, extending in X and Y directions, [0054]; Wen, the vehicle's position in the 3D point cloud map is determined based on the locations of the map points corresponding to the matched feature points, [0063]).
-Regarding claim 14, the combination further discloses the instructions further cause the one or more processors to: generate BEV-domain data associated with the BEV image, wherein the BEV- domain data includes data associated with parking landmarks in the parking zone; localize the vehicle within the parking zone based on the BEV-domain data and the image-domain data to generate localization data; and map the parking zone based on the BEV-domain data, the image-domain data, and the localization data (Popov, output 160 can have a top-down/BEV perspective, extending in X and Y directions, [0054]; Wen, the vehicle's position in the 3D point cloud map is determined based on the locations of the map points corresponding to the matched feature points, [0063]; A pose determination unit, configured to determine the first pose of the vehicle corresponding to the first image frame and the second pose of the vehicle corresponding to the second image frame; and a second map creation unit, configured to create a 3D point cloud map based on the first pose, the second pose, and a set of matched feature points, [0068]).
-Regarding claim 15, the combination further discloses the localize and the map are performed by a simultaneous localization and mapping (SLAM) system (Lev, SLAM, [0046]).
Claim(s) 8, 9, 18 and 19 is/are rejected under 35 U.S.C. 103 as being unpatentable over Popov (U.S. PG-PUB NO. 2025/0045952) in view of Lev (U.S. PG-PUB NO. 2024/0412412) and further in view of Zhang (U.S. PG-PUB NO. 2018/0225866).
-Regarding claim 8, the combination is silent to teaching that the embedding one or more patches includes overlaying a selected first color onto a region including one or more of the extracted features. However, the claimed limitation is well known in the art as evidenced by Zhang.
In the same field of endeavor, Zhang teaches the embedding one or more patches includes overlaying a selected first color onto a region including one or more of the extracted features (Different depths (or ranges of depth) can be displayed on the screen of the computing device, for example, as different colors).
Therefore, it would have been obvious to one of ordinary skills in the art before the effective filing date of the claimed invention to combine the teaching of the combination with the teaching of Zhang in order to improve the depth resolution of the disparity map.
-Regarding claim 9, the combination further discloses obtaining a first point cloud from a first camera of the plurality of cameras; obtaining a second point cloud from a second camera of the plurality of cameras; obtaining transformation characteristics defining a relationship between the first camera and the second camera; and merging the first point cloud with the second point cloud based on the transformation characteristics (Zhang, an optoelectronic module 112 that includes two (or more) depth channels, each of which has a respective image capture device (e.g., camera) 114A, 114B, [0018]; the memory 144 can store two or more 3D color point clouds 146A, 146B generated by the 3D point cloud reconstruction engine 138, [0022]; registration matrix can be obtained from the corresponding 3D key points. The process 216 applies a registration transformation to each 3D point cloud so as to obtain a merged 3D point cloud, [0060]).
-Regarding claim 18, the combination further discloses the embed one or more patches includes overlaying a selected first color onto a region including one or more of the extracted features (Zhang, Different depths (or ranges of depth) can be displayed on the screen of the computing device, for example, as different colors).
-Regarding claim 19, the combination further discloses the instructions further cause the one or more processors to: obtain a first point cloud from a first image sensor of the plurality of image sensors; obtain a second point cloud from a second image sensor of the plurality of image sensors; obtain transformation characteristics defining a relationship between the first image sensor and the second image sensor; and merge the first point cloud with the second point cloud based on the transformation characteristics (Zhang, an optoelectronic module 112 that includes two (or more) depth channels, each of which has a respective image capture device (e.g., camera) 114A, 114B, [0018]; the memory 144 can store two or more 3D color point clouds 146A, 146B generated by the 3D point cloud reconstruction engine 138, [0022]; registration matrix can be obtained from the corresponding 3D key points. The process 216 applies a registration transformation to each 3D point cloud so as to obtain a merged 3D point cloud, [0060]).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to PING Y HSIEH whose telephone number is (571)270-3011. The examiner can normally be reached Monday-Friday, 9am-4pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Jennifer Mehmood can be reached at (571) 272-2976. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/PING Y HSIEH/ Primary Examiner, Art Unit 2664
Prosecution Insights