DETAILED ACTION
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-2, 4-7, 10, 13, 15 and 17-20 are rejected under 35 U.S.C. 103 as being unpatentable over Park (US 12,646,292) in view of Aalamifar (US 12,033,316).
Referring to claims 1 and 18-19, Park discloses a computer implemented method (fig. 1, process 100), comprising:
generating a modal image (fig. 1, sensor data 120 from sensors) corresponding to an input first-person view FPV image (fig. 1, image 130) of a scene (fig. 1, view 132), wherein the modal image is representative of a feature (5:25-29, such as views, environments) of the FPV image;
extracting,
from the FPV image, a first set of feature maps (fig. 1, image features 122) with a first machine-learning model (fig. 1, image encoder 102; 7:40-42, machine learning models), and
from the modal image, a second set of feature maps (fig. 1, transformed features 124) with a second machine-learning model (fig. 1, perspective transformer 104; 9:16-20, MLMs);
concatenating (fig. 1, feature fuser 106) the first set of feature maps with the second set of feature maps to generate a set of tensors (fig. 1, fused features 126);
generating a set of bird-eye view BEV feature maps (fig. 2B, bird eye view 200B; fig. 5, BEV B502) in a BEV plane (fig. 2B, BEV plane 210) with that maps the set of tensors to the set of BEV feature maps based on a correspondence between a set of polar coordinates (figs. 2A/2B, polar coordinates; 9:36-51) associated with the BEV plane and a set of cartesian coordinates (figs. 2A/2B, cartesian grid; 10:46-64) associated with the FPV image; and
decoding the set of BEV feature maps (fig. 2A, transformed features 224) with to generate a BEV map (fig. 2A, image plane 240) with the feature projected thereon for output to an output device (fig. 1, output data 128).
Aalamifar discloses the second machine-learning model is a neural network that was trained using data with a using a 3D mesh (33:22-26, neural network machine learning model trained to use a 3D mesh) of the scene.
Park and Aalamifar are analogous art because they are from the same field of endeavor in neural network and machine learning. Before the time of the filing, it would have been obvious to a person of ordinary skill in the art, having the teaching of Park and Aalamifar before him or her to modify the machine learning model training of Park to include the 3D mesh data of Aalamifar, thereafter the machine learning model is trained with 3D mesh data. The suggestion and/or motivation for doing so would be obtaining the advantage of more real and synthetic data for training neural network during machine learning (32:26-56) as suggested by Aalamifar. Therefore, it would have been obvious to combine Park with Aalamifar to obtain the invention as specified in the instant application claims.
As to claims 2 and 20, Park discloses the method of claim 1, wherein the second machine-learning model is trained with a data (fig. 1, sensor calibrations 140) that is not representative of the feature of the modal image.
As to claim 4, Park discloses the method of claim 1, wherein the FPV image of the scene is received from a sensing unit (fig. 1, sensor data 120 input) and the decoding outputs to the output device that is a display (fig. 1, output data 128 output).
As to claim 5, Park discloses the method of claim 1, comprising processing the BEV feature maps with a third machine-learning model (fig. 1, feature fuser 106; 5:34-61) for stacking the set of BEV feature maps before decoding the set of BEV feature maps with a fourth machine-learning model (fig. 1, MLM 108).
As to claims 6-7, Park discloses the method of claim 5, the method of claim 5, wherein the first, second, and third machine-learning models are a first, second, and third neural network, respectively (fig. 1; 1:59-63, MLMs).
As to claim 10, Park discloses the method of claim 7, wherein the second neural network is trained using a training data of a modality (10:32, such as using lookup table for efficient training).
As to claim 13, Park discloses the method of claim 6, wherein the third neural network is a transformer-based network (fig. 1, transformed features 124) to computer attention metric (11:46-47, attention modeling).
As to claim 15, Park discloses the method of claim 11, comprising: generating a residual BEV feature map (fig. 5, BEV feature maps B504/B506) for the FPV image and the modal image, wherein a fourth machine-learning model (fig. 1, MLM 108) process the residual BEV feature map and the BEV feature maps (fig. 5, perform operation using the BEV feature maps B508).
As to claim 17, Park discloses the method of claim 1, wherein the feature of the modal image is a modality of the FPV image that includes semantic segmentation (11:29-33, semantic segmentation).
Allowable Subject Matter
Claims 3, 9, 11-12, 14, 16 and 21-22 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
The following is a statement of reasons for the indication of allowable subject matter: The claim limitation of “a set of data triplets …” in claim 3, “the second neural network is trained using a synthetic image pattern superimposed on a 3d mesh of the scene” in claim 9, “each tensor generating a contextualized feature column encoder … transforming the contextualized feature column to a BEV ray map …” in claim 14, and “determining a monocular depth value of the FPV … generating a perspective projection of the FPV image … pooling the perspective projection … passing the single channel tensor …” in claim 16.
Conclusion
Applicant’s amendment necessitated the new grounds of rejection presented in this Office action. Accordingly, this action is made final. See MPEP §706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a). A shortened statutory period for reply to this final action is set to expire in THREE MONTHS from the mailing date of this action. In the event a first reply is filled within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date of the advisory action is mailed, and any extension fee pursuant to 37 CFR 1.136 (a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than six months from the date of this final action.
Contact Information
Any inquiry concerning this communication or earlier communications from the examiner should be directed to examiner Cheng-Yuan Tseng whose telephone number is (571)272-9772, and fax number is (571)273-9772. The examiner can normally be reached on Monday through Friday from 09:00 to 17:30 Eastern Time. If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Alicia Harrington can be reached on (571)272-2330. The fax phone number for the organization where this application or proceeding is assigned is (571)273-8300. Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at (866)217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call (800)786-9199 (IN USA OR CANADA) or (571)272-1000.
/CHENG YUAN TSENG/Primary Examiner, Art Unit 2615
Prosecution Insights