DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Status
Claims 1-33 are pending.
Information Disclosure Statement
The IDS filed 09/11/24 has been considered.
Specification
Applicant is reminded of the proper content of an abstract of the disclosure.
A patent abstract is a concise statement of the technical disclosure of the patent and should include that which is new in the art to which the invention pertains. The abstract should not refer to purported merits or speculative applications of the invention and should not compare the invention with the prior art.
If the patent is of a basic nature, the entire technical disclosure may be new in the art, and the abstract should be directed to the entire disclosure. If the patent is in the nature of an improvement in an old apparatus, process, product, or composition, the abstract should include the technical disclosure of the improvement. The abstract should also mention by way of example any preferred modifications or alternatives.
Where applicable, the abstract should include the following: (1) if a machine or apparatus, its organization and operation; (2) if an article, its method of making; (3) if a chemical compound, its identity and use; (4) if a mixture, its ingredients; (5) if a process, the steps.
Extensive mechanical and design details of an apparatus should not be included in the abstract. The abstract should be in narrative form and generally limited to a single paragraph within the range of 50 to 150 words in length.
See MPEP § 608.01(b) for guidelines for the preparation of patent abstracts.
The abstract is currently over 150 words in length, shorted to correct.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claims 1-2, 6-9, 11-13, 17-20, 22-24, 28-31 and 33 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Dana et al. (“One scalar is all you need- absolute depth estimation using monocular self-supervision” Hereinafter “Dana”).
Regarding claim 1, Dana teaches method for training a monocular depth estimation (MDE) network, the method comprising:
obtaining a source dataset comprising a first source image and a first ground truth depth map corresponding to the first source image (Fig. 2, page 3: Source dataset is obtained and a source ground truth depth (GT depth) is obtained which corresponds to the Source dataset. There must be a first source image that is used which the GT corresponds to);
obtaining a target dataset comprising a first target image and a second target image (Fig. 2, page 3: Target image dataset is obtained, this includes a first and second target image acquired at t and t + 1, “Training an MDE using self-supervision aims at reconstructing the view from an image collected at time t ± 1 from an image collected at time t, using the estimated depth from the image at time t and the estimated relative pose between the frames[50,18]” (Section 3.1));
generating an estimated first source depth map corresponding to the first source image using the MDE network (Fig. 2, page 3: MDE network can be seen generating a Source predicted depth which corresponds to the first source image and the Source ground truth (see similar vehicle detections between the predicted map and ground truth depth in box C in the figure));
generating an estimated target depth map corresponding to the first target image using the MDE network (Fig. 2, page 3: The predicted up-to-scale depth map can be seen generated from the MDE network for the target images (bottom arrow). This prediction map must correspond to a first target image of the target dataset);
generating a first estimated relative pose based on the first target image and the second target image using a pose network (Fig. 2, page 3, section 3.1: “Training an MDE using self-supervision aims at reconstructing the view from an image collected at time t ± 1 from an image collected at time t, using the estimated depth from the image at time t and the estimated relative pose between the frames[50,18]”. The pose network generates the relative pose between frames using the first and second target images (to obtain the relative pose between frames, both frames must be input into the pose network, these frames are the first and second target images); and
training the MDE network and the pose network by performing mixed supervision training (As cited below, both fully-supervised and Self-supervised training is used to train the network, so it is mixed supervision training), wherein the performing the mixed supervision training comprises performing fully-supervised training based on the estimated first source depth map and the first ground truth depth map (Page 5, section 3.6: “The network was trained for 15 epochs in a fully-supervised manner using L1 loss between the predicted depthipredabs (see Eq. (8)) and the absolute depth depthiGT values, with a learning rate of 10−4)”. The network is trained in a fully supervised manner using the first source depth map (depthipredabs) and the first ground truth depth map (depthiGT)), and performing self-supervised training based on the estimated target depth map and the first estimated relative pose (Page 3, section 3.1: “Training an MDE using self-supervision aims at reconstructing the view from an image collected at time t ± 1 from an image collected at time t, using the estimated depth from the image at time t and the estimated relative pose between the frames [50,18]. Let us denote by Φ a network that receives as input an image It and by dt its estimated up-to-scale depth output, and by Tt→t±1 the relative pose between two frames It and It±1 estimated by a network Ψ”. The MDE network is trained in a self-supervised manner using the estimated depth at time t and the estimated relative pose).
Regarding claim 2, Dana teaches the method of claim 1, wherein the first estimated relative pose is generated by providing the first target image and the second target image to the pose network (Fig. 2, page 3, section 3.1: “Training an MDE using self-supervision aims at reconstructing the view from an image collected at time t ± 1 from an image collected at time t, using the estimated depth from the image at time t and the estimated relative pose between the frames[50,18]”. The pose network generates the relative pose between frames using the first and second target images (to obtain the relative pose between frames, both frames must be input into the pose network, these frames are the first and second target images)).
Regarding claim 6, Dana teaches the method of claim 1, wherein the source dataset comprises a plurality of source images in a source domain (Fig. 2, page 3: The source dataset can be seen with a plurality of images), and wherein the target dataset comprises a plurality of target images in a target domain (Fig. 2, page 3: The target dataset can be seen with a plurality of images), wherein the plurality of source images are captured using a first sensor (Fig. 2, page 3: A first sensor must capture the source dataset images), and wherein the plurality of target images are captured using a second sensor different from the first sensor (Fig. 1, page 1: A second sensor must capture the target dataset images, the target and source data come from different datasets, the different datasets have different cameras with different FOV’s (Page 5, section 3.8: “KITTI. This dataset [15] is considered the standard benchmark for depth evaluation. The front cameras have a FOVof81°and are located 1.65 m above the ground.
DDAD. This benchmark [21] was collected using six cameras. The front and rear cameras have a FOV of 47°/83° and are located 1.55 m above the ground)).
Regarding claim 7, Dana teaches the method of claim 6, wherein a field of view (FOV) of the plurality of source images is different from a FOV of the plurality of target images (Page 5, section 3.8: “KITTI. This dataset [15] is considered the standard benchmark for depth evaluation. The front cameras have a FOVof81°and are located 1.65 m above the ground.
DDAD. This benchmark [21] was collected using six cameras. The front and rear cameras have a FOV of 47°/83° and are located 1.55 m above the ground)), and
wherein the method further comprises performing FOV conversion on the plurality of source images to generate a plurality of converted source images such that a FOV of the plurality of converted source images matches the FOV of the plurality of target images (Fig. 2, page 3, section 3.2: “To enable self-supervised training on mixed batches of images collected with sensors of different FOVs without breaking the scene geometrical consistency, we adjusted the FOV of the source images to the FOV of the target images, resulting in training data with homogeneous FOV and aspect ratio”).
Regarding claim 8, Dana teaches the method of claim 7, wherein the mixed supervision training comprises training the MDE network to predict depth properties of the target domain based on depth properties of the source domain ((Page 5, section 3.6: “The network was trained for 15 epochs in a fully-supervised manner using L1 loss between the predicted depthipredabs (see Eq. (8)) and the absolute depth depthiGT values, with a learning rate of 10−4)”. The network is trained in a fully supervised manner using the first source depth map (depthipredabs) and the first ground truth depth map (depthiGT)), (Page 3, section 3.1: “Training an MDE using self-supervision aims at reconstructing the view from an image collected at time t ± 1 from an image collected at time t, using the estimated depth from the image at time t and the estimated relative pose between the frames [50,18]. Let us denote by Φ a network that receives as input an image It and by dt its estimated up-to-scale depth output, and by Tt→t±1 the relative pose between two frames It and It±1 estimated by a network Ψ”. The MDE network is trained in a self-supervised manner using the estimated depth at time t and the estimated relative pose), By training the network on both supervised (for the source dataset) and unsupervised (for the target dataset), depth properties of the target domain can be estimated from the MDE using depth properties of the source domain (using the supervised training)).
Regarding claim 9, Dana teaches the method of claim 8, wherein after the mixed supervision training is performed, the method further comprises generating an absolute depth prediction on an input image included in the target domain using the MDE network (Fig. 2, page 3: The predicted absolute depth prediction can be seen as the final output in the bottom fight of Fig. 2. This absolute depth correlates to the target image put into the MDE network to generate the up-to-scale depth prediction, which is processed to generate the absolute depth prediction. The MDE network is used to generate the absolute depth prediction by being part of this process).
Regarding claim 11, Dana teaches the method of claim 1, wherein the first source image is a synthetic image, and wherein the first ground truth depth map is a synthetic depth map (Page 3, section 3.2: “Training an MDE on images collected using different camera FOVs (real or synthetic) introduces significant geometrical differences that a naive self-supervised training regime cannot easily compensate for”. The images can be real or synthetic. If the image is synthetic, the ground truth depth map would also be synthetic).
Regarding claim 12, the content of claim 12 is similar to the content of claim 1, with the additional teachings of a obtaining an input image and generating an estimated depth map by providing the input image to an MDE network. Dana also discloses this information (Page 5, first paragraph: “We used the source test dataset to select the epoch with the lowest AbsRel (see Section 3.9) and reported the selected model accuracy on the target test dataset”. The target test dataset is used on the trained model, so the target test image will act as the input image. As seen in Fig. 2, the estimated depth map will be generated for the input image using the trained MDE). Therefore, claim 12 is rejected for the same reasons of anticipation as claim 1, along with the additional teachings above.
Regarding claim 23, the content of claim 23 is similar to the content of claim 1, with the additional teachings of a device for training an MDE. Dana also discloses this information (Page 4, section 3.5: “In this work we adopted the Φ, Ψ architectures presented in Monodepth2[18], but replaced the backbone of the depth network Φ with MobileNetV2 [39] instead of the original ResNet18 [28]. The MDE was trained using a phototometric loss [18] for 15 epochs, with a learning rate of 10−4, and for another five epochs with a learning rate of 10−5.”. The devices for training the MDE are described in this section. Epochs mean full passes of training data, so a device has to be training the MDE). Therefore, claim 23 is rejected for the same reasons of anticipation as claim 1, along with the additional teachings above.
Regarding claims 13 and 24, the content of claims 13 and 24 is similar to the content of claim 2, therefore it is rejected for the same reasons of anticipation as claim 2.
Regarding claims 17 and 28, the content of claims 17 and 28 is similar to the content of claim 6, therefore it is rejected for the same reasons of anticipation as claim 6.
Regarding claims 18 and 29, the content of claims 18 and 29 is similar to the content of claim 2, therefore it is rejected for the same reasons of anticipation as claim 7.
Regarding claims 19 and 30, the content of claims 19 and 30 is similar to the content of claim 2, therefore it is rejected for the same reasons of anticipation as claim 8.
Regarding claims 20 and 31, the content of claims 20 and 31 is similar to the content of claim 9, therefore it is rejected for the same reasons of anticipation as claim 9.
Regarding claims 22 and 33, the content of claims 22 and 33 is similar to the content of claim 11, therefore it is rejected for the same reasons of anticipation as claim 11.
Claims 10, 21, and 32 are rejected under 35 U.S.C. 102(a)(1) as being unpatentable over Dana et al. (“One scalar is all you need- absolute depth estimation using monocular self-supervision” Hereinafter “Dana”) as evidenced by Guizilini et al. (“3D Packing for Self-Supervised Monocular Depth Estimation” Hereinafter “Guizilini”). Guizillini is relied on only to show what “DDAD” in Dana would be understood to mean in the art and is not proposed as a modification of Dana.
Regarding claim 10, Dana teaches the method of claim 1, wherein the first ground truth depth map is obtained using at least one from among a light detection and ranging (LiDAR)(Page 1, Fig. 1: Dana uses the DDAD dataset for their ground truth depth maps. The DDAD dataset uses LiDAR to obtain the ground truth depth maps “It contains 17,050 training and 4,150 evaluation frames with ground-truth depth maps generated from dense LiDAR measurements using the Luminar-H2 sensor” (Page 2489, section DADD, Guizilini)).
Regarding claims 21 and 32, the content of claims 21 and 32 is similar to the content of claim 10, therefore it is rejected for the same reasons of anticipation as claim 10.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 3, 14, and 25 are rejected under 35 U.S.C. 103 as being unpatentable over Dana et al. (“One scalar is all you need- absolute depth estimation using monocular self-supervision” Hereinafter “Dana”) in view of Guizilini2 et al. (US 11257231 B2 Hereinafter “Guizilini2”).
Regarding claim 3, Dana teaches the method of claim 1, wherein the training further comprises:
Dana does not expressly disclose generating a projected image corresponding to the first target image based on the estimated target depth map and the first estimated relative pose; and performing the self-supervised training based on the first target image and the projected image.
However, Guizilini2 teaches generating a projected image corresponding to the first target image based on the estimated target depth map and the first estimated relative pose (Fig.5, Col. 11, Lines 35-45: “An output of the pose network 520 feeds an image projection 522 into a camera agnostic configuration. The image projection 522 is fed as an input to a view synthesis block 550. This image projection 522 enables training of monocular depth and pose models using much larger datasets by leveraging information from any camera. The context image 504 is also fed as an input to the view synthesis block 550”. The warped target image combines the projected image and the target depth map (532) as seen in Fig. 5. Therefore, the warped target image is the projected image which corresponds to the first target image which is based on the target depth map and the relative pose output by the pose network); and performing the self-supervised training based on the first target image and the projected image (Col. 11-12, Lines 65-5: “In this configuration, the monocular depth/pose estimation framework 500 performs self-supervised learning to predict the warped target image 552 (Î.sub.t) by the view synthesis block 550. The warped target image 552 (Î.sub.t) is provided as feedback to a photometric loss block 508, which also receives the target image (I.sub.t) 502 as input. The monocular depth/pose estimation framework 500 is described in further detail below.”. The warped target image is generated using the first target image and depth map, so self-supervised training is performed based on the first target image and depth map).
At the time the invention was made, it would have been obvious to one of ordinary skill in the art to modify Dana’s mixed supervision training method to include Guizilini2’s generation and use of a projected image for training a self-supervision network because such a modification is taught, suggested, or motivated by the art. More specifically, the motivation to modify Dana to include Guizilini2 is implicitly provided by Guizilini2, stating that generating projected images allows for much larger datasets for training the monocular depth and pose networks (Col. 11, lines 40-45: “This image projection 522 enables training of monocular depth and pose models using much larger datasets by leveraging information from any camera”). Therefore, it would have been obvious to one of ordinary skill in the art at the time of the invention to modify Dana’s mixed supervision training method to include Guizilini2’s generation and use of a projected image for training a self-supervision network with the motivation of leveraging larger datasets. The person of ordinary skill in the art would have recognized the benefit of improved training data by leveraging larger datasets.
Allowable Subject Matter
Claim 4-5, 15-16, 26-27 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure:
Guizilini et al. (US 20210004976 A1) mixed supervision training for monocular depth estimation.
Talker et al. (US 20240169567 A1) mixed supervision training for monocular depth estimation.
AMBRUS et al. (US 20230177849 A1) mixed supervision training for monocular depth estimation.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to STEFANO A DARDANO whose telephone number is (703)756-4543. The examiner can normally be reached Monday - Friday 11:00 - 7:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Greg Morse can be reached at (571) 272-3838. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/STEFANO ANTHONY DARDANO/ Examiner, Art Unit 2663
/GREGORY A MORSE/Supervisory Patent Examiner, Art Unit 2698
Prosecution Insights