DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
Applicant's Amendments filed on 09/16/2026 have been entered and made of record.
Status of Claims
Currently pending Claim(s):
New Claim(s):
1-23
22-23
Response to Arguments
This office action is responsive to Applicant's Arguments/Remarks made in an Amendment received on 09/16/2026.
In view of applicant Argument/Remarks filed 09/16/2026 with respect to claims 1-21, U.S.C. 112(f) claim interpretation has been carefully considered and the claim interpretation to claims 1-21 under 35 U.S.C. 112(f) is withdrawn.
In view of applicant's arguments, Remarks filed on 09/16/2026, with respect to independent claims 1, 6, and 21 under 35 U.S.C. 103, the arguments have been fully considered and are found to be not persuasive. The Applicant argues on pages 11-12 of the Remarks:
PNG
media_image1.png
158
716
media_image1.png
Greyscale
PNG
media_image2.png
626
692
media_image2.png
Greyscale
The Examiner respectfully disagrees. The pending claim does not recite that the encoder module must be the only encoder in the model or that additional encoders are excluded. In response to applicant's argument that the references fail to show certain features of the invention, it is noted that the features upon which applicant relies (i.e., “that a single encoder produces both depth features and the pose features simultaneously.”) are not recited in the rejected claim(s). Although the claims are interpreted in light of the specification, limitations from the specification are not read into the claims. See In re Van Geuns, 988 F.2d 1181, 26 USPQ2d 1057 (Fed. Cir. 1993).
Further, Dong teaches on Page 3 right column Subsection 3.3 first 2 paragraphs “We opt for RepLKNet (Ding et al. 2022), a CNN architecture featuring a notable kernel size of 31 × 31, as the encoder backbone”. Dong identifies RepLKNet as the encoder backbone, as seen in Figure 2 as well. Although Dong references ManyDepths (Ding et al.) in their architecture, this does not mean that Dong falls short in teaching that RepLKNet is the encoder backbone and is used to generate the feature maps (i.e. the first and second representations). Further, the rejection does not rely on Dong alone to teach all the claim limitations, the rejection under 35 USC 103 also relies on Godard to teach a [pose translation of a camera that captured the video based on the] first and second representations, as explained below in the rejection of the independent claims.
The Applicant argues at the end of page 12 to page 13:
PNG
media_image3.png
330
856
media_image3.png
Greyscale
PNG
media_image4.png
400
848
media_image4.png
Greyscale
In response to Applicants statement that “This is impermissible parsing of claim 1 and illustrates the failure of the Examiner to consider the claims as a whole”, the Examiner has revised the rejection of claims 1, 6 and 21 to recite the full claim language in place of abbreviated phrases used in the previous Office Action dated 6/16/2026. This change below in the rejection does not change the basis of the rejection previously made but only identifies the claim language omitted and stated by using ‘…’ as a continuation of a limitation. The rejection considers the limitations in the context of the claim as a whole and relies on the combined teachings of Dong in view of Godard, same as previously stated.
With respect to the Applicant’s remarks regarding Dong is therefore silent as to the feature of claim 1 of: “an encoder module configured to encode first and second images into first and second representations”, the Examiner respectfully disagrees. Dong teaches on Page 3 right column Subsection 3.3 first 2 paragraphs “We opt for RepLKNet (Ding et al. 2022)…as the encoder backbone…As illustrated in Fig. 2(d), RepLKNet generates feature maps of four different scales: F 1, F 2, F 3, F 4 at four stages.”. The generated feature maps, F 1 and F 2 , for example, correspond to the first and second representations, as also seen in figure 2(d) of Dong.
Dong further teaches “a first decoder module configured to decode the first and second representations and generate [first and second] depth maps for the images based on the first and second representations, respectively” on Page 3 left Column Subsection 3.1 first paragraph “Our network comprises a depth network (Fig. 2(a)) and a pose network (Fig. 2(b)). The depth network employs a U-Net structure, encompassing an encoder to extract image features and a decoder to predict dense depth maps.”. The extracted features correspond to the first and second representations and the dense depth map corresponds to the depth map. However, Dong does not teach multiple depth maps.
Godard teaches first and second depth maps in Column 14 [lines 9-11] “the depth-pose hybrid model 455 takes a plurality of input images and feeds each through the depth encoder 470 to extract abstract depth features.”.
Finally, Dong teaches “a second decoder module configured to determine a six degree of freedom pose translation of a camera that captured the video.” based on [the first and second representations]. Dong on Page 3 left column Subsection 3.1 first paragraph discloses that “the pose network predicts the camera transformation between two frames. It has a feature extractor followed by a prediction head, which outputs a six-dimensional vector – three for rotation angles and the other three for translation”. Godard was relied on to teach the first and second representations. Godard in Column 14 [lines 11-18] states, “The abstract depth features from each input image are then concatenated together prior to being input into the pose decoder 480 resulting in a pose for each of the input images or the relative transformations between two subsequent input images. The depth-pose hybrid model 455 is more computationally efficient than the pose estimation model 440 in extracting a pose for each pair of the input images”. The features from each input image that are input to the pose decoder, taught by, Godard correspond to the first and second representations.
Thus, Dong is relied on to teach a six degree of freedom pose translation of a camera, and Godard teaches first and second representations (i.e., supplying features extracted from the respective input images). In other words, these representations are supplied to a pose decoder to determine the pose translation of a camera that captured the video based on the first and second representations. Therefore, the combination of these two references meets the limitations recited in the pending claims.
The Applicant argues on page 14:
PNG
media_image5.png
368
816
media_image5.png
Greyscale
The Examiner respectfully disagrees. As explained above, the pending claim does not recite “one encoder module are shared by the first and second decoder modules”. Thus, due to the Applicant's broad claim language, Applicant's invention is not far removed from the art of record as explained above. Accordingly, these limitations do not render claims patentably distinct over the prior art of record. As a result, it is respectfully submitted that the present application is not in condition for allowance.
Finally, the Applicant argues on page 15:
PNG
media_image6.png
356
810
media_image6.png
Greyscale
The Examiner respectfully disagrees. Dong teaches “train parameters of adapter modules the encoder module using consecutive frames of monocular video based on depth maps [and pose translations] determined by the model based on the consecutive frames of monocular video”. Dong on Page 4 left column Subsection 3.5 Second paragraph discloses that “In Stage 2, we load the weights of the U-Net encoder, the encoder adapters, and the U-Net decoder from Stage 1, and freeze both the encoder and decoder, with only adapter parameters being updated. This paradigm preserves the depth perception ability obtained from Stage 1, as most network parameters are frozen and are unaffected by the erroneous loss caused by object motion”. Godard teaches training the parameter based on pose translation in Column 12 [lines 40-43] “the depth estimation training system 170 includes depth and pose models 175, an image synthesis module 180, an error calculation module 185, an appearance matching loss module”. Therefore, the combination of these two references meets the limitations of the pending claims with the suggestion/motivation provided in the detailed rejection of the claims.
Accordingly, the rejection of claims 1-21 under 35 U.S.C 103 is maintained. This Office action has also been updated with new grounds of rejection addressing newly added claims 22 and 23 with ‘Ranftl et al. (Ranftl, R.; Bochkovskiy, A.; and Koltun, V. 2021, “Vision transformers for dense prediction”, arXiv preprint arXiv:2103.13413)’, ‘Kumar, Varun Ravi, et al. "Omnidet: Surround view cameras based multi-task visual perception network for autonomous driving", IEEE Robotics and Automation Letters 6.2 (2021): 2830-2837’, and ‘Tang et al. (US 11,727,588 B2)’.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 4, 6-8, 10, 15, and 21 are rejected under 35 U.S.C. 103 as being unpatentable over Dong et al. ("PPEA-Depth: Progressive Parameter-Efficient Adaptation for Self-Supervised Monocular Depth Estimation." arXiv preprint arXiv:2312.13066 (2023).) (hereinafter, “Dong”) in view of Godard et al. (US 11,991,342 B2) (hereinafter, “Godard”).
Regarding claim 1, Dong discloses a non-transitory computer readable medium storing a computer model, the computer model comprising (Page 3 Figure 2 “(a) Depth network is a U-Net structure predicting depth taking three consecutive frames. (b) Pose network regresses the camera relative pose given two images. (c) Adapter is a bottleneck structure with a skip connection. (d) Structure of RepLKNet (Ding et al. 2022) backbone. (e) Our encoder adapter design. We attach encoder adapters to pre-trained RepLKBlock and ConvFFN. (f) Our decoder adapter design. Lerp represents linear interpolation.”; Examiner interprets the disclosed network to be implemented through a stored model, as such networks require software instructions for training and execution):
an encoder module (RepLKNet (a CNN architecture featuring a notable kernel size of 31x31, as the encoder backbone) on Page 3 right column Subsection 3.3 first 2 paragraphs equates to an encoder module) configured to encode first and second images (three consecutive frames on Page 3 left column Subsection 3.1 second paragraph equates to first and second images; Figure 2 (d) illustrates image being inputted to encoder) into first and second representations (feature maps of four different scales on Page 3 right column Subsection 3.3 first 2 paragraphs equate to first and second representations) (Page 3 left column Subsection 3.1 second paragraph “Our network takes three consecutive frames from a monocular video as input.”; Page 3 right column Subsection 3.3 first 2 paragraphs “We opt for RepLKNet (Ding et al. 2022), a CNN architecture featuring a notable kernel size of 31 × 31, as the encoder backbone. This selection is attributed to its adaptability concerning input image resolution, comparable accuracy to Swin Transformer (Liu et al. 2021), and enhanced inference speed when applied to downstream tasks. As illustrated in Fig. 2(d), RepLKNet generates feature maps of four different scales: F 1, F 2, F 3, F 4 at four stages.”), respectively,
Figure 2
PNG
media_image7.png
525
1372
media_image7.png
Greyscale
the first and second images being from consecutive frames from video (Page 3 left column Subsection 3.1 second paragraph “Our network takes three consecutive frames from a monocular video as input.”);
a first decoder module (depth network on Page 3 left Column Subsection 3.1 first paragraph equates to the first decoder) configured to decode the first and second representations and generate [first and second] depth maps for the images based on the first and second representations (Page 3 left Column Subsection 3.1 first paragraph “Our network comprises a depth network (Fig. 2(a)) and a pose network (Fig. 2(b)). The depth network employs a U-Net structure, encompassing an encoder to extract image features and a decoder to predict dense depth maps.”; Page 4 right column Subsection 4,2 second paragraph “PPEA-Depth adopts the well-established multi-frame inference and teacher-student distillation training scheme…The main network contains a cost volume construction process, using both the current frame It and its preceding frame It−1 to predict depth Dt.”; Examiner interprets the extracted image features to be the extracted features of the current frame and the preceding frame which equate to the first and second representation), respectively; and
a second decoder module (pose network Page 3 left column Subsection 3.1 first paragraph equates to a second decoder) configured to determine a six degree of freedom pose translation of a camera that captured the video based on the [first and second representations] (Page 3 left column Subsection 3.1 first paragraph “the pose network predicts the camera transformation between two frames. It has a feature extractor followed by a prediction head, which outputs a six-dimensional vector – three for rotation angles and the other three for translation.”).
However, Dong fails to teach a first and second [depth maps] and [pose translation of a camera that captured the video based on the] first and second representations.
Godard teaches a first and second [depth maps] (Column 14 [lines 9-11] “the depth-pose hybrid model 455 takes a plurality of input images and feeds each through the depth encoder 470 to extract abstract depth features.”) and pose translation of a camera that captured the video based on the] first and second representations (Column 14 [lines 11-18] “The abstract depth features from each input image are then concatenated together prior to being input into the pose decoder 480 resulting in a pose for each of the input images or the relative transformations between two subsequent input images. The depth-pose hybrid model 455 is more computationally efficient than the pose estimation model 440 in extracting a pose for each pair of the input images.”).
Therefore, it would have been obvious to one of ordinary skill of the art before the effective filing date to modify Dong’s reference to include a first and second [depth maps] and [pose translation of a camera that captured the video based on the] first and second representations taught by Godard’s reference. The motivation for doing so would have been to reduce the overall computation time by sharing the parameters between the models as suggested by Godard (see Godard, Column 13 [lines 58-60]).
Further, one skilled in the art could have combined the elements described above by known methods with no change to the respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Godard with Dong to obtain the invention specified in claim 1.
Regarding claim 4, which claim 1 is incorporated, Dong discloses wherein the encoder module includes adapter modules trained based on minimizing a photometric loss (Page 3 left column subsection 3.1 second paragraph “The middle frame is reconstructed with its adjacent frames, and the difference between the re-constructed and original images serves as the supervision signal.”; Page 13 left column Subsection D.2 third paragraph “During the training process, the teacher network is super-vised by the image reprojection loss (as mentioned in Section 3.1 in the main paper).”).
Regarding claim 6, Dong discloses a system comprising:
a model including (Page 3 left column Subsection 3.2 first paragraph “Our network comprises a depth network (Fig. 2(a)) and a pose network (Fig. 2(b)).”):
an encoder module (RepLKNet (a CNN architecture featuring a notable kernel size of 31x31, as the encoder backbone) on Page 3 right column Subsection 3.3 first 2 paragraphs equates to an encoder module) configured to encode first and second images (three consecutive frames on Page 3 left column Subsection 3.1 second paragraph equates to first and second images; Figure 2 (d) illustrates image being inputted to encoder) into first and second representations (feature maps of four different scales on Page 3 right column Subsection 3.3 first 2 paragraphs equate to first and second representations) (Page 3 left column Subsection 3.1 second paragraph “Our network takes three consecutive frames from a monocular video as input.”; Page 3 right column Subsection 3.3 first 2 paragraphs “We opt for RepLKNet (Ding et al. 2022), a CNN architecture featuring a notable kernel size of 31 × 31, as the encoder backbone. This selection is attributed to its adaptability concerning input image resolution, comparable accuracy to Swin Transformer (Liu et al. 2021), and enhanced inference speed when applied to downstream tasks. As illustrated in Fig. 2(d), RepLKNet generates feature maps of four different scales: F 1, F 2, F 3, F 4 at four stages.”), respectively;
Figure 2
PNG
media_image7.png
525
1372
media_image7.png
Greyscale
a first decoder module (depth network on Page 3 left Column Subsection 3.1 first paragraph equates to the first decoder) configured to decode the first and second representations and generate [first and second] depth maps for the images based on the first and second representations (Page 3 left Column Subsection 3.1 first paragraph “Our network comprises a depth network (Fig. 2(a)) and a pose network (Fig. 2(b)). The depth network employs a U-Net structure, encompassing an encoder to extract image features and a decoder to predict dense depth maps.”; Page 4 right column Subsection 4,2 second paragraph “PPEA-Depth adopts the well-established multi-frame inference and teacher-student distillation training scheme…The main network contains a cost volume construction process, using both the current frame It and its preceding frame It−1 to predict depth Dt.”; Examiner interprets the extracted image features to be the extracted features of the current frame and the preceding frame which equate to the first and second representation), respectively; and
a second decoder module (pose network Page 3 left column Subsection 3.1 first paragraph equates to a second decoder) configured to determine a six degree of freedom pose translation of a camera that captured the first and second images based on [the first and second representations] (Page 3 left column Subsection 3.1 first paragraph “the pose network predicts the camera transformation between two frames. It has a feature extractor followed by a prediction head, which outputs a six-dimensional vector – three for rotation angles and the other three for translation.”); and
a training module configured to: train the model using pairs of images, each pair of images including at least part of a same scene and captured at different times (Page 3 left column Subsection 3.1 second paragraph “Our network takes three consecutive frames from a monocular video as input. The middle frame is reconstructed with its adjacent frames, and the difference between the re-constructed and original images serves as the supervision signal.”); and
train parameters of adapter modules (adapter on Page 4 left column Subsection 3.5 equates to adapter modules) of the encoder module using consecutive frames of monocular video based on depth maps and [pose translations] determined by the model based on the consecutive frames of monocular video (Page 4 left column Subsection 3.5 Second paragraph “In Stage 2, we load the weights of the U-Net encoder, the encoder adapters, and the U-Net decoder from Stage 1, and freeze both the encoder and decoder, with only adapter parameters being updated. This paradigm preserves the depth perception ability obtained from Stage 1, as most network parameters are frozen and are unaffected by the erroneous loss caused by object motion.”).
However, Dong fails to teach a first and second [depth maps], [pose translation of a camera that captured the first and second images based on the] first and second representations, and [train parameters of adapter modules of the encoder module using consecutive frames of monocular video based on depth maps and] based on pose translations.
Godard teaches a first and second [depth maps] (Column 14 [lines 9-11] “the depth-pose hybrid model 455 takes a plurality of input images and feeds each through the depth encoder 470 to extract abstract depth features.”) , [pose translation of a camera that captured the first and second images based on the] first and second representations (Column 14 [lines 11-18] “The abstract depth features from each input image are then concatenated together prior to being input into the pose decoder 480 resulting in a pose for each of the input images or the relative transformations between two subsequent input images. The depth-pose hybrid model 455 is more computationally efficient than the pose estimation model 440 in extracting a pose for each pair of the input images.”), and [train parameters of adapter modules of the encoder module using consecutive frames of monocular video based on depth maps and] based on pose translations (Column 12 [lines 40-43] “the depth estimation training system 170 includes depth and pose models 175, an image synthesis module 180, an error calculation module 185, an appearance matching loss module”).
Therefore, it would have been obvious to one of ordinary skill of the art before the effective filing date to modify Dong’s reference to include a first and second [depth maps], [pose translation of a camera that captured the first and second images based on the] first and second representations, and [train parameters of adapter modules of the encoder module using consecutive frames of monocular video based on depth maps and] based on pose translations taught by Godard’s reference. The motivation for doing so would have been to reduce the overall computation time by sharing the parameters between the models as suggested by Godard (see Godard, Column 13 [lines 58-60]).
Further, one skilled in the art could have combined the elements described above by known methods with no change to the respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Godard with Dong to obtain the invention specified in claim 6.
Regarding claim 7, which claim 6 is incorporated, Dong discloses wherein the training module is configured to train the parameters of the adapter modules after training the model using the pairs of images (Page 3 left column Subsection 3.1 “Given camera intrinsics K and camera relative pose T between two frames a, b from a video, the pixel correspondence between them can be computed”; Page 4 left column Subsection 3.5 first 2 paragraphs “our progressive adaptation involves two stages. Stage 1 is trained on a dataset that primarily follows the static-scene assumption…Stage 2 is conducted on datasets that predominantly feature dynamic scenes… In Stage 2, we load the weights of the U-Net encoder, the encoder adapters, and the U-Net decoder from Stage 1, and freeze both the encoder and decoder, with only adapter parameters being up-dated.” Page 4, right column Subsection 4.1 first 2 paragraphs “.
Regarding claim 8, which claim 6 is incorporated, Dong discloses wherein the training module is configured to train the parameters of the adapter modules without annotations for the frames of the monocular video (Page 2 left column paragraph 3 “We propose an innovative two-stage self-supervised depth estimation approach, PPEA-Depth, based on parameter-efficient adaptation. We devise lightweight encoder adapters and decoder adapters within our framework.”).
Regarding claim 10, which claim 6 is incorporated, Dong discloses wherein the training module is configured to train the parameters of the adapter modules while all other parameters of the model are fixed (Page 4 left column Subsection 3.5 Second paragraph “In Stage 2, we load the weights of the U-Net encoder, the encoder adapters, and the U-Net decoder from Stage 1, and freeze both the encoder and decoder, with only adapter parameters being updated. This paradigm preserves the depth perception ability obtained from Stage 1, as most network parameters are frozen and are unaffected by the erroneous loss caused by object motion.”)..
Regarding claim 15, which claim 6 is incorporated, Dong discloses wherein the training module is configured to train the parameters of the adapter modules based on minimizing a photometric loss (Page 3 left column subsection 3.1 second paragraph “The middle frame is reconstructed with its adjacent frames, and the difference between the re-constructed and original images serves as the supervision signal.”; Page 13 left column Subsection D.2 third paragraph “During the training process, the teacher network is super-vised by the image reprojection loss (as mentioned in Section 3.1 in the main paper).”).
Regarding claim 21, Dong discloses a method, comprising:
train a model using pairs of images, each pair of images including at least part of a same scene and captured at different times (Page 3 left column Subsection 3.1 second paragraph “Our network takes three consecutive frames from a monocular video as input. The middle frame is reconstructed with its adjacent frames, and the difference between the re-constructed and original images serves as the supervision signal.”),
the model including (Page 3 left column Subsection 3.2 first paragraph “Our network comprises a depth network (Fig. 2(a)) and a pose network (Fig. 2(b)).”):
an encoder module (RepLKNet (a CNN architecture featuring a notable kernel size of 31x31, as the encoder backbone) on Page 3 right column Subsection 3.3 first 2 paragraphs equates to an encoder module) configured to encode first and second images (three consecutive frames on Page 3 left column Subsection 3.1 second paragraph equates to first and second images; Figure 2 (d) illustrates image being inputted to encoder) into first and second representations (feature maps of four different scales on Page 3 right column Subsection 3.3 first 2 paragraphs equate to first and second representations) (Page 3 left column Subsection 3.1 second paragraph “Our network takes three consecutive frames from a monocular video as input.”; Page 3 right column Subsection 3.3 first 2 paragraphs “We opt for RepLKNet (Ding et al. 2022), a CNN architecture featuring a notable kernel size of 31 × 31, as the encoder backbone. This selection is attributed to its adaptability concerning input image resolution, comparable accuracy to Swin Transformer (Liu et al. 2021), and enhanced inference speed when applied to downstream tasks. As illustrated in Fig. 2(d), RepLKNet generates feature maps of four different scales: F 1, F 2, F 3, F 4 at four stages.”), respectively;
Figure 2
PNG
media_image7.png
525
1372
media_image7.png
Greyscale
a first decoder module (depth network on Page 3 left Column Subsection 3.1 first paragraph equates to the first decoder) configured to decode the first and second representations and generate [first and second] depth maps for the images based on the first and second representations (Page 3 left Column Subsection 3.1 first paragraph “Our network comprises a depth network (Fig. 2(a)) and a pose network (Fig. 2(b)). The depth network employs a U-Net structure, encompassing an encoder to extract image features and a decoder to predict dense depth maps.”; Page 4 right column Subsection 4,2 second paragraph “PPEA-Depth adopts the well-established multi-frame inference and teacher-student distillation training scheme…The main network contains a cost volume construction process, using both the current frame It and its preceding frame It−1 to predict depth Dt.”; Examiner interprets the extracted image features to be the extracted features of the current frame and the preceding frame which equate to the first and second representation), respectively; and
a second decoder (pose network Page 3 left column Subsection 3.1 first paragraph equates to a second decoder) module configured to determine a six degree of freedom pose translation of a camera that captured the first and second images based on the first and second representations (Page 3 left column Subsection 3.1 first paragraph “the pose network predicts the camera transformation between two frames. It has a feature extractor followed by a prediction head, which outputs a six-dimensional vector – three for rotation angles and the other three for translation.”); and
training parameters of adapter modules (adapters on Page 4 left column Subsection 3.5 equate to adaptor modules) of the encoder module using consecutive frames of monocular video based on depth maps [and pose translations] determined by the model based on the consecutive frames of monocular video (Page 4 left column Subsection 3.5 Second paragraph “In Stage 2, we load the weights of the U-Net encoder, the encoder adapters, and the U-Net decoder from Stage 1, and freeze both the encoder and decoder, with only adapter parameters being updated. This paradigm preserves the depth perception ability obtained from Stage 1, as most network parameters are frozen and are unaffected by the erroneous loss caused by object motion.”).
However, Dong fails to teach a first and second [depth maps], [pose translation of a camera that captured the first and second images based on the] first and second representations, and [training parameters of adapter modules of the encoder module using consecutive frames of monocular video based on depth maps and] based on pose translations.
Godard teaches a first and second [depth maps] (Column 14 [lines 9-11] “the depth-pose hybrid model 455 takes a plurality of input images and feeds each through the depth encoder 470 to extract abstract depth features.”) , [pose translation of a camera that captured the first and second images based on the] first and second representations (Column 14 [lines 11-18] “The abstract depth features from each input image are then concatenated together prior to being input into the pose decoder 480 resulting in a pose for each of the input images or the relative transformations between two subsequent input images. The depth-pose hybrid model 455 is more computationally efficient than the pose estimation model 440 in extracting a pose for each pair of the input images.”), and [training parameters of adapter modules of the encoder module using consecutive frames of monocular video based on depth maps and] based on pose translations (Column 12 [lines 40-43] “the depth estimation training system 170 includes depth and pose models 175, an image synthesis module 180, an error calculation module 185, an appearance matching loss module”).
Therefore, it would have been obvious to one of ordinary skill of the art before the effective filing date to modify Dong’s reference to include a first and second [depth maps], [pose translation of a camera that captured the first and second images based on the] first and second representations, and [training parameters of adapter modules of the encoder module using consecutive frames of monocular video based on depth maps and] based on pose translations taught by Godard’s reference. The motivation for doing so would have been to reduce the overall computation time by sharing the parameters between the models reduce the overall computation time by sharing the parameters from the pose estimation model and the depth model as suggested by Godard (see Godard, Column 13 [lines 58-60]).
Further, one skilled in the art could have combined the elements described above by known methods with no change to the respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Godard with Dong to obtain the invention specified in claim 21.
Claims 2, 9 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Dong et al. ("PPEA-Depth: Progressive Parameter-Efficient Adaptation for Self-Supervised Monocular Depth Estimation." arXiv preprint arXiv:2312.13066 (2023).) (hereinafter, “Dong”) in view of Godard et al. (US 11,991,342 B2) (hereinafter, “Godard”), and further in view of Wofk et al (US 2022/0343521 A1) (hereinafter, “Wofk”).
Regarding claim 2, which claim 1 is incorporated, Dong discloses wherein the first (depth network on Page 3 left Column Subsection 3.1 first paragraph equates to the first decoder) and second decoder modules (pose network Page 3 left column Subsection 3.1 first paragraph equates to a second decoder) [include dense prediction transformer (DPT) decoders] “(Page 3 left Column Subsection 3.1 first paragraph “Our network comprises a depth network (Fig. 2(a)) and a pose network (Fig. 2(b)). The depth network employs a U-Net structure, encompassing an encoder to extract image features and a decoder to predict dense depth maps.”).
However, both Dong and Godard fail to teach [wherein the first and second decoder modules] include dense prediction transformer (DPT) decoders.
Wofk teaches [wherein the first and second decoder modules] include dense prediction transformer (DPT) decoders (Paragraph [0046] “in addition to using a DPT-Hybrid depth estimator, a DPT-Large depth estimator can be used for higher depth estimation accuracy while a machine-learning framework such as MiDaS-small can be selected for computational efficiency,”).
Therefore, it would have been obvious to one of ordinary skill of the art before the effective filing date to modify Dong in view of Godard to include [wherein the first and second decoder modules] include dense prediction transformer (DPT) decoders taught by Wofk’s reference. The motivation for doing so would have been to obtain higher depth estimation accuracy as suggested by Wofk (see Wofk, paragraph [0046]).
Further, one skilled in the art could have combined the elements described above by known methods with no change to the respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Wofk with Godard and Dong to obtain the invention specified in claim 2.
Regarding claim 9, which claim 6 is incorporated, Dong discloses wherein the adapter modules include an up projection module, [a rectified linear unit (ReLU)], and a down projection module (Page 3 right column first paragraph “adapters follow a bottleneck structure, encompassing two linear projection layers, an activation layer, and a skip connection. The initial projection layer reduces the input feature dimension, and the subsequent one restores it to the original input dimension after the activation layer.”), and
wherein the training module is configured to train parameters of at least one of the up projection module, the ReLU, and the down projection module based on the depth maps and pose translations determined by the model based on the consecutive frames of monocular video (Page 4 left column Subsection 3.5 Second paragraph “In Stage 2, we load the weights of the U-Net encoder, the encoder adapters, and the U-Net decoder from Stage 1, and freeze both the encoder and decoder, with only adapter parameters being updated. This paradigm preserves the depth perception ability obtained from Stage 1, as most network parameters are frozen and are unaffected by the erroneous loss caused by object motion.”).
However, Dong and Godard both fail to teach a rectified linear unit (ReLU).
Wofk teaches a rectified linear unit (ReLU) (Paragraph [0053] “the scale map learner circuitry 315 regresses a dense scale residual map r where values can be negative. In examples disclosed herein, the resulting scale map can be represented as ReLU (1+r) and applied to the input depth {tilde over (z)} to produce the output depth {circumflex over (z)}=ReLU (1+r){tilde over (z)}.”).
Therefore, it would have been obvious to one of ordinary skill of the art before the effective filing date to modify Dong in view of Godard to include a rectified linear unit (ReLU) taught by Wofk’s reference. The motivation for doing so would have been to account for negative values as suggested by Wofk (see Wofk, paragraph [0073]).
Further, one skilled in the art could have combined the elements described above by known methods with no change to the respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Wofk with Godard and Dong to obtain the invention specified in claim 9.
Regarding claim 20 (drawn to a system), claim 20 is rejected the same as claim 2 and the arguments similar to that presented above for claim 2 are equally applicable to the claim 20, and all the other limitations similar to claim 2 are not repeated herein, but incorporated by reference.
Claims 3, 11-14, and 16 are rejected under 35 U.S.C. 103 as being unpatentable over Dong et al. ("PPEA-Depth: Progressive Parameter-Efficient Adaptation for Self-Supervised Monocular Depth Estimation." arXiv preprint arXiv:2312.13066 (2023).) (hereinafter, “Dong”) in view of Godard et al. (US 11,991,342 B2) (hereinafter, “Godard”), and further in view of Tang et al. (US 11,727,588 B2) (hereinafter, “Tang”).
Regarding claim 3, which claim 1 is incorporated, Dong discloses wherein the encoder module includes adapter modules trained [based on minimizing a geometric consistency loss] (Page 4 left column Subsection 3.5 Second paragraph “In Stage 2, we load the weights of the U-Net encoder, the encoder adapters, and the U-Net decoder from Stage 1, and freeze both the encoder and decoder, with only adapter parameters being updated. This paradigm preserves the depth perception ability obtained from Stage 1, as most network parameters are frozen and are unaffected by the erroneous loss caused by object motion.”)
However, Dong and Godard both fail to teach based on minimizing a geometric consistency loss.
Tang teaches based on minimizing a geometric consistency loss (Column 11 [lines 15-19] “The warped depth estimate 614 and the source depth estimate 610 may be input to the residual pose network 604 which may minimize a geometric difference between the warped depth estimate 614 and the source depth estimate 610.”).
Therefore, it would have been obvious to one of ordinary skill of the art before the effective filing date to modify Dong in view of Godard to include minimizing a geometric consistency loss taught by Tang’s reference. The motivation for doing so would have been to improve the accuracy of the warped image as suggested by Tang (see Tang, column 11 [lines 23-28]).
Further, one skilled in the art could have combined the elements described above by known methods with no change to the respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Tang with Godard and Dong to obtain the invention specified in claim 3.
Regarding claim 11, which claim 6 is incorporated, Dong discloses wherein the training module is configured to train the parameters of the adapter modules [based on minimizing a geometric consistency loss] (Page 4 left column Subsection 3.5 Second paragraph “In Stage 2, we load the weights of the U-Net encoder, the encoder adapters, and the U-Net decoder from Stage 1, and freeze both the encoder and decoder, with only adapter parameters being updated. This paradigm preserves the depth perception ability obtained from Stage 1, as most network parameters are frozen and are unaffected by the erroneous loss caused by object motion.”)
However, Dong and Godard both fail to teach based on minimizing a geometric consistency loss.
Tang teaches based on minimizing a geometric consistency loss (Column 11 [lines 15-19] “The warped depth estimate 614 and the source depth estimate 610 may be input to the residual pose network 604 which may minimize a geometric difference between the warped depth estimate 614 and the source depth estimate 610.”).
Therefore, it would have been obvious to one of ordinary skill of the art before the effective filing date to modify Dong in view of Godard to include minimizing a geometric consistency loss taught by Tang’s reference. The motivation for doing so would have been to improve the accuracy of the warped image as suggested by Tang (see Tang, column 11 [lines 23-28]).
Further, one skilled in the art could have combined the elements described above by known methods with no change to the respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Tang with Godard and Dong to obtain the invention specified in claim 11.
Regarding claim 12, which claim 11 is incorporated, Dong and Godard fail to teach a warping module configured to generate a warped depth map based on the first depth map; and a loss module configured to determine the geometric consistency loss based on differences between the warped depth map and the first depth map.
Tang teaches a warping module configured to generate a warped depth map based on the first depth map (Column 11 [lines 7-15] “a warped depth estimate 614 of the target depth estimate 612 may be generated based on the pose estimate 606. The warped depth estimate 614 may be computed by transforming the target depth estimate 612 according to the pose estimate 602. The transformation module (not shown) may warp the target depth estimate 612.”); and
a loss module configured to determine the geometric consistency loss based on differences between the warped depth map and the first depth map (Column 11 [lines 15-19] “The warped depth estimate 614 and the source depth estimate 610 may be input to the residual pose network 604 which may minimize a geometric difference between the warped depth estimate 614 and the source depth estimate 610.”).
Therefore, it would have been obvious to one of ordinary skill of the art before the effective filing date to modify Dong in view of Godard to include a warping module configured to generate a warped depth map based on the first depth map; and a loss module configured to determine the geometric consistency loss based on differences between the warped depth map and the first depth map taught by Tang’s reference. The motivation for doing so would have been to improve the accuracy of the warped image as suggested by Tang (see Tang, column 11 [lines 23-28]).
Further, one skilled in the art could have combined the elements described above by known methods with no change to the respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Tang with Godard and Dong to obtain the invention specified in claim 12.
Regarding claim 13, which claim 12 is incorporated, Dong and Godard fail to teach wherein the warping module is configured to generate the warped depth map further based on the pose translation.
Tang teaches wherein the warping module is configured to generate the warped depth map further based on the pose translation (Column 11 [lines 7-15] “a warped depth estimate 614 of the target depth estimate 612 may be generated based on the pose estimate 606. The warped depth estimate 614 may be computed by transforming the target depth estimate 612 according to the pose estimate 602. The transformation module (not shown) may warp the target depth estimate 612.”).
Therefore, it would have been obvious to one of ordinary skill of the art before the effective filing date to modify Dong in view of Godard to include wherein the warping module is configured to generate the warped depth map further based on the pose translation taught by Tang’s reference. The motivation for doing so would have been to improve the accuracy of the warped image as suggested by Tang (see Tang, column 11 [lines 23-28]).
Further, one skilled in the art could have combined the elements described above by known methods with no change to the respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Tang with Godard and Dong to obtain the invention specified in claim 13.
Regarding claim 14, which claim 13 is incorporated, Dong and Godard fail to teach wherein the warping module is configured to generate the warped depth map based on transforming the first depth map to a three dimensional space and projecting to the second image using the pose translation.
Tang teaches wherein the warping module is configured to generate the warped depth map based on transforming the first depth map to a three dimensional space and projecting to the second image using the pose translation (Column 9 [lines 19-22] “the depth map 508 may be a per-pixel depth map. A view estimation module 510 receives the per-pixel depth map 508 and the six DoF transformation (e.g., relative pose) between the target image 504 and the source image 506.”; Column 11 [lines6-19] “the pose estimate 602 may be used to estimate a residual pose estimate 606…a warped depth estimate 614 of the target depth estimate 612 may be generated based on the pose estimate 606. The warped depth estimate 614 may be computed by transforming the target depth estimate 612 according to the pose estimate 602. The transformation module (not shown) may warp the target depth estimate 612. The warped depth estimate 614 and the source depth estimate 610 may be input to the residual pose network 604 which may minimize a geometric difference between the warped depth estimate 614 and the source depth estimate 610.”; Column 12 [lines 14-17] “the determined residual pose accounts for geometric information and improves a consistency of depth estimates between the target and source views.”).
Therefore, it would have been obvious to one of ordinary skill of the art before the effective filing date to modify Dong in view of Godard to include wherein the warping module is configured to generate the warped depth map based on transforming the first depth map to a three dimensional space and projecting to the second image using the pose translation taught by Tang’s reference. The motivation for doing so would have been to improve the accuracy of the warped image as suggested by Tang (see Tang, column 11 [lines 23-28]).
Further, one skilled in the art could have combined the elements described above by known methods with no change to the respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Tang with Godard and Dong to obtain the invention specified in claim 14.
Regarding claim 16, which claim 15 is incorporated, Dong and Godard fail to teach a warping module configured to generate a warped image based on the first image; and a loss module configured to determine the photometric consistency loss based on differences between the warped image and the first image.
Tang teaches a warping module configured to generate a warped image based on the first image (Column 6 [lines 58-62] “each individual pixel for the target image is warped according to its own depth and pose estimates to generate a reconstructed image (e.g., warped source image)”); and
a loss module configured to determine the photometric consistency loss based on differences between the warped image and the first image (Column 10 [lines 8-11] “A photometric loss is calculated based on the difference between the target image 504 and the warped image 512 (e.g., the warped source image that approximates the target image).”).
Therefore, it would have been obvious to one of ordinary skill of the art before the effective filing date to modify Dong in view of Godard to include a warping module configured to generate a warped image based on the first image; and a loss module configured to determine the photometric consistency loss based on differences between the warped image and the first image taught by Tang’s reference. The motivation for doing so would have been to generate a 3D representation of the target image and use the loss to update the network as suggested by Tang (see Tang, column 3 [lines 43-45] and column 10 [lines 13-15]).
Further, one skilled in the art could have combined the elements described above by known methods with no change to the respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Tang with Godard and Dong to obtain the invention specified in claim 16.
Claims 5 and 18-19 are rejected under 35 U.S.C. 103 as being unpatentable over Dong et al. ("PPEA-Depth: Progressive Parameter-Efficient Adaptation for Self-Supervised Monocular Depth Estimation." arXiv preprint arXiv:2312.13066 (2023).) (hereinafter, “Dong”) in view of Godard et al. (US 11,991,342 B2) (hereinafter, “Godard”), and further in view of Guizilini et al. (US 11,386,567 B2) (hereinafter, “Guizilini”).
Regarding claim 5, which claim 1 is incorporated, Dong and Godard fail to teach wherein the encoder module includes adapter modules trained based on minimizing an edge smoothness loss.
Guizilini teaches wherein the encoder module includes adapter modules trained based on minimizing an edge smoothness loss (Paragraph [0046] “in addition to using a DPT-Hybrid depth estimator, a DPT-Large depth estimator can be used for higher depth estimation accuracy while a machine-learning framework such as MiDaS-small can be selected for computational efficiency,”).
Therefore, it would have been obvious to one of ordinary skill of the art before the effective filing date to modify Dong in view of Godard to include wherein the encoder module includes adapter modules trained based on minimizing an edge smoothness loss taught by Guizilini’s reference. The motivation for doing so would have been to modify the parameters of the models to perform the training as suggested by Guizilini (see Guizilini, Column 5 [lines 8-11]).
Further, one skilled in the art could have combined the elements described above by known methods with no change to the respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Guizilini with Godard and Dong to obtain the invention specified in claim 5.
Regarding claim 18, which claim 6 is incorporated, Dong and Godard fail to teach wherein the training module is configured to train the parameters of the adapter modules based on minimizing an edge smoothness loss.
Guizilini teaches wherein the training module is configured to train the parameters of the adapter modules based on minimizing an edge smoothness loss (Column 13 [lines 15-20] “Ls represents depth smoothness loss and is implemented to regularize the depth in textureless low-image gradient regions, as shown in equation (5). The smoothness loss is an edge-aware term that is weighted for separate pyramid levels starting from 1 and decaying by a factor of two for the separate scales.)”.
Equation 5
PNG
media_image8.png
74
442
media_image8.png
Greyscale
Therefore, it would have been obvious to one of ordinary skill of the art before the effective filing date to modify Dong in view of Godard to include wherein the training module is configured to train the parameters of the adapter modules based on minimizing an edge smoothness loss taught by Guizilini’s reference. The motivation for doing so would have been to modify the parameters of the models to perform the training as suggested by Guizilini (see Guizilini, Column 5 [lines 8-11]).
Further, one skilled in the art could have combined the elements described above by known methods with no change to the respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Guizilini with Godard and Dong to obtain the invention specified in claim 18.
Regarding claim 19, which claim 18 is incorporated, Dong and Godard fail to teach a loss module configured to determine the edge smoothness loss based on first derivatives of pixel values of the first and second depth maps.
Guizilini teaches a loss module configured to determine the edge smoothness loss based on first derivatives of pixel values of the first and second depth maps (Column 16 [lines 25-30] “the training module 230 uses this appearance-based loss as both the first stage loss to account for pixel-level similarities and irregularities along edge regions between a synthesized image derived from depth predictions of the depth model and a target image that is the original input into the depth model 260.”.)
Therefore, it would have been obvious to one of ordinary skill of the art before the effective filing date to modify Dong in view of Godard to include a loss module configured to determine the edge smoothness loss based on first derivatives of pixel values of the first and second depth maps taught by Guizilini’s reference. The motivation for doing so would have been to modify the parameters of the models to perform the training as suggested by Guizilini (see Guizilini, Column 5 [lines 8-11]).
Further, one skilled in the art could have combined the elements described above by known methods with no change to the respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Guizilini with Godard and Dong to obtain the invention specified in claim 19.
Claim 17 is rejected under 35 U.S.C. 103 as being unpatentable over Dong et al. ("PPEA-Depth: Progressive Parameter-Efficient Adaptation for Self-Supervised Monocular Depth Estimation." arXiv preprint arXiv:2312.13066 (2023).) (hereinafter, “Dong”) in view of Godard et al. (US 11,991,342 B2) (hereinafter, “Godard”), further in view of Tang et al. (US 11,727,588 B2) (hereinafter, “Tang”), and Guizilini et al. (US 11,386,567 B2) (hereinafter, “Guizilini”).
Regarding claim 17, which claim 16 is incorporated, Dong, Godard and Tang fail to teach wherein the loss module is configured to determine the photometric loss based on downweighting regions of the first image including moving objects.
Guizilini teaches wherein the loss module is configured to determine the photometric loss based on downweighting regions of the first image including moving objects (Column 13 [lines 1-10] “the training module 230 masks out static pixels by removing pixels that have a warped photometric loss higher than a corresponding unwarped photometric loss, which the training module 230 calculates using the original source image (e.g., 620) without synthesizing the target. The mask (Mp) removes pixels that have appearance loss that does not change between frames, which includes static scenes and dynamic objects moving at a similar speed as the camera.”).
Therefore, it would have been obvious to one of ordinary skill of the art before the effective filing date to modify Dong in view of Godard, and further in view of Tang to include wherein the loss module is configured to determine the photometric loss based on downweighting regions of the first image including moving objects taught by Guizilini’s reference. The motivation for doing so would have been to remove pixels that have appearance loss as suggested by Guizilini (see Guizilini, Column 13 [lines 6-8]).
Further, one skilled in the art could have combined the elements described above by known methods with no change to the respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Guizilini with Godard, Tang, and Dong to obtain the invention specified in claim 17.
Claim 22 is rejected under 35 U.S.C. 103 as being unpatentable over Dong et al. ("PPEA-Depth: Progressive Parameter-Efficient Adaptation for Self-Supervised Monocular Depth Estimation." arXiv preprint arXiv:2312.13066 (2023).) (hereinafter, “Dong”) in view of Godard et al. (US 11,991,342 B2) (hereinafter, “Godard”), further in view of Ranftl et al. (Ranftl, R.; Bochkovskiy, A.; and Koltun, V. 2021. Vision transformers for dense prediction. arXiv preprint arXiv:2103.13413.) (hereinafter; “Ranftl”).
Regarding claim 22, which claim 1 is incorporated, Dong and Godard fail to teach wherein the first decoder module comprises a dense prediction transformer (DPT) decoder configured to combine features from a plurality of transformer blocks of the encoder module by reshaping the features to different resolutions and fusing the reshaped features using convolutional layers.
Ranftl teaches wherein the first decoder module comprises a dense prediction transformer (DPT) decoder (Page 2 left column second paragraph “we introduce the dense prediction transformer (DPT). DPT is a dense prediction architecture that is based on an encoder-decoder design that leverages a transformer as the basic computational building block of the encoder.”) configured to combine features from a plurality of transformer blocks of the encoder module by reshaping the features to different resolutions and fusing the reshaped features using convolutional layers (Page 3 Figure 1 Caption: Figure 1. Left: Architecture overview. The input image is transformed into tokens (orange) either by extracting non-overlapping patches followed by a linear projection of their flattened representation (DPT-Base and DPT-Large) or by applying a ResNet-50 feature extractor (DPT-Hybrid). The image embedding is augmented with a positional embedding and a patch-independent readout token (red) is added. The tokens are passed through multiple transformer stages. We reassemble tokens from different stages into an image-like representation at multiple resolutions (green). Fusion modules (purple) progressively fuse and upsample the representations to generate a fine-grained prediction. Center: Overview of the Reassembles operation. Tokens are assembled into feature maps with 1/s the spatial resolution of the input image. Right: Fusion blocks combine features using residual convolutional units [23] and upsample the feature maps.”).
Therefore, it would have been obvious to one of ordinary skill of the art before the effective filing date to modify Dong in view of Godard to include wherein the first decoder module comprises a dense prediction transformer (DPT) decoder configured to combine features from a plurality of transformer blocks of the encoder module by reshaping the features to different resolutions and fusing the reshaped features using convolutional layers taught by Ranftl’s reference. The motivation for doing so would have been because DPT can better reconstruct fine details while also improving global coherence as suggested by Ranftl (see Ranftl, Page 5 right column paragraph 3).
Further, one skilled in the art could have combined the elements described above by known methods with no change to the respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Ranftl with Dong and Godard to obtain the invention specified in claim 22.
Claim 23 is rejected under 35 U.S.C. 103 as being unpatentable over Dong et al. ("PPEA-Depth: Progressive Parameter-Efficient Adaptation for Self-Supervised Monocular Depth Estimation." arXiv preprint arXiv:2312.13066 (2023).) (hereinafter, “Dong”) in view of Godard et al. (US 11,991,342 B2) (hereinafter, “Godard”), further in view of Kumar et al. (Kumar, Varun Ravi, et al. "Omnidet: Surround view cameras based multi-task visual perception network for autonomous driving." IEEE Robotics and Automation Letters 6.2 (2021): 2830-2837.) (hereinafter, “Kumar”) and Tang et al. (US 11,727,588 B2) (hereinafter, “Tang”).
Regarding claim 23, Dong discloses [wherein the encoder module is shared by a depth estimation branch and a visual odometry branch of the model, and] wherein the training module is configured to train the parameters of the adapter modules by minimizing a combined loss (Page 3 left column subsection 3.1 second paragraph “The middle frame is reconstructed with its adjacent frames, and the difference between the re-constructed and original images serves as the supervision signal.”; Page 13 left column Subsection D.2 third paragraph “During the training process, the teacher network is supervised by the image reprojection loss (as mentioned in Section 3.1 in the main paper). For the student network, the reliable area (¬M) is also supervised by the image reprojection loss, while the unreliable area (M) is instead supervised by a depth consistency loss to enforce a knowledge distillation from the teacher. The depth consistency loss is the L1 difference of the predicted depths between the teacher and the student. In the depth consistency loss, gradients to depths predicted by the teacher are blocked, ensuring the distillation is unidirectional, i.e. only from teacher to student.”) [comprising a geometric consistency loss determined from a warped depth map and a photometric loss determined from a warped image].
However, Dong and Godard both fail to teach wherein the encoder module is shared by a depth estimation branch and a visual odometry branch of the model, and comprising a geometric consistency loss determined from a warped depth map and a photometric loss determined from a warped image.
Kumar teaches wherein the encoder module is shared by a depth estimation branch and a visual odometry branch of the model (Page 2820 Abstract “This work presents a multi-task visual perception network on unrectified fisheye images to enable the vehicle to sense its surrounding environment. It consists of six primary tasks necessary for an autonomous driving system: depth estimation, visual odometry, semantic segmentation, motion segmentation, object detection, and lens soiling detection. We demonstrate that the jointly trained model performs better than the respective single task versions. Our multi-task model has a shared encoder providing a significant computational advantage and has synergized decoders where tasks support each other.”).
Therefore, it would have been obvious to one of ordinary skill of the art before the effective filing date to modify Dong in view of Godard to include wherein the encoder module is shared by a depth estimation branch and a visual odometry branch of the model taught by Kumar’s reference. The motivation for doing so would have been to improve efficiency on the embedded system requiring less memory and data rate to transmit as suggested by Kumar (see Kumar, Page 2833 left column Section III paragraph 2).
However, Dong, Godard and Kumar fail to teach comprising a geometric consistency loss determined from a warped depth map and a photometric loss determined from a warped image.
Tang teaches comprising a geometric consistency loss determined from a warped depth map and a photometric loss determined from a warped image (Column 10 [lines 8-11] “A photometric loss is calculated based on the difference between the target image 504 and the warped image 512 (e.g., the warped source image that approximates the target image”; Column 11 [lines 15-19] “The warped depth estimate 614 and the source depth estimate 610 may be input to the residual pose network 604 which may minimize a geometric difference between the warped depth estimate 614 and the source depth estimate 610.”; Column 12 [lines 20-24] "the residual pose regression may provide an additional supervisory signal for both the pose network and the depth network. The learning of the pose network and the depth network may be simultaneously performed as a combination.")
Therefore, it would have been obvious to one of ordinary skill of the art before the effective filing date to modify Dong in view of Godard, and further in view of Kumar to include comprising a geometric consistency loss determined from a warped depth map and a photometric loss determined from a warped image taught by Tang’s reference. The motivation for doing so would have been to improve the accuracy of the warped image as suggested by Tang (see Tang, column 11 [lines 23-28]).
Further, one skilled in the art could have combined the elements described above by known methods with no change to the respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Tang with Dong, Godard, and Kumar to obtain the invention specified in claim 23.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Guizilini et al. (US 2021/0004974 A1) discloses a system for training a depth model using training data to generate a self-supervised loss from a pair of training images and a supervised loss from the depth data.
Zhu et al. (US 10,552,979 B2) discloses a method for performing data alignment among sensors by collecting a pair of consecutive images in the image data to recognize pixels corresponding to a same point in the point clouds; and establishing an optical flow for visual odometry.
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to UROOJ FATIMA whose telephone number is (571)272-2096. The examiner can normally be reached M-F 8:00-5:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Henok Shiferaw can be reached at (571) 272-4637. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/UROOJ FATIMA/Examiner, Art Unit 2676
/Henok Shiferaw/Supervisory Patent Examiner, Art Unit 2676