DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Status of Claims
Applicant’s Amendments filed on 04/21/2026 has been entered and made of record.
Currently pending Claim(s):
Amended claim(s):
1-21
6, 12, 16-17, and 21
Response to Arguments
This office action is responsive to Applicant’s Arguments/Remarks made in an Amendment received on 04/21/2026.
In view of new claim amendments and applicant arguments, Remarks filed on 04/21/2026, the claim objections to claims 12 and 16 are withdrawn
In view of applicant Argument/Remarks filed 04/21/2026, with respect to claims 1, 6-13, 15, 16-19, and 21, U.S.C. 112(f) claim interpretation have been carefully considered and the arguments are found to be not persuasive. Although, the Applicant argues on Page 9 that structural support for such modules may be found throughout the present application and drawings, such language is not recited in the claims, and as such, the claims continue to be treated under 112(f).
Applicant’s arguments, see Pages 9-14, filed on 04/21/2026, with respect to the rejection(s) of independent claims 1, 6, and 21 under 35 U.S.C. 103 have been fully considered and are persuasive. Therefore, the rejection has been withdrawn. However, upon further consideration, a new grounds of rejection is made in view of Dong et al. ("PPEA-Depth: Progressive Parameter-Efficient Adaptation for Self-Supervised Monocular Depth Estimation." arXiv preprint arXiv:2312.13066 (2023).) and Godard et al. (US 11,991,342 B2).
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action.
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because the claim limitation(s) uses a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier. Such claim limitation(s) is/are:
“encoder module” in claims 1, 6, and 21.
“first decoder module” in claims 1, 6, and 21.
“second decoder module” in claim 1, 6, and 21.
“a training module” in claims 6-11, 15, and 18.
“a warping module” in claims 12, 13, and 16.
“loss module” in claims 16, 17 and 19.
Because this/these claim limitation(s) is/are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, it/they is/are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof.
Claims 1, 6, and 21: “encoder module” corresponds to figure 12, element 230 “A pair of images (denoted I and I′ in FIG. 12) are input to the model 130. The model 130 includes first and second encoders (modules) 1204 and 1208 corresponding to and function as described above regarding the encoder modules 230” (Application Pub, paragraph [0176])
Claims 1, 6, and 21: “first decoder module” corresponds to figure 12, element 1212 “The model 130 includes first and second decoders (modules) 1212 and 1216 corresponding to the decoder module 270. The first and second decoders 1212 and 1216 generate depth maps (D and D′) based on the representations output by the first and second encoders 1204 and 1208, respectively. The depth map D′ is for the image I′, and the depth map D is for the image I.” (Application Pub, paragraph [0177]).
Claims 1, 6, and 21: “second decoder module” corresponds to figure 12, element 1216 “The model 130 includes first and second decoders (modules) 1212 and 1216 corresponding to the decoder module 270. The first and second decoders 1212 and 1216 generate depth maps (D and D′) based on the representations output by the first and second encoders 1204 and 1208, respectively. The depth map D′ is for the image I′, and the depth map D is for the image I.” (Application Pub, paragraph [0177]).
Claims 6-11, 15, and 18: “a training module” corresponds to figure 1, element 50 “FIG. 1 includes a functional block diagram of an example implementation of the herein described two-phase training process performed by a training module 50 for training a machine learning model. The training starts with the training module 50 performing an unsupervised pre-training 110 of a pretext learning model 130 on the pretext task.” (Application Pub, paragraph [0058]).
Claims 12, 13, and 16: “a warping module” corresponds to figure 18, element 1224 “At 1820, the warping module 1224 performs the warping described above, and the loss module 1312 determines the losses as described above including Lself. In various implementations, the warping module 112 and the loss module 1312 may be implemented in the training module 50.” (Application Pub, paragraph [0211]).
Claims 16, 17 and 19: “loss module” corresponds to figure 18, element 1312 “At 1820, the warping module 1224 performs the warping described above, and the loss module 1312 determines the losses as described above including Lself. In various implementations, the warping module 112 and the loss module 1312 may be implemented in the training module 50.” (Application Pub, paragraph [0211]).
Dependent claims 2-5, 14, and 20 are similarly interpreted for their dependency.
If applicant does not intend to have this/these limitation(s) interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation(s) to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation(s) recite(s) sufficient structure to perform the claimed function so as to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 4, 6-8, 10, 15, and 21 are rejected under 35 U.S.C. 103 as being unpatentable over Dong et al. ("PPEA-Depth: Progressive Parameter-Efficient Adaptation for Self-Supervised Monocular Depth Estimation." arXiv preprint arXiv:2312.13066 (2023).) (hereinafter, “Dong”) in view of Godard et al. (US 11,991,342 B2) (hereinafter, “Godard”).
Regarding claim 1, Dong discloses a non-transitory computer readable medium storing a computer model, the computer model comprising (Page 3 Figure 2 “(a) Depth network is a U-Net structure predicting depth taking three consecutive frames. (b) Pose network regresses the camera relative pose given two images. (c) Adapter is a bottleneck structure with a skip connection. (d) Structure of RepLKNet (Ding et al. 2022) backbone. (e) Our encoder adapter design. We attach encoder adapters to pre-trained RepLKBlock and ConvFFN. (f) Our decoder adapter design. Lerp represents linear interpolation.”; Examiner interprets the disclosed network to be implemented through a stored model, as such networks require software instructions for training and execution):
an encoder module (RepLKNet (a CNN architecture featuring a notable kernel size of 31x31, as the encoder backbone) on Page 3 right column Subsection 3.3 first 2 paragraphs equates to an encoder module) configured to encode first and second images (three consecutive frames on Page 3 left column Subsection 3.1 second paragraph equates to first and second images; Figure 2 (d) illustrates image being inputted to encoder) into first and second representations (feature maps of four different scales on Page 3 right column Subsection 3.3 first 2 paragraphs equate to first and second representations) (Page 3 left column Subsection 3.1 second paragraph “Our network takes three consecutive frames from a monocular video as input.”; Page 3 right column Subsection 3.3 first 2 paragraphs “We opt for RepLKNet (Ding et al. 2022), a CNN architecture featuring a notable kernel size of 31 × 31, as the encoder backbone. This selection is attributed to its adaptability concerning input image resolution, comparable accuracy to Swin Transformer (Liu et al. 2021), and enhanced inference speed when applied to downstream tasks. As illustrated in Fig. 2(d), RepLKNet generates feature maps of four different scales: F 1, F 2, F 3, F 4 at four stages.”), respectively,
Figure 2
PNG
media_image1.png
525
1372
media_image1.png
Greyscale
the first and second images being from consecutive frames from video (Page 3 left column Subsection 3.1 second paragraph “Our network takes three consecutive frames from a monocular video as input.”);
a first decoder module (depth network on Page 3 left Column Subsection 3.1 first paragraph equates to the first decoder) configured to decode the first and second representations and generate [first and second] depth maps for the images based on the first and second representations (Page 3 left Column Subsection 3.1 first paragraph “Our network comprises a depth network (Fig. 2(a)) and a pose network (Fig. 2(b)). The depth network employs a U-Net structure, encompassing an encoder to extract image features and a decoder to predict dense depth maps.”; Page 4 right column Subsection 4,2 second paragraph “PPEA-Depth adopts the well-established multi-frame inference and teacher-student distillation training scheme…The main network contains a cost volume construction process, using both the current frame It and its preceding frame It−1 to predict depth Dt.”; Examiner interprets the extracted image features to be the extracted features of the current frame and the preceding frame which equate to the first and second representation), respectively; and
a second decoder module (pose network Page 3 left column Subsection 3.1 first paragraph equates to a second decoder) configured to determine a six degree of freedom pose translation of a camera that captured the video based on the [first and second representations] (Page 3 left column Subsection 3.1 first paragraph “the pose network predicts the camera transformation between two frames. It has a feature extractor followed by a prediction head, which outputs a six-dimensional vector – three for rotation angles and the other three for translation.”).
However, Dong fails to teach a first and second [depth maps] and [pose…based on the] first and second representations.
Godard teaches a first and second [depth maps] (Column 14 [lines 9-11] “the depth-pose hybrid model 455 takes a plurality of input images and feeds each through the depth encoder 470 to extract abstract depth features.”) and [pose translation…based on the] first and second representations (Column 14 [lines 11-18] “The abstract depth features from each input image are then concatenated together prior to being input into the pose decoder 480 resulting in a pose for each of the input images or the relative transformations between two subsequent input images. The depth-pose hybrid model 455 is more computationally efficient than the pose estimation model 440 in extracting a pose for each pair of the input images.”).
Therefore, it would have been obvious to one of ordinary skill of the art before the effective filing date to modify Dong’s reference to include a first and second [depth maps] and [pose…based on the] first and second representations taught by Godard’s reference. The motivation for doing so would have been to reduce the overall computation time by sharing the parameters between the models as suggested by Godard (see Godard, Column 13 [lines 58-60]).
Further, one skilled in the art could have combined the elements described above by known methods with no change to the respective functions, and the combination would have yielded nothing more that predictable results. Therefore, it would have been obvious to combine Godard with Dong to obtain the invention specified in claim 1.
Regarding claim 4, which claim 1 is incorporated, Dong discloses wherein the encoder module includes adapter modules trained based on minimizing a photometric loss (Page 3 left column subsection 3.1 second paragraph “The middle frame is reconstructed with its adjacent frames, and the difference between the re-constructed and original images serves as the supervision signal.”; Page 13 left column Subsection D.2 third paragraph “During the training process, the teacher network is super-vised by the image reprojection loss (as mentioned in Section 3.1 in the main paper).”).
Regarding claim 6, Dong discloses a system comprising:
a model including (Page 3 left column Subsection 3.2 first paragraph “Our network comprises a depth network (Fig. 2(a)) and a pose network (Fig. 2(b)).”):
an encoder module (RepLKNet (a CNN architecture featuring a notable kernel size of 31x31, as the encoder backbone) on Page 3 right column Subsection 3.3 first 2 paragraphs equates to an encoder module) configured to encode first and second images (three consecutive frames on Page 3 left column Subsection 3.1 second paragraph equates to first and second images; Figure 2 (d) illustrates image being inputted to encoder) into first and second representations (feature maps of four different scales on Page 3 right column Subsection 3.3 first 2 paragraphs equate to first and second representations) (Page 3 left column Subsection 3.1 second paragraph “Our network takes three consecutive frames from a monocular video as input.”; Page 3 right column Subsection 3.3 first 2 paragraphs “We opt for RepLKNet (Ding et al. 2022), a CNN architecture featuring a notable kernel size of 31 × 31, as the encoder backbone. This selection is attributed to its adaptability concerning input image resolution, comparable accuracy to Swin Transformer (Liu et al. 2021), and enhanced inference speed when applied to downstream tasks. As illustrated in Fig. 2(d), RepLKNet generates feature maps of four different scales: F 1, F 2, F 3, F 4 at four stages.”), respectively;
Figure 2
PNG
media_image1.png
525
1372
media_image1.png
Greyscale
a first decoder module (depth network on Page 3 left Column Subsection 3.1 first paragraph equates to the first decoder) configured to decode the first and second representations and generate [first and second] depth maps for the images based on the first and second representations (Page 3 left Column Subsection 3.1 first paragraph “Our network comprises a depth network (Fig. 2(a)) and a pose network (Fig. 2(b)). The depth network employs a U-Net structure, encompassing an encoder to extract image features and a decoder to predict dense depth maps.”; Page 4 right column Subsection 4,2 second paragraph “PPEA-Depth adopts the well-established multi-frame inference and teacher-student distillation training scheme…The main network contains a cost volume construction process, using both the current frame It and its preceding frame It−1 to predict depth Dt.”; Examiner interprets the extracted image features to be the extracted features of the current frame and the preceding frame which equate to the first and second representation), respectively; and
a second decoder module (pose network Page 3 left column Subsection 3.1 first paragraph equates to a second decoder) configured to determine a six degree of freedom pose translation of a camera that captured the first and second images based on [the first and second representations] (Page 3 left column Subsection 3.1 first paragraph “the pose network predicts the camera transformation between two frames. It has a feature extractor followed by a prediction head, which outputs a six-dimensional vector – three for rotation angles and the other three for translation.”); and
a training module configured to: train the model using pairs of images, each pair of images including at least part of a same scene and captured at different times (Page 3 left column Subsection 3.1 second paragraph “Our network takes three consecutive frames from a monocular video as input. The middle frame is reconstructed with its adjacent frames, and the difference between the re-constructed and original images serves as the supervision signal.”); and
train parameters of adapter modules (adapter on Page 4 left column Subsection 3.5 equates to adapter modules) of the encoder module using consecutive frames of monocular video based on depth maps and [pose translations] determined by the model based on the consecutive frames of monocular video (Page 4 left column Subsection 3.5 Second paragraph “In Stage 2, we load the weights of the U-Net encoder, the encoder adapters, and the U-Net decoder from Stage 1, and freeze both the encoder and decoder, with only adapter parameters being updated. This paradigm preserves the depth perception ability obtained from Stage 1, as most network parameters are frozen and are unaffected by the erroneous loss caused by object motion.”).
However, Dong fails to teach a first and second [depth maps], [pose…based on the] first and second representations, and [train parameters…] based on pose translations.
Godard teaches a first and second [depth maps] (Column 14 [lines 9-11] “the depth-pose hybrid model 455 takes a plurality of input images and feeds each through the depth encoder 470 to extract abstract depth features.”) , [pose translation…based on the] first and second representations (Column 14 [lines 11-18] “The abstract depth features from each input image are then concatenated together prior to being input into the pose decoder 480 resulting in a pose for each of the input images or the relative transformations between two subsequent input images. The depth-pose hybrid model 455 is more computationally efficient than the pose estimation model 440 in extracting a pose for each pair of the input images.”), and [train parameters…] based on pose translations (Column 12 [lines 40-43] “the depth estimation training system 170 includes depth and pose models 175, an image synthesis module 180, an error calculation module 185, an appearance matching loss module”).
Therefore, it would have been obvious to one of ordinary skill of the art before the effective filing date to modify Dong’s reference to include a first and second [depth maps], [pose…based on the] first and second representations, and [train parameters…] based on pose translations taught by Godard’s reference. The motivation for doing so would have been to reduce the overall computation time by sharing the parameters between the models as suggested by Godard (see Godard, Column 13 [lines 58-60]).
Further, one skilled in the art could have combined the elements described above by known methods with no change to the respective functions, and the combination would have yielded nothing more that predictable results. Therefore, it would have been obvious to combine Godard with Dong to obtain the invention specified in claim 6.
Regarding claim 7, which claim 6 is incorporated, Dong discloses wherein the training module is configured to train the parameters of the adapter modules after training the model using the pairs of images (Page 3 left column Subsection 3.1 “Given camera intrinsics K and camera relative pose T between two frames a, b from a video, the pixel correspondence between them can be computed”; Page 4 left column Subsection 3.5 first 2 paragraphs “our progressive adaptation involves two stages. Stage 1 is trained on a dataset that primarily follows the static-scene assumption…Stage 2 is conducted on datasets that predominantly feature dynamic scenes… In Stage 2, we load the weights of the U-Net encoder, the encoder adapters, and the U-Net decoder from Stage 1, and freeze both the encoder and decoder, with only adapter parameters being up-dated.” Page 4, right column Subsection 4.1 first 2 paragraphs “.
Regarding claim 8, which claim 6 is incorporated, Dong discloses wherein the training module is configured to train the parameters of the adapter modules without annotations for the frames of the monocular video (Page 2 left column paragraph 3 “We propose an innovative two-stage self-supervised depth estimation approach, PPEA-Depth, based on parameter-efficient adaptation. We devise lightweight encoder adapters and decoder adapters within our framework.”).
Regarding claim 10, which claim 6 is incorporated, Dong discloses wherein the training module is configured to train the parameters of the adapter modules while all other parameters of the model are fixed (Page 4 left column Subsection 3.5 Second paragraph “In Stage 2, we load the weights of the U-Net encoder, the encoder adapters, and the U-Net decoder from Stage 1, and freeze both the encoder and decoder, with only adapter parameters being updated. This paradigm preserves the depth perception ability obtained from Stage 1, as most network parameters are frozen and are unaffected by the erroneous loss caused by object motion.”)..
Regarding claim 15, which claim 6 is incorporated, Dong discloses wherein the training module is configured to train the parameters of the adapter modules based on minimizing a photometric loss (Page 3 left column subsection 3.1 second paragraph “The middle frame is reconstructed with its adjacent frames, and the difference between the re-constructed and original images serves as the supervision signal.”; Page 13 left column Subsection D.2 third paragraph “During the training process, the teacher network is super-vised by the image reprojection loss (as mentioned in Section 3.1 in the main paper).”).
Regarding claim 21, Dong discloses a method, comprising:
train a model using pairs of images, each pair of images including at least part of a same scene and captured at different times (Page 3 left column Subsection 3.1 second paragraph “Our network takes three consecutive frames from a monocular video as input. The middle frame is reconstructed with its adjacent frames, and the difference between the re-constructed and original images serves as the supervision signal.”),
the model including (Page 3 left column Subsection 3.2 first paragraph “Our network comprises a depth network (Fig. 2(a)) and a pose network (Fig. 2(b)).”):
an encoder module (RepLKNet (a CNN architecture featuring a notable kernel size of 31x31, as the encoder backbone) on Page 3 right column Subsection 3.3 first 2 paragraphs equates to an encoder module) configured to encode first and second images (three consecutive frames on Page 3 left column Subsection 3.1 second paragraph equates to first and second images; Figure 2 (d) illustrates image being inputted to encoder) into first and second representations (feature maps of four different scales on Page 3 right column Subsection 3.3 first 2 paragraphs equate to first and second representations) (Page 3 left column Subsection 3.1 second paragraph “Our network takes three consecutive frames from a monocular video as input.”; Page 3 right column Subsection 3.3 first 2 paragraphs “We opt for RepLKNet (Ding et al. 2022), a CNN architecture featuring a notable kernel size of 31 × 31, as the encoder backbone. This selection is attributed to its adaptability concerning input image resolution, comparable accuracy to Swin Transformer (Liu et al. 2021), and enhanced inference speed when applied to downstream tasks. As illustrated in Fig. 2(d), RepLKNet generates feature maps of four different scales: F 1, F 2, F 3, F 4 at four stages.”), respectively;
Figure 2
PNG
media_image1.png
525
1372
media_image1.png
Greyscale
a first decoder module (depth network on Page 3 left Column Subsection 3.1 first paragraph equates to the first decoder) configured to decode the first and second representations and generate [first and second] depth maps for the images based on the first and second representations (Page 3 left Column Subsection 3.1 first paragraph “Our network comprises a depth network (Fig. 2(a)) and a pose network (Fig. 2(b)). The depth network employs a U-Net structure, encompassing an encoder to extract image features and a decoder to predict dense depth maps.”; Page 4 right column Subsection 4,2 second paragraph “PPEA-Depth adopts the well-established multi-frame inference and teacher-student distillation training scheme…The main network contains a cost volume construction process, using both the current frame It and its preceding frame It−1 to predict depth Dt.”; Examiner interprets the extracted image features to be the extracted features of the current frame and the preceding frame which equate to the first and second representation), respectively; and
a second decoder (pose network Page 3 left column Subsection 3.1 first paragraph equates to a second decoder) module configured to determine a six degree of freedom pose translation of a camera that captured the first and second images based on the first and second representations (Page 3 left column Subsection 3.1 first paragraph “the pose network predicts the camera transformation between two frames. It has a feature extractor followed by a prediction head, which outputs a six-dimensional vector – three for rotation angles and the other three for translation.”); and
training parameters of adapter modules (adapters on Page 4 left column Subsection 3.5 equate to adaptor modules) of the encoder module using consecutive frames of monocular video based on depth maps [and pose translations] determined by the model based on the consecutive frames of monocular video (Page 4 left column Subsection 3.5 Second paragraph “In Stage 2, we load the weights of the U-Net encoder, the encoder adapters, and the U-Net decoder from Stage 1, and freeze both the encoder and decoder, with only adapter parameters being updated. This paradigm preserves the depth perception ability obtained from Stage 1, as most network parameters are frozen and are unaffected by the erroneous loss caused by object motion.”).
However, Dong fails to teach a first and second [depth maps], [pose…based on the] first and second representations, and [training parameters…] based on pose translations.
Godard teaches a first and second [depth maps] (Column 14 [lines 9-11] “the depth-pose hybrid model 455 takes a plurality of input images and feeds each through the depth encoder 470 to extract abstract depth features.”) , [pose translation…based on the] first and second representations (Column 14 [lines 11-18] “The abstract depth features from each input image are then concatenated together prior to being input into the pose decoder 480 resulting in a pose for each of the input images or the relative transformations between two subsequent input images. The depth-pose hybrid model 455 is more computationally efficient than the pose estimation model 440 in extracting a pose for each pair of the input images.”), and [training parameters…] based on pose translations (Column 12 [lines 40-43] “the depth estimation training system 170 includes depth and pose models 175, an image synthesis module 180, an error calculation module 185, an appearance matching loss module”).
Therefore, it would have been obvious to one of ordinary skill of the art before the effective filing date to modify Dong’s reference to include a first and second [depth maps], [pose…based on the] first and second representations, and [training parameters…] based on pose translations taught by Godard’s reference. The motivation for doing so would have been to reduce the overall computation time by sharing the parameters between the models reduce the overall computation time by sharing the parameters from the pose estimation model and the depth model as suggested by Godard (see Godard, Column 13 [lines 58-60]).
Further, one skilled in the art could have combined the elements described above by known methods with no change to the respective functions, and the combination would have yielded nothing more that predictable results. Therefore, it would have been obvious to combine Godard with Dong to obtain the invention specified in claim 21.
Claims 2, 9 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Dong et al. ("PPEA-Depth: Progressive Parameter-Efficient Adaptation for Self-Supervised Monocular Depth Estimation." arXiv preprint arXiv:2312.13066 (2023).) (hereinafter, “Dong”) in view of Godard et al. (US 11,991,342 B2) (hereinafter, “Godard”), and further in view of Wofk et al (US 2022/0343521 A1) (hereinafter, “Wofk”).
Regarding claim 2, which claim 1 is incorporated, Dong discloses wherein the first (depth network on Page 3 left Column Subsection 3.1 first paragraph equates to the first decoder) and second decoder modules (pose network Page 3 left column Subsection 3.1 first paragraph equates to a second decoder) [include dense prediction transformer (DPT) decoders] “(Page 3 left Column Subsection 3.1 first paragraph “Our network comprises a depth network (Fig. 2(a)) and a pose network (Fig. 2(b)). The depth network employs a U-Net structure, encompassing an encoder to extract image features and a decoder to predict dense depth maps.”).
However, both Dong and Godard fail to teach [wherein the first and second decoder modules] include dense prediction transformer (DPT) decoders.
Wofk teaches [wherein the first and second decoder modules] include dense prediction transformer (DPT) decoders (Paragraph [0046] “in addition to using a DPT-Hybrid depth estimator, a DPT-Large depth estimator can be used for higher depth estimation accuracy while a machine-learning framework such as MiDaS-small can be selected for computational efficiency,”).
Therefore, it would have been obvious to one of ordinary skill of the art before the effective filing date to modify Dong in view of Godard to include [wherein the first and second decoder modules] include dense prediction transformer (DPT) decoders taught by Wofk’s reference. The motivation for doing so would have been to obtain higher depth estimation accuracy as suggested by Wofk (see Wofk, paragraph [0046]).
Further, one skilled in the art could have combined the elements described above by known methods with no change to the respective functions, and the combination would have yielded nothing more that predictable results. Therefore, it would have been obvious to combine Wofk with Godard and Dong to obtain the invention specified in claim 2.
Regarding claim 9, which claim 6 is incorporated, Dong discloses wherein the adapter modules include an up projection module, [a rectified linear unit (ReLU)], and a down projection module (Page 3 right column first paragraph “adapters follow a bottleneck structure, encompassing two linear projection layers, an activation layer, and a skip connection. The initial projection layer reduces the input feature dimension, and the subsequent one restores it to the original input dimension after the activation layer.”), and
wherein the training module is configured to train parameters of at least one of the up projection module, the ReLU, and the down projection module based on the depth maps and pose translations determined by the model based on the consecutive frames of monocular video (Page 4 left column Subsection 3.5 Second paragraph “In Stage 2, we load the weights of the U-Net encoder, the encoder adapters, and the U-Net decoder from Stage 1, and freeze both the encoder and decoder, with only adapter parameters being updated. This paradigm preserves the depth perception ability obtained from Stage 1, as most network parameters are frozen and are unaffected by the erroneous loss caused by object motion.”).
However, Dong and Godard both fail to teach a rectified linear unit (ReLU).
Wofk teaches a rectified linear unit (ReLU) (Paragraph [0053] “the scale map learner circuitry 315 regresses a dense scale residual map r where values can be negative. In examples disclosed herein, the resulting scale map can be represented as ReLU (1+r) and applied to the input depth {tilde over (z)} to produce the output depth {circumflex over (z)}=ReLU (1+r){tilde over (z)}.”).
Therefore, it would have been obvious to one of ordinary skill of the art before the effective filing date to modify Dong in view of Godard to include a rectified linear unit (ReLU) taught by Wofk’s reference. The motivation for doing so would have been to account for negative values as suggested by Wofk (see Wofk, paragraph [0073]).
Further, one skilled in the art could have combined the elements described above by known methods with no change to the respective functions, and the combination would have yielded nothing more that predictable results. Therefore, it would have been obvious to combine Wofk with Godard and Dong to obtain the invention specified in claim 9.
Regarding claim 20 (drawn to a system), claim 20 is rejected the same as claim 2 and the arguments similar to that presented above for claim 2 are equally applicable to the claim 20, and all the other limitations similar to claim 2 are not repeated herein, but incorporated by reference.
Claims 3, 11-14, and 16 are rejected under 35 U.S.C. 103 as being unpatentable over Dong et al. ("PPEA-Depth: Progressive Parameter-Efficient Adaptation for Self-Supervised Monocular Depth Estimation." arXiv preprint arXiv:2312.13066 (2023).) (hereinafter, “Dong”) in view of Godard et al. (US 11,991,342 B2) (hereinafter, “Godard”), and further in view of Tang et al. (US 11,727,588 B2) (hereinafter, “Tang”).
Regarding claim 3, which claim 1 is incorporated, Dong discloses wherein the encoder module includes adapter modules trained [based on minimizing a geometric consistency loss] (Page 4 left column Subsection 3.5 Second paragraph “In Stage 2, we load the weights of the U-Net encoder, the encoder adapters, and the U-Net decoder from Stage 1, and freeze both the encoder and decoder, with only adapter parameters being updated. This paradigm preserves the depth perception ability obtained from Stage 1, as most network parameters are frozen and are unaffected by the erroneous loss caused by object motion.”)
However, Dong and Godard both fail to teach based on minimizing a geometric consistency loss.
Tang teaches based on minimizing a geometric consistency loss (Column 11 [lines 15-19] “The warped depth estimate 614 and the source depth estimate 610 may be input to the residual pose network 604 which may minimize a geometric difference between the warped depth estimate 614 and the source depth estimate 610.”).
Therefore, it would have been obvious to one of ordinary skill of the art before the effective filing date to modify Dong in view of Godard to include minimizing a geometric consistency loss taught by Tang’s reference. The motivation for doing so would have been to improve the accuracy of the warped image as suggested by Tang (see Tang, column 11 [lines 23-28]).
Further, one skilled in the art could have combined the elements described above by known methods with no change to the respective functions, and the combination would have yielded nothing more that predictable results. Therefore, it would have been obvious to combine Tang with Godard and Dong to obtain the invention specified in claim 3.
Regarding claim 11, which claim 6 is incorporated, Dong discloses wherein the training module is configured to train the parameters of the adapter modules [based on minimizing a geometric consistency loss] (Page 4 left column Subsection 3.5 Second paragraph “In Stage 2, we load the weights of the U-Net encoder, the encoder adapters, and the U-Net decoder from Stage 1, and freeze both the encoder and decoder, with only adapter parameters being updated. This paradigm preserves the depth perception ability obtained from Stage 1, as most network parameters are frozen and are unaffected by the erroneous loss caused by object motion.”)
However, Dong and Godard both fail to teach based on minimizing a geometric consistency loss.
Tang teaches based on minimizing a geometric consistency loss (Column 11 [lines 15-19] “The warped depth estimate 614 and the source depth estimate 610 may be input to the residual pose network 604 which may minimize a geometric difference between the warped depth estimate 614 and the source depth estimate 610.”).
Therefore, it would have been obvious to one of ordinary skill of the art before the effective filing date to modify Dong in view of Godard to include minimizing a geometric consistency loss taught by Tang’s reference. The motivation for doing so would have been to improve the accuracy of the warped image as suggested by Tang (see Tang, column 11 [lines 23-28]).
Further, one skilled in the art could have combined the elements described above by known methods with no change to the respective functions, and the combination would have yielded nothing more that predictable results. Therefore, it would have been obvious to combine Tang with Godard and Dong to obtain the invention specified in claim 11.
Regarding claim 12, which claim 11 is incorporated, Dong and Godard fail to teach a warping module configured to generate a warped depth map based on the first depth map; and a loss module configured to determine the geometric consistency loss based on differences between the warped depth map and the first depth map.
Tang teaches a warping module configured to generate a warped depth map based on the first depth map (Column 11 [lines 7-15] “a warped depth estimate 614 of the target depth estimate 612 may be generated based on the pose estimate 606. The warped depth estimate 614 may be computed by transforming the target depth estimate 612 according to the pose estimate 602. The transformation module (not shown) may warp the target depth estimate 612.”); and
a loss module configured to determine the geometric consistency loss based on differences between the warped depth map and the first depth map (Column 11 [lines 15-19] “The warped depth estimate 614 and the source depth estimate 610 may be input to the residual pose network 604 which may minimize a geometric difference between the warped depth estimate 614 and the source depth estimate 610.”).
Therefore, it would have been obvious to one of ordinary skill of the art before the effective filing date to modify Dong in view of Godard to include a warping module configured to generate a warped depth map based on the first depth map; and a loss module configured to determine the geometric consistency loss based on differences between the warped depth map and the first depth map taught by Tang’s reference. The motivation for doing so would have been to improve the accuracy of the warped image as suggested by Tang (see Tang, column 11 [lines 23-28]).
Further, one skilled in the art could have combined the elements described above by known methods with no change to the respective functions, and the combination would have yielded nothing more that predictable results. Therefore, it would have been obvious to combine Tang with Godard and Dong to obtain the invention specified in claim 12.
Regarding claim 13, which claim 12 is incorporated, Dong and Godard fail to teach wherein the warping module is configured to generate the warped depth map further based on the pose translation.
Tang teaches wherein the warping module is configured to generate the warped depth map further based on the pose translation (Column 11 [lines 7-15] “a warped depth estimate 614 of the target depth estimate 612 may be generated based on the pose estimate 606. The warped depth estimate 614 may be computed by transforming the target depth estimate 612 according to the pose estimate 602. The transformation module (not shown) may warp the target depth estimate 612.”).
Therefore, it would have been obvious to one of ordinary skill of the art before the effective filing date to modify Dong in view of Godard to include wherein the warping module is configured to generate the warped depth map further based on the pose translation taught by Tang’s reference. The motivation for doing so would have been to improve the accuracy of the warped image as suggested by Tang (see Tang, column 11 [lines 23-28]).
Further, one skilled in the art could have combined the elements described above by known methods with no change to the respective functions, and the combination would have yielded nothing more that predictable results. Therefore, it would have been obvious to combine Tang with Godard and Dong to obtain the invention specified in claim 13.
Regarding claim 14, which claim 13 is incorporated, Dong and Godard fail to teach wherein the warping module is configured to generate the warped depth map based on transforming the first depth map to a three dimensional space and projecting to the second image using the pose translation.
Tang teaches wherein the warping module is configured to generate the warped depth map based on transforming the first depth map to a three dimensional space and projecting to the second image using the pose translation (Column 9 [lines 19-22] “the depth map 508 may be a per-pixel depth map. A view estimation module 510 receives the per-pixel depth map 508 and the six DoF transformation (e.g., relative pose) between the target image 504 and the source image 506.”; Column 11 [lines6-19] “the pose estimate 602 may be used to estimate a residual pose estimate 606…a warped depth estimate 614 of the target depth estimate 612 may be generated based on the pose estimate 606. The warped depth estimate 614 may be computed by transforming the target depth estimate 612 according to the pose estimate 602. The transformation module (not shown) may warp the target depth estimate 612. The warped depth estimate 614 and the source depth estimate 610 may be input to the residual pose network 604 which may minimize a geometric difference between the warped depth estimate 614 and the source depth estimate 610.”; Column 12 [lines 14-17] “the determined residual pose accounts for geometric information and improves a consistency of depth estimates between the target and source views.”).
Therefore, it would have been obvious to one of ordinary skill of the art before the effective filing date to modify Dong in view of Godard to include wherein the warping module is configured to generate the warped depth map based on transforming the first depth map to a three dimensional space and projecting to the second image using the pose translation taught by Tang’s reference. The motivation for doing so would have been to improve the accuracy of the warped image as suggested by Tang (see Tang, column 11 [lines 23-28]).
Further, one skilled in the art could have combined the elements described above by known methods with no change to the respective functions, and the combination would have yielded nothing more that predictable results. Therefore, it would have been obvious to combine Tang with Godard and Dong to obtain the invention specified in claim 14.
Regarding claim 16, which claim 15 is incorporated, Dong and Godard fail to teach a warping module configured to generate a warped image based on the first image; and a loss module configured to determine the photometric consistency loss based on differences between the warped image and the first image.
Tang teaches a warping module configured to generate a warped image based on the first image (Column 6 [lines 58-62] “each individual pixel for the target image is warped according to its own depth and pose estimates to generate a reconstructed image (e.g., warped source image)”); and
a loss module configured to determine the photometric consistency loss based on differences between the warped image and the first image (Column 10 [lines 8-11] “A photometric loss is calculated based on the difference between the target image 504 and the warped image 512 (e.g., the warped source image that approximates the target image).”).
Therefore, it would have been obvious to one of ordinary skill of the art before the effective filing date to modify Dong in view of Godard to include a warping module configured to generate a warped image based on the first image; and a loss module configured to determine the photometric consistency loss based on differences between the warped image and the first image taught by Tang’s reference. The motivation for doing so would have been to generate a 3D representation of the target image and use the loss to update the network as suggested by Tang (see Tang, column 3 [lines 43-45] and column 10 [lines 13-15]).
Further, one skilled in the art could have combined the elements described above by known methods with no change to the respective functions, and the combination would have yielded nothing more that predictable results. Therefore, it would have been obvious to combine Tang with Godard and Dong to obtain the invention specified in claim 16.
Claims 5 and 18-19 are rejected under 35 U.S.C. 103 as being unpatentable over Dong et al. ("PPEA-Depth: Progressive Parameter-Efficient Adaptation for Self-Supervised Monocular Depth Estimation." arXiv preprint arXiv:2312.13066 (2023).) (hereinafter, “Dong”) in view of Godard et al. (US 11,991,342 B2) (hereinafter, “Godard”), and further in view of Guizilini et al. (US 11,386,567 B2) (hereinafter, “Guizilini”).
Regarding claim 5, which claim 1 is incorporated, Dong and Godard fail to teach wherein the encoder module includes adapter modules trained based on minimizing an edge smoothness loss.
Guizilini teaches wherein the encoder module includes adapter modules trained based on minimizing an edge smoothness loss (Paragraph [0046] “in addition to using a DPT-Hybrid depth estimator, a DPT-Large depth estimator can be used for higher depth estimation accuracy while a machine-learning framework such as MiDaS-small can be selected for computational efficiency,”).
Therefore, it would have been obvious to one of ordinary skill of the art before the effective filing date to modify Dong in view of Godard to include wherein the encoder module includes adapter modules trained based on minimizing an edge smoothness loss taught by Guizilini’s reference. The motivation for doing so would have been to modify the parameters of the models to perform the training as suggested by Guizilini (see Guizilini, Column 5 [lines 8-11]).
Further, one skilled in the art could have combined the elements described above by known methods with no change to the respective functions, and the combination would have yielded nothing more that predictable results. Therefore, it would have been obvious to combine Guizilini with Godard and Dong to obtain the invention specified in claim 5.
Regarding claim 18, which claim 6 is incorporated, Dong and Godard fail to teach wherein the training module is configured to train the parameters of the adapter modules based on minimizing an edge smoothness loss.
Guizilini teaches wherein the training module is configured to train the parameters of the adapter modules based on minimizing an edge smoothness loss (Column 13 [lines 15-20] “Ls represents depth smoothness loss and is implemented to regularize the depth in textureless low-image gradient regions, as shown in equation (5). The smoothness loss is an edge-aware term that is weighted for separate pyramid levels starting from 1 and decaying by a factor of two for the separate scales.)”.
Equation 5
PNG
media_image2.png
74
442
media_image2.png
Greyscale
Therefore, it would have been obvious to one of ordinary skill of the art before the effective filing date to modify Dong in view of Godard to include wherein the training module is configured to train the parameters of the adapter modules based on minimizing an edge smoothness loss taught by Guizilini’s reference. The motivation for doing so would have been to modify the parameters of the models to perform the training as suggested by Guizilini (see Guizilini, Column 5 [lines 8-11]).
Further, one skilled in the art could have combined the elements described above by known methods with no change to the respective functions, and the combination would have yielded nothing more that predictable results. Therefore, it would have been obvious to combine Guizilini with Godard and Dong to obtain the invention specified in claim 18.
Regarding claim 19, which claim 18 is incorporated, Dong and Godard fail to teach a loss module configured to determine the edge smoothness loss based on first derivatives of pixel values of the first and second depth maps.
Guizilini teaches a loss module configured to determine the edge smoothness loss based on first derivatives of pixel values of the first and second depth maps (Column 16 [lines 25-30] “the training module 230 uses this appearance-based loss as both the first stage loss to account for pixel-level similarities and irregularities along edge regions between a synthesized image derived from depth predictions of the depth model and a target image that is the original input into the depth model 260.”.)
Therefore, it would have been obvious to one of ordinary skill of the art before the effective filing date to modify Dong in view of Godard to include a loss module configured to determine the edge smoothness loss based on first derivatives of pixel values of the first and second depth maps taught by Guizilini’s reference. The motivation for doing so would have been to modify the parameters of the models to perform the training as suggested by Guizilini (see Guizilini, Column 5 [lines 8-11]).
Further, one skilled in the art could have combined the elements described above by known methods with no change to the respective functions, and the combination would have yielded nothing more that predictable results. Therefore, it would have been obvious to combine Guizilini with Godard and Dong to obtain the invention specified in claim 19.
Claim 17 is rejected under 35 U.S.C. 103 as being unpatentable over Dong et al. ("PPEA-Depth: Progressive Parameter-Efficient Adaptation for Self-Supervised Monocular Depth Estimation." arXiv preprint arXiv:2312.13066 (2023).) (hereinafter, “Dong”) in view of Godard et al. (US 11,991,342 B2) (hereinafter, “Godard”), further in view of Tang et al. (US 11,727,588 B2) (hereinafter, “Tang”), and Guizilini et al. (US 11,386,567 B2) (hereinafter, “Guizilini”).
Regarding claim 17, which claim 16 is incorporated, Dong, Godard and Tang fail to teach wherein the loss module is configured to determine the photometric loss based on downweighting regions of the first image including moving objects.
Guizilini teaches wherein the loss module is configured to determine the photometric loss based on downweighting regions of the first image including moving objects (Column 13 [lines 1-10] “the training module 230 masks out static pixels by removing pixels that have a warped photometric loss higher than a corresponding unwarped photometric loss, which the training module 230 calculates using the original source image (e.g., 620) without synthesizing the target. The mask (Mp) removes pixels that have appearance loss that does not change between frames, which includes static scenes and dynamic objects moving at a similar speed as the camera.”).
Therefore, it would have been obvious to one of ordinary skill of the art before the effective filing date to modify Dong in view of Godard, and further in view of Tang to include wherein the loss module is configured to determine the photometric loss based on downweighting regions of the first image including moving objects taught by Guizilini’s reference. The motivation for doing so would have been to remove pixels that have appearance loss as suggested by Guizilini (see Guizilini, Column 13 [lines 6-8]).
Further, one skilled in the art could have combined the elements described above by known methods with no change to the respective functions, and the combination would have yielded nothing more that predictable results. Therefore, it would have been obvious to combine Guizilini with Godard, Tang, and Dong to obtain the invention specified in claim 17.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Chakravarty et al. (US 2020/0041276 A1) discloses a variational autoencoder network configured to reconstruct pose vector data and depth maps from camera images, which are then used to perform localization and mapping of a vehicle.
Huang (US 2022/0358359 A1) discloses a jointly trained multi-task network including feature extraction, depth estimation, segmentation, and pose estimation modules. Wherein the pose network determines camera rotation and translation, and outputs optical flow based on the estimated motion information.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to UROOJ FATIMA whose telephone number is (571)272-2096. The examiner can normally be reached M-F 8:00-5:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Henok Shiferaw can be reached at (571) 272-4637. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/UROOJ FATIMA/Examiner, Art Unit 2676
/CHINEYERE WILLS-BURNS/Supervisory Patent Examiner, Art Unit 2673
Prosecution Insights