DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Receipt is acknowledged that application claims priority to foreign application with application number KR10-2023-0145795 dated 10/27/2023. Copies of certified papers required by 37 CFR 1.55 have been received. Priority is acknowledged under 35 USC 119(e) and 37 CFR 1.78.
Claim Interpretation
The claims in this application are given their broadest reasonable interpretation using the
plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification.
Under MPEP 2143.03, "All words in a claim must be considered in judging the patentability of that claim against the prior art." In re Wilson, 424 F.2d 1382, 1385, 165 USPQ 494, 496 (CCPA 1970). As a general matter, the grammar and ordinary meaning of terms as understood by one having ordinary skill in the art used in a claim will dictate whether, and to what extent, the language limits the claim scope. Language that suggests or makes a feature or step optional but does not require that feature or step does not limit the scope of a claim under the broadest reasonable claim interpretation. In addition, when a claim requires selection of an element from a list of alternatives, the prior art teaches the element if one of the alternatives is taught by the prior art. See, e.g., Fresenius USA, Inc. v. Baxter Int’l, Inc., 582 F.3d 1288, 1298, 92 USPQ2d 1163, 1171 (Fed. Cir. 2009).
Claim 7 recite “or” then listing “a difference between the first estimation depth and the second estimation depth as the first loss function; or skip updating, based on the first estimation depth not satisfying the third condition, the plurality of weights included in the MDE model for obtaining the second estimation depth.” Since “or” is disjunctive, any one of the elements found in the prior art is sufficient to reject the claim. While citations have been provided for completeness and rapid prosecution, only one element is required. Because, on balance, it appears the disjunctive interpretation enjoys the most specification support and for that reason the disjunctive interpretation (one of A, B OR C) is being adopted for the purposes of this Office Action. Applicant’s comments and/or amendments relating to this issue are invited to clarify the claim language and the prosecution history.
Claim 17 recite “or” then listing “a difference between the first estimation depth and the second estimation depth as the first loss function; or skip updating, based on the first estimation depth not satisfying the third condition, the plurality of weights included in the MDE model for obtaining the second estimation depth.” Since “or” is disjunctive, any one of the elements found in the prior art is sufficient to reject the claim. While citations have been provided for completeness and rapid prosecution, only one element is required. Because, on balance, it appears the disjunctive interpretation enjoys the most specification support and for that reason the disjunctive interpretation (one of A, B OR C) is being adopted for the purposes of this Office Action. Applicant’s comments and/or amendments relating to this issue are invited to clarify the claim language and the prosecution history.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1- 20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to non-statutory subject matter. When reviewing independent claims 1, 11 and 12 and based upon consideration of all of the relevant factors with respect to the claim as a whole, claims 1- 20 are held to claim an abstract idea without reciting elements that amount to significantly more than the abstract idea and is/are therefore rejected as ineligible subject matter under 35 U.S.C. 101. The Examiner will analyze Claim 1 and similar rationale applies to independent Claim/s 11 and 12. The rationale, under MPEP § 2106, for this finding is explained below:
The claimed invention (1) must be directed to one of the four statutory categories, and (2) must not be wholly directed to subject matter encompassing a judicially recognized exception, as defined below. The following two step analysis is used to evaluate these criteria.
Step 1: Is the claim directed to one of the four patent-eligible subject matter categories: process, machine, manufacture, or composition of matter?
When examining the claim under 35 U.S.C. 101, the Examiner interprets that the claims is related to a machine since the claim is directed to an apparatus.
Step 2a, Prong 1: Does the claim wholly embrace a judicially recognized exception, which includes laws of nature, physical phenomena, and abstract ideas, or is it a particular practical application of a judicial exception?
The Examiner interprets that the judicial exception applies since Claim 1 limitation of update, based on a loss function group applied to the MDE model, a plurality of weights included in the MDE model is directed to an abstract idea. The claim is related to mathematical relationships including generating a depth map, updating a model using a loss function, and outputting an updated model based on a depth map. Under Recentive Analytics, training a neural network on domain-specific data is an abstract idea. Recentive Analytics, Inc. v. Fox Corp., No. 23-2437, 134 F.4th 1205 (Fed. Cir. 2025). If the claim recites a judicial exception (i.e., an abstract idea enumerated in MPEP § 2106.04(a), a law of nature, or a natural phenomenon), the claim requires further analysis in Prong Two.
Step 2a, Prong 2: Does the claim recite additional elements that integrate the judicial exception into a practical application?
The Examiner interprets that Claim 1 limitation does not provide additional elements or combination of additional elements to a practical application since the claims are not adding insignificant extra-solution activity to the judicial exception. Since the claim is generally training a neural network on domain specific data. Specifically, the analysis method does not integrate a judicial exception into practical application. See Genetic Techs. v. Merial LLC, 818 F.3d 1369, 1376, 118 USPQ2d 1541, 1546 (Fed. Cir. 2016) (eligibility "cannot be furnished by the unpatentable law of nature (or natural phenomenon or abstract idea) itself."). For a claim reciting a judicial exception to be eligible, the additional elements (if any) in the claim must "transform the nature of the claim" into a patent-eligible application of the judicial exception, Alice Corp., 573 U.S. at 217, 110 USPQ2d at 1981, either at Prong Two or in Step 2B. The Examiner interprets that Claim 1 limitation does not provide additional elements or combination of additional elements to a practical application since the claims do not provide clear improvement to a technology or to computer functionality. Limiting the training to “a depth map” is a non-qualifying field-of-use limitation. Data gathering (obtaining an input image) and storage (memory storing instructions) are routine activities that do not add a practical application. Reciting an “monocular depth estimation (MDE) model” functionally does not reflect a specific technical improvement to the device. Since the claim is generally linking machine learning to the processing of the images. Specifically, the statement of using machine learning does not integrate an improvement to a technology or to computer functionality. See Genetic Techs. v. Merial LLC, 818 F.3d 1369, 1376, 118 USPQ2d 1541, 1546 (Fed. Cir. 2016) (eligibility "cannot be furnished by the unpatentable law of nature (or natural phenomenon or abstract idea) itself."). For a claim reciting a judicial exception to be eligible, the additional elements (if any) in the claim must "transform the nature of the claim" into a patent-eligible application of the judicial exception, For a claim reciting a judicial exception to be eligible, the additional elements (if any) in the claim must "transform the nature of the claim" into a patent-eligible application of the judicial exception, Alice Corp., 573 U.S. at 217, 110 USPQ2d at 1981, either at Prong Two or in Step 2B. If there are no additional elements in the claim, then it cannot be eligible. In such a case, after making the appropriate rejection, it is a best practice for the examiner to recommend an amendment, if possible, that would resolve eligibility of the claim.
Step 2b: If a judicial exception into a practical application is not recited in the claim, the Examiner must interpret if the claim recites additional elements that amount to significantly more than the judicial exception.
The Examiner interprets that the Claims do not amount to significantly more since the Claims state using machine learning with a high level of generality and the claims lack an inventive concept because the elements, considered individually and as an order combination, are well-understood, routine, and conventional (WURC) in the field. Further, the specification acknowledges that monocular depth estimation (MDE) model, loss functions, and estimated depth maps are known in the art. See, simply appending well-understood, routine, conventional activities previously known to the industry, specified at a high level of generality, to the judicial exception, e.g., a claim to an abstract idea requiring no more than a generic computer to perform generic computer functions that are well-understood, routine and conventional activities previously known to the industry, as discussed in Alice Corp., 573 U.S. at 225, 110 USPQ2d at 1984.
Claims 2-10 and 13-20 depending on the independent claim/s include all the limitation of the independent claim. The Examiner finds that Claim 2 and 13 adds mathematical depth tensor, which are further abstract mathematical concepts. This is seen as an abstract idea related to a mathematical process. The claim describes the further manipulation of data. See MPEP 2106.05(g). The Examiner finds that Claim 3 and 14 add “a minimum discretization value and a maximum discretization" which is a mathematical operation, which are further abstract mathematical concepts. This is seen as an abstract idea related to a mathematical process. The claim describes the further manipulation of data. See MPEP 2106.05(g). The Examiner finds that Claim 4 and 15 involve a mathematical interpretation of the depth tensor based on the discretization value. This is seen as an abstract idea related to a mathematical concept. The claim describes a mathematical augmentation of the discretization value; meaning it fails to integrate the abstract idea into a technical practical application. See MPEP 2106.05(g). The Examiner finds that Claim 5 involves adding a mathematical probability sum, which are further abstract mathematical concepts. This is seen as an abstract idea related to a mathematical concept. The claim describes summing the probability of the uncertainty values of the pixels, these are these are selecting a particular data source or type of data to be manipulated and do not impose meaningful limits on the judicial exception. See MPEP 2106.05(g). Claim 6 and 16 involve a mathematical calculation of loss function, which are further abstract mathematical concepts. This is seen as an abstract idea related to a mathematical concept. The claim describes finding the difference between two depth maps to determine a loss function; meaning it fails to integrate the abstract idea into a technical practical application. See MPEP 2106.05(g). Claim 7 and 17 involve a mathematical calculation of difference between loss estimation depths, which are further abstract mathematical concepts. This is seen as an abstract idea related to a mathematical concept. The claim describes finding the difference between two estimation depths to determine the loss function and next action; meaning it fails to integrate the abstract idea into a technical practical application. See MPEP 2106.05(g). Claim 8 and 18 brings in the variable of pose estimation, which is mere data gathering activity. This is seen as an abstract idea related to a mathematical concept. The claim describes obtaining pose change information based on the input image; meaning it fails to integrate the abstract idea into a technical practical application. See MPEP 2106.05(g). Claim 9 and 19 involve the output of a reconstruction image, which is obtained by applying further abstract mathematical concepts. This is seen as an abstract idea related to a mathematical concept. The claim describes generating a reconstruction image based off data points; meaning it fails to integrate the abstract idea into a technical practical application. See MPEP 2106.05(g). Claim 10 and 20 involve adding a mathematical mean value, which is obtained by applying further abstract mathematical concepts. This is seen as an abstract idea related to a mathematical concept. The claim describes obtaining a mean value associated with the depth maps; meaning it fails to integrate the abstract idea into a technical practical application. See MPEP 2106.05(g).
Thus, Claims 2-10 and 13-20 recite the same abstract idea and therefore are not drawn to the eligible subject matter as they are directed to the abstract idea without significantly more.
For the analogous independent claim 11 to the independent claim 1 and the analogous, the analogous limitations can be analyzed in the same way as above for the claim 1 hence rejected under 101. Moreover, the claim of 11 further recites at the same statutory category of “apparatus” which is a limitation that the examiner interprets that the claims is related to a machine since the claim is directed to a obtain a target depth estimation map by applying the target image to a monocular depth estimation model including updated weights, and is consistent with the abstract ideas, including generating a depth map, updating a model using a loss function, and outputting an updated model based on a depth map, the examiner interprets the addition of the “target image” as mere data gathering activity that would be well known in the art. Under Recentive Analytics, training a neural network on domain-specific data is an abstract idea. Recentive Analytics, Inc. v. Fox Corp., No. 23-2437, 134 F.4th 1205 (Fed. Cir. 2025). This limitation of system just further implement the abstract ideas to be performed by generic computer or software/hardware components of additional elements of the different types of analyzation methods. Therefore, the Examiner interprets that the claims are rejected under 35 U.S.C. 101.
For the analogous independent claim 12 to the independent claim 1 and the analogous limitations can be analyzed in the same way as above for the claim 1 hence rejected under 101. Moreover, the claim of 14 further recites a different statutory category of “system” which is a limitation that the examiner interprets that the claims is related to a process since the claim is directed to a method, and is consistent with the abstract ideas, including generating a depth map, updating a model using a loss function, and outputting an updated model based on a depth map. Under Recentive Analytics, training a neural network on domain-specific data is an abstract idea. Recentive Analytics, Inc. v. Fox Corp., No. 23-2437, 134 F.4th 1205 (Fed. Cir. 2025). This limitation of system just further implement the abstract ideas to be performed by generic computer or software/hardware components of additional elements of the different types of analyzation methods. Therefore, the Examiner interprets that the claims are rejected under 35 U.S.C. 101.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1, 11, and 12 are rejected under 35 U.S.C. 103 as unpatentable over Ambrus et al (US Patent Publication US 2023/0177850 A1 hereafter referred to as Ambrus) in view of Kim et al (US Patent Publication 2022/0036573 A1 hereafter referred to as Kim).
Regarding Claim 1, Ambrus teaches an apparatus (Ambrus ¶0022, ¶0109, discloses an apparatus implementing a method) comprising:
at least one processor (Ambrus ¶0007, ¶0028, ¶0030, ¶0035 discloses program executed by a processor or multiple processors);
and a memory storing instructions (Ambrus Fig 1 118 discloses a memory, ¶0028 discloses a memory storing task information), when executed by the at least one processor (Ambrus ¶0028, ¶0038, ¶0105 discloses the processor executing instructions), cause the apparatus to:
obtain, based on a depth map obtained from a cluster of points (Ambrus ¶0057, ¶0068, ¶0079 discloses the use of a point cloud for the input depth map) at a target time point (Ambrus ¶0067 discloses the use of the network across time to understand a space and ¶0055 discloses multiple frames of input image coming over a video stream which happens over a period of time),
obtain, based on an input image (Ambrus ¶0074, ¶0083, ¶0088 discloses an input image input into a that is input into monocular depth prediction network) associated with the target time point (Ambrus ¶0067 discloses the use of the network across time to understand a space and ¶0055 discloses multiple frames of input image coming over a video stream which happens over a period of time) and that is applied to a monocular depth estimation (MDE) model (Ambrus ¶0074, ¶0083, ¶0088 discloses an input image input into a that is input into monocular depth prediction network), a depth estimation map (Ambrus ¶0073, ¶0069 discloses a depth estimation model included in the monocular depth);
update, based on a loss function group (Ambrus Fig 10, 1004 discloses updating the network with a regression loss) applied to the MDE model (Ambrus ¶0074, ¶0083, ¶0088 discloses an input image input into a that is input into monocular depth prediction network), a plurality of weights included in the MDE model (Ambrus ¶0028 discloses the weights within the model), wherein the loss function group (Ambrus Fig 10, 1004 discloses updating the network with a regression loss) comprises a first loss function (Ambrus Fig 10, 1004 discloses updating the network with a regression loss) and the depth estimation map (Ambrus Fig 9 922 discloses the depth regression loss being obtained from the estimated depth); and
output a signal (Ambrus ¶0024, ¶0053 discloses outputting an uncertainty output for each pixel after the network update) indicating the updated plurality of weights (Ambrus Fig 10, 1004 discloses updating the network with a regression loss, ¶0028 discloses the weights within the model).
Ambrus does not explicitly disclose a depth distribution map, that is obtained based on the depth distribution map.
Kim is in the same field of image analysis for use scene reconstruction for automated driving. Further, Kim teaches a depth distribution map (Kim Fig 7, ¶0016, ¶0021, discloses a depth probability distribution of each of the pixels), that is obtained based on the depth distribution map (Kim Fig 7, ¶0016, ¶0021, discloses a depth probability distribution of each of the pixels).
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Ambrus by incorporating the depth distribution map and estimation of multiple pixels as taught by Kim to make an invention that can automatically estimate the depth of multiple pixels to create a more accurate depth distribution map for scene reconstruction; thus one of ordinary skilled in the art would be motivated to combine the references since there is a need improve the confidence of the estimated depth map by of estimating a depth of an image based on a neural network trained to output two or more statistical values associated with depths for each pixel in the image, to provide a technological improvement. (Kim ¶0118).
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
Regarding Claim 11, Ambrus teaches an apparatus (Ambrus ¶0022, ¶0109, discloses an apparatus implementing a method) comprising:
at least one processor (Ambrus ¶0007, ¶0028, ¶0030, ¶0035 discloses program executed by a processor or multiple processors);
and a memory storing instructions (Ambrus Fig 1 118 discloses a memory, ¶0028 discloses a memory storing task information), when executed by the at least one processor (Ambrus ¶0028, ¶0038, ¶0105 discloses the processor executing instructions), cause the apparatus to:
obtain a target image for testing (Ambrus ¶0072-¶0073 discloses a target image being put into that network with the source image to train the network);
obtain a target depth estimation map by applying the target image to a monocular depth estimation model (Ambrus ¶0073, ¶0069 discloses a depth estimation model included in the monocular depth model of which the target image is an input) including updated weights (Ambrus Fig 10, 1004 discloses updating the network with a regression loss, ¶0028 discloses the weights within the model), wherein target depth estimation map (Ambrus ¶0073, ¶0069 discloses a depth estimation model included in the monocular depth model of which the target image is an input) comprises an estimation depth of each of a plurality of pixels included in the target image (Ambrus ¶0075, ¶0077 discloses finding the monocular depth at each pixel location per pixel in the input image);
obtain a target uncertainty map (Ambrus ¶0067, ¶0074 discloses a uncertainty map based on the model which includes the target image in the input) included in the target depth estimation map (Ambrus ¶0073, ¶0069 discloses a depth estimation model included in the monocular depth model of which the target image is an input); and
output a signal (Ambrus ¶0024, ¶0053 discloses outputting an uncertainty output for each pixel after the network update) indicating the target uncertainty map (Ambrus ¶0067, ¶0074 discloses a uncertainty map based on the model which includes the target image in the input).
Ambrus does not explicitly disclose wherein the target uncertainty map comprises a relative standard deviation value of each of a plurality of estimation depths.
Kim is in the same field of image analysis for use scene reconstruction for automated driving. Further, Kim teaches wherein the target uncertainty map comprises a relative standard deviation value (Kim ¶0066, ¶0080, ¶0081, discloses a standard deviation value of the depth values of the pixels) of each of a plurality of estimation depths (Kim ¶0109, discloses determining the statistical value for all of the pixels in the image).
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Ambrus by incorporating the depth distribution map and estimation of multiple pixels and the confidence of the estimation as taught by Kim to make an invention that can automatically estimate the depth of multiple pixels to create a more accurate depth distribution map for scene reconstruction; thus one of ordinary skilled in the art would be motivated to combine the references since there is a need improve the confidence of the estimated depth map by of estimating a depth of an image based on a neural network trained to output two or more statistical values associated with depths for each pixel in the image, to provide a technological improvement. (Kim ¶0118).
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
Regarding Claim 12, Ambrus teaches a method (Ambrus ¶0005, ¶0006, ¶0009 discloses a method) performed by a processor (Ambrus ¶0007, ¶0028, ¶0030, ¶0035 discloses program executed by a processor or multiple processors), the method comprising:
obtaining, based on a depth map obtained from a cluster of points (Ambrus ¶0057, ¶0068, ¶0079 discloses the use of a point cloud for the input depth map) at a target time point (Ambrus ¶0067 discloses the use of the network across time to understand a space and ¶0055 discloses multiple frames of input image coming over a video stream which happens over a period of time),
obtaining a depth estimation map (Ambrus ¶0073, ¶0069 discloses a depth estimation model included in the monocular depth) by applying an input image that is (Ambrus ¶0074, ¶0083, ¶0088 discloses an input image input into a that is input into monocular depth prediction network) associated with the target time point (Ambrus ¶0067 discloses the use of the network across time to understand a space and ¶0055 discloses multiple frames of input image coming over a video stream which happens over a period of time) to a monocular depth estimation (MDE) model (Ambrus ¶0074, ¶0083, ¶0088 discloses an input image input into a that is input into monocular depth prediction network);
updating, based on a loss function group (Ambrus Fig 10, 1004 discloses updating the network with a regression loss) applied to the MDE model (Ambrus ¶0074, ¶0083, ¶0088 discloses an input image input into a that is input into monocular depth prediction network), a plurality of weights included in the MDE model (Ambrus ¶0028 discloses the weights within the model), wherein the loss function group (Ambrus Fig 10, 1004 discloses updating the network with a regression loss) comprises a first loss function (Ambrus Fig 10, 1004 discloses updating the network with a regression loss) and the depth estimation map (Ambrus Fig 9 922 discloses the depth regression loss being obtained from the estimated depth);and
outputting a signal (Ambrus ¶0024, ¶0053 discloses outputting an uncertainty output for each pixel after the network update) indicating the updated plurality of weights (Ambrus Fig 10, 1004 discloses updating the network with a regression loss, ¶0028 discloses the weights within the model).
Ambrus does not explicitly disclose a depth distribution map, that is obtained based on the depth distribution map.
Kim is in the same field of image analysis for use scene reconstruction for automated driving. Further, Kim teaches a depth distribution map (Kim Fig 7, ¶0016, ¶0021, discloses a depth probability distribution of each of the pixels), that is obtained based on the depth distribution map (Kim Fig 7, ¶0016, ¶0021, discloses a depth probability distribution of each of the pixels).
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Ambrus by incorporating the depth distribution map and estimation of multiple pixels as taught by Kim to make an invention that can automatically estimate the depth of multiple pixels to create a more accurate depth distribution map for scene reconstruction; thus one of ordinary skilled in the art would be motivated to combine the references since there is a need improve the confidence of the estimated depth map by of estimating a depth of an image based on a neural network trained to output two or more statistical values associated with depths for each pixel in the image, to provide a technological improvement. (Kim ¶0118).
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
Claims 2-10 and 13-20 are rejected under 35 U.S.C. 103 as unpatentable over Ambrus in view of Kim in further view of Guizilini et al (US Patent Publication US 2023/0230264 A1 hereafter referred to as Guizilini).
Regarding Claim 2, Ambrus in view of Kim teaches the apparatus of claim 1, wherein the instructions, when executed by the at least one processor (Ambrus ¶0028, ¶0038, ¶0105 discloses the processor executing instructions), further cause the apparatus to:
obtain the depth map (Ambrus ¶0057, ¶0068, ¶0079 discloses the use of a point cloud for the input depth map),
from the cluster of points (Ambrus ¶0057, ¶0068, ¶0079 discloses the use of a point cloud for the input depth map),
the depth distribution map (Kim Fig 7, ¶0016, ¶0021, discloses a depth probability distribution of each of the pixels).
Ambrus in view of Kim does not explicitly disclose by extracting pieces of depth information wherein the pieces of depth information are associated with a plurality of pixels included in the depth map;
obtain a depth tensor by extending a channel of the depth map, wherein the channel of the depth map is extended by a first condition based on the pieces of depth information; and
obtain, based on the depth tensor.
Guizilini is in the same field of image analysis for use scene reconstruction for automated driving. Further, Guizilini teaches by extracting pieces of depth information (Guizilini ¶0058 discloses extracting depth information)
wherein the pieces of depth information (Guizilini ¶0058 discloses extracting depth information) are associated with a plurality of pixels included in the depth map (Guizilini ¶0058, ¶0067 discloses the depth information derived pixel by pixel) ;
obtain a depth tensor (Guizilini ¶0078 discloses a tensor used in the depth model) by extending a channel of the depth map (Guizilini ¶0079 discloses adapting the dimensions of the tensor to extract the features), wherein the channel of the depth map is extended (Guizilini ¶0079 discloses adapting the dimensions of the tensor to extract the features) by a first condition (Guizilini ¶0079 discloses earned correlations associated with the encoded features being the basis for the size adaptation) based on the pieces of depth information (Guizilini ¶0058 discloses extracting depth information) ; and
obtain, based on the depth tensor (Guizilini ¶0078 discloses a tensor used in the depth model).
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Ambrus in view of Kim by incorporating the depth information and depth tensor elements into the depth distribution map as taught by Guizilini to make an invention that can more accurately determine the depth information used in the depth distribution map to produce a more accurate scene reconstruction; thus one of ordinary skilled in the art would be motivated to combine the references since there is a need to perceive distances through estimation of depth using sensor data provides the robotic device with the ability to plan movements through the environment and generally improve situational awareness about the environment.. (Guizilini ¶0002).
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
Regarding Claim 3, Ambrus in view of Kim in view of Guizilini teaches the apparatus of claim 2, wherein the instructions, when executed by the at least one processor (Ambrus ¶0028, ¶0038, ¶0105 discloses the processor executing instructions), further cause the apparatus to:
determine, based on sensing information obtained by a sensor (Ambrus ¶0039 discloses obtaining 3D object information from a sensor), a minimum discretization value (Guizilni ¶0077-¶0078 discloses the common discretization method of encoding the channels to process features including adapting dimensions including a size which can be equated to a minimum or maximum value) and a maximum discretization value that are associated with channels (Guizilni ¶0077-¶0078 discloses the common discretization method of encoding the channels to process features including adapting dimensions including a size which can be equated to a minimum or maximum value) included in an individual pixel (Guizilini ¶0058, ¶0067 discloses the depth information derived pixel by pixel) of the depth tensor (Guizilini ¶0078 discloses a tensor used in the depth model); and
determine a discretization value of each of the channels (Guizilni ¶0077-¶0078 discloses the common discretization method of encoding the channels to process features into separate channels), wherein the discretization value of each of the channels is determined based on an index of each of the channels (Guizilini ¶0079 discloses a tensor with separate data values indicating depths for corresponding locations in the image on a per-pixel basis) included in the individual pixel (Guizilini ¶0058, ¶0067 discloses the depth information derived pixel by pixel) , the minimum discretization value, the maximum discretization value, and a number of the channels (Guizilni ¶0077-¶0078 discloses the common discretization method of encoding the channels to process features including adapting dimensions including a size which can be equated to a minimum or maximum value and each channel being a separate feature). See Claim 2 for rationale (its parent claim).
Regarding Claim 4, Ambrus in view of Kim in view of Guizilini teaches the apparatus of claim 3, wherein the instructions, when executed by the at least one processor (Ambrus ¶0028, ¶0038, ¶0105 discloses the processor executing instructions), further cause the apparatus to:
determine, based on a discretization value (Guizilni ¶0077-¶0078 discloses the common discretization method of encoding the channels to process features into separate channels) of an N-th channel of an individual pixel (Guizilini ¶0058, ¶0067 discloses the depth information derived pixel by pixel) of the depth tensor (Guizilini ¶0078 discloses a tensor used in the depth model) and a discretization value (Guizilni ¶0077-¶0078 discloses the common discretization method of encoding the channels to process features into separate channels) of an (N+1)-th channel of the individual pixel (Guizilini ¶0058, ¶0067 discloses the depth information derived pixel by pixel) following the N-th channel (Guizilini ¶0058, ¶0067 discloses the depth information derived pixel by pixel), a ratio of the discretization value (Guizilini ¶0010 discloses a comparison of images which include the features in the separate channels) of the N-th channel and the discretization value of the (N+1)-th channel (Guzilini ¶0099-¶0100 disclose that pixels in the channels are gone through iteratively represented by the k-th pixel) as a representative discretization value (Guizilni ¶0077-¶0078 discloses the common discretization method of encoding the channels to process features into separate channels) of the N-th channel (Guizilni ¶0077-¶0078 discloses the common discretization method of encoding the channels to process features into separate channels), and
wherein N is a natural number and smaller than or equal to (Guzilini ¶0099-¶0100 disclose that the k-th pixel is a value between 0 and 1 and is less that the total number of channels due to the k+j th calculation) a total number of channels of the depth tensor (Guizilni ¶0077-¶0078 discloses the common discretization method of encoding the channels to process features including adapting dimensions including a size which can be equated to a minimum or maximum value and each channel being a separate feature). See Claim 2 for rationale (its parent claim).
Regarding Claim 5, Ambrus in view of Kim in view of Guizilini teaches the apparatus of claim 4, wherein the depth distribution map comprises pixels (Kim Fig 7, ¶0016, ¶0021, discloses a depth probability distribution of each of the pixels), and wherein a representative discretization value of a pixel (Guizilni ¶0077-¶0078 discloses the common discretization method of encoding the channels to process features into separate channels) of the pixels is associated with channels included in the pixel of pixels of the depth tensor (Guizilini ¶0079 discloses a tensor with separate data values indicating depths for corresponding locations in the image on a per-pixel basis), and
wherein a sum of probabilities (Kim ¶0030, ¶0084, discloses summing the values of pixels in the probability distribution) comprises probabilities that satisfy a second condition (Kim ¶0090 discloses a threshold), wherein the sum of probabilities (Guizilini ¶0079 discloses a tensor with separate data values indicating depths for corresponding locations in the image on a per-pixel basis) is associated with channels included in each of the pixels of the depth tensor (Guizilini ¶0079 discloses a tensor with separate data values indicating depths for corresponding locations in the image on a per-pixel basis) See Claim 2 for rationale (its parent claim).
Regarding Claim 6, Ambrus in view of Kim teaches the apparatus of claim 1, wherein the instructions, when executed by the at least one processor (Ambrus ¶0028, ¶0038, ¶0105 discloses the processor executing instructions), further cause the apparatus to:
obtain a first estimation depth of a first pixel (Kim ¶0013 discloses a first statistical value may be a mean of depth values of a corresponding each of the plural pixels in the input image) included in the depth distribution map (Kim Fig 7, ¶0016, ¶0021, discloses a depth probability distribution of each of the pixels);
obtain a second estimation depth of a second pixel (Kim ¶0013 discloses a second statistical value may be a variance or a standard deviation of the depth values of the corresponding each of the plural pixels in the input image) included in the depth estimation map (Ambrus ¶0073, ¶0069 discloses a depth estimation model included in the monocular depth) , wherein the second pixel is related to a location corresponding to the first pixel (Kim ¶0018, discloses the 3d location information corresponding to the estimated depth information of the pixels);
determine, based on the first estimation depth and the second estimation depth (Kim ¶0013 discloses a first and second statistical value may be a mean of depth values of a corresponding each of the plural pixels in the input image), the first loss function (Kim ¶0013 discloses a second statistical value may be a variance or a standard deviation of the depth values); and
update, based on the determined first loss function (Kim ¶0013 discloses a second statistical value may be a variance or a standard deviation of the depth values), the plurality of weights included in the MDE model (Ambrus Fig 10, 1004 discloses updating the network with a regression loss ¶0028 discloses the weights within the model) for obtaining the second estimation depth (Kim ¶0013 discloses a second statistical value may be a variance or a standard deviation of the depth values of the corresponding each of the plural pixels in the input image).
Ambrus in view of Kim does not explicitly disclose among a plurality of pixels, among a plurality of pixels.
Guizilini is in the same field of image analysis for use scene reconstruction for automated driving. Further, Guizilini teaches among a plurality of pixels (Guizilini ¶0058, ¶0067 discloses the depth information derived pixel by pixel), among a plurality of pixels (Guizilini ¶0058, ¶0067 discloses the depth information derived pixel by pixel).
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Ambrus in view of Kim by incorporating the depth information and depth tensor elements into the depth distribution map as taught by Guizilini to make an invention that can more accurately determine the depth information used in the depth distribution map to produce a more accurate scene reconstruction; thus one of ordinary skilled in the art would be motivated to combine the references since there is a need to perceive distances through estimation of depth using sensor data provides the robotic device with the ability to plan movements through the environment and generally improve situational awareness about the environment.. (Guizilini ¶0002).
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
Regarding Claim 7, Ambrus in view of Kim in view of Guizilini teaches the apparatus of claim 6, wherein the instructions, when executed by the at least one processor (Ambrus ¶0028, ¶0038, ¶0105 discloses the processor executing instructions), further cause the apparatus to:
determine, based on the first estimation depth (Kim ¶0013 discloses a first statistical value may be a mean of depth values of a corresponding each of the plural pixels in the input image) satisfying a third condition (Kim ¶0014 discloses determining a confidence level of the pixel, which can be interpreted as a third condition), a difference between the first estimation depth and the second estimation depth as the first loss function (Kim ¶0013 discloses a second statistical value may be a variance or a standard deviation of the depth values of the corresponding each of the plural pixels in the input image); or
skip updating, based on the first estimation depth not satisfying the third condition (Kim ¶0014 discloses determining to use the value or not based on the confidence value), the plurality of weights included in the MDE model (Ambrus Fig 10, 1004 discloses updating the network with a regression loss ¶0028 discloses the weights within the model) for obtaining the second estimation depth (Kim ¶0013 discloses a second statistical value may be a variance or a standard deviation of the depth values of the corresponding each of the plural pixels in the input image). See Claim 6 for rationale (its parent claim).
Regarding Claim 8, Ambrus in view of Kim teaches the apparatus of claim 1, wherein the instructions, when executed by the at least one processor (Ambrus ¶0028, ¶0038, ¶0105 discloses the processor executing instructions),
by applying an input image at a time point different from the target time point (Ambrus ¶0067 discloses the use of the network across time to understand a space and ¶0055 discloses multiple frames of input image coming over a video stream which happens over a period of time) and an input image at the target time point (Ambrus ¶0067 discloses the use of the network across time to understand a space and ¶0055 discloses multiple frames of input image coming over a video stream which happens over a period of time)
obtain a first cluster of points at the (Ambrus ¶0057, ¶0068, ¶0079 discloses the use of a point cloud for the input depth map) at the target time point (Ambrus ¶0067 discloses the use of the network across time to understand a space and ¶0055 discloses multiple frames of input image coming over a video stream which happens over a period of time) to the depth estimation map (Ambrus ¶0073, ¶0069 discloses a depth estimation model included in the monocular depth);
obtain a second cluster of points (Ambrus ¶0057, ¶0068, ¶0079 discloses the use of a point cloud for the input depth map) at a time point different from the target time point (Ambrus ¶0067 discloses the use of the network across time to understand a space and ¶0055 discloses multiple frames of input image coming over a video stream which happens over a period of time)
to the first cluster of points (Ambrus ¶0057, ¶0068, ¶0079 discloses the use of a point cloud for the input depth map); and
determine, based on the second cluster of points (Ambrus ¶0057, ¶0068, ¶0079 discloses the use of a point cloud for the input depth map), a second loss function (Ambrus Fig 9 972, discloses a vote regression loss) different from the first loss function (Ambrus Fig 10, 1004 discloses updating the network with a regression loss), and wherein the loss function group (Ambrus Fig 9 discloses how both loss functions are used in the training) further comprises the second loss function (Ambrus Fig 9 972, discloses a vote regression loss).
Ambrus in view of Kim does not explicitly disclose obtain pose change information, to a pose estimation model, by applying an inverse of an intrinsic parameter related to a sensor, by applying the pose change information.
Guizilini is in the same field of image analysis for use scene reconstruction for automated driving. Further, Guizilini teaches obtain pose change information (Guzilini ¶0081-¶0082 discloses a pose model that take images as inputs and determines the transformation of the images) to a pose estimation model (Guizilni Fig 3, 306 and Fig 5 discloses a pose model),
by applying an inverse of an intrinsic parameter related to a sensor (Guzilini ¶0079 discloses taking the inverse of the layers of the depth for different scales of the map),
by applying the pose change information (Guzilini ¶0081-¶0082 discloses a pose model that take images as inputs and determines the transformation of the images).
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Ambrus in view of Kim by incorporating the depth information and depth tensor elements into the depth distribution map as taught by Guizilini to make an invention that can more accurately determine the depth information used in the depth distribution map to produce a more accurate scene reconstruction; thus one of ordinary skilled in the art would be motivated to combine the references since there is a need to perceive distances through estimation of depth using sensor data provides the robotic device with the ability to plan movements through the environment and generally improve situational awareness about the environment.. (Guizilini ¶0002).
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
Regarding Claim 9, Ambrus in view of Kim in view of Guizilini teaches the apparatus of claim 8, wherein the instructions, when executed by the at least one processor (Ambrus ¶0028, ¶0038, ¶0105 discloses the processor executing instructions), further cause the apparatus to:
obtain a reconstruction image (Ambrus ¶0025, ¶0069 discloses a 3D reconstruction image) by applying the intrinsic parameter (Guzilini ¶0079 discloses taking the inverse of the layers of the depth for different scales of the map) to the second cluster of points (Ambrus ¶0057, ¶0068, ¶0079 discloses the use of a point cloud for the input depth map); and
determine, based on the input image and the reconstruction image (Ambrus Fig 9 disclose the input image and the reconstructed estimated depth image being used to determine the loss function), the second loss function (Ambrus Fig 9 972, discloses a vote regression loss). See Claim 8 for rationale (its parent claim).
Regarding Claim 10, Ambrus in view of Kim in view of Guizilini teaches the apparatus of claim 8, wherein the instructions, when executed by the at least one processor (Ambrus ¶0028, ¶0038, ¶0105 discloses the processor executing instructions) , further cause the apparatus to:
obtain a first factor (Kim ¶0013 discloses a first statistical value may be a mean of depth values of a corresponding each of the plural pixels in the input image) indicating a mean value (Kim ¶0013 Fig 7 discloses a mean value of depth) associated with channels of a target pixel among pixels (Guizilini ¶0058, ¶0067 discloses the depth information derived pixel by pixel) included in the depth estimation map (Ambrus ¶0073, ¶0069 discloses a depth estimation model included in the monocular depth) and a second factor indicating a standard deviation value (Kim ¶0013 discloses a second statistical value may be a variance or a standard deviation of the depth values of the corresponding each of the plural pixels in the input image) associated with the channels of the target pixel (Guizilini ¶0058, ¶0067 discloses the depth information derived pixel by pixel) ; and
obtain, based on the first factor and the second factor, a relative standard deviation value (Kim ¶0013 discloses a second statistical value may be a variance or a standard deviation of the depth values of the corresponding each of the plural pixels in the input image) indicating uncertainty of the target pixel (Kim ¶0014 discloses determining to use the value or not based on the confidence value). See Claim 8 for rationale (its parent claim).
Regarding Claim 13, Ambrus in view of Kim teaches the method of claim 12, wherein the obtaining the depth distribution map (Kim Fig 7, ¶0016, ¶0021, discloses a depth probability distribution of each of the pixels) comprises:
obtaining the depth map (Ambrus ¶0057, ¶0068, ¶0079 discloses the use of a point cloud for the input depth map) from the cluster of points (Ambrus ¶0057, ¶0068, ¶0079 discloses the use of a point cloud for the input depth map) the depth distribution map (Kim Fig 7, ¶0016, ¶0021, discloses a depth probability distribution of each of the pixels).
Ambrus in view of Kim does not explicitly disclose by extracting pieces of depth information, wherein the pieces of depth information are associated with a plurality of pixels included in the depth map;
obtaining a depth tensor extending a channel of the depth map, wherein the channel of the depth map is extended by a first condition based on the pieces of depth information; and
obtaining, based on the depth tensor.
Guizilini is in the same field of image analysis for use scene reconstruction for automated driving. Further, Guizilini teaches by extracting pieces of depth information (Guizilini ¶0058 discloses extracting depth information)
wherein the pieces of depth information (Guizilini ¶0058 discloses extracting depth information) are associated with a plurality of pixels included in the depth map (Guizilini ¶0058, ¶0067 discloses the depth information derived pixel by pixel);
obtaining a depth tensor (Guizilini ¶0078 discloses a tensor used in the depth model) by extending a channel of the depth map (Guizilini ¶0079 discloses adapting the dimensions of the tensor to extract the features), wherein the channel of the depth map is extended (Guizilini ¶0079 discloses adapting the dimensions of the tensor to extract the features) by a first condition (Guizilini ¶0079 discloses earned correlations associated with the encoded features being the basis for the size adaptation) based on the pieces of depth information (Guizilini ¶0058 discloses extracting depth information); and
obtaining, based on the depth tensor (Guizilini ¶0078 discloses a tensor used in the depth model).
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Ambrus in view of Kim by incorporating the depth information and depth tensor elements into the depth distribution map as taught by Guizilini to make an invention that can more accurately determine the depth information used in the depth distribution map to produce a more accurate scene reconstruction; thus one of ordinary skilled in the art would be motivated to combine the references since there is a need to perceive distances through estimation of depth using sensor data provides the robotic device with the ability to plan movements through the environment and generally improve situational awareness about the environment.. (Guizilini ¶0002).
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
Regarding Claim 14, Ambrus in view of Kim in view of Guizilini teaches the method of claim 13, wherein the obtaining the depth distribution map (Kim Fig 7, ¶0016, ¶0021, discloses a depth probability distribution of each of the pixels) comprises:
determining, based on sensing information obtained by a sensor (Ambrus ¶0039 discloses obtaining 3D object information from a sensor), a minimum discretization value (Guizilni ¶0077-¶0078 discloses the common discretization method of encoding the channels to process features including adapting dimensions including a size which can be equated to a minimum or maximum value) and a maximum discretization value that are associated with channels (Guizilni ¶0077-¶0078 discloses the common discretization method of encoding the channels to process features including adapting dimensions including a size which can be equated to a minimum or maximum value) included in an individual pixel (Guizilini ¶0058, ¶0067 discloses the depth information derived pixel by pixel) of the depth tensor (Guizilini ¶0078 discloses a tensor used in the depth model); and
determining a discretization value of each of the channels (Guizilni ¶0077-¶0078 discloses the common discretization method of encoding the channels to process features into separate channels , wherein the discretization value of each of the channels is determined based on an index of each of the channels (Guizilini ¶0079 discloses a tensor with separate data values indicating depths for corresponding locations in the image on a per-pixel basis) included in the individual pixel (Guizilini ¶0058, ¶0067 discloses the depth information derived pixel by pixel) , the minimum discretization value, the maximum discretization value, and a number of the channels (Guizilni ¶0077-¶0078 discloses the common discretization method of encoding the channels to process features including adapting dimensions including a size which can be equated to a minimum or maximum value and each channel being a separate feature). See Claim 13 for rationale (its parent claim).
Regarding Claim 15, Ambrus in view of Kim in view of Guizilini teaches the method of claim 14, wherein the obtaining the depth distribution map (Kim Fig 7, ¶0016, ¶0021, discloses a depth probability distribution of each of the pixels) comprises:
determining, based on a discretization value (Guizilni ¶0077-¶0078 discloses the common discretization method of encoding the channels to process features into separate channels) of an N-th channel of an individual pixel (Guizilini ¶0058, ¶0067 discloses the depth information derived pixel by pixel) of the depth tensor (Guizilini ¶0078 discloses a tensor used in the depth model) and a discretization value (Guizilni ¶0077-¶0078 discloses the common discretization method of encoding the channels to process features into separate channels) of an (N+1)-th channel of the individual pixel (Guizilini ¶0058, ¶0067 discloses the depth information derived pixel by pixel) following the N-th channel (Guizilini ¶0058, ¶0067 discloses the depth information derived pixel by pixel), a ratio of the discretization value (Guizilini ¶0010 discloses a comparison of images which include the features in the separate channels) of the N-th channel and the discretization value of the (N+1)-th channel (Guzilini ¶0099-¶0100 disclose that pixels in the channels are gone through iteratively represented by the k-th pixel) as a representative discretization value (Guizilni ¶0077-¶0078 discloses the common discretization method of encoding the channels to process features into separate channels) of the N-th channel (Guizilni ¶0077-¶0078 discloses the common discretization method of encoding the channels to process features into separate channels), and
wherein N is a natural number and smaller than or equal to (Guzilini ¶0099-¶0100 disclose that the k-th pixel is a value between 0 and 1 and is less that the total number of channels due to the k+j th calculation) a total number of channels of the depth tensor (Guizilni ¶0077-¶0078 discloses the common discretization method of encoding the channels to process features including adapting dimensions including a size which can be equated to a minimum or maximum value and each channel being a separate feature)
wherein the depth distribution map comprises pixels (Kim Fig 7, ¶0016, ¶0021, discloses a depth probability distribution of each of the pixels), and wherein a representative discretization value of a pixel (Guizilni ¶0077-¶0078 discloses the common discretization method of encoding the channels to process features into separate channels) of the pixels is associated with channels included in the pixel of pixels of the depth tensor (Guizilini ¶0079 discloses a tensor with separate data values indicating depths for corresponding locations in the image on a per-pixel basis), and
wherein a sum of probabilities (Kim ¶0030, ¶0084, discloses summing the values of pixels in the probability distribution) comprises probabilities that satisfy a second condition (Kim ¶0090 discloses a threshold), wherein the sum of probabilities (Guizilini ¶0079 discloses a tensor with separate data values indicating depths for corresponding locations in the image on a per-pixel basis) is associated with channels included in each of the pixels of the depth tensor (Guizilini ¶0079 discloses a tensor with separate data values indicating depths for corresponding locations in the image on a per-pixel basis). See Claim 13 for rationale (its parent claim).
Regarding Claim 16, Ambrus in view of Kim teaches the method of claim 12, wherein the updating the plurality of weights included in the MDE model (Ambrus Fig 10, 1004 discloses updating the network with a regression loss ¶0028 discloses the weights within the model) comprises:
obtaining a first estimation depth of a first pixel (Kim ¶0013 discloses a first statistical value may be a mean of depth values of a corresponding each of the plural pixels in the input image) included in the depth distribution map (Kim Fig 7, ¶0016, ¶0021, discloses a depth probability distribution of each of the pixels);
obtaining a second estimation depth of a second pixel (Kim ¶0013 discloses a second statistical value may be a variance or a standard deviation of the depth values of the corresponding each of the plural pixels in the input image)
included in the depth estimation map (Ambrus ¶0073, ¶0069 discloses a depth estimation model included in the monocular depth) , wherein the second pixel is related to a location corresponding to the first pixel (Kim ¶0018, discloses the 3d location information corresponding to the estimated depth information of the pixels);
determining, based on the first estimation depth and the second estimation depth (Kim ¶0013 discloses a first and second statistical value may be a mean of depth values of a corresponding each of the plural pixels in the input image), the first loss function (Kim ¶0013 discloses a second statistical value may be a variance or a standard deviation of the depth values); and
updating, based on the determined first loss function (Kim ¶0013 discloses a second statistical value may be a variance or a standard deviation of the depth values), the plurality of weights included in the MDE model (Ambrus Fig 10, 1004 discloses updating the network with a regression loss ¶0028 discloses the weights within the model) for obtaining the second estimation depth (Kim ¶0013 discloses a second statistical value may be a variance or a standard deviation of the depth values of the corresponding each of the plural pixels in the input image).
Ambrus in view of Kim does not explicitly disclose among a plurality of pixels, among a plurality of pixels.
Guizilini is in the same field of image analysis for use scene reconstruction for automated driving. Further, Guizilini teaches among a plurality of pixels (Guizilini ¶0058, ¶0067 discloses the depth information derived pixel by pixel), among a plurality of pixels (Guizilini ¶0058, ¶0067 discloses the depth information derived pixel by pixel).
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Ambrus in view of Kim by incorporating the depth information and depth tensor elements into the depth distribution map as taught by Guizilini to make an invention that can more accurately determine the depth information used in the depth distribution map to produce a more accurate scene reconstruction; thus one of ordinary skilled in the art would be motivated to combine the references since there is a need to perceive distances through estimation of depth using sensor data provides the robotic device with the ability to plan movements through the environment and generally improve situational awareness about the environment. (Guizilini ¶0002).
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
Regarding Claim 17, Ambrus in view of Kim in view of Guizilini teaches the method of claim 16, wherein the updating the plurality of weights included in the MDE model (Ambrus Fig 10, 1004 discloses updating the network with a regression loss ¶0028 discloses the weights within the model) comprises:
determining, based on the first estimation depth (Kim ¶0013 discloses a first statistical value may be a mean of depth values of a corresponding each of the plural pixels in the input image) satisfying a third condition (Kim ¶0014 discloses determining a confidence level of the pixel, which can be interpreted as a third condition), a difference between the first estimation depth and the second estimation depth as the first loss function (Kim ¶0013 discloses a second statistical value may be a variance or a standard deviation of the depth values of the corresponding each of the plural pixels in the input image); or
skipping, based on the first estimation depth not satisfying the third condition (Kim ¶0014 discloses determining to use the value or not based on the confidence value) , updating the plurality of weights included in the MDE model (Ambrus Fig 10, 1004 discloses updating the network with a regression loss ¶0028 discloses the weights within the model) for obtaining the second estimation depth (Kim ¶0013 discloses a second statistical value may be a variance or a standard deviation of the depth values of the corresponding each of the plural pixels in the input image). See Claim 16 for rationale (its parent claim).
Regarding Claim 18, Ambrus in view of Kim teaches the method of claim 12, further comprising:
by applying an input image at a time point different from the target time point (Ambrus ¶0067 discloses the use of the network across time to understand a space and ¶0055 discloses multiple frames of input image coming over a video stream which happens over a period of time) and an input image at the target time point (Ambrus ¶0067 discloses the use of the network across time to understand a space and ¶0055 discloses multiple frames of input image coming over a video stream which happens over a period of time)
obtaining a first cluster of points at the (Ambrus ¶0057, ¶0068, ¶0079 discloses the use of a point cloud for the input depth map) at the target time point (Ambrus ¶0067 discloses the use of the network across time to understand a space and ¶0055 discloses multiple frames of input image coming over a video stream which happens over a period of time) to the depth estimation map (Ambrus ¶0073, ¶0069 discloses a depth estimation model included in the monocular depth);
obtaining a second cluster of points (Ambrus ¶0057, ¶0068, ¶0079 discloses the use of a point cloud for the input depth map) at a time point different from the target time point (Ambrus ¶0067 discloses the use of the network across time to understand a space and ¶0055 discloses multiple frames of input image coming over a video stream which happens over a period of time),
to the first cluster of points (Ambrus ¶0057, ¶0068, ¶0079 discloses the use of a point cloud for the input depth map); and
determining, based on the second cluster of points (Ambrus ¶0057, ¶0068, ¶0079 discloses the use of a point cloud for the input depth map), a second loss function (Ambrus Fig 9 972, discloses a vote regression loss) different from the first loss function (Ambrus Fig 10, 1004 discloses updating the network with a regression loss), and wherein the loss function group (Ambrus Fig 9 discloses how both loss functions are used in the training) further comprises the second loss function (Ambrus Fig 9 972, discloses a vote regression loss).
Ambrus in view of Kim does not explicitly disclose obtaining pose change information, to a pose estimation model, by applying an inverse of an intrinsic parameter related to a sensor, by applying the pose change information.
Guizilini is in the same field of image analysis for use scene reconstruction for automated driving. Further, Guizilini teaches obtaining pose change information (Guzilini ¶0081-¶0082 discloses a pose model that take images as inputs and determines the transformation of the images) to a pose estimation model (Guizilni Fig 3, 306 and Fig 5 discloses a pose model)
by applying an inverse of an intrinsic parameter related to a sensor (Guzilini ¶0079 discloses taking the inverse of the layers of the depth for different scales of the map)
by applying the pose change information (Guzilini ¶0081-¶0082 discloses a pose model that take images as inputs and determines the transformation of the images).
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Ambrus in view of Kim by incorporating the depth information and depth tensor elements into the depth distribution map as taught by Guizilini to make an invention that can more accurately determine the depth information used in the depth distribution map to produce a more accurate scene reconstruction; thus one of ordinary skilled in the art would be motivated to combine the references since there is a need to perceive distances through estimation of depth using sensor data provides the robotic device with the ability to plan movements through the environment and generally improve situational awareness about the environment.. (Guizilini ¶0002).
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
Regarding Claim 19, Ambrus in view of Kim in view of Guizilini teaches the method of claim 18, wherein the determining the second loss function(Ambrus Fig 9 972, discloses a vote regression loss) comprises:
obtaining a reconstruction image (Ambrus ¶0025, ¶0069 discloses a 3D reconstruction image) by applying the intrinsic parameter (Guzilini ¶0079 discloses taking the inverse of the layers of the depth for different scales of the map) to the second cluster of points (Ambrus ¶0057, ¶0068, ¶0079 discloses the use of a point cloud for the input depth map); and
determining, based on the input image and the reconstruction image (Ambrus Fig 9 disclose the input image and the reconstructed estimated depth image being used to determine the loss function), the second loss function (Ambrus Fig 9 972, discloses a vote regression loss). See Claim 18 for rationale (its parent claim).
Regarding Claim 20, Ambrus in view of Kim in view of Guizilini teaches the method of claim 18, further comprising:
obtaining a first factor (Kim ¶0013 discloses a first statistical value may be a mean of depth values of a corresponding each of the plural pixels in the input image) indicating a mean value (Kim ¶0013 Fig 7 discloses a mean value of depth) associated with channels of a target pixel among pixels (Guizilini ¶0058, ¶0067 discloses the depth information derived pixel by pixel) included in the depth estimation map (Ambrus ¶0073, ¶0069 discloses a depth estimation model included in the monocular depth) and a second factor indicating a standard deviation value (Kim ¶0013 discloses a second statistical value may be a variance or a standard deviation of the depth values of the corresponding each of the plural pixels in the input image) associated with the channels of the target pixel (Guizilini ¶0058, ¶0067 discloses the depth information derived pixel by pixel) ; and
obtaining, based on the first factor and the second factor, a relative standard deviation value (Kim ¶0013 discloses a second statistical value may be a variance or a standard deviation of the depth values of the corresponding each of the plural pixels in the input image) indicating uncertainty of the target pixel (Kim ¶0014 discloses determining to use the value or not based on the confidence value). See Claim 18 for rationale (its parent claim).
Reference Cited
The prior art made of record and not relied upon is considered pertinent to applicant’s disclosure.
US-20210398302-A1 to Guizilini discloses a method for scene reconstruction using a depth estimation and pose model.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to RACHEL LYNN ROBERTS whose telephone number is (571)272-6413. The examiner can normally be reached Monday- Friday 7:30am- 5:00pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Oneal Mistry can be reached on (313) 446-4912. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/RACHEL L ROBERTS/Examiner, Art Unit 2674
/Ross Varndell/Primary Examiner, Art Unit 2674