DETAIL OFFICE ACTIONS
The United States Patent & Trademark Office appreciates the response filed for the current application that is submitted on 10/27/2025. The United States Patent & Trademark Office reviewed the following documents submitted and has made the following comments below.
Amendment
Applicant submitted amendments on 10/27/2025. The Examiner acknowledges the amendment and has reviewed the claims accordingly.
Priority
Applicant claims the benefit of US Provisional Application No. PRO 63/339,090 filed 05-06-2022. Claims 1-20 have been afforded the benefit of this filing date.
Information Disclosure Statement
The IDS(s) dated 07/21/2023 that have been previously considered remain placed in the application file.
Overview
Claims 1-20 are pending in this application and have been considered below.
Claims 1, 4-12, and 15-20 are rejected.
Claim 2-3 and 13-14 is objected to.
Applicant Arguments:
In regards to the argument on Argument 1, Applicant/s state/s “However, this rejection of the second limitation is improper. The mapper of Petrozskaya is a model that produces a map or 3D mesh (see e.g., paragraph 147 and 428). Thus, even if the word 'parameter' is considered equivalent with the word 'feature,' model parameters of a model do not teach or suggest image features of an image because a parameter of a model is fundamentally different from a parameter/feature of an image. Thus, claim 1 (unamended) is not taught or suggested by the combination of Petrovskaya and Zhong.” therefore, the rejection of 35 U.S.C. 103 should be withdrawn (See Remarks, page 8, paragraph 6).
In regards to the argument on Argument 2, Applicant/s state/s “claim 1 is amended herein to recite inter alia "the image features representing visual information of the reference image and the set of source images." Petrozskaya does not teach or suggest this. For example, model parameters of a model (e.g., "desired grid resolution, a truncation distance, the contribution of different weighing components," as recited in Petrozskaya) do not teach or suggest ‘image features representing visual information of the reference image and the set of source images,’ as recited in amened claim 1.” therefore, the rejection of 35 U.S.C. 103 should be withdrawn (See Remarks, page 9, paragraph 1).
In regards to the argument on Argument 3, Applicant/s state/s “this rejection is improper. Firstly, the Office Action asserts that the 4 dimensional state of the Scaling Series (described in paragraph 263) includes model parameters of the mapper (described in paragraph 147) and a data log of "dynamic obstacles" (described in paragraph 269). The Office Action also asserts that these model parameters and the data log of dynamic obstacles are arranged in the Scaling Series state based on relative post distance. Petrozskaya does not teach or suggest this.” therefore, the rejection of 35 U.S.C. 103 should be withdrawn (See Remarks, page 9, paragraph 3).
In regards to the argument on Argument 4, Applicant/s state/s “Petrozskaya does not teach or suggest that the dimensions of the Scaling Series state are arranged according to pose.” therefore, the rejection of 35 U.S.C. 103 should be withdrawn (See Remarks, page 9, paragraph 3).
In regards to the argument on Argument 5, Applicant/s state/s “Firstly, even though the Scaling Series state can be 4D, this does not teach or suggest a 4D feature volume. Secondly, the Scaling Series state with model parameters of the mapper does not teach or suggest a 4D feature volume with image features of images, where the image features represent visual information of those images. Thirdly, the Scaling Series state with model parameters of the mapper and the data log of dynamic obstacles does not teach or suggest that the model parameters and the data log are arranged in the Scaling Series state "based on relative pose distances between the reference image and the set of source images," as recited in amended claim 1. Thus, the combination of Petrovskaya and Zhong does not teach or suggest amended claim 1.” therefore, the rejection of 35 U.S.C. 103 should be withdrawn (See Remarks, page 10, paragraph 3).
Examiner’s Responses:
In response to Argument 1, Applicant’s arguments, see Remarks, filed 10/27/25, with respect to the rejection(s) of claims 1 and 11 under 35 U.S.C. 103 have been fully considered and are persuasive. Therefore, the rejection has been withdrawn due to the amendment. However, upon further consideration, a new ground(s) of rejection is made for Claim 1 and 11 and its dependent claims under 35 U.S.C. 103 in view of Petrovskaya et al. (US Patent Publication 2016 20160148433 A1 hereafter referred to as Petrovskaya) in view of Zhong et al. (US Patent No 11062471 B1 hereafter referred to as Zhong) in further view of Sun et al (Sun, Penghui, Suping Wu, and Kui Lin. "Attention-guided multi-view stereo network for depth estimation." 2020 IEEE 22nd International Conference on High Performance Computing and Communications; IEEE 18th International Conference on Smart City; IEEE 6th International Conference on Data Science and Systems (HPCC/SmartCity/DSS). IEEE, 2020. hereafter referred to as Sun).
The Examiner finds that Petrovskaya teaches on the amended claim language “reference image of an environment” and “a set of one or more source images of the environment” in Claim 1 and 11.
Specifically, Petrovskaya teaches a system that maps an environment based on an image in ¶0082 ¶0073 and using sets of training data related to the obstacles in the images in ¶0269 and ¶0252-¶0253 to generate a four dimensional feature volume in ¶0263. Applicant argues that model parameters of a model do not teach or suggest image features of an image because a parameter of a model is fundamentally different from a parameter/feature of an image. However, the Examiner interprets that Petrozskaya teaches the main concept of taking a reference image from an environment using training images to derive parameters from the images to predict poses of the objects in the environment, the additional details of the function and characteristics of the main concepts as stated above by the applicant in the amendments is taught by Sun in the details of the rejection below. The Examiner will maintain prior art Petrozskaya and details of the rejection are below.
The Examiner finds that Zhong teaches on the amendment claim language “reducing the 4D feature volume to generate a three dimensional (3D) cost volume” in claim 1, and 11.
Specifically, Zhong teaches reducing the 4D feature volume to a 3D feature volume in Col 6 Line 64-64 and Col 3 Line 14-15. Applicant argues that claim 1 (unamended) is not taught or suggested by the combination of Petrovskaya and Zhong. However, the Examiner interprets that Zhong does teach the main concept of the dimensionality reduction, the additional details of the functions of the main concepts as stated above by the applicant in the amendments is taught by Sun in the details of the rejection below. The Examiner will maintain prior art Zhong and details of the rejection are below.
In response to Argument 2, Applicant’s arguments, see Remarks, filed 10/27/25, with respect to the rejection(s) of claims 1 and 11 under 35 U.S.C. 103 have been fully considered and are persuasive. Therefore, the rejection has been withdrawn due to the amendment. However, upon further consideration, a new ground(s) of rejection is made for Claim 1 and 11 and its dependent claims under 35 U.S.C. 103 in view of Petrovskaya et al. (US Patent Publication 2016 20160148433 A1 hereafter referred to as Petrovskaya) in view of Zhong et al. (US Patent No 11062471 B1 hereafter referred to as Zhong) in further view of Sun et al (Sun, Penghui, Suping Wu, and Kui Lin. "Attention-guided multi-view stereo network for depth estimation." 2020 IEEE 22nd International Conference on High Performance Computing and Communications; IEEE 18th International Conference on Smart City; IEEE 6th International Conference on Data Science and Systems (HPCC/SmartCity/DSS). IEEE, 2020. hereafter referred to as Sun).
The Examiner finds that Petrovskaya teaches on the amended claim language “reference image of an environment” and “a set of one or more source images of the environment” in Claim 1 and 11.
Specifically, Petrovskaya teaches a system that maps an environment based on an image in ¶0082 ¶0073 and using sets of training data related to the obstacles in the images in ¶0269 and ¶0252-¶0253 to generate a four dimensional feature volume in ¶0263. Applicant argues that model parameters of a model do not teach or suggest image features of an image because a parameter of a model is fundamentally different from a parameter/feature of an image. However, the Examiner interprets that Petrozskaya teaches the main concept of taking a reference image from an environment using training images to derive parameters from the images to predict poses of the objects in the environment, the additional details of the function and characteristics of the main concepts as stated above by the applicant in the amendments is taught by Sun in the details of the rejection below. The Examiner will maintain prior art Petrozskaya and details of the rejection are below.
In response to Argument 3, Applicant’s arguments, see Remarks, filed 10/27/25, with respect to the rejection(s) of claims 1 and 11 under 35 U.S.C. 103 have been fully considered and are persuasive. Therefore, the rejection has been withdrawn due to the amendment. However, upon further consideration, a new ground(s) of rejection is made for Claim 1 and 11 and its dependent claims under 35 U.S.C. 103 in view of Petrovskaya et al. (US Patent Publication 2016 20160148433 A1 hereafter referred to as Petrovskaya) in view of Zhong et al. (US Patent No 11062471 B1 hereafter referred to as Zhong) in further view of Sun et al (Sun, Penghui, Suping Wu, and Kui Lin. "Attention-guided multi-view stereo network for depth estimation." 2020 IEEE 22nd International Conference on High Performance Computing and Communications; IEEE 18th International Conference on Smart City; IEEE 6th International Conference on Data Science and Systems (HPCC/SmartCity/DSS). IEEE, 2020. hereafter referred to as Sun).
The Examiner finds that Petrovskaya teaches on the amended claim language “generating a four dimensional (4D) feature volume” and “metadata” in Claim 1 and 11.
Specifically, Petrovskaya teaches in ¶0264-¶0265 that the data logs have first and subsequent poses data that is used to obtain the separation distance between poses to determine the estimated pose. Petrovskaya also teaches an feature volume in relation to the data frames color in ¶0159. Applicant argues that data log of dynamic obstacles are arranged in the Scaling Series state based on relative post distance and that Petrozskaya does not teach or suggest this. However, the Examiner interprets that Petrozskaya teaches the main concept of data logs have first and subsequent poses data that is used to obtain the separation distance between poses to determine the estimated pose, the additional details of the function and characteristics of the image features of the main concepts as stated above by the applicant in the amendments is taught by Sun in the details of the rejection below. The Examiner will maintain prior art Petrozskaya and details of the rejection are below.
In response to Argument 4, Applicant’s arguments, see Remarks, filed 10/27/25, with respect to the rejection(s) of claims 1 and 11 under 35 U.S.C. 103 have been fully considered and are persuasive. Therefore, the rejection has been withdrawn due to the amendment. However, upon further consideration, a new ground(s) of rejection is made for Claim 1 and 11 and its dependent claims under 35 U.S.C. 103 in view of Petrovskaya et al. (US Patent Publication 2016 20160148433 A1 hereafter referred to as Petrovskaya) in view of Zhong et al. (US Patent No 11062471 B1 hereafter referred to as Zhong) in further view of Sun et al (Sun, Penghui, Suping Wu, and Kui Lin. "Attention-guided multi-view stereo network for depth estimation." 2020 IEEE 22nd International Conference on High Performance Computing and Communications; IEEE 18th International Conference on Smart City; IEEE 6th International Conference on Data Science and Systems (HPCC/SmartCity/DSS). IEEE, 2020. hereafter referred to as Sun).
The Examiner finds that Petrovskaya teaches on the amended claim language “arranged in the 4D feature volume based on relative pose distances” in Claim 1 and 11.
Specifically, Petrovskaya teaches in ¶0264-¶0265 that the data logs have first and subsequent poses data that is used to obtain the separation distance between poses to determine the estimated pose in the scaling series. Applicant argues that Petrozskaya does not teach or suggest that the dimensions of the Scaling Series state are arranged according to pose. However, the Examiner interprets that Petrozskaya teaches the main concept of data logs have first and subsequent poses data that is used to obtain the separation distance between poses to determine the estimated pose, the additional details of the function and characteristics of the image features of the main concepts as stated above by the applicant in the amendments is taught by Sun in the details of the rejection below. The Examiner will maintain prior art Petrozskaya and details of the rejection are below.
In response to Argument 5, Applicant’s arguments, see Remarks, filed 10/27/25, with respect to the rejection(s) of claims 1 and 11 and its dependent claims under 35 U.S.C. 103 have been fully considered and are persuasive. Therefore, the rejection has been withdrawn due to the amendment. However, upon further consideration, a new ground(s) of rejection is made for Claim 1 and 11 and its dependent claims under 35 U.S.C. 103 in view of Petrovskaya et al. (US Patent Publication 2016 20160148433 A1 hereafter referred to as Petrovskaya) in view of Zhong et al. (US Patent No 11062471 B1 hereafter referred to as Zhong) in further view of Sun et al (Sun, Penghui, Suping Wu, and Kui Lin. "Attention-guided multi-view stereo network for depth estimation." 2020 IEEE 22nd International Conference on High Performance Computing and Communications; IEEE 18th International Conference on Smart City; IEEE 6th International Conference on Data Science and Systems (HPCC/SmartCity/DSS). IEEE, 2020. hereafter referred to as Sun).
The Examiner finds that Petrovskaya teaches on the amended claim language “reference image of an environment” and “a set of one or more source images of the environment” in Claim 1 and 11.
Specifically, Petrovskaya teaches a system that maps an environment based on an image in ¶0082 ¶0073 and using sets of training data related to the obstacles in the images in ¶0269 and ¶0252-¶0253 to generate a four dimensional feature volume in ¶0263. Petrovskaya teaches in ¶0264-¶0265 that the data logs have first and subsequent poses data that is used to obtain the separation distance between poses to determine the estimated pose. Petrovskaya also teaches an feature volume in relation to the data frames color in ¶0159. Petrovskaya teaches in ¶0264-¶0265 that the data logs have first and subsequent poses data that is used to obtain the separation distance between poses to determine the estimated pose in the scaling series. Applicant argues that the combination of Petrovskaya and Zhong does not teach or suggest amended claim 1. However, the Examiner interprets that Petrozskaya and Zhong teaches the main concept of taking a reference image from an environment using training images to derive parameters from the images to predict poses of the objects in the environment by generation a 4D feature volume associated with features, reducing the volume and applying a depth estimation cost modle, the additional details of the function and characteristics of the main concepts as stated above by the applicant in the amendments is taught by Sun in the details of the rejection below. The Examiner will maintain prior art Petrozskaya and details of the rejection are below.
The Examiner finds that Zhong teaches on the amendment claim language “reducing the 4D feature volume to generate a three dimensional (3D) cost volume” in claim 1, and 11.
Specifically, Zhong teaches reducing the 4D feature volume to a 3D feature volume in Col 6 Line 64-64 and Col 3 Line 14-15. Applicant argues that the combination of Petrovskaya and Zhong does not teach or suggest amended claim 1. However, the Examiner interprets that Zhong does teach the main concept of the dimensionality reduction, the additional details of the functions of the main concepts as stated above by the applicant in the amendments is taught by Sun in the details of the rejection below. The Examiner will maintain prior art Zhong and details of the rejection are below.
Claim Interpretation
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification.
Under MPEP 2143.03, "All words in a claim must be considered in judging the patentability of that claim against the prior art." In re Wilson, 424 F.2d 1382, 1385, 165 USPQ 494, 496 (CCPA 1970). As a general matter, the grammar and ordinary meaning of terms as understood by one having ordinary skill in the art used in a claim will dictate whether, and to what extent, the language limits the claim scope. Language that suggests or makes a feature or step optional but does not require that feature or step does not limit the scope of a claim under the broadest reasonable claim interpretation. In addition, when a claim requires selection of an element from a list of alternatives, the prior art teaches the element if one of the alternatives is taught by the prior art. See, e.g., Fresenius USA, Inc. v. Baxter Int’l, Inc., 582 F.3d 1288, 1298, 92 USPQ2d 1163, 1171 (Fed. Cir. 2009).
Claim 5 recite “at least one of ” then listing “a ray direction of one of the source images
PNG
media_image1.png
13
19
media_image1.png
Greyscale
; a reference plane depth
PNG
media_image2.png
14
21
media_image2.png
Greyscale
; a source plane depth
PNG
media_image3.png
13
21
media_image3.png
Greyscale
; a relative ray angle Bon;
PNG
media_image4.png
13
27
media_image4.png
Greyscale
a relative pose distance p°'n; or a depth validity mask”. Since “at least one of” is disjunctive, any one of the elements found in the prior art is sufficient to reject the claim. While citations have been provided for completeness and rapid prosecution, only one element is required. Because, on balance, it appears the disjunctive interpretation enjoys the most specification support and for that reason the disjunctive interpretation (one of A, B OR C) is being adopted for the purposes of this Office Action. Applicant’s comments and/or amendments relating to this issue are invited to clarify the claim language and the prosecution history.
Claim 9 recite “at least one of ” then listing “the 3D representation is generated without performing a 3D convolution or generating the 3D representation includes fusing the 2D depth map of the reference image with another 2D depth map.”. Since “at least one of” is disjunctive, any one of the elements found in the prior art is sufficient to reject the claim. While citations have been provided for completeness and rapid prosecution, only one element is required. Because, on balance, it appears the disjunctive interpretation enjoys the most specification support and for that reason the disjunctive interpretation (one of A, B OR C) is being adopted for the purposes of this Office Action. Applicant’s comments and/or amendments relating to this issue are invited to clarify the claim language and the prosecution history.
Claim 15 recite “at least one of ” then listing “a ray direction of one of the source images
PNG
media_image1.png
13
19
media_image1.png
Greyscale
; a reference plane depth
PNG
media_image2.png
14
21
media_image2.png
Greyscale
; a source plane depth
PNG
media_image3.png
13
21
media_image3.png
Greyscale
; a relative ray angle Bon;
PNG
media_image4.png
13
27
media_image4.png
Greyscale
a relative pose distance p°'n; or a depth validity mask”. Since “at least one of” is disjunctive, any one of the elements found in the prior art is sufficient to reject the claim. While citations have been provided for completeness and rapid prosecution, only one element is required. Because, on balance, it appears the disjunctive interpretation enjoys the most specification support and for that reason the disjunctive interpretation (one of A, B OR C) is being adopted for the purposes of this Office Action. Applicant’s comments and/or amendments relating to this issue are invited to clarify the claim language and the prosecution history.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1, 4-11, and 14-20 are rejected under 35 U.S.C. 103 as obvious over Petrovskaya et al. (US Patent Publication 2016 20160148433 A1 hereafter referred to as Petrovskaya) in view of Zhong et al. (US Patent No 11062471 B1 hereafter referred to as Zhong) in further view of Sun et al (Sun, Penghui, Suping Wu, and Kui Lin. "Attention-guided multi-view stereo network for depth estimation." 2020 IEEE 22nd International Conference on High Performance Computing and Communications; IEEE 18th International Conference on Smart City; IEEE 6th International Conference on Data Science and Systems (HPCC/SmartCity/DSS). IEEE, 2020. hereafter referred to as Sun).
Regarding Claim 1, Petrovskaya teaches a method, (¶0068, Petrozskaya, "methods for acquiring and applying a depth determination of an environment in e.g., various augmented reality applications.") comprising:
receiving a reference image of an environment (¶0082, Petrozskaya, "a mapping system may receive the raw depth frame, image frame, and/or capture device orientation data" Figure 1 ¶0073, Petrozskaya, "Initially 100a, a user 110, may scan a capture device 105a (illustrated here as a device similar to that depicted in FIG. 5) about an environment 150." ¶0073, Petrozskaya, "This raw data may be recorded on the capture device 105a into a data log (including, e.g., depth, RGB, and IMU data) as the user walks through and/or scans the environment 150.") and a set of one or more source images of the environment (¶0269, Petrozskaya, , "A training set of positive and negative examples can be constructed as follows. The training data log may be collected so that it contains a good number of dynamic obstacles ( e.g., people moving around, objects out of place, and so on) ground truth poses can be computed by initializing the first frame (e.g., by global localization and/or manually) and tracking from that pose for the rest of the log, making sure the pose alignment appears correct throughout. ");
generating a four dimensional (4D) feature volume (¶0263, Petrozskaya, "modify the Scaling Series state to be 4 dimensional (e.g., (x, y, z, a)), and use the 4 dimensional state with the down vector in local coordinates as") that includes the image features (¶0147, Petrozskaya, "the mapper uses a number of different parameters, e.g.: a desired grid resolution, a truncation distance, the contribution of different weighing components (as in EQN. 6), various parameters for pose estimation, etc." Examiner is interpreting parameter as equivalent with feature) and metadata (¶0269, Petrozskaya, "data log may be collected so that it contains a good number of dynamic obstacles ( e.g., people moving around, objects out of place, and so on).") associated with the reference image and set of source images (¶0073, Petrozskaya, "As the user 110 moves the capture device 105a, the capture device 105a may acquire a plurality of depth frames 115a, 115b, 115c using the depth sensor. Each depth frame may provide depth values for each point in the capture device's 105a field of view. This raw data may be recorded on the capture device 105a into a data log (including, e.g., depth, RGB, and IMU data) as the user walks through and/or scans the environment 150." Examiner interprets the plurality of depth frame as equivalent to the source images), the image features (¶0147, Petrozskaya, "the mapper uses a number of different parameters, e.g.: a desired grid resolution, a truncation distance, the contribution of different weighing components (as in EQN. 6), various parameters for pose estimation, etc." Examiner is interpreting parameter as equivalent with feature) and the metadata (¶0269, Petrozskaya, "data log may be collected so that it contains a good number of dynamic obstacles ( e.g., people moving around, objects out of place, and so on).") arranged in the 4D feature volume (¶0263, Petrozskaya, "modify the Scaling Series state to be 4 dimensional (e.g., (x, y, z, a)), and use the 4 dimensional state with the down vector in local coordinates as") based on relative pose distances between the reference image and the set of source images (¶0123-¶0126, Petrozskaya teaches using a map and a volumetric grid to estimate distances ¶0260, Petrozskaya, " For example, given a pose of the camera and a scan of data, the system may need to determine whether this pose is correct or not.");
applying a depth estimation model to the 3D cost volume (¶0068, Petrozskaya, "A user may passively or actively scan a device (e.g., a tablet device, a mobile phone device, etc.) about the environment acquiring depth data for various regions. The system may integrate these scans into an internal three-dimensional model.") and data based on the reference image to generate a two dimensional (2D) depth map for the reference image (¶0166, Petrozskaya, "The process of assigning texture coordinates to all mesh vertices is called "mesh unwrapping" as it "unwraps" the 3D mesh onto the plane of the 2D texture.").
Petrovskaya does not explicitly teach reducing the 4D feature volume to generate a three dimensional (3D) cost volume.
Zhong is in the same field of depth image analysis. Further, Zhong teaches reducing the 4D feature volume (Col 6 Line 64-65, Zhong, "the feature dimension may be reduced accordingly.") to generate a three dimensional (3D) cost volume (Col 3, Line 14-15, Zhong, "3D cost volume is then processed to generate a disparity map for each stereo image pair.").
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Petrovskaya by incorporating reducing the 4D feature volume to a 3D feature volume, mapping the pose distance in order, and implementing a neural network that is taught by Zhong, to make an invention that can calculate the depth in images using fewer features than other methods similar in the art; thus, one of ordinary skilled in the art would be motivated to combine the references since by reducing the features needed for the environmental reconstruction the computation speed is decreased so that the analysis can be done in real time (Zhong, Col 1, Lines 15-35).
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
Petrovskaya and Zhong in combination do not explicitly disclose receiving image features of the reference image and the set of source images;
the image features representing visual information of the reference image and the set of source images.
Sun is in the same field of 3D reconstruction of images. Further, Sun teaches receiving image features (Sun Pg 3 Col 1 ¶01 discloses image features with detailed information including edge texture details) of the reference image (Sun Fig 1 and Pg 2 Col 2 ¶01 discloses a reference image and a set of adjacent source images)
and the set of source images (Sun Fig 1 and Pg 2 Col 2 ¶01 discloses a reference image and a set of adjacent source images);
the image features (Sun Pg 3 Col 1 ¶01 discloses image features with detailed information including edge texture details) representing visual information (Sun Pg 3 Col 1 ¶01 discloses image features with detailed information including edge texture details and Pg 3 Col 1 ¶03 discloses using visual selective attention for depth map refinement) of the reference image and the set of source images (Sun Fig 1 and Pg 2 Col 2 ¶01 discloses a reference image and a set of adjacent source images).
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Petrovskaya in view of Zhong by incorporating a reference and source image with image features representing visual information as taught by Sun, to make an invention that can calculate the depth in images using less computational power than other methods similar in the art; thus, one of ordinary skilled in the art would be motivated to combine the references since there is a need to produce a reconstruction with a refined predicted depth map but with the use of less computational power (Sun, Pg 2, Col 1 ¶01).
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
Regarding Claim 4, Petrovskaya in view of Zhong in view of Sun discloses the method of claim 1, wherein the image features and metadata (¶0267, Petrozskaya, "Various parameters for Scaling Series (including final uncertainty 11*, number of particles per II-neighborhood, etc.) can be learned by maximizing the number of successful localizations per second of computational time for a large set of data frames from a data log."), are arranged in the 4D feature volume according to ascending or descending order of relative pose distance (Col 2, Lines 44-52, Zhong, "In general, each disparity level is associated with 45 a different distance from the viewpoint. The output of the stereo matching neural network system is a disparity map indicating a disparity value for each pixel for a pair of stereo images. Generally, the disparity value corresponds to depth or distance ( disparity is actually inversely proportional to 50 depth) from a viewpoint (e.g., camera) to an object visible at the pixel in an image captured by the camera."). See rationale for Claim 1, its parent claim.
Regarding Claim 5, Petrovskaya in view of Zhong in view of Sun discloses the method of claim 1, wherein the metadata in the 4D feature volume (¶0263, Petrozskaya, "modify the Scaling Series state to be 4 dimensional (e.g., (x, y, z, a)), and use the 4 dimensional state with the down vector in local coordinates as") includes at least one of:
a ray direction of the reference image
r
k
,
i
,
j
0
(¶0336, Petrozskaya, "Thus, instead of just checking the ray P, to Pk for obstacles, some embodiments may check a rectangle of width w that inscribes this ray (and is aligned with the ray's direction),"); ;
a ray direction of one of the source images of the set of source images (Sun Fig 1 and Pg 2 Col 2 ¶01 discloses a reference image and a set of adjacent source images)
r
k
,
i
,
j
n
(¶0336, Petrozskaya, "Thus, instead of just checking the ray P, to Pk for obstacles, some embodiments may check a rectangle of width w that inscribes this ray (and is aligned with the ray's direction),");
a reference plane depth
z
k
,
i
,
j
0
(¶0228, Petrozskaya, "but may allow for point-to-plane distance computations.");
a source plane depth
z
k
,
i
,
j
n
(¶0228, Petrozskaya, "but may allow for point-to-plane distance computations.");
a relative ray angle
θ
0
,
n
(¶0136, Petrozskaya, "The incidence weight may depend on the angle at which the ray of the pixel hits the surface.");
a relative pose distance
p
0
,
n
(¶0034, Petrozskaya, "is an idealized two-dimensional representation depicting a "Point-to-Plane" metric for pose assessment as may be applied in some embodiments;"); or
a depth validity mask
m
k
,
i
,
j
n
(¶0192, Petrozskaya, "A data point may be considered an outlier, e.g., if it does not match up to a valid model point, is too far from the matched model point (more than some threshold
k
d
i
s
t
,), does not have a valid normal, or its normal is too different from the model normal. "). See rationale for Claim 1, its parent claim.
Regarding Claim 6, Petrovskaya in view of Zhong in view of Sun discloses the method of claim 1, wherein the depth estimation model includes a 2D convolutional neural network (Col 5, Lines 49-50, Zhong, "The matching neural network 130 uses 2D convolutions to compute costs and learn cost 50 aggregation.") including an encoder-decoder architecture augmented with the cost volume (Col 4, Line 20-23, Zhong, "The cost aggregation volume neural network model 115 comprises an encoder-decoder matching neural network that learns to match pixels between pairs of left and right 2D feature maps to build a cost volume 103."). See rationale for Claim 1, its parent claim.
Regarding Claim 7, Petrovskaya in view of Zhong in view of Sun discloses the method of claim 1, wherein reducing the 4D feature volume includes reducing volumetric cells (Col 6 Line 64-65, Zhong, "the feature dimension may be reduced accordingly.") of the 4D feature volume in parallel into a feature map (¶0128, Petrozskaya, "3D volumetric grid, where the value of each lattice point in the grid ( or cell corner, in some embodiments the value of non-lattice points is the interpolated value from the surrounding lattice points) is the estimated signed distance d to the map surface ( e.g., to the surface of an object)."). See rationale for Claim 1, its parent claim.
Regarding Claim 8, Petrovskaya in view of Zhong in view of Sun discloses the method of claim 1, further comprising generating a 3D representation of the environment (¶0366, Petrozskaya, "The 3D maps produced using the disclosed techniques may be metrically accurate, allowing the user to easily measure dimensions of rooms") based on the 2D depth map of the reference image (¶0166, Petrozskaya, "The process of assigning texture coordinates to all mesh vertices is called "mesh unwrapping" as it "unwraps" the 3D mesh onto the plane of the 2D texture."). See rationale for Claim 1, its parent claim.
Regarding Claim 9, Petrovskaya in view of Zhong in view of Sun discloses the method of claim 8, wherein at least one of:
the 3D representation is generated without performing a 3D convolution (Col 1, Lines 42-44, Zhong, "A neural network may be used to generate disparity maps in real time by matching image features in stereo images using only 2D convolutions.") or
generating the 3D representation includes fusing the 2D depth map of the reference image with another 2D depth map (Col 3, Lines 11-14, Zhong, "a 3D efficient cost aggregation volume is generated by combining the 2D cost maps corresponding to the plurality of disparity levels."). See rationale for Claim 1, its parent claim.
Regarding Claim 10, Petrovskaya in view of Zhong in view of Sun discloses the method of claim 1, wherein the image features of the reference image are generated by a first feature extractor model (Col 4, Lines 3-5, Zhong, "The feature extraction neural network model 110 may be configured to generate one or more feature maps 101 and/or 102 in parallel for the stereo image pair.") and the data based on the reference image includes second image features of the reference image generated by a second feature extractor model different from the first feature extractor model (Col 6, Lines 27-33, Zhong, "second feature maps are received by the cost aggregation volume neural network model 115. In an embodiment, the second feature maps are extracted from a 30 second image in the stereo image pair by the feature extraction neural network model 110, where each second feature map corresponds to a different disparity level in a set of disparity levels."). See rationale for Claim 1, its parent claim.
Regarding Claim 11, Petrovskaya teaches a non-transitory computer-readable medium (¶0436 , Petrovskaya "The memory 6310 and storage devices 6320 are computer-readable storage media that may store instructions that implement at least portions of the various embodiments. In addition, the data structures and message structures may be stored or transmitted via a data transmission medium, e.g., a signal on a communications link. Various communications links may be used, e.g., the Internet, a local area network, a wide area network, or a point-to-point dial-up connection. Thus, computer readable media can include computer-readable storage media (e.g., "non transitory" media) and computer-readable transmission media.") storing instructions that, when executed by a computing system, cause the computing system to perform operations (¶0437, Petrovskaya "The instructions stored in memory 6310 can be implemented as software and/or firmware to program the processor(s) 6305 to carry out actions described above.") comprising:
receiving a reference image of an environment (¶0082, Petrozskaya, "a mapping system may receive the raw depth frame, image frame, and/or capture device orientation data" Figure 1 ¶0073, Petrozskaya, "Initially 100a, a user 110, may scan a capture device 105a (illustrated here as a device similar to that depicted in FIG. 5) about an environment 150." ¶0073, Petrozskaya, "This raw data may be recorded on the capture device 105a into a data log (including, e.g., depth, RGB, and IMU data) as the user walks through and/or scans the environment 150.") and a set of one or more source images of the environment (¶0269, Petrozskaya, , "A training set of positive and negative examples can be constructed as follows. The training data log may be collected so that it contains a good number of dynamic obstacles ( e.g., people moving around, objects out of place, and so on) ground truth poses can be computed by initializing the first frame (e.g., by global localization and/or manually) and tracking from that pose for the rest of the log, making sure the pose alignment appears correct throughout. ");
generating a four dimensional (4D) feature volume (¶0263, Petrozskaya, "modify the Scaling Series state to be 4 dimensional (e.g., (x, y, z, a)), and use the 4 dimensional state with the down vector in local coordinates as") that includes the image features (¶0147, Petrozskaya, "the mapper uses a number of different parameters, e.g.: a desired grid resolution, a truncation distance, the contribution of different weighing components (as in EQN. 6), various parameters for pose estimation, etc." Examiner is interpreting parameter as equivalent with feature) and metadata (¶0269, Petrozskaya, "data log may be collected so that it contains a good number of dynamic obstacles ( e.g., people moving around, objects out of place, and so on).") associated with the reference image and set of source images (¶0073, Petrozskaya, "As the user 110 moves the capture device 105a, the capture device 105a may acquire a plurality of depth frames 115a, 115b, 115c using the depth sensor. Each depth frame may provide depth values for each point in the capture device's 105a field of view. This raw data may be recorded on the capture device 105a into a data log (including, e.g., depth, RGB, and IMU data) as the user walks through and/or scans the environment 150." Examiner interprets the plurality of depth frame as equivalent to the source images), the image features (¶0147, Petrozskaya, "the mapper uses a number of different parameters, e.g.: a desired grid resolution, a truncation distance, the contribution of different weighing components (as in EQN. 6), various parameters for pose estimation, etc." Examiner is interpreting parameter as equivalent with feature) and the metadata (¶0269, Petrozskaya, "data log may be collected so that it contains a good number of dynamic obstacles ( e.g., people moving around, objects out of place, and so on).") arranged in the 4D feature volume (¶0263, Petrozskaya, "modify the Scaling Series state to be 4 dimensional (e.g., (x, y, z, a)), and use the 4 dimensional state with the down vector in local coordinates as") based on relative pose distances between the reference image and the set of source images (¶0123-¶0126, Petrozskaya teaches using a map and a volumetric grid to estimate distances ¶0260, Petrozskaya, " For example, given a pose of the camera and a scan of data, the system may need to determine whether this pose is correct or not.");
applying a depth estimation model to the 3D cost volume (¶0068, Petrozskaya, "A user may passively or actively scan a device (e.g., a tablet device, a mobile phone device, etc.) about the environment acquiring depth data for various regions. The system may integrate these scans into an internal three-dimensional model.") and data based on the reference image to generate a two dimensional (2D) depth map for the reference image (¶0166, Petrozskaya, "The process of assigning texture coordinates to all mesh vertices is called "mesh unwrapping" as it "unwraps" the 3D mesh onto the plane of the 2D texture.").
Petrovskaya does not explicitly teach reducing the 4D feature volume to generate a three dimensional (3D) cost volume.
Zhong is in the same field of depth image analysis. Further, Zhong teaches reducing the 4D feature volume (Col 6 Line 64-65, Zhong, "the feature dimension may be reduced accordingly.") to generate a three dimensional (3D) cost volume (Col 3, Line 14-15, Zhong, "3D cost volume is then processed to generate a disparity map for each stereo image pair.").
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Petrovskaya by incorporating reducing the 4D feature volume to a 3D feature volume, mapping the pose distance in order, and implementing a neural network that is taught by Zhong, to make an invention that can calculate the depth in images using fewer features than other methods similar in the art; thus, one of ordinary skilled in the art would be motivated to combine the references since by reducing the features needed for the environmental reconstruction the computation speed is decreased so that the analysis can be done in real time (Zhong, Col 1, Lines 15-35).
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
Petrovskaya and Zhong in combination do not explicitly disclose receiving image features of the reference image and the set of source images;
the image features representing visual information of the reference image and the set of source images.
Sun is in the same field of 3D reconstruction of images. Further, Sun teaches receiving image features (Sun Pg 3 Col 1 ¶01 discloses image features with detailed information including edge texture details) of the reference image (Sun Fig 1 and Pg 2 Col 2 ¶01 discloses a reference image and a set of adjacent source images)
and the set of source images (Sun Fig 1 and Pg 2 Col 2 ¶01 discloses a reference image and a set of adjacent source images);
the image features (Sun Pg 3 Col 1 ¶01 discloses image features with detailed information including edge texture details) representing visual information (Sun Pg 3 Col 1 ¶01 discloses image features with detailed information including edge texture details and Pg 3 Col 1 ¶03 discloses using visual selective attention for depth map refinement) of the reference image and the set of source images (Sun Fig 1 and Pg 2 Col 2 ¶01 discloses a reference image and a set of adjacent source images).
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Petrovskaya in view of Zhong by incorporating a reference and source image with image features representing visual information as taught by Sun, to make an invention that can calculate the depth for 3D reconstruction in images using less computational power than other methods similar in the art; thus, one of ordinary skilled in the art would be motivated to combine the references since there is a need to produce a reconstruction with a refined predicted depth map but with the use of less computational power (Sun, Pg 2, Col 1 ¶01).
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
Regarding Claim 14, Petrovskaya in view of Zhong in view of Sun discloses the non-transitory computer-readable medium of claim 11, wherein the image features and metadata (¶0267, Petrozskaya, "Various parameters for Scaling Series (including final uncertainty 11*, number of particles per II-neighborhood, etc.) can be learned by maximizing the number of successful localizations per second of computational time for a large set of data frames from a data log.") are arranged in the 4D feature volume according to ascending or descending order of relative pose distance (Col 2, Lines 44-52, Zhong, "In general, each disparity level is associated with 45 a different distance from the viewpoint. The output of the stereo matching neural network system is a disparity map indicating a disparity value for each pixel for a pair of stereo images. Generally, the disparity value corresponds to depth or distance ( disparity is actually inversely proportional to 50 depth) from a viewpoint (e.g., camera) to an object visible at the pixel in an image captured by the camera."). See rationale for Claim 11, its parent claim.
Regarding Claim 15, Petrovskaya in view of Zhong in view of Sun discloses the non-transitory computer-readable medium of claim 11, wherein the metadata in the 4D feature volume includes at least one of:
a ray direction of the reference image
r
k
,
i
,
j
0
(¶0336, Petrozskaya, "Thus, instead of just checking the ray P, to Pk for obstacles, some embodiments may check a rectangle of width w that inscribes this ray (and is aligned with the ray's direction),"); ;
a ray direction of one of the source images of the set of source images (Sun Fig 1 and Pg 2 Col 2 ¶01 discloses a reference image and a set of adjacent source images)
r
k
,
i
,
j
n
(¶0336, Petrozskaya, "Thus, instead of just checking the ray P, to Pk for obstacles, some embodiments may check a rectangle of width w that inscribes this ray (and is aligned with the ray's direction),");
a reference plane depth
z
k
,
i
,
j
0
(¶0228, Petrozskaya, "but may allow for point-to-plane distance computations.");
a source plane depth
z
k
,
i
,
j
n
(¶0228, Petrozskaya, "but may allow for point-to-plane distance computations.");
a relative ray angle
θ
0
,
n
(¶0136, Petrozskaya, "The incidence weight may depend on the angle at which the ray of the pixel hits the surface.");
a relative pose distance
p
0
,
n
(¶0034, Petrozskaya, "is an idealized two-dimensional representation depicting a "Point-to-Plane" metric for pose assessment as may be applied in some embodiments;"); or
a depth validity mask
m
k
,
i
,
j
n
(¶0192, Petrozskaya, "A data point may be considered an outlier, e.g., if it does not match up to a valid model point, is too far from the matched model point (more than some threshold
k
d
i
s
t
,), does not have a valid normal, or its normal is too different from the model normal. "). See rationale for Claim 11, its parent claim.
Regarding Claim 16, Petrovskaya in view of Zhong in view of Sun discloses the non-transitory computer-readable medium of claim 11, wherein the depth estimation model includes a 2D convolutional neural network (Col 5, Lines 49-50, Zhong, "The matching neural network 130 uses 2D convolutions to compute costs and learn cost 50 aggregation.") including an encoder-decoder architecture augmented with the cost volume (Col 4, Line 20-23, Zhong, "The cost aggregation volume neural network model 115 comprises an encoder-decoder matching neural network that learns to match pixels between pairs of left and right 2D feature maps to build a cost volume 103."). See rationale for Claim 11, its parent claim.
Regarding Claim 17, Petrovskaya in view of Zhong in view of Sun discloses the non-transitory computer-readable medium of claim 11, wherein reducing the 4D feature volume includes reducing volumetric cells (Col 6 Line 64-65, Zhong, "the feature dimension may be reduced accordingly.") of the 4D feature volume in parallel into a feature map (¶0128, Petrozskaya, "3D volumetric grid, where the value of each lattice point in the grid ( or cell corner, in some embodiments the value of non-lattice points is the interpolated value from the surrounding lattice points) is the estimated signed distance d to the map surface ( e.g., to the surface of an object)."). See rationale for Claim 11, its parent claim.
Regarding Claim 18, Petrovskaya in view of Zhong in view of Sun discloses the non-transitory computer-readable medium of claim 11, further comprising generating a 3D representation of the environment (¶0366, Petrozskaya, "The 3D maps produced using the disclosed techniques may be metrically accurate, allowing the user to easily measure dimensions of rooms") based on the 2D depth map of the reference image (¶0166, Petrozskaya, "The process of assigning texture coordinates to all mesh vertices is called "mesh unwrapping" as it "unwraps" the 3D mesh onto the plane of the 2D texture."). See rationale for Claim 11, its parent claim.
Regarding Claim 19, Petrovskaya in view of Zhong in view of Sun discloses the non-transitory computer-readable medium of claim 18, wherein the 3D representation is generated without performing a 3D convolution (Col 1, Lines 42-44, Zhong, "A neural network may be used to generate disparity maps in real time by matching image features in stereo images using only 2D convolutions."). See rationale for Claim 11, its parent claim.
Regarding Claim 20, Petrovskaya in view of Zhong in vie of Sun discloses the non-transitory computer-readable medium of claim 18, wherein generating the 3D representation includes fusing the 2D depth map of the reference image with another 2D depth map (Col 3, Lines 11-14, Zhong, "a 3D efficient cost aggregation volume is generated by combining the 2D cost maps corresponding to the plurality of disparity levels."). See rationale for Claim 11, its parent claim.
Allowable Subject Matter
Claims 2, 3, 12, and 13 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Conclusion
48. THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
49. A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
50. Any inquiry concerning this communication or earlier communications from the examiner should be directed to RACHEL LYNN ROBERTS whose telephone number is (571)272-6413. The examiner can normally be reached Monday- Friday 7:30am- 5:00pm. 32. Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice. 33. If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Oneal Mistry can be reached on 313-446-4912. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300. 34. Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/RACHEL L ROBERTS/Examiner, Art Unit 2674
/ONEAL R MISTRY/Supervisory Patent Examiner, Art Unit 2674