DETAILED ACTION
This action is in response to the remarks and amendments filed on April 7th, 2026. Claims 1-20 are pending and have been examined.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
Applicant’s arguments, see “REMARKS”, filed April 7th, 2026, with respect to the 35 U.S.C. 112(a) and 112(b) rejections of claims 1 and 13 have been fully considered and are persuasive. After further consideration, a semi-ground truth depth map is considered a term of art. For the purposes of examination, a semi ground truth depth map will be interpreted as a depth map that contains depth measurements at points/pixels in an image which is based on real world data, that is computed with minimal human intervention. The 35 U.S.C. 112(a) and 112(b) rejections of claims 1 and 13 have been withdrawn.
Applicant's arguments with respect to the 35 U.S.C. 103 rejections have been fully considered but they are not persuasive.
Applicant alleges that "Lee and Jae-Han, alone or in combination, do not disclose or suggest that the second depth information is estimated by the other device based on the obtained image and a semi-ground truth (semi-GT) depth map, wherein the semi-GT depth map is based on the first depth information" and "Jae-Han does not disclose or suggest generating a semi-ground-truth map that is based on the first depth information”. Examiner respectfully disagrees. Jae-Han teaches that a relative depth map Rn is estimated based off of depth maps Dn and Dn-1 (Jae-Han Figure 3 and Section 3.3). Depth map Dn is considered first depth information and is created based on the obtained image and is therefore grounded in real world data, and is considered a semi-ground truth depth map. Furthermore, since Rn is created based on Dn, and Dn is created based on the obtained image, Rn is also created based on the obtained image. Therefore, the rejection is maintained.
Applicant alleges that “Jae-Han describes internal feature maps and depth maps within a single neural-network architecture, where an intermediate depth map (e.g., D3) is used as input to other decoder blocks that produce relative depth maps (R3-R6). These internal maps are learned representations within one model; they are not generated from an externally computed first depth information for specific points and then used by a separate device" and "The Examiner suggests that Jae-Han's internal decoder blocks are separate devices… However, Jae-Han teaches a monolithic neural network in which D3 and R3-R6 are simply different layers of the same model.” Examiner respectfully disagrees. A device is defined as "a piece of equipment or mechanism designed to serve a special purpose or perform a special function". As can be seen in figure 2, the decoder part of the Jae-Han's neural network is broken up into sections, each section containing a dense block followed by a variable number of whole strip masking blocks. Under broadest reasonable interpretation, each of these sections could be considered a device, as they each perform different functions and output different depth maps. Therefore, the rejection is maintained.
Applicant alleges that “The Examiner suggests that… the intermediate depth map D3 is a semi-ground-truth depth map… There is no disclosure that D3 is constructed from first depth information for at least one point obtained from object-specific height information. Rather, D3 is just another learned depth map produced by the same network from training data.” Examiner respectfully disagrees. Jae-Han teaches that depth map D3 is created based on the obtained image, with points in the image given a depth based on their estimated real world depth. This is analogous to a semi ground truth depth map, as it is semi accurate information which is based on real world data which requires minimal human intervention. Furthermore, it was not suggested that D3 is constructed from specific height information. Rather, Lee is cited to teach creating first depth information from height information, and Jae-Han is cited to teach using that first depth information and the obtained image to create second depth information. Therefore, the rejection is maintained.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1, 4, 10, 12-14, and 20 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by US20190340775 (herein after referred to by its primary author, Lee) in view of “Monocular Depth Estimation Using Relative Depth Maps” (herein after referred to by its primary author, Jae-Han).
In regards to claim 1, Lee teaches a device including at least one processor configured to implement steps of a method of verifying estimated depth information including: obtaining an image of a scene, wherein the image comprises at least one object of interest having a set of points; (Lee Paragraph [0048] “At operation 506, the example process 500 may include receiving an image of an environment, according to any of the techniques discussed herein.”) obtaining a height information for at least one point from the set of points of the object of interest; estimating a first depth information for the at least one point, based on the obtained height information, and detecting a corresponding position of the at least one point in the obtained image (Lee Figure 5A steps 506-514; Paragraph [0034] “In some examples, to identify the depth of the detected object, a monocular height may be used that takes as input an object classification and/or the ROI 210. U.S. application Ser. No. 15/453,569, titled “Object Height Estimation from Monocular Images” and filed Mar. 8, 2017 describes such a model and is incorporated herein by reference. The monocular image model may include a machine-learned model such as, for example, a convolutional neural network (CNN). In some examples, the monocular image model may receive an image (e.g., the ROI 210) and/or object classification as input and may output a probability distribution similar to the example probability distribution 300.” Examiner note: This section shows that a height can be used to estimate a probability of estimated depths.); receiving, from an other device, a second depth information for the at least one point Lee Figure 5A 512); and determining a validity of the estimated second depth information, based on determining a measure of dissimilarity between the first depth information and the second depth information for the at least one point. (Lee Paragraph [0036] “In the depicted example, and in a system using the improved techniques discussed herein, the techniques may include identifying, from the probability distribution 300, a probability that corresponds to a depth measurement associated with a LIDAR point. For example, in FIG. 3, LIDAR point 306 is associated with a lowest probability, LIDAR point 302 is associated with a slightly higher probability, and LIDAR point 304 is associated with a highest probability of the three LIDAR points depicted.” Examiner note: The probability distribution from the monocular image model is used and compared to the point from the LIDAR measurement (the second depth information) and the most similar point is picked. This is analogous to determining validity based on dissimilarity, as the point is picked (analogous to determining to be valid) based on the measure of similarity (analogous to dissimilarity).)
Lee fails to teach wherein the second depth information is estimated by the other device based on the obtained image and a semi-ground truth (semi-GT) depth map, wherein the semi-GT depth map is based on the first depth information.
However, Jae-Han teaches obtaining an image of a scene (Jae-Han Figures 1&2 “Input Image”); estimating a first depth information for the at least one point, Jae-Han Figures 1&2 “D3”); receiving, from an other device, a second depth information for the at least one point, wherein the second depth information is estimated by the other device based on the obtained image and a semi-ground truth (semi-GT) depth map, wherein the semi-GT depth map is based on the first depth information (Jae-Han Figures 1&2 “R3, R4, R5, or R6”; Section 3.3. “First, in decoder 6, the relative depths for all pixel pairs in the lowest-resolution depth map D3 are estimated.” Examiner note: The “decoder part” in figure 2 is made up of many different dense blocks, where dense block D1 estimates the first depth information, and dense blocks D6, D7, D8, or D9 estimates the corresponding second depth information. These separate blocks could be reasonably interpreted as separate devices, as some result in ordinary depth maps, and others result in relative depth maps, as can be seen in figure 1. Furthermore, section 3.3. teaches that decoder 6 uses the depth map D3 to create the relative depth map R3. The depth map D3 is considered to be analogous to a semi ground truth depth map, as they are both semi accurate predictions of the real depth for points/pixels within an image, which is computed automatically with no human intervention.); determine a validity of the estimated second depth information (Jae-Han Section 3.4 “Thus, by combining all these maps at multiple resolutions, we obtain a faithful depth map that takes advantages of those component maps… For the optimal combination, we minimize the mean squared error, subject to constraints on weighting parameters (e.g. nonnegativity of weights), using the interior point method [1].”)
Jae-Han is considered to be analogous to the claimed invention because they are both in the same field of depth map comparison and combination. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have modified the system of Lee to include the teachings of Jae-Han, to provide the benefit of reduced error (Jae-Han Table 3).
In regards to claim 4, Lee in view of Jae-Han teaches the device according to claim 1, wherein: determining the validity comprises verifying the second depth information, for the at least one point, based on the measure of dissimilarity being below a first threshold. (Lee Paragraph [0020] “The techniques may compare this [distance between the primary and secondary depth estimate] to a threshold difference that may be statically defined (e.g., 1.5 meters, 3 meters) or that may be associated with a classification of the detected object (e.g., 6 meters for semi-trucks, 3 meters for pickup trucks, 2 meters for passenger vehicles, 1 meter for compact vehicles).”; Paragraph [0021] “If the difference is less than or equal to the threshold difference (e.g., the difference between the two estimates is 1 meter and the detected object is a passenger vehicle associated with a threshold difference of 2 meters), the techniques may identify the estimates as both corresponding to the detected object.”)
In regards to claim 10, Lee in view of Jae-Han teaches the device according to claim 1, wherein: the measure of dissimilarity is determined based on one or more of: computing a mean square error; computing a mean absolute error; computing an absolute relative error. (Lee Paragraph [0036] “In the depicted example, and in a system using the improved techniques discussed herein, the techniques may include identifying, from the probability distribution 300, a probability that corresponds to a depth measurement associated with a LIDAR point. For example, in FIG. 3, LIDAR point 306 is associated with a lowest probability, LIDAR point 302 is associated with a slightly higher probability, and LIDAR point 304 is associated with a highest probability of the three LIDAR points depicted.” Examiner note: In this reference, the probability is analogous to a predicted error, since a probability is a measure of how likely it is that a measurement is a certain value.)
In regards to claim 12, Lee in view of Jae-Han teaches the device according to claim 1, wherein: the scene is a static scene and the at least one object of interest is located within the static scene. (Lee Figure 5A Step 506)
In regards to claim 13, Lee in view of Jae-Han renders obvious the claim language as in the consideration of claim 1.
In regards to claim 14, Lee in view of Jae-Han renders obvious the claim language as in the consideration of claims 1 and 13.
In regards to claim 20, Lee in view of Jae-Han renders obvious the claim language as in the consideration of claims 4 and 13.
Claims 2 and 15 are rejected under 35 U.S.C. 103 as being unpatentable over Lee in view Jae-Han, and further in view of US10733482 (herein after referred to by its primary author, Lee2).
In regards to claim 2, Lee in view of Jae-Han teaches the device according to claim 1, but fails to teach wherein: estimating the first depth information includes: receiving position information with respect to a ground plane, of a camera capturing the image of the scene; determining, in the obtained image, a set of pixels corresponding to the position of the at least one point for which the height information is obtained; and estimating the first depth information based on the determined set of pixels in the obtained image and the received position information of the camera.
However, Lee2 teaches wherein estimating the first depth information includes: receiving position information with respect to a ground plane, of a camera capturing the image of the scene (Lee2 Column 7 Line 20 Equation; Column 7 Line 25 “K is the image capture device’s intrinsic matrix” Examiner note: The intrinsic matrix of the image capture device is analogous to the position information of a camera); determining, in the obtained image, a set of pixels corresponding to the position of the at least one point for which the height information is obtained (Lee2 Column 7 Line 20 equation; Column 7 Line 24 “where (x,y,z) is a point in the environment relative to the image capture device”); and estimating the first depth information based on the determined set of pixels in the obtained image and the received position information of the camera (Lee2 Column 7 line 40; Column 7 line 43 “where d.sub.i is the depth associated with pixel (u, v).sub.i.”).
Lee2 is considered to be analogous to the claimed invention because they are both in the same field of estimating measurements from RGB images. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have modified the system of Lee in view Jae-Han to include the height estimation teachings of Lee2, to provide the advantage of reduced expense when compared to LIDAR ranging systems (Lee2 Background “For example, various autonomous systems, such as autonomous vehicles and autonomous drones, utilize data indicative of object dimensions and/or locations for collision and obstacle avoidance. In order to effectively navigate a three dimensional environment, such autonomous systems need information about the obstacle sizes (e.g. any or all of a height, width, or length) and/or locations. Additionally, these systems require estimates of how such objects interact with the environment… Though various sensors, such as radar and LIDAR, can provide location information of objects in an environment, they are much more expensive than simple camera systems.”)
In regards to claim 15, Lee in view of Jae-Han and Lee2 renders obvious the claim limitations as in the consideration of claims 2 and 13.
Claims 3, 5-6, 11, an 16-18 are rejected under 35 U.S.C. 103 as being unpatentable over Lee in view Jae-Han, and further in view of US20150302570 (herein after referred to by its primary author, Shirakyan).
In regards to claim 3, Lee in view Jae-Han teaches determining a validity of the adjusted second depth information, based on determining a measure of dissimilarity between the first depth information and the adjusted second depth information for the at least one point. (Lee Paragraph [0036] “In the depicted example, and in a system using the improved techniques discussed herein, the techniques may include identifying, from the probability distribution 300, a probability that corresponds to a depth measurement associated with a LIDAR point. For example, in FIG. 3, LIDAR point 306 is associated with a lowest probability, LIDAR point 302 is associated with a slightly higher probability, and LIDAR point 304 is associated with a highest probability of the three LIDAR points depicted.”)
Lee in view Jae-Han fails to teach adjusting the second depth information, for the at least one point.
However, Shirakyan teaches adjusting the second depth information, for the at least one point (Shirakyan Paragraph [0025] “The depth correction system 104 further includes an adjustment determination component 112 that determines per-pixel correction values for the pixels utilizing depth calibration data 114 for a non-linear error model calibrated for the depth sensor 102.”)
Shirakyan is considered to be analogous to the claimed invention because they are both in the same field of depth sensor verification. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have modified the system of Lee in view Jae-Han to include the depth information adjustments of Shirakyan, to provide the advantage of accounting for factory errors, which result in systemic biases (Shirakyan Paragraph [0027] “Thus, at runtime responsive to the input depth image 106 being produced by the depth sensor 102, the adjustment determination component 112 can compute the per-pixel correction values (e.g., errors for each pixel in the input depth image 106) based upon the reported depth values of the pixels. Moreover, the compensation component 116 can subtract the per-pixel correction values from the corresponding depth values (e.g., the per-pixel errors can be subtracted from the raw readings from the input depth image 106). Accordingly, the depth correction system 104 can compensate for systemic biases (negative and/or positive).”)
In regards to claim 5, Lee in view of Jae-Han and Shirakyan teaches the device according to claim 3, wherein: adjusting the second depth information comprises fine-tuning a depth estimation system of the other device, based on an optimization technique. (Shirakyan Paragraph [0025] “The depth correction system 104 further includes an adjustment determination component 112 that determines per-pixel correction values for the pixels utilizing depth calibration data 114 for a non-linear error model calibrated for the depth sensor 102.” Examiner note: This depth correction unit of this reference is able to adjust values per-pixel, this is analogous to fine-tuning the depth estimation, as each pint receives its own tuning.)
In regards to claim 6, Lee in view of Jae-Han and Shirakyan teaches the device according to claims 3, wherein the at least one processor is further configured to implement steps of: optimizing a depth estimation system of the other device based on the first depth information (Shirakyan Paragraph [0025] “The depth correction system 104 further includes an adjustment determination component 112 that determines per-pixel correction values for the pixels utilizing depth calibration data 114 for a non-linear error model calibrated for the depth sensor 102.”; Figure 3 114 Examiner note: The calibration data is analogous to the first depth information.); and receiving, from the other device, an adjusted second depth information estimated based on its optimized depth estimation system. (Shirakyan Paragraph [0026] “Moreover, the depth correction system 104 includes a compensation component 116 that applies the per-pixel correction values to the depth values to generate the corrected depth image 108. For instance, the compensation component 116 can subtract the per-pixel correction values from the corresponding depth values to produce the corrected depth image 108. The corrected depth image 108 can further be outputted by an output component 118 of the depth correction system 104. The output component 118, for example, can cause the corrected depth image 108 to be displayed on a display screen, retained in a data store, transmitted to a disparate computing device (or computing devices), a combination thereof, and so forth.”)
In regards to claim 11, Lee in view of Jae-Han teaches the device according to claim 1, but fails to teach sending an image to a remote device; and obtaining the measure of dissimilarity, from the remote device.
However, Shirakyan teaches sending an image to a remote device; and obtaining the measure of dissimilarity, from the remote device. (Shirakyan Figure 3 Input image, Corrected Depth Image; Paragraph [0026] “The output component 118, for example, can cause the corrected depth image 108 to be displayed on a display screen, retained in a data store, transmitted to a disparate computing device (or computing devices), a combination thereof, and so forth.”)
Shirakyan is considered to be analogous to the claimed invention because they are both in the same field of depth sensor verification. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have modified the system of Lee in view of Jae-Han to include the depth information adjustments of Shirakyan, to provide the advantage of accounting for factory errors, which result in systemic biases (Shirakyan Paragraph [0027] “Thus, at runtime responsive to the input depth image 106 being produced by the depth sensor 102, the adjustment determination component 112 can compute the per-pixel correction values (e.g., errors for each pixel in the input depth image 106) based upon the reported depth values of the pixels. Moreover, the compensation component 116 can subtract the per-pixel correction values from the corresponding depth values (e.g., the per-pixel errors can be subtracted from the raw readings from the input depth image 106). Accordingly, the depth correction system 104 can compensate for systemic biases (negative and/or positive).”)
In regards to claim 16, Lee in view of Jae-Han and Shirakyan renders obvious the claim language as in the consideration of claims 3 and 13.
In regards to claim 17, Lee in view of Jae-Han and Shirakyan renders obvious the claim language as in the consideration of claims 5 and 16.
In regards to claim 18, Lee in view of Jae-Han and Shirakyan renders obvious the claim language as in the consideration of claims 6 and 16.
Claims 7-9 and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Lee in view of Jae-Han, and further in view of WO2017194962 (herein after referred to by its primary author, Davison).
In regards to claim 7, Lee in view of Jae-Han teaches the device according to claim 1, determining a respective first depth information for a subset of points from the set of points wherein respective first depth information is based on a set of sparse depth maps with sparse depth measurements at the subset of points (Jae-Han Figure 4; Figure 4 Description “A sparse comparison matrix P4,3 is restored to a dense matrix ˜P4,3 by the ALS algorithm. Then, ˜P4,3 is reshaped and normalized to a relative depth map R4.” Examiner note: The sparse comparison matrix P4,3 is analogous to a sparse depth map, as they are both sparse depth representations.) but fails to teach determining a first three-dimensional, 3D, depth-map representation for the at least one object of interest, based on determining a respective first depth information for a subset of points from the set of points; and determining a second 3D depth-map representation for the at least one object of interest, based on determining a respective second depth information for the subset of points from the set of points.
However, Davison teaches determining a first three-dimensional, 3D, depth-map representation for the at least one object of interest, based on determining a respective first depth information for a subset of points from the set of points; and determining a second 3D depth-map representation for the at least one object of interest, based on determining a respective second depth information for the subset of points from the set of points. (Davison Figure 2 “Rendered Depth Map” and “Measured Depth Map”)
Davison is considered to be analogous to the claimed invention because they are both in the same field of depth sensor verification. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have modified the system of Lee in view of Jae-Han to include the depth maps of Davison, to provide the advantage of low cost robots that only map with RGB cameras (Davison Page 8 Paragraph 3 “Therefore, using only a single forward-looking camera to provide detailed maps suitable for precise autonomous navigation, the present apparatus and method may be employed for free-space and obstacle mapping by low- cost robots.”)
In regards to claim 8, Lee in view of Jae-Han and Davison teaches the device according to claim 7, wherein the at least one processor is further configured to implement a step of: determining a measure of dissimilarity between the first 3D depth-map representation and the second 3D depth-map representation for the at least one object of interest. (Davison Figure 2 251, 260 Examiner note: The error between the two depth maps is computer, as seen in step 260.)
In regards to claim 9, Lee in view of Jae-Han and Davison teaches the device according to claim 8, wherein the at least one processor is further configured to implement steps of: verifying the second 3D depth-map representation, based on the measure of dissimilarity being below a second threshold; or adjusting the second 3D depth-map representation, by adjusting one or more of second depth information, based on the measure of dissimilarity being above the second threshold. (Davison Figure 2 261-271; Page 7 Paragraph 3 “The resultant updated rendered depth map 250 is compared (block 251) to the original measured depth map 240, and the nonlinear error 260 between the two is used in conjunction with the partial derivative gradient values235 derived from the rendering process (block 231) to reduce the cost function (block 261).” Examiner note: The surface model that is used to create the rendered depth map is updated, then the rendered depth map is re-rendered (analogous to adjusted) based on the error used to update the surface model.)
In regards to claim 19, Lee in view of Jae-Han and Davison renders obvious the claim language as in the consideration of claims 7 and 13.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure:
“Simultaneous Depth Estimation and Surgical Tool Segmentation in Laparoscopic Images” teaches using a semi-ground truth segmentation map to determine the depth of a medical image.
“Semi-Supervised Deep Learning for Monocular Depth Map Prediction” teaches predicting depth from stereo images, wherein during training sparse ground truth data is used.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to CALEB LOGAN ESQUINO whose telephone number is (703)756-1462. The examiner can normally be reached M-Fr 8:00AM-4:00PM EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Andrew Bee can be reached at (571) 270-5183. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/CALEB L ESQUINO/Examiner, Art Unit 2677
/ANDREW W BEE/Supervisory Patent Examiner, Art Unit 2677
Prosecution Insights