DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Receipt is acknowledged that application claims priority to foreign application with application number CN 202211737780.9 dated 12/30/2022. Copies of certified papers required by 37 CFR 1.55 have been received. Priority is acknowledged under 35 USC 119(e) and 37 CFR 1.78.
Response to Amendment
The amendment filed 04/08/2026 has been entered. Applicant’s amendments to the claims have overcome each and every objection previously set forth in the Non-Final Office Action mailed 01/23/2026. Claims 1, 3-9, 11-15, and 17-20 remain pending in the application, with claims 2, 10, and 16 having been cancelled.
Response to Arguments
Applicant's arguments filed 04/08/2026 have been fully considered but they are not persuasive. On pg. 13 of the Remarks filed 04/08/2026, Applicant argues that “Li fails to disclose and teach obtaining the projected depth values that include the measurement unit information by converting the spatial coordinate values of the point clouds” and “Li fails to disclose the measurement unit information of the LiDAR”. Applicant further argues that amended claim 1 is unobvious and patentable over Cai. Examiner disagrees and submits that Li in view of Cai teaches the amended portion “the projected depth value comprising measurement unit information”. Cai teaches the use of LiDAR data to obtain point cloud data (LIDAR sensor in para 116). Thus, the depth values are expressed in measurement units (¶138: “The depth values 715 can be obtained via the camera tracker 714 in meters or another measurement system such as feet”). This is consistent with examples provided in the original disclosure of the instant application in ¶46: “since the point clouds and the spatial coordinate value are obtained by lidar, the spatial coordinate value includes a measurement unit information of lidar”. Therefore, as described in further detail in the rejection below, Li in view of Cai discloses the measurement unit information of the LiDAR data, resulting in projected depth values that include the measurement unit information in the combination.
On pg. 14-15 of the Remarks filed 04/08/2026, Applicant argues that “Currently amended claimed in claim 1 discloses that the scaling factor is calculated based on the first image and the second image. However, in Li, the scaling factor is obtained using only a single reference image”. In the rejection previously set forth, and maintained below, this limitation is taught in the combination of Li in view of Cai. In response to applicant's arguments against the references individually, one cannot show nonobviousness by attacking references individually where the rejections are based on combinations of references. See In re Keller, 642 F.2d 413, 208 USPQ 871 (CCPA 1981); In re Merck & Co., 800 F.2d 1091, 231 USPQ 375 (Fed. Cir. 1986). As previously set forth in the Non-Final Office Action mailed 01/23/2026, Li fails to teach wherein the pose matrix is generated by the first image and the second image. This deficiency is remedied in combination with Cai. Therefore, the rejection is maintained.
On pg. 15-16 of the Remarks filed 04/08/2026, Applicant argues that “Li fails to disclose how to calculate the loss value based on a reconstructed initial projection image, the first image and the second image captured by the camera device.” Examiner disagrees. Li teaches calculating a loss value according to 1) the first image (¶78: “depth loss component LSDL(f) corresponding to the reference image 201”; see LSDL(f) in the total loss in ¶89), 2) the initial projection image (depth consistency loss is based on the converted depth map, ¶86, and the depth consistency loss is in the total loss in ¶89), and 3) the second image (¶80: “depth loss component LSDL(g) corresponding to the reference image 202”; see LSDL(g) in the total loss in ¶89). Applicant further argues that the present application simplifies the loss value calculation process. The examiner notes that there is no explicit claim language that requires that the first image, initial projection image, and second image are the only loss value components and excludes previous calculations that feed into the total loss value. Claim 1 broadly recites “calculating” different values “based on” or “according to” other determined values. In response to applicant's arguments that the present application avoids potential problems caused by the method of Li (pg. 16, ¶3-4), a recitation of the intended use of the claimed invention must result in a structural difference between the claimed invention and the prior art in order to patentably distinguish the claimed invention from the prior art. Although the claims are interpreted in light of the specification, limitations from the specification are not read into the claims. See In re Van Geuns, 988 F.2d 1181, 26 USPQ2d 1057 (Fed. Cir. 1993). In view of the foregoing, the rejection is maintained.
On pg. 17-18 of the Remarks filed 04/08/2026, Applicant argues that “the pose matrix disclosed in the present application is different from the pose between the input image 305A and the input image 305B disclosed by Cai.” The pose matrix is taught by Li in view of Cai. Cai is solely relied upon to teach wherein the pose matrix is generated by two close video frames (¶92). Similar to the present application, Li teaches a pose matrix used for converting a spatial coordinate value (¶68). A person or ordinary skill in the art could use a pose matrix generated by two images in the method of Li to convert spatial coordinate values. In response to applicant's arguments against the references individually, one cannot show nonobviousness by attacking references individually where the rejections are based on combinations of references. See In re Keller, 642 F.2d 413, 208 USPQ 871 (CCPA 1981); In re Merck & Co., 800 F.2d 1091, 231 USPQ 375 (Fed. Cir. 1986). Therefore, the rejection is maintained.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 4, 9, and 15 are rejected under 35 U.S.C. 103 as being unpatentable over Li et al. (U.S. Patent No. 2022/0198693 A1), hereinafter Li, in view of Cai et al. (U.S. Patent No. 2024/0177329 A1), hereinafter Cai.
Regarding claim 1, Li teaches a method for identifying depths of images (Li, abstract: “obtained trained depth estimation model may provide more accurate depth information”) using a computer device (Li, para 121: “may be implemented as computer software programs”), comprising:
obtaining a plurality of images to be identified (Li, para 116: “The sequence of target images is captured by a target camera”);
obtaining a plurality of target depth images based on the plurality of images to be identified (Li, para 118: “at block 620, a sequence of depth maps corresponding to the sequence of target images may be generated. The method 600 may also include generating a three-dimensional image of the target object based on the sequence of depth maps”), and determining depth information of the plurality of images to be identified by inputting the plurality of images to be identified into a pre-trained image identification model (Li, trained depth estimation model in FIG. 6 attached below, para 117: “a depth map corresponding to at least one target image in the sequence of target images is generated using the trained depth estimation model 104”), a training method of the pre-trained image identification model (Li, see training of the model in para 66 and FIG. 4A) comprising:
obtaining point cloud of a scene and a spatial coordinate value of each point in the point cloud (Li, sparse point cloud, para 39: “The sequence image set 102 and the sparse point cloud 103 may be provided together to the model training apparatus 140 for training the depth estimation model 104”), and obtaining a first image and a second image of the scene captured by a camera device (Li, para 116: “The sequence of target images is captured by a target camera”; reference images 201 and 202);
inputting the first image into a preset depth identification network (Li, see images input to the depth estimation model in FIG. 4A), and obtaining an initial depth image (Li, dense depth map, para 39: “The depth estimation model 104 may be trained to generate a dense depth map of a target object based on different images of the target object”);
converting the spatial coordinate value according to a pose matrix generated by an image and an internal reference matrix of the camera device, and obtaining a projected depth value and a projected coordinate value of each point in the point cloud based on the spatial coordinate value converted (Li, projection performed using the world and camera coordinate system, para 68: “The sparse depth map is obtained by projecting points in the sparse point cloud onto a specific reference image by means of using coordinate transformation. Taking the sparse depth map 401 corresponding to the reference image 201 as an example, firstly, all the points Pnw relative to the world coordinate system are transformed into points Pnf relative to the camera coordinate system”);
calculating a scaling factor for each point in the point cloud (Li, para 73: “The model training apparatus 140 may include a scaling layer 410, the scaling layer 410 may scale the dense depth map 403-1 to a dense depth map 403-2 according to the first scaling factor”;
θ
in Eqn 4 in para 75) according to the projected depth value (Li, Dsf value from para 68-69 – the sparse depth values), a number of points in the point cloud (Li, n value, para 67: “n represents the nth point in the sparse point cloud 103”), and an initial depth value of an initial pixel point corresponding to the projected coordinate value of the initial depth image (Li, Df value referenced in para 75 – the dense depth values; unf is the 2D coordinates of each point, so corresponding values are used, see para 69-70);
calculating a target depth value for each point in the point clouds according to the scaling factor and the initial depth value (Li, para 73: “The model training apparatus 140 may include a scaling layer 410, the scaling layer 410 may scale the dense depth map 403-1 to a dense depth map 403-2 according to the first scaling factor”);
generating an initial projection image (Li, converted depth map 406 in FIG. 4A) based on the pose matrix, the internal reference matrix (Li, the matrices adjust depth values throughout the method, therefore later calculated depth values are all based on them; see also para 82 where the coordinate systems are considered again), the target depth value (Li, dense depth map 403-2 feeds into conversion layer), the second image (Li, para 83: “convert the dense depth map 403-2 into a converted depth map 406 corresponding to the reference image 202”), and a pixel coordinate value of a target pixel point corresponding to the projected coordinate value in the second image (Li, conversion layer is based on corresponding pixel coordinates in both images, see para 83-84, para 84: “pixel coordinates (x.sub.f, y.sub.f) in the reference image 201 corresponding to the pixel coordinates (x.sub.g, y.sub.g) of the reference image 202”); and
calculating a loss value of the preset depth identification network according to the first image, the initial projection image, and the second image (Li, para 89: “When the reference images in the sequence image set 102 are sequentially input to the model training apparatus 140, the total loss may be determined correspondingly as shown in formula (11), so as to train the depth estimation model 104”; formula is according to the initial projection image due to its basis on the distorted depth maps, see para 86-88), and obtaining the pre-trained image identification model by adjusting the preset depth identification network based on the loss value (Li, para 5: “training a depth estimation model at least based on the first loss and the second loss, to obtain the trained depth estimation model”).
PNG
media_image1.png
228
348
media_image1.png
Greyscale
PNG
media_image2.png
532
445
media_image2.png
Greyscale
Li fails to explicitly teach 1) wherein a plurality of point clouds are obtained of a road scene (emphasis added); 2) the projected depth value comprising measurement unit information; and 3) wherein the pose matrix is generated by the first image and the second image (Li teaches in the para 68 citation above that the projection is based on one reference image).
However, Cai teaches a depth estimation method (Cai, abstract: “predicted depth value for each pixel of the image”; see FIG. 8 below), including 1) wherein a plurality of point clouds (Cai, 6DOF data used in the method of FIG. 8, see para 147: “In some cases, the tracker is a six-degree-of-freedom (6DOF)”; data may be LIDAR/SLAM data for input images, para 116: “For example, 6DoF SLAM can use feature point associations from an input image (or other sensor data, such as a radar sensor, LIDAR sensor, etc.) to determine the pose (position and orientation) of the image sensor 402 and/or system 400 for the input image. 6DoF mapping can also be performed to update the SLAM map…keyframes can be selected from input images or a video stream to represent an observed scene. For every keyframe, a respective 6DoF camera pose associated with the image can be determined”) of a road scene (Cai, para 60: “machine learning model that can be deployed and implemented on any device such as an autonomous vehicle”; para 82: “For instance, a road participant (e.g., pedestrian, vehicle) can be easily separated from the background (e.g., road, building) given the different patterns of their depth values”) are obtained and used to determine depth values (see para 147 citation above); 2) depth value comprising measurement unit information (Cai, para 138: “The depth values 715 can be obtained via the camera tracker 714 in meters or another measurement system such as feet”); and 3) a pose matrix generated by two input images (Cai, para 92: “consider two neighboring or close video frames, I.sub.t and I.sub.s (e.g., input images 305A and 305B). Suppose that pixel p.sub.t ϵ I.sub.t and pixel p.sub.s ϵ I.sub.s are two different views of the same point of an object. In such a case, p.sub.t and p.sub.s are related geometrically as indicated in Equation 2 below, where h(p)=[h, w, 1] denotes the homogeneous coordinates”).
PNG
media_image3.png
476
441
media_image3.png
Greyscale
In the method taught by Li, the obtained point cloud generates two sparse depth maps (Li, para 67 and FIG. 4A). A person of ordinary skill in the art would have been able to obtain two sparse depth maps from two sets of point cloud data. Additionally, Li teaches a world coordinate system based on one reference image. A person of ordinary skill in the art would have been able to consider two images when determining a world coordinate system. Thus, it would have been obvious to a person having ordinary skill in the art, before the effective filing date of the claimed invention, to have combined the point cloud data with each image of road scenes and the pose matrix, taught by Cai, with the method of Li in order to 1) have 3D data at the specific pose when each image was taken, making each sparse depth map more representative of its reference image, 2) utilize the method in an autonomous vehicle, and 3) determine a transformed pose based on both images relevant to the prediction process (see citations used in the rejection above).
Additionally, Cai teaches the use of depth sensors, specifically LiDAR, to obtain depth values in meters. Li teaches obtaining point cloud data, but fails to explicitly teach a sensor obtaining data in measurement units. Cai teaches a known technique of using LiDAR sensors to collect point cloud data. A person having ordinary skill in the art, before the effective filing date of the claimed invention, could have applied the known technique, as taught by Cai, in the same way to the method of Li and achieved predictable results of obtaining depth information (depth values and subsequent projected depth values) for the surrounding environment in measurement units to determine accurate distances between a vehicle and surrounding objects.
Regarding claim 4 (dependent on claim 1), Li in view of Cai teaches wherein a calculation formula of the scaling factor is represented as:
PNG
media_image4.png
144
733
media_image4.png
Greyscale
Li teaches this in formula 4 in para 75, attached below. Refer to the claim rejection of claim 1 for further details.
PNG
media_image5.png
381
587
media_image5.png
Greyscale
Regarding claim 9, Li teaches a computer device (Li, see FIG. 7 and para 119-121) comprising: a processor (Li, para 121: “processing unit 701”); and a storage device storing a plurality of instructions (Li, para 121: “computer software programs, which are tangibly contained in a machine-readable medium, such as the storage unit 708”). All further claim limitations are met and rendered obvious by Li in view of Cai because the method steps of claim 1 are the same as the steps executed by the computer device in claim 9.
Regarding claim 15, Li teaches a non-transitory storage medium having stored thereon at least one computer-readable instructions (Li, para 135: “The computer-readable storage medium may be a tangible device that may hold and store instructions used by an instruction execution device”), which when executed by a processor of a computer device (Li, see FIG. 7 and para 119-121), causes the processor to perform a method for detecting image sizes (Detects depth values in image, see citations below). All further claim limitations are met and rendered obvious by Li in view of Cai because the method steps of claim 1 are the same as the method steps performed by the processor in claim 15.
Allowable Subject Matter
Claims 3, 5-8, 11-14, and 17-20 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
The following is a statement of reasons for the indication of allowable subject matter: Regarding claim 3, Li in view of Cai fails to teach further comprising: obtaining a camera coordinate matrix by multiplying a spatial homogeneous matrix corresponding to the spatial coordinate value by the pose matrix; determining a vertical coordinate value of the camera coordinate matrix as the projected depth value; obtaining a camera pose matrix by multiplying the camera coordinate matrix by the internal reference matrix; and performing a division operation on each element value of the camera pose matrix by the projected depth value, and obtaining the projected coordinate value. Li teaches in para 68-69 that “the depth dnf of each point relative to the camera coordinate system of the reference image 201 may be obtained, i.e. a component of Pnf along a Z axis” and the transformation in formula 1. Thus, the projected depth value is not the vertical coordinate value of the camera coordinate matrix. Cai teaches an equation demonstrating the pose between two images, wherein the equation is based on a spatial homogeneous matrix and camera intrinsic matrix, see para 92, but the specific aspects of claim 3 are not disclosed.
While similar methods, such as Zhao et al. (U.S. Patent No. 2023/0145498 A1), hereinafter Zhao, disclose related methods for depth projection (see para 67 of Zhao, recited in part below), they are insufficient to remedy the deficiencies of Li in view of Cai. Therefore, the prior art fails to teach as a whole or in reasonable combination wherein converting the spatial coordinate value according to the pose matrix generated by the first image and the second image and the internal reference matrix of the camera device, and obtaining the projected depth value and the projected coordinate value of each point in the point clouds based on the spatial coordinate value converted further comprises: obtaining a camera coordinate matrix by multiplying a spatial homogeneous matrix corresponding to the spatial coordinate value by the pose matrix; determining a vertical coordinate value of the camera coordinate matrix as the projected depth value; obtaining a camera pose matrix by multiplying the camera coordinate matrix by the internal reference matrix; and performing a division operation on each element value of the camera pose matrix by the projected depth value, and obtaining the projected coordinate value.
Regarding claim 5, Zhao teaches a similar method (Zhao, para 67: “for a given homogeneous coordinate of a point on the source image 304, the image reprojection system 106 reprojects the point based on source camera intrinsic parameters, the target camera intrinsic parameters, the relative camera pose between the two cameras, and a corresponding depth value. For example, in various implementations, the image reprojection system 106 applies an inverse intrinsic matrix for the source camera, a depth rescaling factor, the relative camera matrix (i.e., relative camera pose), and the intrinsic matrix for the target camera to the source image point to reproject a point in the source image to align with the target image”). Li teaches obtaining the initial projection image by adjusting the pixel coordinate value of the target pixel point to be corresponding target coordinate value in the second image (Li, conversion of the dense depth map, para 83: “convert the dense depth map 403-2 into a converted depth map 406 corresponding to the reference image 202”); however, Li and the existing prior art fail to teach as a whole or in reasonable combination wherein generating the initial projection image based on the pose matrix, the internal reference matrix, the target depth value, the second image, and the pixel coordinate value of the target pixel point corresponding to the projected coordinate value in the second image further comprises: constructing a homogeneous coordinate matrix according to the pixel coordinate value of the target pixel point; obtaining an inverse matrix of the internal reference matrix; calculating a target coordinate value of the target pixel point according to the pose matrix, the inverse matrix, the internal reference matrix, the homogeneous coordinate matrix, and the target depth value; and obtaining the initial projection image by adjusting the pixel coordinate value of the target pixel point to be corresponding target coordinate value in the second image.
Regarding claim 7, Li teaches wherein the loss value is based on two types of difference values (Li, para 65: “the first constraint may be a sparse depth loss (SDL) related to the difference between depth maps with different densities, and the second loss may be a depth consistency loss (DCL) related to the depth consistency of different images”); however, Li and the existing prior art fail to teach as a whole wherein calculating the loss value of the preset depth identification network according to the first image, the initial projection image and the second image further comprises: calculating a first pixel difference value between a pixel value of each of pixel points in the first image and a pixel value of corresponding pixel points in the initial projection image; obtaining a first difference image by adjusting the pixel value of each of pixel points in the first image to be corresponding first pixel difference value; calculating a second pixel difference value between the pixel value of each of pixel points in the first image and a pixel value of corresponding pixel points in the second image, and generating a second difference image corresponding to the first image according to the second pixel difference value; obtaining a target image by adjusting the second pixel difference value of the second difference image according to a comparison result of the second pixel difference value with the corresponding first pixel difference value and a preset value; and calculating the loss value according to a pixel value of each of pixel points in the target image and the corresponding first pixel difference value of corresponding pixel points in the first difference image.
For similar reasons, the reasons for the indication of allowable subject matter apply to the corresponding device and storage medium claims, 11-13 and 17-19, respectively. In view of the foregoing, the prior art references alone or in reasonable combination are insufficient to teach the invention as a whole, as claimed in claims 3, 5, 7, 11-13, and 17-19. Due to their dependence on claims 5, 7, 13, or 19, claims 6, 8, 14, and 20 are similarly objected to.
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to EMMA E DRYDEN whose telephone number is (571)272-1179. The examiner can normally be reached M-F 9-5 EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, ANDREW BEE can be reached at (571) 270-5183. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/EMMA E DRYDEN/Examiner, Art Unit 2677
/ANDREW W BEE/Supervisory Patent Examiner, Art Unit 2677
Prosecution Insights