DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claims 1-4, 6, 7, 11-13, and 17-20 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Kamal et al. (U.S. Pub. No. 2019/0182472).
Re claims 1 and 17: Kamal et al. disclose an apparatus for estimating depth (i.e., “depth data generation system”, Abstract), the apparatus comprising/method for estimating depth comprising:
at least one memory (i.e., “storage facility 106”, Paragraph [0030]; and “storage device 1206”, Paragraph [0155]); and
at least one processor coupled to the at least one memory and configured to (i.e., “depth map convergence facility 104”, Paragraph [0030]; and “processor 1204”, Paragraph [0155]):
obtain/obtaining first depth data (i.e., “each node 302 may include equipment and/or devices for performing at least one depth map capture technique (described in more detail below) to capture a depth map”, Paragraph [0052]) from a first depth data source (i.e., FIG. 3, “node 302-1”, Paragraph [0051]), wherein the first depth data is associated with a first field of view (FOV) (i.e., “a scope of capture 306 of node 302-1”, Paragraph [0054]);
obtain/obtaining second depth data (i.e., “each node 302 may include equipment and/or devices for performing at least one depth map capture technique (described in more detail below) to capture a depth map”, Paragraph [0052]) from a second depth data source (i.e., FIG. 3, “node 302-2”, Paragraph [0061]), wherein the second depth data is associated with a second FOV, the second FOV being different from the first FOV (See for example, FIG. 3, scope of capture of node 302-2, “In the setup of implementation 300, each of nodes 302 may be positioned so as to capture all or substantially all of the circular area designated as real-world scene 304 from the perspective (i.e., angle, distance, etc.) afforded by the respective fixed node position of the node. For example, all of the respective areas of nodes 302 may be overlapping with the respective areas of all the other nodes 302 in an area (e.g., a circular area) designated as real-world scene 304”, Paragraph [0054]);
generate/generating FOV adjusted depth data based on the second depth data associated with the second FOV (i.e., “system 100 may provide the virtual reality media content representative of real-world scene 304 as experienced from a dynamically selectable viewpoint corresponding to an arbitrary location within real-world scene 304”, Paragraph [0067]), wherein the FOV adjusted depth data is associated with a target FOV, the target FOV being different from the second FOV (See for example, Paragraphs [0063] and [0068]);
generate/generating a fused depth seed (i.e., “third depth data point”, Paragraph [0130]) based on the FOV adjusted depth data and at least one of the first depth data or an additional FOV adjusted depth data, wherein the fused depth seed is associated with the target FOV (See for example, Paragraphs [0130]-[0131]); and
determine/determining a depth map (i.e., “converged depth map”, Paragraph [0130]) based on the fused depth seed.
Re claims 2 and 18: Kamal et al. disclose wherein the target FOV is the first FOV (i.e., “an “arbitrary location” may refer to any point in space at the real-world event. For example, arbitrary locations are not limited to fixed node positions (e.g., where nodes 302 are disposed) around real-world scene 304, but also include all the positions between nodes 302 and even positions where nodes such as nodes 302 may not be able to be positioned (e.g., in the middle of real-world scene 304)”, Paragraph [0068]).
Re claims 3 and 19: Kamal et al. disclose wherein the target FOV is different from the first FOV (i.e., “an “arbitrary location” may refer to any point in space at the real-world event. For example, arbitrary locations are not limited to fixed node positions (e.g., where nodes 302 are disposed) around real-world scene 304, but also include all the positions between nodes 302 and even positions where nodes such as nodes 302 may not be able to be positioned (e.g., in the middle of real-world scene 304)”, Paragraph [0068]).
Re claims 4 and 20: Kamal et al. disclose wherein the at least one processor is further configured to generate/generating the additional FOV adjusted depth data based on the first depth data associated with the first FOV (i.e., the third depth data point is associated first depth data, Paragraph [0131]), wherein the additional FOV adjusted depth data is associated with the target FOV (i.e., Paragraph [0130]).
Re claim 6: Kamal et al. disclose wherein the at least one processor is further configured to:
obtain an input frame (i.e., “field of view 604”, Paragraph [0085]) associated with the target FOV (i.e., “the media player device may detect user input (e.g., moving or turning the display screen upon which the field of view is presented)”, Paragraph [0085]); and
determine the depth map further based on the input frame, wherein the depth map is associated with the input frame (i.e., “nodes 504 may include video capture devices (e.g., visible light video cameras, etc.) configured to capture texture data (e.g., 2D video data) of objects 506 included in real-world scene 502 that, when combined with depth data representative of objects 506, may be used to generate dynamic volumetric models of the surfaces of objects 506 included in real-world scene 502”, Paragraph [0081]; and Paragraph [0085]).
Re claim 7: Kamal et al. disclose wherein, to generate the FOV adjusted depth data based on the second depth data associated with the second FOV, the at least one processor is configured to filter the second depth data (See for example, “if the first depth data point has been assigned a relatively high confidence value and the second depth data point has been assigned a relatively low confidence value, system 100 may generate a third depth data point between the first and second depth data points that is closer to the first depth data point than to the second depth data point”, Paragraph [0131]) to remove at least one depth value associated with at least one pixel of the second depth data (i.e., when value is fail type, Paragraph [0134]), wherein the at least one pixel is associated with the target FOV (i.e., third depth data point based on confidence, Paragraphs [0131]-[0132] and [0134]-[0135]).
Re claim 11: Kamal et al. disclose wherein, to filter the second depth data, the at least one processor is configured to: determine whether a confidence value associated with the at least one depth value is less than a confidence value threshold (i.e., “Confidence values may be numerical values (e.g., percentage values, numbers on a particular scale, etc.), binary pass/fail-type values”, Paragraph [0037]); and
based on a determination that the confidence value associated with the at least one depth value is less than the confidence value threshold (i.e., when value is fail type, Paragraph [0134]), remove the at least one depth value in the FOV adjusted depth data (i.e., third depth data point based on confidence, Paragraphs [0131]-[0132] and [0134]-[0135]).
Re claim 12: Kamal et al. disclose wherein the first depth data source comprises at least one of: one or more cameras; a six degrees-of-freedom (6DoF) tracking system; a 3DoF tracking system; a Light Detection and Ranging (LiDAR) sensor; a structured light (SL) depth sensor; an indirect time of flight (iToF) sensor; a direct ToF (dToF) sensor; or a depth from stereo (DFS) system (i.e., “Depth map capture subsystem 204 may include any suitable hardware or combination of hardware and software configured to capture a depth map of object 202 from a fixed position at which depth map capture subsystem 204 is disposed. More specifically, depth map capture subsystem 204 may include hardware devices such as optical emitters (e.g., lasers or other devices for generating stimulated emission of electromagnetic radiation at a suitable frequency, camera flash equipment or other devices for generating pulses of light to bathe a real-world scene in light, etc.), optical sensors (e.g., video cameras, infrared (“IR”) sensors, time-of-flight sensors, etc.), and other hardware equipment configured to perform at least one depth map capture technique for capturing a depth map representative of surfaces of objects (e.g., such as object 202) within a real-world scene”, Paragraph [0045]).
Re claim 13: Kamal et al. disclose wherein the second depth data source comprises at least one of: the one or more cameras; the 6DoF tracking system; the 3DoF tracking system; the LiDAR sensor; the SL depth sensor; the iToF sensor; the dToF sensor; or the DFS system (i.e., “Depth map capture subsystem 204 may include any suitable hardware or combination of hardware and software configured to capture a depth map of object 202 from a fixed position at which depth map capture subsystem 204 is disposed. More specifically, depth map capture subsystem 204 may include hardware devices such as optical emitters (e.g., lasers or other devices for generating stimulated emission of electromagnetic radiation at a suitable frequency, camera flash equipment or other devices for generating pulses of light to bathe a real-world scene in light, etc.), optical sensors (e.g., video cameras, infrared (“IR”) sensors, time-of-flight sensors, etc.), and other hardware equipment configured to perform at least one depth map capture technique for capturing a depth map representative of surfaces of objects (e.g., such as object 202) within a real-world scene”, Paragraph [0045]).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 5 and 8-10 are rejected under 35 U.S.C. 103 as being unpatentable over Kamal et al. in view of Sun et al. (U.S. Pub. No. 2020/0090358). The teachings of Kamal et al. have been discussed above.
As to claim 5, Kamal et al. does not explicitly disclose wherein, to generate the FOV adjusted depth data based on the second depth data associated with the second FOV, the at least one processor is configured to: project the second depth data from the second FOV into a three-dimensional (3D) representation of second depth data; and re-project the 3D representation of the second depth data into the target FOV.
Sun et al. teaches generating FOV adjusted depth data (i.e., “optimized depth map DA”, Paragraph [0046]) based on the second depth data associated with the second FOV, the at least one processor is configured to (i.e., “depth processing unit 130”, Paragraph [0016]):
project the second depth data from the second FOV into a three-dimensional (3D) representation of second depth data (i.e., “Transform coordinates of pixels in the first depth map D1 to coordinates in a first three-dimensional coordinate system of the first depth capture device 110 (or 210) to generate first three-dimensional coordinates of the pixels in the first depth map according to internal parameters of the first depth capture device 110 (or 210)”, Paragraph [0048]); and re-project the 3D representation of the second depth data into the target FOV (i.e., “Transform the first three-dimensional coordinates of the pixels in the first depth map D1 to second three-dimensional coordinates in a second three-dimensional coordinate system of the second depth capture device 120 (or 220)”, Paragraph [0049]).
Kamal et al. and Sun et al. are analogous art because they are from the field of digital image processing for processing depth maps.
Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to modify Kamal et al. by incorporating the generating the FOV adjusted depth data based on the second depth data associated with the second FOV, the at least one processor is configured to: project the second depth data from the second FOV into a three-dimensional (3D) representation of second depth data, and re-project the 3D representation of the second depth data into the target FOV, as taught by Sun et al.
The suggestion/motivation for doing so would have been to provide a more accurate depth map, and to provide a wider range of depth data.
Therefore, it would have been obvious to combine Sun et al. with Kamal et al. to obtain the invention as specified in claim 5.
As to claim 8, Kamal et al. does not explicitly disclose wherein, to filter the second depth data, the at least one processor is configured to: determine whether at least one of a density or a quality of the at least one depth value satisfies a filtering condition; and based on a determination that the density of the quality of the at least one depth value satisfies the filtering condition, include the at least one depth value in the FOV adjusted depth data.
Sun et al. teaches wherein, to filter the second depth data (See for example, “suppose the first pixel in the first depth map corresponds to the second pixel in the second depth map, and the second pixel has been given an unreasonable depth value, the depth processing unit would render a depth value to the third pixel corresponding to the first pixel and the second pixel in the optimized depth map according to the depth value of the first pixel”, Paragraph [0053]), the at least one processor is configured to (i.e., “depth processing unit 130”, Paragraph [0016]): determine whether at least one of a density or a quality of the at least one depth value satisfies a filtering condition (i.e., “confidence”, Paragraph [0032]); and based on a determination that the density of the quality of the at least one depth value satisfies the filtering condition, include the at least one depth value in the FOV adjusted depth data (i.e., “the optimized depth map DA may be based on the first depth map D1”, Paragraph [0032]).
Therefore, in view of Sun et al., it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Kamal et al. by to filter the second depth data, the at least one processor is configured to: determine whether at least one of a density or a quality of the at least one depth value satisfies a filtering condition, and based on a determination that the density of the quality of the at least one depth value satisfies the filtering condition, include the at least one depth value in the FOV adjusted depth data, as taught by Sun et al., in order to provide a more accurate depth map.
As to claim 9, Sun et al. teaches wherein the filtering condition is associated with the second depth data source and an additional filtering condition is associated with the first depth data source, the additional filtering condition being different from the filtering condition (See for example, Paragraph [0031]; and “the confidence of the first depth map D1 is likely to be higher than the confidence of the second depth map D2”, Paragraph [0032]).
As to claim 10, Sun et al. teaches wherein the filtering condition comprises a confidence mask associated with the second depth data source (See for example, “the pixel PD2A has not been given a depth value, then the depth processing unit 130 would render a depth value to the corresponding pixel PDAA in the optimized depth map DA according to the depth value of the pixel PD1A”, Paragraph [0029]).
Claims 14 and 15 are rejected under 35 U.S.C. 103 as being unpatentable over Kamal et al. in view of Shalumov et al. (U.S. Pub. No. 2022/0292289). The teachings of Kamal et al. have been discussed above.
As to claim 14, Kamal et al. does not explicitly disclose wherein the target FOV is associated with a machine learning model configured to generate one or more depth maps.
Shalumov et al. teaches a target FOV that is associated with a machine learning model configured to generate one or more depth maps (i.e., “the panorama image is generated by transforming the image from each camera into three dimensional point cloud in a local coordinate frame of the camera using the depth data 20 output from the neural network 18”, Paragraph [0038]).
Kamal et al. and Shalumov et al. are analogous art because they are from the filed of digital image processing for processing depth data.
Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to modify Kamal et al. by incorporating the target FOV is associated with a machine learning model configured to generate one or more depth maps, as taught by Shalumov et al.
The suggestion/motivation for doing so would have been to provide a more accurate depth data estimation.
Therefore, it would have been obvious to combine Shalumov et al. with Kamal et al. to obtain the invention as specified in claim 14.
As to claim 15, Kamal et al. does not explicitly disclose wherein the at least one processor is configured to determine the depth map using the machine learning model.
Shalumov et al. teaches at least one processor (i.e., “depth estimation system 30”, Paragraph [0019]) that is configured to determine the depth map using the machine learning model (i.e., “depth data 20 output from the neural network 18”, Paragraph [0038]).
Therefore, in view of Shalumov et al., it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Kamal et al. by incorporating the at least one processor is configured to determine the depth map using the machine learning model, as taught by Shalumov et al., in order to provide a more accurate depth data estimation.
Claim 16 is rejected under 35 U.S.C. 103 as being unpatentable over Kamal et al. in view of Uddin et al. (“Unsupervised Deep Event Stereo for Depth Estimation”, IEEE Transactions on Circuits and Systems for Video Technology, Vol. 32, No. 11, November 2022, pp. 7489-7504). The teachings of Kamal et al. have been discussed above.
As to claim 16, Kamal et al. does not explicitly disclose wherein the first depth data associated with the first FOV and the second depth data associated with the second FOV are obtained asynchronously.
Uddin et al. teaches first depth data associated with first FOV and second depth data associated with second FOV that are obtained asynchronously (See for example, Fig. 2, Stereo Events (From Left and Right Cameras), “The proposed method takes asynchronous and sparse stereo event data as inputs and embeds them into event features using the event embedding sub-network”, page 7492).
Kamal et al. and Uddin et al. are analogous art because they are from the field of digital image processing for processing depth data.
Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to modify Kamal et al. by incorporating the first depth data associated with the first FOV and the second depth data associated with the second FOV are obtained asynchronously, as taught by Uddin et al.
The suggestion/motivation for doing so would have been to produce better depth estimation when visualizing events in low light scenes.
Therefore, it would have been obvious to combine Uddin et al. with Kamal et al. to obtain the invention as specified in claim 16.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JOSE M TORRES whose telephone number is (571)270-1356. The examiner can normally be reached Monday thru Friday; 10:00 AM to 6:00 PM EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Jennifer Mehmood can be reached at 571-272-2976. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JOSE M TORRES/Examiner, Art Unit 2664 03/31/2026
Prosecution Insights