DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
This Office Action is in response to the Applicants’ Amendment filed on July 08, 2026. Claims 1-20 are currently presented in the instant application.
Response to Arguments
Applicant’s arguments with respect to claims 1-20 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument. Claims 1- 2, 8, 15-16 and 20 have been amended. Therefore, the 35 USC 103 have been withdrawn.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 15 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Steinbrücker et al. (US 20230147759 A1) in view of VAREKAMP et al. (US 20190385352 A1) and further in view of Du et al. (US 20190213778 A1).
Regarding claim 1, Steinbrücker discloses a method (Steinbrücker, see FIG. 12 and at least par. [0188]) comprising:
at a device including one or more processors and non-transitory memory (Steinbrücker, see par. [0139]);
storing, in a first buffer, a first mesh of a physical environment and, in a second buffer, a second mesh of the physical environment in a second mesh buffer (Steinbrücker, see at least par. [0133] The reconstruction 318 may include representations of the physical world in one or more data formats including, for example, voxels, meshes, planes, etc. The different formats may represent alternative representations of the same portions of the physical world or may represent different portions of the physical world. In the illustrated example, on the left side of the reconstruction 318, portions of the physical world are presented as a global surface; on the right side of the reconstruction 318, portions of the physical world are presented as meshes.) associated with a timestamp (Steinbrücker, see at par. [0147] FIG. 5B illustrates a schematic of a viewing optics assembly 48 and attendant components. Oriented to user eyes 49, in some embodiments, two eye tracking cameras 50 detect metrics of the user eyes 49 such as eye shape, eyelid occlusion, pupil direction and glint on the user eyes 49. In some embodiments, a depth sensor 51, such as a time of flight sensor, emits relay signals to the world to determine distance to given objects. In some embodiments, world cameras 52 record a greater-than-peripheral view to map the environment 32 and detect inputs that may affect AR content. Camera 53 may further capture a specific timestamp of physical world images within a field of view of the user. Each of the world cameras 52, the camera 53 and the depth sensor 51 have respective fields of view of 54, 55, and 56 to collect data from and record a physical world scene, such as physical world environment 32 depicted in FIG. 3A.);
rendering a first depth map based on the first mesh and camera extrinsics of an image of the physical environment and a second depth map based on the second mesh and the camera extrinsics of the image of the physical environment (Steinbrücker, see at least par. [0147] FIG. 5B illustrates a schematic of a viewing optics assembly 48 and attendant components. Oriented to user eyes 49, in some embodiments, two eye tracking cameras 50 detect metrics of the user eyes 49 such as eye shape, eyelid occlusion, pupil direction and glint on the user eyes 49. In some embodiments, a depth sensor 51, such as a time of flight sensor, emits relay signals to the world to determine distance to given objects. In some embodiments, world cameras 52 record a greater-than-peripheral view to map the environment 32 and detect inputs that may affect AR content. Camera 53 may further capture a specific timestamp of physical world images within a field of view of the user. Each of the world cameras 52, the camera 53 and the depth sensor 51 have respective fields of view of 54, 55, and 56 to collect data from and record a physical world scene, such as physical world environment 32 depicted in FIG. 3A.);
Steinbrücker does not disclose rendering a first depth map based on the first mesh and camera extrinsics of an image of the physical environment and a second depth map based on the second mesh and the camera extrinsics of the image of the physical environment; and generating a blended depth map based on the first depth map, the second depth map, and a difference between a time of the image of the physical environment and the time stamp. However,
VAREKAMP discloses:
rendering a first depth map based on the first mesh and camera extrinsics of an image of the physical environment (VAREKAMP, see at least par. [0082], [0082] In many embodiments the first texture map and the first mesh may be generated from a capture of a real life scene. The capture may be by a suitable set of cameras. For example, a single stereo camera or range camera may capture a real life scene and generate an image and a depth (/disparity) map. In other embodiments, a plurality of cameras at different positions may capture a two-dimensional image and a depth map may be generated from the captured images, e.g. by disparity estimation.) and a second depth map based on the second mesh and the camera extrinsics of the image of the physical environment (VAREKAMP, see at least par. [0105], Equivalently, the weighting of the first light intensity value from the first view transformer 207 may be a monotonically increasing function of a magnitude of the gradient in the second mesh. The same may symmetrically be applicable to the weighting of the second light intensity value from the second view transformer 209. The weight of each texture may specifically be inversely proportional to the local gradient magnitude in the depth/disparity map or mesh that is associated with each texture.);
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to modify the system and method of Steinbrücker, with rendering a first depth map based on the first mesh and camera extrinsics of an image of the physical environment and a second depth map based on the second mesh and the camera extrinsics of the image of the physical environment, as provided by VAREKAMP. The modification provides an improved system and method for generating a depth map and improved techniques and systems for calculating and processing depth information, particularly for systems that need to resolve parallax problem, thereby to allows increased flexibility, increased accuracy, reduced complexity, improved computational efficiency, improved compatibility with existing graphic processing approaches, improved image quality, improved de-occlusion performance, and/or improved performance would be advantageous. (VAREKAMP, see par. [0016]).
Steinbrücker in view of VAREKAMP does not disclose generating a blended depth map based on the first depth map, the second depth map, and a difference between a time of the image of the physical environment and the time stamp. However,
Du discloses:
generating a blended depth map based on the first depth map, the second depth map, and a difference between a time of the image of the physical environment and the time stamp (Du, see at least par. [0060] In the example environment (201) of FIG. 2a, the depth maps produced by the depth generators (270-273) are provided (e.g., over a network) to a fusion component (280). The fusion component (280) can be implemented as software and/or hardware. In general, the fusion component (280) performs some or all of the following operations: receiving depth maps from depth generators; receiving texture maps from the depth generators (e.g., passed through the depth generators), camera pods, or another component; generating a dynamic 3D model; and transmitting model data for the dynamic 3D model and texture maps to rendering components. Specifically, for a given frame (e.g., associated with a timestamp or time slice), the fusion component (280) combines depth maps from different input viewpoints for the given frame to generate a 3D model for the given frame, which may include updating a reference 3D model (which is based on 3D model data for one or more previous frames) and estimating a current 3D model (based on the depth maps for the given frame). For example, the 3D model is a mesh of vertices for triangles or other volumetric representation. In some example implementations, the fusion component (280) runs as software on a separate computer system or separate virtual machine. Model data for the dynamic 3D model generated by the fusion component is transmitted (e.g., over a network) to one or more rendering components (290), which can also be called renderers or rendering engines.).
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to modify the system and method of Steinbrücker, with generating a blended depth map based on the first depth map, the second depth map, and a difference between a time of the image of the physical environment and the time stamp, as provided by Du. The modification provides an improved system and method for generating a depth map and greatly enhances the user's enjoyment in using the XR system and in order to improve the quality of results when fusing depth maps to generate dynamic three-dimensional (“3D”) models of a computer-represented environment, when applying texture details to dynamic 3D models, or when rendering views of textured, dynamic 3D models. (Du, see par. [0005]).
Regarding claim 15. Steinbrücker in view of VAREKAMP in view of Du discloses a device (Steinbrücker, see FIGs 1-2 and par. [0122] ) to perform same method of claim 1. Therefore, claim 15 is further rejected based on the same rationale as claim 1 set forth above and incorporated herein.
Regarding claim 20. Steinbrücker in view of VAREKAMP in view of Du discloses a non-transitory memory storing one or more programs, which, when executed by one or more processors of a device, cause the device (Steinbrücker, see par.[0021]) to perform same method of claim 1. Therefore, claim 20 is further rejected based on the same rationale as claim 1 set forth above and incorporated herein.
Claims 3-7, 9, 13, 17 are rejected under 35 U.S.C. 103 as being unpatentable over Steinbrücker et al. (US 20230147759 A1) in view of VAREKAMP et al. (US 20190385352 A1) and further in view of Du et al. (US 20190213778 A1), as applied claim 1 above and further in view of Bleyer et al. (US 20210358084 A1).
Regarding claim 3. Steinbrücker in view of VAREKAMP in view of Du discloses the method of claim 1 (as rejected above), and Steinbrücker in view of VAREKAMP in view of Du does not disclose further comprising, in response to detecting an update trigger, overwriting the first mesh buffer with the second mesh and overwriting the second mesh buffer with a third mesh stored in a third mesh buffer. However,
Bleyer discloses:
further comprising, in response to detecting an update trigger, overwriting the first mesh buffer with the second mesh and overwriting the second mesh buffer with a third mesh stored in a third mesh buffer (Bleyer, see par. [0110-0112]).
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to modify the system and method of Steinbrücker, with further comprising, in response to detecting an update trigger, overwriting the first mesh buffer with the second mesh and overwriting the second mesh buffer with a third mesh stored in a third mesh buffer, as provided by Bleyer. The modification provides an improved system and method for generating a depth map and improved techniques and systems for calculating and processing depth information, particularly for systems that need to resolve parallax problems. (Bleyer, see par. [0012).
Regarding claim 4. Steinbrücker in view of VAREKAMP, further in view of Du and further in view of Bleyer discloses the method of claim 3 (as rejected above), and Steinbrücker in view of VAREKAMP, further in view of Du and further in view of Bleyer further discloses wherein the update trigger includes a temporal trigger detected when a threshold amount of time has elapsed since an initialization or a most recent update trigger (Bleyer, see pars. [0082, 0110-0112], [0082] However, generating a depth map from the stereo pair of images 410A for performing parallax correction is computationally expensive, particularly for high-resolution stereo pairs of images (which may be desirable for providing parallax-corrected pass-through views of an environment). For example, FIG. 4A illustrates the HMD 400 capturing the stereo pair of images 410A at a first timepoint (e.g., indicated according to time axis, t). In some instances, because of the computational expense associated with performing the depth calculations 420A to generate a depth map, the depth map 415A is not completed until a period of time has elapsed (e.g., several frames) since the first timepoint at which the HMD 400 captured the stereo pair of images 410A. The temporal delay associated with conventional depth calculations (e.g., depth calculations 420A) may cause high latency in operations that depend on updated depth information (e.g., providing parallax-corrected pass-through views of an environment).).
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to modify the system and method of Steinbrücker, with wherein the update trigger includes a temporal trigger detected when a threshold amount of time has elapsed since an initialization or a most recent update trigger, as provided by Bleyer. The modification provides an improved system and method for generating a depth map and improved techniques and systems for calculating and processing depth information, particularly for systems that need to resolve parallax problems. (Bleyer, see par. [0012).
Regarding claim 5. Steinbrücker in view of VAREKAMP, further in view of Du and further in view of Bleyer discloses the method of claim 3 (as rejected above), and Steinbrücker in view of VAREKAMP, further in view of Du and further in view of Bleyer further discloses wherein the update trigger includes a motion-based trigger detected based on motion of the device (Bleyer, see at least par. [0123] Additionally, or alternatively, the HMD 800 may detect changes in the position of the HMD 800 relative to an environment based on changes in the depth maps over time. A difference in the depth information of two sequentially generated depth maps may also indicate that the perspective of the HMD 800 is changing with respect to at least a portion of the real-world environment. For instance, a difference in depth information between consecutively generated depth maps may indicate that objects in the environment are moving with respect to the HMD 800, which may trigger an increase in the frequency at which the HMD 800 generates depth maps (e.g., to more frequently update depth information for the moving objects in the environment).).
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to modify the system and method of Steinbrücker, with wherein the update trigger includes a motion-based trigger detected based on motion of the device, as provided by Bleyer. The modification provides an improved system and method for generating a depth map and improved techniques and systems for calculating and processing depth information, particularly for systems that need to resolve parallax problems. (Bleyer, see par. [0012).
Regarding claim 6. Steinbrücker in view of VAREKAMP, further in view of Du and further in view of Bleyer discloses the method of claim 3 (as rejected above), and Steinbrücker in view of VAREKAMP, further in view of Du and further in view of Bleyer further discloses wherein the update trigger includes an eye-based trigger detected based on an eye characteristic of a user (Bleyer, see at least par. [0075] By performing these different transforms, the HMD 300 is able to perform three-dimensional (3D) geometric transforms on the raw camera images to transform the perspectives of the raw images in a manner so as to correlate with the perspectives of the user's pupils 330 and 335. Additionally, the 3D geometric transforms rely on depth computations in which the objects in the HMD 300's environment are mapped out to determine their depths as well as the pose 375).
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to modify the system and method of Steinbrücker, with wherein the update trigger includes an eye-based trigger detected based on an eye characteristic of a user, as provided by Bleyer. The modification provides an improved system and method for generating a depth map and improved techniques and systems for calculating and processing depth information, particularly for systems that need to resolve parallax problems. (Bleyer, see par. [0012).
Regarding claim 7. Steinbrücker in view of VAREKAMP, further in view of Du and further in view of Bleyer discloses the method of claim 3 (as rejected above), and Steinbrücker in view of VAREKAMP, further in view of Du and further in view of Bleyer further discloses wherein the update trigger includes a content-based trigger detected based on virtual content displayed by the device (Bleyer, see at least par. [0123] Additionally, or alternatively, the HMD 800 may detect changes in the position of the HMD 800 relative to an environment based on changes in the depth maps over time. A difference in the depth information of two sequentially generated depth maps may also indicate that the perspective of the HMD 800 is changing with respect to at least a portion of the real-world environment. For instance, a difference in depth information between consecutively generated depth maps may indicate that objects in the environment are moving with respect to the HMD 800, which may trigger an increase in the frequency at which the HMD 800 generates depth maps (e.g., to more frequently update depth information for the moving objects in the environment).).
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to modify the system and method of Steinbrücker, with wherein the update trigger includes a content-based trigger detected based on virtual content displayed by the device, as provided by Bleyer. The modification provides an improved system and method for generating a depth map and improved techniques and systems for calculating and processing depth information, particularly for systems that need to resolve parallax problems. (Bleyer, see par. [0012]).
Regarding claim 9. Steinbrücker in view of VAREKAMP, further in view of Du and further in view of Bleyer discloses the method of claim 3 (as rejected above), and Steinbrücker in view of VAREKAMP, further in view of Du and further in view of Bleyer further discloses further comprising updating the third mesh buffer in response to receiving mesh data (Bleyer, see at least par. [0039] High frequency depth map generation (e.g., providing a depth map for each captured stereo pair of images) may improve user experiences that depend on near-real-time depth calculations, such as, for example, providing parallax-corrected pass-through images of a user's environment, hand or other object tracking, surface reconstruction mesh building or updating, streaming stereoscopic video, and/or others at a higher frame rate or with lower latency than would otherwise be possible using traditional techniques.).
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to modify the system and method of Steinbrücker, with further comprising updating the third mesh buffer in response to receiving mesh data, as provided by Bleyer. The modification provides an improved system and method for generating a depth map and improved techniques and systems for calculating and processing depth information, particularly for systems that need to resolve parallax problems. (Bleyer, see par. [0012]).
Regarding claim 13. Steinbrücker in view of VAREKAMP, further in view of Du and further in view of Bleyer discloses the method of claim 3 (as rejected above), and Steinbrücker in view of VAREKAMP, further in view of Du and further in view of Bleyer further discloses further comprising transforming the image of the environment based on the blended depth map and a difference between a first perspective of the image of the environment and a second perspective (Bleyer, see pars. [0122], A change in the pose of the HMD 800 may cause portions of real-world objects that were not previously visible to the user to become visible according to the user's new perspective. Therefore, in some implementations, the HMD 800 increases the frequency with which the HMD 800 generates depth maps in response to detecting a change in the pose of the HMD 800 (e.g., in order to more accurately capture the changes in the user's perspective/perception of the real-world environment). [0123] Additionally, or alternatively, the HMD 800 may detect changes in the position of the HMD 800 relative to an environment based on changes in the depth maps over time. A difference in the depth information of two sequentially generated depth maps may also indicate that the perspective of the HMD 800 is changing with respect to at least a portion of the real-world environment. For instance, a difference in depth information between consecutively generated depth maps may indicate that objects in the environment are moving with respect to the HMD 800, which may trigger an increase in the frequency at which the HMD 800 generates depth maps (e.g., to more frequently update depth information for the moving objects in the environment).
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to modify the system and method of Steinbrücker, with further comprising transforming the image of the environment based on the blended depth map and a difference between a first perspective of the image of the environment and a second perspective, as provided by Bleyer. The modification provides an improved system and method for generating a depth map and improved techniques and systems for calculating and processing depth information, particularly for systems that need to resolve parallax problems. (Bleyer, see par. [0012]).
Regarding claim 17, the device of claim 17 performs same method of claim 3. Therefore, claim 17 is further rejected based on the same rationale as claim 3 set forth above and incorporated herein.
Claim 8 is rejected under 35 U.S.C. 103 as being unpatentable over Steinbrücker et al. (US 20230147759 A1) in view of VAREKAMP et al. (US 20190385352 A1) and further in view of Du et al. (US 20190213778 A1) and further in view of Bleyer et al. (US 20210358084 A1), as applied claim 3 above, and further in view of Moore et al. (US 20170352244 A1).
Regarding claim 8. Steinbrücker in view of VAREKAMP, further in view of Du and further in view of Bleyer discloses the method of claim 3 (as rejected above), but Steinbrücker in view of VAREKAMP, further in view of Du and further in view of Bleyer does not disclose in response to determining that the third mesh has changed more than a threshold amount after the update trigger, suppressing a subsequent update trigger for a particular amount of time.
Moore discloses:
in response to determining that the third mesh is has changed more than a threshold amount after the update trigger, suppressing a subsequent update trigger for a particular amount of time (Moore, see pars. [0093-0098]).
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to modify the system and method of Steinbrücker, with in response to determining that the third mesh is has changed more than a threshold amount after the update trigger, suppressing a subsequent update trigger for a particular amount of time, as provided by Moore. The modification provides an improved system and method for generating a blended depth map based on the first depth map, the second depth map, and a difference between a time of the image of the physical environment and a time of the second mesh, thereby to remedy such an occurrence and possibly help prevent any loss of confidence in the hazard detector (and/or home automation system), the hazard detector and remote server system may be configured to provide notifications indicative of the hazard in certain situations even if the hazardous condition is no longer detected by the hazard detector (Moore, see par. [0022]).
Claim 10 is rejected under 35 U.S.C. 103 as being unpatentable over Steinbrücker et al. (US 20230147759 A1) in view of VAREKAMP et al. (US 20190385352 A1) and further in view of Du et al. (US 20190213778 A1) and further in view of Bleyer et al. (US 20210358084 A1), as applied claim 3 above, and further in view of Tsukizaki et al. (US 20020075262 A1).
Regarding claim 10. Steinbrücker in view of VAREKAMP, further in view of Du and further in view of Bleyer discloses the method of claim 3 (as rejected above), but Steinbrücker in view of VAREKAMP, further in view of Du and further in view of Bleyer does not discloses further comprising, in response to detecting a subsequent update trigger, overwriting the first mesh buffer with the third mesh and overwriting the second mesh buffer with a fourth mesh stored in the third mesh buffer. However,
Tsukizaki discloses:
further comprising, in response to detecting a subsequent update trigger, overwriting the first mesh buffer with the third mesh and overwriting the second mesh buffer with a fourth mesh stored in the third mesh buffer (Tsukizaki, see at least par. [0077] As described above, according to the first and second embodiments of the present invention, the frame F is divided into the mesh M, and it is determined whether or not overwrite rendering is possible for each mesh M in a predetermined range E where a predetermined object A should be rendered. More specifically, the Z value of each mesh Mc is subjected to clearing processing on every other dot (every other mesh), or other objects are inhibited from being rendered. By doing so, it is possible to visibly display an object hidden by a shade of another object without carrying out a determination process relative to a positional relationship between a virtual viewpoint and object and a semi-transparency processing for making object semitransparent at high speed and low cost.).
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to modify the system and method of Steinbrücker, with further comprising, in response to detecting a subsequent update trigger, overwriting the first mesh buffer with the third mesh and overwriting the second mesh buffer with a fourth mesh stored in the third mesh buffer, as provided by Tsukizaki,. The modification provides an improved system and method for generating a depth map and improved techniques and systems for perform a display process at high speed and low cost, and to a computer-readable recording medium on which a rendering program executed by a computer is recorded, a program processor for executing a rendering program, and a rendering program executed by a computer. (Tsukizaki, see par. [0014]).
Claims 11-12, 14 and 18-19 are rejected under 35 U.S.C. 103 as being unpatentable over Steinbrücker et al. (US 20230147759 A1) in view of VAREKAMP et al. (US 20190385352 A1) and further in view of Du et al. (US 20190213778 A1) and further in view of Bleyer et al. (US 20210358084 A1), as applied claims 1 and 15 above, and further in view of Cole et al. (US 20160255327 A1).
Regarding claim 11. Steinbrücker in view of VAREKAMP, further in view of Du and further in view of Bleyer discloses, but Steinbrücker in view of VAREKAMP, further in view of Du and further in view of Bleyer does not disclose further comprising, in response to determining that a portion of the first mesh was not used in rendering the first depth map, updating the portion of the first mesh with a corresponding portion of the second mesh. However,
Cole discloses:
further comprising, in response to determining that a portion of the first mesh was not used in rendering the first depth map, updating the portion of the first mesh with a corresponding portion of the second mesh (Cole, see at least [0098], generating depth map of the environment of interest, generating 3D mesh models and UV maps, processing image content received from one or more camera devices positioned at one or more location in the environment, e.g., encoding image in one or more different formats, extract occluded image data in accordance with the features of the present invention, and communicating the image content as well as environmental model information and UV maps to one or more playback devices in accordance with the features of the invention. In some embodiments the processing system 608 may include a server with the server responding to requests for content and/o environmental information for use in rendering content, e.g., depth maps corresponding to environment of interest, and/or 3D environmental mesh models, UV maps and/or imaging content.).
Regarding claim 12. Steinbrücker in view of VAREKAMP, further in view of Du and further in view of Cole discloses (as rejected claim 1), but Steinbrücker in view of VAREKAMP, further in view of Du and further in view of Cole further discloses further comprising, in response to determining that a portion of the first mesh was not used in rendering the first depth map and a portion of the second mesh was not used in rendering the second depth map, updating the portion of the first mesh and the portion of the second mesh with corresponding portions of a third mesh (Cole, see pars. [0092] While the depth map generated from each image corresponds to a portion of the environment to be mapped, in some embodiments the depth maps generated from individual images are processed, e.g., stitched together, to form a composite map of the complete environment scanned using the light field cameras. Thus by using the light field cameras a relatively complete environmental map can be, and in some embodiments is generated. [0093] In the case of light field cameras, an array of micro-lenses captures enough information that one can refocus images after acquisition. It is also possible to shift, after image capture, one's viewpoint within the sub-apertures of the main lens, effectively obtaining multiple views. In the case of a light field camera, depth cues from both defocus and correspondence are available simultaneously in a single capture. This can be useful when attempting to fill in occluded information/scene portions not captured by the stereoscopic cameras.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to modify the system and method of Burn, with generating a blended depth map based on the first depth map, the second depth map, and further comprising, in response to determining that a portion of the first mesh was not used in rendering the first depth map and a portion of the second mesh was not used in rendering the second depth map, updating the portion of the first mesh and the portion of the second mesh with corresponding portions of a third mesh, as provided by Cole. The modification provides an improved system and method for generating a depth map and improved techniques and systems for calculating and processing depth information, particularly for systems that need to allow a playback device to receive and/or use images of non-occluded portions of an environment along with at least some image content corresponding to occluded portions of the environment. (Cole, see par. [0008]).
Regarding claim 14. Steinbrücker in view of VAREKAMP, further in view of Du and further in view of Cole discloses (as rejected claim 1 above), and Steinbrücker in view of VAREKAMP, further in view of Du and further in view of Cole does not disclose further comprising: generating virtual content based on the blended depth map; compositing the virtual content with the image of the physical environment to generate a display image; and displaying the display image. However,
Cole discloses:
further comprising:
generating virtual content based on the blended depth map (Cole, see at least par. [0087] In some embodiments the camera rig 801 may be mounted on a support structure such that it can be rotated around a vertical axis. In various embodiments the camera rig 801 may be deployed in an environment of interest, e.g., such as a stadium, auditorium, or another place where an event to be captured is taking place. In some embodiments the light field cameras of the camera rig 801 are used to capture images of the environment of interest, e.g., a 360 degree scene area of interest, and generate depth maps which can be used in simulating a 3D environment and displaying stereoscopic imaging content.);
compositing the virtual content with the image of the physical environment to generate a display image (Cole, see at least the second part of par. [0089], Such depth maps and/or composite depth map may, and in some embodiments are, provided to a playback device for use in displaying stereoscopic imaging content and simulating a 3D environment which can be experienced by the viewers.); and
displaying the display image (Cole, see par. [0093]).
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to modify the system and method of Steinbrücker, with generating a blended depth map based on the first depth map, the second depth map, and further comprising: generating virtual content based on the blended depth map; compositing the virtual content with the image of the physical environment to generate a display image; and displaying the display image, as provided by Cole. The modification provides an improved system and method for generating a depth map and improved techniques and systems for calculating and processing depth information, particularly for systems that need to allow a playback device to receive and/or use images of non-occluded portions of an environment along with at least some image content corresponding to occluded portions of the environment. (Cole, see par. [0008]).
Regarding claim 18, the device of claim 18 performs same method of claim 11. Therefore, claim 18 is further rejected based on the same rationale as claim 11 set forth above and incorporated herein.
Regarding claim 19, the device of claim 19 performs same method of claim 12. Therefore, claim 19 is further rejected based on the same rationale as claim 12 set forth above and incorporated herein.
Allowable Subject Matter
Claims 2 and 16 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Steinbrücker in view VAREKAMP and further in view of Du discloses claims 1 and 15 (as rejected above). However, the limitations:
wherein generating the blended depth map includes generating a weighted sum of an inverse of the first depth map and an inverse of the second depth map, wherein a weighting factor of the weighted sum applied to the inverse of the second depth map is proportional to the difference between a time of the image of the physical environment and the timestamp, taken as a whole render the claim patentably distinct over prior arts.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to KIM THANH THI TRAN whose telephone number is (571)270-1408. The examiner can normally be reached Monday-Friday 8:00am-5:00pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, ALICIA HARRINGTON can be reached at 5712722330. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/KIM THANH T TRAN/Examiner, Art Unit 2615
/JAMES A THOMPSON/Primary Examiner, Art Unit 2615
