Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claim 5 is rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 5 recites the limitation "wherein the determining the first depth is based at least in part on the seed point.” There is insufficient antecedent basis for this limitation in the claim, as no reference to a seed point is previously claimed.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 1-3, 5-7, 9 is/are rejected under 35 U.S.C. 103 as being unpatentable over Kutliroff et al. (US 20180018805, hereinafter “Kut”) in view of Jiang et al. (US 20150193935).
Re claim 1, Kut teaches a device comprising:
a display; one or more image components for capturing image data associated with a physical environment surrounding a user; one or more processors; and non-transitory computer-readable media storing computer-executable instructions, which when executed by the one or more processors cause the one or more processors to perform operations comprising (see [0024], display element and depth camera).
receiving plurality of frames associated with the physical environment from the one or more image components (see [0024], capture 3D image frames of the scene from multiple perspectives or camera poses).
performing segmentation on the plurality of frames to determine a first portion of the plurality of the frames ([0037] FIG. 8 is a more detailed block diagram of a 3D segmentation circuit 236, configured in accordance with certain embodiments of the present disclosure. The segmentation circuit is shown to include a plane fitting circuit 802, a floor/wall/ceiling determination circuit 804, a floor/wall/ceiling removal circuit 806, and a connected components clustering circuit 808. The 3D segmentation circuit 236 is configured to segment out each of the objects detected by the object detection circuit 234, by finding the points of the 3D reconstruction that correspond to the contours of the objects. To this end, recall that each object captured by a 3D camera in a given image is associated with a point cloud, and that point cloud is represented by a set of points included in the depth map provided by the 3D camera. As will be appreciated in light of this disclosure, two or more objects detected in a scene, which are grouped together based on a measure of similarity of features, can be fused by combining point clouds associated with each of the detected objects)
determining a normal for the first portion of the plurality of the frames ([0038] The plane fitting circuit 802 is configured to scan the 3D reconstruction of the scene for planar surfaces. One method to accomplish this is to calculate normal vectors to the surfaces by scanning the depth maps and calculating the cross product of differences of neighboring depth pixels. The normal vectors are then clustered into groups based on spatial proximity and the values of the vectors)
determining, based at least in part on the normal, a first depth between a three-dimensional point of the plurality of frames and the first portion ([0027] FIG. 3 illustrates an example color image and depth map, as may be provided by depth camera 110, in accordance with certain embodiments of the present disclosure. An RGB image frame 302 is shown alongside a corresponding depth map frame 304 of a scene of a furnished room provided by the depth camera. The scene, for which a context-based 3D reconstruction is desired, includes a variety of objects such as a table, lamp, sofa, etc.,. The RGB frame 302 captures color data represented by RGB pixels of the image, although other color schemes are possible such as luminance/chrominance (also referred to as YUV). In the depth map frame 304, each pixel represents the distance between that region of the scene and the camera. This can be done through the intensity value of each pixel. In the depth map frame 304, for example, the pixel indicates proximity to the camera. In particular, regions with darker (more intense) pixels 310 are relatively far from the camera, while lighter (less intense) pixels 312 are closer. In addition, the operational parameters of the camera, such as focal length and principal axis, are known or provided so that each depth pixel can be projected into a 3D position in the scene relative to the camera. Furthermore, the RGB and depth components of the camera may be calibrated, both spatially and temporally, to improve the accuracy of the RGB values and the depth values of each individual pixel) and ([0029] As new RGB and depth frames of the scene 120 are captured by depth camera 110 they are provided to the camera pose calculation circuit 402. The camera pose calculation circuit 402 is configured to compute and update the position and orientation (or pose) of the camera. The calculated pose of the camera is the 3D transformation from the position and orientation of the camera in a previous frame, to its position and orientation in the current frame. Three parameters may describe the translation of the camera between consecutive frames (e.g., x, y and z). Three additional parameters may describe the change in orientation (e.g., yaw, pitch and roll angle) for a total of six degrees of freedom (6DOF) that are computed to determine the updated pose of the camera relative to its pose in the previous frame. Determination of the camera pose for each frame can be used to establish a global coordinate system, consistent across all captured frames, in which the 3D points extracted from the depth map can be projected and accumulated by depth pixel accumulation circuit 406. The calculation of the camera pose may be performed in real-time)
and generating a three-dimensional scene reconstruction from the plurality of frames based at least in part on the first depth, the three-dimensional point, and the first portion ([0025] At a top level, the 3D scene reconstruction system 130 may be configured to generate a 3D reconstruction of the scene comprising points in 3D space corresponding to structures or objects within the scene. In accordance with an embodiment, the 3D scene reconstruction system 130 is configured to recognize objects within the scene as belonging to a class of objects, and to further recognize multiple instances of those objects appearing within the scene, thus enabling fusion of the data associated with each instance, to fill gaps and improve reconstruction accuracy, as will be explained in greater detail below. In some embodiments, the resulting 3D reconstructed scene may also be presented to the user on display element 112).
Kut does not explicitly teach wherein determining, based on at least in part on the normal, a first depth between a 3d center point of the plurality of frames and the first portion because Kut does not explicitly teach a 3d center point of the plurality of frames.
However, Jiang teaches wherein determining, based on at least in part on the normal, a first depth between a 3d center point of the plurality of frames and the first portion ([0022] FIG. 5 illustrates a 3D point Q on a plane .pi. and the 2D projection q, q' of the 3D point Q on two respective images I, I' with different views of the plane .pi. to illustrate determining the pose of the master device 110A using the following notation) and (0035] For a 3D point Q on the plane .pi., its 2D projection q and the center of the camera 114, illustrated in FIG. 5 as point O, forms a ray. After intersecting the ray with the plane .pi., the coordinate for the 3D point Q can be determined The plurality of reference points, correspond to a plurality of 3D points, which form a 3D plane. Generally, there are two solutions for homography decomposition. To select the correct solution, the two resulting 3D planes are stored and used to estimate the master device 110A pose in subsequent frames. When the average projection error for one plane is greater than the average projection error for the other plane, e.g., one is 1.2 times greater; the plane having the larger projection error is eliminated. If the initial image is roughly a front view of the planar object 102, the plane normal n may be used to select the correct 3D plane. After the correct 3D plane is selected, a world coordinate frame is defined to align the z-axis with the plane normal n with the origin on the plane. Thus, with the 3D-2D homography H determined, the pose of the master device 110A with respect to the object 102 is determined based on the rotation matrix R and translation vector t as discussed above with reference to equation 6. If desired, other pose determination techniques may be used).
Jiang teaches a first depth between a 3d center point of plurality of frames and a first portion based at least in part on the normal.
Kut and Jiang teaches claim 1. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Kut’s 3d reconstruction system including determining a first depth value between a point of the plurality of frames and a first portion to explicitly include a 3d center point of the plurality of frames, as taught by Jiang, as the references are in the analogous art of 3D reconstruction systems based on a plurality of image frames. An advantage of the modification is that it achieves the result of using the center of the camera as a reference point for ray projection from a known origin for pose estimation of the camera, as taught by Jiang.
Re claim 2, Kut and Jiang teach claim 1. Furthermore, Kut teaches comprising one or more position sensors for capturing orientation and motion data associated with the device and wherein determining the first depth is based at least in part on the orientation and motion data ([0028] FIG. 4 is a more detailed block diagram of a 3D reconstruction circuit 232, configured in accordance with certain embodiments of the present disclosure. The reconstruction circuit is shown to include a camera pose calculation circuit 402, a depth pixel accumulation circuit 406 and, in some embodiments, inertial sensors 404 such as, for example, a gyroscope and/or an accelerometer. An example rendering 500 of a 3D reconstruction of the scene shown in FIG. 3 is illustrated in FIG. 5. This 3D reconstruction is composed of a relatively large number of points in 3D space, corresponding to structures within the scene, and may be represented in one of several ways including, for example, a signed distance function in a volumetric structure, or, equivalently, a polygonal mesh. A 3D reconstruction of a scene generally includes at least two operations: calculation of the camera pose, and accumulation of the depth pixels into a global coordinate system. The 3D reconstruction circuit 232 is configured to perform an initial reconstruction and an enhanced reconstruction. The initial reconstruction is based on the RGB and depth frames received from the camera. A second enhanced reconstruction of the scene is performed after fused object instances are generated and substituted into the scene, as will be described in greater detail below) and ([0029] As new RGB and depth frames of the scene 120 are captured by depth camera 110 they are provided to the camera pose calculation circuit 402. The camera pose calculation circuit 402 is configured to compute and update the position and orientation (or pose) of the camera. The calculated pose of the camera is the 3D transformation from the position and orientation of the camera in a previous frame, to its position and orientation in the current frame. Three parameters may describe the translation of the camera between consecutive frames (e.g., x, y and z). Three additional parameters may describe the change in orientation (e.g., yaw, pitch and roll angle) for a total of six degrees of freedom (6DOF) that are computed to determine the updated pose of the camera relative to its pose in the previous frame. Determination of the camera pose for each frame can be used to establish a global coordinate system, consistent across all captured frames, in which the 3D points extracted from the depth map can be projected and accumulated by depth pixel accumulation circuit 406. The calculation of the camera pose may be performed in real-time).
Re claim 3, Kut and Jiang teaches claim 1. Furthermore, Kut teaches wherein the portion of the plurality of frames include at least one of an object, surface, or plane ([0037] FIG. 8 is a more detailed block diagram of a 3D segmentation circuit 236, configured in accordance with certain embodiments of the present disclosure. The segmentation circuit is shown to include a plane fitting circuit 802, a floor/wall/ceiling determination circuit 804, a floor/wall/ceiling removal circuit 806, and a connected components clustering circuit 808. The 3D segmentation circuit 236 is configured to segment out each of the objects detected by the object detection circuit 234, by finding the points of the 3D reconstruction that correspond to the contours of the objects. To this end, recall that each object captured by a 3D camera in a given image is associated with a point cloud, and that point cloud is represented by a set of points included in the depth map provided by the 3D camera. As will be appreciated in light of this disclosure, two or more objects detected in a scene, which are grouped together based on a measure of similarity of features, can be fused by combining point clouds associated with each of the detected objects)
Re claim 5, Kut and Jiang teach claim 1. Furthermore, Kut teaches receiving point cloud data associated with a simultaneous localization and mapping tracking operation and capturing of the plurality of frames and wherein determining the first depth is based at least in part on the seed point ([0030] In some embodiments, the camera pose may be calculated using an RGB-based Simultaneous Localization and Mapping (SLAM) algorithm which is configured to extract feature descriptors from each RGB frame, match corresponding features across multiple frames and calculate the 6DOF camera pose for each frame through triangulation. Alternatively, data from inertial sensors 404, such as gyroscopes and accelerometers, may be used, either independently, or in combination with the results of the RGB SLAM technique to obtain a more robust estimate of the camera pose) and ([0037] FIG. 8 is a more detailed block diagram of a 3D segmentation circuit 236, configured in accordance with certain embodiments of the present disclosure. The segmentation circuit is shown to include a plane fitting circuit 802, a floor/wall/ceiling determination circuit 804, a floor/wall/ceiling removal circuit 806, and a connected components clustering circuit 808. The 3D segmentation circuit 236 is configured to segment out each of the objects detected by the object detection circuit 234, by finding the points of the 3D reconstruction that correspond to the contours of the objects. To this end, recall that each object captured by a 3D camera in a given image is associated with a point cloud, and that point cloud is represented by a set of points included in the depth map provided by the 3D camera. As will be appreciated in light of this disclosure, two or more objects detected in a scene, which are grouped together based on a measure of similarity of features, can be fused by combining point clouds associated with each of the detected objects), and ([0031] In some embodiments, the calculation of the camera pose may be based on the depth maps. An Iterative Closest Point algorithm (ICP) can be applied to successive depth maps captured by the camera, to align two sets of point clouds and compute the transformation between them. As will be appreciated, point cloud data is effectively provided by the imaging device by virtue of the depth map. Assuming the two point cloud sets from successive frames represent static scenes, the computed transformation describes the movement of the camera that captured the two point cloud sets. By computing the camera transformation describing successive point cloud frames, the global camera pose for each frame can be computed, and the associated point clouds can be accumulated (by depth pixel accumulation circuit 406) into a single data structure representing the 3D reconstruction of the entire scene. By repeating this process for each successive depth frame captured by the camera, each point in the scene is effectively sampled multiple times, and consequently, a weighted average of their 3D positions can be computed, generating a higher quality 3D reconstruction of the scene. Furthermore, the segmentation techniques described below, which rely on depth data, may operate directly on the depth maps provided by the camera or on the 3D reconstruction based on aligning successive depth frames and averaging the 3D positions of the accumulated points. In some embodiments, the camera pose may be estimated using other known techniques in light of the present disclosure), and (see [0028], initial reconstruction based on RGB and depth frames received from the camera as initial seed of a cloud-based reconstruction).
Re claim 6, Kut and Jiang teach claim 1. Furthermore, Jiang teaches wherein the center point of the plurality of frames is determined based at least in part on a capture position associated with individual frames of the plurality of frames (([0022] FIG. 5 illustrates a 3D point Q on a plane .pi. and the 2D projection q, q' of the 3D point Q on two respective images I, I' with different views of the plane .pi. to illustrate determining the pose of the master device 110A using the following notation) and (0035] For a 3D point Q on the plane .pi., its 2D projection q and the center of the camera 114, illustrated in FIG. 5 as point O, forms a ray. After intersecting the ray with the plane .pi., the coordinate for the 3D point Q can be determined The plurality of reference points, correspond to a plurality of 3D points, which form a 3D plane. Generally, there are two solutions for homography decomposition. To select the correct solution, the two resulting 3D planes are stored and used to estimate the master device 110A pose in subsequent frames. When the average projection error for one plane is greater than the average projection error for the other plane, e.g., one is 1.2 times greater; the plane having the larger projection error is eliminated. If the initial image is roughly a front view of the planar object 102, the plane normal n may be used to select the correct 3D plane. After the correct 3D plane is selected, a world coordinate frame is defined to align the z-axis with the plane normal n with the origin on the plane. Thus, with the 3D-2D homography H determined, the pose of the master device 110A with respect to the object 102 is determined based on the rotation matrix R and translation vector t as discussed above with reference to equation 6. If desired, other pose determination techniques may be used). Jiang teaches a first depth between a 3d center point of plurality of frames and a first portion based at least in part on the normal).
Jiang teaches wherein the center point of the plurality of frames is determined based at least in part on a capture position (camera center) associated with individual frames of the plurality of frames (image captured from camera). For motivation, see claim 1.
Re claim 7, Kut and Jiang teach claim 1. Furthermore, Kut further teaches performing the segmentation on the plurality of frames determines a second portion of the plurality of the frames, ([0037] FIG. 8 is a more detailed block diagram of a 3D segmentation circuit 236, configured in accordance with certain embodiments of the present disclosure. The segmentation circuit is shown to include a plane fitting circuit 802, a floor/wall/ceiling determination circuit 804, a floor/wall/ceiling removal circuit 806, and a connected components clustering circuit 808. The 3D segmentation circuit 236 is configured to segment out each of the objects detected by the object detection circuit 234, by finding the points of the 3D reconstruction that correspond to the contours of the objects. To this end, recall that each object captured by a 3D camera in a given image is associated with a point cloud, and that point cloud is represented by a set of points included in the depth map provided by the 3D camera. As will be appreciated in light of this disclosure, two or more objects detected in a scene, which are grouped together based on a measure of similarity of features, can be fused by combining point clouds associated with each of the detected objects).
the operations further comprise:
determining a normal for the second portion of the plurality of the frames; ([0038] The plane fitting circuit 802 is configured to scan the 3D reconstruction of the scene for planar surfaces. One method to accomplish this is to calculate normal vectors to the surfaces by scanning the depth maps and calculating the cross product of differences of neighboring depth pixels. The normal vectors are then clustered into groups based on spatial proximity and the values of the vectors)
determining a second depth, based at least in part on the normal for the second portion, between the three-dimensional point of the plurality of frames and the second portion ([0027] FIG. 3 illustrates an example color image and depth map, as may be provided by depth camera 110, in accordance with certain embodiments of the present disclosure. An RGB image frame 302 is shown alongside a corresponding depth map frame 304 of a scene of a furnished room provided by the depth camera. The scene, for which a context-based 3D reconstruction is desired, includes a variety of objects such as a table, lamp, sofa, etc.,. The RGB frame 302 captures color data represented by RGB pixels of the image, although other color schemes are possible such as luminance/chrominance (also referred to as YUV). In the depth map frame 304, each pixel represents the distance between that region of the scene and the camera. This can be done through the intensity value of each pixel. In the depth map frame 304, for example, the pixel indicates proximity to the camera. In particular, regions with darker (more intense) pixels 310 are relatively far from the camera, while lighter (less intense) pixels 312 are closer. In addition, the operational parameters of the camera, such as focal length and principal axis, are known or provided so that each depth pixel can be projected into a 3D position in the scene relative to the camera. Furthermore, the RGB and depth components of the camera may be calibrated, both spatially and temporally, to improve the accuracy of the RGB values and the depth values of each individual pixel) and ([0029] As new RGB and depth frames of the scene 120 are captured by depth camera 110 they are provided to the camera pose calculation circuit 402. The camera pose calculation circuit 402 is configured to compute and update the position and orientation (or pose) of the camera. The calculated pose of the camera is the 3D transformation from the position and orientation of the camera in a previous frame, to its position and orientation in the current frame. Three parameters may describe the translation of the camera between consecutive frames (e.g., x, y and z). Three additional parameters may describe the change in orientation (e.g., yaw, pitch and roll angle) for a total of six degrees of freedom (6DOF) that are computed to determine the updated pose of the camera relative to its pose in the previous frame. Determination of the camera pose for each frame can be used to establish a global coordinate system, consistent across all captured frames, in which the 3D points extracted from the depth map can be projected and accumulated by depth pixel accumulation circuit 406. The calculation of the camera pose may be performed in real-time)
and generating the three-dimensional scene reconstruction from the plurality of frames based at least in part on the second depth and the second portion ([0025] At a top level, the 3D scene reconstruction system 130 may be configured to generate a 3D reconstruction of the scene comprising points in 3D space corresponding to structures or objects within the scene. In accordance with an embodiment, the 3D scene reconstruction system 130 is configured to recognize objects within the scene as belonging to a class of objects, and to further recognize multiple instances of those objects appearing within the scene, thus enabling fusion of the data associated with each instance, to fill gaps and improve reconstruction accuracy, as will be explained in greater detail below. In some embodiments, the resulting 3D reconstructed scene may also be presented to the user on display element 112).
Furthermore, Jiang teaches wherein the three dimensional point of the plurality of frames is the 3d center point ([0022] FIG. 5 illustrates a 3D point Q on a plane .pi. and the 2D projection q, q' of the 3D point Q on two respective images I, I' with different views of the plane .pi. to illustrate determining the pose of the master device 110A using the following notation) and (0035] For a 3D point Q on the plane .pi., its 2D projection q and the center of the camera 114, illustrated in FIG. 5 as point O, forms a ray. After intersecting the ray with the plane .pi., the coordinate for the 3D point Q can be determined The plurality of reference points, correspond to a plurality of 3D points, which form a 3D plane. Generally, there are two solutions for homography decomposition. To select the correct solution, the two resulting 3D planes are stored and used to estimate the master device 110A pose in subsequent frames. When the average projection error for one plane is greater than the average projection error for the other plane, e.g., one is 1.2 times greater; the plane having the larger projection error is eliminated. If the initial image is roughly a front view of the planar object 102, the plane normal n may be used to select the correct 3D plane. After the correct 3D plane is selected, a world coordinate frame is defined to align the z-axis with the plane normal n with the origin on the plane. Thus, with the 3D-2D homography H determined, the pose of the master device 110A with respect to the object 102 is determined based on the rotation matrix R and translation vector t as discussed above with reference to equation 6. If desired, other pose determination techniques may be used). Jiang teaches a second depth between a 3d center point of plurality of frames and a second portion based at least in part on the normal of a portion plane.
For motivation, see claim 1.
Re claim 9, Kut and Jiang teach claim 7. Furthermore, Kut teaches further comprising determining the 3d center point of the plurality of frames based at least in part on cloud data output by a simultaneous localization and mapping operation associated with the device ([0030] In some embodiments, the camera pose may be calculated using an RGB-based Simultaneous Localization and Mapping (SLAM) algorithm which is configured to extract feature descriptors from each RGB frame, match corresponding features across multiple frames and calculate the 6DOF camera pose for each frame through triangulation. Alternatively, data from inertial sensors 404, such as gyroscopes and accelerometers, may be used, either independently, or in combination with the results of the RGB SLAM technique to obtain a more robust estimate of the camera pose) and ([0037] FIG. 8 is a more detailed block diagram of a 3D segmentation circuit 236, configured in accordance with certain embodiments of the present disclosure. The segmentation circuit is shown to include a plane fitting circuit 802, a floor/wall/ceiling determination circuit 804, a floor/wall/ceiling removal circuit 806, and a connected components clustering circuit 808. The 3D segmentation circuit 236 is configured to segment out each of the objects detected by the object detection circuit 234, by finding the points of the 3D reconstruction that correspond to the contours of the objects. To this end, recall that each object captured by a 3D camera in a given image is associated with a point cloud, and that point cloud is represented by a set of points included in the depth map provided by the 3D camera. As will be appreciated in light of this disclosure, two or more objects detected in a scene, which are grouped together based on a measure of similarity of features, can be fused by combining point clouds associated with each of the detected objects), and ([0031] In some embodiments, the calculation of the camera pose may be based on the depth maps. An Iterative Closest Point algorithm (ICP) can be applied to successive depth maps captured by the camera, to align two sets of point clouds and compute the transformation between them. As will be appreciated, point cloud data is effectively provided by the imaging device by virtue of the depth map. Assuming the two point cloud sets from successive frames represent static scenes, the computed transformation describes the movement of the camera that captured the two point cloud sets. By computing the camera transformation describing successive point cloud frames, the global camera pose for each frame can be computed, and the associated point clouds can be accumulated (by depth pixel accumulation circuit 406) into a single data structure representing the 3D reconstruction of the entire scene. By repeating this process for each successive depth frame captured by the camera, each point in the scene is effectively sampled multiple times, and consequently, a weighted average of their 3D positions can be computed, generating a higher quality 3D reconstruction of the scene. Furthermore, the segmentation techniques described below, which rely on depth data, may operate directly on the depth maps provided by the camera or on the 3D reconstruction based on aligning successive depth frames and averaging the 3D positions of the accumulated points. In some embodiments, the camera pose may be estimated using other known techniques in light of the present disclosure).
Claim(s) 4 is/are rejected under 35 U.S.C. 103 as being unpatentable over Kutliroff et al. (US 20180018805, hereinafter “Kut”) in view of Jiang et al. (US 20150193935) and Kopf et al. (US 20180302612).
Re claim 4, Kut and Jiang teach claim 1. Furthermore, Kut and Jiang do not explicitly teach wherein determining the normal of the portion of the plurality of the frames further comprises:
determining a normal direction for individual pixels of the portion; and estimating the normal based at least in part on the normal directions.
However, Kopf teaches wherein determining the normal of the portion of the plurality of the frames further comprises:
determining a normal direction for individual pixels of the portion; and estimating the normal based at least in part on the normal directions ([0078] An optional step in the three-dimensional reconstruction pipeline is to compute a normal map. In one embodiment, the goal is merely to compute a normal that looks plausible and good enough for simple artist-driven lighting effects, rather than an accurate normal map of the scene. A process for generating the normal map is illustrated in FIG. 5. In this embodiment, the normal map estimation module 128 generates 502 a first (base) normal map from the depth values in the two-layer panorama that is accurate with respect to surface slopes but not textures on each surface. The normal map estimation module 128 also generates 504 a second (details) map from the luminance values that has surface details that are artistically-driven to produce desired effects in response to lighting. The normal map estimation module 128 then transforms 506 the second normal map (with the texture details) onto the first normal map (with accurate orientation) to get a combined normal map that serves are a good approximation for the scene) and (see [0011], generating a normal map for the multi-layered geometric mesh, the normal map estimating for each pixel, an angle normal to a surface depicted by the pixel).
Kut, Jiang, and Kopf teaches claim 4. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Kut and Jiang’s 3d reconstruction system including calculating normal to explicitly teach determining a normal direction for individual pixels of the portion and estimating the normal based at least in part on the normal directions, as taught by Kopf, as the references are in the analogous art of 3d reconstruction systems based on frame image data. An advantage of the modification is that it achieves the result of estimating normal maps based on determining normal directions for each pixel in order to estimate a normal map good enough for the scene, as taught by Kopf.
Claim(s) 8 is/are rejected under 35 U.S.C. 103 as being unpatentable over Kutliroff et al. (US 20180018805, hereinafter “Kut”) in view of Jiang et al. (US 20150193935) and Lucas et al. (US 20190088004).
Re claim 8, Kut and Jiang teach claim 7. Furthermore, Jiang teaches wherein generating the three-dimensional scene reconstruction further comprises:
generating, based at least in part on the first depth, the second depth, and the three-dimensional center point, a ray from three-dimensional center points and intersecting with at least one of the plurality of frames (see [0035], in reference to Fig. 5, wherein for a planar surface with a normal, a ray from a 3d center point of a camera intersects with at least one frame of a plurality of frames)
Kut and Jiang do not explicitly teach a plurality of rays from the 3d center point.
However, Lucas teaches a plurality of rays from a 3d center point (see [0060], in view of Fig. 10, wherein a stereo matching technique is then applied to perform depth estimation for seeds in the initial point cloud. Rays are traced from the camera center of a first view or image, through a seed point on the first image. Ray positions are projected onto the view of another overlapping second camera, or second view or image, to determine a linear range or bracket of potential depth estimates along the ray and within the second view or image. By one form, the process is repeated for all pairs of cameras with overlapping fields of view, although other alternatives could be used as mentioned below. Stereo-matching confidence scores are then used to select the best depth estimate for the seed point being analyzed. By one form, and as mentioned above, this involves an initial selection by using the DAISY score, while the depth estimate is then refined using the CENSUS score. Occluded points are then discarded. This results in the data that could be used to form an initial point cloud, where each included pixel location is provided with 2D pixel coordinates and a depth value. The confidence scores are described in detail below).
Kut, Jiang, and Lucas teaches claim 8. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Kut and Jiang’s 3d reconstruction system including a ray from the 3d center point and intersecting with at least one of the plurality of frames to explicitly include a plurality of rays from the 3d center point, as taught by Lucas, as the references are in the analogous art of 3d reconstruction. An advantage of the modification is that it achieves the result of using ray tracing to project a plurality of rays from a 3d center point into a scene to detect positions where the ray hit objects within the scene, as taught by Lucas.
Claim(s) 10-11, 13-14, 16 is/are rejected under 35 U.S.C. 103 as being unpatentable over Kutliroff et al. (US 20180018805, hereinafter “Kut”) in view of Molyneaux et al. (US 20190197786, hereinafter “Moly”).
Re claim 10, Kut teaches a method comprising:
receiving a frame associated with the physical environment from the one or more image components (see [0024], capture 3D image frames of the scene from multiple perspectives or camera poses).
estimating a first normal for a first pixel of the frame; estimating a second normal for a second pixel of the frame and assigning the first pixel and the second pixel to a common surface ([0038] The plane fitting circuit 802 is configured to scan the 3D reconstruction of the scene for planar surfaces. One method to accomplish this is to calculate normal vectors to the surfaces by scanning the depth maps and calculating the cross product of differences of neighboring depth pixels. The normal vectors are then clustered into groups based on spatial proximity and the values of the vectors).
determining a depth associated with the surface ([0027] FIG. 3 illustrates an example color image and depth map, as may be provided by depth camera 110, in accordance with certain embodiments of the present disclosure. An RGB image frame 302 is shown alongside a corresponding depth map frame 304 of a scene of a furnished room provided by the depth camera. The scene, for which a context-based 3D reconstruction is desired, includes a variety of objects such as a table, lamp, sofa, etc.,. The RGB frame 302 captures color data represented by RGB pixels of the image, although other color schemes are possible such as luminance/chrominance (also referred to as YUV). In the depth map frame 304, each pixel represents the distance between that region of the scene and the camera. This can be done through the intensity value of each pixel. In the depth map frame 304, for example, the pixel indicates proximity to the camera. In particular, regions with darker (more intense) pixels 310 are relatively far from the camera, while lighter (less intense) pixels 312 are closer. In addition, the operational parameters of the camera, such as focal length and principal axis, are known or provided so that each depth pixel can be projected into a 3D position in the scene relative to the camera. Furthermore, the RGB and depth components of the camera may be calibrated, both spatially and temporally, to improve the accuracy of the RGB values and the depth values of each individual pixel) and ([0029] As new RGB and depth frames of the scene 120 are captured by depth camera 110 they are provided to the camera pose calculation circuit 402. The camera pose calculation circuit 402 is configured to compute and update the position and orientation (or pose) of the camera. The calculated pose of the camera is the 3D transformation from the position and orientation of the camera in a previous frame, to its position and orientation in the current frame. Three parameters may describe the translation of the camera between consecutive frames (e.g., x, y and z). Three additional parameters may describe the change in orientation (e.g., yaw, pitch and roll angle) for a total of six degrees of freedom (6DOF) that are computed to determine the updated pose of the camera relative to its pose in the previous frame. Determination of the camera pose for each frame can be used to establish a global coordinate system, consistent across all captured frames, in which the 3D points extracted from the depth map can be projected and accumulated by depth pixel accumulation circuit 406. The calculation of the camera pose may be performed in real-time)
outputting the surface and the depth for use in generating a three-dimensional scene reconstruction of a physical environment ([0025] At a top level, the 3D scene reconstruction system 130 may be configured to generate a 3D reconstruction of the scene comprising points in 3D space corresponding to structures or objects within the scene. In accordance with an embodiment, the 3D scene reconstruction system 130 is configured to recognize objects within the scene as belonging to a class of objects, and to further recognize multiple instances of those objects appearing within the scene, thus enabling fusion of the data associated with each instance, to fill gaps and improve reconstruction accuracy, as will be explained in greater detail below. In some embodiments, the resulting 3D reconstructed scene may also be presented to the user on display element 112).
Kut does not explicitly teach responsive to determining that the first normal is less than or equal to a threshold difference of the second normal, assigning the first pixel and the second pixel to a common surface.
However, Moly teaches responsive to determining that the first normal is less than or equal to a threshold difference of the second normal, assigning the first pixel and the second pixel to a common surface ([0199] FIG. 22 shows portions 1400 of the plane extraction system 1300, illustrating details on plane extraction 1310, according to some embodiments. The plane extraction 1310 may include dividing each of the mesh bricks 1308 into sub-bricks 1402. Plane detection 1404 may be conducted for each of the sub-bricks 1402. For example, plane detection 1404 may compare primitive normals of each mesh triangle in a sub-brick; merge those mesh triangles, with primitive normal differences smaller than a predetermined threshold value, into one mesh triangle; and identify a mesh triangle with an area larger than a predetermined area value as a plane).
Kut and Moly teach claim 10. It would have been obvious to one ordinary skill in the art before the effective filing date of the claimed invention to modify Kut’s 3d reconstruction system including grouping pixels based on normal values to explicitly include determining that a first normal is less than or equal to a threshold difference of the second normal, and assigning the first and second pixels to a common surface, as taught by Moly, as the references are in the analogous art of 3d reconstruction systems. An advantage of the modification is that it achieves the result of using thresholds in order to group pixels based on normal thresholds, so that similar pixels can be grouped as belonging to the same surface, as taught by Moly.
Re claim 11, Kut and Moly teach claim 10. Furthermore, Moly teaches estimating a third normal for a third pixel of the frame, responsive to determining that the third normal is less than or equal to the threshold difference of the second normal or the first normal, assigning the third pixel to the common surface ([0199] FIG. 22 shows portions 1400 of the plane extraction system 1300, illustrating details on plane extraction 1310, according to some embodiments. The plane extraction 1310 may include dividing each of the mesh bricks 1308 into sub-bricks 1402. Plane detection 1404 may be conducted for each of the sub-bricks 1402. For example, plane detection 1404 may compare primitive normals of each mesh triangle in a sub-brick; merge those mesh triangles, with primitive normal differences smaller than a predetermined threshold value, into one mesh triangle; and identify a mesh triangle with an area larger than a predetermined area value as a plane). For motivation, see claim 10.
Re claim 13, Kut and Moly teach claim 10. Furthermore, Kut teaches assigning a class associated with the surface ([0025] At a top level, the 3D scene reconstruction system 130 may be configured to generate a 3D reconstruction of the scene comprising points in 3D space corresponding to structures or objects within the scene. In accordance with an embodiment, the 3D scene reconstruction system 130 is configured to recognize objects within the scene as belonging to a class of objects, and to further recognize multiple instances of those objects appearing within the scene, thus enabling fusion of the data associated with each instance, to fill gaps and improve reconstruction accuracy, as will be explained in greater detail below. In some embodiments, the resulting 3D reconstructed scene may also be presented to the user on display element 112) and ([0037] FIG. 8 is a more detailed block diagram of a 3D segmentation circuit 236, configured in accordance with certain embodiments of the present disclosure. The segmentation circuit is shown to include a plane fitting circuit 802, a floor/wall/ceiling determination circuit 804, a floor/wall/ceiling removal circuit 806, and a connected components clustering circuit 808. The 3D segmentation circuit 236 is configured to segment out each of the objects detected by the object detection circuit 234, by finding the points of the 3D reconstruction that correspond to the contours of the objects. To this end, recall that each object captured by a 3D camera in a given image is associated with a point cloud, and that point cloud is represented by a set of points included in the depth map provided by the 3D camera. As will be appreciated in light of this disclosure, two or more objects detected in a scene, which are grouped together based on a measure of similarity of features, can be fused by combining point clouds associated with each of the detected objects).
Re claim 14, Kut and Moly teach claim 13. Furthermore, Kut teaches wherein the class includes at least one of an object, wall, ceiling, or floor (([0025] At a top level, the 3D scene reconstruction system 130 may be configured to generate a 3D reconstruction of the scene comprising points in 3D space corresponding to structures or objects within the scene. In accordance with an embodiment, the 3D scene reconstruction system 130 is configured to recognize objects within the scene as belonging to a class of objects, and to further recognize multiple instances of those objects appearing within the scene, thus enabling fusion of the data associated with each instance, to fill gaps and improve reconstruction accuracy, as will be explained in greater detail below. In some embodiments, the resulting 3D reconstructed scene may also be presented to the user on display element 112) and ([0037] FIG. 8 is a more detailed block diagram of a 3D segmentation circuit 236, configured in accordance with certain embodiments of the present disclosure. The segmentation circuit is shown to include a plane fitting circuit 802, a floor/wall/ceiling determination circuit 804, a floor/wall/ceiling removal circuit 806, and a connected components clustering circuit 808. The 3D segmentation circuit 236 is configured to segment out each of the objects detected by the object detection circuit 234, by finding the points of the 3D reconstruction that correspond to the contours of the objects. To this end, recall that each object captured by a 3D camera in a given image is associated with a point cloud, and that point cloud is represented by a set of points included in the depth map provided by the 3D camera. As will be appreciated in light of this disclosure, two or more objects detected in a scene, which are grouped together based on a measure of similarity of features, can be fused by combining point clouds associated with each of the detected objects).
Re claim 16, Kut and Moly teach claim 10. Furthermore, Kut teaches planarizing point associated with the surface ([0038] The plane fitting circuit 802 is configured to scan the 3D reconstruction of the scene for planar surfaces. One method to accomplish this is to calculate normal vectors to the surfaces by scanning the depth maps and calculating the cross product of differences of neighboring depth pixels. The normal vectors are then clustered into groups based on spatial proximity and the values of the vectors. Next, a plane is fitted to each cluster).
Claim(s) 12, 15 is/are rejected under 35 U.S.C. 103 as being unpatentable over Kutliroff et al. (US 20180018805, hereinafter “Kut”) in view of Molyneaux et al. (US 20190197786, hereinafter “Moly”), and Kopf et al. (US 20180302612).
Re claim 12, Kut and Moly teach claim 10. Furthermore, Kut and Moly do not explicitly teach wherein determining the depth associated with the surface further comprises selecting a depth that minimizes a photoconsistency error, computed integrally for an overall region of the surface.
However, Kopf teaches wherein determining the depth associated with the surface further comprises selecting a depth that minimizes a photoconsistency error, computed integrally for an overall region of the surface (see [0043], normal map estimation module 128 estimates normal for surfaces) and ([0056] To compute the near envelope n the dense reconstruction module 124 first identifies 302 anchor pixels with reliable depth estimates. The dense reconstruction module 124 computes the anchors from two sources: (1) stereo matching where it is reliable, and (2) the point features already computed in the point cloud during the sparse reconstruction. Stereo matching is known to work well at strong image gradients. This is used by computing a lightweight MVS result and carefully considering the pixels around image edges for anchors. The dense reconstruction module 124 obtains a noisy depth map by independently computing the minimum of E.sub.data for each pixel, e.g., dropping the smoothness term from Eq. 1. The dense reconstruction module 124 uses a geometric consistency filter with aggressive settings to discard a significant portion of the incorrectly estimated depth values. An edge detector is used to compute an edge mask. If the image edge coincides with a depth edge, it is desirable to ensure that the depth belonging to the front layer is selected. This may be achieved by dilating the edge mask by 1 pixel and applying a 5×5 morphological minimum filter to the masked pixels inside the detected edges. In addition to these anchors computed from stereo, the dense reconstruction module 124 also adds all the sparse point features that are observed in the image to the set of anchor pixels).
Kut, Moly, and Kopf teach claim 12. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Kut and Moly’s 3d reconstruction system including calculating normals to explicitly teach selecting a depth that minimizes a photoconsistency error, computed integrally for an overall region of the surface, as taught by Kopf, as the references are in the analogous art of 3d reconstruction systems using calculated surface normal. An advantage of the modification is that it achieves the result of using a filter to discard depth values that are inconsistent integrally over a surface to maintain correct estimated depth values and improve quality by filtering, as taught by Kopf.
Re claim 15, Kut and Moly teach claim 10. Kut and Moly do not explicitly teach denoising points associated with the surface.
However, Kopf teaches denoising points associated with the surface (see [0070] in reference to Fig. 29, denoised mesh by planarizing the mesh) and ([0366] The reconstruction filter 4902 may include computer executable instructions for producing a depth map based on the depth information 4904. The depth map may include one or more pixels. Each pixel may indicate a distance to a point of a surface in a physical world. In some embodiments, the reconstruction filter 4902 may composite depth information 4904 and data from the raycast engine 4906. In some embodiments, the reconstruction filter 4902 may reduce or remove noise from the depth information 4904 based, at least in part, on the data from the raycast engine 4902 and/or from the composite of the depth information 4904 and the data from the raycast engine 4906. In some embodiments, the reconstruction filter 4902 may up-sample the depth information 4904 using deep-learning techniques).
Kut, Moly, and Kopf teaches claim 15. For motivation, see claim 12.
Claim(s) 17, 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Kutliroff et al. (US 20180018805, hereinafter “Kut”) in view of Jiang et al. (US 20150193935) and Molyneaux et al. (US 20190197786, hereinafter “Moly”).
Re claim 17, Kut teaches one or more non-transitory computer-readable media storing computer-executable instructions, which when executed by one or more processors cause the one or more processors to perform operations comprising:
receiving plurality of frames associated with the physical environment from the one or more image components (see [0024], capture 3D image frames of the scene from multiple perspectives or camera poses).
performing segmentation on the plurality of frames to determine a first portion of the plurality of the frames ([0037] FIG. 8 is a more detailed block diagram of a 3D segmentation circuit 236, configured in accordance with certain embodiments of the present disclosure. The segmentation circuit is shown to include a plane fitting circuit 802, a floor/wall/ceiling determination circuit 804, a floor/wall/ceiling removal circuit 806, and a connected components clustering circuit 808. The 3D segmentation circuit 236 is configured to segment out each of the objects detected by the object detection circuit 234, by finding the points of the 3D reconstruction that correspond to the contours of the objects. To this end, recall that each object captured by a 3D camera in a given image is associated with a point cloud, and that point cloud is represented by a set of points included in the depth map provided by the 3D camera. As will be appreciated in light of this disclosure, two or more objects detected in a scene, which are grouped together based on a measure of similarity of features, can be fused by combining point clouds associated with each of the detected objects)
determining a normal for individual pixels of the first portion and assigning the individual pixels to a surface ([0038] The plane fitting circuit 802 is configured to scan the 3D reconstruction of the scene for planar surfaces. One method to accomplish this is to calculate normal vectors to the surfaces by scanning the depth maps and calculating the cross product of differences of neighboring depth pixels. The normal vectors are then clustered into groups based on spatial proximity and the values of the vectors) and ([0025] At a top level, the 3D scene reconstruction system 130 may be configured to generate a 3D reconstruction of the scene comprising points in 3D space corresponding to structures or objects within the scene. In accordance with an embodiment, the 3D scene reconstruction system 130 is configured to recognize objects within the scene as belonging to a class of objects, and to further recognize multiple instances of those objects appearing within the scene, thus enabling fusion of the data associated with each instance, to fill gaps and improve reconstruction accuracy, as will be explained in greater detail below. In some embodiments, the resulting 3D reconstructed scene may also be presented to the user on display element 112).
determining a depth associated with the surface ([0027] FIG. 3 illustrates an example color image and depth map, as may be provided by depth camera 110, in accordance with certain embodiments of the present disclosure. An RGB image frame 302 is shown alongside a corresponding depth map frame 304 of a scene of a furnished room provided by the depth camera. The scene, for which a context-based 3D reconstruction is desired, includes a variety of objects such as a table, lamp, sofa, etc.,. The RGB frame 302 captures color data represented by RGB pixels of the image, although other color schemes are possible such as luminance/chrominance (also referred to as YUV). In the depth map frame 304, each pixel represents the distance between that region of the scene and the camera. This can be done through the intensity value of each pixel. In the depth map frame 304, for example, the pixel indicates proximity to the camera. In particular, regions with darker (more intense) pixels 310 are relatively far from the camera, while lighter (less intense) pixels 312 are closer. In addition, the operational parameters of the camera, such as focal length and principal axis, are known or provided so that each depth pixel can be projected into a 3D position in the scene relative to the camera. Furthermore, the RGB and depth components of the camera may be calibrated, both spatially and temporally, to improve the accuracy of the RGB values and the depth values of each individual pixel) and ([0029] As new RGB and depth frames of the scene 120 are captured by depth camera 110 they are provided to the camera pose calculation circuit 402. The camera pose calculation circuit 402 is configured to compute and update the position and orientation (or pose) of the camera. The calculated pose of the camera is the 3D transformation from the position and orientation of the camera in a previous frame, to its position and orientation in the current frame. Three parameters may describe the translation of the camera between consecutive frames (e.g., x, y and z). Three additional parameters may describe the change in orientation (e.g., yaw, pitch and roll angle) for a total of six degrees of freedom (6DOF) that are computed to determine the updated pose of the camera relative to its pose in the previous frame. Determination of the camera pose for each frame can be used to establish a global coordinate system, consistent across all captured frames, in which the 3D points extracted from the depth map can be projected and accumulated by depth pixel accumulation circuit 406. The calculation of the camera pose may be performed in real-time)
and generating a three-dimensional scene reconstruction from the plurality of frames based at least in part on the depth, a three-dimensional point, and the surface (([0025] At a top level, the 3D scene reconstruction system 130 may be configured to generate a 3D reconstruction of the scene comprising points in 3D space corresponding to structures or objects within the scene. In accordance with an embodiment, the 3D scene reconstruction system 130 is configured to recognize objects within the scene as belonging to a class of objects, and to further recognize multiple instances of those objects appearing within the scene, thus enabling fusion of the data associated with each instance, to fill gaps and improve reconstruction accuracy, as will be explained in greater detail below. In some embodiments, the resulting 3D reconstructed scene may also be presented to the user on display element 112).
Kut does not explicitly teach wherein the 3d point is a 3d center point.
However, Jiang teaches wherein the 3d point is a 3d center point ([0022] FIG. 5 illustrates a 3D point Q on a plane .pi. and the 2D projection q, q' of the 3D point Q on two respective images I, I' with different views of the plane .pi. to illustrate determining the pose of the master device 110A using the following notation) and (0035] For a 3D point Q on the plane .pi., its 2D projection q and the center of the camera 114, illustrated in FIG. 5 as point O, forms a ray. After intersecting the ray with the plane .pi., the coordinate for the 3D point Q can be determined The plurality of reference points, correspond to a plurality of 3D points, which form a 3D plane. Generally, there are two solutions for homography decomposition. To select the correct solution, the two resulting 3D planes are stored and used to estimate the master device 110A pose in subsequent frames. When the average projection error for one plane is greater than the average projection error for the other plane, e.g., one is 1.2 times greater; the plane having the larger projection error is eliminated. If the initial image is roughly a front view of the planar object 102, the plane normal n may be used to select the correct 3D plane. After the correct 3D plane is selected, a world coordinate frame is defined to align the z-axis with the plane normal n with the origin on the plane. Thus, with the 3D-2D homography H determined, the pose of the master device 110A with respect to the object 102 is determined based on the rotation matrix R and translation vector t as discussed above with reference to equation 6. If desired, other pose determination techniques may be used).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Kut’s 3d reconstruction system including determining a first depth value between a point of the plurality of frames and a first portion to explicitly include a 3d center point of the plurality of frames, as taught by Jiang, as the references are in the analogous art of 3D reconstruction systems based on a plurality of image frames. An advantage of the modification is that it achieves the result of using the center of the camera as a reference point for ray projection from a known origin for pose estimation of the camera, as taught by Jiang.
Kut and Jiang do not explicitly teach assigning the individual pixels to a surface based at least in part on a threshold normal difference between the individual pixels.
However, Moly teaches assigning the individual pixels to a surface based at least in part on a threshold normal difference between the individual pixels ([0199] FIG. 22 shows portions 1400 of the plane extraction system 1300, illustrating details on plane extraction 1310, according to some embodiments. The plane extraction 1310 may include dividing each of the mesh bricks 1308 into sub-bricks 1402. Plane detection 1404 may be conducted for each of the sub-bricks 1402. For example, plane detection 1404 may compare primitive normals of each mesh triangle in a sub-brick; merge those mesh triangles, with primitive normal differences smaller than a predetermined threshold value, into one mesh triangle; and identify a mesh triangle with an area larger than a predetermined area value as a plane).
Kut, Jiang, and Moly teach claim 17. It would have been obvious to one ordinary skill in the art before the effective filing date of the claimed invention to modify Kut and Jiang’s 3d reconstruction system including grouping pixels based on normal values to explicitly include assigning the individual pixels to a surface based at least in part on a threshold normal difference between the individual pixels, as taught by Moly, as the references are in the analogous art of 3d reconstruction systems. An advantage of the modification is that it achieves the result of using thresholds in order to group pixels based on normal thresholds, so that similar pixels can be grouped as belonging to the same surface, as taught by Moly.
Re claim 20, Kut, Jiang and Moly teach claim 17. Furthermore, Kut teaches determining a normal associated with the surface based at least in part on the normal of the individual pixels; and wherein determining the depth is based at least in part on the normal associated with a surface (([0027] FIG. 3 illustrates an example color image and depth map, as may be provided by depth camera 110, in accordance with certain embodiments of the present disclosure. An RGB image frame 302 is shown alongside a corresponding depth map frame 304 of a scene of a furnished room provided by the depth camera. The scene, for which a context-based 3D reconstruction is desired, includes a variety of objects such as a table, lamp, sofa, etc.,. The RGB frame 302 captures color data represented by RGB pixels of the image, although other color schemes are possible such as luminance/chrominance (also referred to as YUV). In the depth map frame 304, each pixel represents the distance between that region of the scene and the camera. This can be done through the intensity value of each pixel. In the depth map frame 304, for example, the pixel indicates proximity to the camera. In particular, regions with darker (more intense) pixels 310 are relatively far from the camera, while lighter (less intense) pixels 312 are closer. In addition, the operational parameters of the camera, such as focal length and principal axis, are known or provided so that each depth pixel can be projected into a 3D position in the scene relative to the camera. Furthermore, the RGB and depth components of the camera may be calibrated, both spatially and temporally, to improve the accuracy of the RGB values and the depth values of each individual pixel) and ([0029] As new RGB and depth frames of the scene 120 are captured by depth camera 110 they are provided to the camera pose calculation circuit 402. The camera pose calculation circuit 402 is configured to compute and update the position and orientation (or pose) of the camera. The calculated pose of the camera is the 3D transformation from the position and orientation of the camera in a previous frame, to its position and orientation in the current frame. Three parameters may describe the translation of the camera between consecutive frames (e.g., x, y and z). Three additional parameters may describe the change in orientation (e.g., yaw, pitch and roll angle) for a total of six degrees of freedom (6DOF) that are computed to determine the updated pose of the camera relative to its pose in the previous frame. Determination of the camera pose for each frame can be used to establish a global coordinate system, consistent across all captured frames, in which the 3D points extracted from the depth map can be projected and accumulated by depth pixel accumulation circuit 406. The calculation of the camera pose may be performed in real-time).
Claim(s) 18-19 is/are rejected under 35 U.S.C. 103 as being unpatentable over Kutliroff et al. (US 20180018805, hereinafter “Kut”) in view of Jiang et al. (US 20150193935), Molyneaux et al. (US 20190197786, hereinafter “Moly”), and Kopf et al. (US 20180302612).
Re claim 18, Kut, Jiang, and Moly teach claim 17. Furthermore, Kut teaches planarizing points associated with the surface ([0038] The plane fitting circuit 802 is configured to scan the 3D reconstruction of the scene for planar surfaces. One method to accomplish this is to calculate normal vectors to the surfaces by scanning the depth maps and calculating the cross product of differences of neighboring depth pixels. The normal vectors are then clustered into groups based on spatial proximity and the values of the vectors. Next, a plane is fitted to each cluster).
Kut, Jiang, and Moly do not explicitly teach denoising points associated with the surface.
However, Kopf teaches denoising points associated with the surface (see [0070] in reference to Fig. 29, denoised mesh by planarizing the mesh) and ([0366] The reconstruction filter 4902 may include computer executable instructions for producing a depth map based on the depth information 4904. The depth map may include one or more pixels. Each pixel may indicate a distance to a point of a surface in a physical world. In some embodiments, the reconstruction filter 4902 may composite depth information 4904 and data from the raycast engine 4906. In some embodiments, the reconstruction filter 4902 may reduce or remove noise from the depth information 4904 based, at least in part, on the data from the raycast engine 4902 and/or from the composite of the depth information 4904 and the data from the raycast engine 4906. In some embodiments, the reconstruction filter 4902 may up-sample the depth information 4904 using deep-learning techniques).
Kut, Jiang, Moly, and Kopf teaches claim 18. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Kut, Jiang, and Moly’s 3d reconstruction system including calculating normals of a surface to explicitly teach denoising points associated with a surface, as taught by Kopf, as the references are in the analogous art of 3d reconstruction systems using calculated surface normal. An advantage of the modification is that it achieves the result of using a filter to discard depth values that are inconsistent integrally over a surface to maintain correct estimated depth values and improve quality by filtering, as taught by Kopf.
Re claim 19, Kut, Jiang, and Moly teach claim 17. Kut, Jiang, and Moly do not explicitly teach wherein determining the depth associated with the surface further comprises selecting a depth that minimizes a photoconsistency error, computed integrally for an overall region of
the surface.
However, Kopf teaches teach wherein determining the depth associated with the surface further comprises selecting a depth that minimizes a photoconsistency error, computed integrally for an overall region of the surface (see [0043], normal map estimation module 128 estimates normal for surfaces) and ([0056] To compute the near envelope n the dense reconstruction module 124 first identifies 302 anchor pixels with reliable depth estimates. The dense reconstruction module 124 computes the anchors from two sources: (1) stereo matching where it is reliable, and (2) the point features already computed in the point cloud during the sparse reconstruction. Stereo matching is known to work well at strong image gradients. This is used by computing a lightweight MVS result and carefully considering the pixels around image edges for anchors. The dense reconstruction module 124 obtains a noisy depth map by independently computing the minimum of E.sub.data for each pixel, e.g., dropping the smoothness term from Eq. 1. The dense reconstruction module 124 uses a geometric consistency filter with aggressive settings to discard a significant portion of the incorrectly estimated depth values. An edge detector is used to compute an edge mask. If the image edge coincides with a depth edge, it is desirable to ensure that the depth belonging to the front layer is selected. This may be achieved by dilating the edge mask by 1 pixel and applying a 5×5 morphological minimum filter to the masked pixels inside the detected edges. In addition to these anchors computed from stereo, the dense reconstruction module 124 also adds all the sparse point features that are observed in the image to the set of anchor pixels).
Kut, Jiang, Moly, and Kopf teach claim 19. For motivation, see claim 18.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Peter Hoang whose telephone number is (571)270-1346. The examiner can normally be reached Monday-Friday 8:00 am - 5:00 pm PST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Hajnik F. Daniel can be reached at (571) 272-7642. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/PETER HOANG/ Primary Examiner, Art Unit 2616
Prosecution Insights