Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-7, 9, 11-17, and 19 is/are rejected under 35 U.S.C. 103 as being unpatentable over Murez (U.S. PGPUB 20210279943) in view of Serlie (U.S. PGPUB 20140003691), Molyneaux et al. (U.S. PGPUB 20210042988), and further in view of Kim et al. (U.S. PGPUB 20130202162).
With respect to claim 1, Murez discloses an apparatus for three-dimensional reconstruction (3DR) of a scene (paragraph 51, Referring to FIG. 1, an exemplary XR system 100 according to one embodiment is illustrated), the apparatus comprising: at least one memory; and at least one processor coupled to the at least one memory (paragraph 51, an interconnected auxiliary computing system or controller 6 (also referred to as an interconnected auxiliary computing system or controller component 6) which may be configured to be worn as a belt pack or the like on the user, it is deemed inherent that such a computing system or controller comprises a memory and processor coupled to the memory) and configured to:
extract a plurality of two-dimensional (2D) features from a plurality of image frames of a scene, wherein each image frame of the plurality of image frames comprises a respective view of the scene (paragraph 58, The presently disclosed method begins by obtaining a sequence of frames of RGB images, such as images obtained by the cameras 22, 24 and 26, or other suitable cameras. Then, features from each of the frames is extracted using a 2D CNN);
unproject the plurality of 2D features from a 2D space onto a three-dimensional (3D) space to obtain a plurality of 3D features (paragraph 58, These features are then back-projected into a 3D volume using the known camera intrinsics and extrinsics, paragraph 61, At step 122, the features 120a, 120b, 120c are then back-projected into a respective 3D voxel volume 124). However, Murez does not expressly disclose determining a plurality of weights for the plurality of 3D features based on geometric information and visual similarity information for the plurality of image frames, wherein each weight of the plurality of weights is associated with a respective feature 3D of the plurality of 3D features; determining a plurality of voxel features for the plurality of image frames based, respectively, on the plurality of weights for the plurality of 3D features; and generating the 3DR of the scene based on the plurality of voxel features.
Serlie, who also deal with 3D reconstruction, disclose a method for determining a plurality of weights for the plurality of 3D features, wherein each weight of the plurality of weights is associated with a respective feature 3D of the plurality of 3D features (paragraph 52, The system may comprise a weighting subsystem 4 for weighting an image element value of the image element, based on the distance);
determining a plurality of voxel features for the plurality of image frames based, respectively, on the plurality of weights for the plurality of 3D features (paragraph 52, to obtain a weighted image element value); and
generating the 3DR of the scene based on the plurality of voxel features (paragraph 53, The system may comprise a view generator 5 for generating a view 8 of the image volume dataset 1, based on the weighted image element value, in which a view element value of the view 8 is based on the weighted image element value. It is possible that the distance computing subsystem 3 and the weighting subsystem 4 have processed a plurality of image elements, or all of the image elements, so that an at least partially weighted image volume dataset is obtained. The view generator 5 may be arranged for generating a view of such (at least partially) weighted image volume dataset).
Murez and Serlie are in the same field of endeavor, namely computer graphics.
Before the effective filing date of the claimed invention, it would have been obvious to apply the method of determining a plurality of weights for the plurality of 3D features, wherein each weight of the plurality of weights is associated with a respective feature 3D of the plurality of 3D features; determining a plurality of voxel features for the plurality of image frames based, respectively, on the plurality of weights for the plurality of 3D features; and generating the 3DR of the scene based on the plurality of voxel features, as taught by Serlie, to the Murez system, because it would be advantageous to have an improved volume visualization (paragraph 5 of Serlie).
Molyneaux et al., who also deal with 3D reconstruction, disclose a method for determining a plurality of weights for the plurality of 3D features based on geometric information (paragraph 60, In embodiments in which the signed distance function is a truncated signed distance function, the maximum absolute value for a distance in a voxel may be truncated to some maximum, T, such that the signed distance would lie in the interval from −T to T. Further, each voxel may include a weight, indicating a certainty that the distance for the voxel accurately reflects the distance to a surface, paragraph 123, In the illustrated example, voxels before the surface 904 but outside the truncated distance −T are assigned with a signed distance of the truncated distance −T and a weight of “1” because it is certain that everything between the sensor and the surface is empty. Voxels between the truncated distance −T and the surface 904 are assigned with a signed distance between the truncated distance −T and 0, and a weight of “1” because it is certain to be outside an object. Voxels between the surface 904 and a predetermined depth behind the surface 904 are assigned with a signed distance between 0 and the truncated distance T, and a weight between “1” and “0” because the farther away a voxel behind the surface, the less certain is whether it represents inside of an object or empty space. After the predetermined depth, all voxels lying behind the surface receive a zero update).
Murez, Serlie, and Molyneaux et al. are in the same field of endeavor, namely computer graphics.
Before the effective filing date of the claimed invention, it would have been obvious to apply the method of determining a plurality of weights for the plurality of 3D features based on geometric information, as taught by Molyneaux et al., to the Murez as modified by Serlie system, because this may remove a surface from the 3D representation of the environment based on the signed distances and weights in the updated pixels (paragraph 140 of Molyneaux et al.).
Kim et al., who also deal with 3D reconstruction, disclose a method for determining a plurality of weights for the plurality of 3D features based visual similarity information for the plurality of image frames (paragraph 45, patch similarities between the target frame and each of the database frames are obtained, it is considered that a database frame having a higher similarity has more super-resolution pixel information required for the target frame, and such similarity is used as a weighting factor for computation processing for super-resolution reconstruction).
Murez, Serlie, Molyneaux et al., and Kim et al. are in the same field of endeavor, namely computer graphics.
Before the effective filing date of the claimed invention, it would have been obvious to apply the method of determining a plurality of weights for the plurality of 3D features based visual similarity information for the plurality of image frames, as taught by Kim et al., to the Murez as modified by Serlie and Molyneaux et al. system, because when a weighting factor is determined according to a similarity, as a result, pixels in a patch of a database frame having a high similarity have high contribution in relation to a super-resolution pixel value of a corresponding patch of the target frame (paragraph 48 of Kim et al.).
With respect to claim 2, Murez as modified by Serlie, Molyneaux et al., and Kim et al. disclose the apparatus of claim 1, wherein the visual similarity information is based on a visual similarity of two or more 2D features of the plurality of 2D features of the plurality of 2D features from different images frames of the plurality of image frames (Kim et al.: paragraph 26, FIG. 1B illustrates results of extracting a facial region and feature point regions from several continuous frames, paragraph 27, at least one feature point region in the facial region, the present invention is not limited thereto, paragraph 31, similarities between the target frame and each of the remaining frames are calculated in units of the unit patches divided in operation S106. That is, similarities between each unit patch of the target frame and unit patches of the remaining frames corresponding to the unit patch of the target frame are calculated). Thus, Kim et al. suggest at least one feature point. It would have been obvious to include at least two feature points (two or more features) because this would improve accuracy of the 3D reconstruction.
With respect to claim 3, Murez as modified by Serlie, Molyneaux et al., and Kim et al. disclose the apparatus of claim 1, wherein the geometric information is based on a plurality of distances, wherein each distance of the plurality of distances is between a respective voxel associated with a respective voxel feature of the plurality of voxel features to a location of a respective image sensor (Molyneaux et al.: paragraph 121, Such a depth image may include a grid of pixels (not shown) in the plane parallel to the x-coordinate and y-coordinate. Each pixel may indicate a distance, in a particular direction, from the image sensor 906 to the surface 904, paragraph 122, The XR system may update the grid of voxels based on the depth image captured by the sensor 906). It would have been obvious for the geometric information to be based on a plurality of distances, because the reconstruction may be more complete and less noisy than the original sensor data by using spatial and temporal averaging (i.e. averaging data from multiple viewpoints over time) (paragraph 117 of Molyneaux et al.), which uses a plurality of distance data.
With respect to claim 4, Murez as modified by Serlie, Molyneaux et al., and Kim et al. disclose the apparatus of claim 3, wherein the respective image sensor is a camera (Molyneaux et al.: paragraph 91, world cameras 52 record a greater-than-peripheral view to map the environment 32 and detect inputs that may affect AR content). It would have been obvious for the image sensor to be a camera, because this would use common, readily available equipment for performing image capture.
With respect to claim 5, Murez as modified by Serlie, Molyneaux et al., and Kim et al. disclose the apparatus of claim 1, wherein the geometric information is based on a plurality of view angles, wherein each view angle of the plurality of view angles is between two respective rays each associated with a respective image frame of the plurality of image frames (Molyneaux et al.: paragraph 115, the perception module 160 may include components that format information to provide the component 164. An example of such a component may be raycasting component 160f. A use component (e.g., component 164), for example, may query for information about the physical world from a particular point of view. Raycasting component 160f may select from one or more representations of the physical world data within a field of view from that point of view). It would have been obvious for the geometric information is based on a plurality of view angles, wherein each view angle of the plurality of view angles is between two respective rays each associated with a respective image frame of the plurality of image frames because this would perform processing for visual occlusion, physics-based interactions, and environment reasoning (paragraph 113 of Molyneaux et al.).
With respect to claim 6, Murez as modified by Serlie, Molyneaux et al., and Kim et al. disclose the apparatus of claim 1, wherein the at least one processor is configured to determine the plurality of weights for the plurality of 3D features further based on at least one of semantic labels (Murez: paragraph 56, the voxel volume is passed through a three-dimensional (3D) CNN configured to refine the features and predict the TSDF values. Additional heads may be added to predict color, semantic, and instance labels with minimal extra compute resource), depth predictions, depth uncertainties, or 3D scans associated with the plurality of 2D features (Murez: paragraph 59, The networks are trained and evaluated on real scans of indoor rooms from the Scannet and RIO datasets).
With respect to claim 7, Murez as modified by Serlie, Molyneaux et al., and Kim et al. disclose the apparatus of claim 1, wherein the at least one processor is configured to group one or more voxel features of the plurality of voxel features together to generate a group of voxel features based on the one or more voxel features having a visual similarity across different views of the scene (Kim et al.: paragraph 47, In operation S206, a 2D super-resolution image of the target frame is acquired by calculating a weighting factor Wn by Equation 2 from the similarity Sn calculated in operation S205, adjusting a pixel belonging to a patch of the target frame to a pixel brightness value Is reconstructed to a super resolution by Equation 3 using the calculated weighting factor Wn, and repeating the pixel brightness value adjusting process for all pixels included in the facial region of the target frame [grouping by pixel brightness], wherein unprojecting the plurality of 2D features from the 2D space onto the 3D space is based on the group of voxel features (Murez: paragraph 58, These features are then back-projected into a 3D volume using the known camera intrinsics and extrinsics, paragraph 61, At step 122, the features 120a, 120b, 120c are then back-projected into a respective 3D voxel volume 124). Murez in combination with Kim et al. disclose unprojecting the grouped features from Kim et al.
With respect to claim 9, Murez as modified by Serlie, Molyneaux et al., and Kim et al. disclose the apparatus of claim 1, wherein each weight of the plurality of weights is a positive value (Murez: For the weights we use a binary mask Wt(i,j,k)∈{0, 1} which stores if voxel (i,j,k) is inside or outside the view frustum of the camera).
With respect to claim 11, Murez as modified by Serlie, Molyneaux et al., and Kim et al. disclose a method for three-dimensional reconstruction (3DR) of a scene, the method executed by the system of claim 1; see rationale for rejection of claim 1.
With respect to claim 12, Murez as modified by Serlie, Molyneaux et al., and Kim et al. disclose the method of claim 11 as executed by the system of claim 2; see rationale for rejection of claim 2.
With respect to claim 13, Murez as modified by Serlie, Molyneaux et al., and Kim et al. disclose the method of claim 11 as executed by the system of claim 3; see rationale for rejection of claim 3.
With respect to claim 14, Murez as modified by Serlie, Molyneaux et al., and Kim et al. disclose the method of claim 13 as executed by the system of claim 4; see rationale for rejection of claim 4.
With respect to claim 15, Murez as modified by Serlie, Molyneaux et al., and Kim et al. disclose the method of claim 11 as executed by the system of claim 5; see rationale for rejection of claim 5.
With respect to claim 16, Murez as modified by Serlie, Molyneaux et al., and Kim et al. disclose the method of claim 11 as executed by the system of claim 6; see rationale for rejection of claim 6.
With respect to claim 17, Murez as modified by Serlie, Molyneaux et al., and Kim et al. disclose the method of claim 11 as executed by the system of clam 7; see rationale for rejection of claim 7.
With respect to claim 19, Murez as modified by Serlie, Molyneaux et al., and Kim et al. disclose the method of claim 11 as executed by the system of claim 9; see rationale for rejection of claim 9.
Claim(s) 8 and 18 is/are rejected under 35 U.S.C. 103 as being unpatentable over Murez (U.S. PGPUB 20210279943) in view of Serlie (U.S. PGPUB 20140003691), Molyneaux et al. (U.S. PGPUB 20210042988), Kim et al. (U.S. PGPUB 20130202162), and further in view of Berman et al. (U.S. PGPUB 20120189158).
With respect to claim 8, Murez as modified by Serlie, Molyneaux et al., and Kim et al. disclose the apparatus of claim 1. However, Murez as modified by Serlie, Molyneaux et al., and Kim et al. do not expressly disclose the at least one processor is configured to determine the plurality of weights for the plurality of 3D features further based on a lookup table, the lookup table mapping the geometric information to weights of the plurality of weights.
Berman et al., who also deal with 3D reconstruction, disclose a method wherein the at least one processor is configured to determine the plurality of weights for the plurality of 3D features further based on a lookup table, the lookup table mapping the geometric information to weights of the plurality of weights (paragraph 27, The weight coefficient for each voxel can be pre-calculated and saved in the memory 124 in the form of, for example, a table 126).
Murez, Serlie, Molyneaux et al., Kim et al., and Berman et al. are in the same field of endeavor, namely computer graphics.
Before the effective filing date of the claimed invention, it would have been obvious to apply the method wherein the at least one processor is configured to determine the plurality of weights for the plurality of 3D features further based on a lookup table, the lookup table mapping the geometric information to weights of the plurality of weights, as taught by Berman et al., to the Murez as modified by Serlie, Molyneaux et al., and Kim et al. system, because the effort required to reconstruct these tables is noticeably less than the effort required to storing them in their original complete from (paragraph 24 of Berman et al.).
With respect to claim 18, Murez as modified by Serlie, Molyneaux et al., Kim et al., and Berman et al. disclose the method of claim 11 as executed by the system of claim 8; see rationale for rejection of claim 8.
Claim(s) 10 and 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Murez (U.S. PGPUB 20210279943) in view of Serlie (U.S. PGPUB 20140003691), Molyneaux et al. (U.S. PGPUB 20210042988), Kim et al. (U.S. PGPUB 20130202162), and further in view of Shi et al. (CN 116258835).
With respect to claim 10, Murez as modified by Serlie, Molyneaux et al., and Kim et al. disclose the apparatus of claim 1. However, Murez as modified by Serlie, Molyneaux et al., and Kim et al. do not expressly disclose a sum of all weights of the plurality of weights is equal to one.
Shi et al., who also deal with 3D reconstruction, disclose a method wherein a sum of all weights of the plurality of weights is equal to one (paragraph 61, Feature weighting: By learning independent linear layers, parameterized by corresponding
weight vectors, relative weights are generated, where represents the dot product operation, and the total weights are 1).
Murez, Serlie, Molyneaux et al., Kim et al., and Shi et al. are in the same field of endeavor, namely computer graphics.
Before the effective filing date of the claimed invention, it would have been obvious to apply the method wherein a sum of all weights of the plurality of weights is equal to one, as taught by Shi et al., to the Murez as modified by Serlie, Molyneaux et al., and Kim et al. system, because this would normalize the weights to simplify calculations.
With respect to claim 20, Murez as modified by Serlie, Molyneaux et al., Kim et al., and Shi et al. disclose the method of claim 11 as executed by the system of claim 10; see rationale for rejection of claim 10.
Response to Arguments
Applicant’s arguments with respect to claim(s) 1 and 11 have been considered but are moot in view of the new ground(s) of rejection.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
U.S. PGPUB 20120251003 to Perbet et al. for a method of calculating a set of weights from a measure of similarity between pixels and performing a 3D reconstruction.
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ANDREW GUS YANG whose telephone number is (571)272-5514. The examiner can normally be reached M-F 9 AM - 5:30 PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kent Chang can be reached at (571)272-7667. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ANDREW G YANG/Primary Examiner, Art Unit 2614
4/3/26
Prosecution Insights