DETAILED ACTION
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA . In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
Applicant’s submission filed on 02/05/2026 has been entered. Claims 7, 16, and 19 were amended. Claims 1-20 are pending in the application.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-2, 10-11, and 19-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Cole et al. (US 2019/0147642) in view of Peng et al. ("Sparse-to-dense multi-encoder shape completion of unstructured point cloud," IEEE Access 8 (2020): 30969-30978), Wang et al. (US 2014/0043329), Ambrus et al. (US 2023/0177850), and Martinez (US 2019/0114824).
Regarding claim 1, Cole teaches/suggests: A computer-implemented method for completing three dimensional face reconstruction comprising:
receiving image data (Cole [0018] “a user may submit a photograph 105 that includes an image of a face to the system 100”);
analyzing the image data and extracting two dimensional facial features (Cole [0019] “The facial recognition engine 110 is configured to generate image features 112 that correspond to the face of the user depicted in the photograph 105”);
generate a three dimensional facial feature point cloud of complete facial features (Cole [0020] “The 3D estimator neural network 112 processes the image features 112 received from the facial recognition engine 110 to generate data 122 that specifies an estimated 3D shape and an estimated texture of the face of the user depicted in the photograph 105” [0031] “Other types of artificial neural networks could also be used for the 3D estimator neural network, so long as the neural network is configured to receive as input an image or features derived from an image, and produces an output … point clouds”),
wherein the three dimensional facial feature point cloud is utilized to control a computing device to complete a downstream task (Cole [0021] “The 3D rendering engine 130 is configured to generate the avatar 115 corresponding to the face of the user depicted in the photograph 105 based on the data 122 specifying the estimated 3D shape and texture” [The rendering meets the downstream task.]).
Cole does not teach/suggest:
receiving image data associated with multiple two dimensional non-frontal face images;
inputting the sparse three dimensional facial feature point clouds into an encoder-decoder architecture to generate a three dimensional facial feature point cloud of complete facial features,
Peng, however, teaches/suggests sparse three dimensional
receiving image data associated with multiple two dimensional non-frontal face images (Cole Fig. 1: the illustrated non-frontal photograph; Peng Abstract “Unstructured point clouds are a representative shape representation of real-world scenes in 3D vision and graphics. Incompletion inevitably arises, due to the way the set of unorganized points is captured, e.g., as fusion of depth images”);
inputting the sparse three dimensional facial feature point clouds into an encoder-decoder architecture to generate a three dimensional facial feature point cloud of complete facial features (Cole [0020] “The 3D estimator neural network 112 processes the image features 112 received from the facial recognition engine 110 to generate data 122 that specifies an estimated 3D shape and an estimated texture of the face of the user depicted in the photograph 105” Peng §III-C ¶2 “we adopt PointNet++ [12] as the encoder to encode and decode the sparse point cloud ... we use a three-layer MLP to decode the expanded feature and reduce the feature dimension to 3 to regress the 3D coordinates of the final point cloud”),
Cole and Peng are silent regarding:
constructing sparse three dimensional facial feature point clouds based on the two dimensional facial features;
Wang, however, teaches/suggests:
constructing sparse three dimensional facial feature point clouds based on the two dimensional facial features (Wang [0031] “sparse point clouds of the user's face will be recovered accordingly ... landmark feature points between the 2D face model and 3D face model may be detected and registered”);
Before the effective filing date of the claimed invention, it would have been obvious for one of ordinary skill in the art to modify the sparse point clouds of Cole as modified by Peng to be constructed as taught/suggested by Wang to be input into the PointNet++.
Peng further teaches/suggests in parallel (Peng §III-C ¶2 “The hierarchical feature learning structure of PointNet++ has been proven to be able to learn the local and global features of point cloud simultaneously”). Cole as modified by Peng and Wang does not teach/suggest including an encoder employing two or more types of neural networks in parallel. Ambrus and Martinez, teach/suggest two or more types of neural networks (Ambrus [0080] “Another implementation of the aggregation layer 860 may be a dynamic graph (DG) convolutional neural network (CNN) that maintains a permutation invariance of point sets; however, the aggregation layer 860 may be designed to capture a local geometric structure by encoding features in edges between points” Martinez [0009] “The innovation enables mapping with a feed-forward neural network that defines two criteria, one that learns to detect important shape landmark points on an image”). Before the effective filing date of the claimed invention, the substitution of one known element (the dynamic graph CNN of Ambrus and the feed-forward NN of Martinez) for another (the PointNet++ of Peng) would have been obvious to one of ordinary skill in the art because such substitutions would have yielded predictable results, namely to learn the local and global features.
As such, Cole as modified by Peng, Wang, Ambrus, and Martinez teaches/suggests an encoder employing two or more types of neural networks in parallel (Peng §III-C ¶2 “The hierarchical feature learning structure of PointNet++ has been proven to be able to learn the local and global features of point cloud simultaneously” Ambrus [0080] “Another implementation of the aggregation layer 860 may be a dynamic graph (DG) convolutional neural network (CNN) that maintains a permutation invariance of point sets; however, the aggregation layer 860 may be designed to capture a local geometric structure by encoding features in edges between points” Martinez [0009] “The innovation enables mapping with a feed-forward neural network that defines two criteria, one that learns to detect important shape landmark points on an image”).
Regarding claim 2, Cole as modified by Peng, Wang, Ambrus, and Martinez teaches/suggests: The computer-implemented method of claim 1, wherein the multiple two dimensional non-frontal face images include occlusions that are caused by at least one of: an individual who is being captured within the images (Cole [0018] “The avatar 115 is an estimation of the face in the photograph 105 from a perspective that is different from the perspective in the photograph 105”) and an object that is located in between at least one camera and the individual who is being captured within the images [This is yet to be considered because of the “at least one of” recitation.].
Claims 10 and 11 recite limitation(s) similar in scope to those of claims 1 and 2, respectively, and are rejected for the same reason(s). Cole as modified by Peng, Wang, Ambrus, and Martinez further teaches/suggests a memory storing instructions and a processor (Cole [0088] “The essential elements of a computer are a processor for performing instructions and one or more memory devices for storing instructions and data”).
Claim 19 recites limitation(s) similar in scope to those of claim 1, and is rejected for the same reason(s). Cole as modified by Peng, Wang, Ambrus, and Martinez further teaches/suggests a non-transitory computer readable storage medium storing instructions (Cole [0088] “The essential elements of a computer are a processor for performing instructions and one or more memory devices for storing instructions and data”).
Regarding claim 20, Cole as modified by Peng, Wang, Ambrus, and Martinez teaches/suggests: The non-transitory computer readable storage medium of claim 19, wherein an output vector of the encoder is fed into a decoder of the encoder-decoder architecture to decode sparse three dimensional facial feature point clouds and to generate the three dimensional facial feature point cloud of complete facial features (Cole [0018] “The avatar 115 is an estimation of the face in the photograph 105 from a perspective that is different from the perspective in the photograph 105” Peng §III-A ¶1 “we encode the input point cloud to get a high-dimensional feature vector, and then decode the feature vector to output a sparse point cloud with complete shape ... and get the final dense complete point cloud through encoding, feature expansion and decoding”). The same rationale to combine as set forth in the rejection of claim 1 is incorporated herein.
Claim(s) 3-5 and 12-14 is/are rejected under 35 U.S.C. 103 as being unpatentable over Cole et al. (US 2019/0147642) in view of Peng et al. ("Sparse-to-dense multi-encoder shape completion of unstructured point cloud," IEEE Access 8 (2020): 30969-30978), Wang et al. (US 2014/0043329), Ambrus et al. (US 2023/0177850), and Martinez (US 2019/0114824) as applied to claims 2 and 11 above, and further in view of Hu et al. (US 2021/0209839).
Regarding claim 3, Cole, Peng, Wang, Ambrus, and Martinez are silent regarding: The computer-implemented method of claim 2, wherein analyzing the image data includes extracting a fixed number of facial landmarks, wherein the fixed number of facial landmarks include the occlusions and a shape completion matrix is used to estimate true locations of occluded facial feature points. Hu, however, teaches/suggests extracting a fixed number of facial landmarks (Hu [0048] “the circuitry 202 may acquire a plurality of pre-defined landmark points on the aligned 3D mean-shape model 324”), wherein the fixed number of facial landmarks include the occlusions and a shape completion matrix is used to estimate true locations of occluded facial feature points (Hu [0058] “As the second aligned 3D mean-shape model 402B may be associated with the non-frontal view of the face of the user 110 in which the face of the user 110 may be titled towards the left-side, a portion of the left-side of the face may be occluded … vertices 410A, 410B, 410C, 410D, 410E, and 410F may be the left-most vertices on the parallel lines in the second aligned 3D mean-shape model 402B. The circuitry 202 may determine landmark points 412A, 412B, 412C, 412D, 412E, and 412F in the second 2D projection 406B as landmark points on the contour of the face, which may correspond to the vertices 410A, 410B, 410C, 410D, 410E, and 410F”). Before the effective filing date of the claimed invention, it would have been obvious for one of ordinary skill in the art to modify the image features of Cole as modified by Peng, Wang, Ambrus, and Martinez to be extracted as taught/suggested by Hu for the shape completion.
Regarding claim 4, Cole as modified by Peng, Wang, Ambrus, Martinez, and Hu teaches/suggests: The computer-implemented method of claim 3, wherein facial features that correspond to the facial landmarks in the multiple two dimensional non-frontal face images are matched (Hu [0045] “The circuitry 202 may use the face modeler to estimate the affine transformation to align the set of feature points on several facial features”). Aligning is considered matching. The same rationale to combine as set forth in the rejection of claim 3 is incorporated herein.
Regarding claim 5, Cole as modified by Peng, Wang, Ambrus, Martinez, and Hu teaches/suggests: The computer-implemented method of claim 4, wherein constructing the sparse three dimensional facial feature point clouds includes using the matching correspondences of the facial features to the facial landmarks in the multiple two dimensional non-frontal face images to create a three dimensional reconstruction of sparse feature points (Peng Abstract “Unstructured point clouds are a representative shape representation of real-world scenes in 3D vision and graphics. Incompletion inevitably arises, due to the way the set of unorganized points is captured, e.g., as fusion of depth images” §III-C ¶2 “we adopt PointNet++ [12] as the encoder to encode and decode the sparse point cloud” Hu [0047] “The set of landmark points 328 in the generated 2D projection 326 may be points that may define key face features of the aligned 3D mean-shape model 324”). The same rationales to combine as set forth in the rejection of claims 1 and 3 are incorporated herein.
Claims 12-14 recite limitation(s) similar in scope to those of claims 3-5, respectively, and are rejected for the same reason(s).
Claim(s) 6-7 and 15-16 is/are rejected under 35 U.S.C. 103 as being unpatentable over Cole et al. (US 2019/0147642) in view of Peng et al. ("Sparse-to-dense multi-encoder shape completion of unstructured point cloud," IEEE Access 8 (2020): 30969-30978), Wang et al. (US 2014/0043329), Ambrus et al. (US 2023/0177850), Martinez (US 2019/0114824), and Hu et al. (US 2021/0209839) as applied to claims 5 and 14 above, and further in view of Zhang et al. (US 2023/0206603).
Regarding claim 6, Cole, Peng, Wang, Ambrus, Martinez, and Hu are silent regarding: The computer-implemented method of claim 5, wherein inputting the sparse three dimensional facial feature point clouds into the encoder-decoder architecture include inputting the sparse three dimensional facial feature point clouds with a variable number of points to generate a complete dense point cloud of a missing part of the face of the individual who is being captured within the images. Zhang, however, teaches/suggests point clouds with a variable number of points (Zhang [0065] “Three missing point clouds of different scales generated by sampling the farthest point are input into the multi-resolution encoder module to extract feature”). Before the effective filing date of the claimed invention, it would have been obvious for one of ordinary skill in the art to modify the sparse point clouds of Cole as modified by Peng, Wang, Ambrus, Martinez, and Hu to have different scales as taught/suggested by Zhang for the shape completion.
As such, Cole as modified by Peng, Wang, Ambrus, Martinez, Hu, and Zhang teaches/suggests inputting the sparse three dimensional facial feature point clouds with a variable number of points to generate a complete dense point cloud of a missing part of the face of the individual who is being captured within the images (Peng Abstract “Unstructured point clouds are a representative shape representation of real-world scenes in 3D vision and graphics. Incompletion inevitably arises, due to the way the set of unorganized points is captured, e.g., as fusion of depth images” §III-C ¶2 “we adopt PointNet++ [12] as the encoder to encode and decode the sparse point cloud ... we use a three-layer MLP to decode the expanded feature and reduce the feature dimension to 3 to regress the 3D coordinates of the final point cloud” Zhang [0065] “Three missing point clouds of different scales generated by sampling the farthest point are input into the multi-resolution encoder module to extract feature”).
Regarding claim 7, Cole as modified by Peng, Wang, Ambrus, Martinez, Hu, and Zhang teaches/suggests: The computer-implemented method of claim 6, wherein the encoder of the encoder-decoder architecture employs a graph convolutional neural network to understand a specific geometry of the sparse three dimensional facial feature point clouds and uses a fully connected class of a feedforward artificial neural network to learn an overall geometry of the face of the individual who is being captured within the images (Cole [0018] “a user may submit a photograph 105 that includes an image of a face to the system 100” Peng §III-C ¶2 “The hierarchical feature learning structure of PointNet++ has been proven to be able to learn the local and global features of point cloud simultaneously” Ambrus [0080] “Another implementation of the aggregation layer 860 may be a dynamic graph (DG) convolutional neural network (CNN) that maintains a permutation invariance of point sets; however, the aggregation layer 860 may be designed to capture a local geometric structure by encoding features in edges between points” Martinez [0009] “The innovation enables mapping with a feed-forward neural network that defines two criteria, one that learns to detect important shape landmark points on an image” [0044] “the deep neural network used four convolutional layers, two max pooling layers and two fully connected layers”). The same rationale to combine as set forth in the rejection of claim 1 is incorporated herein.
Claims 15 and 16 recite limitation(s) similar in scope to those of claims 6 and 7, respectively, and are rejected for the same reason(s).
Allowable Subject Matter
Claims 8-9 and 17-18 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
The following is a statement of reasons for the indication of allowable subject matter: The limitation “wherein the encoder encodes node features along with information of neighbor nodes to use both local and global information to learn an overall geometry of the face of the individual who is being captured within the images,” taken as a whole, renders the claims patentably distinct over the prior art.
Response to Arguments
Applicant's arguments filed on 02/05/2026 have been fully considered but they are not persuasive. Applicant argues “Peng does not disclose constructing sparse three dimensional facial feature point clouds based on the two dimensional facial features … the Office's attempts at Official Notice are improper and traversed.” See Remarks, pp. 8-10. However, Wang is now cited to replace the official notice.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure:
US 2018/0253593 – 3D human face model
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a). A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ANH-TUAN V NGUYEN whose telephone number is 571-270-7513. The examiner can normally be reached on M-F 9AM-5PM ET. Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice. If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, JASON CHAN can be reached on 571-272-3022. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ANH-TUAN V NGUYEN/
Primary Examiner, Art Unit 2619
Prosecution Insights