FLEXIBLE 3D LANDMARK DETECTION

DETAILED ACTION Notice of Pre-AIA or AIA Status The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA . Response to Amendment The amendment filed August 6, 2026 has been entered. Claims 1-20 remain pending in the application. Applicant’s amendments to the Specification have overcome each and every objection previously set forth in the Non-Final Office Action mailed May 6, 2026. Response to Arguments Applicant’s arguments, see Page 10 of Remarks, filed August 6, 2026, with respect to the rejection(s) of claim(s) 1-20 under 35 USC 102 and 103 have been fully considered and are persuasive. Therefore, the rejection has been withdrawn. However, upon further consideration, a new ground(s) of rejection is made in view of Zheng et al. (ImFace: A Nonlinear 3D Morphable Face Model with Implicit Neural Representations). Claim Objections Claims 8, 10, and 13 are objected to because of the following informalities: In claim 8, it is unclear whether “a canonical shape” in line 3 refers to the same canonical shape from claim 1. In claim 10, it is unclear whether “a canonical shape” in line 2 refers to the same canonical shape from claim 1. In claim 13, it is unclear whether “a canonical shape” in line 5 refers to the same canonical shape from claim 11. In claim 13, it is also unclear whether “a set of query points” in line 6 refers to the same set of query points from claim 11. Appropriate correction is required. Claim Rejections - 35 USC § 103 In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status. The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action: A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made. Claims 1-4 and 7-10 are rejected under 35 U.S.C. 103 as being unpatentable over Xiao et al. (Recurrent 3D-2D Dual Learning for Large-pose Facial Landmark Detection) in view of Zheng et al. (ImFace: A Nonlinear 3D Morphable Face Model with Implicit Neural Representations), hereinafter Xiao and Zheng respectively. Regarding claim 1, Xiao teaches a computer-implemented method for performing landmark detection (Paragraph 4 in 2nd Col. of Page 3 – “Fig. 2 shows the entire framework of the RDR model for large-pose 2D landmark detection”), the method comprising: determining a first set of parameters associated with a depiction of a first face in a first image (Paragraph 4 in 2nd Col. of Page 3, Paragraph 2 in 1st Col. of Page 4 – “Fig. 2 shows the entire framework of the RDR model for large-pose 2D landmark detection. It first receives a 2D face image and passes it through a 3D parameter initialization module to directly predict parameters for accurate 3D face initialization… Using good initial parameters can improve the overall performance and accelerate the refinement process. Our model initializes the 3D fitting parameters, i.e., Q = {P, αid, αexp}, by direct regression via the 3D parameter initialization module, Fig. 2”; Note: the predicted initial parameters are equivalent to the first set of parameters); obtaining a set of query points (set of vertices on the neutral 3D face [Wingdings font/0xE0] see quote below) defined on a canonical shape (neutral 3D face [Wingdings font/0xE0] see quote) (Fig. 2, Paragraph 1-2 in 1st Col. of Page 3 – “In the 3D face fitting process within our model, the Dynamic Expression Model (DEM) [23, 4] is employed…In 3D face modeling, each 3D face is described by a set of N vertices F…An expression blendshape, Bi ∈ R3× N, is formed by transforming a neutral 3D face to a 3D face that has a typical expression with the preserved identity. In this work, we use a neutral shape B0 and another 46 expression blendshapes with different expressions to represent a 3D face”; Note: The neutral 3D face is equivalent to the canonical shape. The set of vertices on the neutral 3D face are the query points); generating, via execution of a prediction network included in the first machine learning model (duel refinement module [Wingdings font/0xE0] see quote below), a first set of three-dimensional (3D) landmarks (3D key vertices [Wingdings font/0xE0] see quote below) on the first face based on the first set of parameters (Fig. 2, Paragraph 2 in 1st Col. of Page 3, Paragraph 3 in 1st Col. of Page 4, Paragraph 2 in 2nd Col. of Page 4 – “In 3D face modeling, each 3D face is described by a set of N vertices…This network predicts the parameters upon the features from pool5 by using three independent fc5-fc6’s, each of which predicts a specific 3D parameter of αid,P and αexp respectively. Given the predicted identity parameter αid, the user-specific expression blendshapes B are obtained via Eqn. (2) and used in the following 3D face refinement process. The initial 3D face model is generated with Eqn. (1) and the initial 2D landmark locations are inferred directly via 3D-to-2D landmark projection with Eqn. (4)… We propose to detect 2D landmarks through two steps: first the projection from the 3D key vertices gives Sp and then ‘fine-tuning’ on Sp”; Note: a 3D face model is generated based on the 3D parameters. The 3D face model is made up of vertices, which include key vertices that are equivalent to the 3D landmarks. The dual refinement module, shown in Fig. 2, is equivalent to the first machine learning model and is implied to include a prediction network since it predicts landmarks), wherein, for each query point in the set of query points, a corresponding 3D landmark is generated based on (i) the first set of parameters and (ii) position data corresponding to the query point (Fig. 2, Paragraph 2 in 1st Col. of Page 3, Paragraph 3 in 1st Col. of Page 4, Paragraph 2 in 2nd Col. of Page 4 – “In 3D face modeling, each 3D face is described by a set of N vertices F = [v1, . . . , vN] ∈ R3×N…An expression blendshape, Bi ∈ R3× N, is formed by transforming a neutral 3D face to a 3D face that has a typical expression with the preserved identity …Given a rotation matrix, Rφ,γ,θ ∈ R3×3 and a translational vector t3d ∈ R3, the 3D face reconstruction process of F is formally described as PNG media_image1.png 38 346 media_image1.png Greyscale …This network predicts the parameters upon the features from pool5 by using three independent fc5-fc6’s, each of which predicts a specific 3D parameter of αid,P and αexp respectively. Given the predicted identity parameter αid, the user-specific expression blendshapes B are obtained via Eqn. (2) and used in the following 3D face refinement process. The initial 3D face model is generated with Eqn. (1) and the initial 2D landmark locations are inferred directly via 3D-to-2D landmark projection with Eqn. (4)… We propose to detect 2D landmarks through two steps: first the projection from the 3D key vertices gives Sp and then ‘fine-tuning’ on Sp”; Note: for every vertex on the neutral 3D face (query point), a corresponding vertex on the 3D face is generated (3D landmark). The 3D landmarks are based on parameters, αid and αexp, and position data of the vertices from a neutral 3D face (query points)); projecting, based on the first set of parameters, the first set of 3D landmarks onto the first image to generate a first set of two-dimensional (2D) landmarks (Paragraph 1 in 2nd Col. of Page 3, Paragraph 2 in 2nd Col. of Paragraph 4 – “Using the 3D face model to detect 2D facial landmarks needs to project 3D faces onto the 2D image plane. Under weak perspective projection [3, 29, 11], the 2D locations of all vertices can be formulated as PNG media_image2.png 31 490 media_image2.png Greyscale where Q = {P,αid,αexp} represents the parameters for 3D face fitting and 3D-to-2D projection…We propose to detect 2D landmarks through two steps: first the projection from the 3D key vertices gives Sp and then ‘fine-tuning’ on Sp gives final 2D locations S”; Note: the 3D key vertices, which are the 3D landmarks, are projected onto the image to generate 2D landmarks); and training the first machine learning model based on one or more losses associated with the first set of 2D landmarks to generate a first trained machine learning model (Fig. 2, Paragraph 3 in 2nd Col. of Page 5 – “The overall loss for dual 3D-2D refinement accumulated through K recurrent iterations is then formally described as PNG media_image3.png 115 491 media_image3.png Greyscale … Here β and λ are trade-off parameters to ensure that both 3D vertices and 2D landmarks are close to the ground-truth landmark locations”; Note: the dual refinement module, shown in Fig. 2, is equivalent to the first machine learning model. It is trained with the loss equation shown in the quote). PNG media_image4.png 512 1257 media_image4.png Greyscale Screenshot of Fig. 2 (taken from Xiao) Xiao does not teach converting, via execution of a position encoder included in a first machine learning model, each query point in the set of query points into a respective vector-based position encoding; nor “the respective vector-based position encoding” in the limitation: “wherein, for each query point in the set of query points, a corresponding 3D landmark is generated based on (i) the first set of parameters and (ii) the respective vector-based position encoding corresponding to the query point”. However, Zheng teaches converting, via execution of a position encoder included in a first machine learning model, each query point in the set of query points into a respective vector-based position encoding (Fig. 2b, Paragraph 2 in 2nd Col. of Page 3, Paragraph 3 in 2nd Col. of Page 3, Paragraph 1-2 in 1st Col. of Page 4, Paragraph 2 in 2nd Col. of Page 4 – “All the fields above are implemented by a shared Mini Nets architecture, where the entire facial deformation or geometry is further decomposed into a number of semantically meaningful parts and encoded by a set of local field functions, so that rich details can be sufficiently captured…The facial deformations incurred by expressions are represented by ExpNet E, which learns an observation-to-canonical warping for every face scan: PNG media_image5.png 31 265 media_image5.png Greyscale where l ∈ Rk×3 denotes k 3D landmarks on an observed face generated by a Landmark-Net η : (zexp,zid) → l … To model shape morphs among individuals, IDNet I further warps the canonical space to a template shape space shared by all faces: PNG media_image6.png 32 356 media_image6.png Greyscale where l′ ∈ Rk×3 denotes k landmarks on the canonical face generated by another Landmark-Net conditioned only on the identity embedding η′ : zid → l′, and p′′ is the deformed point in the template space…five landmarks located at the outer eye corners, mouth corners, and nose tip are utilized to describe the local regions (ln ∈ R3)5n=1, and each region is assigned a tiny MLP with sinusoidal activations [49] to generate the local field, denoted as ψn. To capture high-frequency local variations, we leverage sinusoidal positional encoding γ [36] on the coordinate x − ln. At the end of a Mini-Nets block, a lightweight Fusion Network conditioned on the absolute coordinate of input x is equipped”; Note: the query points x are converted into vector-based position encodings γ. Execution of a position encoder is implied because the position encodings would not be generated otherwise. The model shown in Fig. 2b below is equivalent to the first machine learning model in this case); and wherein, for each query point in the set of query points, a corresponding 3D landmark is generated based on (i) the first set of parameters and (ii) the respective vector-based position encoding corresponding to the query point (Fig. 2b, Paragraph 3 in 2nd Col. of Page 3, Paragraph 1-2 in 1st Col. of Page 4, Paragraph 2 in 2nd Col. of Page 4 – “The facial deformations incurred by expressions are represented by ExpNet E, which learns an observation-to-canonical warping for every face scan: PNG media_image5.png 31 265 media_image5.png Greyscale where l ∈ Rk×3 denotes k 3D landmarks on an observed face generated by a Landmark-Net η : (zexp,zid) → l … To model shape morphs among individuals, IDNet I further warps the canonical space to a template shape space shared by all faces: PNG media_image6.png 32 356 media_image6.png Greyscale where l′ ∈ Rk×3 denotes k landmarks on the canonical face generated by another Landmark-Net conditioned only on the identity embedding η′ : zid → l′, and p′′ is the deformed point in the template space…To capture high-frequency local variations, we leverage sinusoidal positional encoding γ [36] on the coordinate x − ln. At the end of a Mini-Nets block, a lightweight Fusion Network conditioned on the absolute coordinate of input x is equipped”; Note: for each query point l’, a corresponding 3D landmark p’’ is generated based on parameters (zexp, zid) and vector-based position encoding γ, which is shown to be used in the mini-nets, ExpNet and IDNet, in Fig. 2b below). PNG media_image7.png 247 738 media_image7.png Greyscale Screenshot of Fig. 2b (taken from Zheng) It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Xiao to incorporate the teachings of Zheng to convert each query point into a vector-based position encoding for the benefit of better encapsulating the spatial relationships of the facial features. Encoding the positions makes it easier for the machine learning model to use and manipulate the data, such as for recognizing patterns related to the points on the face or capturing “rich details” (Zheng: Paragraph 2 in 2nd Col. of Page 3) and “high-frequency local variations” (Zheng: Paragraph 2 in 2nd Col. of Page 4). Additionally, a person of ordinary skill in the art before the effective filing date of the claimed invention would have recognized that the position data of the query points of Xiao could have been substituted for the vector-based position encodings of Zheng because both the position data and vector-based position encodings serve the purpose of representing points and locations on the 3D model. Furthermore, a person of ordinary skill in the art would have been able to carry out the substitution. Finally, the substitution achieves the predictable result of using the positions to generate 3D landmarks. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to substitute the position data of Xiao for the vector-based position encodings of Zheng according to known methods to yield the predictable result of using the positions to generate 3D landmarks. Regarding claim 2, Xiao in view of Zheng teaches the computer-implemented method of claim 1. Xiao further teaches training, based on the one or more losses, a second machine learning model that generates the first set of parameters to generate a second trained machine learning model (Paragraph 2-3 in 1st Col. of Page 4 – “Our model initializes the 3D fitting parameters, i.e., Q = {P,αid,αexp}, by direct regression via the 3D parameter initialization module, Fig. 2… This direct 3D parameter initialization module is trained by minimizing the discrepancy between the predicted parameters Q0 and ground truth Q∗: PNG media_image8.png 43 224 media_image8.png Greyscale ”; Note: the 3D parameter initialization module, which is equivalent to the second machine learning model, is trained based on the loss equation in the quote). Regarding claim 3, Xiao in view of Zheng teaches the computer-implemented method of claim 1. Xiao further teaches determining a second set of parameters associated with a depiction of a second face in a second image (Paragraph 4 in 2nd Col. of Page 3, Paragraph 2 in 1st Col. of Page 4 – “Fig. 2 shows the entire framework of the RDR model for large-pose 2D landmark detection. It first receives a 2D face image and passes it through a 3D parameter initialization module to directly predict parameters for accurate 3D face initialization… Using good initial parameters can improve the overall performance and accelerate the refinement process. Our model initializes the 3D fitting parameters, i.e., Q = {P, αid, αexp}, by direct regression via the 3D parameter initialization module, Fig. 2”; Note: the predicted initial parameters are equivalent to the second set of parameters); and generating, via execution of the first trained machine learning model, a second set of 3D landmarks on the second face based on the second set of parameters (Fig. 2, Paragraph 2 in 1st Col. of Page 3, Paragraph 3 in 1st Col. of Page 4, Paragraph 2 in 2nd Col. of Page 4, Paragraph 3 in 2nd Col. of page 6 – “In 3D face modeling, each 3D face is described by a set of N vertices…This network predicts the parameters upon the features from pool5 by using three independent fc5-fc6’s, each of which predicts a specific 3D parameter of αid,P and αexp respectively. Given the predicted identity parameter αid, the user-specific expression blendshapes B are obtained via Eqn. (2) and used in the following 3D face refinement process. The initial 3D face model is generated with Eqn. (1) and the initial 2D landmark locations are inferred directly via 3D-to-2D landmark projection with Eqn. (4)…We propose to detect 2D landmarks through two steps: first the projection from the 3D key vertices gives Sp and then ‘fine-tuning’ on Sp…Our trained model is thus also evaluated on AFLW-PIFA testing images and archives NME of 4.11”; Note: a 3D face model is generated based on the 3D parameters. The 3D face model is made up of vertices, which include key vertices that are equivalent to the 3D landmarks. The dual refinement module, shown in Fig. 2, is equivalent to the first machine learning model, and it is trained). Regarding claim 4, Xiao in view of Zheng teaches the computer-implemented method of claim 3. Xiao further teaches reconstructing a 3D shape of the second face based on the second set of 3D landmarks (Paragraph 1 in 2nd Col. of Page 3, Paragraph 3 in 2nd Col. of Page 4 – “Within Q, we use P = {φ,γ,θ,t3d,f} to collectively denote the rotations, translation and 3D-to-2D projection parameters…Two key components within the dual refinement module, the 3D face refinement component and the 2D landmark refinement component, utilize the extracted deep features and perform refinement on 3D face model and the projected 2D landmark locations accordingly…The PE-LSTM refines the 3D face model by predicting the necessary updates ΔPk and Δαexpk”; Note: the 3D face model, which is comprised of a 3D shape, is refined/reconstructed based on the updated 3D parameters, which include vertex landmarks of the 3D model). Regarding claim 7, Xiao in view of Zheng teaches the computer-implemented method of claim 3. Xiao further teaches projecting, based on the second set of parameters, the second set of 3D landmarks onto the second image to generate a second set of 2D landmarks (Paragraph 1 in 2nd Col. of Page 3, Paragraph 2 in 2nd Col. of Paragraph 4, Paragraph 3 in 2nd Col. of Page 6 – “Using the 3D face model to detect 2D facial landmarks needs to project 3D faces onto the 2D image plane. Under weak perspective projection [3, 29, 11], the 2D locations of all vertices can be formulated as PNG media_image2.png 31 490 media_image2.png Greyscale where Q = {P,αid,αexp} represents the parameters for 3D face fitting and 3D-to-2D projection…We propose to detect 2D landmarks through two steps: first the projection from the 3D key vertices gives Sp and then ‘fine-tuning’ on Sp gives final 2D locations S…Our trained model is thus also evaluated on AFLW-PIFA testing images and archives NME of 4.11”; Note: the 3D key vertices, which are the 3D landmarks, are projected onto the image to generate 2D landmarks. The parameters and landmarks are of a second set since the model is already trained). Regarding claim 8, Xiao in view of Zheng teaches the computer-implemented method of claim 1. Xiao further teaches wherein generating the first set of 3D landmarks comprises: converting a set of points on a canonical shape into position data (Paragraph 2 in 1st Col. of Page 3, Paragraph 1 in 2nd Col. of Page 3 – “In 3D face modeling, each 3D face is described by a set of N vertices F = [v1,...,vN] ∈ R3× N. Each vertex v has a 3D-coordinate [x,y,z]T ∈ R3… An expression blendshape, Bi ∈ R3× N, is formed by transforming a neutral 3D face to a 3D face that has a typical expression with the preserved identity. In this work, we use a neutral shape B0 and another 46 expression blendshapes with different expressions to represent a 3D face, similar to [4]. Given a rotation matrix, Rφ,γ,θ ∈ R3× 3 and a translational vector t3d ∈ R3, the 3D face reconstruction process of F is formally described as PNG media_image9.png 33 352 media_image9.png Greyscale where the rotation parameters φ,γ,θ correspond to pitch, yaw and roll rotation angles respectively… Q = {P,αid,αexp} represents the parameters for 3D face fitting and 3D-to-2D projection”; Note: the neutral 3D face is comprised of vertices, which are equivalent to a set of points on a canonical shape. The neutral 3D face is equivalent to the canonical shape. The points are represented as coordinates and transformation parameters, which are position data); and generating, via execution of the first machine learning model, the first set of 3D landmarks (Fig. 2, Paragraph 2 in 1st Col. of Page 3, Paragraph 3 in 1st Col. of Page 4, Paragraph 2 in 2nd Col. of Page 4 – “In 3D face modeling, each 3D face is described by a set of N vertices…This network predicts the parameters upon the features from pool5 by using three independent fc5-fc6’s, each of which predicts a specific 3D parameter of αid,P and αexp respectively. Given the predicted identity parameter αid, the user-specific expression blendshapes B are obtained via Eqn. (2) and used in the following 3D face refinement process. The initial 3D face model is generated with Eqn. (1) and the initial 2D landmark locations are inferred directly via 3D-to-2D landmark projection with Eqn. (4)… We propose to detect 2D landmarks through two steps: first the projection from the 3D key vertices gives Sp and then ‘fine-tuning’ on Sp”; Note: a 3D face model is generated and is made up of vertices, which include key vertices that are equivalent to the 3D landmarks. The dual refinement module, shown in Fig. 2, is equivalent to the first machine learning model) based on (i) a set of features associated with the first image (Paragraph 2 in 1st Col of Page 3 – “the expression blendshapes B of a specific person can be described as PNG media_image10.png 34 173 media_image10.png Greyscale ”; Note: the expression blendshape of the person in the image is equivalent to the features), (ii) the first set of parameters (Paragraph 1 in 2nd Col. of Page 3 – “Q = {P,αid,αexp} represents the parameters for 3D face fitting and 3D-to-2D projection”), and (iii) the set of position data (Paragraph 2 in 1st Col of Page 3 – “In 3D face modeling, each 3D face is described by a set of N vertices F = [v1,...,vN] ∈ R3× N. Each vertex v has a 3D-coordinate [x,y,z]T ∈ R3… Given a rotation matrix, Rφ,γ,θ ∈ R3× 3 and a translational vector t3d ∈ R3, the 3D face reconstruction process of F is formally described”; Note: the vertex coordinates and the transformations are equivalent to the position data). Xiao does not teach the “position encodings” in the limitation: “converting a set of points on a canonical shape into a set of position encodings and generating, via execution of the first machine learning model, the first set of 3D landmarks based on (i) a set of features associated with the first image, (ii) the first set of parameters, and (iii) the set of position encodings”. However, Zheng teaches converting a set of points on a canonical shape into a set of position encodings (Fig. 2b, Paragraph 2 in 2nd Col. of Page 3, Paragraph 3 in 2nd Col. of Page 3, Paragraph 1-2 in 1st Col. of Page 4, Paragraph 2 in 2nd Col. of Page 4 – “All the fields above are implemented by a shared Mini Nets architecture, where the entire facial deformation or geometry is further decomposed into a number of semantically meaningful parts and encoded by a set of local field functions, so that rich details can be sufficiently captured…The facial deformations incurred by expressions are represented by ExpNet E, which learns an observation-to-canonical warping for every face scan: PNG media_image5.png 31 265 media_image5.png Greyscale where l ∈ Rk×3 denotes k 3D landmarks on an observed face generated by a Landmark-Net η : (zexp,zid) → l … To model shape morphs among individuals, IDNet I further warps the canonical space to a template shape space shared by all faces: PNG media_image6.png 32 356 media_image6.png Greyscale where l′ ∈ Rk×3 denotes k landmarks on the canonical face generated by another Landmark-Net conditioned only on the identity embedding η′ : zid → l′, and p′′ is the deformed point in the template space…five landmarks located at the outer eye corners, mouth corners, and nose tip are utilized to describe the local regions (ln ∈ R3)5n=1, and each region is assigned a tiny MLP with sinusoidal activations [49] to generate the local field, denoted as ψn. To capture high-frequency local variations, we leverage sinusoidal positional encoding γ [36] on the coordinate x − ln. At the end of a Mini-Nets block, a lightweight Fusion Network conditioned on the absolute coordinate of input x is equipped”; Note: the points x from the canonical face are converted into vector-based position encodings γ). A person of ordinary skill in the art before the effective filing date of the claimed invention would have recognized that the position data of Xiao could have been substituted for the position encodings of Zheng because both the position data and position encodings serve the purpose of representing points and locations on the 3D model. Furthermore, a person of ordinary skill in the art would have been able to carry out the substitution. Finally, the substitution achieves the predictable result of using the positions to generate 3D landmarks. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to substitute the position data of Xiao for the position encodings of Zheng according to known methods to yield the predictable result of using the positions to generate 3D landmarks. Additionally, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Xiao to incorporate the teachings of Zheng to encode the position data for the benefit of ease of use and manipulation of the position data by the machine learning model and “To capture high-frequency local variations” (Zheng: Paragraph 2 in 2nd Col. of Page 4). Regarding claim 9, Xiao in view of Zheng teaches the computer-implemented method of claim 1. Xiao further teaches wherein projecting the first set of 3D landmarks onto the first image comprises: transforming the first set of 3D landmarks into a second set of 3D landmarks based on a head pose included in the first set of parameters (Paragraph 3 in 1st Col. of Page 2, Paragraph 2 in 1st Col. of Page 3, Paragraph 1 in 2nd Col. of Page 3, Paragraph 3 in 2nd Col. of Page 4 – “RDR is designed to perform 3D faces fitting based on the dynamic expression model [23, 15, 4]. Thus it can also be used directly for virtual avatar manipulation [15, 4] by simply being fed the related 3D parameters, i.e., coefficients of expression blendshapes, head poses and camera projection…In 3D face modeling, each 3D face is described by a set of N vertices F…Within Q, we use P = {φ,γ,θ,t3d,f} to collectively denote the rotations, translation and 3D-to-2D projection parameters…The dual refinement module, illustrated in Fig. 2, takes deep shape-indexed features [13] extracted from deconv5 around the predicted 2D landmark locations as input. Two key components within the dual refinement module, the 3D face refinement component and the 2D landmark refinement component, utilize the extracted deep features and perform refinement on 3D face model and the projected 2D landmark locations accordingly… The PE-LSTM refines the 3D face model by predicting the necessary updates ΔPk and Δαexpk”; Note: the 3D face model vertices/landmarks are updated/transformed based on the parameters P, which include transformation parameters and head pose. The transformed vertex landmarks on the 3D model are equivalent to the second set of 3D landmarks); and projecting the second set of 3D landmarks based on a focal length included in the first set of parameters to generate the first set of 2D landmarks (Paragraph 1 in 2nd Col. of Page 3, Paragraph 2 in 2nd Col. of Page 4, Paragraph 1 in 1st Col. of Page 5 – “Using the 3D face model to detect 2D facial landmarks needs to project 3D faces onto the 2D image plane…Within Q, we use P = {φ,γ,θ,t3d,f} to collectively denote the rotations, translation and 3D-to-2D projection parameters… We propose to detect 2D landmarks through two steps: first the projection from the 3D key vertices gives Sp and then “fine-tuning” on Sp gives final 2D locations S…deep features extracted from hidden layers of the network are sufficiently informative for estimating head pose, focal length and expression related parameters”; Note: 3D key vertices/landmarks are projected to generated 2D landmarks based on projection parameters. Parameters include focal length. It would have been obvious to one of ordinary skill in the art to use the focal length parameter to project the 3D landmarks because focal length helps determine the distance and perspective for the projection, which would lead to a more accurate 3D-2D mapping). Regarding claim 10, Xiao in view of Zheng teaches the computer-implemented method of claim 1. Xiao further teaches wherein the first set of 3D landmarks comprises a set of offsets from a canonical shape (Paragraph 2 in 1st Col. of Page 3 – “In 3D face modeling, each 3D face is described by a set of N vertices F…An expression blendshape, Bi ∈ R3× N, is formed by transforming a neutral 3D face to a 3D face that has a typical expression with the preserved identity. In this work, we use a neutral shape B0 and another 46 expression blendshapes with different expressions to represent a 3D face, similar to [4]. Given a rotation matrix, Rφ,γ,θ ∈ R3× 3 and a translational vector t3d ∈ R3, the 3D face reconstruction process of F is formally described as PNG media_image9.png 33 352 media_image9.png Greyscale where the rotation parameters φ,γ,θ correspond to pitch, yaw and roll rotation angles respectively”; Note: the 3D landmarks of the 3D face mesh are comprised of transformed vertices, which are equivalent to a set of offsets from canonical shape. The neutral 3D face is equivalent to the canonical shape). Claims 5, 11, 13-15, and 19-20 are rejected under 35 U.S.C. 103 as being unpatentable over Xiao in view of Zheng and Guler et al. (US 20210241522 A1), hereinafter Guler. Regarding claim 5, Xiao in view of Zheng teaches the computer-implemented method of claim 3. Xiao does not teach determining, based on the second set of 3D landmarks and the second set of parameters, a set of visibilities of the second set of 3D landmarks within the second image. However, Guler teaches determining, based on a set of 3D landmarks and a set of parameters, a set of visibilities of the set of 3D landmarks within an image (Paragraph 0055 – “the method may further comprise identifying object landmarks, within the predetermined set of object landmarks, which are not visible in the two-dimensional image. Object landmarks, which are to be expected to be present due to the identified object type, may not be present in the image. For example, if the object type is a human, parts of the human body may not be visible due to posture or other items obstructing the view”; Note: visible landmarks are determined based on a set of landmarks and parameters, such as human posture. The 3D landmarks were previously taught by Xiao). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Xiao to incorporate the teachings of Guler to determine a set of visibilities of 3D landmarks based on the landmarks and parameters for the benefit of being able to remove unnecessary landmarks and to only store and display the visible ones, which would help increase accuracy when outputting the landmarks. Regarding claim 11, Xiao teaches a process (Paragraph 4 in 2nd Col. of Page 3 – “Fig. 2 shows the entire framework of the RDR model for large-pose 2D landmark detection”) with the steps of: determining a first set of parameters associated with a depiction of a first face in a first image (Paragraph 4 in 2nd Col. of Page 3, Paragraph 2 in 1st Col. of Page 4 – “Fig. 2 shows the entire framework of the RDR model for large-pose 2D landmark detection. It first receives a 2D face image and passes it through a 3D parameter initialization module to directly predict parameters for accurate 3D face initialization… Using good initial parameters can improve the overall performance and accelerate the refinement process. Our model initializes the 3D fitting parameters, i.e., Q = {P, αid, αexp}, by direct regression via the 3D parameter initialization module, Fig. 2”; Note: the predicted initial parameters are equivalent to the first set of parameters); obtaining a set of query points (set of vertices on the neutral 3D face [Wingdings font/0xE0] see quote below) defined on a canonical shape (neutral 3D face [Wingdings font/0xE0] see quote) (Fig. 2, Paragraph 1-2 in 1st Col. of Page 3 – “In the 3D face fitting process within our model, the Dynamic Expression Model (DEM) [23, 4] is employed…In 3D face modeling, each 3D face is described by a set of N vertices F…An expression blendshape, Bi ∈ R3× N, is formed by transforming a neutral 3D face to a 3D face that has a typical expression with the preserved identity. In this work, we use a neutral shape B0 and another 46 expression blendshapes with different expressions to represent a 3D face”; Note: The neutral 3D face is equivalent to the canonical shape. The set of vertices on the neutral 3D face are the query points); generating, via execution of a prediction network included in the first machine learning model (duel refinement module [Wingdings font/0xE0] see quote below), a first set of three-dimensional (3D) landmarks (3D key vertices [Wingdings font/0xE0] see quote below) on the first face based on the first set of parameters (Fig. 2, Paragraph 2 in 1st Col. of Page 3, Paragraph 3 in 1st Col. of Page 4, Paragraph 2 in 2nd Col. of Page 4 – “In 3D face modeling, each 3D face is described by a set of N vertices…This network predicts the parameters upon the features from pool5 by using three independent fc5-fc6’s, each of which predicts a specific 3D parameter of αid,P and αexp respectively. Given the predicted identity parameter αid, the user-specific expression blendshapes B are obtained via Eqn. (2) and used in the following 3D face refinement process. The initial 3D face model is generated with Eqn. (1) and the initial 2D landmark locations are inferred directly via 3D-to-2D landmark projection with Eqn. (4)… We propose to detect 2D landmarks through two steps: first the projection from the 3D key vertices gives Sp and then ‘fine-tuning’ on Sp”; Note: a 3D face model is generated based on the 3D parameters. The 3D face model is made up of vertices, which include key vertices that are equivalent to the 3D landmarks. The dual refinement module, shown in Fig. 2, is equivalent to the first machine learning model and is implied to include a prediction network since it predicts landmarks), wherein, for each query point in the set of query points, a corresponding 3D landmark is generated based on (i) the first set of parameters and (ii) position data corresponding to the query point (Fig. 2, Paragraph 2 in 1st Col. of Page 3, Paragraph 3 in 1st Col. of Page 4, Paragraph 2 in 2nd Col. of Page 4 – “In 3D face modeling, each 3D face is described by a set of N vertices F = [v1, . . . , vN] ∈ R3×N…An expression blendshape, Bi ∈ R3× N, is formed by transforming a neutral 3D face to a 3D face that has a typical expression with the preserved identity …Given a rotation matrix, Rφ,γ,θ ∈ R3×3 and a translational vector t3d ∈ R3, the 3D face reconstruction process of F is formally described as PNG media_image1.png 38 346 media_image1.png Greyscale …This network predicts the parameters upon the features from pool5 by using three independent fc5-fc6’s, each of which predicts a specific 3D parameter of αid,P and αexp respectively. Given the predicted identity parameter αid, the user-specific expression blendshapes B are obtained via Eqn. (2) and used in the following 3D face refinement process. The initial 3D face model is generated with Eqn. (1) and the initial 2D landmark locations are inferred directly via 3D-to-2D landmark projection with Eqn. (4)… We propose to detect 2D landmarks through two steps: first the projection from the 3D key vertices gives Sp and then ‘fine-tuning’ on Sp”; Note: for every vertex on the neutral 3D face (query point), a corresponding vertex on the 3D face is generated (3D landmark). The 3D landmarks are based on parameters, αid and αexp, and position data of the vertices from a neutral 3D face (query points)); projecting, based on the first set of parameters, the first set of 3D landmarks onto the first image to generate a first set of two-dimensional (2D) landmarks (Paragraph 1 in 2nd Col. of Page 3, Paragraph 2 in 2nd Col. of Paragraph 4 – “Using the 3D face model to detect 2D facial landmarks needs to project 3D faces onto the 2D image plane. Under weak perspective projection [3, 29, 11], the 2D locations of all vertices can be formulated as PNG media_image2.png 31 490 media_image2.png Greyscale where Q = {P,αid,αexp} represents the parameters for 3D face fitting and 3D-to-2D projection…We propose to detect 2D landmarks through two steps: first the projection from the 3D key vertices gives Sp and then ‘fine-tuning’ on Sp gives final 2D locations S”; Note: the 3D key vertices, which are the 3D landmarks, are projected onto the image to generate 2D landmarks); and training the first machine learning model based on one or more losses associated with the first set of 2D landmarks to generate a first trained machine learning model (Fig. 2, Paragraph 3 in 2nd Col. of Page 5 – “The overall loss for dual 3D-2D refinement accumulated through K recurrent iterations is then formally described as PNG media_image3.png 115 491 media_image3.png Greyscale … Here β and λ are trade-off parameters to ensure that both 3D vertices and 2D landmarks are close to the ground-truth landmark locations”; Note: the dual refinement module, shown in Fig. 2, is equivalent to the first machine learning model. It is trained with the loss equation shown in the quote). Xiao does not teach converting, via execution of a position encoder included in a first machine learning model, each query point in the set of query points into a respective vector-based position encoding; nor “the respective vector-based position encoding” in the limitation: “wherein, for each query point in the set of query points, a corresponding 3D landmark is generated based on (i) the first set of parameters and (ii) the respective vector-based position encoding corresponding to the query point”. However, Zheng teaches converting, via execution of a position encoder included in a first machine learning model, each query point in the set of query points into a respective vector-based position encoding (Fig. 2b, Paragraph 2 in 2nd Col. of Page 3, Paragraph 3 in 2nd Col. of Page 3, Paragraph 1-2 in 1st Col. of Page 4, Paragraph 2 in 2nd Col. of Page 4 – “All the fields above are implemented by a shared Mini Nets architecture, where the entire facial deformation or geometry is further decomposed into a number of semantically meaningful parts and encoded by a set of local field functions, so that rich details can be sufficiently captured…The facial deformations incurred by expressions are represented by ExpNet E, which learns an observation-to-canonical warping for every face scan: PNG media_image5.png 31 265 media_image5.png Greyscale where l ∈ Rk×3 denotes k 3D landmarks on an observed face generated by a Landmark-Net η : (zexp,zid) → l … To model shape morphs among individuals, IDNet I further warps the canonical space to a template shape space shared by all faces: PNG media_image6.png 32 356 media_image6.png Greyscale where l′ ∈ Rk×3 denotes k landmarks on the canonical face generated by another Landmark-Net conditioned only on the identity embedding η′ : zid → l′, and p′′ is the deformed point in the template space…five landmarks located at the outer eye corners, mouth corners, and nose tip are utilized to describe the local regions (ln ∈ R3)5n=1, and each region is assigned a tiny MLP with sinusoidal activations [49] to generate the local field, denoted as ψn. To capture high-frequency local variations, we leverage sinusoidal positional encoding γ [36] on the coordinate x − ln. At the end of a Mini-Nets block, a lightweight Fusion Network conditioned on the absolute coordinate of input x is equipped”; Note: the query points x are converted into vector-based position encodings γ. Execution of a position encoder is implied because the position encodings would not be generated otherwise. The model shown in Fig. 2b above is equivalent to the first machine learning model in this case); and wherein, for each query point in the set of query points, a corresponding 3D landmark is generated based on (i) the first set of parameters and (ii) the respective vector-based position encoding corresponding to the query point (Fig. 2b, Paragraph 3 in 2nd Col. of Page 3, Paragraph 1-2 in 1st Col. of Page 4, Paragraph 2 in 2nd Col. of Page 4 – “The facial deformations incurred by expressions are represented by ExpNet E, which learns an observation-to-canonical warping for every face scan: PNG media_image5.png 31 265 media_image5.png Greyscale where l ∈ Rk×3 denotes k 3D landmarks on an observed face generated by a Landmark-Net η : (zexp,zid) → l … To model shape morphs among individuals, IDNet I further warps the canonical space to a template shape space shared by all faces: PNG media_image6.png 32 356 media_image6.png Greyscale where l′ ∈ Rk×3 denotes k landmarks on the canonical face generated by another Landmark-Net conditioned only on the identity embedding η′ : zid → l′, and p′′ is the deformed point in the template space…To capture high-frequency local variations, we leverage sinusoidal positional encoding γ [36] on the coordinate x − ln. At the end of a Mini-Nets block, a lightweight Fusion Network conditioned on the absolute coordinate of input x is equipped”; Note: for each query point l’, a corresponding 3D landmark p’’ is generated based on parameters (zexp, zid) and vector-based position encoding γ, which is shown to be used in the mini-nets, ExpNet and IDNet, in Fig. 2b above). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Xiao to incorporate the teachings of Zheng to convert each query point into a vector-based position encoding for the benefit of better encapsulating the spatial relationships of the facial features. Encoding the positions makes it easier for the machine learning model to use and manipulate the data, such as for recognizing patterns related to the points on the face or capturing “rich details” (Zheng: Paragraph 2 in 2nd Col. of Page 3) and “high-frequency local variations” (Zheng: Paragraph 2 in 2nd Col. of Page 4). Additionally, a person of ordinary skill in the art before the effective filing date of the claimed invention would have recognized that the position data of the query points of Xiao could have been substituted for the vector-based position encodings of Zheng because both the position data and vector-based position encodings serve the purpose of representing points and locations on the 3D model. Furthermore, a person of ordinary skill in the art would have been able to carry out the substitution. Finally, the substitution achieves the predictable result of using the positions to generate 3D landmarks. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to substitute the position data of Xiao for the vector-based position encodings of Zheng according to known methods to yield the predictable result of using the positions to generate 3D landmarks. Xiao modified by Zheng does not teach one or more non-transitory computer-readable media storing instructions that, when executed by one or more processors, cause the one or more processors to perform the steps. However, Guler teaches one or more non-transitory computer-readable media storing instructions that, when executed by one or more processors, cause the one or more processors to perform the steps (Paragraph 0103 – “the processing apparatus 702 is coupled to the memory 704 and is operable to read/write data to/from the memory 704. The memory 704 may comprise a single memory unit or a plurality of memory units, upon which the computer readable instructions (or code) is stored… Examples of non-volatile memory include ROM, PROM, EEPROM, flash memory, optical storage, magnetic storage”; Note: optical and magnetic storage are examples of non-transitory computer-readable media storage). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Xiao to incorporate the teachings of Guler to have non-transitory computer readable media storing instructions to be executed by a processor for the benefit of having a reliable and persistent storage so that the process can be performed again in the future. Regarding claim 13, Xiao in view of Zheng and Guler teaches the one or more non-transitory computer-readable media of claim 11. Xiao further teaches wherein the instructions further cause the one or more processors to perform the steps of: generating, via execution of a second machine learning model, a set of points on a canonical shape, wherein the set of points corresponds to displacements of a set of query points associated with the first set of 3D landmarks (Fig. 2, Paragraph 1-2 in 1st Col. of Page 3 – “In the 3D face fitting process within our model, the Dynamic Expression Model (DEM) [23, 4] is employed…In 3D face modeling, each 3D face is described by a set of N vertices F…An expression blendshape, Bi ∈ R3× N, is formed by transforming a neutral 3D face to a 3D face that has a typical expression with the preserved identity. In this work, we use a neutral shape B0 and another 46 expression blendshapes with different expressions to represent a 3D face, similar to [4]. Given a rotation matrix, Rφ,γ,θ ∈ R3× 3 and a translational vector t3d ∈ R3, the 3D face reconstruction process of F is formally described as PNG media_image9.png 33 352 media_image9.png Greyscale where the rotation parameters φ,γ,θ correspond to pitch, yaw and roll rotation angles respectively”; Note: the Dynamic Expression Model is equivalent to the second machine learning model. The set of vertices on the neutral 3D face is equivalent to a set of points on a canonical shape. The neutral 3D face is equivalent to the canonical shape. The points are transformed/displaced from initial vertex positions, which are the query points since they are related to landmarks of the face; see facial landmark points in Fig. 2); further generating the first set of 3D landmarks based on the set of points (Fig. 2, Paragraph 2 in 1st Col. of Page 3, Paragraph 3 in 1st Col. of Page 4, Paragraph 2 in 2nd Col. of Page 4 – “In 3D face modeling, each 3D face is described by a set of N vertices…This network predicts the parameters upon the features from pool5 by using three independent fc5-fc6’s, each of which predicts a specific 3D parameter of αid,P and αexp respectively. Given the predicted identity parameter αid, the user-specific expression blendshapes B are obtained via Eqn. (2) and used in the following 3D face refinement process. The initial 3D face model is generated with Eqn. (1) and the initial 2D landmark locations are inferred directly via 3D-to-2D landmark projection with Eqn. (4)… We propose to detect 2D landmarks through two steps: first the projection from the 3D key vertices gives Sp and then ‘fine-tuning’ on Sp”; Note: the 3D face model is generated and made up of vertices/points, which include key vertices that are equivalent to the 3D landmarks); and training the second machine learning model based on the one or more losses (Fig. 2, Paragraph 1 in 1st Col. of Page 3, Paragraph 2 in 1st Col. of Page 5, Paragraph 2 in 2nd Col. of Page 5 – “In the 3D face fitting process within our model, the Dynamic Expression Model (DEM) [23, 4] is employed… we minimize PNG media_image11.png 36 239 media_image11.png Greyscale for all losses related to the expression parameters, where αi is the estimated coefficient for the expression blendshape Bi. In the 3D face model refinement step, the updated expression coefficients at the k-th stage αkexp are obtained… The difference between predicted 3D fitting parameters P, αexp and their ground truth is also considered in training the 3D refinement component. The overall loss becomes PNG media_image12.png 40 489 media_image12.png Greyscale ”; Note: a loss is used to train the dynamic expression model for updating the expression parameters in 3D fitting). Regarding claim 14, Xiao in view of Zheng and Guler teaches the one or more non-transitory computer-readable media of claim 11. Xiao further teaches wherein the instructions further cause the one or more processors to perform the steps of: determining a second set of parameters associated with a depiction of a second face in a second image (Paragraph 4 in 2nd Col. of Page 3, Paragraph 2 in 1st Col. of Page 4 – “Fig. 2 shows the entire framework of the RDR model for large-pose 2D landmark detection. It first receives a 2D face image and passes it through a 3D parameter initialization module to directly predict parameters for accurate 3D face initialization… Using good initial parameters can improve the overall performance and accelerate the refinement process. Our model initializes the 3D fitting parameters, i.e., Q = {P, αid, αexp}, by direct regression via the 3D parameter initialization module, Fig. 2”; Note: the predicted initial parameters are equivalent to the second set of parameters); and generating, via execution of the first trained machine learning model, a second set of 3D landmarks on the second face based on the second set of parameters (Fig. 2, Paragraph 2 in 1st Col. of Page 3, Paragraph 3 in 1st Col. of Page 4, Paragraph 2 in 2nd Col. of Page 4, Paragraph 3 in 2nd Col. of page 6 – “In 3D face modeling, each 3D face is described by a set of N vertices…This network predicts the parameters upon the features from pool5 by using three independent fc5-fc6’s, each of which predicts a specific 3D parameter of αid,P and αexp respectively. Given the predicted identity parameter αid, the user-specific expression blendshapes B are obtained via Eqn. (2) and used in the following 3D face refinement process. The initial 3D face model is generated with Eqn. (1) and the initial 2D landmark locations are inferred directly via 3D-to-2D landmark projection with Eqn. (4)…We propose to detect 2D landmarks through two steps: first the projection from the 3D key vertices gives Sp and then ‘fine-tuning’ on Sp…Our trained model is thus also evaluated on AFLW-PIFA testing images and archives NME of 4.11”; Note: a 3D face model is generated based on the 3D parameters. The 3D face model is made up of vertices, which include key vertices that are equivalent to the 3D landmarks. The dual refinement module, shown in Fig. 2, is equivalent to the first machine learning model, and it is trained). Regarding claim 15, Xiao in view of Zheng and Guler teaches the one or more non-transitory computer-readable media of claim 14. Xiao further teaches wherein the instructions further cause the one or more processors to perform the step of reconstructing a 3D shape of the second face based on the second set of 3D landmarks (Paragraph 1 in 2nd Col. of Page 3, Paragraph 3 in 2nd Col. of Page 4 – “Within Q, we use P = {φ,γ,θ,t3d,f} to collectively denote the rotations, translation and 3D-to-2D projection parameters…Two key components within the dual refinement module, the 3D face refinement component and the 2D landmark refinement component, utilize the extracted deep features and perform refinement on 3D face model and the projected 2D landmark locations accordingly…The PE-LSTM refines the 3D face model by predicting the necessary updates ΔPk and Δαexpk”; Note: the 3D face model, which is comprised of a 3D shape, is refined/reconstructed based on the updated 3D parameters, which include vertex landmarks of the 3D model). Regarding claim 19, Xiao in view of Zheng and Guler teaches the one or more non-transitory computer-readable media of claim 11. Xiao further teaches wherein the first set of parameters comprises at least one of a camera parameter or a pose of a head associated with the first face (Paragraph 3 in 1st Col. of Page 2 – “RDR is designed to perform 3D faces fitting based on the dynamic expression model [23, 15, 4]. Thus it can also be used directly for virtual avatar manipulation [15, 4] by simply being fed the related 3D parameters, i.e., coefficients of expression blendshapes, head poses and camera projection”). Regarding claim 20, Xiao teaches a system (Paragraph 3 in 1st Col. of Paragraph 8 – “We test our model on GeForce GTX Titan GPU and Intel i7-4930K CPU”; Note: it is implied there is a computer system to test the model), comprising: one or more processors configured to perform the steps of (Paragraph 3 in 1st Col. of Paragraph 8 – “We test our model on GeForce GTX Titan GPU and Intel i7-4930K CPU”): determining a first machine learning model, wherein the first machine learning model is trained based on one or more losses (Fig. 2, Paragraph 3 in 2nd Col. of Page 5 – “The overall loss for dual 3D-2D refinement accumulated through K recurrent iterations is then formally described as PNG media_image3.png 115 491 media_image3.png Greyscale … Here β and λ are trade-off parameters to ensure that both 3D vertices and 2D landmarks are close to the ground-truth landmark locations”; Note: the dual refinement module, shown in Fig. 2, is equivalent to the first machine learning model. It is trained with the loss equation shown in the quote) associated with a projection of a first set of three-dimensional (3D) landmarks generated by the first machine learning model onto a two-dimensional (2D) space (Paragraph 1 in 2nd Col. of Page 3, Paragraph 2 in 2nd Col. of Paragraph 4 – “Using the 3D face model to detect 2D facial landmarks needs to project 3D faces onto the 2D image plane. Under weak perspective projection [3, 29, 11], the 2D locations of all vertices can be formulated as PNG media_image2.png 31 490 media_image2.png Greyscale where Q = {P,αid,αexp} represents the parameters for 3D face fitting and 3D-to-2D projection…we propose the dual refinement module. Concretely, we introduce a compensation term ΔS that accommodates future ‘fine-tuning’ on the 2D landmarks locations obtained via 3D-to-2D projection…We propose to detect 2D landmarks through two steps: first the projection from the 3D key vertices gives Sp and then ‘fine-tuning’ on Sp gives final 2D locations S”; Note: the 3D key vertices, which are the 3D landmarks, are projected onto the image to generate 2D landmarks); determining a set of parameters associated with a depiction of a face in an image (Paragraph 4 in 2nd Col. of Page 3, Paragraph 2 in 1st Col. of Page 4 – “Fig. 2 shows the entire framework of the RDR model for large-pose 2D landmark detection. It first receives a 2D face image and passes it through a 3D parameter initialization module to directly predict parameters for accurate 3D face initialization… Using good initial parameters can improve the overall performance and accelerate the refinement process. Our model initializes the 3D fitting parameters, i.e., Q = {P, αid, αexp}, by direct regression via the 3D parameter initialization module, Fig. 2”; Note: the predicted initial parameters are equivalent to the first set of parameters); obtaining a set of query points (set of vertices on the neutral 3D face [Wingdings font/0xE0] see quote below) defined on a canonical shape (neutral 3D face [Wingdings font/0xE0] see quote) (Fig. 2, Paragraph 1-2 in 1st Col. of Page 3 – “In the 3D face fitting process within our model, the Dynamic Expression Model (DEM) [23, 4] is employed…In 3D face modeling, each 3D face is described by a set of N vertices F…An expression blendshape, Bi ∈ R3× N, is formed by transforming a neutral 3D face to a 3D face that has a typical expression with the preserved identity. In this work, we use a neutral shape B0 and another 46 expression blendshapes with different expressions to represent a 3D face”; Note: The neutral 3D face is equivalent to the canonical shape. The set of vertices on the neutral 3D face are the query points); generating, via execution of a prediction network included in the first machine learning model, a second set of three-dimensional (3D) landmarks (3D key vertices [Wingdings font/0xE0] see quote below) on the face based on the set of parameters (Fig. 2, Paragraph 2 in 1st Col. of Page 3, Paragraph 3 in 1st Col. of Page 4, Paragraph 2 in 2nd Col. of Page 4, Paragraph 3 in 2nd Col. of page 6 – “In 3D face modeling, each 3D face is described by a set of N vertices…This network predicts the parameters upon the features from pool5 by using three independent fc5-fc6’s, each of which predicts a specific 3D parameter of αid,P and αexp respectively. Given the predicted identity parameter αid, the user-specific expression blendshapes B are obtained via Eqn. (2) and used in the following 3D face refinement process. The initial 3D face model is generated with Eqn. (1) and the initial 2D landmark locations are inferred directly via 3D-to-2D landmark projection with Eqn. (4)…We propose to detect 2D landmarks through two steps: first the projection from the 3D key vertices gives Sp and then ‘fine-tuning’ on Sp…Our trained model is thus also evaluated on AFLW-PIFA testing images and archives NME of 4.11”; Note: a 3D face model is generated based on the 3D parameters. The 3D face model is made up of vertices, which include key vertices that are equivalent to the 3D landmarks. The dual refinement module, shown in Fig. 2, is equivalent to the first machine learning model, and it is trained. It is implied to include a prediction network since it predicts landmarks. The second set of landmarks are generated when executing the model after training is completed), wherein, for each query point in the set of query points, a corresponding 3D landmark is generated based on (i) the set of parameters and (ii) position data corresponding to the query point (Fig. 2, Paragraph 2 in 1st Col. of Page 3, Paragraph 3 in 1st Col. of Page 4, Paragraph 2 in 2nd Col. of Page 4 – “In 3D face modeling, each 3D face is described by a set of N vertices F = [v1, . . . , vN] ∈ R3×N…An expression blendshape, Bi ∈ R3× N, is formed by transforming a neutral 3D face to a 3D face that has a typical expression with the preserved identity …Given a rotation matrix, Rφ,γ,θ ∈ R3×3 and a translational vector t3d ∈ R3, the 3D face reconstruction process of F is formally described as PNG media_image1.png 38 346 media_image1.png Greyscale …This network predicts the parameters upon the features from pool5 by using three independent fc5-fc6’s, each of which predicts a specific 3D parameter of αid,P and αexp respectively. Given the predicted identity parameter αid, the user-specific expression blendshapes B are obtained via Eqn. (2) and used in the following 3D face refinement process. The initial 3D face model is generated with Eqn. (1) and the initial 2D landmark locations are inferred directly via 3D-to-2D landmark projection with Eqn. (4)… We propose to detect 2D landmarks through two steps: first the projection from the 3D key vertices gives Sp and then ‘fine-tuning’ on Sp”; Note: for every vertex on the neutral 3D face (query point), a corresponding vertex on the 3D face is generated (3D landmark). The 3D landmarks are based on parameters, αid and αexp, and position data of the vertices from a neutral 3D face (query points)); and reconstructing a 3D shape of the face based on the second set of 3D landmarks (Paragraph 1 in 2nd Col. of Page 3, Paragraph 3 in 2nd Col. of Page 4 – “Within Q, we use P = {φ,γ,θ,t3d,f} to collectively denote the rotations, translation and 3D-to-2D projection parameters…Two key components within the dual refinement module, the 3D face refinement component and the 2D landmark refinement component, utilize the extracted deep features and perform refinement on 3D face model and the projected 2D landmark locations accordingly…The PE-LSTM refines the 3D face model by predicting the necessary updates ΔPk and Δαexpk”; Note: the 3D face model, which is comprised of a 3D shape, is refined/reconstructed based on the updated 3D parameters, which include vertex landmarks of the 3D model). Xiao does not teach converting, via execution of a position encoder included in the first machine learning model, each query point in the set of query points into a respective vector-based position encoding; nor “the respective vector-based position encoding” in the limitation: “wherein, for each query point in the set of query points, a corresponding 3D landmark is generated based on (i) the set of parameters and (ii) the respective vector-based position encoding corresponding to the query point”. However, Zheng teaches converting, via execution of a position encoder included in the first machine learning model, each query point in the set of query points into a respective vector-based position encoding (Fig. 2b, Paragraph 2 in 2nd Col. of Page 3, Paragraph 3 in 2nd Col. of Page 3, Paragraph 1-2 in 1st Col. of Page 4, Paragraph 2 in 2nd Col. of Page 4 – “All the fields above are implemented by a shared Mini Nets architecture, where the entire facial deformation or geometry is further decomposed into a number of semantically meaningful parts and encoded by a set of local field functions, so that rich details can be sufficiently captured…The facial deformations incurred by expressions are represented by ExpNet E, which learns an observation-to-canonical warping for every face scan: PNG media_image5.png 31 265 media_image5.png Greyscale where l ∈ Rk×3 denotes k 3D landmarks on an observed face generated by a Landmark-Net η : (zexp,zid) → l … To model shape morphs among individuals, IDNet I further warps the canonical space to a template shape space shared by all faces: PNG media_image6.png 32 356 media_image6.png Greyscale where l′ ∈ Rk×3 denotes k landmarks on the canonical face generated by another Landmark-Net conditioned only on the identity embedding η′ : zid → l′, and p′′ is the deformed point in the template space…five landmarks located at the outer eye corners, mouth corners, and nose tip are utilized to describe the local regions (ln ∈ R3)5n=1, and each region is assigned a tiny MLP with sinusoidal activations [49] to generate the local field, denoted as ψn. To capture high-frequency local variations, we leverage sinusoidal positional encoding γ [36] on the coordinate x − ln. At the end of a Mini-Nets block, a lightweight Fusion Network conditioned on the absolute coordinate of input x is equipped”; Note: the query points x are converted into vector-based position encodings γ. Execution of a position encoder is implied because the position encodings would not be generated otherwise. The model shown in Fig. 2b above is equivalent to the first machine learning model in this case); and wherein, for each query point in the set of query points, a corresponding 3D landmark is generated based on (i) the set of parameters and (ii) the respective vector-based position encoding corresponding to the query point (Fig. 2b, Paragraph 3 in 2nd Col. of Page 3, Paragraph 1-2 in 1st Col. of Page 4, Paragraph 2 in 2nd Col. of Page 4 – “The facial deformations incurred by expressions are represented by ExpNet E, which learns an observation-to-canonical warping for every face scan: PNG media_image5.png 31 265 media_image5.png Greyscale where l ∈ Rk×3 denotes k 3D landmarks on an observed face generated by a Landmark-Net η : (zexp,zid) → l … To model shape morphs among individuals, IDNet I further warps the canonical space to a template shape space shared by all faces: PNG media_image6.png 32 356 media_image6.png Greyscale where l′ ∈ Rk×3 denotes k landmarks on the canonical face generated by another Landmark-Net conditioned only on the identity embedding η′ : zid → l′, and p′′ is the deformed point in the template space…To capture high-frequency local variations, we leverage sinusoidal positional encoding γ [36] on the coordinate x − ln. At the end of a Mini-Nets block, a lightweight Fusion Network conditioned on the absolute coordinate of input x is equipped”; Note: for each query point l’, a corresponding 3D landmark p’’ is generated based on parameters (zexp, zid) and vector-based position encoding γ, which is shown to be used in the mini-nets, ExpNet and IDNet, in Fig. 2b above). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Xiao to incorporate the teachings of Zheng to convert each query point into a vector-based position encoding for the benefit of better encapsulating the spatial relationships of the facial features. Encoding the positions makes it easier for the machine learning model to use and manipulate the data, such as for recognizing patterns related to the points on the face or capturing “rich details” (Zheng: Paragraph 2 in 2nd Col. of Page 3) and “high-frequency local variations” (Zheng: Paragraph 2 in 2nd Col. of Page 4). Additionally, a person of ordinary skill in the art before the effective filing date of the claimed invention would have recognized that the position data of the query points of Xiao could have been substituted for the vector-based position encodings of Zheng because both the position data and vector-based position encodings serve the purpose of representing points and locations on the 3D model. Furthermore, a person of ordinary skill in the art would have been able to carry out the substitution. Finally, the substitution achieves the predictable result of using the positions to generate 3D landmarks. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to substitute the position data of Xiao for the vector-based position encodings of Zheng according to known methods to yield the predictable result of using the positions to generate 3D landmarks. Xiao modified by Zheng does not teach one or more memories that store instructions, and one or more processors that are coupled to the one or more memories and, when executing the instructions, are configured to perform the steps. However, Guler teaches one or more memories that store instructions, and one or more processors that are coupled to the one or more memories and, when executing the instructions, are configured to perform the steps (Paragraph 0101 – “The apparatus 700 comprises a processing apparatus 702 and memory 704 according to an exemplary embodiment. Computer-readable code 706 may be stored on the memory 704 and may, when executed by the processing apparatus 702, cause the apparatus 700 to perform methods as described herein”). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Xiao to incorporate the teachings of Guler to have memory storing instructions to be executed by a processor for the benefit of being able to test the process and perform it again in the future. Claim 6 is rejected under 35 U.S.C. 103 as being unpatentable over Xiao in view of Zheng and He et al. (Super-Resolution of Faces Using Texture Mapping on a Generic 3D Model), hereinafter He. Regarding claim 6, Xiao in view of Zheng teaches the computer-implemented method of claim 3. Xiao further teaches a mesh that is generated based on the second set of 3D landmarks (Fig. 2 Caption, Paragraph 1 in 2nd Col. of Page 3, Paragraph 3 in 2nd Col. of Page 4 – “RDR first directly predicts initial 3D face fitting parameters with the 3D parameter initialization module, generates an initial 3D face mesh…Within Q, we use P = {φ,γ,θ,t3d,f} to collectively denote the rotations, translation and 3D-to-2D projection parameters…Two key components within the dual refinement module, the 3D face refinement component and the 2D landmark refinement component, utilize the extracted deep features and perform refinement on 3D face model and the projected 2D landmark locations accordingly…The PE-LSTM refines the 3D face model by predicting the necessary updates ΔPk and Δαexpk”; Note: the 3D face mesh is refined/re-generated based on the updated 3D parameters, which include vertex landmarks in the 3D model). Xiao does not teach generating a texture for the second face based on a projection of a set of pixel values from the second image onto a mesh. However, He teaches generating a texture for the second face based on a projection of a set of pixel values from the second image onto a mesh (Paragraph 4 in 1st Col. of Page 4 – “we employ texture mapping technology instead, where detailed texture could be rendered on a 3D shape with substantially fewer vertices… In essence, texture map is a 2D image which contains all the texture of a human face and texture coordinates determine how the texture is mapped from the 2D texture map onto the final 3D mesh…Revisiting how pixels of input image were filled into texture map as described in section 2.4: Firstly, each pixel of input image was mapped into texture map with sub-pixel level. Then scattered data interpolation method was applied to get grid data”; Note: a texture is generated based on a mapping/projection of pixels of an image onto a texture map that goes onto a mesh). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Xiao to incorporate the teachings of He to generate a texture for the face based on a projection of pixel values from the image onto a mesh for the benefit of being able to create a realistic and visually appealing 3D reconstruction of the face, including color and other appearance attributes. Claim 12 is rejected under 35 U.S.C. 103 as being unpatentable over Xiao in view of Zheng, Guler, and Malafeev et al. (US 20230144458 A1), hereinafter Malafeev. Regarding claim 12, Xiao in view of Zheng and Guler teaches the one or more non-transitory computer-readable media of claim 11. Xiao does not teach wherein the instructions further cause the one or more processors to perform the steps of: applying, via execution of a second machine learning model, a transformation to a second image to generate the first image; and training the second machine learning model based on the one or more losses. However, Malafeev teaches applying, via execution of a second machine learning model, a transformation to a second image to generate the first image (Paragraph 0044 – “object detection may be performed using one or more MLMs, which may be separate from or integrated with one or more MLMs used for detecting the one or more locations of the one or more landmarks. For example, during a first step, the face 132 may be detected. Next, fiducial facial landmarks may be detected for the detected face using the object location(s). In one or more embodiments, the detected object location(s) may be used to crop the image 122, and the cropped image 122 may be analyzed to determine the locations of the landmarks”; Note: the MLM that crops the image is equivalent to the second machine learning model. The cropping is a transformation, and the cropped image is equivalent to the first image); and training the second machine learning model based on the one or more losses (Paragraph 0086, 0088 – “the training engine 404 may determine to end training using any suitable approach, such as determining the MLM(s) 106 has converged (e.g., using the loss function data 414)… The output analyzer 408 of the training engine 404 may be configured to generate the loss function data 414 from the output data 412. The output data 412 may represent one or more outputs from one or more of the MLM(s)”; Note: the machine learning model is trained based on a loss function). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Xiao to incorporate the teachings of Malafeev to transform the image for the benefit of removing irrelevant parts of the image so that the focus is on the face, which would make detecting landmarks easier. It also would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Xiao to incorporate the teachings of Malafeev to train the machine learning model using a loss for the benefit of correcting errors in the model’s predictions and increasing accuracy for later use. Claims 16-17 are rejected under 35 U.S.C. 103 as being unpatentable over Xiao in view of Zheng, Guler, and He. Regarding claim 16, Xiao in view of Zheng and Guler teaches the one or more non-transitory computer-readable media of claim 15. Xiao does not teach wherein the instructions further cause the one or more processors to perform the step of generating a texture for the second face based on a projection of a set of pixel values from the second image onto the 3D shape. However, He teaches generating a texture for the second face based on a projection of a set of pixel values from the second image onto the 3D shape (Paragraph 4 in 1st Col. of Page 4 – “we employ texture mapping technology instead, where detailed texture could be rendered on a 3D shape with substantially fewer vertices… In essence, texture map is a 2D image which contains all the texture of a human face and texture coordinates determine how the texture is mapped from the 2D texture map onto the final 3D mesh…Revisiting how pixels of input image were filled into texture map as described in section 2.4: Firstly, each pixel of input image was mapped into texture map with sub-pixel level. Then scattered data interpolation method was applied to get grid data”; Note: a texture is generated based on a mapping/projection of pixels of an image onto a texture map that goes onto a mesh, which is equivalent to the 3D shape). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Xiao to incorporate the teachings of He to generate a texture for the face based on a projection of pixel values from the image onto the 3D shape for the benefit of being able to create a realistic and visually appealing 3D reconstruction of the face, including color and other appearance attributes. Regarding claim 17, Xiao in view of Zheng and Guler teaches the one or more non-transitory computer-readable media of claim 14. Xiao further teaches a mesh that is generated based on the second set of 3D landmarks (Fig. 2 Caption, Paragraph 1 in 2nd Col. of Page 3, Paragraph 3 in 2nd Col. of Page 4 – “RDR first directly predicts initial 3D face fitting parameters with the 3D parameter initialization module, generates an initial 3D face mesh…Within Q, we use P = {φ,γ,θ,t3d,f} to collectively denote the rotations, translation and 3D-to-2D projection parameters…Two key components within the dual refinement module, the 3D face refinement component and the 2D landmark refinement component, utilize the extracted deep features and perform refinement on 3D face model and the projected 2D landmark locations accordingly…The PE-LSTM refines the 3D face model by predicting the necessary updates ΔPk and Δαexpk”; Note: the 3D face mesh is refined/re-generated based on the updated 3D parameters, which include vertex landmarks in the 3D model). Xiao does not teach generating a texture for the second face based on a projection of a set of pixel values from the second image onto a mesh. However, He teaches generating a texture for the second face based on a projection of a set of pixel values from the second image onto a mesh (Paragraph 4 in 1st Col. of Page 4 – “we employ texture mapping technology instead, where detailed texture could be rendered on a 3D shape with substantially fewer vertices… In essence, texture map is a 2D image which contains all the texture of a human face and texture coordinates determine how the texture is mapped from the 2D texture map onto the final 3D mesh…Revisiting how pixels of input image were filled into texture map as described in section 2.4: Firstly, each pixel of input image was mapped into texture map with sub-pixel level. Then scattered data interpolation method was applied to get grid data”; Note: a texture is generated based on a mapping/projection of pixels of an image onto a texture map that goes onto a mesh). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Xiao to incorporate the teachings of He to generate a texture for the face based on a projection of pixel values from the image onto a mesh for the benefit of being able to create a realistic and visually appealing 3D reconstruction of the face, including color and other appearance attributes. Claim 18 is rejected under 35 U.S.C. 103 as being unpatentable over Xiao in view of Zheng, Guler, and Marks et al. (UGLLI Face Alignment: Estimating Uncertainty with Gaussian Log-Likelihood Loss), hereinafter Marks. Regarding claim 18, Xiao in view of Zheng and Guler teaches the one or more non-transitory computer-readable media of claim 11. Xiao does not teach wherein the one or more losses comprise a Gaussian negative likelihood loss. However, Marks teaches wherein the one or more losses comprise a Gaussian negative likelihood loss (Paragraph 4 in 1st Col. of Page 5 – “for each landmark in every input image, the network outputs a Gaussian distribution (parameterized by µij and Lij). The goal of training is for the network to learn a mapping from input images to Gaussian distributions, such that the likelihood of the groundtruth landmark locations (over all landmarks and all training images) is as large as possible. Maximizing the likelihood (1) is equivalent to minimizing the negative log likelihood, so we use the negative log likelihood as our loss function. Our loss function Lij at each hourglass i for the landmark j can be expressed as the sum of two terms, T1 and T2: PNG media_image13.png 74 478 media_image13.png Greyscale ”). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Xiao to incorporate the teachings of Marks to have the loss function be a Gaussian negative likelihood loss because “Estimating uncertainty not only enables the identification of failure cases in real world scenarios, but also allows downstream tasks to be adjusted either automatically or manually based upon the estimated uncertainty. Therefore, while it is certainly important to improve the accuracy of face alignment systems, it is equally important to predict their uncertainty” (Marks: Paragraph 2 in 1st Col. of Page 3). Conclusion The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Liu et al. (Superimposition-guided Facial Reconstruction from Skull) teaches a method of detecting landmarks on portrait photos and corresponding synthesized images for facial reconstruction. Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a). A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action. Any inquiry concerning this communication or earlier communications from the examiner should be directed to MICHELLE HAU MA whose telephone number is (571)272-2187. The examiner can normally be reached M-Th 7-5:30. Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice. If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, King Poon can be reached at (571) 270-0728. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300. Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000. /MICHELLE HAU MA/ Examiner, Art Unit 2617 /KING Y POON/Supervisory Patent Examiner, Art Unit 2617