DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale or otherwise available to the public before the effective filing date of the claimed invention.
Claims 1 and 8 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Ye; Shu-Hao et al. (US 10970526 B1; hereinafter simply referred to as Ye).
Regarding independent claim 1, Ye teaches:
A computer-implemented method for performing landmark detection (See Col 3 Lines 55 – 67, and Figure 1, wherein a computer implemented method (facial image reconstruction system, ‘10’ in figure 1, which uses convolutional neural networks, necessarily implemented via a computer) performs landmark detection using the facial landmark detection model ‘110’ in figure 1).
applying, via execution of a first machine learning model, a first transformation to a first image depicting a first face to generate a second image (See Col 3, Lines 35 – 54, and Col 5 Lines 39 – 51, and Figure 1, wherein a first machine learning model (feature fusion convolutional autoencoder, ‘100’ in figure 1, comprising convolutional decoder ‘106’ in figure 1) takes an input facial image, thermal facial image, and performs a transformation/reconstruction, to obtain a second image, being the reconstructed visible facial image)
determining, via execution of a second machine learning model, a first set of landmarks on the first face based on the second image (See Col 3, Lines 35 – 54, and Col 5 Lines 39 – 51, and Figure 1 wherein a second machine learning model (facial landmark detection model, ‘110’ in figure 1) determines facial landmark models (first set of landmarks) of the first face (face in the thermal facial image) based on the second image (reconstructed visible facial image)).
and training the first machine learning model based on one or more losses associated with the first set of landmarks to generate a first trained machine learning model. (Col 5 Lines 39 – 67, Col 6 Lines 1 – 2, Col 3, Lines 35 – 54 and Figure 1, and Formulas 3 and 4 wherein the first machine learning model, being the convolutional autoencoder ‘100’, in figure 1, is trained (parameter weight adjustment through backpropagation) using the hybrid loss function, ‘111’ in figure 1, that are associated with the first set of landmarks, to generate a first trained machine learning model (being the post adjustment convolutional autoencoder)).
Regarding dependent claim 8, Ye teaches:
Determining, within the second image, a first set of positions that corresponds to the first set of landmarks; (See Col 4 Lines 19 – 67, Col 5 Lines 1 – 51, and Figure 2 wherein a first set of positions are determined corresponding to the first set of landmarks, being the characteristic vectors containing positions of the landmarks)
applying a second transformation that is an inverse of the first transformation to the first set of positions to generate a second set of positions in the first image; (See Col 6 Lines 3 – 40, Col 4 Lines 19 – 67, Figures 2, 4A and 4B, wherein a second transformation being the up-sampling is done to the first set of positions (characteristic vectors) being the inverse of the first transformation, being the down-sampling, to generate a second set of positions in the first image being the up-sampled characteristic vector)
and computing the one or more losses based on the second set of positions and a set of ground truth positions associated with the first set of landmarks. (See Col 6 Lines 3 – 40 wherein computing the one or more losses comprises one or more losses on the second set of positions (up-sampled characteristic vector to obtain reconstructed visible facial image used to obtain losses) and a set of ground truth positions associated with the first set of landmarks (real visible facial image, comprising characteristic vectors associated with first set of landmarks, used to obtain losses)).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 2, 3, and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Ye; Shu-Hao et al. (US 10970526 B1; hereinafter simply referred to as Ye) in view of Wang; Hong et al. (WO 2023061122 A1; hereinafter simply referred to as Wang; translated via Espacenet)
Regarding dependent claim 2, Ye does not explicitly disclose:
Training the second machine learning model based on the one or more losses to generate a second trained machine learning model.
However, Wang teaches of training the second machine learning model based on the one or more losses to generate a second trained machine learning model. (See ¶ 153 wherein the second machine learning model (second convolutional neural network) is trained (updated through backpropagation) using the one or more losses (contrastive loss) to generate a second trained machine learning model (updated second convolutional neural network)).
As taught by Wang having multiple trained machine learning models trained using losses allows for one machine learning model to focus on one task and another machine learning model to focus on another task. (See ¶ 141 wherein the first convolutional neural network focus on extracting facial features from overall image samples and the second convolutional neural network focuses on extracting facial features using local image samples). As both the teachings of Ye and Wang deal with the technical field of image processing regarding facial images, it would have been obvious to one or ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of Ye with Wang to teach of training the second machine learning model based on the one or more losses to generate a second trained machine learning model in order for the two machine learning models to focus on different tasks while still being update/trained using the one or more losses.
Regarding dependent claim 3, Ye in view of Wang teaches:
Applying, via execution the first trained machine learning model, a second transformation to a second face depicted in a third image to generate a fourth image; and determining, via execution of the second trained machine learning model, a second set of landmarks on the second face based on the fourth image. (See Ye Col 3, Lines 35 – 54, and Col 5 Lines 39 – 67, and Figure 1, wherein a batch of multiple thermal facial images (including a third image) are input into the first trained machine learning model (post parameter weight adjustment convolutional autoencoder) to generate a fourth image (reconstructed visible facial image), and determining a second set of landmarks (facial landmark models) using the second trained machine learning mode (trained facial landmark detection model, ‘110’ in figure 1)).
Regarding independent claim 20, Ye teaches:
Determining a first machine learning model and a second machine learning model, wherein the first machine learning model is trained based on one or more losses associated with a first set of landmarks generated by the second machine learning model; (See Col 5 Lines 39 – 67, Col 6 Lines 1 – 2, Col 3, Lines 35 – 54 and Figure 1, and Formulas 3 and 4 wherein the first machine learning model, being the convolutional autoencoder ‘100’, in figure 1, is trained (parameter weight adjustment through backpropagation) using the loss functions, ‘111 – 111C’ in figure 1, output from the second machine learning model (facial landmark detection model) that are associated with the first set of landmarks, to generate a first trained machine learning model (being the post adjustment convolutional autoencoder)).
applying, via execution of the first machine learning model, a transformation to a first image depicting a first face to generate a second image, (See Col 3, Lines 35 – 54, and Col 5 Lines 39 – 51, and Figure 1, wherein a first machine learning model (feature fusion convolutional autoencoder, ‘100’ in figure 1, comprising convolutional decoder ‘106’ in figure 1) takes an input facial image, thermal facial image, and performs a transformation/reconstruction, to obtain a second image, being the reconstructed visible facial image)
and determining, via execution of the second machine learning model, a second set of landmarks on the first face based on the second image (See Col 3, Lines 35 – 54, and Col 5 Lines 39 – 51, and Figure 1 wherein a second machine learning model (facial landmark detection model, ‘110’ in figure 1) determines facial landmark models (second set of landmarks) of the first face (face in the thermal facial image) based on the second image (reconstructed visible facial image)).
and training the first machine learning model based on one or more losses associated with the first set of landmarks to generate a first trained machine learning model. (Col 5 Lines 39 – 67, Col 6 Lines 1 – 2, Col 3, Lines 35 – 54 and Figure 1, and Formulas 3 and 4 wherein the first machine learning model, being the convolutional autoencoder ‘100’, in figure 1, is trained (parameter weight adjustment through backpropagation) using the hybrid loss function, ‘111’ in figure 1, that are associated with the first set of landmarks, to generate a first trained machine learning model (being the post adjustment convolutional autoencoder)).
Ye does not explicitly disclose a computer system, comprising: one or more memories that store instructions; and one or more processors that are coupled to the one or more memories and, when executing the instructions, are configured to perform facial landmark detection, the second machine learning model is trained based on one or more losses associated with a first set of landmarks generated by the second machine learning model from input that includes a transformed image generated via execution of the first machine learning model.
However Wang teaches of a computer system, comprising: one or more memories that store instructions; and one or more processors that are coupled to the one or more memories and, when executing the instructions, are configured to perform facial landmark detection, (See ¶ 213, 217, 35, wherein a computer system comprises one or more memories (storage mediums) that store instructions executed by a processor to perform facial landmark detection)
and the second machine learning model is trained based on one or more losses associated with a first set of landmarks generated by the second machine learning model from input that includes a transformed image generated via execution of the first machine learning model. (See ¶ 149 – 153, 190 wherein the second machine learning model (second convolutional neural network) is trained (updated through backpropagation) using the one or more losses (contrastive loss) to generate a second trained machine learning model (updated second convolutional neural network) based on a first set of landmarks, partial face features).
As taught by Wang having multiple trained machine learning models trained using losses allows for one machine learning model to focus on one task and another machine learning model to focus on another task. (See ¶ 141 wherein the first convolutional neural network focus on extracting facial features from overall image samples and the second convolutional neural network focuses on extracting facial features using local image samples). As both the teachings of Ye and Wang deal with the technical field of image processing regarding facial images, it would have been obvious to one or ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of Ye with Wang to teach of training the second machine learning model based on the one or more losses to generate a second trained machine learning model in order for the two machine learning models to focus on different tasks while still being update/trained using the one or more losses.
Claim 4 is rejected under 35 U.S.C. 103 as being unpatentable over Ye; Shu-Hao et al. (US 10970526 B1; hereinafter simply referred to as Ye) in view of Wang; Hong et al. (WO 2023061122 A1; hereinafter simply referred to as Wang; translated via Espacenet) further in view of Guler; Riza et al. (US 20240346763 A1; hereinafter simply referred to as Guler).
Regarding dependent claim 4, Ye in view of Wang does not explicitly disclose:
Inputting, into the second trained machine learning model, (i) a set of points on a canonical shape and (ii) the fourth image; and generating, by the second trained machine learning model, the second set of landmarks as a set of positions of the set of points within the fourth image.
However, Guler teaches of inputting, into the second trained machine learning model, (i) a set of points on a canonical shape and (ii) the fourth image; (See ¶ 126 – 130, figure 5 and 8, wherein the second trained machine learning model (third and fourth machine learning models as part of the head-specific canonical reconstruction network, ‘530’ in figure 5) receives input of the cropped region, ‘810’ in figure 8, (fourth image) and a set of point on a canonical shape (head specific canonical pose comprising points))
and generating, by the second trained machine learning model, the second set of landmarks as a set of positions of the set of points within the fourth image. (See ¶ 129 wherein head-specific tensor and position encoding values are generated corresponding to facial features (second set of landmarks) based on their corresponding points, via the head-specific canonical reconstruction network, ‘530’, in figure 5).
As taught by Guler, the generation of a second set of landmarks using positions of a set of points within an image allows for a determination of whether or not certain pixels are classified as being inside the head-specific canonical pose or outside of it. (See ¶ 129 wherein a determination is made on whether or not a pixel is classified as being inside of the head-specific canonical pose based on the position encodings values of the facial features). As both the teachings of Ye in view of Wang and Guler deal with the technical field of image processing regarding the processing of facial image data, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of Ye in view of Wang with Guler to teach of inputting, into the second trained machine learning model, (i) a set of points on a canonical shape and (ii) the fourth image; and generating, by the second trained machine learning model, the second set of landmarks as a set of positions of the set of points within the fourth image in order for a determination to be made of whether or not certain pixels are classified as being inside the head-specific canonical pose or outside of it.
Claim 6 is rejected under 35 U.S.C. 103 as being unpatentable over Ye; Shu-Hao et al. (US 10970526 B1; hereinafter simply referred to as Ye) in view of Deore; Milind et al. (US 20210240808 A1; hereinafter simply referred to as Deore).
Regarding dependent claim 6, Ye does not explicitly disclose:
Converting, via execution of a feature detector included in the second machine learning model, the second image into a set of features; and generating, via execution of a prediction network included in the second machine learning model, the first set of landmarks as a set of positions within the second image, wherein the set of positions corresponds to a set of key points on the first face.
However, Deore; teaches of converting via execution of a feature detector included in the second machine learning model, the second image into a set of features. (See ¶ 32 wherein a facial image is converted into a set of features (features such as eyes and nose etc.) wherein the conversion is done by a Deep neural net, DNN, (second machine learning model) using a feature detector (the feature detector being the part of the DNN that detects/classifies the image facial features)).
and generating, via execution of a prediction network included in the second machine learning model, the first set of landmarks as a set of positions within the second image, wherein the set of positions corresponds to a set of key points on the first face. (See ¶ 33 wherein the first set of landmarks (facial feature landmarks) are given a set of positions (coordinates) within the second image (image data) wherein the set of positions corresponds to a set of key points on the first face, being the coordinate points corresponding to the facial landmarks such as the eyes and nose. All executed by the DNN (second machine learning mode) using a prediction network (being the part of the DNN that determines/predicts the coordinates)).
As taught by Deore, the determination of the facial features and their positions on the face using a machine learning model allows for the facial feature information to be used in an authentication of a user. (See ¶ 30 wherein the facial features of a user are used as a form of authentication). As both the teachings of Ye and Deore deal with the technical field of image processing regarding processing of facial image information, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention to combine the teachings of Ye with Deore to teach of converting, via execution of a feature detector included in the second machine learning model, the second image into a set of features; and generating, via execution of a prediction network included in the second machine learning model, the first set of landmarks as a set of positions within the second image, wherein the set of positions corresponds to a set of key points on the first face in order for the determined facial features to be used for authentication of a user.
Claim 7 is rejected under 35 U.S.C. 103 as being unpatentable over Ye; Shu-Hao et al. (US 10970526 B1; hereinafter simply referred to as Ye) in view of Deore; Milind et al. (US 20210240808 A1; hereinafter simply referred to as Deore) further in view of Yuksel; Ali et al. (US 20250014189 A1; hereinafter simply referred to as Yuksel).
Regarding dependent claim 7, Ye in view of Deore does not explicitly disclose:
Determining the first set of landmarks further comprises generating a set of confidence values associated with the set of positions.
However, Yuksel teaches of determining the set of landmarks further comprises generating a set of confidence values associated with the set of positions. (See ¶ 42 and 43 wherein a set of facial landmarks positions are determined and confidence values are generated for each facial landmark and its corresponding position).
As taught by Yuksel, the use of confidence values associated with the set of landmark positions allows for only selecting the landmarks with a sufficiently high confidence value leading to more accurate of reliable determination of landmark points. (See ¶ 45 wherein only landmark positions with a high enough confidence are selected resulting in an increase in accuracy and reliability of landmark determination). As both the teachings of Ye and Yuksel deal with the technical field of image processing regarding the processing of facial image data, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of Ye with Yuksel to teach of determining the first set of landmarks further comprises generating a set of confidence values associated with the set of positions in order for the reliability and accuracy of landmark detection to be increased.
Claim 9 is rejected under 35 U.S.C. 103 as being unpatentable over Ye; Shu-Hao et al. (US 10970526 B1; hereinafter simply referred to as Ye) in view of Bhagavatula; Chandrasekhar et al. (US 20190012578 A1; hereinafter simply referred to as Bhagavatula).
Regarding dependent claim 9, Ye does not explicitly disclose:
The first machine learning model comprises a spatial transformer neural network.
However, Bhagavatula teaches of the first machine learning model comprises a spatial transformer neural network. (See ¶ 49 wherein a spatial transformer is used in a machine learning convolutional neural network for facial landmark generation).
As taught by Bhagavatula, the use of a spatial transformer neural network allows for the generation of a 3D model of a face. (See ¶ 49 wherein a spatial transformer network together with a convolutional neural network allow for the generation of a 3D model of a face). As both the teachings of Ye and Bhagavatula deal with the technical field of image processing regarding the processing of facial image data, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of Ye with Bhagavatula to teach of the first machine learning model comprises a spatial transformer neural network in order to allow for the generation of 3D models of a face.
Claim 10 is rejected under 35 U.S.C. 103 as being unpatentable over Ye; Shu-Hao et al. (US 10970526 B1; hereinafter simply referred to as Ye) in view of Xiang; Xiaoyu et al. (US 20230377095 A1; hereinafter simply referred to as Xiang).
Regarding dependent claim 10, Ye does not explicitly disclose:
The first transformation comprises an affine transformation.
However, Xiang teaches of the first transformation comprises an affine transformation. (See ¶ 25 wherein the first transformation comprises an affine transformation (generation of affine-aligned face image)).
As taught by Xiang the transformation being an affine transformation allows for the production of identity facial features. (See ¶ 25 wherein a affine-aligned face image (image transformed using affine transformation) allows for the production of identity facial features). As both the teachings of Ye and Xiang deal with the technical field of image processing regarding processing of facial image data, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of Ye with Xiang to teach of the first transformation comprises an affine transformation in order for identity facial features to be determined.
Claim 11, 15, and 17 are rejected under 35 U.S.C. 103 as being unpatentable over Ye; Shu-Hao et al. (US 10970526 B1; hereinafter simply referred to as Ye) further in view of Guler; Riza et al. (US 20240346763 A1; hereinafter simply referred to as Guler).
Regarding independent claim 11, Ye teaches:
Applying, via execution of a first machine learning model, a first transformation to a first image depicting a first face to generate a second image (See Col 3, Lines 35 – 54, and Col 5 Lines 39 – 51, and Figure 1, wherein a first machine learning model (feature fusion convolutional autoencoder, ‘100’ in figure 1, comprising convolutional decoder ‘106’ in figure 1) takes an input facial image, thermal facial image, and performs a transformation/reconstruction, to obtain a second image, being the reconstructed visible facial image)
determining, via execution of a second machine learning model, a first set of landmarks on the first face based on the second image (See Col 3, Lines 35 – 54, and Col 5 Lines 39 – 51, and Figure 1 wherein a second machine learning model (facial landmark detection model, ‘110’ in figure 1) determines facial landmark models (first set of landmarks) of the first face (face in the thermal facial image) based on the second image (reconstructed visible facial image)).
and training the first machine learning model based on one or more losses associated with the first set of landmarks to generate a first trained machine learning model. (Col 5 Lines 39 – 67, Col 6 Lines 1 – 2, Col 3, Lines 35 – 54 and Figure 1, and Formulas 3 and 4 wherein the first machine learning model, being the convolutional autoencoder ‘100’, in figure 1, is trained (parameter weight adjustment through backpropagation) using the hybrid loss function, ‘111’ in figure 1, that are associated with the first set of landmarks, to generate a first trained machine learning model (being the post adjustment convolutional autoencoder))
Ye fails to specifically disclose one or more non-transitory computer readable media storing instructions that, when executed by one or more processors, cause the one or more processors to perform the steps of facial landmark recognition.
However, Guler teaches of one or more non-transitory computer readable media storing instructions that, when executed by one or more processors, cause the one or more processors to perform the steps of facial landmark recognition (See ¶ 159, 126 wherein a non-transitory computer-readable storage medium contains instructions executed by at least one processor, causing the processor to execute the instructions to perform the steps of identifying facial landmarks in an image).
As taught by Guler the use of a non-transitory computer-readable storage medium that stores instructions executed by a processor allows for instructions to be executed that perform various functions such as volumetric reconstruction and predicting a volumetric reconstruction tensor. (See ¶ 159 wherein the use of a non-transitory computer-readable storage medium that stores instruction to be executed by a processor allows for various image processing operations to be executed such as volumetric reconstruction and predicting a volumetric reconstruction tensor). As both the teachings of Ye and Guler deal with the technical field of image processing regarding facial image data, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of Ye with Guler to teach of the use of one or more non-transitory computer readable media storing instructions that, when executed by one or more processors, cause the one or more processors to perform the steps of facial landmark recognition in order to allow for instructions to be executed that perform various functions such as volumetric reconstruction and predicting a volumetric reconstruction tensor.
Regarding dependent claim 15, Ye in view of Guler teaches:
Converting the second image into a set of features and a set of parameters; (See Guler ¶ 126, 127 and figure 8 wherein the second image (cropped image region ‘810’ in figure 8) is converted into a set of features (facial features) and a set of parameters (head-specific volumetric reconstruction tensor))
converting a set of points on a canonical shape into a set of position encodings; (See Guler ¶ 128 wherein a set of points (set of points from the head-specific volumetric reconstruction tensor) are converted into a set of position encodings, wherein the points are in a canonical shape (head-specific canonical pose))
and generating, based on the set of features and the set of position encodings, a set of three-dimensional (3D) positions that is (i) included in the first set of landmarks and (ii) in a canonical space associated with the canonical shape (See Guler ¶ 126- 130, figures 5 and 8, wherein based on the set of features (facial features) and position encodings, a set of 3D positions is generated (generated 3D mesh comprising 3D positions) included in the first set of landmarks (facial features that make up the head 3D mesh) in a canonical space associated with the canonical shape (head-specific canonical pose representing a pose of a head of an object in a canonical space)).
Regarding dependent claim 17, Ye in view of Guler teaches:
Determining the first set of landmarks further comprises determining the set of points based on a set of displacements of a set of query points associated with the first set of landmarks. (See Guler ¶ 105 – 107 wherein determining the first set of landmarks (facial landmarks) comprises determining the set of points based on a set of displacements (deviations) of a set of query points (ground truth canonical space representation comprising points) associated with the first set of landmarks, being the determination of the individual points belonging to the canonical volume based on the comparison to the ground truth space representation).
Claims 12 and 13 are rejected under 35 U.S.C. 103 as being unpatentable over Ye; Shu-Hao et al. (US 10970526 B1; hereinafter simply referred to as Ye) in view of Guler; Riza et al. (US 20240346763 A1; hereinafter simply referred to as Guler) further in view of Wang; Hong et al. (WO 2023061122 A1; hereinafter simply referred to as Wang; translated via Espacenet).
Regarding dependent claim 12, Ye in view of Guler does not explicitly disclose:
Training the second machine learning model based on the one or more losses to generate a second trained machine learning model.
However, Wang teaches of training the second machine learning model based on the one or more losses to generate a second trained machine learning model. (See ¶ 153 wherein the second machine learning model (second convolutional neural network) is trained (updated through backpropagation) using the one or more losses (contrastive loss) to generate a second trained machine learning model (updated second convolutional neural network)).
As taught by Wang having multiple trained machine learning models trained using losses allows for one machine learning model to focus on one task and another machine learning model to focus on another task. (See ¶ 141 wherein the first convolutional neural network focus on extracting facial features from overall image samples and the second convolutional neural network focuses on extracting facial features using local image samples). As both the teachings of Ye in view of Guler and Wang deal with the technical field of image processing regarding facial images, it would have been obvious to one or ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of Ye in view of Guler with Wang to teach of training the second machine learning model based on the one or more losses to generate a second trained machine learning model in order for the two machine learning models to focus on different tasks while still being update/trained using the one or more losses.
Regarding dependent claim 13, Ye in view of Guler and Wang teaches:
Applying, via execution the first trained machine learning model, a second transformation to a second face depicted in a third image to generate a fourth image; (See Ye Col 3, Lines 35 – 54, and Col 5 Lines 39 – 67, and Figure 1, wherein a batch of multiple thermal facial images (including a third image) are input into the first trained machine learning model (post parameter weight adjustment convolutional autoencoder) to generate a fourth image (reconstructed visible facial image))
inputting, into the second trained machine learning model, (i) a set of points on a canonical shape and (ii) the fourth image; (See Guler ¶ 126 – 130, figure 5 and 8, wherein the second trained machine learning model (third and fourth machine learning models as part of the head-specific canonical reconstruction network, ‘530’ in figure 5) receives input of the cropped region, ‘810’ in figure 8, (fourth image) and a set of point on a canonical shape (head specific canonical pose comprising points))
and generating, by the second trained machine learning model, a second set of landmarks on the second face as a set of positions of the set of points within the fourth image (See ¶ 129 wherein head-specific tensor and position encoding values are generated corresponding to facial features (second set of landmarks) based on their corresponding points, via the head-specific canonical reconstruction network, ‘530’, in figure 5).
Claim 18 is rejected under 35 U.S.C. 103 as being unpatentable over Ye; Shu-Hao et al. (US 10970526 B1; hereinafter simply referred to as Ye) in view of Guler; Riza et al. (US 20240346763 A1; hereinafter simply referred to as Guler) further in view of Yuksel; Ali et al. (US 20250014189 A1; hereinafter simply referred to as Yuksel).
Regarding dependent claim 18, Ye in view of Guler does not explicitly disclose:
The first set of landmarks comprises a set of positions within the second image and a set of confidence values associated with the set of positions.
However, Yuksel teaches The first set of landmarks comprises a set of positions within the second image and a set of confidence values associated with the set of positions (See ¶ 41 - 43 wherein a set of facial landmarks positions are determined and confidence values are generated for each facial landmark and its corresponding position, within the second image (cropped face portion of image)).
As taught by Yuksel, the use of confidence values associated with the set of landmark positions allows for only selecting the landmarks with a sufficiently high confidence value leading to more accurate of reliable determination of landmark points. (See ¶ 45 wherein only landmark positions with a high enough confidence are selected resulting in an increase in accuracy and reliability of landmark determination). As both the teachings of Ye and Yuksel deal with the technical field of image processing regarding the processing of facial image data, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of Ye with Yuksel to teach of determining the first set of landmarks further comprises generating a set of confidence values associated with the set of positions in order for the reliability and accuracy of landmark detection to be increased.
Claim 19 is rejected under 35 U.S.C. 103 as being unpatentable over Ye; Shu-Hao et al. (US 10970526 B1; hereinafter simply referred to as Ye) in view of Guler; Riza et al. (US 20240346763 A1; hereinafter simply referred to as Guler) further in view of Cashman; Thomas (US 20230281945 A1; hereinafter simply referred to as Cashman).
Regarding dependent claim 19, Ye in view of Guler does not explicitly disclose:
The one or more losses comprise a Gaussian negative likelihood loss.
However, Cashman teaches of the one or more losses comprise a Gaussian negative likelihood loss. (See ¶ 52 wherein the one or more losses comprise a Gaussian negative likelihood loss (Gaussian negative log likelihood loss, GNLL)).
As taught by Cashman the use of GNLL allows for the training of a neural network to detect landmarks with less uncertainty. (See ¶ 52 and 116 wherein the use of GNLL allows for training of a neural network to detect landmarks with more certainty). As both the teachings of Ye in view of Guler and Cashman deal with the technical field of image processing regarding the processing of facial image data, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of Ye in view of Guler with Cashman to teach of the one or more losses comprise a Gaussian negative likelihood loss in order to allow for the training of a neural network and for the detection of landmarks to be more completed with more certainty.
Allowable Subject Matter
Claims 5, 14, and 16 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
The following is a statement of reasons for the indications of allowable subject matter:
Regarding clams 5 and 14, the reason of allowable subject matter is that the prior art fails to teach or reasonably suggest the limitations of claims 1 and 11 respectively further comprising applying the first transformation to the first image comprises: generating, via execution of the first machine learning model, a set of parameters corresponding to the first transformation based on the first image; generating a sampling grid associated with the first image based on the set of parameters, wherein the sampling grid specifies a set of spatial locations to be sampled from the first image; and applying a sampling kernel to each spatial location included in the set of spatial locations to generate a pixel value for a corresponding spatial location in the second image.
Regarding clams 16, the reason of allowable subject matter is that the prior art fails to teach or reasonably suggest the limitations of claim15 further comprising determining the first set of landmarks further comprises: applying, based on the set of parameters, one or more additional transformations to the set of 3D positions to generate a first set of two-dimensional (2D) positions that is (i) included in the first set of landmarks and (ii) in a first 2D space associated with the second image; and applying a second transformation that is an inverse of the first transformation to the first set of 2D positions to generate a second set of 2D positions that is (i) included in the first set of landmarks and (ii) in a second 2D space associated with the first image.
Prior Art Made of Record
The prior art made of record and not relied upon is considered pertinent to Applicant’s disclosure and is as follows:
U.S. Patent Application No. US 20230162407 A1 (Kalarot) discloses the determination of a landmark loss by comparing the facial landmarks of a modified image to an input of facial landmarks, and using the losses to train a machine learning model. (Kalarot [0073]).
Furthermore please See attached PTO-892.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ALEJANDRO HERNANDEZ whose telephone number is (703)756-1876. The examiner can normally be reached M-F 8 am - 5 pm ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, John M Villecco can be reached at (571) 272-7319. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ALEJANDRO HERNANDEZ/Examiner, Art Unit 2661
/JOHN VILLECCO/Supervisory Patent Examiner, Art Unit 2661
Prosecution Insights