DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claims 1-6 are pending in this application.
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
Specification
The title of the invention is not descriptive. A new title is required that is clearly indicative of the invention to which the claims are directed.
Priority
Receipt is acknowledged of certified copies of papers required by 37 CFR 1.55 granting priority to KR10-2023-0168440 originally filed on November 28, 2023.
35 U.S.C. § 112 Sixth Paragraph - Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action.
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because the claim limitation(s) uses a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier. Such claim limitations are: “steps” in claims 4-6.
Because this/these claim limitation(s) is/are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, it/they is/are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof.
If applicant does not intend to have this/these limitation(s) interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation(s) to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation(s) recite(s) sufficient structure to perform the claimed function so as to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1-2 and 4-5 are rejected under 35 U.S.C. 102(a)(2) as being anticipated by Sidiya et al. (US PGPub US2023/0169626A1, filed on November 30, 2021), hereby referred to as “Sidiya”.
Consider Claim 1.
Sidiya teaches:
1. An apparatus for reconstructing facial images, comprising: (Sidiya: abstract, A neural network system for restoring images, a method and a non-transitory computer-readable storage medium thereof are provided. The neural network system includes an encoder and a generative adversarial network (GAN) prior network. The encoder includes a plurality of encoder blocks, where each encoder block includes at least one transformer block and one convolution layer, where the encoder receives an input image and generates a plurality of encoder features and a plurality of latent vectors. Additionally, the GAN prior network includes a plurality of pre-trained generative prior layers, where the GAN prior network receives the plurality of encoder features and the plurality of latent vectors from the encoder and generates an output image with super-resolution. [0035]-[0044], Figure 1, [0044] As shown in FIG. 1 , the GAN prior network 102 includes a plurality of generative prior layers. The plurality of generative prior layers may include generative prior layers 102-1, 102-2, 102-3, . . . , and 102-7 that are stacked to each other. The number of the plurality of generative prior layers is not limited to 7. FIG. 1 is only for illustrating.)
1. a generator network including an encoder and a decoder, (Sidiya: [0035]-[0044], Figure 1, [0045] The generative prior layer 102-1 receives inputs including the encoder features f5 from the encoder block 101-5, the encoder features f6 from the encoder block 101-6, and the latent vector c1 from the fully connected layer FC 103, and then generates an output feature. The generative prior layer 102-2 receives the output feature from the generative prior layer 102-1. In addition to the output feature of the generative prior layer 102-1, the generative prior layer 102-2 receives the encoder features f4 from the encoder block 101-4 and the latent vector c2 from the fully connected layer FC 103. After receiving the inputs, the generative prior layer 102-2 generates an output feature and sends the output feature to the generative prior layer 102-3 that subsequently follows the generative prior layer 102-2. [0052] FIG. 2 is a block diagram illustrating a neural network system including an encoder with transformer blocks, a GAN prior network, and a decoder in accordance with one or more examples of the present disclosure. In addition to the encoder and the GAN prior network, the neural network system in FIG. 2 includes a decoder as well. The overall architecture of the neural network system in FIG. 2 includes the encoder and the decoder that separated with the trained weights of the GAN prior network. The GAN prior network is connected to the encoder and the decoder with skip connections. The encoder network is built using successive transformer blocks composed of self-attention layers and residual blocks. The decoder network includes convolution layers followed by pixel shuffle layers for features up-sampling. The output of each encoder block with a specific resolution is concatenated with the output of the corresponding block in the GAN prior network, then a convolution layer followed by a transformer block is applied to the results and the output is fed to the next block in the GAN prior network. In addition, the output of the last layer of the GAN prior network is used as an input to the decoder with the output of the initial encoder block in the encoder.)
1. wherein the encoder analyzes an occluded image to extract feature values, (Sidiya: [0037], [0038] The encoder block 101-1 receives an input image having a low-resolution and extracts encoder features f1 from the input image. The input image may be a face image. The encoder features f1 are sent to both the GAN prior network 102 and the encoder block 101-2 that subsequently follows the encoder block 101-1. In an example, the encoder features f1 may have a resolution of 64×64 as shown in FIG. 1 . In the encoder block 101-1, the convolution layer EC 1 and the plurality of transformer blocks T11-T16 are stacked to each other, and the convolution layer EC 1 is followed by the plurality of transformer blocks T11-T16. The number of the plurality of transformer blocks in the encoder 101 is not limited to 6. [0039] The encoder block 101-2 receives the encoder features f1 from the encoder block 101-1 and generates the encoder features f2. The encoder features f2 are sent to both the GAN prior network 102 and the encoder block 101-3 that subsequently follows the encoder block 101-2. In an example, the encoder features f2 may have a resolution of 32×32 as shown in FIG. 1 . In the encoder block 101-2, the convolution layer EC 2 is followed by the transformer block T21.)
1. and the decoder restores a final facial image. (Sidiya: [0052], Figure 2, [0054] The decoder 204 includes a plurality of decoder blocks. The plurality of decoder blocks include the decoder blocks 204-1, 204-2, and 204-3 as shown in FIG. 2 . Each decoder block include a convolution layer and a pixel shuffle layer that follows the convolution layer. For example, the decoder block 204-1 includes a convolution layer 2041-1 and a pixel shuffle layer 2041-2, the decoder block 204-2 includes a convolution layer 2042-1 and a pixel shuffle layer 2042-2, the decoder block 204-3 includes a convolution layer 2043-1 and a pixel shuffle layer 2043-2. [0055] The convolution layer 2041-1 in the decoder block 204-1 receives inputs including the output feature from the generative prior layer 202-7 and the encoder feature f1, and then generates an output feature. The pixel shuffle layer 2041-2 receives the output feature of the convolution layer 2041-1 and up-samples the output feature. For example, the pixel shuffle layer 2041-2 up-samples the output feature of the convolution layer 2041-1 to 64×64 and sends the up-sampled feature to the decoder block 204-2 that follows the decoder block 204-1. [0056]-[0060])
Consider Claim 2.
Sidiya teaches: 2. The apparatus for reconstructing facial images according to claim 1, wherein the encoder and the decoder are connected in a skip-connection manner using a spatial style map, and wherein the spatial style map is generated by inputting a style value into a mapping network. (Sidya: [0060] FIG. 3 is a block diagram illustrating a transformer block in the neural network system shown in FIG. 1 , FIG. 2 or FIG. 5 in accordance with an example of the present disclosure. As shown in FIG. 3 , the transformer block 300 includes a self-attention layer 301 with a skip connection, a convolution layer 302, a Leaky Rectified Linear Activation (LReLU) layer 303, and a convolution layer 304. The LReFU layer 303 is sandwiched between the convolution layer 302 and the convolution layer 304. [0061] The output and input of the self-attention layer 301 are added to each other using a skip connection and the added result passed through a residual block to form the overall operations of the transformer block 301. For example, the added result is then sent to the convolution layer 302. The convolution layer 302 generates a first convolution output and sends the first convolution output to the LReFU layer 303. Further, the LReFU layer 303 generates an LReFU output and sends the LReFU output to the convolution layer 304, and the convolution layer 304 generates a second convolution output. The input of the convolution layer 302 and the second convolution output of the convolution layer 304 are added to each other using a skip connection to generate an output of the transformer block 300. [0062] FIG. 4 is a block diagram illustrating a self-attention layer in the transformer block shown in FIG. 3 in accordance with an example of the present disclosure. The self-attention layer 301 may include a plurality of projection layers, e.g., separable depth-wise convolution layers, each of which respectively learns query, key, and value features. The query, key, and value features may be embeddings related to inputs of the self-attention layer. The outputs of the projection layers are divided into small patches through a patch division layer 402. K, Q and V may be respectively matrices of a set of key features, query features and value features. After division, the key features K is transposed using a transpose layer 403, the query features Q and the transpose of key features K are multiplied, and an attention map is obtained through a softmax layer 404. Moreover, the attention map is multiplied by the value features V and the output is merged using an inverse of the patch division operation through a patch merge layer 405 and a final convolution is applied using a convolution layer 406 to generate the output of the self-attention layer 301. The patch division layer 402 divides feature maps to patch block so as to reduce the computational cost without losing results performance.)
Consider Claim 4.
Sidiya teaches:
4. A method for reconstructing facial images, comprising the steps of: (Sidiya: abstract, A neural network system for restoring images, a method and a non-transitory computer-readable storage medium thereof are provided. The neural network system includes an encoder and a generative adversarial network (GAN) prior network. The encoder includes a plurality of encoder blocks, where each encoder block includes at least one transformer block and one convolution layer, where the encoder receives an input image and generates a plurality of encoder features and a plurality of latent vectors. Additionally, the GAN prior network includes a plurality of pre-trained generative prior layers, where the GAN prior network receives the plurality of encoder features and the plurality of latent vectors from the encoder and generates an output image with super-resolution. [0035]-[0044], Figure 1, [0044] As shown in FIG. 1 , the GAN prior network 102 includes a plurality of generative prior layers. The plurality of generative prior layers may include generative prior layers 102-1, 102-2, 102-3, . . . , and 102-7 that are stacked to each other. The number of the plurality of generative prior layers is not limited to 7. FIG. 1 is only for illustrating.)
4. analysing, by an encoder of a generator network, (Sidiya: [0035]-[0044], Figure 1, [0045] The generative prior layer 102-1 receives inputs including the encoder features f5 from the encoder block 101-5, the encoder features f6 from the encoder block 101-6, and the latent vector c1 from the fully connected layer FC 103, and then generates an output feature. The generative prior layer 102-2 receives the output feature from the generative prior layer 102-1. In addition to the output feature of the generative prior layer 102-1, the generative prior layer 102-2 receives the encoder features f4 from the encoder block 101-4 and the latent vector c2 from the fully connected layer FC 103. After receiving the inputs, the generative prior layer 102-2 generates an output feature and sends the output feature to the generative prior layer 102-3 that subsequently follows the generative prior layer 102-2. [0052] FIG. 2 is a block diagram illustrating a neural network system including an encoder with transformer blocks, a GAN prior network, and a decoder in accordance with one or more examples of the present disclosure. In addition to the encoder and the GAN prior network, the neural network system in FIG. 2 includes a decoder as well. The overall architecture of the neural network system in FIG. 2 includes the encoder and the decoder that separated with the trained weights of the GAN prior network. The GAN prior network is connected to the encoder and the decoder with skip connections. The encoder network is built using successive transformer blocks composed of self-attention layers and residual blocks. The decoder network includes convolution layers followed by pixel shuffle layers for features up-sampling. The output of each encoder block with a specific resolution is concatenated with the output of the corresponding block in the GAN prior network, then a convolution layer followed by a transformer block is applied to the results and the output is fed to the next block in the GAN prior network. In addition, the output of the last layer of the GAN prior network is used as an input to the decoder with the output of the initial encoder block in the encoder.)
4. an occluded image to extract feature values, (Sidiya: [0037], [0038] The encoder block 101-1 receives an input image having a low-resolution and extracts encoder features f1 from the input image. The input image may be a face image. The encoder features f1 are sent to both the GAN prior network 102 and the encoder block 101-2 that subsequently follows the encoder block 101-1. In an example, the encoder features f1 may have a resolution of 64×64 as shown in FIG. 1 . In the encoder block 101-1, the convolution layer EC 1 and the plurality of transformer blocks T11-T16 are stacked to each other, and the convolution layer EC 1 is followed by the plurality of transformer blocks T11-T16. The number of the plurality of transformer blocks in the encoder 101 is not limited to 6. [0039] The encoder block 101-2 receives the encoder features f1 from the encoder block 101-1 and generates the encoder features f2. The encoder features f2 are sent to both the GAN prior network 102 and the encoder block 101-3 that subsequently follows the encoder block 101-2. In an example, the encoder features f2 may have a resolution of 32×32 as shown in FIG. 1 . In the encoder block 101-2, the convolution layer EC 2 is followed by the transformer block T21.)
4. and reconstructing, by a decoder of the generator network, a final facial image. (Sidiya: [0052], Figure 2, [0054] The decoder 204 includes a plurality of decoder blocks. The plurality of decoder blocks include the decoder blocks 204-1, 204-2, and 204-3 as shown in FIG. 2 . Each decoder block include a convolution layer and a pixel shuffle layer that follows the convolution layer. For example, the decoder block 204-1 includes a convolution layer 2041-1 and a pixel shuffle layer 2041-2, the decoder block 204-2 includes a convolution layer 2042-1 and a pixel shuffle layer 2042-2, the decoder block 204-3 includes a convolution layer 2043-1 and a pixel shuffle layer 2043-2. [0055] The convolution layer 2041-1 in the decoder block 204-1 receives inputs including the output feature from the generative prior layer 202-7 and the encoder feature f1, and then generates an output feature. The pixel shuffle layer 2041-2 receives the output feature of the convolution layer 2041-1 and up-samples the output feature. For example, the pixel shuffle layer 2041-2 up-samples the output feature of the convolution layer 2041-1 to 64×64 and sends the up-sampled feature to the decoder block 204-2 that follows the decoder block 204-1. [0056]-[0060])
Consider Claim 5.
Sidiya teaches: 5. The method for reconstructing facial images according to claim 4, wherein the encoder and the decoder are connected in a skip-connection manner using a spatial style map, and wherein the spatial style map is generated by inputting a style value into a mapping network. (Sidya: [0060] FIG. 3 is a block diagram illustrating a transformer block in the neural network system shown in FIG. 1 , FIG. 2 or FIG. 5 in accordance with an example of the present disclosure. As shown in FIG. 3 , the transformer block 300 includes a self-attention layer 301 with a skip connection, a convolution layer 302, a Leaky Rectified Linear Activation (LReLU) layer 303, and a convolution layer 304. The LReFU layer 303 is sandwiched between the convolution layer 302 and the convolution layer 304. [0061] The output and input of the self-attention layer 301 are added to each other using a skip connection and the added result passed through a residual block to form the overall operations of the transformer block 301. For example, the added result is then sent to the convolution layer 302. The convolution layer 302 generates a first convolution output and sends the first convolution output to the LReFU layer 303. Further, the LReFU layer 303 generates an LReFU output and sends the LReFU output to the convolution layer 304, and the convolution layer 304 generates a second convolution output. The input of the convolution layer 302 and the second convolution output of the convolution layer 304 are added to each other using a skip connection to generate an output of the transformer block 300. [0062] FIG. 4 is a block diagram illustrating a self-attention layer in the transformer block shown in FIG. 3 in accordance with an example of the present disclosure. The self-attention layer 301 may include a plurality of projection layers, e.g., separable depth-wise convolution layers, each of which respectively learns query, key, and value features. The query, key, and value features may be embeddings related to inputs of the self-attention layer. The outputs of the projection layers are divided into small patches through a patch division layer 402. K, Q and V may be respectively matrices of a set of key features, query features and value features. After division, the key features K is transposed using a transpose layer 403, the query features Q and the transpose of key features K are multiplied, and an attention map is obtained through a softmax layer 404. Moreover, the attention map is multiplied by the value features V and the output is merged using an inverse of the patch division operation through a patch merge layer 405 and a final convolution is applied using a convolution layer 406 to generate the output of the self-attention layer 301. The patch division layer 402 divides feature maps to patch block so as to reduce the computational cost without losing results performance.)
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent may not be obtained though the invention is not identically disclosed or described as set forth in section 102 of this title, if the differences between the subject matter sought to be patented and the prior art are such that the subject matter as a whole would have been obvious at the time the invention was made to a person having ordinary skill in the art to which said subject matter pertains. Patentability shall not be negatived by the manner in which the invention was made.
Claims 1-6 are rejected under 35 U.S.C. 103 as being unpatentable over Sidiya et al. (US PGPub US2023/0169626A1), hereby referred to as “Sidiya”, in view of Fu et al. (US PGPub 2021/0232803), hereby referred to as “Fu”.
Consider Claim 1 and 3.
Sidiya teaches:
1. An apparatus for reconstructing facial images, comprising: (Sidiya: abstract, A neural network system for restoring images, a method and a non-transitory computer-readable storage medium thereof are provided. The neural network system includes an encoder and a generative adversarial network (GAN) prior network. The encoder includes a plurality of encoder blocks, where each encoder block includes at least one transformer block and one convolution layer, where the encoder receives an input image and generates a plurality of encoder features and a plurality of latent vectors. Additionally, the GAN prior network includes a plurality of pre-trained generative prior layers, where the GAN prior network receives the plurality of encoder features and the plurality of latent vectors from the encoder and generates an output image with super-resolution. [0035]-[0044], Figure 1, [0044] As shown in FIG. 1 , the GAN prior network 102 includes a plurality of generative prior layers. The plurality of generative prior layers may include generative prior layers 102-1, 102-2, 102-3, . . . , and 102-7 that are stacked to each other. The number of the plurality of generative prior layers is not limited to 7. FIG. 1 is only for illustrating.)
1. a generator network including an encoder and a decoder, (Sidiya: [0035]-[0044], Figure 1, [0045] The generative prior layer 102-1 receives inputs including the encoder features f5 from the encoder block 101-5, the encoder features f6 from the encoder block 101-6, and the latent vector c1 from the fully connected layer FC 103, and then generates an output feature. The generative prior layer 102-2 receives the output feature from the generative prior layer 102-1. In addition to the output feature of the generative prior layer 102-1, the generative prior layer 102-2 receives the encoder features f4 from the encoder block 101-4 and the latent vector c2 from the fully connected layer FC 103. After receiving the inputs, the generative prior layer 102-2 generates an output feature and sends the output feature to the generative prior layer 102-3 that subsequently follows the generative prior layer 102-2. [0052] FIG. 2 is a block diagram illustrating a neural network system including an encoder with transformer blocks, a GAN prior network, and a decoder in accordance with one or more examples of the present disclosure. In addition to the encoder and the GAN prior network, the neural network system in FIG. 2 includes a decoder as well. The overall architecture of the neural network system in FIG. 2 includes the encoder and the decoder that separated with the trained weights of the GAN prior network. The GAN prior network is connected to the encoder and the decoder with skip connections. The encoder network is built using successive transformer blocks composed of self-attention layers and residual blocks. The decoder network includes convolution layers followed by pixel shuffle layers for features up-sampling. The output of each encoder block with a specific resolution is concatenated with the output of the corresponding block in the GAN prior network, then a convolution layer followed by a transformer block is applied to the results and the output is fed to the next block in the GAN prior network. In addition, the output of the last layer of the GAN prior network is used as an input to the decoder with the output of the initial encoder block in the encoder.)
1. wherein the encoder analyzes an occluded image to extract feature values, (Sidiya: [0037], [0038] The encoder block 101-1 receives an input image having a low-resolution and extracts encoder features f1 from the input image. The input image may be a face image. The encoder features f1 are sent to both the GAN prior network 102 and the encoder block 101-2 that subsequently follows the encoder block 101-1. In an example, the encoder features f1 may have a resolution of 64×64 as shown in FIG. 1 . In the encoder block 101-1, the convolution layer EC 1 and the plurality of transformer blocks T11-T16 are stacked to each other, and the convolution layer EC 1 is followed by the plurality of transformer blocks T11-T16. The number of the plurality of transformer blocks in the encoder 101 is not limited to 6. [0039] The encoder block 101-2 receives the encoder features f1 from the encoder block 101-1 and generates the encoder features f2. The encoder features f2 are sent to both the GAN prior network 102 and the encoder block 101-3 that subsequently follows the encoder block 101-2. In an example, the encoder features f2 may have a resolution of 32×32 as shown in FIG. 1 . In the encoder block 101-2, the convolution layer EC 2 is followed by the transformer block T21.)
1. and the decoder restores a final facial image. (Sidiya: [0052], Figure 2, [0054] The decoder 204 includes a plurality of decoder blocks. The plurality of decoder blocks include the decoder blocks 204-1, 204-2, and 204-3 as shown in FIG. 2 . Each decoder block include a convolution layer and a pixel shuffle layer that follows the convolution layer. For example, the decoder block 204-1 includes a convolution layer 2041-1 and a pixel shuffle layer 2041-2, the decoder block 204-2 includes a convolution layer 2042-1 and a pixel shuffle layer 2042-2, the decoder block 204-3 includes a convolution layer 2043-1 and a pixel shuffle layer 2043-2. [0055] The convolution layer 2041-1 in the decoder block 204-1 receives inputs including the output feature from the generative prior layer 202-7 and the encoder feature f1, and then generates an output feature. The pixel shuffle layer 2041-2 receives the output feature of the convolution layer 2041-1 and up-samples the output feature. For example, the pixel shuffle layer 2041-2 up-samples the output feature of the convolution layer 2041-1 to 64×64 and sends the up-sampled feature to the decoder block 204-2 that follows the decoder block 204-1. [0056]-[0060])
3. The apparatus for reconstructing facial images according to claim 1, wherein the generator network is trained with three loss functions. (Sydia: [0030] A CNN that incorporates a pre-trained network of StyleGAN may achieve great results of image super-resolution. The CNN may be composed of encoder and decoder networks separated by trained weights of a generative model. Both encoder and decoder are built with successive convolution layers. In addition, the decoder may contain a pixel shuffle layers to up-sample input features. [0031] The prior information is combined by adding skip connection or concatenation operation between the encoder and the pretrained StyleGAN network as well as between the GAN prior network and the decoder. The network is trained end to end with perceptual loss, mean square loss and cross entropy loss. It is trained for 200 thousand iterations. Such CNN may lack ability to utilize non-local information for face reconstruction. [0063] In some examples, during the training of the neural network system, the weights of the generative prior network may be kept fixed. The neural network system is trained for an up-sampling factor of 4 from 64×64 to 256×256. The neural network system is trained for 200,000 iterations using mean square loss, perceptual loss and cross entropy loss. [0064] In some examples, the dataset used to train the neural network system is a synthetic dataset, composed of paired low-resolution and high-resolution image faces which simulate degradation found in real-world face images. FIG. 6 shows comparison among output images obtained respectively through bicubic up-sampling, PSFRGAN, GFP-GAN and the neural network system in accordance with an example of the present disclosure. As shown in FIG. 6, 601 shows an output image obtained using bicubic up-sampling, 602 shows an output image obtained using PSFRGAN, 603 shows an output image using GFP-GAN, and 604 shows an output image obtained using the neural network system in accordance with the present disclosure.)
Sydia does not teach the limitations from Claim 3 that the three loss functions are:
3. The apparatus for reconstructing facial images according to claim 1, wherein the generator network is trained with a style consistency loss function, an adversarial loss function, and an identity preserving loss function, wherein the style consistency loss function allows the encoder to analyze the occluded image to extract feature values, wherein the adversarial loss function ensures that the reconstructed image is indistinguishable from the original image in visual terms, and wherein the identity preserving loss function ensures that the identity of the reconstructed face is as similar as possible to the identity of the original image.
Fu teaches:
1. An apparatus for reconstructing facial images, comprising: (Fu: abstract, An apparatus and corresponding method for frontal face synthesis. The apparatus comprises a decoder that synthesizes a high-resolution (HR) frontal-view (FV) image of a face from received features of a low-resolution (LR) non-frontal-view (NFV) image of the face. The HR FV image is of a higher resolution relative to a lower resolution of the LR NFV image. The decoder includes a main path and an auxiliary path. The auxiliary path produces auxiliary-path features from the received features and feeds the auxiliary-path features produced into the main path for synthesizing the HR FV image. The auxiliary-path features represent a HR NFV image of the face at the higher resolution. As such, an HR identity-preserved frontal face can be synthesized from one or many LR faces with various poses and may be used in types of commercial applications, such as video surveillance. [0029]-[0042], Figures 1A-C;)
1. a generator network including an encoder and a decoder, (Fu: [0027], An example embodiment enables frontal view synthesis from single or multiple low-resolution (LR) faces with various poses. Generally speaking, an example embodiment may be directed to a super-resolution (SR) integrated generative adversarial network (SRGAN) that learns face frontalization and super-resolution collaboratively to synthesize high-quality, identity-preserved frontal faces, as disclosed further below. Super-resolution recovers a high-resolution image from a low-resolution image by upscaling and/or improving details within the low-resolution image. An example embodiment learns a generator network, such as disclosed below with regard to FIG. 1C-1, that includes a deep encoder and a SR integrated decoder. Features extracted by the deep encoder are passed to the decoder for reconstruction. An example embodiment of a decoder is specially designed to first super-resolve (i.e., recover a high-resolution image from a low-resolution image) non-frontal-view (NFV) images, such as side-view (SV) images, and ultimately utilize the information to reconstruct high-resolution HR frontal-view (FV) faces. To train the model, a three-level loss (i.e., pixel, patch, and global) provides fine-to-coarse coverage that learns a precise non-linear transformation between a LR NFV image of face(s) and an HR FV image of the face(s). Moreover, SRGAN accepts multiple LR profile faces as input by adding an orthogonal constraint in the generator to penalize redundant latent representations and, hence, diversify the learned features space. With these techniques, an example embodiment can generate frontal faces faithful to ground-truth, as disclosed in further detail below. [0029]-[0042], Figures 1A-C; [0029] FIG. 1A is block diagram of an example embodiment of an apparatus 100 for frontal face synthesis. The apparatus 100 may be referred to interchangeably herein as a super-resolution (SR) integrated generative adversarial network (SRGAN). The apparatus 100 comprises a decoder 102 configured to synthesize a high-resolution (HR) frontal-view (FV) image 104 of a face 106 from received features 108 of a low-resolution (LR) non-frontal-view (NFV) image 110 of the face 106. The HR FV image 104 may also be referred to interchangeably herein as a super-resolved front face ISF. The HR FV image 104 is of a higher resolution relative to a lower resolution of the LR NFV image 110. The decoder 102 includes a main path 112 and an auxiliary path 114. The auxiliary path 114 is configured to produce auxiliary-path features 116 from the received features 108 and feed the auxiliary-path features 116 produced into the main path 112 for synthesizing the HR FV image 104. The auxiliary-path features represent a HR NFV image 144 of the face 106 at the higher resolution. The auxiliary path 114 may be referred to interchangeably herein as a super-resolution (SR) module or side-view SR branch. Super-resolution includes recovering a high-resolution image from a low-resolution image by upscaling and/or improving details within the low-resolution image. The auxiliary path may be considered to be a “branch” as it splits off from the main path 112.)
1. wherein the encoder analyzes an occluded image to extract feature values, (Fu: [0040] The apparatus 100 further comprises an encoder 146. The encoder 146 may include a combination 148 of a plurality of convolutional layers configured to produce a feature map 152 of features extracted from the LR NFV image 110. The encoder may further include a pixel-wise sum operator 154 configured to generate the received features 108 by performing a pixel-wise sum of the LR NFV image 110 and the feature map 152 produced and to pass the received features 108 to the main path 112 and auxiliary path 114 via an output 156 of the encoder 146. The main path 112 and auxiliary path 114 are split at the output 156. [0041] The auxiliary-path features 116 produced and fed by the auxiliary path 114 into the main path 112 increase high-frequency information of the face 106 in the HR FV image 104. The high-frequency information may be related to a periocular, nose, or mouth region of the face 106, or combination thereof. [0042] FIG. 1C-2 is a block diagram of an example embodiment of the apparatus 100 of FIG. 1A in an operational mode. As such, elements of the apparatus 100 shown in FIG. 1C-1 that are used in the training mode of the apparatus 100 (e.g., the convolutional layer 117, ground- truth images 128 and 145, etc.) are not included in the block diagram of FIG. 1C-2 as such elements are not employed in the operational mode.)
1. and the decoder restores a final facial image. (Fu: [0035] Continuing with FIG. 1C-1, the decoder 102 includes the main path 112 and the auxiliary path 114. The auxiliary path 114 is configured to produce auxiliary-path features 116 from the received features 108 and feed the auxiliary-path features 116 produced into the main path 112 for synthesizing the HR FV image 104. The auxiliary-path features 116 represent a HR NFV image 144 of the face 106. The main path 112 includes multiple successive main path convolutional stages 118 a, 118 b, 118 c of respective successive convolutional layers. [0042] FIG. 1C-2 is a block diagram of an example embodiment of the apparatus 100 of FIG. 1A in an operational mode. As such, elements of the apparatus 100 shown in FIG. 1C-1 that are used in the training mode of the apparatus 100 (e.g., the convolutional layer 117, ground- truth images 128 and 145, etc.) are not included in the block diagram of FIG. 1C-2 as such elements are not employed in the operational mode. As in FIG. 1C-1, disclosed above, the apparatus 100 in FIG. 1C-2 comprises a decoder 102 configured to synthesize the HR FV 104 of the face 106 from received features 108 of the LR NFV image 110 of the face 106. The HR FV image 104 is of the higher resolution relative to the lower resolution of the LR NFV image 110. The decoder 102 includes the main path 112 and the auxiliary path 114. The auxiliary path 114 is configured to produce the auxiliary-path features 116 from the received features 108 and feed the auxiliary-path features 116 produced into the main path 112 for synthesizing the HR FV image 104. The auxiliary-path features 116 represent the HR NFV image 144 (i.e., ISP), disclosed above, namely a synthesized HR image of a NFV of the face 106 from the LR NFV image 110 of the face 106.)
3. The apparatus for reconstructing facial images according to claim 1, wherein the generator network is trained with a style consistency loss function, an adversarial loss function, and an identity preserving loss function, (Examiner Note: Lu’s teachings for three different parameters across a given HR FV image and a ground truth frontal facial image and Lu clearly teaches that the style consistency loss function is represented by the pixel and local level losses; while the global level losses include both adversarial and identify preserving loss functions. “The pixel-level and local-level losses represent differences between corresponding pixels and corresponding patches, respectively, of the given HR FV and ground-truth, frontal face images. The global-level losses include adversarial and identity-preserving losses” [0004, claim 2]. Fu: [0004] The main path includes multiple successive main path convolutional stages of respective successive convolutional layers. The multiple successive main path convolutional stages are configured to increase resolution of the received features of the LR NFV image successively. The multiple successive main path convolutional stages include weights that may be trained based on back-propagated pixel-level, local-level, and global-level losses. The back-propagated pixel-level, local-level, and global-level losses are determined based on differences between a given HR FV image and a ground-truth, frontal face image. The given HR FV image is synthesized by the apparatus in a training phase of the apparatus. The pixel-level and local-level losses represent differences between corresponding pixels and corresponding patches, respectively, of the given HR FV and ground-truth, frontal face images. The global-level losses include adversarial and identity-preserving losses. [0034] Continuing with reference to FIG. 1C-1, the block diagram includes a legend 101 of symbols employed in the block diagram. As disclosed above with reference to FIG. 1A, the apparatus 100 comprises a decoder 102 configured to synthesize a HR FV 104 of a face 106 from received features 108 of a LR NFV image 110 of the face 106. The HR FV image 104 is of a higher resolution relative to a lower resolution of the LR NFV image 110. The block diagram of FIG. 1C-1 discloses a framework of the apparatus 100 wherein, given a non-frontal (i.e., profile) LR face ILP, that is, the LR NFV image 110, single-input (SI) SRGAN, disclosed in detail further below, synthesizes a high-quality frontal face ISF, that is the HR FV image 104, by integrating a side-view SR, that is, the auxiliary path 114. Further, the three-level loss, namely, pixel-level 122, local-level 124, and global-level 126 losses (i.e., Lpix, Lpatch, and Lglobal, respectively), provides fine-to-coarse coverage, as disclosed further below.)
3. wherein the style consistency loss function allows the encoder to analyze the occluded image to extract feature values, wherein the adversarial loss function ensures that the reconstructed image is indistinguishable from the original image in visual terms, and wherein the identity preserving loss function ensures that the identity of the reconstructed face is as similar as possible to the identity of the original image. (Fu: [0004] The main path includes multiple successive main path convolutional stages of respective successive convolutional layers. The multiple successive main path convolutional stages are configured to increase resolution of the received features of the LR NFV image successively. The multiple successive main path convolutional stages include weights that may be trained based on back-propagated pixel-level, local-level, and global-level losses. The back-propagated pixel-level, local-level, and global-level losses are determined based on differences between a given HR FV image and a ground-truth, frontal face image. The given HR FV image is synthesized by the apparatus in a training phase of the apparatus. The pixel-level and local-level losses represent differences between corresponding pixels and corresponding patches, respectively, of the given HR FV and ground-truth, frontal face images. The global-level losses include adversarial and identity-preserving losses. [0034] Continuing with reference to FIG. 1C-1, the block diagram includes a legend 101 of symbols employed in the block diagram. As disclosed above with reference to FIG. 1A, the apparatus 100 comprises a decoder 102 configured to synthesize a HR FV 104 of a face 106 from received features 108 of a LR NFV image 110 of the face 106. The HR FV image 104 is of a higher resolution relative to a lower resolution of the LR NFV image 110. The block diagram of FIG. 1C-1 discloses a framework of the apparatus 100 wherein, given a non-frontal (i.e., profile) LR face ILP, that is, the LR NFV image 110, single-input (SI) SRGAN, disclosed in detail further below, synthesizes a high-quality frontal face ISF, that is the HR FV image 104, by integrating a side-view SR, that is, the auxiliary path 114. Further, the three-level loss, namely, pixel-level 122, local-level 124, and global-level 126 losses (i.e., Lpix, Lpatch, and Lglobal, respectively), provides fine-to-coarse coverage, as disclosed further below.)
It would have been obvious before the effective filing date of the claimed invention to one of ordinary skill in the art to Sidiya’s method and system for restoring and image reconstruction using a generative adversarial network to leverage the loss parameters disclosed by Fu for a method and system for frontal face synthesis. The determination of obviousness is predicated upon the following findings: they are both directed towards the same field of endeavor of facial image reconstruction and synthesis using a generative adversarial network and lend themselves for combination. One skilled in the art would have been motivated to modify Sidiya in order to leverage the loss parameters presented by Fu, in order to ensure a higher accuracy in machine learning and facial image reconstruction. Furthermore, the prior art collectively includes each element claimed (though not all in the same reference), and one of ordinary skill in the art could have combined the elements in the manner explained above using known engineering design, interface and/or programming techniques, without changing a “fundamental” operating principle of Sidiya, while the teaching of Fu continues to perform the same function as originally taught prior to being combined, in order to produce the repeatable and predictable result of taking into account pixel-level, local level, adversarial and identity-preserving losses in order to generate a higher quality facial reconstruction for machine learning and training as suggested by Fu. It is for at least the aforementioned reasons that the examiner has reached a conclusion of obviousness with respect to the claim in question.
Consider Claim 2.
The combination of Sidiya and Fu teaches:
2. The apparatus for reconstructing facial images according to claim 1, wherein the encoder and the decoder are connected in a skip-connection manner using a spatial style map, and wherein the spatial style map is generated by inputting a style value into a mapping network. (Sidya: [0060] FIG. 3 is a block diagram illustrating a transformer block in the neural network system shown in FIG. 1 , FIG. 2 or FIG. 5 in accordance with an example of the present disclosure. As shown in FIG. 3 , the transformer block 300 includes a self-attention layer 301 with a skip connection, a convolution layer 302, a Leaky Rectified Linear Activation (LReLU) layer 303, and a convolution layer 304. The LReFU layer 303 is sandwiched between the convolution layer 302 and the convolution layer 304. [0061] The output and input of the self-attention layer 301 are added to each other using a skip connection and the added result passed through a residual block to form the overall operations of the transformer block 301. For example, the added result is then sent to the convolution layer 302. The convolution layer 302 generates a first convolution output and sends the first convolution output to the LReFU layer 303. Further, the LReFU layer 303 generates an LReFU output and sends the LReFU output to the convolution layer 304, and the convolution layer 304 generates a second convolution output. The input of the convolution layer 302 and the second convolution output of the convolution layer 304 are added to each other using a skip connection to generate an output of the transformer block 300. [0062] FIG. 4 is a block diagram illustrating a self-attention layer in the transformer block shown in FIG. 3 in accordance with an example of the present disclosure. The self-attention layer 301 may include a plurality of projection layers, e.g., separable depth-wise convolution layers, each of which respectively learns query, key, and value features. The query, key, and value features may be embeddings related to inputs of the self-attention layer. The outputs of the projection layers are divided into small patches through a patch division layer 402. K, Q and V may be respectively matrices of a set of key features, query features and value features. After division, the key features K is transposed using a transpose layer 403, the query features Q and the transpose of key features K are multiplied, and an attention map is obtained through a softmax layer 404. Moreover, the attention map is multiplied by the value features V and the output is merged using an inverse of the patch division operation through a patch merge layer 405 and a final convolution is applied using a convolution layer 406 to generate the output of the self-attention layer 301. The patch division layer 402 divides feature maps to patch block so as to reduce the computational cost without losing results performance. Fu: [0040] The apparatus 100 further comprises an encoder 146. The encoder 146 may include a combination 148 of a plurality of convolutional layers configured to produce a feature map 152 of features extracted from the LR NFV image 110. The encoder may further include a pixel-wise sum operator 154 configured to generate the received features 108 by performing a pixel-wise sum of the LR NFV image 110 and the feature map 152 produced and to pass the received features 108 to the main path 112 and auxiliary path 114 via an output 156 of the encoder 146. The main path 112 and auxiliary path 114 are split at the output 156. [0041] The auxiliary-path features 116 produced and fed by the auxiliary path 114 into the main path 112 increase high-frequency information of the face 106 in the HR FV image 104. The high-frequency information may be related to a periocular, nose, or mouth region of the face 106, or combination thereof. [0076])
Consider Claim 3.
The combination of Sydia and Fu teaches:
3. The apparatus for reconstructing facial images according to claim 1, wherein the generator network is trained with a style consistency loss function, an adversarial loss function, and an identity preserving loss function, wherein the style consistency loss function allows the encoder to analyze the occluded image to extract feature values, wherein the adversarial loss function ensures that the reconstructed image is indistinguishable from the original image in visual terms, and wherein the identity preserving loss function ensures that the identity of the reconstructed face is as similar as possible to the identity of the original image. (Sydia: [0030] A CNN that incorporates a pre-trained network of StyleGAN may achieve great results of image super-resolution. The CNN may be composed of encoder and decoder networks separated by trained weights of a generative model. Both encoder and decoder are built with successive convolution layers. In addition, the decoder may contain a pixel shuffle layers to up-sample input features. [0031] The prior information is combined by adding skip connection or concatenation operation between the encoder and the pretrained StyleGAN network as well as between the GAN prior network and the decoder. The network is trained end to end with perceptual loss, mean square loss and cross entropy loss. It is trained for 200 thousand iterations. Such CNN may lack ability to utilize non-local information for face reconstruction. [0063] In some examples, during the training of the neural network system, the weights of the generative prior network may be kept fixed. The neural network system is trained for an up-sampling factor of 4 from 64×64 to 256×256. The neural network system is trained for 200,000 iterations using mean square loss, perceptual loss and cross entropy loss. [0064] In some examples, the dataset used to train the neural network system is a synthetic dataset, composed of paired low-resolution and high-resolution image faces which simulate degradation found in real-world face images. FIG. 6 shows comparison among output images obtained respectively through bicubic up-sampling, PSFRGAN, GFP-GAN and the neural network system in accordance with an example of the present disclosure. As shown in FIG. 6, 601 shows an output image obtained using bicubic up-sampling, 602 shows an output image obtained using PSFRGAN, 603 shows an output image using GFP-GAN, and 604 shows an output image obtained using the neural network system in accordance with the present disclosure. Fu: [0004] The main path includes multiple successive main path convolutional stages of respective successive convolutional layers. The multiple successive main path convolutional stages are configured to increase resolution of the received features of the LR NFV image successively. The multiple successive main path convolutional stages include weights that may be trained based on back-propagated pixel-level, local-level, and global-level losses. The back-propagated pixel-level, local-level, and global-level losses are determined based on differences between a given HR FV image and a ground-truth, frontal face image. The given HR FV image is synthesized by the apparatus in a training phase of the apparatus. The pixel-level and local-level losses represent differences between corresponding pixels and corresponding patches, respectively, of the given HR FV and ground-truth, frontal face images. The global-level losses include adversarial and identity-preserving losses. [0034] Continuing with reference to FIG. 1C-1, the block diagram includes a legend 101 of symbols employed in the block diagram. As disclosed above with reference to FIG. 1A, the apparatus 100 comprises a decoder 102 configured to synthesize a HR FV 104 of a face 106 from received features 108 of a LR NFV image 110 of the face 106. The HR FV image 104 is of a higher resolution relative to a lower resolution of the LR NFV image 110. The block diagram of FIG. 1C-1 discloses a framework of the apparatus 100 wherein, given a non-frontal (i.e., profile) LR face ILP, that is, the LR NFV image 110, single-input (SI) SRGAN, disclosed in detail further below, synthesizes a high-quality frontal face ISF, that is the HR FV image 104, by integrating a side-view SR, that is, the auxiliary path 114. Further, the three-level loss, namely, pixel-level 122, local-level 124, and global-level 126 losses (i.e., Lpix, Lpatch, and Lglobal, respectively), provides fine-to-coarse coverage, as disclosed further below.)
Consider Claim 4 and 6.
Sidiya teaches:
4. A method for reconstructing facial images, comprising the steps of: (Sidiya: abstract, A neural network system for restoring images, a method and a non-transitory computer-readable storage medium thereof are provided. The neural network system includes an encoder and a generative adversarial network (GAN) prior network. The encoder includes a plurality of encoder blocks, where each encoder block includes at least one transformer block and one convolution layer, where the encoder receives an input image and generates a plurality of encoder features and a plurality of latent vectors. Additionally, the GAN prior network includes a plurality of pre-trained generative prior layers, where the GAN prior network receives the plurality of encoder features and the plurality of latent vectors from the encoder and generates an output image with super-resolution. [0035]-[0044], Figure 1, [0044] As shown in FIG. 1 , the GAN prior network 102 includes a plurality of generative prior layers. The plurality of generative prior layers may include generative prior layers 102-1, 102-2, 102-3, . . . , and 102-7 that are stacked to each other. The number of the plurality of generative prior layers is not limited to 7. FIG. 1 is only for illustrating.)
4. analysing, by an encoder of a generator network, (Sidiya: [0035]-[0044], Figure 1, [0045] The generative prior layer 102-1 receives inputs including the encoder features f5 from the encoder block 101-5, the encoder features f6 from the encoder block 101-6, and the latent vector c1 from the fully connected layer FC 103, and then generates an output feature. The generative prior layer 102-2 receives the output feature from the generative prior layer 102-1. In addition to the output feature of the generative prior layer 102-1, the generative prior layer 102-2 receives the encoder features f4 from the encoder block 101-4 and the latent vector c2 from the fully connected layer FC 103. After receiving the inputs, the generative prior layer 102-2 generates an output feature and sends the output feature to the generative prior layer 102-3 that subsequently follows the generative prior layer 102-2. [0052] FIG. 2 is a block diagram illustrating a neural network system including an encoder with transformer blocks, a GAN prior network, and a decoder in accordance with one or more examples of the present disclosure. In addition to the encoder and the GAN prior network, the neural network system in FIG. 2 includes a decoder as well. The overall architecture of the neural network system in FIG. 2 includes the encoder and the decoder that separated with the trained weights of the GAN prior network. The GAN prior network is connected to the encoder and the decoder with skip connections. The encoder network is built using successive transformer blocks composed of self-attention layers and residual blocks. The decoder network includes convolution layers followed by pixel shuffle layers for features up-sampling. The output of each encoder block with a specific resolution is concatenated with the output of the corresponding block in the GAN prior network, then a convolution layer followed by a transformer block is applied to the results and the output is fed to the next block in the GAN prior network. In addition, the output of the last layer of the GAN prior network is used as an input to the decoder with the output of the initial encoder block in the encoder.)
4. an occluded image to extract feature values, (Sidiya: [0037], [0038] The encoder block 101-1 receives an input image having a low-resolution and extracts encoder features f1 from the input image. The input image may be a face image. The encoder features f1 are sent to both the GAN prior network 102 and the encoder block 101-2 that subsequently follows the encoder block 101-1. In an example, the encoder features f1 may have a resolution of 64×64 as shown in FIG. 1 . In the encoder block 101-1, the convolution layer EC 1 and the plurality of transformer blocks T11-T16 are stacked to each other, and the convolution layer EC 1 is followed by the plurality of transformer blocks T11-T16. The number of the plurality of transformer blocks in the encoder 101 is not limited to 6. [0039] The encoder block 101-2 receives the encoder features f1 from the encoder block 101-1 and generates the encoder features f2. The encoder features f2 are sent to both the GAN prior network 102 and the encoder block 101-3 that subsequently follows the encoder block 101-2. In an example, the encoder features f2 may have a resolution of 32×32 as shown in FIG. 1 . In the encoder block 101-2, the convolution layer EC 2 is followed by the transformer block T21.)
4. and reconstructing, by a decoder of the generator network, a final facial image. (Sidiya: [0052], Figure 2, [0054] The decoder 204 includes a plurality of decoder blocks. The plurality of decoder blocks include the decoder blocks 204-1, 204-2, and 204-3 as shown in FIG. 2 . Each decoder block include a convolution layer and a pixel shuffle layer that follows the convolution layer. For example, the decoder block 204-1 includes a convolution layer 2041-1 and a pixel shuffle layer 2041-2, the decoder block 204-2 includes a convolution layer 2042-1 and a pixel shuffle layer 2042-2, the decoder block 204-3 includes a convolution layer 2043-1 and a pixel shuffle layer 2043-2. [0055] The convolution layer 2041-1 in the decoder block 204-1 receives inputs including the output feature from the generative prior layer 202-7 and the encoder feature f1, and then generates an output feature. The pixel shuffle layer 2041-2 receives the output feature of the convolution layer 2041-1 and up-samples the output feature. For example, the pixel shuffle layer 2041-2 up-samples the output feature of the convolution layer 2041-1 to 64×64 and sends the up-sampled feature to the decoder block 204-2 that follows the decoder block 204-1. [0056]-[0060])
6. The method for reconstructing facial images according to claim 4, wherein the generator network is trained with three loss functions. (Sydia: [0030] A CNN that incorporates a pre-trained network of StyleGAN may achieve great results of image super-resolution. The CNN may be composed of encoder and decoder networks separated by trained weights of a generative model. Both encoder and decoder are built with successive convolution layers. In addition, the decoder may contain a pixel shuffle layers to up-sample input features. [0031] The prior information is combined by adding skip connection or concatenation operation between the encoder and the pretrained StyleGAN network as well as between the GAN prior network and the decoder. The network is trained end to end with perceptual loss, mean square loss and cross entropy loss. It is trained for 200 thousand iterations. Such CNN may lack ability to utilize non-local information for face reconstruction. [0063] In some examples, during the training of the neural network system, the weights of the generative prior network may be kept fixed. The neural network system is trained for an up-sampling factor of 4 from 64×64 to 256×256. The neural network system is trained for 200,000 iterations using mean square loss, perceptual loss and cross entropy loss. [0064] In some examples, the dataset used to train the neural network system is a synthetic dataset, composed of paired low-resolution and high-resolution image faces which simulate degradation found in real-world face images. FIG. 6 shows comparison among output images obtained respectively through bicubic up-sampling, PSFRGAN, GFP-GAN and the neural network system in accordance with an example of the present disclosure. As shown in FIG. 6, 601 shows an output image obtained using bicubic up-sampling, 602 shows an output image obtained using PSFRGAN, 603 shows an output image using GFP-GAN, and 604 shows an output image obtained using the neural network system in accordance with the present disclosure.)
Sydia does not teach the limitations from Claim 3 that the three loss functions are:
6. The method for reconstructing facial images according to claim 4, wherein the generator network is trained with a style consistency loss function, an adversarial loss function, and an identity preserving loss function, wherein the style consistency loss function allows the encoder to analyze the occluded image to extract feature values, wherein the adversarial loss function ensures that the reconstructed image is indistinguishable from the original image in visual terms, and wherein the identity preserving loss function ensures that the identity of the reconstructed face is as similar as possible to the identity of the original image. (Sydia: [0030] A CNN that incorporates a pre-trained network of StyleGAN may achieve great results of image super-resolution. The CNN may be composed of encoder and decoder networks separated by trained weights of a generative model. Both encoder and decoder are built with successive convolution layers. In addition, the decoder may contain a pixel shuffle layers to up-sample input features. [0031] The prior information is combined by adding skip connection or concatenation operation between the encoder and the pretrained StyleGAN network as well as between the GAN prior network and the decoder. The network is trained end to end with perceptual loss, mean square loss and cross entropy loss. It is trained for 200 thousand iterations. Such CNN may lack ability to utilize non-local information for face reconstruction. [0063] In some examples, during the training of the neural network system, the weights of the generative prior network may be kept fixed. The neural network system is trained for an up-sampling factor of 4 from 64×64 to 256×256. The neural network system is trained for 200,000 iterations using mean square loss, perceptual loss and cross entropy loss. [0064] In some examples, the dataset used to train the neural network system is a synthetic dataset, composed of paired low-resolution and high-resolution image faces which simulate degradation found in real-world face images. FIG. 6 shows comparison among output images obtained respectively through bicubic up-sampling, PSFRGAN, GFP-GAN and the neural network system in accordance with an example of the present disclosure. As shown in FIG. 6, 601 shows an output image obtained using bicubic up-sampling, 602 shows an output image obtained using PSFRGAN, 603 shows an output image using GFP-GAN, and 604 shows an output image obtained using the neural network system in accordance with the present disclosure. Fu: [0004] The main path includes multiple successive main path convolutional stages of respective successive convolutional layers. The multiple successive main path convolutional stages are configured to increase resolution of the received features of the LR NFV image successively. The multiple successive main path convolutional stages include weights that may be trained based on back-propagated pixel-level, local-level, and global-level losses. The back-propagated pixel-level, local-level, and global-level losses are determined based on differences between a given HR FV image and a ground-truth, frontal face image. The given HR FV image is synthesized by the apparatus in a training phase of the apparatus. The pixel-level and local-level losses represent differences between corresponding pixels and corresponding patches, respectively, of the given HR FV and ground-truth, frontal face images. The global-level losses include adversarial and identity-preserving losses. [0034] Continuing with reference to FIG. 1C-1, the block diagram includes a legend 101 of symbols employed in the block diagram. As disclosed above with reference to FIG. 1A, the apparatus 100 comprises a decoder 102 configured to synthesize a HR FV 104 of a face 106 from received features 108 of a LR NFV image 110 of the face 106. The HR FV image 104 is of a higher resolution relative to a lower resolution of the LR NFV image 110. The block diagram of FIG. 1C-1 discloses a framework of the apparatus 100 wherein, given a non-frontal (i.e., profile) LR face ILP, that is, the LR NFV image 110, single-input (SI) SRGAN, disclosed in detail further below, synthesizes a high-quality frontal face ISF, that is the HR FV image 104, by integrating a side-view SR, that is, the auxiliary path 114. Further, the three-level loss, namely, pixel-level 122, local-level 124, and global-level 126 losses (i.e., Lpix, Lpatch, and Lglobal, respectively), provides fine-to-coarse coverage, as disclosed further below.)
Fu teaches:
4. A method for reconstructing facial images, comprising the steps of: (Fu: abstract, An apparatus and corresponding method for frontal face synthesis. The apparatus comprises a decoder that synthesizes a high-resolution (HR) frontal-view (FV) image of a face from received features of a low-resolution (LR) non-frontal-view (NFV) image of the face. The HR FV image is of a higher resolution relative to a lower resolution of the LR NFV image. The decoder includes a main path and an auxiliary path. The auxiliary path produces auxiliary-path features from the received features and feeds the auxiliary-path features produced into the main path for synthesizing the HR FV image. The auxiliary-path features represent a HR NFV image of the face at the higher resolution. As such, an HR identity-preserved frontal face can be synthesized from one or many LR faces with various poses and may be used in types of commercial applications, such as video surveillance. [0029]-[0042], Figures 1A-C;)
4. analysing, by an encoder of a generator network, (Fu: [0027], An example embodiment enables frontal view synthesis from single or multiple low-resolution (LR) faces with various poses. Generally speaking, an example embodiment may be directed to a super-resolution (SR) integrated generative adversarial network (SRGAN) that learns face frontalization and super-resolution collaboratively to synthesize high-quality, identity-preserved frontal faces, as disclosed further below. Super-resolution recovers a high-resolution image from a low-resolution image by upscaling and/or improving details within the low-resolution image. An example embodiment learns a generator network, such as disclosed below with regard to FIG. 1C-1, that includes a deep encoder and a SR integrated decoder. Features extracted by the deep encoder are passed to the decoder for reconstruction. An example embodiment of a decoder is specially designed to first super-resolve (i.e., recover a high-resolution image from a low-resolution image) non-frontal-view (NFV) images, such as side-view (SV) images, and ultimately utilize the information to reconstruct high-resolution HR frontal-view (FV) faces. To train the model, a three-level loss (i.e., pixel, patch, and global) provides fine-to-coarse coverage that learns a precise non-linear transformation between a LR NFV image of face(s) and an HR FV image of the face(s). Moreover, SRGAN accepts multiple LR profile faces as input by adding an orthogonal constraint in the generator to penalize redundant latent representations and, hence, diversify the learned features space. With these techniques, an example embodiment can generate frontal faces faithful to ground-truth, as disclosed in further detail below. [0029]-[0042], Figures 1A-C; [0029] FIG. 1A is block diagram of an example embodiment of an apparatus 100 for frontal face synthesis. The apparatus 100 may be referred to interchangeably herein as a super-resolution (SR) integrated generative adversarial network (SRGAN). The apparatus 100 comprises a decoder 102 configured to synthesize a high-resolution (HR) frontal-view (FV) image 104 of a face 106 from received features 108 of a low-resolution (LR) non-frontal-view (NFV) image 110 of the face 106. The HR FV image 104 may also be referred to interchangeably herein as a super-resolved front face ISF. The HR FV image 104 is of a higher resolution relative to a lower resolution of the LR NFV image 110. The decoder 102 includes a main path 112 and an auxiliary path 114. The auxiliary path 114 is configured to produce auxiliary-path features 116 from the received features 108 and feed the auxiliary-path features 116 produced into the main path 112 for synthesizing the HR FV image 104. The auxiliary-path features represent a HR NFV image 144 of the face 106 at the higher resolution. The auxiliary path 114 may be referred to interchangeably herein as a super-resolution (SR) module or side-view SR branch. Super-resolution includes recovering a high-resolution image from a low-resolution image by upscaling and/or improving details within the low-resolution image. The auxiliary path may be considered to be a “branch” as it splits off from the main path 112.)
4. an occluded image to extract feature values; (Fu: [0040] The apparatus 100 further comprises an encoder 146. The encoder 146 may include a combination 148 of a plurality of convolutional layers configured to produce a feature map 152 of features extracted from the LR NFV image 110. The encoder may further include a pixel-wise sum operator 154 configured to generate the received features 108 by performing a pixel-wise sum of the LR NFV image 110 and the feature map 152 produced and to pass the received features 108 to the main path 112 and auxiliary path 114 via an output 156 of the encoder 146. The main path 112 and auxiliary path 114 are split at the output 156. [0041] The auxiliary-path features 116 produced and fed by the auxiliary path 114 into the main path 112 increase high-frequency information of the face 106 in the HR FV image 104. The high-frequency information may be related to a periocular, nose, or mouth region of the face 106, or combination thereof. [0042] FIG. 1C-2 is a block diagram of an example embodiment of the apparatus 100 of FIG. 1A in an operational mode. As such, elements of the apparatus 100 shown in FIG. 1C-1 that are used in the training mode of the apparatus 100 (e.g., the convolutional layer 117, ground- truth images 128 and 145, etc.) are not included in the block diagram of FIG. 1C-2 as such elements are not employed in the operational mode.)
4. and reconstructing, by a decoder of the generator network, a final facial image. (Fu: [0035] Continuing with FIG. 1C-1, the decoder 102 includes the main path 112 and the auxiliary path 114. The auxiliary path 114 is configured to produce auxiliary-path features 116 from the received features 108 and feed the auxiliary-path features 116 produced into the main path 112 for synthesizing the HR FV image 104. The auxiliary-path features 116 represent a HR NFV image 144 of the face 106. The main path 112 includes multiple successive main path convolutional stages 118 a, 118 b, 118 c of respective successive convolutional layers. [0042] FIG. 1C-2 is a block diagram of an example embodiment of the apparatus 100 of FIG. 1A in an operational mode. As such, elements of the apparatus 100 shown in FIG. 1C-1 that are used in the training mode of the apparatus 100 (e.g., the convolutional layer 117, ground- truth images 128 and 145, etc.) are not included in the block diagram of FIG. 1C-2 as such elements are not employed in the operational mode. As in FIG. 1C-1, disclosed above, the apparatus 100 in FIG. 1C-2 comprises a decoder 102 configured to synthesize the HR FV 104 of the face 106 from received features 108 of the LR NFV image 110 of the face 106. The HR FV image 104 is of the higher resolution relative to the lower resolution of the LR NFV image 110. The decoder 102 includes the main path 112 and the auxiliary path 114. The auxiliary path 114 is configured to produce the auxiliary-path features 116 from the received features 108 and feed the auxiliary-path features 116 produced into the main path 112 for synthesizing the HR FV image 104. The auxiliary-path features 116 represent the HR NFV image 144 (i.e., ISP), disclosed above, namely a synthesized HR image of a NFV of the face 106 from the LR NFV image 110 of the face 106.)
6. The method for reconstructing facial images according to claim 4, wherein the generator network is trained with a style consistency loss function, an adversarial loss function, and an identity preserving loss function, (Examiner Note: Lu’s teachings for three different parameters across a given HR FV image and a ground truth frontal facial image and Lu clearly teaches that the style consistency loss function is represented by the pixel and local level losses; while the global level losses include both adversarial and identify preserving loss functions. “The pixel-level and local-level losses represent differences between corresponding pixels and corresponding patches, respectively, of the given HR FV and ground-truth, frontal face images. The global-level losses include adversarial and identity-preserving losses” [0004, claim 2]. Fu: [0004] The main path includes multiple successive main path convolutional stages of respective successive convolutional layers. The multiple successive main path convolutional stages are configured to increase resolution of the received features of the LR NFV image successively. The multiple successive main path convolutional stages include weights that may be trained based on back-propagated pixel-level, local-level, and global-level losses. The back-propagated pixel-level, local-level, and global-level losses are determined based on differences between a given HR FV image and a ground-truth, frontal face image. The given HR FV image is synthesized by the apparatus in a training phase of the apparatus. The pixel-level and local-level losses represent differences between corresponding pixels and corresponding patches, respectively, of the given HR FV and ground-truth, frontal face images. The global-level losses include adversarial and identity-preserving losses. [0034] Continuing with reference to FIG. 1C-1, the block diagram includes a legend 101 of symbols employed in the block diagram. As disclosed above with reference to FIG. 1A, the apparatus 100 comprises a decoder 102 configured to synthesize a HR FV 104 of a face 106 from received features 108 of a LR NFV image 110 of the face 106. The HR FV image 104 is of a higher resolution relative to a lower resolution of the LR NFV image 110. The block diagram of FIG. 1C-1 discloses a framework of the apparatus 100 wherein, given a non-frontal (i.e., profile) LR face ILP, that is, the LR NFV image 110, single-input (SI) SRGAN, disclosed in detail further below, synthesizes a high-quality frontal face ISF, that is the HR FV image 104, by integrating a side-view SR, that is, the auxiliary path 114. Further, the three-level loss, namely, pixel-level 122, local-level 124, and global-level 126 losses (i.e., Lpix, Lpatch, and Lglobal, respectively), provides fine-to-coarse coverage, as disclosed further below.)
6. wherein the style consistency loss function allows the encoder to analyze the occluded image to extract feature values, wherein the adversarial loss function ensures that the reconstructed image is indistinguishable from the original image in visual terms, and wherein the identity preserving loss function ensures that the identity of the reconstructed face is as similar as possible to the identity of the original image. (Fu: [0004] The main path includes multiple successive main path convolutional stages of respective successive convolutional layers. The multiple successive main path convolutional stages are configured to increase resolution of the received features of the LR NFV image successively. The multiple successive main path convolutional stages include weights that may be trained based on back-propagated pixel-level, local-level, and global-level losses. The back-propagated pixel-level, local-level, and global-level losses are determined based on differences between a given HR FV image and a ground-truth, frontal face image. The given HR FV image is synthesized by the apparatus in a training phase of the apparatus. The pixel-level and local-level losses represent differences between corresponding pixels and corresponding patches, respectively, of the given HR FV and ground-truth, frontal face images. The global-level losses include adversarial and identity-preserving losses. [0034] Continuing with reference to FIG. 1C-1, the block diagram includes a legend 101 of symbols employed in the block diagram. As disclosed above with reference to FIG. 1A, the apparatus 100 comprises a decoder 102 configured to synthesize a HR FV 104 of a face 106 from received features 108 of a LR NFV image 110 of the face 106. The HR FV image 104 is of a higher resolution relative to a lower resolution of the LR NFV image 110. The block diagram of FIG. 1C-1 discloses a framework of the apparatus 100 wherein, given a non-frontal (i.e., profile) LR face ILP, that is, the LR NFV image 110, single-input (SI) SRGAN, disclosed in detail further below, synthesizes a high-quality frontal face ISF, that is the HR FV image 104, by integrating a side-view SR, that is, the auxiliary path 114. Further, the three-level loss, namely, pixel-level 122, local-level 124, and global-level 126 losses (i.e., Lpix, Lpatch, and Lglobal, respectively), provides fine-to-coarse coverage, as disclosed further below.)
It would have been obvious before the effective filing date of the claimed invention to one of ordinary skill in the art to Sidiya’s method and system for restoring and image reconstruction using a generative adversarial network to leverage the loss parameters disclosed by Fu for a method and system for frontal face synthesis. The determination of obviousness is predicated upon the following findings: they are both directed towards the same field of endeavor of facial image reconstruction and synthesis using a generative adversarial network and lend themselves for combination. One skilled in the art would have been motivated to modify Sidiya in order to leverage the loss parameters presented by Fu, in order to ensure a higher accuracy in machine learning and facial image reconstruction. Furthermore, the prior art collectively includes each element claimed (though not all in the same reference), and one of ordinary skill in the art could have combined the elements in the manner explained above using known engineering design, interface and/or programming techniques, without changing a “fundamental” operating principle of Sidiya, while the teaching of Fu continues to perform the same function as originally taught prior to being combined, in order to produce the repeatable and predictable result of taking into account pixel-level, local level, adversarial and identity-preserving losses in order to generate a higher quality facial reconstruction for machine learning and training as suggested by Fu. It is for at least the aforementioned reasons that the examiner has reached a conclusion of obviousness with respect to the claim in question.
Consider Claim 5.
The combination of Sidiya and Fu teaches:
5. The method for reconstructing facial images according to claim 4, wherein the encoder and the decoder are connected in a skip-connection manner using a spatial style map, and wherein the spatial style map is generated by inputting a style value into a mapping network. (Sidya: [0060] FIG. 3 is a block diagram illustrating a transformer block in the neural network system shown in FIG. 1 , FIG. 2 or FIG. 5 in accordance with an example of the present disclosure. As shown in FIG. 3 , the transformer block 300 includes a self-attention layer 301 with a skip connection, a convolution layer 302, a Leaky Rectified Linear Activation (LReLU) layer 303, and a convolution layer 304. The LReFU layer 303 is sandwiched between the convolution layer 302 and the convolution layer 304. [0061] The output and input of the self-attention layer 301 are added to each other using a skip connection and the added result passed through a residual block to form the overall operations of the transformer block 301. For example, the added result is then sent to the convolution layer 302. The convolution layer 302 generates a first convolution output and sends the first convolution output to the LReFU layer 303. Further, the LReFU layer 303 generates an LReFU output and sends the LReFU output to the convolution layer 304, and the convolution layer 304 generates a second convolution output. The input of the convolution layer 302 and the second convolution output of the convolution layer 304 are added to each other using a skip connection to generate an output of the transformer block 300. [0062] FIG. 4 is a block diagram illustrating a self-attention layer in the transformer block shown in FIG. 3 in accordance with an example of the present disclosure. The self-attention layer 301 may include a plurality of projection layers, e.g., separable depth-wise convolution layers, each of which respectively learns query, key, and value features. The query, key, and value features may be embeddings related to inputs of the self-attention layer. The outputs of the projection layers are divided into small patches through a patch division layer 402. K, Q and V may be respectively matrices of a set of key features, query features and value features. After division, the key features K is transposed using a transpose layer 403, the query features Q and the transpose of key features K are multiplied, and an attention map is obtained through a softmax layer 404. Moreover, the attention map is multiplied by the value features V and the output is merged using an inverse of the patch division operation through a patch merge layer 405 and a final convolution is applied using a convolution layer 406 to generate the output of the self-attention layer 301. The patch division layer 402 divides feature maps to patch block so as to reduce the computational cost without losing results performance. Fu: [0040] The apparatus 100 further comprises an encoder 146. The encoder 146 may include a combination 148 of a plurality of convolutional layers configured to produce a feature map 152 of features extracted from the LR NFV image 110. The encoder may further include a pixel-wise sum operator 154 configured to generate the received features 108 by performing a pixel-wise sum of the LR NFV image 110 and the feature map 152 produced and to pass the received features 108 to the main path 112 and auxiliary path 114 via an output 156 of the encoder 146. The main path 112 and auxiliary path 114 are split at the output 156. [0041] The auxiliary-path features 116 produced and fed by the auxiliary path 114 into the main path 112 increase high-frequency information of the face 106 in the HR FV image 104. The high-frequency information may be related to a periocular, nose, or mouth region of the face 106, or combination thereof. [0076])
Consider Claim 6.
The combination of Sidiya and Fu teaches:
6. The method for reconstructing facial images according to claim 4, wherein the generator network is trained with a style consistency loss function, an adversarial loss function, and an identity preserving loss function, wherein the style consistency loss function allows the encoder to analyze the occluded image to extract feature values, wherein the adversarial loss function ensures that the reconstructed image is indistinguishable from the original image in visual terms, and wherein the identity preserving loss function ensures that the identity of the reconstructed face is as similar as possible to the identity of the original image. (Sydia: [0030] A CNN that incorporates a pre-trained network of StyleGAN may achieve great results of image super-resolution. The CNN may be composed of encoder and decoder networks separated by trained weights of a generative model. Both encoder and decoder are built with successive convolution layers. In addition, the decoder may contain a pixel shuffle layers to up-sample input features. [0031] The prior information is combined by adding skip connection or concatenation operation between the encoder and the pretrained StyleGAN network as well as between the GAN prior network and the decoder. The network is trained end to end with perceptual loss, mean square loss and cross entropy loss. It is trained for 200 thousand iterations. Such CNN may lack ability to utilize non-local information for face reconstruction. [0063] In some examples, during the training of the neural network system, the weights of the generative prior network may be kept fixed. The neural network system is trained for an up-sampling factor of 4 from 64×64 to 256×256. The neural network system is trained for 200,000 iterations using mean square loss, perceptual loss and cross entropy loss. [0064] In some examples, the dataset used to train the neural network system is a synthetic dataset, composed of paired low-resolution and high-resolution image faces which simulate degradation found in real-world face images. FIG. 6 shows comparison among output images obtained respectively through bicubic up-sampling, PSFRGAN, GFP-GAN and the neural network system in accordance with an example of the present disclosure. As shown in FIG. 6, 601 shows an output image obtained using bicubic up-sampling, 602 shows an output image obtained using PSFRGAN, 603 shows an output image using GFP-GAN, and 604 shows an output image obtained using the neural network system in accordance with the present disclosure. Fu: [0004] The main path includes multiple successive main path convolutional stages of respective successive convolutional layers. The multiple successive main path convolutional stages are configured to increase resolution of the received features of the LR NFV image successively. The multiple successive main path convolutional stages include weights that may be trained based on back-propagated pixel-level, local-level, and global-level losses. The back-propagated pixel-level, local-level, and global-level losses are determined based on differences between a given HR FV image and a ground-truth, frontal face image. The given HR FV image is synthesized by the apparatus in a training phase of the apparatus. The pixel-level and local-level losses represent differences between corresponding pixels and corresponding patches, respectively, of the given HR FV and ground-truth, frontal face images. The global-level losses include adversarial and identity-preserving losses. [0034] Continuing with reference to FIG. 1C-1, the block diagram includes a legend 101 of symbols employed in the block diagram. As disclosed above with reference to FIG. 1A, the apparatus 100 comprises a decoder 102 configured to synthesize a HR FV 104 of a face 106 from received features 108 of a LR NFV image 110 of the face 106. The HR FV image 104 is of a higher resolution relative to a lower resolution of the LR NFV image 110. The block diagram of FIG. 1C-1 discloses a framework of the apparatus 100 wherein, given a non-frontal (i.e., profile) LR face ILP, that is, the LR NFV image 110, single-input (SI) SRGAN, disclosed in detail further below, synthesizes a high-quality frontal face ISF, that is the HR FV image 104, by integrating a side-view SR, that is, the auxiliary path 114. Further, the three-level loss, namely, pixel-level 122, local-level 124, and global-level 126 losses (i.e., Lpix, Lpatch, and Lglobal, respectively), provides fine-to-coarse coverage, as disclosed further below.)
Conclusion
The prior art made of record in form PTO-892 and not relied upon is considered pertinent to applicant's disclosure.
PNG
media_image1.png
275
912
media_image1.png
Greyscale
Any inquiry concerning this communication or earlier communications from the examiner should be directed to TAHMINA ANSARI whose telephone number is 571-270-3379. The examiner can normally be reached on IFP Flex - Monday through Friday 9 to 5.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, O’NEAL MISTRY can be reached on 313-446-4912. The fax phone numbers for the organization where this application or proceeding is assigned are 571-273-8300 for regular communications and 571-273-8300 for After Final communications. TC 2600’s customer service number is 571-272-2600.
Any inquiry of a general nature or relating to the status of this application or proceeding should be directed to the receptionist whose telephone number is 571-272-2600.
2674
June 25, 2026
/TAHMINA N ANSARI/Primary Examiner, Art Unit 2674
Prosecution Insights