DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
This office action is in response to Applicant’s communication filed on 5/23/2025. Claims 1-20 have been examined.
Double Patenting
The nonstatutory double patenting rejection is based on a judicially created doctrine grounded in public policy (a policy reflected in the statute) so as to prevent the unjustified or improper timewise extension of the “right to exclude” granted by a patent and to prevent possible harassment by multiple assignees. A nonstatutory obviousness-type double patenting rejection is appropriate where the conflicting claims are not identical, but at least one examined application claim is not patentably distinct from the reference claim(s) because the examined application claim is either anticipated by, or would have been obvious over, the reference claim(s). See, e.g., In re Berg, 140 F.3d 1428, 46 USPQ2d 1226 (Fed. Cir. 1998); In re Goodman, 11 F.3d 1046, 29 USPQ2d 2010 (Fed. Cir. 1993); In re Longi, 759 F.2d 887, 225 USPQ 645 (Fed. Cir. 1985); In re Van Ornum, 686 F.2d 937, 214 USPQ 761 (CCPA 1982); In re Vogel, 422 F.2d 438, 164 USPQ 619 (CCPA 1970); and In re Thorington, 418 F.2d 528, 163 USPQ 644 (CCPA 1969).
A timely filed terminal disclaimer in compliance with 37 CFR 1.321(c) or 1.321(d) may be used to overcome an actual or provisional rejection based on a nonstatutory double patenting ground provided the conflicting application or patent either is shown to be commonly owned with this application, or claims an invention made as a result of activities undertaken within the scope of a joint research agreement.
Effective January 1, 1994, a registered attorney or agent of record may sign a terminal disclaimer. A terminal disclaimer signed by the assignee must fully comply with 37 CFR 3.73(b).
Claims 1,5-8,10-20 are rejected on the ground of nonstatutory obviousness-type double patenting as being unpatentable over claims 1-20 of the Patent No. US 12,321,495 B2
Claims 2-4,9 are rejected on the ground of nonstatutory obviousness-type double patenting as being unpatentable over claims 1-20 of the Patent No. US 12,321,495 B2 in view of Maximov
Below are the analysis to the claims.
Claims of Instant application
Claims of Patent No. US 12.321,495
Claims 1,15
A non-transitory computer readable medium /method
encoding, from a digital image depicting a face of a person, a face anonymization guide comprising features representing one or more of gender, ethnicity, age, or expression associated with the face in the digital image;
generating a masked digital image from the digital image by masking face pixels for the face depicted in the digital image; and
generating, from the masked digital image utilizing a generative neural network, an anonymized digital image by inpainting a synthetic face within the digital image according to the face anonymization guide.
Claims 1,15
A non-transitory computer readable medium/method
encoding, from a digital image depicting a face of a person, a face anonymization guide utilizing a face anonymization neural network comprising a face attribute encoder and a texture encoder to: extract, utilizing the face attribute encoder, a first set of features representing one or more of gender, ethnicity, or age associated with the face in the digital image….;
generating a masked digital image from the digital image by masking face pixels for the face depicted in the digital image; and
generating, from the masked digital image and utilizing a decoder of the face anonymization neural network, an anonymized digital image by replacing the face pixels masked in the digital image with synthetic face pixels synthesized according to the face anonymization guide, wherein generating the anonymized digital image comprises: extracting a masked image vector from the masked digital image; and combining the masked image vector and the face anonymization guide utilizing the face anonymization neural network.
Claim 3
wherein inpainting the synthetic face within the digital image comprising replacing masked face pixels of the masked digital image with synthetic face pixels forming the synthetic face..
Claim 1
generating, from the masked digital image and utilizing a decoder of the face anonymization neural network, an anonymized digital image by replacing the face pixels masked in the digital image with synthetic face pixels synthesized according to the face anonymization guide
Claims 5,20
wherein generating the anonymized digital image comprises: extracting a masked image vector from the masked digital image; and combining the masked image vector and the face anonymization guide utilizing the generative neural network.
Claims 1,15
wherein generating the anonymized digital image comprises: extracting a masked image vector from the masked digital image; and combining the masked image vector and the face anonymization guide utilizing the face anonymization neural network.
Claim 6
wherein generating the anonymized digital image comprises synthesizing synthetic face pixels utilizing a decoder of the generative neural network according to the face anonymization guide.
Claim 6
wherein generating the anonymized digital image comprises synthesizing the synthetic face pixels utilizing the decoder of the face anonymization neural network according to the face anonymization guide
Claim 7
wherein encoding the face anonymization guide comprises: utilizing a first encoder of the generative neural network to extract a first feature set and a second encoder of the generative neural network to extract a second feature set from the digital image; and combining the first feature set and the second feature set into a guide feature set.
Claim 1
encoding, from a digital image depicting a face of a person, a face anonymization guide utilizing a face anonymization neural network comprising a face attribute encoder and a texture encoder to: extract, utilizing the face attribute encoder, a first set of features representing one or more of gender, ethnicity, or age associated with the face in the digital image; extract, utilizing the texture encoder, a second set of features representing image texture of the digital image; and combine the first set of features and the second set of features into the face anonymization guide;
Claim 8
A system comprising: one or more memory devices storing a generative neural network and a digital image depicting a face of a person; and one or more processors configured to cause the system to anonymize the digital image depicting the face of the person by:
generating a face anonymization guide by encoding features representing one or more of gender, ethnicity, age, or expression of the face in the digital image; and
generating, utilizing the generative neural network, an anonymized digital image by inpainting a synthetic face within the digital image according to the face anonymization guide that preserves the one or more of gender, ethnicity, age, or expression of the face in the digital image.
Claim 8
A system comprising: one or more memory devices storing a face anonymization neural network and a digital image depicting a face of a person; and one or more processors configured to cause the system to anonymize the digital image depicting the face of the person by:
encoding, from the digital image depicting the face of the person utilizing the face anonymization neural network, a face anonymization guide utilizing the face anonymization neural network comprising a face attribute encoder and a texture encoder to: extract, utilizing the face attribute encoder, a first set of features representing one or more of gender, ethnicity, or age associated with the face in the digital image; generating, from the masked digital image and utilizing a decoder of the face anonymization neural network, an anonymized digital image by replacing the face pixels masked in the digital image with synthetic face pixels synthesized according to the face anonymization guide,
Claim 10,
wherein generating the face anonymization guide comprises utilizing a first encoder of the generative neural network to encode the features.
Claim 8,
encoding, from the digital image depicting the face of the person utilizing the face anonymization neural network, a face anonymization guide utilizing the face anonymization neural network comprising a face attribute encoder and a texture encoder to: extract, utilizing the face attribute encoder, a first set of features representing one or more of gender, ethnicity, or age associated with the face in the digital image
Claim 11,
extract, utilizing the face attribute encoder, a first set of features representing one or more of gender, ethnicity, or age associated with the face in the digital image; extract, utilizing the texture encoder, a second set of features representing image texture of the digital image; and combine the first set of features and the second set of features into the face anonymization guide
Claim 8
extract, utilizing the face attribute encoder, a first set of features representing one or more of gender, ethnicity, or age associated with the face in the digital image; extract, utilizing the texture encoder, a second set of features representing image texture of the digital image; and combine the first set of features and the second set of features into the face anonymization guide
Claim 12,
wherein generating the anonymized digital image comprises: generating a masked digital image from the digital image by masking face pixels for the face depicted in the digital image; extracting a masked image vector from the masked digital image utilizing the generative neural network; combining the masked image vector and the face anonymization guide to generate a combined image mask-feature guide vector; and generating the anonymized digital image utilizing a decoder of the generative neural network from the combined image mask-feature guide vector.
Claim 8
generating a masked digital image from the digital image by masking face pixels for the face depicted in the digital image; and generating, from the masked digital image and utilizing a decoder of the face anonymization neural network, an anonymized digital image by replacing the face pixels masked in the digital image with synthetic face pixels synthesized according to the face anonymization guide, wherein generating the anonymized digital image comprises: extracting a masked image vector from the masked digital image; and combining the masked image vector and the face anonymization guide utilizing the face anonymization neural network.
Claims 13,18
wherein generating the masked digital image comprises: detecting an additional face depicted in the digital image in addition to the face of the person; removing pixels of the additional face to remove the additional face from a boundary around the face depicted in the digital image; and masking the face pixels for the face depicted within the boundary.
Claims 12,18
wherein generating the masked digital image comprises: detecting an additional face depicted in the digital image in addition to the face of the person; removing pixels of the additional face to remove the additional face from a boundary around the face depicted in the digital image; and masking the face pixels for the face depicted within the boundary.
Claim 14,
wherein generating the masked digital image comprises: generating a boundary around the face depicted in the digital image; aligning the boundary to orient the face pixels by removing tilt; and masking the face pixels aligned within the boundary
Claim 4
wherein generating the masked digital image comprises: generating a boundary around the face depicted in the digital image; aligning the boundary to orient the face pixels by removing tilt; and masking the face pixels aligned within the boundary.
Claim 16
wherein encoding the face anonymization guide comprises generating a feature vector representing gender, ethnicity, age, and expression for the face in the digital image.
Claim 16
wherein encoding the face anonymization guide comprises generating a feature vector representing gender, ethnicity, age, and expression for the face in the digital image.
Claim 17,
wherein encoding the face anonymization guide comprises: utilizing an encoder of the generative neural network to encode texture features for the face depicted in the digital image; and combining the features representing one or more of gender, ethnicity, age, or expression and the texture features into the face anonymization guide.
Claim 15
encoding, from a digital image depicting a face of a person, a face anonymization guide utilizing a face anonymization neural network comprising a face attribute encoder and a texture encoder to: extract, utilizing the face attribute encoder, a first set of features representing one or more of gender, ethnicity, or age associated with the face in the digital image; extract, utilizing the texture encoder, a second set of features representing image texture of the digital image; and combine the first set of features and the second set of features into the face anonymization guide
Claim 19
wherein generating the anonymized digital image comprises: extracting a masked image vector from the masked digital image utilizing an encoder of the generative neural network; generating a mapped noise vector by extracting features from a noise vector utilizing a mapper associated with the generative neural network; and synthesizing the synthetic face from the face anonymization guide, the masked image vector, and the mapped noise vector utilizing a decoder of the generative neural network.
Claim 19
wherein generating the anonymized digital image comprises: extracting the masked image vector from the masked digital image utilizing an encoder of the face anonymization neural network; generating a mapped noise vector by extracting features from a noise vector utilizing a mapper associated with the face anonymization neural network; and synthesizing the synthetic face pixels from the face anonymization guide, the masked image vector, and the mapped noise vector utilizing the decoder of the face anonymization neural network.
With regards to claims 1,8,15, the Patent No. US 12,321,495 B2 teaches generating, from the masked digital image and utilizing a decoder of the face anonymization neural network, an anonymized digital image by replacing the face pixels masked in the digital image with synthetic face pixels synthesized according to the face anonymization guide ( Claim 1) which is equivalent to inpainting a synthetic face within the digital image and the Patent No. US 12,321,495 B2 teaches generating a face anonymization guide which is equivalent to encoding face anonymization guide by encoding features.
With regards to claims 2,9, the Patent No. US 12,321,495 B2 teaches wherein inpainting the synthetic face within the digital image (claim1) . However, Patent No. US 12,321,495 B2 does not explicitly teach wherein inpainting is performed without copying or blending pixels originating from another digital image. Maximov teaches wherein inpainting is performed without copying or blending pixels originating from another digital image (Section – Masked background image).
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Patent No. US 12,321,495 B2 to include the teachings of Maximov. The motivation for doing so is to allow the system to remove the identifying characteristics of faces and bodies while producing high quality images and videos that can be used for any computer vision task, such as detection or tracking (Maximov – Abstract).
With regards to claim 3, the Patent No. US 12,321,495 B2 does not explicitly teach forming the synthetic face. However, Maximov teaches forming the synthetic face (Section – Masked background image).
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Patent No. US 12,321,495 B2 to include the teachings of Maximov. The motivation for doing so is to allow the system to remove the identifying characteristics of faces and bodies while producing high quality images and videos that can be used for any computer vision task, such as detection or tracking (Maximov – Abstract).
With regards to claim 4, the Patent No. US 12,321,495 B2 teaches wherein inpainting the synthetic face within the digital image according to the face anonymization guide ( Claim1). However, the Patent No. US 12,321,495 B2 does not explicitly teach generating the synthetic face to preserve the one or more of gender, ethnicity, age, or expression associated with the face in the digital image. However, Maximov teaches generating the synthetic face to preserve the one or more of gender, ethnicity, age, or expression associated with the face in the digital image (Section - Landmark image).
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Patent No. US 12,321,495 B2 to include the teachings of Maximov. The motivation for doing so is to allow the system to remove the identifying characteristics of faces and bodies while producing high quality images and videos that can be used for any computer vision task, such as detection or tracking (Maximov – Abstract).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-6,8-10,12,15,17,20 are rejected under 35 U.S.C. 103 as being unpatentable over Maximov et al. “ CIAGAN: Conditional Identity Anonymization Generative Adversarial networks “ – 11-30-2020 ( Maximov hereinafter) in view of Tao e al. “AnonymousNet: Natural Face De-Identification with Measurable Privacy” 4-19-2019 ( Tao hereinafter)
Regarding claim 1,
Maximov teaches a non-transitory computer readable medium storing executable instructions which, when executed by a processing device, cause the processing device to perform operations comprising:
[..] face anonymization guide comprising features representing one or more of gender, ethnicity, age, or expression associated with the face in the digital image (Abstract - Our model is able to remove the identifying characteristics of faces and bodies while producing high quality images and videos that can be used for any computer vision task, Section 3.1 - Pose preservation and temporal consistency. We propose using a landmark-based representation of the input face (or body) – Section - Landmark image - we use only the face silhouette, the mouth, and the bridge of the nose (see Fig. 2). This allows the network the freedom of choosing several facial characteristics, such as eye distance or eye shape, while at the same time, preserving expressions that are dependent on the mouth region, e.g. smiling or laughing, and the global pose is given by the nose position. The landmarks are represented as a binary image which is fed as input to the generator).
generating a masked digital image from the digital image by masking face pixels for the face depicted in the digital image (Fig.2, Our CIAGAN model takes as input the image, its landmark, the masked face and the desired identity - Section - Masked Background Image - Our goal is to generate only the face region of an image and inpainting it to the original image background. and
generating, from the masked digital image utilizing a generative neural network, an anonymized digital image by inpainting a synthetic face within the digital image according to the face anonymization guide (Section – Masked Background Image - Our goal is to generate only the face region of an image and inpainting it to the original image background. we provide the generative model with the masked background image together with the landmark image - .Section 3.3 - we also need to blend the generated faces seamlessly with a background. To this end, we use the conditional GAN framework [15], where we condition the generator with a landmark and a masked image (background), as explained in Section 3.2. The generator uses an encoder-decoder architecture The encoder brings the landmark and the masked image to a low-dimensional representation (bottleneck), where it is combined with identity representation while the decoder takes combined representation and up samples it to generate the anonymized RGB image – Section _ Landmark Image -we use only the face silhouette, the mouth, and the bridge of the nose (see Fig. 2). This allows the network the freedom of choosing several facial characteristics, such as eye distance or eye shape, while at the same time, preserving expressions that are dependent on the mouth region, e.g. smiling or laughing, and the global pose is given by the nose position. The landmarks are represented as a binary image which is fed as input to the generator ).
However, Maximov does not explicitly teach
encoding, from a digital image depicting a face of a person, a face anonymization guide comprising features representing one or more of gender, ethnicity, age, or expression associated with the face in the digital image
Tao teaches
encoding, from a digital image depicting a face of a person, a face anonymization guide comprising features representing one or more of gender, ethnicity, age, or expression associated with the face in the digital image ( Abstract, The framework encompasses four stages: facial attribute estimation, privacy-metric-oriented face obfuscation- Section _Stage I - Google Net is not trained for one label, but rather is fed with 40 facial attributes at the same time and outputs multiple classification results accordingly –- Figure 4: Overview of the facial attribute prediction pipeline. We train a deep Convolutional Neural Network (CNN) fed by preprocessed and labelled images, and then extract deep features from the fully connected (FC) layer of the CNN and accordingly train random forest classifiers to predict facial attributes (the full attribute list can be found in Figure 6). - Fig.6 – list extracted attributes include Male, Young< smiling)
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Maximov to include the teachings of Tao. The motivation for doing so is to allow system to enhance image obfuscation (Tao - Section I).
Regarding claim 2,
Maximov further teaches
wherein inpainting the synthetic face within the digital image is performed without copying or blending pixels originating from another digital image (Section – Masked background image - Our goal is to generate only the face region of an image and inpainting it to the original image background. we provide the generative model with the masked background image together with the landmark image. The masked background image still contains the forehead region of the head. Once the generator has access to this information, it can learn to match the skin appearance of the generated face to the forehead skin color).
Regarding claim 3,
Maximov further teaches
wherein inpainting the synthetic face within the digital image comprising replacing masked face pixels of the masked digital image with synthetic face pixels forming the synthetic face (Section – Masked background image - Our goal is to generate only the face region of an image and inpainting it to the original image background- we provide the generative model with the masked background image together with the landmark image. The masked background image still contains the forehead region of the head).
Regarding claim 4,
Maximov further teaches
wherein inpainting the synthetic face within the digital image according to the face anonymization guide comprises generating the synthetic face to preserve the one or more of gender, ethnicity, age, or expression associated with the face in the digital image (Section - Landmark image - the generator is conditioned on the face shape allowing us to preserve the input pose in the output - To hide as much identity information as possible but still preserve the pose, instead of using all 68 landmarks [18], we use only the face silhouette the mouth, and the bridge of the nose (see Fig. 2). This allows the network the freedom of choosing several facial characteristics, such as eye distance or eye shape, while at the same time, preserving expressions that are dependent of the mouth region, e.g. smiling or laughing, and the global pose is given by the nose position).
Regarding claim 5,
Maximov further teaches
wherein generating the anonymized digital image comprises: extracting a masked image vector from the masked digital image; and combining the masked image vector and the face anonymization guide utilizing the generative neural network (Section 3.3 - The encoder brings the landmark and the masked image to a low-dimensional representation (masked image vector ) where it is combined with identity representation while the decoder takes combined representation and up samples it to generate the anonymized RGB image . Section 5 - The generator uses an encoder-decoder architecture, and receives as input a 6-dimensional image that is created by concatenating the landmark image with the masked background image. It encodes the input image into a 3-dimensional tensor
that has 256 channels. In the bottleneck, the encoded image is concatenated with the identity embedding (that is an output of the transposed convolution network). Finally, the network decodes the combined embedding, to produce the anonymized version of the source image – Note; the examiner interprets the face anonymization guide as identify /landmark information).
Regarding claim 6,
Maximov further teaches
wherein generating the anonymized digital image comprises synthesizing synthetic face pixels utilizing a decoder of the generative neural network according to the face anonymization guide (Section 5 – . The generator uses an encoder-decoder architecture, and receives as input a 6-dimensional image that is created by concatenating the landmark image with the masked background image. It encodes the input image into a 3-dimensional tensor that has 256 channels. In the bottleneck, the encoded image is concatenated with the identity embedding (that is an output of the transposed convolution network). Finally, the network decodes the combined embedding, to produce the anonymized version of the source image – Section 3.4 – The network produces a parametrized version of the identity and feeds it into the bottleneck of the generator. In this way, the generator learns to generate a face with some of the characteristics of the desired identity while keeping the pose of the real image. In other words, the generated image is a composition of both the landmark identity and the desired identify).
Regarding claim 8,
Maximov teaches a system comprising:
one or more memory devices storing a generative neural network and a digital image depicting a face of a person; and one or more processors configured to cause the system to anonymize the digital image depicting the face of the person by: generating a face anonymization guide [..] representing one or more of gender, ethnicity, age, or expression of the face in the digital image ( Abstract - Our model is able to remove the identifying characteristics of faces and bodies while producing high quality images and videos that can be used for any computer vision task, Section 3.1 - Pose preservation and temporal consistency. We propose using a landmark-based representation of the input face (or body). Section – Landmark image - we use only the face silhouette, the mouth, and the bridge of the nose (see Fig. 2). This allows the network the freedom of choosing several facial characteristics, such as eye distance or eye shape, while at the same time, preserving expressions that are dependent on the mouth region, e.g. smiling or laughing, and the global pose is given by the nose position. The landmarks are represented as a binary image which is fed as input to the generator); and
generating, utilizing the generative neural network, an anonymized digital image by inpainting a synthetic face within the digital image according to the face anonymization guide that preserves the one or more of gender, ethnicity, age, or expression of the face in the digital image(Section – Masked Background Image - Our goal is to generate only the face region of an image and inpainting it to the original image background. we provide the generative model with the masked background image together with the landmark image - .Section 3.3 - we also need to blend the generated faces seamlessly with a background. To this end, we use the conditional GAN framework [15], where we condition the generator with a landmark and a masked image (background), as explained in Section 3.2. The generator uses an encoder-decoder architecture The encoder brings the landmark and the masked image to a low-dimensional representation (bottleneck), where it is combined with identity representation while the decoder takes combined representation and up samples it to generate the anonymized RGB image – Section _ Landmark Image -we use only the face silhouette, the mouth, and the bridge of the nose (see Fig. 2). This allows the network the freedom of choosing several facial characteristics, such as eye distance or eye shape, while at the same time, preserving expressions that are dependent on the mouth region, e.g. smiling or laughing, and the global pose is given by the nose position. The landmarks are represented as a binary image which is fed as input to the generator).
Maximov does not explicitly teach generating a face anonymization guide by encoding features representing one or more of gender, ethnicity, age, or expression of the face in the digital image
Tao teaches
generating a face anonymization guide by encoding features representing one or more of gender, ethnicity, age, or expression of the face in the digital image ( Abstract - The framework encompasses four stages: facial attribute estimation, privacy-metric-oriented face obfuscation- Section _Stage I - Google Net is not trained for one label, but rather is fed with 40 facial attributes at the same time and outputs multiple classification results accordingly –- Figure 4: Overview of the facial attribute prediction pipeline. We train a deep Convolutional Neural Network (CNN) fed by preprocessed and labelled images, and then extract deep features from the fully connected (FC) layer of the CNN and accordingly train random forest classifiers to predict facial attributes (the full attribute list can be found in Figure 6). - Fig.6 – list extracted attributes include Male, Young< smiling).
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Maximov to include the teachings of Tao. The motivation for doing so is to allow system to enhance image obfuscation ( Tao - Section I).
Regarding claim 9,
Maximov further teaches
wherein inpainting the synthetic face within the digital image is performed without copying or blending pixels originating from another digital image (Section – Masked background image - Our goal is to generate only the face region of an image and inpainting it to the original image background. we provide the generative model with the masked background image together with the landmark image. The masked background image still contains the forehead region of the head. Once the generator has access to this information, it can learn to match the skin appearance of the generated face to the forehead skin color).
Regarding claim 10,
Maximov further teaches
wherein generating the face anonymization guide comprises utilizing a first encoder of the generative neural network to encode the features (Section 5 - The generator uses an encoder-decoder architecture, and receives as input a 6-dimensional image that is created by concatenating the landmark image with the masked background image. It encodes the input image into a 3-dimensional tensor that has 256 channels. In the bottleneck, the encoded image is concatenated with the identity embedding (that is an output of the transposed convolution network). Finally, the network decodes the combined embedding, to produce the anonymized version of the source image).
Regarding claim 12,
Maximov further teaches
wherein generating the anonymized digital image comprises: generating a masked digital image from the digital image by masking face pixels for the face depicted in the digital image; extracting a masked image vector from the masked digital image utilizing the generative neural network; combining the masked image vector and the face anonymization guide to generate a combined image mask-feature guide vector; and generating the anonymized digital image utilizing a decoder of the generative neural network from the combined image mask-feature guide vector (Section – Masked Background Image - Our goal is to generate only the face region of an image and inpainting it to the original image background. we provide the generative model with the masked background image together with the landmark image. The masked background image still contains the forehead region of the head. Section 3.3 - The generator uses an encoder-decoder architecture [22]. The encoder brings the landmark and the masked image to a low-dimensional representation (masked image vector ) where it is combined with identity representation while the decoder takes combined representation and up samples it to generate the anonymized RGB image – Section 5 - The generator uses an encoder-decoder architecture, and receives as input a 6-dimensional image that is created by concatenating the landmark image with the masked background image. It encodes the input image into a 3-dimensional tensor that has 256 channels. In the bottleneck, the encoded image is concatenated with the identity embedding (that is an output of the transposed convolution network). Finally, the network decodes the combined embedding, to produce the anonymized version of the source image).
Regarding claim 15,
Maximov teaches a computer implemented method comprising:
[..] face anonymization guide comprising features representing one or more of gender, ethnicity, age, or expression associated with the face in the digital image (Abstract - Our model is able to remove the identifying characteristics of faces and bodies while producing high quality images and videos that can be used for any computer vision task, Section 3.1 - Pose preservation and temporal consistency. We propose using a landmark-based representation of the input face (or body) – Section - Landmark image - we use only the face silhouette, the mouth, and the bridge of the nose (see Fig. 2). This allows the network the freedom of choosing several facial characteristics, such as eye distance or eye shape, while at the same time, preserving expressions that are dependent on the mouth region, e.g. smiling or laughing, and the global pose is given by the nose position. The landmarks are represented as a binary image which is fed as input to the generator).
generating a masked digital image from the digital image by masking face pixels for the face depicted in the digital image (Fig.2, Our CIAGAN model takes as input the image, its landmark, the masked face and the desired identity - Section - Masked Background Image - Our goal is to generate only the face region of an image and inpainting it to the original image background. and
generating, from the masked digital image utilizing a generative neural network, an anonymized digital image by inpainting a synthetic face within the digital image according to the face anonymization guide (Section – Masked Background Image - Our goal is to generate only the face region of an image and inpainting it to the original image background. we provide the generative model with the masked background image together with the landmark image - .Section 3.3 - we also need to blend the generated faces seamlessly with a background. To this end, we use the conditional GAN framework [15], where we condition the generator with a landmark and a masked image (background), as explained in Section 3.2. The generator uses an encoder-decoder architecture The encoder brings the landmark and the masked image to a low-dimensional representation (bottleneck), where it is combined with identity representation while the decoder takes combined representation and up samples it to generate the anonymized RGB image – Section _ Landmark Image -we use only the face silhouette, the mouth, and the bridge of the nose (see Fig. 2). This allows the network the freedom of choosing several facial characteristics, such as eye distance or eye shape, while at the same time, preserving expressions that are dependent on the mouth region, e.g. smiling or laughing, and the global pose is given by the nose position. The landmarks are represented as a binary image which is fed as input to the generator ).
Maximov does not explicitly teach
encoding, from a digital image depicting a face of a person, a face anonymization guide comprising features representing one or more of gender, ethnicity, age, or expression associated with the face in the digital image
Tao teaches
encoding, from a digital image depicting a face of a person, a face anonymization guide comprising features representing one or more of gender, ethnicity, age, or expression associated with the face in the digital image ( Abstract, The framework encompasses four stages: facial attribute estimation, privacy-metric-oriented face obfuscation- Section _Stage I – Google Net is not trained for one label, but rather is fed with 40 facial attributes at the same time and outputs multiple classification results accordingly –- Figure 4: Overview of the facial attribute prediction pipeline. We train a deep Convolutional Neural Network (CNN) fed by preprocessed and labelled images, and then extract deep features from the fully connected (FC) layer of the CNN and accordingly train random forest classifiers to predict facial attributes (the full attribute list can be found in Figure 6). - Fig.6 – list extracted attributes include Male, Young< smiling)
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Maximov to include the teachings of Tao. The motivation for doing so is to allow system to enhance image obfuscation. ( Tao - Section I).
Regarding claim 17,
Maximov in view of Tao teaches
wherein encoding the face anonymization guide comprises: utilizing an encoder of the generative neural network to encode texture features for the face depicted in the digital image; and combining the features representing one or more of gender, ethnicity, age, or expression and the texture features into the face anonymization guide( Tao - Abstract, The framework encompasses four stages: facial attribute estimation, privacy-metric-oriented face obfuscation- Section _Stage I - We adopt Google Net [55] for facial attribute extraction - Google Net is not trained for one label, but rather is fed with 40 facial attributes at the same time and outputs multiple classification results accordingly Figure 4 outlines the facial attribute prediction pipeline, where labelled images are first fed into the model, features are extracted from the fully connected (FC) layer, and then 40 random forest classifiers [32] are trained and facial attributes are subsequently obtained.–- Figure 4: Overview of the facial attribute prediction pipeline. We train a deep Convolutional Neural Network (CNN) fed by preprocessed and labelled images, and then extract deep features from the fully connected (FC) layer of the CNN and accordingly train random forest classifiers to predict facial attributes (the full attribute list can be found in Figure 6). - Fig.6 – list extracted attributes include Male, Young< smiling – Section State II - the
Privacy-Preserving Attribute Selection (PPAS) Algorithm, a method to select and update facial attributes – Note: same motivation as claim 15).
Regarding claim 20,
Maximov further teaches
wherein generating the anonymized digital image comprises: extracting a masked image vector from the masked digital image; and combining the masked image vector and the face anonymization guide utilizing the generative neural network (Section 3.3 - The encoder brings the landmark and the masked image to a low-dimensional representation (masked image vector ) where it is combined with identity representation while the decoder takes combined representation and up samples it to generate the anonymized RGB image . Section 5 - The generator uses an encoder-decoder architecture, and receives as input a 6-dimensional image that is created by concatenating the landmark image with the masked background image. It encodes the input image into a 3-dimensional tensor
that has 256 channels. In the bottleneck, the encoded image is concatenated with the identity embedding (that is an output of the transposed convolution network). Finally, the network decodes the combined embedding, to produce the anonymized version of the source image – Note; the examiner interprets the face anonymization guide as identify /landmark information).
Claim 7 is rejected under 35 U.S.C. 103 as being unpatentable over Maximov in view of Tao further in view of Kim et al. Publication No. WO 2023/247732 (Kim hereinafter)
Regarding claim 7,
Maximov in view of Tao teaches wherein encoding the face anonymization guide (Tao - Abstract, Section I, Fig.4 & 6). However, Maximov in view of Tao does not explicitly teach
utilizing a first encoder of the generative neural network to extract a first feature set and a second encoder of the generative neural network to extract a second feature set from the digital image; and combining the first feature set and the second feature set into a guide feature set.
Kim teaches
utilizing a first encoder of the generative neural network to extract a first feature set and a second encoder of the generative neural network to extract a second feature set from the digital image; and (¶ 0041 - The variable parameters of the synthetic model may include pose parameters encoding a position and/or orientation of the model with respect to a virtual camera, and deformation parameters for example encoding adjustments applied to the base geometry of the model, thereby causing the model to exhibit different facial expressions, mouth movements, and so on. The pose parameters may encode a position of the model in three-dimensions and rigid rotations about three axes - ¶ 0042 - The synthetic model data 116 may include sampled pose/expression data 118, including specific parameter values corresponding to a range of poses and expressions for the synthetic model. The sampled pose/expression data 188 may for example include parameter values corresponding to several minutes or hours of animation of the synthetic model. The range of poses and expressions may be sampled using any suitable strategy, for example randomly or based on a predetermined space-filling design to ensure relevant regions of the parameter space for poses and expressions are covered in an efficient manner. The sampled poses and expressions may be independent of the poses and expressions exhibited by the target face in the target footage - ¶ 0046 - An encoder-decoder network within the meaning of the present disclosure may be a neural network including one or more encoder networks and one or more decoder networks - 1 0053 -the second path through the encoder-decoder network 200 ( comprising the encoder network 202 and the model decoder network 206) may be trained to process two-dimensional renders 212 of a three-dimensional synthetic face model 214 exhibiting a range of poses and expressions, to determine corresponding sets of parameter values 216 for the synthetic face model 214).
combining the first feature set and the second feature set into a guide feature set combining the first feature set and the second feature set into a guide feature set (¶ 0040 - The combination of the reflectance model and lighting model may enable the irradiance at a given point on the surface of the face to be determined. For example, the reflectance model may be a Lambertian reflectance model which treats the surface of the model as a perfect diffuse surface that scatters incident illumination equally in all directions - See Also Claim 3).
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Maximov in view of Tao to include the teachings of Kim. The motivation for doing so is to allow system to provide direct control over expressions exhibited by the target human face via the parameter values of the synthetic face model, without the need for intermediate rendering of the synthetic face model (Kim – ¶ 0010).
Claim 11 is rejected under 35 U.S.C. 103 as being unpatentable over Maximov in view of Tao further in view of Wen et al. “IdentityDP: Differential Private Identification Protection for Face Images” – 3-2-2021 ( Wen hereinafter)
Regarding claim 11,
Maximov in view of Tao teaches wherein encoding the face anonymization guide (Tao - Abstract, Section I, Fig.4 & 6)
However, Maximov in view of Tao does not explicitly teach
utilizing a second encoder of the generative neural network to encode additional features of the face depicted in the digital image; and combining the features from the first encoder and the additional features from the second encoder into the face anonymization guide.
Wen teaches
utilizing a second encoder of the generative neural network to encode additional features of the face depicted in the digital image; and combining the features from the first encoder and the additional features from the second encoder into the face anonymization guide (Section : Stage I- The key idea is to explicitly guide the generation process by an appropriate representation of that intention. Therefore, our network consists of 3 components: (1) Identity Encoder; (2) Attribute Encoder; (3) Fusion Generator. Attribute Encoder: Attribute representation, which determines pose, expression, illumination, background and so on, intuitively carries more spatial information than identity. Johnson et al. [49] illustrate that low-level features tend to preserve image content and overall spatial structure, and high-level features tend to preserve color, texture, and exact shape. Fusion Network; After obtaining the disentangled identity and attribute representations, we would like to learn a way to integrate them to reproduce the original face image, which will be used in our subsequent steps. Note: the examiner interprets the identity and attribute presentation as the representation that guide the generation process and uses the fuse representations to reconstruct the anonymized face).
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Maximov in view of Tao to include the teachings of Wen. The motivation for doing so is to allow system to effectively obfuscate the identity-related information of faces, preserve significant visual similarity, and generate high-quality images that can be used for identity-agnostic computer vision task (Wen – Abstract)
Claims 13,18 are rejected under 35 U.S.C. 103 as being unpatentable over Maximov in view of Tao further in view of Cazzani et al. Publication No. US 2024/0129435 ( Cazzani hereinafter)
Regarding claim 13,
Maximov further teaches wherein generating the masked digital image comprises: detecting an additional face depicted in the digital image in addition to the face of the person (Section – Masked Background image – Multiple faces in the same image) . However, Maximov does not explicitly teach
removing pixels of the additional face to remove the additional face from a boundary around the face depicted in the digital image; and masking the face pixels for the face depicted within the boundary.
Cazzani teaches
detecting additional faces depicted in the digital image in addition to the face of the person; removing pixels of the additional faces so that a boundary for the face depicted in the digital image does not include the pixels of the additional faces (¶ 0014 - Pose can be determined using processes such as Procrustes analysis, but those skilled in the art will appreciate that there are other techniques to determine the pose of each person in an image. Knowing the pose (head and/or body) of a person in the image can help distinguish the most "prominent" person in the scene, even when other participants are at similar distances from the camera. Once the most prominent person ( or intended participant) in the scene is identified, a privacy curtain may be "built" behind or around that person by removing, via pixel manipulation, other persons from the scene or image)
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Maximov to include the teachings of Cazzani. The motivation for doing so is to allow system to utilize segmentation efficiently to precisely hide persons who are behind the intended participant or that are not as prominent as the intended participant (Cazzani – ¶ 0015).
Regarding claim 18,
Maximov further teaches wherein generating the masked digital image comprises: detecting an additional face depicted in the digital image in addition to the face of the person (Section – Masked Background image – Multiple faces in the same image ) . However, Maximov does not explicitly teach
removing pixel values for pixels of the additional face in the digital image; generating a boundary for the face depicted in the digital image including pixels with removed pixel values for the additional face; and masking the face pixels for the face depicted within the boundary..
Cazzani teaches
detecting an additional face depicted in the digital image in addition to the face of the person; removing pixel values for pixels of the additional face in the digital image; generating a boundary for the face depicted in the digital image including pixels with removed pixel values for the additional face; and masking the face pixels for the face depicted within the boundary (¶ 0014 - Pose can be determined using processes such as Procrustes analysis, but those skilled in the art will appreciate that there are other techniques to determine the pose of each person in an image. Knowing the pose (head and/or body) of a person in the image can help distinguish the most "prominent" person in the scene, even when other participants are at similar distances from the camera. Once the most prominent person ( or intended participant) in the scene is identified, a privacy curtain may be "built" behind or around that person by removing, via pixel manipulation, other persons from the scene or image).
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Maximov to include the teachings of Cazzani. The motivation for doing so is to allow system to utilize segmentation efficiently to precisely hide persons who are behind the intended participant or that are not as prominent as the intended participant (Cazzani – ¶ 0015).
Claim 14 is rejected under 35 U.S.C. 103 as being unpatentable over Maximov in view of Tao further in view of Georgievskaya et al. Publication No. US 2022/0335252 A1 ( Georgievskaya hereinafter)
Regarding claim 14,
Maximov further teaches wherein generating the masked digital image Section - Masked Background Image). However, Maximov does not explicitly teach
generating a boundary around the face depicted in the digital image; aligning the boundary to orient the face pixels by removing tilt; and masking the face pixels aligned within the boundary
Georgievskaya teaches
generating a boundary around the face depicted in the digital image; aligning the boundary to orient the face pixels by removing tilt; and masking the face pixels aligned within the boundary (¶ 0076 - FIG. 4 depicts standardizing a facial image 400 comprising representation of a face 410 and non-skin information 412, in accordance with various embodiments. A surrounding area of a face 402 in the facial image 400 is cropped to obtain a cropped facial image 404. An angle of orientation of the face 402 in the facial image 400 is estimated and correction is performed to obtain a corrected rotation of the face in the facial image 406. The facial image, wherein the rotation of the face is corrected 406 is aligned to be in the center to obtain a correction of a position of the face relative to the facial image 408 coordinates of the facial key points are used to ensure the face is geometrically aligned, such that the head rotation and inclines are automatically estimated and corrected, the face position is centered in the facial image, and finally the background non-skin information is eliminated to obtain the standardized image 406 – ¶ 0077 - mask is applied to a standardized image to build a mask of skin pixels 502, and one or more areas comprising skin-pixels and one or more areas comprises pixels related to non-skin information are separated. The separation is performed by at least one of: evaluation of color dependencies and distributions and facial key points-based areas definition in image or by a convolutional neural network configured to segment skin pixels and a masked facial image 504 is obtained as an end result. ¶ 0039 - The facial key points enable localization of a plurality of facial regions including eyebrows, eyes, mouth, nose).
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Maximov to include the teachings of Georgievskaya. The motivation for doing so is to allow system to concatenate the masked facial image with the generated photo-realistic skin patterns. (Georgievskaya– Abstract).
Claim 16 is rejected under 35 U.S.C. 103 as being unpatentable over Maximov in view of Tao further in view of KUTA et al. US 2022/0172416 A1 ( KUTA hereinafter) .
Regarding claim 16,
Maximov in view of Tao teaches wherein encoding the face anonymization guide (Tao - Abstract, Section I, Fig.4 & 6). However, Maximov in view of Tao does not explicitly teach
generating a feature vector representing gender, ethnicity, age, and expression for the face in the digital image.
KUTA teaches
generating a feature vector representing gender, ethnicity, age, and expression for the face in the digital image (Abstract, Claim 1 - using, by the at least one processor, a first neural network (NN), to encode the facial image into a first latent vector; ¶ 0012 - the latent vector may include a plurality of scalar parameters. The plurality of scalar parameters may include a first group of parameters, representing features pertaining to, or helping to specify the identity of a person. For example, the first group of parameters may represent an age, an ethnicity and/or a gender of a person depicted in the obtained image Additionally, or alternatively, the first group may include one or more parameters pertaining to facial features of specific individuals. Such parameters may, for example, represent a distance between pupils, a width and/or length of a nose, a complexion, a hairline).
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Maximov in view of Tao to include the teachings of KUTA. The motivation for doing so is to allow system to anonymize an image of a face so as to prevent identification of the depicted face in order to comply with privacy regulations and /or reduce potential risk in case of date breach (KUTA – ¶ 0003).
Claim 19 is rejected under 35 U.S.C. 103 as being unpatentable over Maximov in view of Tao further in view of Ren et al. Patent No. US 11,580,673 B1 ( Ren hereinafter).
Regarding claim 19,
Maximov does not explicitly teach
wherein generating the anonymized digital image comprises: extracting a masked image vector from the masked digital image utilizing an encoder of the generative neural network; generating a mapped noise vector by extracting features from a noise vector utilizing a mapper associated with the generative neural network; and synthesizing the synthetic face from the face anonymization guide, the masked image vector, and the mapped noise vector utilizing a decoder of the generative neural network.
However, Ren teaches
extracting a masked image vector from the masked digital image utilizing an encoder of the generative adversarial neural network (Col.3, lines 10- 30 - an image synthesis model ( e.g., an algorithm or process) that uses an additional constraint on initially projected features by injecting a mask embedding vector into a latent features vector. The image synthesis model can perform a more efficient initial feature projection by producing latent features that are compatible with the mask input, thereby preserving or improving output image quality over other models. methods for using a mask embedding vector and a latent features vector, realistic high-resolution images may be generated with semantic control. For example, a conditional generative adversarial network (cGAN) or another system or algorithm may take as input, a semantic mask (e.g., an edge map showing a breast shape or position) and a latent features vector at an initial latent projection layer, and may use the input to generate a realistic high-resolution image (e.g., a mammogram). In another example, a cGAN or another system or algorithm may take as input, a semantic mask (e.g., an binary image represent facial features or facial expressions) and a latent features vector at an initial latent projection layer, and may use the input to generate a realistic high-resolution image of a human face).
generating a mapped noise vector by extracting features from a noise vector utilizing a mapper associated with the generative adversarial neural network (Col.5, lines 15-25 the generator may generate a mask embedding vector and then concatenate the mask 20 embedding vector to a latent noise vector as input of a feature projection path. The mask is an image providing constraints. For example, the mask may be an edge map, a segmentation map, a gradient field, etc).
synthesizing the synthetic face pixels from the face anonymization guide, the masked image vector, and the mapped noise vector utilizing a decoder of the generative adversarial neural network (Abstract - receiving, as input, a mask embedding vector and a latent features vector, wherein the mask embedding vector acts as a semantic constraint; generating, using a trained image synthesis algorithm and the input, a realistic image, wherein the realistic image is constrained by the mask embedding vector; and outputting, by the trained image synthesis algorithm, the realistic image to a display or a storage device - Col.5, lines 20-35 -Col.5, lines 15-25 the generator may generate a mask embedding vector and then concatenate the mask 20 embedding vector to a latent noise vector as input of a feature projection path. The mask is an image providing constraints. For example, the mask may be an edge map, a segmentation map, a gradient field, etc the subject matter described herein may involve techniques, methods, systems, or devices that use mask embedding in semantic cGANs for realistic image synthesis. For example, an example image synthesis system may take both a mask and a random latent features vector as input and may use this input to generate highly stochastic images with fine-grained details. In this example, the highly stochastic images may be human faces and the mask may be data for representing human facial features and/or human facial expressions (e.g., frowns, smiles, etc.)
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Maximov to include the teachings of Ren. The motivation for doing so is to allow system to perform a more efficient initial feature projection by producing latent features that are compatible with the mask input, thereby preserving or improving output image quality over other models (Ren - Col.3, lines 1-30 - Ren).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to YOUNES NAJI whose telephone number is (571)272-2659. The examiner can normally be reached Monday - Friday 8:30 AM -5:30 PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Oscar A Louie can be reached at (571) 270-1684. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/YOUNES NAJI/Primary Examiner, Art Unit 2445
Prosecution Insights