DETAILED ACTION
This Office Action is in response to the Applicants' communication filed on April 13, 2026, which amends the independent claims 1 and 9, adds new dependent claims 15-18, and presents arguments, is hereby acknowledged. Claims 1-18 are currently pending and have been examined.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
Applicant’s arguments filed on April 13, 2026, have been fully considered.
Applicant argues that by this response, the independent claims 1, 5, and 13 are hereby amended to add a new limitation “determine a descriptor associated with the facial image data of the human being, wherein the descriptor includes information describing the characteristic of the mimic expression feature and the characteristic of each identification feature of a selectable subset of the plurality of identification features, wherein: the selectable subset of the plurality of identification features does not allow a unique identification of the human being; the first device is further configured to provide an obfuscated image, wherein in the obfuscated image the facial image data of the human being is replaced with placeholder data;
the first device is further configured to provide the obfuscated image and the descriptor to a second device; and the second device is configured to produce, based on jointly processing the obfuscated image and the descriptor, an anonymized image that includes a replacement of at least a portion of the placeholder data of the obfuscated image with artificial facial image data corresponding to the descriptor” (emphasize added for the new limitations) in order to overcome the 35 U.S.C. §103 rejection.
Examiner replies that the amended claims with new limitation may overcome the cited portions of the prior arts. However, a newly found art, Whitehill, etc. (US 20150324633 A1) teaches that determine a descriptor associated with the facial image data of the human being, wherein the descriptor includes information describing the characteristic of the mimic expression feature and the characteristic of each identification feature of a selectable subset of the plurality of identification features (See Whitehill: Fig. 1, and [0012], “In one particular approach, the original facial image is encoded as a feature set. The feature set contains personal identity components that contribute to recognizability of the facial image and expression components that contribute to the emotional expression of the facial image. A perturbation transform is applied to the feature set. The perturbation transform substantially perturbs the personal identity components and substantially preserves at least some of the expression components. The perturbed feature set is decoded to obtain the synthesized facial image. In this way, the synthesized facial image is anonymized while still retaining some of the emotional expression of the original facial image”; [0029], “The image filter outputs are passed to a feature selection module at 110. The feature selection module, whose parameters are found using machine learning methods, can include the use of a machine learning technique that is trained on a database of spontaneous expressions by subjects that have been manually labeled for facial actions from the Facial Action Coding System (FACS). The feature selection module 110 processes the image filter outputs for each of the plurality of image windows to choose a subset of the characteristics or parameters to pass to the classification module at 112. The feature selection module results for the two or more image windows can optionally be combined and processed by a classifier process at 112 to produce a joint decision regarding the posterior probability of the presence of an action unit in the face shown in the image. The classifier process can utilize machine learning on the database of spontaneous facial expressions. At 114, a promoted output of the process 100 can be a score for each of the action units that quantifies the observed “content” of each of the action units in the face shown in the image”; [0010], “The present invention overcomes the limitations of the prior art by perturbing recognizable facial images, in a manner such that the perturbed facial images are no longer recognizable while still preserving at least part of the emotional expression or other attributes of the original facial image or the subject or circumstances”; and [0014], “In yet another aspect, facial images are anonymized while preserving attributes of the facial image other than facial expression. For example, facial images may be perturbed so that they are no longer recognizable as the original subject, but while still preserving gender, age or other attributes of the original facial image”. Note that the feature set is mapped to the “descriptor”, the expression components are mapped to the mimic expression feature, the personal identification components are mapped to the personal identification features, the perturbation is made to the personal identification components to render the output image un-identifiable).
Examiner further replies that the newly found art, Whitehill, etc. (US 20150324633 A1) also teaches that the second device is configured to produce, based on jointly processing the obfuscated image and the descriptor, an anonymized image that includes a replacement of at least a portion of the placeholder data of the obfuscated image with artificial facial image data corresponding to the descriptor (See Whitehill: Fig. 1, and [0010], “The present invention overcomes the limitations of the prior art by perturbing recognizable facial images, in a manner such that the perturbed facial images are no longer recognizable while still preserving at least part of the emotional expression or other attributes of the original facial image or the subject or circumstances”; and [0043], “The feature set is input to the transform module 424, which applies a perturbation transform to the feature set that substantially perturbs its personal identity components but substantially preserves at least some of its expression components. The output of the transform module 424 is a perturbed feature set, which serves as an input to the decoder module 426. The decoder module 426 decodes the perturbed feature set to obtain the synthesized facial image 430. The synthesized facial image 430, which is now “anonymized,” together with a facial expression category label of the original facial image 400 (e.g., happy, sad, etc.), can be used to train an AFER system”. Note that the perturbation is applied to the personal identification components, and the output synthetic image is “anonymized” but keep the facial expression category label of the original facial image, which means that the portion of the facial image with personal identification features is replaced (placeholder) by an artificial image). The remaining arguments of the applicant are mooted in view of the newly found art.
Examiner respectfully further replies that the Applicant's arguments have been fully considered and a new ground of rejections have been made. Accordingly, new grounds of rejection are set forth below. Since the new grounds of rejection are necessitated by Applicant's amendments to the claims, the present action is made final.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 3-9, and 11-18 are rejected under 35 U.S.C. 103 as being unpatentable over Jung, etc. (US 8203609 B2) in view of Astarabadi, etc. (US 20200358983 A1), further in view of Lee, etc. (US 20140147023 A1), Zavesky (US 20130141530 A1), and Whitehill, etc. (US 20150324633 A1).
Regarding claim 1, Jung teaches that a personal data processing system (See Jung: Fig. 24, and Col. 21 Lines 20-29, "FIG. 24 illustrates an exemplary embodiment of a system 1600. The system includes an imaging device 1610, an anonymization device 1650, and a broadcast device. The imaging device is operable to capture an image depicting a class of subjects 1605 that may have a temporal presence in an area 1603. The broadcast device is operable to display a human-perceivable indication of a selected anonymization policy to at least a portion of the area. The anonymization policy includes obscuring a property of the class of subjects depicted in the captured image") comprising:
a first device (See Jung: Fig. 3, and Col. 12 Lines 16-21, "In an embodiment, the imaging device 320 operable to acquire an image of a subject further includes a digital imaging device operable to acquire an image of a subject of the potential image subject(s) 305. In another embodiment, the imaging device operable to acquire an image of a subject further includes a digital camera operable to acquire an image of a subject”) configured to:
capture image data including facial image data of a human being (See Jung: Fig. 3, and Col. 12 Lines 16-27, "In an embodiment, the imaging device 320 operable to acquire an image of a subject further includes a digital imaging device operable to acquire an image of a subject of the potential image subject(s) 305. In another embodiment, the imaging device operable to acquire an image of a subject further includes a digital camera operable to acquire an image of a subject. For example, the digital camera may include a lens 322, an imaging element 324, an analog to digital converter 326, and/or a processor 328. In a further embodiment, the imaging device operable to acquire an image of a subject further includes an imaging device operable to capture an image of a subject of the potential image subject(s) 305"),
the facial image data of a human being including a plurality of identification features each having a characteristic (See Jung: Fig. 24, and Col. 21 Lines 19-32, "In a further embodiment, the anonymization device further includes an anonymization device having a selector circuit operable to select an anonymization policy, where the anonymization policy includes obscuring at least one of a variable, a recognizable, a distinguishable, and/or a unique aspect of a property of the class of subjects depicted in the captured image. In another embodiment, the anonymization device further includes an anonymization device having a selector circuit operable to select an anonymization policy, where the anonymization policy includes obscuring at least one of a variable, a recognizable, a distinguishable, and/or a unique aspect of a property of each instance of the class of subjects depicted in the captured image". Note that the unique aspect of the property of the subject may be the identification features of the face image) and a mimic expression feature having a characteristic;
detect the facial image data of the human being in the image data (See Jung: Figs. 3-5, and Col. 14 Lines 66-67 ~ Col. 15 Lines 1-18, "FIG. 5 illustrates an embodiment where the display 440B of the image capture device 420 indicating a "Green" color-based visual indication of the anonymization policy, corresponding to decreasing a fidelity of facial properties and license plate properties depicted in acquired images to be indistinguishable. The decreased fidelity of the illustrated embodiment includes a decreased fidelity of facial properties of human subjects 408A, 408B, and 408C implemented by obscuring portions of their faces and/or heads. The faces and/or heads of human subjects may be located in the image using techniques known to those skilled in the art, including artificial intelligence, and/or facial recognition techniques. FIG. 5 also illustrates a decreased fidelity of the identification number "XYZ 123" property of a license plate of a car subject 406 in the image that was implemented by an obscuring portion over the license identification number. In an embodiment, the image 401 may be used to study human and/or vehicle traffic patterns in the area 403 while preserving the anonymity of human subjects and/or registered owners of vehicles". Note that using face recognition technique to recognize face in the image may be the face detection); and
determine a descriptor associated with the facial image data of the human being, wherein the descriptor includes information describing the characteristic of the mimic expression feature and the characteristic of each identification feature of a selectable subset of the plurality of identification features, wherein:
the selectable subset of the plurality of identification features does not allow a unique identification of the human being (See Jung: Fig. 3, and Col. 13 Lines 17-36, "In an embodiment, the anonymizer circuit 330 operable to generate an anonymized image that includes a decreased fidelity of a property of the subject of the acquired image in response to an anonymization policy further includes an anonymizer circuit operable to generate an anonymized image that includes at least one of a decreased resolution of a property, an obscuring of a property, a blackout of a property, and/or a removal of a property of the subject of the acquired image in response to an anonymization policy. In a further embodiment, the anonymizer circuit further includes an anonymizer circuit operable to generate an anonymized image that includes a decreased fidelity of an aspect, a property, and/or an attribute of the subject of the acquired image in response to an anonymization policy. In another embodiment, the anonymizer circuit further includes an anonymizer circuit operable to generate an anonymized image that includes a decreased fidelity of at least one of a face, a license plate, a label, and/or a recognizable property associated with the subject of the acquired image in response to an anonymization policy". Note that the blackout or removal properties may be the identification features that does not allow unique identification of the human being);
the first device is further configured to provide an obfuscated image (See Jung: Fig. 3-5, and Col. 14 Lines 58-67 ~ Col. 15 Lines 1-18, "FIG. 5 illustrates an image 401 of the exemplary embodiment of an environment 400 of FIG. 4 with a decreased fidelity of a property of a subject. In an embodiment, the decreased fidelity may be implemented and/or practiced using systems, devices, apparatus, and/or methods disclosed herein. For example, the system 300 described in conjunction with FIG. 3 may be used to implement and/or practice the decreased fidelity of a property of a subject depicted in an image 405. For example, FIG. 5 illustrates an embodiment where the display 440B of the image capture device 420 indicating a "Green" color-based visual indication of the anonymization policy, corresponding to decreasing a fidelity of facial properties and license plate properties depicted in acquired images to be indistinguishable. The decreased fidelity of the illustrated embodiment includes a decreased fidelity of facial properties of human subjects 408A, 408B, and 408( implemented by obscuring portions of their faces and/or heads. The faces and/or heads of human subjects may be located in the image using techniques known to those skilled in the art, including artificial intelligence, and/or facial recognition techniques. FIG. 5 also illustrates a decreased fidelity of the identification number "XYZ 123" property of a license plate of a car subject 406 in the image that was implemented by an obscuring portion over the license identification number. In an embodiment, the image 401 may be used to study human and/or vehicle traffic patterns in the area 403 while preserving the anonymity of human subjects and/or registered owners of vehicles"),
wherein in the obfuscated image the facial image data of the human being is replaced with placeholder data;
the first device is further configured to provide the obfuscated image and the descriptor to a second device (See Jung: Fig. 3, and Col. 11 Lines 52-66, "FIG. 3 illustrates an exemplary embodiment of an environment 300 in which an embodiment may be implemented. The exemplary environment includes an area 303 that includes, or that may include, one or more subjects 305 whose image may be acquired by an imaging device. The environment also includes a system 310 that includes an imaging device 320, an anonymizer circuit 330, and a display 340. The imaging device is operable to acquire an image of a subject. The anonymizer circuit is operable to generate an anonymized image that includes a decreased fidelity of a property of a subject of the acquired image in response to an anonymization policy. The display is operable to provide a human-perceivable indication of the anonymization policy. In an embodiment, the display includes colored lights indicating the anonymization policy then in effect". Note that the obscured image generation and display portion, etc. of the system may be regarded as a second device); and
the second device configured to produce, based on jointly processing the obfuscated image and the descriptor (See Jung: Fig. 3, and Col. 12 Lines 51-63, "In an embodiment, the anonymizer circuit 330 operable to generate an anonymized image that includes a decreased fidelity of a property of the subject of the acquired image in response to an anonymization policy further includes an anonymizer circuit operable to: generate an anonymized image that includes a decreased fidelity of a property of the subject of the acquired image in response to an anonymization policy; and restrict a dissemination of the acquired image. For example, a restriction of a dissemination of the acquired image may include blocking a dissemination of the image where the image does not include the decreased fidelity of a property of a subject of the acquired image in response to an anonymization policy. By way of further example, a restriction of a dissemination of the acquired image may include requiring a special permission before a dissemination of the image where the image does not include the decreased fidelity of a property of a subject of the acquired image in response to an anonymization policy". Note that the anonymized image may be the obfuscated image, and the fidelity of the subject property may be the descriptor which may be the set of features of the facial images), an anonymized image that includes a replacement of at least a portion of the placeholder data of the obfuscated image with artificial facial image data corresponding to the descriptor.
However, Jung fails to explicitly disclose that a mimic expression feature having a characteristic; determine a descriptor associated with the facial image data of the human being wherein the descriptor includes information describing the characteristic of the mimic expression feature and the characteristic of each identification feature of a selectable subset of the plurality of identification features; wherein in the obfuscated image the facial image data of the human being is replaced with placeholder data; and an anonymized image that includes a replacement of at least a portion of the placeholder data of the obfuscated image with artificial facial image data corresponding to the descriptor.
However, Astarabadi teaches that a mimic expression feature having a characteristic (See Astarabadi: Figs. 2A-D, and [0179], "The second device can: transform the first face model for the first user and the first feed of facial landmark containers into a first synthetic face image feed that mimics the first user's face depicted in the first video feed; and render this first synthetic face image feed over the first background. Concurrently, the second device can: implement a generic face model (e.g., for a cartoon character, a cat, a dog, a mouse) to transform the third feed of facial landmark containers into a third synthetic face image feed that mimics a generic face (e.g., the cartoon character, the cat, the dog, the mouse); and render this third synthetic face image-adjacent the first synthetic face image feed-over the first background. In this implementation, once the first device (or the remote computer system) generates this new face model for the third face, the second device can download this new face model and transition to generating the third synthetic face image feed according to this new face model rather than the generic face model". Note that mimic a generic face may be the mimic expressions for the facial image).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention was effectively filed to modify Jung to have a mimic expression feature having a characteristic as taught by Astarabadi in order to enable the user to select 2D color profiles, schema, and background to match his/her audience and/or to improve the users confidence when engaging in video calls with others (See Astarabadi: Fig. 1, and [0015], "Furthermore, the first device can enable the first user to elect or customize other synthetic content rendered with a synthetic reconstruction of her face at the second device, such as including: a custom background that differs from the first user's true environment during the video call; or makeup, facial hair schema, hair style, clothing and/or accessory schema that differ from the first user's true appearance during the video call. The second device can then incorporate this synthetic content elected by the first user into the first synthetic video feed during the video call, thereby enabling the first user to control how she is presented to the second user"). Jung teaches a method and system that may capture an image with many subjects on the see ne, select the anonymization policies, apply the anonymization policies to the captured images to generate anonymized images with identification information such as human faces blacked out or obscured, and output or display the anonymized image to the user or viewer; while Astarabadi teaches a system and method that may generate a synthetic facial image with mimic expressions to represent the first user and present the synthetic image to the second user. Therefore, it is obvious to one of ordinary skill in the art to modify Jung by Astarabadi to have mimic expression in the generic facial image generation and use the synthetic facial image to anonymize the human images. The motivation to modify Jung by Astarabadi is "Use of known technique to improve similar devices (methods, or products) in the same way".
However, Jung, modified by Astarabadi, fails to explicitly disclose that determine a descriptor associated with the facial image data of the human being wherein the descriptor includes information describing the characteristic of the mimic expression feature and the characteristic of each identification feature of a selectable subset of the plurality of identification features; wherein in the obfuscated image the facial image data of the human being is replaced with placeholder data; and an anonymized image that includes a replacement of at least a portion of the placeholder data of the obfuscated image with artificial facial image data corresponding to the descriptor.
However, Lee teaches that determine a descriptor associated with the facial image data of the human being (See Lee: Fig. 2, and [0045], "FIG. 2 (a) shows an input image, FIG. 2 (b) shows a facial image detected by the face detection unit 110, FIG. 2 (c) shows a facial image that is normalized by the normalization unit 120, FIG. 2 (d) shows a facial image expressing key points extracted by the key point position setting unit 130, and FIG. 2 (e) shows a vector K extracted by the key point descriptor extraction unit 140, which may be expressed as K={kl, ... , kM} (where kl denotes a descriptor extracted from the first key point and kM denotes a descriptor extracted from the M-th key point). FIG. 2 (f) shows a case where a change in the facial image is detected by the matching unit 150 and the input facial image IN is recognized as the same face as a pre-stored facial image S1". Note that the descriptors for the facial images are the facial image key point features aggregated in sets of vector, combining with the second art of Astarabadi, the descriptors may also include the mimic features of the facial images) wherein the descriptor includes information describing the characteristic of the mimic expression feature and the characteristic of each identification feature of a selectable subset of the plurality of identification features.
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention was effectively filed to modify Jung to have determine a descriptor associated with the facial image data of the human being (See Lee: Fig. 1, and [0009], "In addition, another object of the present disclosure is to improve the performance of face recognition by recognizing a face with reference to descriptors of key points extracted from respectively predetermined positions of previously stored facial images and an input facial image, in a manner that addresses the problems of conventional methods"). Jung teaches a method and system that may capture an image with many subjects on the scene, select the anonymization policies, apply the anonymization policies to the captured images to generate anonymized images with identification information such as human faces blacked out or obscured, and output or display the anonymized image to the user or viewer; while Lee teaches a system and method that may determine the descriptors for the input facial image characterization and use the facial image descriptors to identify the faces of the human subjects accurately. Therefore, it is obvious to one of ordinary skill in the art to modify Jung by Lee to have descriptors associated with the facial images and the descriptors characterize the facial image features including various expressions for facial identifications. The motivation to modify Jung by Lee is "Use of known technique to improve similar devices (methods, or products) in the same way".
However, Jung, modified by Astarabadi and Lee, fails to explicitly disclose that wherein the descriptor includes information describing the characteristic of the mimic expression feature and the characteristic of each identification feature of a selectable subset of the plurality of identification features; wherein in the obfuscated image the facial image data of the human being is replaced with placeholder data; and an anonymized image that includes a replacement of at least a portion of the placeholder data of the obfuscated image with artificial facial image data corresponding to the descriptor.
However, Zavesky teaches that wherein in the obfuscated image the facial image data of the human being is replaced with placeholder data (See Zavesky: Fig. 1, and [0011], "Further, the disclosed systems and methods may accommodate consumer-quality media (e.g., social media from mobile devices, video from an event, and user-generated video) as candidates for object replacement. For example, a consumer entertainment application may utilize the disclosed systems and methods for replacement of faces, bodies, clothing, etc., based on a consumer's personal photo collection. The disclosed systems and methods may also be utilized in security and surveillance applications to replace unknown or unapproved faces, products, or objects with generic placeholders. For example, an object representation database may include faces, products, and objects that have not been approved for release or that need to be prevented from display. When the disclosed system is executed, context rules may identify that the particular faces, products, or objects are to be replaced with placeholders. Thus, the system may automatically determine when the particular face, products, or objects should be replaced in an image or video". Note that when the face image is not approved for use, and placeholder is displayed or placed in the face image position); and
a replacement of at least a portion of the placeholder data of the obfuscated image with artificial facial image data corresponding to the descriptor (See Zavesky: Figs. 3-5, and [0033], "The data provider 310 may also provide context rules 314 for the replacement process 300. For example, the data provider 310 may specify conditions (e.g., if, when, and how) for replacing a source object in the video data 312 with the replacement object 316. To illustrate, the data provider 310 may specify audio or music requirements, video semantic requirements, lighting requirements, texture requirements, other visual requirements, or any combination thereof. The context rules may be any combination of logical statements (e.g., AND, NOT, OR), statistical priors (e.g., choose object A 50% of the time and object B 50% of the time to replace a specific source object), strict geometric constraints (e.g., a team flag should only be substituted in the top third of an image frame of the video data 312), or any combination thereof. Other context rules may be provided from a controlling system that may have external knowledge of demographic information (e.g., viewership age), scheduled transmission time of video data 312 or video stream consumption (e.g. primetime IV broadcast), or any combination thereof. For example, during post-production, context rules 314 provided by a data provider (e.g., a movie director) may specify that a main character's face (e.g., Tom Cruise) should replace a specific stunt double's face in scenes where the movie director wants the final output video to include the main character performing a particular stunt (i.e., in specific video frames)").
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention was effectively filed to modify Jung to have wherein in the obfuscated image the facial image data of the human being is replaced with placeholder data; and a replacement of at least a portion of the placeholder data of the obfuscated image with artificial facial image data corresponding to the descriptor as taught by Zavesky in order to enable providing a content producer utilizing a replacing system to perform post-production editing to show a Railway logo in commercial without cost associated with reshooting commercial (See Zavesky: Fig. 2, and [0031], "Computed properties and information on the object (i.e., generated 3D model 218, computed audio and visual context 220, computed color and lighting conditions 222, and computed BRDF and texture conditions 224) may be stored 226 in a database. The stored properties and information may enable the system 100 of FIG. 1 to accurately perform object replacement. For example, the system 100 of FIG. 1 may leverage information stored in the database (e.g., existing objects and corresponding properties) to supplement or augment generating a replacement object (i.e., a second object to replace a target first object)"). Jung teaches a method and system that may capture an image with many subjects on the scene, select the anonymization policies, apply the anonymization policies to the captured images to generate anonymized images with identification information such as human faces blacked out or obscured, and output or display the anonymized image to the user or viewer; while Zavesky teaches a system and method that may determine if there is a need to replace the first object with a second object (face) based on some rule, put a placeholder in the position of the first object, and replace the first object (e.g., a first face) with the second object (e.g., a second face). Therefore, it is obvious to one of ordinary skill in the art to modify Jung by Zavesky to have a placeholder for the anonymized facial image and replacing the placeholder with a anonymized image. The motivation to modify Jung by Zavesky is "Use of known technique to improve similar devices (methods, or products) in the same way".
However, Jung, modified by Astarabadi, Lee, and Zavesky, fails to explicitly disclose that wherein the descriptor includes information describing the characteristic of the mimic expression feature and the characteristic of each identification feature of a selectable subset of the plurality of identification features; and an anonymized image that includes a replacement of at least a portion of the placeholder data of the obfuscated image with artificial facial image data corresponding to the descriptor.
However, Whitehill teaches that wherein the descriptor includes information describing the characteristic of the mimic expression feature and the characteristic of each identification feature of a selectable subset of the plurality of identification features (See Whitehill: Fig. 1, and [0012], “In one particular approach, the original facial image is encoded as a feature set. The feature set contains personal identity components that contribute to recognizability of the facial image and expression components that contribute to the emotional expression of the facial image. A perturbation transform is applied to the feature set. The perturbation transform substantially perturbs the personal identity components and substantially preserves at least some of the expression components. The perturbed feature set is decoded to obtain the synthesized facial image. In this way, the synthesized facial image is anonymized while still retaining some of the emotional expression of the original facial image”; [0029], “The image filter outputs are passed to a feature selection module at 110. The feature selection module, whose parameters are found using machine learning methods, can include the use of a machine learning technique that is trained on a database of spontaneous expressions by subjects that have been manually labeled for facial actions from the Facial Action Coding System (FACS). The feature selection module 110 processes the image filter outputs for each of the plurality of image windows to choose a subset of the characteristics or parameters to pass to the classification module at 112. The feature selection module results for the two or more image windows can optionally be combined and processed by a classifier process at 112 to produce a joint decision regarding the posterior probability of the presence of an action unit in the face shown in the image. The classifier process can utilize machine learning on the database of spontaneous facial expressions. At 114, a promoted output of the process 100 can be a score for each of the action units that quantifies the observed “content” of each of the action units in the face shown in the image”; [0010], “The present invention overcomes the limitations of the prior art by perturbing recognizable facial images, in a manner such that the perturbed facial images are no longer recognizable while still preserving at least part of the emotional expression or other attributes of the original facial image or the subject or circumstances”; and [0014], “In yet another aspect, facial images are anonymized while preserving attributes of the facial image other than facial expression. For example, facial images may be perturbed so that they are no longer recognizable as the original subject, but while still preserving gender, age or other attributes of the original facial image”. Note that the feature set is mapped to the “descriptor”, the expression components are mapped to the mimic expression feature, the personal identification components are mapped to the personal identification features, the perturbation is made to the personal identification components to render the output image un-identifiable); and
an anonymized image that includes a replacement of at least a portion of the placeholder data of the obfuscated image with artificial facial image data corresponding to the descriptor (See Whitehill: Fig. 1, and [0010], “The present invention overcomes the limitations of the prior art by perturbing recognizable facial images, in a manner such that the perturbed facial images are no longer recognizable while still preserving at least part of the emotional expression or other attributes of the original facial image or the subject or circumstances”; and [0043], “The feature set is input to the transform module 424, which applies a perturbation transform to the feature set that substantially perturbs its personal identity components but substantially preserves at least some of its expression components. The output of the transform module 424 is a perturbed feature set, which serves as an input to the decoder module 426. The decoder module 426 decodes the perturbed feature set to obtain the synthesized facial image 430. The synthesized facial image 430, which is now “anonymized,” together with a facial expression category label of the original facial image 400 (e.g., happy, sad, etc.), can be used to train an AFER system”. Note that the perturbation is applied to the personal identification components, and the output synthetic image is “anonymized” but keep the facial expression category label of the original facial image, which means that the portion of the facial image with personal identification features is replaced (placeholder) by an artificial image).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention was effectively filed to modify Jung to have wherein the descriptor includes information describing the characteristic of the mimic expression feature and the characteristic of each identification feature of a selectable subset of the plurality of identification features; and an anonymized image that includes a replacement of at least a portion of the placeholder data of the obfuscated image with artificial facial image data corresponding to the descriptor as taught by Whitehill in order to improve the techniques to generate anonymized facial images (See Whitehill: Fig. 1, and [0009], "Therefore, there is a need for improved techniques to generate anonymized facial images"). Jung teaches a method and system that may capture an image with many subjects on the scene, select the anonymization policies, apply the anonymization policies to the captured images to generate anonymized images with identification information such as human faces blacked out or obscured, and output or display the anonymized image to the user or viewer; while Whitehill teaches a system and method that may perturb the personal identification components in the original facial image, reserve the expression components in the original facial image, and generate a synthetic “anonymized” image for the user to keep the PII privately and securely. Therefore, it is obvious to one of ordinary skill in the art to modify Jung by Whitehill to have perturbation applied to the PPI and reservations applied to the mimic expression to generate the ”anonymized” synthetic images for the users. The motivation to modify Jung by Whitehill is "Use of known technique to improve similar devices (methods, or products) in the same way".
Regarding claim 3, Jung, Astarabadi, Lee, Zavesky, and Whitehill teach all the features with respect to claim 1 as outlined above. Further, Jung teaches that the personal data processing system according to claim 1, wherein the second device includes a storage configured to store the descriptor and the obfuscated image (See Jung: Fig. 24, and Col. 22 Lines 49-60, "For example, the anonymization policy may be applied to the class of subjects depicted in the captured image before the captured image is saved in a storage device, such as before an anonymized captured image 1642 is saved in a storage device 1640 of the imaging device 1610 and/or before an anonymized captured image 1694 is saved in a storage device 1690 of the anonymization device 1650. In another example, a captured image may be transitorily saved in a volatile memory (not shown) before the anonymization policy is applied and the image persistently saved in a more permanent memory and/or non-volatile media, such as the storage 1640 and/or the storage 1690").
Regarding claim 4, Jung, Astarabadi, Lee, Zavesky, and Whitehill teach all the features with respect to claim 3 as outlined above. Further, Jung teaches that the personal data processing system according to claim 3, wherein the second device is configured to retrieve the descriptor from the storage and to provide at least one of the information or a part of the information of the descriptor or the obfuscated image to a user (See Jung: Fig. 20, and Col. 20 Lines 14-29, "FIG. 20 illustrates an alternative embodiment of the exemplary operational flow 1300 of FIG. 19. The ambiguation operation 1310 may include at least one additional operation. The at least one additional operation may include an operation 1312 and/or an operation 1314. The operation 1312 obscures a person, article, object, face, license plate, and/or label of a subject depicted in a received image in response to a privacy policy. The operation 1314 obscures a property of a subject depicted in a received digital and/or analog image in response to a privacy policy. The communication operation 1320 may include at least one additional operation, such as the operation 1322. The operation 1322 sends a signal indicative of the privacy policy and is receivable by at least one of a remotely located image capture device, a privacy policy indicator at a location physically distant from the device, and/or a local privacy policy indicator").
Regarding claim 5, Jung, Astarabadi, Lee, Zavesky, and Whitehill teach all the features with respect to claim 1 as outlined above. Further, Jung teaches that the personal data processing system of claim 1, wherein the second device is further configured to send the anonymized image to a third device configured to display the anonymized image to a user (See Jung: Figs. 3-5, and Col. 14 Lines 58-67 ~ Col. 15 Lines 1-18, "FIG. 5 illustrates an image 401 of the exemplary embodiment of an environment 400 of FIG. 4 with a decreased fidelity of a property of a subject. In an embodiment, the decreased fidelity may be implemented and/or practiced using systems, devices, apparatus, and/or methods disclosed herein. For example, the system 300 described in conjunction with FIG. 3 may be used to implement and/or practice the decreased fidelity of a property of a subject depicted in an image 405. For example, FIG. 5 illustrates an embodiment where the display 440B of the image capture device 420 indicating a "Green" color-based visual indication of the anonymization policy, corresponding to decreasing a fidelity of facial properties and license plate properties depicted in acquired images to be indistinguishable. The decreased fidelity of the illustrated embodiment includes a decreased fidelity of facial properties of human subjects 408A, 408B, and 408C implemented by obscuring portions of their faces and/or heads. The faces and/or heads of human subjects may be located in the image using techniques known to those skilled in the art, including artificial intelligence, and/or facial recognition techniques. FIG. 5 also illustrates a decreased fidelity of the identification number "XYZ 123" property of a license plate of a car subject 406 in the image that was implemented by an obscuring portion over the license identification number. In an embodiment, the image 401 may be used to study human and/or vehicle traffic patterns in the area 403 while preserving the anonymity of human subjects and/or registered owners of vehicles". Note that the obscured image 405 with identifiable information blacked out is output and displayed to the user).
Regarding claim 6, Jung, Astarabadi, Lee, Zavesky, and Whitehill teach all the features with respect to claim 1 as outlined above. Further, Zavesky teaches that the personal data processing system of claim 1, wherein the image data includes video data (See Zavesky: Fig. 1, and [0015], "Referring to FIG. 1, a particular illustrative embodiment of a system to digitally replace objects in images or video is disclosed and generally designated 100. The system 100 includes a three-dimensional (3D) model generator 120 coupled to a database 130. The 3D model generator 120 may be configured to generate a 3D model of an object (e.g., representative first object 112) depicted in video content 110 and to store the generated 3D model 132 of the first object 112 in the database 130. The database 130 may also include a library 134 of 3D models generated for a plurality of objects. The library 134 of 3D models may be categorized and organized based on visual properties of the objects, scene or background data surrounding the objects, other information descriptive of the objects, or a combination thereof").
Regarding claim 7, Jung, Astarabadi, Lee, Zavesky, and Whitehill teach all the features with respect to claim 1 as outlined above. Further, Astarabadi teaches that the personal data processing system according to claim 1, wherein at least one of the first device or the second device are real-time devices configured to process data in real-time (See Astarabadi: Fig. 1, and [0067], "Like the first method SlO0 described above, Blocks of the second method S200 can be executed by native or browser-based applications executing on a set of computing devices (e.g., smartphones, tablets, laptop computers) during a video call between two users in order: to compress a first video feed of a first user into a first lightweight (e.g., sub-kilobyte) feed of constellations of facial landmarks at a first device; and to reconstruct this first video feed at a second device by injecting this feed of facial landmark constellations and a first (pseudo-) unique face model of the first user into a synthetic face generator, which outputs a first stream of synthetic, photorealistic images of the first user that the second device then renders in near real-time. Simultaneously, the second device can compress a second video feed of the second user into a second lightweight constellation of facial landmarks; and the first device can reconstruct this second video feed by injecting this feed of facial landmark constellations and a second (pseudo-) unique face model of the second user into a synthetic face generator, which outputs a second stream of synthetic, photorealistic images of the second user that the first device then renders in near real-time").
Regarding claim 8, Jung, Astarabadi, Lee, Zavesky, and Whitehill teach all the features with respect to claim 1 as outlined above. Further, Astarabadi teaches that the personal data processing system according to claim 1, wherein the first device and the second device are connected via a communication network configured to transfer data including the image data and the descriptor from the first device to the second device (See Astarabadi: Fig. 1, and [0011], "In particular, rather than transmit and receive data-rich video feeds during a video call, a first device executing the first method S100 can instead extract facial landmark constellations from a video feed and transmit a feed of facial landmark constellations to a second device. The second device can then leverage a local model-such as a generic model or a model specific to the first user-to reconstruct a photorealistic representation of the first user's face and then render this photorealistic synthetic video feed in near real-time. The second user may thus experience the video call as though a color video was received from the first user's device but without necessitating a high-bandwidth, low-latency data connection between the first and second devices. The second and first devices can concurrently execute the same process in reverse to extract facial landmark constellations from a second video feed recorded at the second device, to transmit this second feed of facial landmark constellations to the first device, and to leverage a local model to reconstruct a photorealistic synthetic video feed of the second user. More specifically, by extracting facial landmark constellations from a high-definition video feed according to the method S100, the first device can compress this high-definition video feed by multiple orders of magnitude (e.g., by approximately 100 times). Transmission of a feed of facial landmark constellations-at the same frame rate as the original high-definition video (e.g., 24 frames per second)-from the first device to the second device during a video call may therefore require significantly less bandwidth than the original high-definition video (e.g., 10 kilobits per second rather than 1.5 Megabits per second). Therefore, the first method S100 can enable a high-quality video call with significantly less upload bandwidth to transmit a representation of a first video from the first device to a computer network and significantly less download bandwidth required to download this representation of the first video to the second device, and vice versa").
Regarding claim 9, Jung, Astarabadi, Lee, Zavesky, and Whitehill teach all the features with respect to claim 1 as outlined above. Further, Jung, Astarabadi, Lee, Zavesky, and Whitehill teach that a method of operating a personal data processing system, the method (See Jung: Fig. 24, and Col. 21 Lines 20-29, "FIG. 24 illustrates an exemplary embodiment of a system 1600. The system includes an imaging device 1610, an anonymization device 1650, and a broadcast device. The imaging device is operable to capture an image depicting a class of subjects 1605 that may have a temporal presence in an area 1603. The broadcast device is operable to display a human-perceivable indication of a selected anonymization policy to at least a portion of the area. The anonymization policy includes obscuring a property of the class of subjects depicted in the captured image") comprising:
capturing image data including facial image data of a human being (See Jung: Fig. 3, and Col. 13 Lines 16-27, "In an embodiment, the imaging device 320 operable to acquire an image of a subject further includes a digital imaging device operable to acquire an image of a subject of the potential image subject(s) 305. In another embodiment, the imaging device operable to acquire an image of a subject further includes a digital camera operable to acquire an image of a subject. For example, the digital camera may include a lens 322, an imaging element 324, an analog to digital converter 326, and/or a processor 328. In a further embodiment, the imaging device operable to acquire an image of a subject further includes an imaging device operable to capture an image of a subject of the potential image subject(s) 305"),
the facial image data of a human being including a plurality of identification features each having a characteristic (See Jung: Fig. 24, and Col. 21 Lines 19-32, "In a further embodiment, the anonymization device further includes an anonymization device having a selector circuit operable to select an anonymization policy, where the anonymization policy includes obscuring at least one of a variable, a recognizable, a distinguishable, and/or a unique aspect of a property of the class of subjects depicted in the captured image. In another embodiment, the anonymization device further includes an anonymization device having a selector circuit operable to select an anonymization policy, where the anonymization policy includes obscuring at least one of a variable, a recognizable, a distinguishable, and/or a unique aspect of a property of each instance of the class of subjects depicted in the captured image". Note that the unique aspect of the property of the subject may be the identification features of the face image) and a mimic expression feature having a characteristic (See Astarabadi: Figs. 2A-D, and [0179], "The second device can: transform the first face model for the first user and the first feed of facial landmark containers into a first synthetic face image feed that mimics the first user's face depicted in the first video feed; and render this first synthetic face image feed over the first background. Concurrently, the second device can: implement a generic face model (e.g., for a cartoon character, a cat, a dog, a mouse) to transform the third feed of facial landmark containers into a third synthetic face image feed that mimics a generic face (e.g., the cartoon character, the cat, the dog, the mouse); and render this third synthetic face image-adjacent the first synthetic face image feed-over the first background. In this implementation, once the first device (or the remote computer system) generates this new face model for the third face, the second device can download this new face model and transition to generating the third synthetic face image feed according to this new face model rather than the generic face model". Note that mimic a generic face may be the mimic expressions for the facial image);
detecting the facial image data of the human being in the image data (See Jung: Figs. 3-5, and Col. 14 Lines 66-67 ~ Col. 15 Lines 1-18, "FIG. 5 illustrates an embodiment where the display 440B of the image capture device 420 indicating a "Green" color-based visual indication of the anonymization policy, corresponding to decreasing a fidelity of facial properties and license plate properties depicted in acquired images to be indistinguishable. The decreased fidelity of the illustrated embodiment includes a decreased fidelity of facial properties of human subjects 408A, 408B, and 408C implemented by obscuring portions of their faces and/or heads. The faces and/or heads of human subjects may be located in the image using techniques known to those skilled in the art, including artificial intelligence, and/or facial recognition techniques. FIG. 5 also illustrates a decreased fidelity of the identification number "XYZ 123" property of a license plate of a car subject 406 in the image that was implemented by an obscuring portion over the license identification number. In an embodiment, the image 401 may be used to study human and/or vehicle traffic patterns in the area 403 while preserving the anonymity of human subjects and/or registered owners of vehicles". Note that using face recognition technique to recognize face in the image may be the face detection);
determining a descriptor associated with the facial image data of the human being, wherein the descriptor (See Whitehill: Fig. 1, and [0012], “In one particular approach, the original facial image is encoded as a feature set. The feature set contains personal identity components that contribute to recognizability of the facial image and expression components that contribute to the emotional expression of the facial image. A perturbation transform is applied to the feature set. The perturbation transform substantially perturbs the personal identity components and substantially preserves at least some of the expression components. The perturbed feature set is decoded to obtain the synthesized facial image. In this way, the synthesized facial image is anonymized while still retaining some of the emotional expression of the original facial image”; [0029], “The image filter outputs are passed to a feature selection module at 110. The feature selection module, whose parameters are found using machine learning methods, can include the use of a machine learning technique that is trained on a database of spontaneous expressions by subjects that have been manually labeled for facial actions from the Facial Action Coding System (FACS). The feature selection module 110 processes the image filter outputs for each of the plurality of image windows to choose a subset of the characteristics or parameters to pass to the classification module at 112. The feature selection module results for the two or more image windows can optionally be combined and processed by a classifier process at 112 to produce a joint decision regarding the posterior probability of the presence of an action unit in the face shown in the image. The classifier process can utilize machine learning on the database of spontaneous facial expressions. At 114, a promoted output of the process 100 can be a score for each of the action units that quantifies the observed “content” of each of the action units in the face shown in the image”; [0010], “The present invention overcomes the limitations of the prior art by perturbing recognizable facial images, in a manner such that the perturbed facial images are no longer recognizable while still preserving at least part of the emotional expression or other attributes of the original facial image or the subject or circumstances”; and [0014], “In yet another aspect, facial images are anonymized while preserving attributes of the facial image other than facial expression. For example, facial images may be perturbed so that they are no longer recognizable as the original subject, but while still preserving gender, age or other attributes of the original facial image”. Note that the feature set is mapped to the “descriptor”, the expression components are mapped to the mimic expression feature, the personal identification components are mapped to the personal identification features, the perturbation is made to the personal identification components to render the output image un-identifiable) includes information describing the characteristic of the mimic expression feature and the characteristic of each identification feature of a selectable subset of the plurality of identification features (See Lee: Fig. 2, and [0045], "FIG. 2 (a) shows an input image, FIG. 2 (b) shows a facial image detected by the face detection unit 110, FIG. 2 (c) shows a facial image that is normalized by the normalization unit 120, FIG. 2 (d) shows a facial image expressing key points extracted by the key point position setting unit 130, and FIG. 2 (e) shows a vector K extracted by the key point descriptor extraction unit 140, which may be expressed as K={kl, ..., kM} (where kl denotes a descriptor extracted from the first key point and kM denotes a descriptor extracted from the M-th key point). FIG. 2 (f) shows a case where a change in the facial image is detected by the matching unit 150 and the input facial image IN is recognized as the same face as a pre-stored facial image S1". Note that the descriptors for the facial images are the facial image key point features aggregated in sets of vector, combining with the second art of Astarabadi, the descriptors may also include the mimic features of the facial images),
wherein the selectable subset of the plurality of identification features substantially does not allow a unique identification of the human being (See Jung: Fig. 3, and Col. 13 Lines 17-36, "In an embodiment, the anonymizer circuit 330 operable to generate an anonymized image that includes a decreased fidelity of a property of the subject of the acquired image in response to an anonymization policy further includes an anonymizer circuit operable to generate an anonymized image that includes at least one of a decreased resolution of a property, an obscuring of a property, a blackout of a property, and/or a removal of a property of the subject of the acquired image in response to an anonymization policy. In a further embodiment, the anonymizer circuit further includes an anonymizer circuit operable to generate an anonymized image that includes a decreased fidelity of an aspect, a property, and/or an attribute of the subject of the acquired image in response to an anonymization policy. In another embodiment, the anonymizer circuit further includes an anonymizer circuit operable to generate an anonymized image that includes a decreased fidelity of at least one of a face, a license plate, a label, and/or a recognizable property associated with the subject of the acquired image in response to an anonymization policy". Note that the blackout or removal properties may be the identification features that does not allow unique identification of the human being);
providing an obfuscated image (See Jung: Fig. 3-5, and Col. 14 Lines 58-67 ~ Col. 15 Lines 1-18, "FIG. 5 illustrates an image 401 of the exemplary embodiment of an environment 400 of FIG. 4 with a decreased fidelity of a property of a subject. In an embodiment, the decreased fidelity may be implemented and/or practiced using systems, devices, apparatus, and/or methods disclosed herein. For example, the system 300 described in conjunction with FIG. 3 may be used to implement and/or practice the decreased fidelity of a property of a subject depicted in an image 405. For example, FIG. 5 illustrates an embodiment where the display 440B of the image capture device 420 indicating a "Green" color-based visual indication of the anonymization policy, corresponding to decreasing a fidelity of facial properties and license plate properties depicted in acquired images to be indistinguishable. The decreased fidelity of the illustrated embodiment includes a decreased fidelity of facial properties of human subjects 408A, 408B, and 408C implemented by obscuring portions of their faces and/or heads. The faces and/or heads of human subjects may be located in the image using techniques known to those skilled in the art, including artificial intelligence, and/or facial recognition techniques. FIG. 5 also illustrates a decreased fidelity of the identification number "XYZ 123" property of a license plate of a car subject 406 in the image that was implemented by an obscuring portion over the license identification number. In an embodiment, the image 401 may be used to study human and/or vehicle traffic patterns in the area 403 while preserving the anonymity of human subjects and/or registered owners of vehicles"),
wherein in the obfuscated image the facial image data of the human being is replaced with placeholder data (See Zavesky: Fig. 1, and [0011], "Further, the disclosed systems and methods may accommodate consumer-quality media (e.g., social media from mobile devices, video from an event, and user-generated video) as candidates for object replacement. For example, a consumer entertainment application may utilize the disclosed systems and methods for replacement of faces, bodies, clothing, etc., based on a consumer's personal photo collection. The disclosed systems and methods may also be utilized in security and surveillance applications to replace unknown or unapproved faces, products, or objects with generic placeholders. For example, an object representation database may include faces, products, and objects that have not been approved for release or that need to be prevented from display. When the disclosed system is executed, context rules may identify that the particular faces, products, or objects are to be replaced with placeholders. Thus, the system may automatically determine when the particular face, products, or objects should be replaced in an image or video". Note that when the face image is not approved for use, and placeholder is displayed or placed in the face image position); and
producing, based on jointly processing the obfuscated image and the descriptor (See Jung: Fig. 3, and Col. 12 Lines 51-63, "In an embodiment, the anonymizer circuit 330 operable to generate an anonymized image that includes a decreased fidelity of a property of the subject of the acquired image in response to an anonymization policy further includes an anonymizer circuit operable to: generate an anonymized image that includes a decreased fidelity of a property of the subject of the acquired image in response to an anonymization policy; and restrict a dissemination of the acquired image. For example, a restriction of a dissemination of the acquired image may include blocking a dissemination of the image where the image does not include the decreased fidelity of a property of a subject of the acquired image in response to an anonymization policy. By way of further example, a restriction of a dissemination of the acquired image may include requiring a special permission before a dissemination of the image where the image does not include the decreased fidelity of a property of a subject of the acquired image in response to an anonymization policy". Note that the anonymized image may be the obfuscated image, and the fidelity of the subject property may be the descriptor which may be the set of features of the facial images),
an anonymized image that includes a replacement of at least a portion of the placeholder data of the obfuscated image with artificial facial image data (See Whitehill: Fig. 1, and [0010], “The present invention overcomes the limitations of the prior art by perturbing recognizable facial images, in a manner such that the perturbed facial images are no longer recognizable while still preserving at least part of the emotional expression or other attributes of the original facial image or the subject or circumstances”; and [0043], “The feature set is input to the transform module 424, which applies a perturbation transform to the feature set that substantially perturbs its personal identity components but substantially preserves at least some of its expression components. The output of the transform module 424 is a perturbed feature set, which serves as an input to the decoder module 426. The decoder module 426 decodes the perturbed feature set to obtain the synthesized facial image 430. The synthesized facial image 430, which is now “anonymized,” together with a facial expression category label of the original facial image 400 (e.g., happy, sad, etc.), can be used to train an AFER system”. Note that the perturbation is applied to the personal identification components, and the output synthetic image is “anonymized” but keep the facial expression category label of the original facial image, which means that the portion of the facial image with personal identification features is replaced (placeholder) by an artificial image) corresponding to the descriptor (See Zavesky: Figs. 3-5, and [0033], "The data provider 310 may also provide context rules 314 for the replacement process 300. For example, the data provider 310 may specify conditions (e.g., if, when, and how) for replacing a source object in the video data 312 with the replacement object 316. To illustrate, the data provider 310 may specify audio or music requirements, video semantic requirements, lighting requirements, texture requirements, other visual requirements, or any combination thereof. The context rules may be any combination of logical statements (e.g., AND, NOT, OR), statistical priors ( e.g., choose object A 50% of the time and object B 50% of the time to replace a specific source object), strict geometric constraints (e.g., a team flag should only be substituted in the top third of an image frame of the video data 312), or any combination thereof. Other context rules may be provided from a controlling system that may have external knowledge of demographic information (e.g., viewership age), scheduled transmission time of video data 312 or video stream consumption (e.g. primetime IV broadcast), or any combination thereof. For example, during post-production, context rules 314 provided by a data provider (e.g., a movie director) may specify that a main character's face (e.g., Tom Cruise) should replace a specific stunt double's face in scenes where the movie director wants the final output video to include the main character performing a particular stunt (i.e., in specific video frames)").
Regarding claim 11, Jung, Astarabadi, Lee, Zavesky, and Whitehill teach all the features with respect to claim 9 as outlined above. Further, Jung teaches that the method according to claim 9, comprising:
storing the descriptor and the obfuscated image in a storage (See Jung: Fig. 24, and Col. 22 Lines 49-60, "For example, the anonymization policy may be applied to the class of subjects depicted in the captured image before the captured image is saved in a storage device, such as before an anonymized captured image 1642 is saved in a storage device 1640 of the imaging device 1610 and/or before an anonymized captured image 1694 is saved in a storage device 1690 of the anonymization device 1650. In another example, a captured image may be transitorily saved in a volatile memory (not shown) before the anonymization policy is applied and the image persistently saved in a more permanent memory and/or non-volatile media, such as the storage 1640 and/or the storage 1690").
Regarding claim 12, Jung, Astarabadi, Lee, Zavesky, and Whitehill teach all the features with respect to claim 11 as outlined above. Further, Jung teaches that the method according to claim 11, comprising:
retrieving the descriptor from the storage and providing at least one of the information or a part of the information of the descriptor or the obfuscated image to a user (See Jung: Fig. 20, and Col. 20 Lines 14-29, "FIG. 20 illustrates an alternative embodiment of the exemplary operational flow 1300 of FIG. 19. The ambiguation operation 1310 may include at least one additional operation. The at least one additional operation may include an operation 1312 and/or an operation 1314. The operation 1312 obscures a person, article, object, face, license plate, and/or label of a subject depicted in a received image in response to a privacy policy. The operation 1314 obscures a property of a subject depicted in a received digital and/or analog image in response to a privacy policy. The communication operation 1320 may include at least one additional operation, such as the operation 1322. The operation 1322 sends a signal indicative of the privacy policy and is receivable by at least one of a remotely located image capture device, a privacy policy indicator at a location physically distant from the device, and/or a local privacy policy indicator").
Regarding claim 13, Jung, Astarabadi, Lee, Zavesky, and Whitehill teach all the features with respect to claim 9 as outlined above. Further, Jung teaches that the method according to claim 9, comprising:
sending the anonymized image to a third device configured to display the anonymized image to a user (See Jung: Figs. 3-5, and Col. 14 Lines 58-67 ~ Col. 15 Lines 1-18, "FIG. 5 illustrates an image 401 of the exemplary embodiment of an environment 400 of FIG. 4 with a decreased fidelity of a property of a subject. In an embodiment, the decreased fidelity may be implemented and/or practiced using systems, devices, apparatus, and/or methods disclosed herein. For example, the system 300 described in conjunction with FIG. 3 may be used to implement and/or practice the decreased fidelity of a property of a subject depicted in an image 405. For example, FIG. 5 illustrates an embodiment where the display 440B of the image capture device 420 indicating a "Green" color-based visual indication of the anonymization policy, corresponding to decreasing a fidelity of facial properties and license plate properties depicted in acquired images to be indistinguishable. The decreased fidelity of the illustrated embodiment includes a decreased fidelity of facial properties of human subjects 408A, 408B, and 408C implemented by obscuring portions of their faces and/or heads. The faces and/or heads of human subjects may be located in the image using techniques known to those skilled in the art, including artificial intelligence, and/or facial recognition techniques. FIG. 5 also illustrates a decreased fidelity of the identification number "XYZ 123" property of a license plate of a car subject 406 in the image that was implemented by an obscuring portion over the license identification number. In an embodiment, the image 401 may be used to study human and/or vehicle traffic patterns in the area 403 while preserving the anonymity of human subjects and/or registered owners of vehicles". Note that the obscured image 405 with identifiable information blacked out is output and displayed to the user).
Regarding claim 14, Jung, Astarabadi, Lee, Zavesky, and Whitehill teach all the features with respect to claim 9 as outlined above. Further, Jung teaches that the method according to claim 9, comprising:
using the personal data processing system in at least one of a data analysis system or a communication system (See Jung: Fig. 1, and Col. 7 Lines 57-67 ~ Col. 8 Lines 1-7, "Other external input or output devices 39, such as a joystick, game pad, satellite dish, scanner or the like may be connected to the processing unit 21 through a USB port 40 and USB port interface 41, to the system bus 23. Alternatively, the other external input and output devices 39 may be connected by other interfaces, such as a parallel port, game port or other port. The computing device 20 may further include or be capable of connecting to a flash card memory (not shown) through an appropriate connection port (not shown). The computing device 20 may further include or be capable of connecting with a network through a network port 42 and network interface 43, and through wireless port 46 and corresponding wireless interface 47 may be provided to facilitate communication with other peripheral devices, including other computers, printers, and so on (not shown). It will be appreciated that the various components and connections shown are exemplary and other components and means of establishing communications links may be used". Note that the whole system with many components connected by the network may be the communication system).
Regarding claim 15, Jung, Astarabadi, Lee, Zavesky, and Whitehill teach all the features with respect to claim 1 as outlined above. Further, Astarabadi teaches that the personal data processing system according to claim 1, wherein the descriptor further includes at least one of: a facial image data bounding box, a facial image data landmark, or a facial image data attribute (See Astarabadi: Fig. 1, and [0009], “As shown in FIG. 1, a first method S100 for video conferencing includes, at a first device associated with a first user: recording a first sequence of video frames in Block Silo; for a first video frame, in the first sequence of video frames, recorded at alignment feature first time, detecting a first constellation of facial landmarks in the first video frame in Block S120 and storing locations of the first constellation of facial landmarks in a first container in Block S122; and transmitting the first container and a first audio packet, recorded at approximately the first time, to a second device in Block S130. The first method S100 also includes, at the second device associated with a second user: receiving the first container and the first audio packet in Block S132; deforming a face reconstruction model into alignment with locations of the first constellation of facial landmarks in the first container to generate a deformed 3D face mesh depicting the first user in Block S140; projecting the deformed 3D face mesh onto an image plane to generate a first synthetic video frame depicting the first user in Block S142; populating the first synthetic video frame with a background image in Block S144; and rendering the first synthetic video frame and outputting the first audio packet at approximately the first time in Block S150”).
Regarding claim 16, Jung, Astarabadi, Lee, Zavesky, and Whitehill teach all the features with respect to claim 1 as outlined above. Further, Astarabadi teaches that the personal data processing system according to claim 1, wherein the placeholder data comprises at least one of: image data having a predefined color, image data having a predefined pattern, or pixelated image data (See Astarabadi: Fig. 2, and [0085], “In particular, in Block S202, the remote computer system can train the conditional generative adversarial network to output a synthetic face image based on a set of input conditions, including: a facial landmark container, which captures relative locations (and/or sizes, orientations) of facial landmarks that represent a facial expression; and a face model, which contains a (pseudo-) unique set of coefficients characterizing a unique human face and secondary physiognomic features (e.g., face shape, skin tone, facial hair, makeup, freckles, wrinkles, eye color, hair color, hair style, and/or jewelry). Therefore, the remote computer system can input values from a facial landmark container and coefficients from a face model into the conditional generative adversarial network to generate a synthetic face image that depicts a face—(uniquely) represented by coefficients in the face model—exhibiting a facial expression represented by the facial landmark container”. Note that the unique features like face shape, hair styles, etc. are mapped to the predefined color or patterns).
Regarding claim 17, Jung, Astarabadi, Lee, Zavesky, and Whitehill teach all the features with respect to claim 9 as outlined above. Further, Astarabadi teaches that the method according to claim 9, wherein the descriptor further includes at least one of: a facial image data bounding box, a facial image data landmark, or a facial image data attribute (See Astarabadi: Fig. 1, and [0009], “As shown in FIG. 1, a first method S100 for video conferencing includes, at a first device associated with a first user: recording a first sequence of video frames in Block Silo; for a first video frame, in the first sequence of video frames, recorded at alignment feature first time, detecting a first constellation of facial landmarks in the first video frame in Block S120 and storing locations of the first constellation of facial landmarks in a first container in Block S122; and transmitting the first container and a first audio packet, recorded at approximately the first time, to a second device in Block S130. The first method S100 also includes, at the second device associated with a second user: receiving the first container and the first audio packet in Block S132; deforming a face reconstruction model into alignment with locations of the first constellation of facial landmarks in the first container to generate a deformed 3D face mesh depicting the first user in Block S140; projecting the deformed 3D face mesh onto an image plane to generate a first synthetic video frame depicting the first user in Block S142; populating the first synthetic video frame with a background image in Block S144; and rendering the first synthetic video frame and outputting the first audio packet at approximately the first time in Block S150”).
Regarding claim 18, Jung, Astarabadi, Lee, Zavesky, and Whitehill teach all the features with respect to claim 9 as outlined above. Further, Astarabadi teaches that the method according to claim 9, wherein the placeholder data comprises at least one of: image data having a predefined color, image data having a predefined pattern, or pixelated image data (See Astarabadi: Fig. 2, and [0085], “In particular, in Block S202, the remote computer system can train the conditional generative adversarial network to output a synthetic face image based on a set of input conditions, including: a facial landmark container, which captures relative locations (and/or sizes, orientations) of facial landmarks that represent a facial expression; and a face model, which contains a (pseudo-) unique set of coefficients characterizing a unique human face and secondary physiognomic features (e.g., face shape, skin tone, facial hair, makeup, freckles, wrinkles, eye color, hair color, hair style, and/or jewelry). Therefore, the remote computer system can input values from a facial landmark container and coefficients from a face model into the conditional generative adversarial network to generate a synthetic face image that depicts a face—(uniquely) represented by coefficients in the face model—exhibiting a facial expression represented by the facial landmark container”. Note that the unique features like face shape, hair styles, etc. are mapped to the predefined color or patterns).
Claims 2 and 10 are rejected under 35 U.S.C. 103 as being unpatentable over Jung, etc. (US 8203609 B2) in view of Astarabadi, etc. (US 20200358983 A1), further in view of Lee, etc. (US 20140147023 A1), Zavesky (US 20130141530 A1), Whitehill, etc. (US 20150324633 A1), and Balzer, etc. (US 20190377901 A1).
Regarding claim 2, Jung, Astarabadi, Lee, Zavesky, and Whitehill teach all the features with respect to claim 1 as outlined above. However, Jung, modified by Astarabadi, Lee, Zavesky, and Whitehill, fails to explicitly disclose that the personal data processing system according to claim 1, wherein the descriptor further comprises metadata including non-image data captured when capturing the image data and associated with the image data.
However, Balzer teaches that the personal data processing system according to claim 1, wherein the descriptor further comprises metadata including non-image data captured when capturing the image data and associated with the image data (See Balzer: Fig. 1, and [0038], "In another example, the PII pixels are replaced with the metadata directly inside the image. This is performed by including a known marker in the file that is unlikely to occur in real images such as five pixels of specific colors in a specific combination. After this, the metadata for identifying the PII is placed within the pixels, after performing a direct conversion of the metadata text to image data. A known indicator, perhaps the same as the first, can be used to identify the end of the region. This solution has an advantage of not requiring external metadata (and therefore reducing storage requirements). However, a PII region may not be sufficiently large for storing the metadata. In this case, the PII region could be expanded. A disadvantage of expansion is that non-PII regions are obscured. Another difficulty is that PII may be spread over multiple lines of an image so that each line would include an essentially independent region (this occurs because image file formats tend to store data line-by-line.) The challenge of multiple lines may be addressed by including a reference to the first line within the subsequent lines, to reduce the data processing required. For this embodiment, the missing data is obtained from the server 104").
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention was effectively filed to modify Jung to have the personal data processing system according to claim 1, wherein the descriptor further comprises metadata including non-image data captured when capturing the image data and associated with the image data as taught by Balzer in order to ensure privacy for the users (See Balzer: Fig.1, and [0021], "Controlling access to subsequent users and recipients of documents can protect personally identifiable information of individuals and ensures privacy for the users. As used herein, personally identifiable information (PII) includes anything that particularly identifies a particular entity (for example, a user, organization or content, etc.)"). Jung teaches a method and system that may capture an image with many subjects on the scene, select the anonymization policies, apply the anonymization policies to the captured images to generate anonymized images with identification information such as human faces blacked out or obscured, and output or display the anonymized image to the user or viewer; while Balzer teaches a system and method that may protect PII (personal identifiable information) by obfuscating the PII with metadata inserted into the image files. Therefore, it is obvious to one of ordinary skill in the art to modify Jung by Balzer to have metadata added in the facial image files and protect the PII. The motivation to modify Jung by Balzer is "Use of known technique to improve similar devices (methods, or products) in the same way".
Regarding claim 10, Jung, Astarabadi, Lee, Zavesky, and Whitehill teach all the features with respect to claim 9 as outlined above. Further, Balzer teaches that the method according to claim 9, comprising:
capturing metadata including non-image data captured when capturing the image data and associated with the image data (See Balzer: Fig. 1, and [0038], "In another example, the PII pixels are replaced with the metadata directly inside the image. This is performed by including a known marker in the file that is unlikely to occur in real images such as five pixels of specific colors in a specific combination. After this, the metadata for identifying the PII is placed within the pixels, after performing a direct conversion of the metadata text to image data. A known indicator, perhaps the same as the first, can be used to identify the end of the region. This solution has an advantage of not requiring external metadata (and therefore reducing storage requirements). However, a PII region may not be sufficiently large for storing the metadata. In this case, the PII region could be expanded. A disadvantage of expansion is that non-PII regions are obscured. Another difficulty is that PII may be spread over multiple lines of an image so that each line would include an essentially independent region (this occurs because image file formats tend to store data line-by-line.) The challenge of multiple lines may be addressed by including a reference to the first line within the subsequent lines, to reduce the data processing required. For this embodiment, the missing data is obtained from the server 104").
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to GORDON G LIU whose telephone number is (571)270-0382. The examiner can normally be reached Monday - Friday 8:00-5:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Devona E Faulk can be reached at 571-272-7515. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/GORDON G LIU/Primary Examiner, Art Unit 2618
Prosecution Insights