DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Receipt is acknowledged of certified copies of papers required by 37 CFR 1.55.
Information Disclosure Statement
The information disclosure statement(s) (IDS) submitted on 12/26/2024 & 12/05/2025 is/are in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Specification
The title of the invention is not descriptive. A new title is required that is clearly indicative of the invention to which the claims are directed to. The examiner recommends incorporation of the primary inventive concept or field of endeavor into the title of the invention. This application is related to anonymization of faces captured in images which should be reflected in the title.
Applicant is reminded of the proper language and format for an abstract of the disclosure.
The abstract should be in narrative form and generally limited to a single paragraph on a separate sheet within the range of 50 to 150 words in length. The abstract should describe the disclosure sufficiently to assist readers in deciding whether there is a need for consulting the full patent text for details.
The language should be clear and concise and should not repeat information given in the title. It should avoid using phrases which can be implied, such as, “The disclosure concerns,” “The disclosure defined by this invention,” “The disclosure describes,” etc. In addition, the form and legal phraseology often used in patent claims, such as “means” and “said,” should be avoided (emphasis added).
Claim Objections
Claim 12 is objected to because of the following informalities:
Claim 12 recites “An image processing method comprising causing a computer to:” The examiner believes the applicant intended to recite “An image processing method causing a computer to:”.
Appropriate correction is required.
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action.
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because the claim limitation(s) uses a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier. Such claim limitation(s) is/are:
Claim 1 first recites the limitation “an image conversion unit that performs an anonymization process…”
Claim 1 first recites the limitation “an image determination unit that determines whether…”
Because this/these claim limitation(s) is/are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, it/they is/are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof.
“an image conversion unit”
Structure: image conversion unit 140 is included in image processing device 100, which is realized by a hardware processor such as a CPU [¶0029; Fig. 2]
Algorithm: the image conversion unit 140 replaces the face of a person depicted in each image with the face of another person without changing the direction information of the person, or applies a mosaicking process on the face depending on the size and distance of the face. The image conversion unit 140 acquires a captured image included in the captured image data 172 that has been processed by the image processing unit 130 (S100). It then selects one of the faces depicted in the acquired captured image (S102). When the size of a face is greater than or equal to a first threshold Th1 (S104), then it replaces the face with the face of another person (S106). When the size is less than a first threshold Th1, then it determines to perform mosaic processing on the face (S110). Both are considered "anonymization processes". The image conversion unit 140 additionally determines that, for each face depicted, whether the distance of the face is less than or equal to a second threshold Th2 (S108), and if so, to replace the face with the face of another person (S106), and when the distance is greater than the second threshold Th2 (S110), to perform mosaic processing. This process is repeated for every face in the image (S112) and then stores image data in the storage unit (S114). The image conversion unit may select faces to be utilized as training data by performing mosaic processing on faces for which direction information has failed to be acquired among faces depicted in each image. When the face direction FD or gaze direction ED do not substantially match between the captured and converted image, the image conversion unit 140 performs the conversion process again on the face of captured image and excludes the face from being utilized as learning data. [¶0037-42, 45, 47, 0057-61; Fig. 10]
“an image determination unit”
Structure: image determination unit 150 is included in image processing device 100, which is realized by a hardware processor such as a CPU [¶0029; Fig. 2]
Algorithm: image determination unit 150 determines the continuity of the time-series in-vehicle images and out-vehicle images. The image determination unit 150 first acquires time-series converted images (S200) and then selects faces of a person tracked as the same person before conversion (S202). It then extracts feature points representing a face from the face of a person depicted in the converted image, where feature points for the right eye (REP), left eye (LEP), nose (NP), right corner of the mouth (RMP), left corner of the mouth (LMP), and ears (EP) of a face from the face of a person detected in the converted image, and extracts these feature points for the face of each person tracked through the time-series of converted image and collates these feature points. More specifically, the image determination unit 150 extracts feature points of a person in the converted image at a time point t and t+1 and then performs collation by determining whether these two sets of extracted feature points substantially match via translation and rotation (S204). If the extracted feature points match from the collation process, then the image determination unit 150 determines that the tracked face is still the same face after conversion. If the features do not substantially match, the image determination unit 150 determines that the faces tracked as the same person are not the same person after conversion and then causes the image conversion unit 140 to reconvert faces again (S208). In the case that the faces do not match, then it may restrict the application of a predetermined process to the time-series converted images and exclude them from being utilized as training data. When the faces do match, the image determination unit 150 then determines if the image is an in-vehicle image (S206) whether the gaze and face direction of these faces match those of the captured images prior to conversion (S210). If the images are out-vehicle images, then the image determination unit 150 determines whether the face direction of the faces matches that of the images prior to conversion (S12). If there is no match for out-vehicle images, then the process proceeds to step S208. When there is a match determined in either S10 or S212, the image determination unit 150 determines that the faces have been converted normally and then determines whether the process has been performed across all depicted faces in the converted images (S214). The image determination unit 150 then stores the converted image data 174 which have face continuity and consistency confirmed in the storage unit 170 for annotation 176 (S216). [¶0042-53; 62-67; Fig. 11]
If applicant does not intend to have this/these limitation(s) interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation(s) to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation(s) recite(s) sufficient structure to perform the claimed function so as to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claim 7 rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Regarding claim 7, it recites “the image determination unit determines whether the predetermined requirement is satisfied for a face of a person who is facing forward in a traveling direction of a vehicle…”. The language in the specification implies that this means the face of a person facing toward the front of the vehicle (i.e., opposing directions) [¶0048], however the claim language implies facing in the same direction as the vehicle. This claim language is unclear. For the purposes of compact prosecution, the examiner will interpret this claim in accordance with how it is described in the specification.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 12 & 13 is/are rejected under 35 U.S.C. 103 as being unpatentable over Simhadri et al (US 2021/0209734 A1), hereinafter referred to as “Simhadri”, in view of Sako et al (US 2014/0112534 A1), hereinafter referred to as “Sako”.
Regarding claim 12, Simhadri teach An image processing method (the method outlined in Fig. 2 for performing an automated facial anonymization procedure [¶0052]) comprising causing a computer (automated face anonymization system 102 comprises a processor 108 and memory 110 [¶0035; Fig. 1]) to:
perform an anonymization process on a plurality of input images captured in a time series (faces extracted from frames of a video stream from step 202 are anonymized in act 212 [¶0052-53, 58; Fig. 2]);
determine whether the plurality of input images on which the anonymization process has been performed satisfy a predetermined requirement (in act 1504, a structural similarity index (SSIM) is computed between faces of current frame and a previously anonymized frame which is then compared to a threshold in act 1506 to determine if the SSIM satisfies a predetermined requirement for a certain degree of similarity when the SSIM ≥ threshold [¶0123-125; Fig. 15]); and
perform a predetermined process on the plurality of input images on which the anonymization process has been performed in a case where it is determined that the plurality of input images on which the anonymization process has been performed satisfy the predetermined requirement (in act 1506, if the SSIM is greater than or equal to a predetermined threshold (i.e., satisfying the predetermined requirement), then the process proceeds to act 1510 (i.e., a predetermined process) to update trackers to localize faces without running a particular detection subroutine and subsequently runs both act 1312 of anonymizing local faces and act 1512 for continuing or reinitializing tracking [¶0126-129; Fig. 15]),
and
the predetermined requirement includes that a face of a person tracked as the same person in the plurality of input images is the same face in each of the plurality of input images on which the anonymization process has been performed (when the SSIM is greater than a predetermined threshold in act 1506 [¶0125-0126; Fig. 15] – the examiner notes that having a similarity greater than a given threshold implies faces being sufficiently similar (i.e., the same person)).
Simhadri discloses an anonymization process of pixelating or otherwise blurring a face of a person but fails to recite replacing a person's face with another face as a form of anonymization. Sako, however, is analogous art pertinent to the field of endeavor of the present application and describe a process for anonymizing a person's face by replacing it with a natural face based off attributes of the original face before replacement. More specifically, Sako teach wherein the anonymization process includes a process of changing a face of a person depicted in the plurality of input images to a face of another person, (Sako: when it is determined that a face image is to be replaced with a natural face image (S109, Yes), the face image replacement unit 18 replaces the face image region detected by the face region detection unit 14 based on human attributes determined by the determination unit 16 (S118) [¶0054-57; Fig. 4]).
Sako further explain their motivation behind performing this face replacement process, as replacing the face of a person with a natural face according to human attributes (age, gender, etc.) avoids any unnatural or uncanny images while still protecting the privacy of persons captured in images or videos [¶0033]. It would have been obvious to one of ordinary skill in the art to implement the face replacement process taught by Sako to provide contextual continuity (i.e., preserving aspects of the original identity of the person being anonymized) while still protecting their identity.
With respect to claim 13, Simhadri teach A non-transitory computer-readable storage medium (memory 110 [¶0035; Fig. 1]) having stored thereon a program causing a computer (automated face anonymization system 102 comprises a processor 108 and memory 110 [¶0035; Fig. 1]) to:
perform an anonymization process on a plurality of input images captured in a time series (faces extracted from frames of a video stream from step 202 are anonymized in act 212 [¶0052-53, 58; Fig. 2]);
determine whether the plurality of input images on which the anonymization process has been performed satisfy a predetermined requirement (in act 1504, a structural similarity index (SSIM) is computed between faces of current frame and a previously anonymized frame which is then compared to a threshold in act 1506 to determine if the SSIM satisfies a predetermined requirement for a certain degree of similarity when the SSIM ≥ threshold [¶0123-125; Fig. 15]); and
perform a predetermined process on the plurality of input images on which the anonymization process has been performed in a case where it is determined that the plurality of input images on which the anonymization process has been performed satisfy the predetermined requirement (in act 1506, if the SSIM is greater than or equal to a predetermined threshold (i.e., satisfying the predetermined requirement), then the process proceeds to act 1510 (i.e., a predetermined process) to update trackers to localize faces without running a particular detection subroutine and subsequently runs both act 1312 of anonymizing local faces and act 1512 for continuing or reinitializing tracking [¶0126-129; Fig. 15]),
and
the predetermined requirement includes that a face of a person tracked as the same person in the plurality of input images is the same face in each of the plurality of input images on which the anonymization process has been performed (when the SSIM is greater than a predetermined threshold in act 1506 [¶0125-0126; Fig. 15] – the examiner notes that having a similarity greater than a given threshold implies faces being sufficiently similar (i.e., the same person)).
Simhadri discloses an anonymization process of pixelating or otherwise blurring a face of a person but fails to recite replacing a person's face with another face as a form of anonymization. Sako, however, is analogous art pertinent to the field of endeavor of the present application and describe a process for anonymizing a person's face by replacing it with a natural face based off attributes of the original face before replacement. More specifically, Sako teach wherein the anonymization process includes a process of changing a face of a person depicted in the plurality of input images to a face of another person, (Sako: when it is determined that a face image is to be replaced with a natural face image (S109, Yes), the face image replacement unit 18 replaces the face image region detected by the face region detection unit 14 based on human attributes determined by the determination unit 16 (S118) [¶0054-57; Fig. 4]).
Sako further explain their motivation behind performing this face replacement process, as replacing the face of a person with a natural face according to human attributes (age, gender, etc.) avoids any unnatural or uncanny images while still protecting the privacy of persons captured in images or videos [¶0033]. It would have been obvious to one of ordinary skill in the art to implement the face replacement process taught by Sako to provide contextual continuity (i.e., preserving aspects of the original identity of the person being anonymized) while still protecting their identity.
Allowable Subject Matter
Claims 1-11 are allowed.
Regarding claim 1, the primary reason for allowance is that the prior art fails to teach or reasonably suggest “an image conversion unit” and “an image determination unit” as interpreted under 35 U.S.C. § 112(f).
The closest cited prior art, Simhadri, teach a process of anonymizing a face image by applying a pixelation process (analogous to mosaicking) on subjects captured in a video. This anonymization process tracks facial key points to estimate a facial pose and uses a structural similarity metric to ensure sameness between frames of a face tracked across the time series. Simhadri, however, fails to teach a process of anonymization by replacing a face with another face, and is silent utilizing size and distance thresholds for determining whether to perform face replacement or mosaicking for anonymization, nor does it specify a process for checking if a converted face image is taken inside a vehicle.
The next closest cited prior art, Sako, teach a process of anonymization by replacing a face of a user with another face based on human attributes of the original face. This process preserves certain aspects of the original face (age, gender, orientation, expression) after face replacement is applied. Sako, however, still fails to recite a size and distance threshold for deciding which anonymization process to use, nor does it specify checking to see if a converted face image is taken inside a vehicle.
The closest prior art made of record, Lim; Sunok (US 2020/0042775 A1) describe a de-identification process wherein a size of a face is ascertained for performing a facial anonymization process, however this threshold comparison is used to decide to not run the anonymization process when the size of a face is less than a threshold, which teaches away from the disclosed image conversion unit of the present application.
No prior art discloses, alone or in reasonable combination teach the algorithms associated with the image conversion unit and the image determination unit as detailed in Figs. 10 & 11 of the disclosed application, respectively.
Claims 2-10 are allowable by virtue of their dependence on a claim previously indicated as allowable.
Turning to claim 11, an identical analysis to that of claim 1 is applied as it recites substantially identical subject matter.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure:
Hsu et al (US 2021/0392116 A1) describe a video camera for anonymizing a video stream by detecting and pixelating a subject’s face.
Hong et al (KR 102411943 B1) teach a facial de-identification system that captures feature points from the original face to better inform the masking with a new face
Muraki et al (“Anonymizing Face Images by Using Similarity-Based Metric”, 2013, IEEE) describe a method for anonymizing a face using comparably similar faces to that of the original user to obscure a faces original identity.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Michael M. Sofroniou whose telephone number is (571)272-0287. The examiner can normally be reached M-F: 8:30 AM - 5:00 PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, John M. Villecco can be reached at (571) 272-7319. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MICHAEL M SOFRONIOU/Examiner, Art Unit 2661
/JOHN VILLECCO/Supervisory Patent Examiner, Art Unit 2661
Prosecution Insights