DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 3/15/2024 in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claim 13 is rejected under 35 U.S.C. 101 because it includes two subsets: a non-volatile computer readable storage medium and a non-transitory computer readable medium. In accordance with MPEP 2103.03, the USPTO generally accepts “non-transitory computer readable medium” as passing the first part of the Alice analysis. By claiming “non-volatile…” in the disjunctive, the claim apparently is intended to mean something beyond non-transitory. Because this the claim understood to include at least some transitory forms, it includes non-statutory subject matter. Accordingly, it falls outside the four categories of statutory subject matter and is rejected on that basis.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claim(s) 1, 4, 5, 7-9, 11, 13, 14, 17-19, and 21 is/are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Fu et al (US20200015575).
Regarding claim 1, Fu discloses a facial image processing method, comprising:
acquiring a lip mask in a to-be-processed facial image, wherein the lip mask is a mask of a lip area (para. [0130], the final lip segmentation result; para. [0141], Canny edge detection and contour smoothness is used to get the final lip region mask);
acquiring an initial image of lipstick trial makeup based on luminance of the to-be-processed facial image subjected to luminance and color separation (para. [0164], For lip color matching, the lip region, ILip, and lip color image, Icolor are first converted to CIELAB L*a*b* space such that luminance (L-channel, IL) is isolated from all color channels (a* and b* channels, Ia & Ib)), and a color of a target lipstick sample subjected to luminance and color separation (para. [0164], The pre-process module 50 modifies the input lip region LI′ by changing the lip color 60 to the desired lipstick color 62 and luminance enhancement through the dynamic texture generation method described herein); and
performing image fusion on the initial image of lipstick trial makeup and the to-be-processed facial image based on the lip mask to acquire a lipstick trial makeup image (para. [0024], combining the output effect with the first image to provide a resultant image having the lip color and the at least one output effect applied to the lip; para. [0157], that image having the color may then be combined with the original facial image of the user and any images of the user generated with an end effect to provide a resultant image having the pre-defined lip color and the output effect(s) desired; para. [0168]).
Regarding claim 4, Fu discloses a method further comprising:
adjusting luminance of the to-be-processed facial image subjected to luminance and color separation (para. [0042], modifying the input lip region to have a desired lipstick color and luminance enhancement using the luminance channel of L*a*b* color space for the desired lipstick color and a weighted coverage factor), wherein said adjusting the luminance of the to-be-processed facial image subjected to luminance and color separation comprises:
acquiring initial luminance of the to-be-processed facial image (para. [0032], calculating an average brightness of the input image); and
adjusting the initial luminance using a lipstick trial makeup target luminance coefficient (para. [0042], weighted coverage factor) to acquire the luminance of the to-be-processed facial image subjected to luminance and color separation (para. [0152], In step 2025a, illuminance/brightness parameters over a range can be used to provide varying degrees of transparency, smoothness and/or shine for various gloss or shine effects as in step 2025b; para. [0164]).
Regarding claim 5, Fu discloses a method wherein the lipstick trial makeup target luminance coefficient is calculated based on preset luminance of the target lipstick sample and luminance of the lip area of the to-be-processed facial image (para. [0032], an average brightness of the input image; para. [0042], a desired lipstick color and luminance enhancement using the luminance channel of L*a*b* color space).
Regarding claim 7, Fu discloses a method further comprising:
performing image fusion on the initial image of lipstick trial makeup and the to-be-processed facial image based on the lip mask to acquire an intermediate image (para. [0024], combining the output effect with the first image to provide a resultant image having the lip color and the at least one output effect applied to the lip; para. [0028]; para. [0157], that image having the color may then be combined with the original facial image of the user and any images of the user generated with an end effect to provide a resultant image having the pre-defined lip color and the output effect(s) desired);
acquiring an intensity coefficient of lipstick glossy texture effect corresponding to lipstick texture of the target lipstick sample (para. [0148], First the method generates a glossy effect. It detects light distribution of the lip and applies smoothness and transparency; para. [0152], In step 2025a, illuminance/brightness parameters over a range can be used to provide varying degrees of transparency, smoothness and/or shine for various gloss or shine effects as in step 2025b; para. [0158]);
calculating a luminance adjustment amount of the lip area (output of Appendix C) based on the intensity coefficient of lipstick glossy texture effect (MAX_EFFECT and RANGE in Appendix C), the lip mask (mask map m of exact lip region in Appendix C), luminance of the to-be-processed facial image and luminance of the lip area (o (x) of the inputs in Appendix C); and
adjusting luminance of the intermediate image based on the luminance adjustment amount to acquire the lipstick trial makeup image (para. [0149], The pseudo code of the detailed algorithm to add transparency and shine effect is provided in Appendix C).
Regarding claim 8, Fu discloses a method wherein said calculating the luminance adjustment amount of the lip area based on the intensity coefficient of lipstick glossy texture effect, the lip mask, luminance of the to-be-processed facial image and luminance of the lip area comprises:
determining a maximum value among the luminance of the to-be-processed facial image and the luminance of the lip area (for x=0 to size of (i) in Appendix C); and
calculating the luminance adjustment amount of the lip area based on the maximum value, the intensity coefficient of lipstick glossy texture effect and the lip mask (output of Appendix C).
Regarding claim 9, Fu discloses a method wherein said acquiring the lip mask in the to-be-processed facial image comprises:
performing face key point alignment on the to-be-processed facial image, and determining the lip area based on lip key points among the face key points (fig. 21; 320 in fig. 26a; para. [0114], Such landmarks are common to all faces and so are detected and evaluated using precise localization of their fiducial points (e.g. nose tip, mouth and eye corners) in color images of face foregrounds; para. [0137]);
retaining the lip area (fig. 24), triangulating areas in the to-be-processed facial image except the lip area (figs. 21 and 22, it would be possible to include/exclude any landmark points when determining desired areas), and acquiring a binary image by converting (para. [0133]-[0134]);
performing edge smoothing on the binary image by using luminance channel information of the to-be-processed facial image as a guide map (390 in fig. 26a; para. [0040], The various embodiments of the method may further comprise using an edge detection and contouring to smooth the overall probability map; para. [0141]); and
determining the lip mask based on the binary image subjected to the edge smoothing (395 in fig. 26a).
Regarding claim 11, Fu discloses a method wherein prior to said acquiring the initial image of lipstick trial makeup based on the luminance of the to-be-processed facial image subjected to luminance and color separation, and the color of the target lipstick sample subjected to luminance and color separation, the method further comprises:
in response to a color space type of the to-be-processed facial image being different from a color space type of the target lipstick sample, converting the color space type of the to-be-processed facial image and the color space type of the target lipstick sample to a same color space type (para. [0024], [0157], [0164]).
Regarding claims 13 and 14, the claims recite similar subject matter as claim 1 and are rejected for the same reasons as stated above.
Regarding claim 17, the claims recite similar subject matter as claim 4 and is rejected for the same reasons as stated above.
Regarding claim 18, the claims recite similar subject matter as claim 7 and is rejected for the same reasons as stated above.
Regarding claim 19, the claims recite similar subject matter as claim 9 and is rejected for the same reasons as stated above.
Regarding claim 21, the claims recite similar subject matter as claim 11 and is rejected for the same reasons as stated above.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 2 and 15 is/are rejected under 35 U.S.C. 103 as being unpatentable over Fu et al (US20200015575) in view of Nguyen et al (US20150145882).
Regarding claim 2, Fu teaches a method wherein said performing image fusion on the initial image of lipstick trial makeup and the to-be-processed facial image based on the lip mask to acquire the lipstick trial makeup image comprises:
acquiring a first fusion weight corresponding to the initial image of lipstick trial makeup associated with the lip mask (para. [0042], weighted coverage factor; para. [0138], mixture component weights; para. [0165]).
Fu fails to teach a second fusion weight corresponding to the to-be-processed facial image, wherein the first fusion weight is associated with the lip mask; and
performing image fusion on the initial image of lipstick trial makeup and the to-be-processed facial image based on the first fusion weight and the second fusion weight to acquire the lipstick trial makeup image.
However Nguyen teaches acquiring a first fusion weight corresponding to the initial image of lipstick trial makeup (para. [0161], Pixels having such a non-zero weight compose a lip shape; para. [0172]) and a second fusion weight corresponding to the to-be-processed facial image (para. [0276], weighting and summing the pixel component of the specific type among the channel images included in the coating material application layer and the pixel component of the specific type among the channel images included in the source image, in which the weighting is weighting using a compositing parameter corresponding to the specific type), wherein the first fusion weight is associated with the lip mask (figs. 24B and 26B); and
performing image fusion on the initial image of lipstick trial makeup and the to-be-processed facial image based on the first fusion weight and the second fusion weight to acquire the lipstick trial makeup image (para. [0072], specifying a region using an outline, generates a lipstick application layer indicating a lip region in which lipstick is applied using the selected cosmetic item, and composites the lipstick application layer with a human image; para. [0087], [0148]).
Therefore taking the combined teachings of Fu and Nguyen as a whole, it would have been obvious to one of ordinary skill in the art at the time the invention was filed to incorporate the steps of Nguyen into the method of Fu. The motivation to combine Nguyen and Fu would be to more realistically reproduce a state in which make-up having various optical properties is applied to a subject illuminated by ambient light (para. [0009] of Nguyen).
Regarding claim 15, the claims recite similar subject matter as claim 2 and is rejected for the same reasons as stated above.
Claim(s) 10 and 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Fu et al (US20200015575).
Regarding claim 10, Fu fails to explicitly teach a method wherein prior to said performing image fusion on the initial image of lipstick trial makeup and the to-be-processed facial image based on the lip mask to acquire a lipstick trial makeup image, the method further comprises:
acquiring color information of a lip concealer sample, and acquiring a base makeup image based on the luminance of the to-be-processed facial image subjected to luminance and color separation, and a color of the lip concealer sample subjected to luminance and color separation; and
performing image fusion on the base makeup image and the to-be-processed facial image based on the lip mask to acquire the adjusted to-be-processed facial image.
However Fu does teach acquiring color information of a desired lipstick (para. [0031], displaying a first selected color for a first type of makeup), and acquiring a base makeup image based on the luminance of the to-be-processed facial image subjected to luminance and color separation (para. [0164], For lip color matching, the lip region, ILip, and lip color image, Icolor are first converted to CIELAB L*a*b* space such that luminance (L-channel, IL) is isolated from all color channels (a* and b* channels, Ia & Ib)), and a color of the lipstick subjected to luminance and color separation (para. [0164], The pre-process module 50 modifies the input lip region LI′ by changing the lip color 60 to the desired lipstick color 62 and luminance enhancement through the dynamic texture generation method described herein); and
performing image fusion on the base makeup image and the to-be-processed facial image based on the lip mask to acquire the adjusted to-be-processed facial image (para. [0024], combining the output effect with the first image to provide a resultant image having the lip color and the at least one output effect applied to the lip; para. [0157], that image having the color may then be combined with the original facial image of the user and any images of the user generated with an end effect to provide a resultant image having the pre-defined lip color and the output effect(s) desired; para. [0168]).
It would be obvious to apply the steps using a lip concealer sample in place of lipstick to provide more helpful and streamlined ways for users to interact with a virtual try-on system, and to show more realistic effects than can be achieved by a simple color overlay (para. [0011]).
Regarding claim 20, the claims recite similar subject matter as claim 10 and is rejected for the same reasons as stated above.
Allowable Subject Matter
Claims 3, 6, 16 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Related Art
Freeman et al (US20190122404) – see figs. 5E, 7D, 19, 24, 30A-32C; para. [0169]-[0171], [0201]
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to LEON VIET Q NGUYEN whose telephone number is (571)270-1185. The examiner can normally be reached Mon-Fri 11AM-7PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Gregory Morse can be reached at 571-272-3838. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/LEON VIET Q NGUYEN/ Primary Examiner, Art Unit 2663