DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Double Patenting
The nonstatutory double patenting rejection is based on a judicially created doctrine grounded in public policy (a policy reflected in the statute) so as to prevent the unjustified or improper timewise extension of the “right to exclude” granted by a patent and to prevent possible harassment by multiple assignees. A nonstatutory double patenting rejection is appropriate where the conflicting claims are not identical, but at least one examined application claim is not patentably distinct from the reference claim(s) because the examined application claim is either anticipated by, or would have been obvious over, the reference claim(s). See, e.g., In re Berg, 140 F.3d 1428, 46 USPQ2d 1226 (Fed. Cir. 1998); In re Goodman, 11 F.3d 1046, 29 USPQ2d 2010 (Fed. Cir. 1993); In re Longi, 759 F.2d 887, 225 USPQ 645 (Fed. Cir. 1985); In re Van Ornum, 686 F.2d 937, 214 USPQ 761 (CCPA 1982); In re Vogel, 422 F.2d 438, 164 USPQ 619 (CCPA 1970); In re Thorington, 418 F.2d 528, 163 USPQ 644 (CCPA 1969).
A timely filed terminal disclaimer in compliance with 37 CFR 1.321(c) or 1.321(d) may be used to overcome an actual or provisional rejection based on nonstatutory double patenting provided the reference application or patent either is shown to be commonly owned with the examined application, or claims an invention made as a result of activities undertaken within the scope of a joint research agreement. See MPEP § 717.02 for applications subject to examination under the first inventor to file provisions of the AIA as explained in MPEP § 2159. See MPEP § 2146 et seq. for applications not subject to examination under the first inventor to file provisions of the AIA . A terminal disclaimer must be signed in compliance with 37 CFR 1.321(b).
The filing of a terminal disclaimer by itself is not a complete reply to a nonstatutory double patenting (NSDP) rejection. A complete reply requires that the terminal disclaimer be accompanied by a reply requesting reconsideration of the prior Office action. Even where the NSDP rejection is provisional the reply must be complete. See MPEP § 804, subsection I.B.1. For a reply to a non-final Office action, see 37 CFR 1.111(a). For a reply to final Office action, see 37 CFR 1.113(c). A request for reconsideration while not provided for in 37 CFR 1.113(c) may be filed after final for consideration. See MPEP §§ 706.07(e) and 714.13.
The USPTO Internet website contains terminal disclaimer forms which may be used. Please visit www.uspto.gov/patent/patents-forms. The actual filing date of the application in which the form is filed determines what form (e.g., PTO/SB/25, PTO/SB/26, PTO/AIA /25, or PTO/AIA /26) should be used. A web-based eTerminal Disclaimer may be filled out completely online using web-screens. An eTerminal Disclaimer that meets all requirements is auto-processed and approved immediately upon submission. For more information about eTerminal Disclaimers, refer to www.uspto.gov/patents/apply/applying-online/eterminal-disclaimer.
Claims 1-20 are rejected on the ground of nonstatutory double patenting as being unpatentable over claim1-4, 6-20 of U.S. Patent No.12299775. Although the claims at issue are not identical, they are not patentably distinct from each other because they are obvious of variances of one another. Below is the detailed correspondence between the instant claims and the claims of U.S. Patent No.12299775.
Instant Application
1
2
3
4
5
6
7
8
9
10
U.S. Patent No.12299775
1
1
3
4
6
6
7
8
9
10
Instant Application
11
12
13
14
15
16
17
18
19
20
U.S. Patent No.12299775
11
12
13
14
15
16
17
18
19
20
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-4, 8-10, 15-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Kuo (US 2020/0242833 A1) in view of Su et al. (US 2020/0143230 A1).
Regarding claim 1, Kuo teaches:
A method comprising:
accessing, by a user system, an image that depicts a real-world person; ([0036], “At block 310, the computing device 102 obtains a digital image depicting an individual.”)
retrieving, by the user system, an augmented reality (AR) fashion item; ([0038], “At block 330, the computing device 102 obtains selection of a makeup effect from a user.”)
generating a modified image by combining the AR fashion item with the image that depicts the real-world person; ([0039], “At block 370, the computing device 102 adjusts visual characteristics of the makeup effect based on the surface properties of the makeup effect and the lighting conditions of the region of interest. For some embodiments, the computing device 102 adjusts the visual characteristics by adjusting the color of the makeup effect based on the angle of lighting incident on the individual depicted in the digital image, the lighting intensity, and/or the color of the lighting incident on the individual depicted in the digital image. For some embodiments, the computing device 102 adjusts the color of the makeup effect on a pixel-by-pixel basis in the region of interest. At block 380, the computing device 102 performs virtual application of the adjusted makeup effect to the region of interest in the digital image.”) and
However, Kuo does not, but Su teaches:
applying a fitting machine learning model to the modified image to adjust a fit of the AR fashion item on the real-world person depicted in the image. ([0051]-[0052] teaches training a neural network to apply lighting effects to an image based on the acquired lighting properties to make it looks like real world scenario: “Exemplarily, before executing the image lighting method provided by the embodiments of the present disclosure, the embodiments of the present disclosure further include using a label image to train the convolutional neural network, to obtain the trained convolutional neural network. The label image includes an original portrait image and a lighted portrait image. The convolutional neural network obtained by means of training by means of the label image may implement lighting for any portrait image. In an embodiment, the lighting type for the portrait image includes rimming light, photographic studio light, stage light, monochromatic light, two-color light, or polychromatic light. FIG. 2 is a schematic diagram of an image lighting effect provided by embodiments of the present disclosure. As compared with the portrait image that is not lighted, the portrait image after the rimming light luminous effect processing, five sense organs of the face can be more stereoscopic. As compared with the portrait image that is not lighted, the portrait image after the photographic studio light luminous effect processing, light rays for the facial part are fuller and more uniform. Corresponding to different lighting types, different label images are included; different label images are used to train the convolutional neural network, to obtain the convolutional neural networks corresponding to different lighting types. When requiring to perform lighting in a dimming light type on the initial image, the convolutional neural network corresponding to the dimming light type is correspondingly adopted. When requiring to perform lighting in a dimming light type on the initial image, the convolutional neural network corresponding to the dimming light type is correspondingly adopted. In actual applications, the lighting types for the portrait image may also include other types.”)
Kuo teaches applying lighting effects to a person face image with added makeup feature based on acquired lighting properties. Su teaches using a trained neural network to applying lighting effects to features on person face image, which will produce more natural lighting effect.(Su, [0091])
It would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to have combined the teachings of Kuo with the specific teachings of Su to use a neural network to apply the acquired lighting properties to a person’s face image with added makeup features. The benefit would be to produce more natural lighting effect.(Su, [0091])
Regarding claim 2, Kuo in view of Su teaches:
The method of claim 1, further comprising: estimating a lighting adjustment for the AR fashion item by applying a machine learning model to the image and the AR fashion item using lighting properties of the real-world person depicted in the image, the modified image being generated using the estimated lighting adjustment. (Su, [0091], “Regarding the image lighting method provided by this embodiment, by using the deep learnt convolutional neural network technology, using the convolutional neural network to perform feature extraction on the initial image, obtaining an adaptive bilateral grid matrix of the initial image based on the maximum pooling result map and the minimum pooling result map of the initial image, and obtaining a target image of the initial image according to the bilateral grid matrix, a lighting effect of the image is more natural.” FIG. 5 and FIG. 4, [0051]-[0052] teaches training a neural network to apply lighting effects to an image based on the acquired lighting properties. The combination rationale of claim 1 is applied here. )
Regarding claim 3, Kuo in view of Su teaches:
The method of claim 2, further comprising: adjusting one or more pixel values of the AR fashion item based on parameters output by the machine learning model corresponding to the estimated lighting adjustment. (Su, [0145]-[0147], “In an embodiment, as shown in FIG. 8, the image lighting apparatus further includes: a shrinkage module 607, configured to shrink the initial image. In an embodiment, as shown in FIG. 8, the image lighting apparatus further includes: an amplification module 608, configured to amplify the target image, to obtain a target image having the same size as the initial image.” Kuo teaches putting (overlaying) makeup on a face image and adjusting the face image along with the makeup based on the lighting properties. Su teaches adjusting a face image with features based on lighting properties. The generated image is amplified to its initial image size. It would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to have combined the teachings of Kuo with the specific teachings of Su to use a neural network to apply the acquired lighting properties to a person’s face image with added makeup features. The benefit would be to produce more natural lighting effect.(Su, [0091]).)
Regarding claim 4, Kuo in view of Su teaches:
The method of claim 3, further comprising: overlaying the AR fashion item with the adjusted one or more pixel values on a portion of the image that depicts the real-world person. (Su, [0145]-[0147], “In an embodiment, as shown in FIG. 8, the image lighting apparatus further includes: a shrinkage module 607, configured to shrink the initial image. In an embodiment, as shown in FIG. 8, the image lighting apparatus further includes: an amplification module 608, configured to amplify the target image, to obtain a target image having the same size as the initial image.” Kuo teaches putting (overlaying) makeup on a face image and adjusting the face image along with the makeup based on the lighting properties. Su teaches adjusting a face image with features based on lighting properties. The generated image is amplified to its initial image size. The combination rationale of claim 3 is applied here.)
Regarding claim 8, Kuo in view of Su teaches:
The method of claim 1, wherein the AR fashion item is retrieved in response to input that selects the AR fashion item from a list of AR fashion items. (Kuo, FIG. 5.)
Regarding claim 9, Kuo in view of Su teaches:
The method of claim 1, wherein an estimated lighting adjustment of the modified image comprises at least one of gamma correction, contrast enhancement, brightness modification, or color correction. (Kuo, [0025], “For some embodiments, the content analyzer 106 determines the lighting conditions by estimating such parameters as the angle of lighting incident on the individual depicted in the digital image, a lighting intensity, a color of the lighting incident on the individual depicted in the digital image, and so on. One or more of these parameters are later utilized for adjusting a visual characteristic (e.g., color, intensity) of a makeup effect prior to being applied to the facial region of the individual.”)
Regarding claim 10, Kuo in view of Su teaches:
The method of claim 1, wherein the image comprises a frame of a real-time video feed captured by a camera of the user system. (Kuo, [0024], “Alternatively, the digital image may be derived from a still image of a video” [0003], “when capturing a digital image of an individual, it can be difficult to achieve a realistic result when performing virtual application of makeup effects.”)
Regarding claim 15, Kuo in view of Su teaches:
A system comprising:at least one processor of a user system configured to perform operations (Kuo, [0005], “Another embodiment is a system that comprises a memory storing instructions and a processor coupled to the memory. The processor is configured by the instructions to obtain a digital image depicting an individual and determine lighting conditions of the content in the digital image.”) comprising: the rest of claim 15 recites similar limitations of claim 1, thus is rejected accordingly.
Regarding claim 16, Kuo in view of Su teaches:
The system of claim 15, the operations further comprising: blending the image, the AR fashion item, and a lighting adjustment using a blending model to generate a modified image that depicts the real-world person wearing the AR fashion item. (Kuo [0039], “At block 370, the computing device 102 adjusts visual characteristics of the makeup effect based on the surface properties of the makeup effect and the lighting conditions of the region of interest. For some embodiments, the computing device 102 adjusts the visual characteristics by adjusting the color of the makeup effect based on the angle of lighting incident on the individual depicted in the digital image, the lighting intensity, and/or the color of the lighting incident on the individual depicted in the digital image. For some embodiments, the computing device 102 adjusts the color of the makeup effect on a pixel-by-pixel basis in the region of interest. At block 380, the computing device 102 performs virtual application of the adjusted makeup effect to the region of interest in the digital image.” FIG. 3)
Regarding claim 19, Kuo in view of Su teaches:
The system of claim 15, the operations further comprising: applying a fitting machine learning model to the modified image to adjust a fit of the AR fashion item on the real-world person depicted in the image. (Su, [0091], “Regarding the image lighting method provided by this embodiment, by using the deep learnt convolutional neural network technology, using the convolutional neural network to perform feature extraction on the initial image, obtaining an adaptive bilateral grid matrix of the initial image based on the maximum pooling result map and the minimum pooling result map of the initial image, and obtaining a target image of the initial image according to the bilateral grid matrix, a lighting effect of the image is more natural.” FIG. 5 and FIG. 4, [0051]-[0052] teaches training a neural network to apply lighting effects to an image based on the acquired lighting properties. The combination rationale of claim 15 is applied here. )
Regarding claim 20, Kuo in view of Su teaches:
A non-transitory machine-readable storage medium that includes instructions that, when executed by one or more processors of a user system, cause the user system to perform operations (Kuo [0005], “Another embodiment is a system that comprises a memory storing instructions and a processor coupled to the memory. The processor is configured by the instructions to obtain a digital image depicting an individual and determine lighting conditions of the content in the digital image.”) comprising: the rest of claim 20 recites similar limitations of claim 1, thus is rejected accordingly.
Claim 17-18 recite similar limitations of claim 3-4 respectively, thus are rejected using the same rejection rationale respectively.
Allowable Subject Matter
Claims 5-7, 11-14 would be allowable if rewritten to overcome the rejection(s) under 35 U.S.C. 101, set forth in this Office action and to include all of the limitations of the base claim and any intervening claims.
The following is a statement of reasons for the indication of allowable subject matter: none of the references along or in combinations teaches the limitations of “wherein the machine learning model comprises an encoder and a decoder, further comprising: generating a plurality of features by applying the encoder to the image and the AR fashion item.” Recited in claim 5.
none of the references along or in combinations teaches the limitations of “applying one or more machine learning models to the real-time video feed to generate tracking information of the real-world person depicted in the real-time video feed; continuously updating the real-time video feed; and modifying placement of the AR fashion item, adjusted based on an estimated lighting adjustment, on the depiction of the real-world person.” Recited in claim 11.
none of the references along or in combinations teaches the limitations of “training a machine learning model by performing training operations comprising: accessing training data comprising training images that depict real-world objects, a training AR object, and corresponding ground-truth images that depict the real-world objects applied with the training AR objects having lighting adjustments; applying the machine learning model to an individual training image of the training images and the training AR object to estimate training lighting adjustments; combining the training AR object with the individual training image based on the estimated training lighting adjustments to generate a training modified image; computing a deviation between the training modified image and an individual ground-truth image of the ground-truth images corresponding to the individual training image; and updating one or more parameters of the machine learning model based on the computed deviation.” Recited in claim 12.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to YANNA WU whose telephone number is (571)270-0725. The examiner can normally be reached Monday-Thursday 8:00-5:30 ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Alicia Harrington can be reached at 5712722330. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/YANNA WU/Primary Examiner, Art Unit 2615
Prosecution Insights