DETAILED ACTION
This Office Action is in response to the Applicants' communication filed on August 25, 2026, which amends the independent claims 1, 8 and 15, cancels the dependent claims 5, 12 and 19, and presents arguments, is hereby acknowledged. Claims 1-8, 10-11, 13-18 and 20 are currently pending and have been examined.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
Applicant’s arguments filed on August 25, 2026, have been fully considered.
Applicant argues that by this response, the independent claims 1, 8 and 15 are hereby amended to add a new limitation “presenting, by the computing system, an interface for receiving a modification to one or more attributes of the adjusted eyelash extension template, wherein the one or more attributes include one or more of a length of the adjusted eyelash extension template, a density of the adjusted eyelash extension template, a thickness of the adjusted eyelash extension template, or a curve of the adjusted eyelash extension template; adjusting, by the computing system, the adjusted eyelash extension template based the received modification to the one or more attributes of the adjusted eyelash extension template to create a further adjusted eyelash extension template; and generating, by the computing system, an augmented reality presentation that includes the image of the face of the subject and the further adjusted eyelash extension template.” in order to overcome the 35 U.S.C. §103 rejection.
Examiner replies that the independent claim 1 is amended by adding three new limitations that may overcome the cited portions of the prior arts. However, the first newly added limitation “presenting, by the computing system, an interface for receiving a modification to one or more attributes of the adjusted eyelash extension template, wherein the one or more attributes include one or more of a length of the adjusted eyelash extension template, a density of the adjusted eyelash extension template, a thickness of the adjusted eyelash extension template, or a curve of the adjusted eyelash extension template” was originally cited in the dependent claim 5 which is now cancelled, and the cited limitation is added into the independent claim 1, and this limitation is rejected by the on-record secondary art Lotti (US 12198289 B1)and it is still rejected by the secondary art. The remaining two newly added limitations may overcome the current rejection, but a new art has been found, Toyoda, etc. (US 20130113832 A1), teaches that adjusting, by the computing system, the adjusted eyelash extension template based the received modification to the one or more attributes of the adjusted eyelash extension template to create a further adjusted eyelash extension template (See Toyota: Fig. 2, and [0043], “The form alteration unit 16 determines whether the form that is configured by connecting the tips of plural eyelashes such as the upper eyelashes from the inner corner of the eye to the outer corner of the eye generated by the eyelash generation unit 15 has been altered by the user. If the form has been altered, the form alteration unit 16 obtains information on the form altered by the user, and alters the eyelash length and the eyelash direction (angle on a horizontal plane) based on the information on the altered form. It is noted that the eyelash form and the manner of altering the eyelash form are described in detail below”; and [0038], “It is noted that setting information relating to the eyelash shape may include parameter information such as the eyelash thickness, the eyelash length, the curvature of the eyelashes when they are curled, the number of eyelash strands, and the eyelash form, for example”); and generating, by the computing system, an augmented reality presentation that includes the image of the face of the subject and the further adjusted eyelash extension template (See Toyota: Fig. 2, and [0046], “The simulation image generation unit 18 generates an eye image using the eyelashes generated by the eyelash generation unit 15 or the form altered by the form alteration unit 16. Also, the simulation image generation unit 18 may generate the eye image using the curvature altered by the curvature alteration unit 17”). The remaining arguments of the applicant are mooted in view of the newly found art.
Examiner respectfully further replies that the Applicant's arguments have been fully considered and a new ground of rejections has been made. Accordingly, new grounds of rejection are set forth below. Since the new grounds of rejection are necessitated by Applicant's amendments to the claims, the present action is made final.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-8, 10-11, 13-18 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Fu, etc. (US 20190014884 A1) in view of Lotti (US 12198289 B1), further in view of Toyoda, etc. (US 20130113832 A1).
Regarding claim 1, Fu teaches that a computer-implemented method of creating an eyelash extension, the method (See Fu: Fig. 38, and [0267], "For the various systems and methods herein, a general system structure as shown in FIG. 38 may be used and methods combined in various ways (such as in FIG. 37) or with other techniques and improvements described above. As shown in FIG. 38, a memory capability (cloud- or hardware server-based) may be employed to store in a preferably secure location all various data and data sets described herein, including eye templates, ground truth data, product data, style and makeup look information, user information, training and learning information of all types and anything else the system requires for operation. The data should be accessible to programmable system software based on the algorithms and pseudo code noted herein, and executable on a processor in a controller herein") comprising:
receiving, by a computing system, an image of a face of a subject (See Fu: Fig. 3, and [0113], "FIG. 3 shows a flow diagram of a method for virtually removing makeup in accordance with an embodiment of the present disclosure, generally referred to as embodiment 1000. Referring to FIG. 3, in Step 1010, an input is acquired by a user. The user input can be any facial image as that term is defined herein, including a single image, a repeat photo, a single video frame or a video having multiple frames, captured by the user using a mobile device or uploaded to the system of the present disclosure. Once acquired, the facial image is detected or identified from the user input. It is preferred that in the removal method, only one image is input, and that the face is detected once the image is uploaded, however, the invention is not limited to only one image input");
detecting, by the computing system, one or more eye landmarks within the image (See Fu: Fig. 3, and [0114], "Upon detection of the face of the image, in Step 1020, the facial landmarks are located using the input image. Landmarks can be preset and selected such as top of the chin, outside edge of each eye, inner edge of each eyebrow, and the like. Such landmarks are common to all faces and so are detected and evaluated using precise localization of their fiducial points (e.g. nose tip, mouth and eye corners) in color images of face foregrounds");
adjusting, by the computing system (See Fu: Figs. 28A-E, and [0173], "These effects are created using the following method. Eye templates are created as shown in FIGS. 28a-28e, wherein each template is respectively, an eye shadow template 438 (FIG. 28a), an eye middle template 440 (FIG. 28b), an eye corner template 442 (FIG. 28c), an eye tail template 444 (FIG. 28d) and an eyelash template 446 (FIG. 28e). Each such template is created by manually labeling landmarks using points on the templates according to a landmark protocol. An Example is shown in FIG. 29 wherein points identified as points 448 are applied to eye shadow template 438. The landmark locations of the points 448 are saved as a text file"; and Fig. 30, and [0175], "This eyeshadow application is illustrated as a flow chart 10000 shown in FIG. 30. In Step 10010, the landmarks are detected from a current frame and in Step 10020, the eye region is cropped from the landmarks. At the same time or prior thereto, the eye shadow template is loaded and pre-annotated with landmarks in a landmarks location file in Step 10030. Such annotated files and templates are saved in a learning database. In Step 10040, 100 points are generated around the eye region by linear interpolation based on the annotated landmarks of Step 10030. In Step 10050, 100 points are generated around the eye region of the current image frame by linear interpolation based on the detected landmarks from Step 10010. The 100 points from the template in Step 10040 are forward warped onto the 100 points of the eye region in the image from Step 10050. This creates the eye image with the templates applied in Step 10060, and the template is cropped back on the image to show the image frame with the eye shadow applied in Step 10070"), an eyelash extension template based on the one or more eye landmarks to create an adjusted eyelash extension template; and
generating, by the computing system, an augmented reality presentation that includes the image of the face of the subject and the adjusted eyelash extension template (See Fu: Fig. 7, and [0156], "In each instance, the method after such histogram matching for a first effect, e.g., a level of gloss or shiny in steps 2025a and 2025b, the method provides a first image having the output effect. If multiple output effects are desired, steps 2020 and a further one of the other parallel steps 2030a and 2030b to provide shimmer and/or a natural effect in 2040a and 2040b or combinations thereof (or additional effects as desired as would be understood by one skilled in the art based on this disclosure) can be repeated to provide one or more additional images, each of which has the related output effect as desired. The first image having the first output effect and/or the additional images with their respective output effects are combined and blended with the original facial image of the user in step 2040 to create a resultant image in step 2050 having each of the output effects combined on the facial image of the user"; and [0010], "Further, while facial landmarks detection presents many potential attractive applications in augmented reality, virtual reality, human-computer interaction, and so on, and there are now applications that let people wear virtual make-up and recognize the faces using certain end points as facial landmarks, there are still issues with such developing technology from an accuracy standpoint. For example, when using these techniques there are always two primary problems that severely influence performance of such an application in a video: shaking problems and lag problems");
presenting, by the computing system, an interface for receiving a modification to one or more attributes of the adjusted eyelash extension template, wherein the one or more attributes include one or more of a length of the adjusted eyelash extension template, a density of the adjusted eyelash extension template, a thickness of the adjusted eyelash extension template, or a curve of the adjusted eyelash extension template;
adjusting, by the computing system, the adjusted eyelash extension template based the received modification to the one or more attributes of the adjusted eyelash extension template to create a further adjusted eyelash extension template; and
generating, by the computing system, an augmented reality presentation that includes the image of the face of the subject and the further adjusted eyelash extension template.
However, Fu fails to explicitly disclose that (adjusting) an eyelash extension template based on the one or more eye landmarks to create an adjusted eyelash extension template; presenting, by the computing system, an interface for receiving a modification to one or more attributes of the adjusted eyelash extension template, wherein the one or more attributes include one or more of a length of the adjusted eyelash extension template, a density of the adjusted eyelash extension template, a thickness of the adjusted eyelash extension template, or a curve of the adjusted eyelash extension template; adjusting, by the computing system, the adjusted eyelash extension template based the received modification to the one or more attributes of the adjusted eyelash extension template to create a further adjusted eyelash extension template; and generating, by the computing system, an augmented reality presentation that includes the image of the face of the subject and the further adjusted eyelash extension template;
However, Lotti teaches that (adjusting) an eyelash extension template based on the one or more eye landmarks (See Lotti: Fig. 9, and Col. 42 Lines 54-67 ~ Col. 43 Lines 1-17, "In an embodiment, 3D model 900 may be a morphological model. A morphological model can represent the shape and structure of objects (e.g., human faces) using morphological data. In some embodiments, morphological data can describe the form and structural relationships between geometry (e.g., vertices, lines, planes and/or landmarks) of the model and enables manipulation of the geometry based on those relationships. In some embodiments, a morphological model may include a template model (e.g., 3D template model) of a human face. The template model may be initialized with template 3D model values (e.g., template landmark data) reflecting average values (e.g., average positions, sizes, colors, etc.) for an object, such as a human face. The template 3D model values may be derived from a representative collection of objects, such as human faces or features thereof. In some embodiments, the template model can be used as a reference model that can be compared to values representing a subject's unique face. In some embodiments, the comparison can generate difference information (e.g., metric) reflecting differences (e.g., deltas or deviations) between the template 3D model values, and in particular the template landmark data, and values representing corresponding points and/or facial features of the subject's face. The difference information can be stored as part of 3D landmark data 812. To generate the 3D model of the subject's face, conversion system 820 may adjust the template model based on the difference information corresponding to a particular subject, which can contribute to computational efficiency in generating a 3D model. In some embodiments, a morphological model can be used with a PCA model to generate a 3D model, as described further below"); and
presenting, by the computing system, an interface for receiving a modification to one or more attributes of the adjusted eyelash extension template, wherein the one or more attributes include one or more of a length of the adjusted eyelash extension template, a density of the adjusted eyelash extension template, a thickness of the adjusted eyelash extension template, or a curve of the adjusted eyelash extension template (See Lotti: Figs. 1-2, and Col. 7 Lines 58-67 ~ Col. 8 Lines 1-9, "In some embodiments, a client device, such as client device 110, can implement or include one or more applications, such as application 119 executed at client device 110. In some embodiments, application 119 can be used to communicate (e.g., send and receive information) with beauty products platform 120 beauty products platform 120. In some embodiments, application 119 can implement user interfaces (Uls) (e.g., graphical user interfaces (GUls)), such as UI 112 that may be webpages rendered by a web browser and displayed on the client device 110 in a web browser window. In another embodiment, the Uls 112 of client application 119 may be included in a stand-alone application downloaded to the client device 110 and natively running on the client device 110 (also referred to as a "native application" or "native client application" herein). In some embodiments, preview module 151 can be implemented as part of application 119. In other embodiments, preview module 151 can be separate from application 119 and application 119 can interface with preview module 151"; Col. 19 Lines 1-21, "Lash configuration information (also referred to as "lash map" herein) can refer to information related to the selection of artificial lash extensions and/or the application of artificial lash extensions at the eye area of a user. In some embodiments, lash configuration information can identify the particular artificial lash extensions of a set of lash extensions (e.g., length, style, and/or color), a location at the underside of the natural lashes at which each particular artificial lash extension of the set of artificial lash extensions is to be applied, and/or the order of each artificial lash extension in the set of artificial lash extensions. In some embodiments and as described further below, lash configuration information can include one or more of style information, length information, color information, placement information, or order information for an eye or pair of eyes of a user. An example of lash configuration information is illustrated in element 235. Although described with respect to artificial lash extensions for purposes of illustration, rather than limitation, lash configuration information can apply to false eyelashes, generally, in some embodiments": and Fig. 6, and Col. 36 Lines 46-67 ~ Col. 37 Lines 1-3, "In some embodiments, an artificial lash extension can include a base. For example, artificial lash extension 600 includes base 606. In some embodiments, artificial lash extension 700 may or may not (as illustrated) include a base similar to base 606 of artificial lash extension 600. The base can include a top side (e.g., facing out of the page and towards the reader), a bottom side, a back side, a front side, and two ends (e.g., two lateral sides). In some embodiments, one or more of the multiple artificial hairs of artificial lash extension protrude out the front side of the base. When arranged at the underside of a natural lash, the backside of the artificial lash extension can point towards the user's eye. The thickness (e.g., between the top side and bottom side of the base can be between approximately 0.05 millimeters (mm) and approximately 0.15 mm (e.g., 0.05 mm+/-0.01 mm). In some embodiments, the thickness of the base can be less than 0.05 mm. In some embodiments, the low profile of the base is designed to allow the artificial lash extension to be light weight to better adhere to the underside of the natural lash and prevent obstruction of a user's view. The low profile of the base can at least in part be attributed to an attachment operation that forms the base and/or attaches clusters of artificial hairs to the base. For example, the attachment operation can include an application of heat that, at least in part, creates a base with a low profile").
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention was effectively filed to modify Fu to have (adjusting) an eyelash extension template based on the one or more eye landmarks to create an adjusted eyelash extension template; and presenting, by the computing system, an interface for receiving a modification to one or more attributes of the adjusted eyelash extension template, wherein the one or more attributes include one or more of a length of the adjusted eyelash extension template, a density of the adjusted eyelash extension template, a thickness of the adjusted eyelash extension template, or a curve of the adjusted eyelash extension template as taught by Lotti in order to enable previewing beauty products using static images e.g. static images, and ensuring precision in size, location, and/or orientation of a virtual item, leading to deficiencies in realism and ability to accurately gage final appearance (See Lotti: Figs. lA-B, and Col 6 Lines 29-51, "In some embodiments, beauty product previews, (e.g., enhanced by AR technology) can refer to methods for visualizing and trying on beauty products. AR beauty product previews can allow beauty products to be preview without actually applying the products. These AR previews can provide a personalized glimpse into how products such as cosmetics, hair colors, and nail polishes will look on the user. Skincare previews through AR can simulate the potential effects of various treatments and serums on the user's skin, illustrating improvements in texture, tone, and hydration. Haircare previews can allow individuals to see themselves with different hair colors, cuts, or styles, thereby facilitating informed decisions about new products or treatments. In the realm of cosmetics, AR technology enables users to preview a wide range of makeup products including foundation, lipstick, eyeshadow, and mascara, showcasing the products' colors and finishes on their own faces in real-time. Nail care previews through AR can display an array of polish colors and finishes on the user's own nails. In some embodiments, false eyelashes previews can show false eyelashes (e.g., artificial lash extensions) as an AR artifact placed as an overlay of a 2D image (e.g., still image or video frame) representing a user's face"). Fu teaches a method and system that may provide systems and methods for virtual facial makeup simulation through virtual makeup removal and virtual makeup add-ons, virtual end effects and simulated textures; while Lotti teaches a system and method that may use augmented reality to preview false eyelashes by dynamically adjusting the eyelash extension templates. Therefore, it is obvious to one of ordinary skill in the art to modify Fu by Lotti to adjust an eyelash extension template based on the one or more eye landmarks. The motivation to modify Fu by Lotti is "Use of known technique to improve similar devices (methods, or products) in the same way".
However, Fu, modified by Lotti, fails to explicitly disclose that adjusting, by the computing system, the adjusted eyelash extension template based the received modification to the one or more attributes of the adjusted eyelash extension template to create a further adjusted eyelash extension template; and generating, by the computing system, an augmented reality presentation that includes the image of the face of the subject and the further adjusted eyelash extension template.
However, Toyoda teaches that adjusting, by the computing system, the adjusted eyelash extension template based the received modification to the one or more attributes of the adjusted eyelash extension template to create a further adjusted eyelash extension template (See Toyota: Fig. 2, and [0043], “The form alteration unit 16 determines whether the form that is configured by connecting the tips of plural eyelashes such as the upper eyelashes from the inner corner of the eye to the outer corner of the eye generated by the eyelash generation unit 15 has been altered by the user. If the form has been altered, the form alteration unit 16 obtains information on the form altered by the user, and alters the eyelash length and the eyelash direction (angle on a horizontal plane) based on the information on the altered form. It is noted that the eyelash form and the manner of altering the eyelash form are described in detail below”; and [0038], “It is noted that setting information relating to the eyelash shape may include parameter information such as the eyelash thickness, the eyelash length, the curvature of the eyelashes when they are curled, the number of eyelash strands, and the eyelash form, for example”); and
generating, by the computing system, an augmented reality presentation that includes the image of the face of the subject and the further adjusted eyelash extension template (See Toyota: Fig. 2, and [0046], “The simulation image generation unit 18 generates an eye image using the eyelashes generated by the eyelash generation unit 15 or the form altered by the form alteration unit 16. Also, the simulation image generation unit 18 may generate the eye image using the curvature altered by the curvature alteration unit 17”).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention was effectively filed to modify Fu to have adjusting, by the computing system, the adjusted eyelash extension template based the received modification to the one or more attributes of the adjusted eyelash extension template to create a further adjusted eyelash extension template; and generating, by the computing system, an augmented reality presentation that includes the image of the face of the subject and the further adjusted eyelash extension template as taught by Lotti in order to change the shape of the eyelash freely to investigate the user preference property with respect to the finishing of mascara from the produced around-the-eye image (See Toyota: Figs. 1-2, and [0147], " As can be appreciated, according to an aspect of the present invention, the eyelash shape may be freely changed to generate an eye image and the generated eye image may be used to conduct a user preference survey with respect to a mascara finish in a simple manner at low cost. For example, a mascara finish preference may be quantified using numeric values representing the length, thickness, number of strands, curl, and overall form of the eyelashes, and the quantified data may be numerically analyzed. Also, an image of a finish reflecting these numeric values may be represented"). Fu teaches a method and system that may provide systems and methods for virtual facial makeup simulation through virtual makeup removal and virtual makeup add-ons, virtual end effects and simulated textures; while Toyota teaches a system and method that may generate an adjusted eyelash model by altering its shape parameters. Therefore, it is obvious to one of ordinary skill in the art to modify Fu by Toyota to freely change the eyelash shape parameters to generate the adjusted eyelash template (model). The motivation to modify Fu by Toyota is "Use of known technique to improve similar devices (methods, or products) in the same way".
Regarding claim 2, Fu, Lotti and Toyota teach all the features with respect to claim 1 as outlined above. Further, Fu teaches that the computer-implemented method of claim 1, wherein generating the augmented reality presentation includes:
cropping, by the computing system, the image of the face of the subject to an eye region based on the one or more eye landmarks (See Fu: Figs. 21 and 28A-E, and [0174], "To create the eye shadow add-on, the eye region landmarks are extracted from a landmarks detector for the frame. The eye region is cropped by the interested landmarks from 37 to 42 and 43 to 48. 100 points are generated b linear interpolation around the eye region from detected annotated landmarks (the landmarks may be annotated using an annotation system as described herein). From this, 100 points are generated around the eye shadow template based on the manually annotated landmarks. The template is applied to the frame by forward warping the 100 points from frame to template. There are many available warping algorithms such as forward warping, inverse warping and similarity transformation, affine transformation and the like. In the preferred embodiment, forward warping with affine transformation was adopted. Following this, the eye region image is cropped back into the original frame"); and
detecting, by the computing system, one or more extension attachment points based on the cropped image of the eye region (See Fu: Figs. 21 and 28A-E, and [0174], "To create the eye shadow add-on, the eye region landmarks are extracted from a landmarks detector for the frame. The eye region is cropped by the interested landmarks from 37 to 42 and 43 to 48. 100 points are generated b linear interpolation around the eye region from detected annotated landmarks (the landmarks may be annotated using an annotation system as described herein). From this, 100 points are generated around the eye shadow template based on the manually annotated landmarks. The template is applied to the frame by forward warping the 100 points from frame to template. There are many available warping algorithms such as forward warping, inverse warping and similarity transformation, affine transformation and the like. In the preferred embodiment, forward warping with affine transformation was adopted. Following this, the eye region image is cropped back into the original frame". Note that after cropping, more points are added, which is mapped to "adjusting").
Regarding claim 3, Fu, Lotti and Toyota teach all the features with respect to claim 2 as outlined above. Further, Fu, Lotti and Toyota teach that the computer-implemented method of claim 2, wherein the eyelash extension template includes a two-dimensional image of the eyelash extension (See Fu: Figs. 28A-E, and [0173], "These effects are created using the following method. Eye templates are created as shown in FIGS. 28a-28e, wherein each template is respectively, an eye shadow template 438 (FIG. 28a), an eye middle template 440 (FIG. 28b), an eye corner template 442 (FIG. 28c), an eye tail template 444 (FIG. 28d) and an eye lash template 446 (FIG. 28e). Each such template is created by manually labeling landmarks using points on the templates according to a landmark protocol. An Example is shown in FIG. 29 wherein points identified as points 448 are applied to eye shadow template 438. The landmark locations of the points 448 are saved as a text file"); and
wherein generating the augmented reality presentation (See Fu: Fig. 3, and [0097], "Virtual Facial Makeup Simulation For Augmented Personalized Tutorials") includes:
adjusting the two-dimensional image of the eyelash extension based on the adjusted eyelash extension template (See Lotti: Fig. 9, and Col. 42 Lines 54-67 ~ Col. 43 Lines 1-17, "In an embodiment, 3D model 900 may be a morphological model. A morphological model can represent the shape and structure of objects (e.g., human faces) using morphological data. In some embodiments, morphological data can describe the form and structural relationships between geometry (e.g., vertices, lines, planes and/or landmarks) of the model and enables manipulation of the geometry based on those relationships. In some embodiments, a morphological model may include a template model (e.g., 3D template model) of a human face. The template model may be initialized with template 3D model values (e.g., template landmark data) reflecting average values (e.g., average positions, sizes, colors, etc.) for an object, such as a human face. The template 3D model values may be derived from a representative collection of objects, such as human faces or features thereof. In some embodiments, the template model can be used as a reference model that can be compared to values representing a subject's unique face. In some embodiments, the comparison can generate difference information (e.g., metric) reflecting differences (e.g., deltas or deviations) between the template 3D model values, and in particular the template landmark data, and values representing corresponding points and/or facial features of the subject's face. The difference information can be stored as part of 3D landmark data 812. To generate the 3D model of the subject's face, conversion system 820 may adjust the template model based on the difference information corresponding to a particular subject, which can contribute to computational efficiency in generating a 3D model. In some embodiments, a morphological model can be used with a PCA model to generate a 3D model, as described further below"); and
rendering the adjusted two-dimensional image of the eyelash extension based on the one or more extension attachment points (See Fu: Fig. 7, and [0156], "In each instance, the method after such histogram matching for a first effect, e.g., a level of gloss or shiny in steps 2025a and 2025b, the method provides a first image having the output effect. If multiple output effects are desired, steps 2020 and a further one of the other parallel steps 2030a and 2030b to provide shimmer and/or a natural effect in 2040a and 2040b or combinations thereof (or additional effects as desired as would be understood by one skilled in the art based on this disclosure) can be repeated to provide one or more additional images, each of which has the related output effect as desired. The first image having the first output effect and/or the additional images with their respective output effects are combined and blended with the original facial image of the user in step 2040 to create a resultant image in step 2050 having each of the output effects combined on the facial image of the user"; and [0010], "Further, while facial landmarks detection presents many potential attractive applications in augmented reality, virtual reality, human-computer interaction, and so on, and there are now applications that let people wear virtual make-up and recognize the faces using certain end points as facial landmarks, there are still issues with such developing technology from an accuracy standpoint. For example, when using these techniques there are always two primary problems that severely influence performance of such an application in a video: shaking problems and lag problems").
Regarding claim 4, Fu, Lotti and Toyota teach all the features with respect to claim 2 as outlined above. Further, Lotti teaches that the computer-implemented method of claim 2, wherein the eyelash extension template includes a three-dimensional model of the eyelash extension (See Lotti: Fig. 1, and Col. 10 Lines 10-32, "In some embodiments, beauty products platform 120 can implement preview module 151. In some embodiments, lash preview module 151 can implement one or more features, operations and/or embodiments as described herein. For example, in some embodiments, a video segment of a video stream having a representation of the subject's face can be received by the beauty products platform 120 and/or preview module 151. A computer vision operation can be performed on the video segment to track one or more points (e.g., two-dimensional (2D) points) of the video segment across multiple video frames. In some embodiments, beauty products platform 120 and/or preview module 151 can receive an indication of a user selection of false eyelashes among a set of false eyelashes for preview using AR. Using the tracked points, the beauty products platform 120 and/or preview module 151 can modify the video segment to include an overlay of a 3D model of false eyelashes (e.g., virtual element) proximate the tracked points (e.g., corresponding to the eye area of the subject). The modified video segment including the overlay of the 3D model of false eyelashes proximate the tracked points can be provided to the client device 110 for presentation in real-time in an AR environment"); and
wherein generating the augmented reality presentation (See Lotti: Fig. 3, and Col. 21 Lines 34-56, "FIG. 3 illustrates a flow diagram of an example method 300 for using augmented reality to preview artificial lash extensions, in accordance with some embodiments. Method 300 may be performed by processing logic that may comprise hardware (e.g., circuitry, dedicated logic, etc.), computer-readable instructions such as software or firmware (e.g., run on a general-purpose computing system or a dedicated machine), or a combination thereof. Method 300 may also be associated with sets of instructions stored on a non-transitory computer-readable medium (e.g., magnetic or optical disk, etc.). The instructions, when executed by a processing device, may cause the processing device to perform operations of method 300. In an embodiment, the operations of method 300 are performed by computing system 1200 of FIG. 12. In some embodiments, the operations of method 300 are performed by preview module 151 of FIG. lA-1B. In an embodiment, operations of a particular method depicted in FIG. 3 can be performed simultaneously or in different orders than depicted. Various embodiments may include additional operations not depicted in FIG. 3 or a subset of operations depicted in FIG. 3. In some embodiments, method 300 can be performed with the same, different, fewer, or more operations in the same or different order") includes:
adjusting the three-dimensional model of the eyelash extension based on the adjusted eyelash extension template (See Lotti: Fig. 9, and Col. 42 Lines 54-67 ~ Col. 43 Lines 1-17, "In an embodiment, 3D model 900 may be a morphological model. A morphological model can represent the shape and structure of objects (e.g., human faces) using morphological data. In some embodiments, morphological data can describe the form and structural relationships between geometry (e.g., vertices, lines, planes and/or landmarks) of the model and enables manipulation of the geometry based on those relationships. In some embodiments, a morphological model may include a template model (e.g., 3D template model) of a human face. The template model may be initialized with template 3D model values (e.g., template landmark data) reflecting average values (e.g., average positions, sizes, colors, etc.) for an object, such as a human face. The template 3D model values may be derived from a representative collection of objects, such as human faces or features thereof. In some embodiments, the template model can be used as a reference model that can be compared to values representing a subject's unique face. In some embodiments, the comparison can generate difference information (e.g., metric) reflecting differences (e.g., deltas or deviations) between the template 3D model values, and in particular the template landmark data, and values representing corresponding points and/or facial features of the subject's face. The difference information can be stored as part of 3D landmark data 812. To generate the 3D model of the subject's face, conversion system 820 may adjust the template model based on the difference information corresponding to a particular subject, which can contribute to computational efficiency in generating a 3D model. In some embodiments, a morphological model can be used with a PCA model to generate a 3D model, as described further below"); and
rendering the adjusted three-dimensional model of the eyelash extension based on the one or more extension attachment points (See Lotti: Fig. 3, and Col. 24 Lines 25-42, "At block 306, the processing logic can receive an indication of a user selection of the false eyelashes among a set of false eyelashes. For example, a user can select set of false eyelashes (e.g., artificial lash extensions) associated with specific lash configuration information that can identify one or more a style, length, color, placement, order, etc. In some embodiments, a user can select various artificial lash extensions or sets of artificial lash extensions and preview the artificial lash extensions or sets of artificial lash extensions. The preview can include showing the artificial lash extensions or sets of artificial lash extensions as virtual elements applied to a correct location in video frames of the video segment at the underside of natural eyelashes of the user. In some embodiments, by realistically rendering the artificial lash extensions as an overlay on video frames representing the user (e.g., still image or frames of the video stream/video segment) the artificial lash extensions as applied to the user can be virtually previewed").
Regarding claim 6, Fu, Lotti and Toyota teach all the features with respect to claim 1 as outlined above. Further, Fu teaches that the computer-implemented method of claim 1, further comprising:
presenting, by the computing system, one or more recommended eyelash extension templates based on the detected eye landmarks (See Fu: Fig. 11, and [0207], "In FIG. 11, the makeup recommendation system generates personalized step-by-step makeup instructions using real products in the database. The trained models for different makeup styles 4040 may be taken from the deep learning system 4000 and annotation system 5000 which can be input into the makeup recommendation system 6000 to provide a personalized makeup recommendation 7050, and also optionally a virtual makeup tutorial may be provided as described below. The make-up recommendation 7050 can be derived from a makeup recommender 7020 from the trained system and models such as trained models 4040, although a separate trained model may be created solely for use with a recommendation system. Product matching 7030 can also be used using a makeup product database, which may be the same or different from the makeup database 7045 (as shown in FIG. 1, it is the same database). The makeup recommender and/or product matching can result in the personalized makeup recommendation 7050. Virtual tutorials may also be generated using segmented video pathways or taking information from product searching and identification using a trained products classifier from a beauty products database as discussed below").
Regarding claim 7, Fu, Lotti and Toyota teach all the features with respect to claim 1 as outlined above. Further, Fu teaches that the computer-implemented method of claim 1, further comprising:
transmitting, by the computing system, instructions to an eyelash extension creation system to cause the eyelash extension creation system to apply the eyelash extension to the subject based on the adjusted eyelash extension template (See Fu: Fig. 38, and [0279], "It should also be noted that implementations of the systems and methods can be provided as one or more computer-readable programs embodied on or in one or more articles of manufacture. The program instructions can be encoded on an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus"; and Fig. 11, and [0207], "In FIG. 11, the makeup recommendation system generates personalized step-by-step makeup instructions using real products in the database. The trained models for different makeup styles 4040 may be taken from the deep learning system 4000 and annotation system 5000 which can be input into the makeup recommendation system 6000 to provide a personalized makeup recommendation 7050, and also optionally a virtual makeup tutorial may be provided as described below. The make-up recommendation 7050 can be derived from a makeup recommender 7020 from the trained system and models such as trained models 4040, although a separate trained model may be created solely for use with a recommendation system. Product matching 7030 can also be used using a makeup product database, which may be the same or different from the makeup database 7045 (as shown in FIG. 1, it is the same database). The makeup recommender and/or product matching can result in the personalized makeup recommendation 7050. Virtual tutorials may also be generated using segmented video pathways or taking information from product searching and identification using a trained products classifier from a beauty products database as discussed below").
Regarding claim 8, Fu, Lotti and Toyota teach all the features with respect to claim 1 as outlined above. Further, Fu, Lotti and Toyota teach that a non-transitory computer-readable medium having computer-executable instructions stored thereon that, in response to execution by one or more processors of a computing system, cause the computing system to perform actions for creating an eyelash extension, the actions (See Fu: Fig. 38, and [0267], "For the various systems and methods herein, a general system structure as shown in FIG. 38 may be used and methods combined in various ways (such as in FIG. 37) or with other techniques and improvements described above. As shown in FIG. 38, a memory capability (cloud- or hardware server-based) may be employed to store in a preferably secure location all various data and data sets described herein, including eye templates, ground truth data, product data, style and makeup look information, user information, training and learning information of all types and anything else the system requires for operation. The data should be accessible to programmable system software based on the algorithms and pseudo code noted herein, and executable on a processor in a controller herein") comprising:
receiving, by the computing system, an image of a face of a subject (See Fu: Fig. 3, and [0113], "FIG. 3 shows a flow diagram of a method for virtually removing makeup in accordance with an embodiment of the present disclosure, generally referred to as embodiment 1000. Referring to FIG. 3, in Step 1010, an input is acquired by a user. The user input can be any facial image as that term is defined herein, including a single image, a repeat photo, a single video frame or a video having multiple frames, captured by the user using a mobile device or uploaded to the system of the present disclosure. Once acquired, the facial image is detected or identified from the user input. It is preferred that in the removal method, only one image is input, and that the face is detected once the image is uploaded, however, the invention is not limited to only one image input");
detecting, by the computing system, one or more eye landmarks within the image (See Fu: Fig. 3, and [0114], "Upon detection of the face of the image, in Step 1020, the facial landmarks are located using the input image. Landmarks can be preset and selected such as top of the chin, outside edge of each eye, inner edge of each eyebrow, and the like. Such landmarks are common to all faces and so are detected and evaluated using precise localization of their fiducial points (e.g. nose tip, mouth and eye corners) in color images of face foregrounds");
adjusting, by the computing system (See Fu: Figs. 28A-E, and [0173], "These effects are created using the following method. Eye templates are created as shown in FIGS. 28a-28e, wherein each template is respectively, an eye shadow template 438 (FIG. 28a), an eye middle template 440 (FIG. 28b), an eye corner template 442 (FIG. 28c), an eye tail template 444 (FIG. 28d) and an eye lash template 446 (FIG. 28e). Each such template is created by manually labeling landmarks using points on the templates according to a landmark protocol. An Example is shown in FIG. 29 wherein points identified as points 448 are applied to eye shadow template 438. The landmark locations of the points 448 are saved as a text file"; and Fig. 30, and [0175], "This eyeshadow application is illustrated as a flow chart 10000 shown in FIG. 30. In Step 10010, the landmarks are detected from a current frame and in Step 10020, the eye region is cropped from the landmarks. At the same time or prior thereto, the eye shadow template is loaded and pre-annotated with landmarks in a landmarks location file in Step 10030. Such annotated files and templates are saved in a learning database. In Step 10040, 100 points are generated around the eye region by linear interpolation based on the annotated landmarks of Step 10030. In Step 10050, 100 points are generated around the eye region of the current image frame by linear interpolation based on the detected landmarks from Step 10010. The 100 points from the template in Step 10040 are forward warped onto the 100 points of the eye region in the image from Step 10050. This creates the eye image with the templates applied in Step 10060, and the template is cropped back on the image to show the image frame with the eye shadow applied in Step 10070"), an eyelash extension template based on the one or more eye landmarks to create an adjusted eyelash extension template (See Lotti: Fig. 9, and Col. 42 Lines 54-67 ~ Col. 43 Lines 1-17, "In an embodiment, 3D model 900 may be a morphological model. A morphological model can represent the shape and structure of objects (e.g., human faces) using morphological data. In some embodiments, morphological data can describe the form and structural relationships between geometry (e.g., vertices, lines, planes and/or landmarks) of the model and enables manipulation of the geometry based on those relationships. In some embodiments, a morphological model may include a template model (e.g., 3D template model) of a human face. The template model may be initialized with template 3D model values (e.g., template landmark data) reflecting average values (e.g., average positions, sizes, colors, etc.) for an object, such as a human face. The template 3D model values may be derived from a representative collection of objects, such as human faces or features thereof. In some embodiments, the template model can be used as a reference model that can be compared to values representing a subject's unique face. In some embodiments, the comparison can generate difference information (e.g., metric) reflecting differences (e.g., deltas or deviations) between the template 3D model values, and in particular the template landmark data, and values representing corresponding points and/or facial features of the subject's face. The difference information can be stored as part of 3D landmark data 812. To generate the 3D model of the subject's face, conversion system 820 may adjust the template model based on the difference information corresponding to a particular subject, which can contribute to computational efficiency in generating a 3D model. In some embodiments, a morphological model can be used with a PCA model to generate a 3D model, as described further below");
generating, by the computing system, an augmented reality presentation that includes the image of the face of the subject and the adjusted eyelash extension template (See Fu: Fig. 7, and [0156], "In each instance, the method after such histogram matching for a first effect, e.g., a level of gloss or shiny in steps 2025a and 2025b, the method provides a first image having the output effect. If multiple output effects are desired, steps 2020 and a further one of the other parallel steps 2030a and 2030b to provide shimmer and/or a natural effect in 2040a and 2040b or combinations thereof (or additional effects as desired as would be understood by one skilled in the art based on this disclosure) can be repeated to provide one or more additional images, each of which has the related output effect as desired. The first image having the first output effect and/or the additional images with their respective output effects are combined and blended with the original facial image of the user in step 2040 to create a resultant image in step 2050 having each of the output effects combined on the facial image of the user"; and [0010], "Further, while facial landmarks detection presents many potential attractive applications in augmented reality, virtual reality, human-computer interaction, and so on, and there are now applications that let people wear virtual make-up and recognize the faces using certain end points as facial landmarks, there are still issues with such developing technology from an accuracy standpoint. For example, when using these techniques there are always two primary problems that severely influence performance of such an application in a video: shaking problems and lag problems").
presenting, by the computing system, an interface for receiving a modification to one or more attributes of the adjusted eyelash extension template, wherein the one or more attributes include one or more of a length of the adjusted eyelash extension template, a density of the adjusted eyelash extension template, a thickness of the adjusted eyelash extension template, or a curve of the adjusted eyelash extension template (See Lotti: Figs. 1-2, and Col. 7 Lines 58-67 ~ Col. 8 Lines 1-9, "In some embodiments, a client device, such as client device 110, can implement or include one or more applications, such as application 119 executed at client device 110. In some embodiments, application 119 can be used to communicate (e.g., send and receive information) with beauty products platform 120 beauty products platform 120. In some embodiments, application 119 can implement user interfaces (Uls) (e.g., graphical user interfaces (GUls)), such as UI 112 that may be webpages rendered by a web browser and displayed on the client device 110 in a web browser window. In another embodiment, the Uls 112 of client application 119 may be included in a stand-alone application downloaded to the client device 110 and natively running on the client device 110 (also referred to as a "native application" or "native client application" herein). In some embodiments, preview module 151 can be implemented as part of application 119. In other embodiments, preview module 151 can be separate from application 119 and application 119 can interface with preview module 151"; Col. 19 Lines 1-21, "Lash configuration information (also referred to as "lash map" herein) can refer to information related to the selection of artificial lash extensions and/or the application of artificial lash extensions at the eye area of a user. In some embodiments, lash configuration information can identify the particular artificial lash extensions of a set of lash extensions (e.g., length, style, and/or color), a location at the underside of the natural lashes at which each particular artificial lash extension of the set of artificial lash extensions is to be applied, and/or the order of each artificial lash extension in the set of artificial lash extensions. In some embodiments and as described further below, lash configuration information can include one or more of style information, length information, color information, placement information, or order information for an eye or pair of eyes of a user. An example of lash configuration information is illustrated in element 235. Although described with respect to artificial lash extensions for purposes of illustration, rather than limitation, lash configuration information can apply to false eyelashes, generally, in some embodiments": and Fig. 6, and Col. 36 Lines 46-67 ~ Col. 37 Lines 1-3, "In some embodiments, an artificial lash extension can include a base. For example, artificial lash extension 600 includes base 606. In some embodiments, artificial lash extension 700 may or may not (as illustrated) include a base similar to base 606 of artificial lash extension 600. The base can include a top side (e.g., facing out of the page and towards the reader), a bottom side, a back side, a front side, and two ends (e.g., two lateral sides). In some embodiments, one or more of the multiple artificial hairs of artificial lash extension protrude out the front side of the base. When arranged at the underside of a natural lash, the backside of the artificial lash extension can point towards the user's eye. The thickness (e.g., between the top side and bottom side of the base can be between approximately 0.05 millimeters (mm) and approximately 0.15 mm (e.g., 0.05 mm+/-0.01 mm). In some embodiments, the thickness of the base can be less than 0.05 mm. In some embodiments, the low profile of the base is designed to allow the artificial lash extension to be light weight to better adhere to the underside of the natural lash and prevent obstruction of a user's view. The low profile of the base can at least in part be attributed to an attachment operation that forms the base and/or attaches clusters of artificial hairs to the base. For example, the attachment operation can include an application of heat that, at least in part, creates a base with a low profile");
adjusting, by the computing system, the adjusted eyelash extension template based the received modification to the one or more attributes of the adjusted eyelash extension template to create a further adjusted eyelash extension template (See Toyota: Fig. 2, and [0043], “The form alteration unit 16 determines whether the form that is configured by connecting the tips of plural eyelashes such as the upper eyelashes from the inner corner of the eye to the outer corner of the eye generated by the eyelash generation unit 15 has been altered by the user. If the form has been altered, the form alteration unit 16 obtains information on the form altered by the user, and alters the eyelash length and the eyelash direction (angle on a horizontal plane) based on the information on the altered form. It is noted that the eyelash form and the manner of altering the eyelash form are described in detail below”; and [0038], “It is noted that setting information relating to the eyelash shape may include parameter information such as the eyelash thickness, the eyelash length, the curvature of the eyelashes when they are curled, the number of eyelash strands, and the eyelash form, for example”); and
generating, by the computing system, an augmented reality presentation that includes the image of the face of the subject and the further adjusted eyelash extension template (See Toyota: Fig. 2, and [0046], “The simulation image generation unit 18 generates an eye image using the eyelashes generated by the eyelash generation unit 15 or the form altered by the form alteration unit 16. Also, the simulation image generation unit 18 may generate the eye image using the curvature altered by the curvature alteration unit 17”).
Regarding claim 9, Fu, Lotti and Toyota teach all the features with respect to claim 8 as outlined above. Further, Fu teaches that the non-transitory computer-readable medium of claim 8, wherein generating the augmented reality presentation includes:
cropping, by the computing system, the image of the face of the subject to an eye region based on the one or more eye landmarks (See Fu: Figs. 21 and 28A-E, and [0174], "To create the eye shadow add-on, the eye region landmarks are extracted from a landmarks detector for the frame. The eye region is cropped by the interested landmarks from 37 to 42 and 43 to 48. 100 points are generated b linear interpolation around the eye region from detected annotated landmarks (the landmarks may be annotated using an annotation system as described herein). From this, 100 points are generated around the eye shadow template based on the manually annotated landmarks. The template is applied to the frame by forward warping the 100 points from frame to template. There are many available warping algorithms such as forward warping, inverse warping and similarity transformation, affine transformation and the like. In the preferred embodiment, forward warping with affine transformation was adopted. Following this, the eye region image is cropped back into the original frame"); and
detecting, by the computing system, one or more extension attachment points based on the cropped image of the eye region (See Fu: Figs. 21 and 28A-E, and [0174], "To create the eye shadow add-on, the eye region landmarks are extracted from a landmarks detector for the frame. The eye region is cropped by the interested landmarks from 37 to 42 and 43 to 48. 100 points are generated b linear interpolation around the eye region from detected annotated landmarks (the landmarks may be annotated using an annotation system as described herein). From this, 100 points are generated around the eye shadow template based on the manually annotated landmarks. The template is applied to the frame by forward warping the 100 points from frame to template. There are many available warping algorithms such as forward warping, inverse warping and similarity transformation, affine transformation and the like. In the preferred embodiment, forward warping with affine transformation was adopted. Following this, the eye region image is cropped back into the original frame". Note that after cropping, more points are added, which is mapped to "adjusting").
Regarding claim 10, Fu, Lotti and Toyota teach all the features with respect to claim 9 as outlined above. Further, Fu, Lotti and Toyota teach that the non-transitory computer-readable medium of claim 9, wherein the eyelash extension template includes a two-dimensional image of the eyelash extension (See Fu: Figs. 28A-E, and [0173], "These effects are created using the following method. Eye templates are created as shown in FIGS. 28a-28e, wherein each template is respectively, an eye shadow template 438 (FIG. 28a), an eye middle template 440 (FIG. 28b), an eye corner template 442 (FIG. 28c), an eye tail template 444 (FIG. 28d) and an eye lash template 446 (FIG. 28e). Each such template is created by manually labeling landmarks using points on the templates according to a landmark protocol. An Example is shown in FIG. 29 wherein points identified as points 448 are applied to eye shadow template 438. The landmark locations of the points 448 are saved as a text file"); and
wherein generating the augmented reality presentation (See Fu: Fig. 3, and [0097], "Virtual Facial Makeup Simulation For Augmented Personalized Tutorials") includes:
adjusting the two-dimensional image of the eyelash extension based on the adjusted eyelash extension template (See Lotti: Fig. 9, and Col. 42 Lines 54-67 ~ Col. 43 Lines 1-17, "In an embodiment, 3D model 900 may be a morphological model. A morphological model can represent the shape and structure of objects (e.g., human faces) using morphological data. In some embodiments, morphological data can describe the form and structural relationships between geometry (e.g., vertices, lines, planes and/or landmarks) of the model and enables manipulation of the geometry based on those relationships. In some embodiments, a morphological model may include a template model (e.g., 3D template model) of a human face. The template model may be initialized with template 3D model values (e.g., template landmark data) reflecting average values (e.g., average positions, sizes, colors, etc.) for an object, such as a human face. The template 3D model values may be derived from a representative collection of objects, such as human faces or features thereof. In some embodiments, the template model can be used as a reference model that can be compared to values representing a subject's unique face. In some embodiments, the comparison can generate difference information (e.g., metric) reflecting differences (e.g., deltas or deviations) between the template 3D model values, and in particular the template landmark data, and values representing corresponding points and/or facial features of the subject's face. The difference information can be stored as part of 3D landmark data 812. To generate the 3D model of the subject's face, conversion system 820 may adjust the template model based on the difference information corresponding to a particular subject, which can contribute to computational efficiency in generating a 3D model. In some embodiments, a morphological model can be used with a PCA model to generate a 3D model, as described further below"); and
rendering the adjusted two-dimensional image of the eyelash extension based on the one or more extension attachment points (See Fu: Fig. 7, and [0156], "In each instance, the method after such histogram matching for a first effect, e.g., a level of gloss or shiny in steps 2025a and 2025b, the method provides a first image having the output effect. If multiple output effects are desired, steps 2020 and a further one of the other parallel steps 2030a and 2030b to provide shimmer and/or a natural effect in 2040a and 2040b or combinations thereof (or additional effects as desired as would be understood by one skilled in the art based on this disclosure) can be repeated to provide one or more additional images, each of which has the related output effect as desired. The first image having the first output effect and/or the additional images with their respective output effects are combined and blended with the original facial image of the user in step 2040 to create a resultant image in step 2050 having each of the output effects combined on the facial image of the user"; and [0010], "Further, while facial landmarks detection presents many potential attractive applications in augmented reality, virtual reality, human-computer interaction, and so on, and there are now applications that let people wear virtual make-up and recognize the faces using certain end points as facial landmarks, there are still issues with such developing technology from an accuracy standpoint. For example, when using these techniques there are always two primary problems that severely influence performance of such an application in a video: shaking problems and lag problems").
Regarding claim 11, Fu, Lotti and Toyota teach all the features with respect to claim 8 as outlined above. Further, Lotti teaches that the non-transitory computer-readable medium of claim 9, wherein the eyelash extension template includes a three-dimensional model of the eyelash extension (See Lotti: Fig. 1, and Col. 10 Lines 10-32, "In some embodiments, beauty products platform 120 can implement preview module 151. In some embodiments, lash preview module 151 can implement one or more features, operations and/or embodiments as described herein. For example, in some embodiments, a video segment of a video stream having a representation of the subject's face can be received by the beauty products platform 120 and/or preview module 151. A computer vision operation can be performed on the video segment to track one or more points (e.g., two-dimensional (2D) points) of the video segment across multiple video frames. In some embodiments, beauty products platform 120 and/or preview module 151 can receive an indication of a user selection of false eyelashes among a set of false eyelashes for preview using AR. Using the tracked points, the beauty products platform 120 and/or preview module 151 can modify the video segment to include an overlay of a 3D model of false eyelashes (e.g., virtual element) proximate the tracked points (e.g., corresponding to the eye area of the subject). The modified video segment including the overlay of the 3D model of false eyelashes proximate the tracked points can be provided to the client device 110 for presentation in real-time in an AR environment"); and
wherein generating the augmented reality presentation (See Lotti: Fig. 3, and Col. 21 Lines 34-56, "FIG. 3 illustrates a flow diagram of an example method 300 for using augmented reality to preview artificial lash extensions, in accordance with some embodiments. Method 300 may be performed by processing logic that may comprise hardware (e.g., circuitry, dedicated logic, etc.), computer-readable instructions such as software or firmware (e.g., run on a general-purpose computing system or a dedicated machine), or a combination thereof. Method 300 may also be associated with sets of instructions stored on a non-transitory computer-readable medium (e.g., magnetic or optical disk, etc.). The instructions, when executed by a processing device, may cause the processing device to perform operations of method 300. In an embodiment, the operations of method 300 are performed by computing system 1200 of FIG. 12. In some embodiments, the operations of method 300 are performed by preview module 151 of FIG. lA-1B. In an embodiment, operations of a particular method depicted in FIG. 3 can be performed simultaneously or in different orders than depicted. Various embodiments may include additional operations not depicted in FIG. 3 or a subset of operations depicted in FIG. 3. In some embodiments, method 300 can be performed with the same, different, fewer, or more operations in the same or different order") includes:
adjusting the three-dimensional model of the eyelash extension based on the adjusted eyelash extension template (See Lotti: Fig. 9, and Col. 42 Lines 54-67 ~ Col. 43 Lines 1-17, "In an embodiment, 3D model 900 may be a morphological model. A morphological model can represent the shape and structure of objects (e.g., human faces) using morphological data. In some embodiments, morphological data can describe the form and structural relationships between geometry (e.g., vertices, lines, planes and/or landmarks) of the model and enables manipulation of the geometry based on those relationships. In some embodiments, a morphological model may include a template model (e.g., 3D template model) of a human face. The template model may be initialized with template 3D model values (e.g., template landmark data) reflecting average values (e.g., average positions, sizes, colors, etc.) for an object, such as a human face. The template 3D model values may be derived from a representative collection of objects, such as human faces or features thereof. In some embodiments, the template model can be used as a reference model that can be compared to values representing a subject's unique face. In some embodiments, the comparison can generate difference information (e.g., metric) reflecting differences (e.g., deltas or deviations) between the template 3D model values, and in particular the template landmark data, and values representing corresponding points and/or facial features of the subject's face. The difference information can be stored as part of 3D landmark data 812. To generate the 3D model of the subject's face, conversion system 820 may adjust the template model based on the difference information corresponding to a particular subject, which can contribute to computational efficiency in generating a 3D model. In some embodiments, a morphological model can be used with a PCA model to generate a 3D model, as described further below"); and
rendering the adjusted three-dimensional model of the eyelash extension based on the one or more extension attachment points (See Lotti: Fig. 3, and Col. 24 Lines 25-42, "At block 306, the processing logic can receive an indication of a user selection of the false eyelashes among a set of false eyelashes. For example, a user can select set of false eyelashes (e.g., artificial lash extensions) associated with specific lash configuration information that can identify one or more a style, length, color, placement, order, etc. In some embodiments, a user can select various artificial lash extensions or sets of artificial lash extensions and preview the artificial lash extensions or sets of artificial lash extensions. The preview can include showing the artificial lash extensions or sets of artificial lash extensions as virtual elements applied to a correct location in video frames of the video segment at the underside of natural eyelashes of the user. In some embodiments, by realistically rendering the artificial lash extensions as an overlay on video frames representing the user (e.g., still image or frames of the video stream/video segment) the artificial lash extensions as applied to the user can be virtually previewed").
Regarding claim 13, Fu, Lotti and Toyota teach all the features with respect to claim 8 as outlined above. Further, Fu teaches that the non-transitory computer-readable medium of claim 8, wherein the actions further comprise:
presenting, by the computing system, one or more recommended eyelash extension templates based on the detected eye landmarks (See Fu: Fig. 11, and [0207], "In FIG. 11, the makeup recommendation system generates personalized step-by-step makeup instructions using real products in the database. The trained models for different makeup styles 4040 may be taken from the deep learning system 4000 and annotation system 5000 which can be input into the makeup recommendation system 6000 to provide a personalized makeup recommendation 7050, and also optionally a virtual makeup tutorial may be provided as described below. The make-up recommendation 7050 can be derived from a makeup recommender 7020 from the trained system and models such as trained models 4040, although a separate trained model may be created solely for use with a recommendation system. Product matching 7030 can also be used using a makeup product database, which may be the same or different from the makeup database 7045 (as shown in FIG. 1, it is the same database). The makeup recommender and/or product matching can result in the personalized makeup recommendation 7050. Virtual tutorials may also be generated using segmented video pathways or taking information from product searching and identification using a trained products classifier from a beauty products database as discussed below").
Regarding claim 14, Fu, Lotti and Toyota teach all the features with respect to claim 8 as outlined above. Further, Fu teaches that the non-transitory computer-readable medium of claim 8, wherein the actions further comprise:
transmitting, by the computing system, instructions to an eyelash extension creation system to cause the eyelash extension creation system to apply the eyelash extension to the subject based on the adjusted eyelash extension template (See Fu: Fig. 38, and [0279], "It should also be noted that implementations of the systems and methods can be provided as one or more computer-readable programs embodied on or in one or more articles of manufacture. The program instructions can be encoded on an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus"; and Fig. 11, and [0207], "In FIG. 11, the makeup recommendation system generates personalized step-by-step makeup instructions using real products in the database. The trained models for different makeup styles 4040 may be taken from the deep learning system 4000 and annotation system 5000 which can be input into the makeup recommendation system 6000 to provide a personalized makeup recommendation 7050, and also optionally a virtual makeup tutorial may be provided as described below. The make-up recommendation 7050 can be derived from a makeup recommender 7020 from the trained system and models such as trained models 4040, although a separate trained model may be created solely for use with a recommendation system. Product matching 7030 can also be used using a makeup product database, which may be the same or different from the makeup database 7045 (as shown in FIG. 1, it is the same database). The makeup recommender and/or product matching can result in the personalized makeup recommendation 7050. Virtual tutorials may also be generated using segmented video pathways or taking information from product searching and identification using a trained products classifier from a beauty products database as discussed below").
Regarding claim 15, Fu, Lotti and Toyota teach all the features with respect to claim 1 as outlined above. Further, Fu, Lotti and Toyota teach that a computing system (See Fu: Fig. 38, and [0267], "For the various systems and methods herein, a general system structure as shown in FIG. 38 may be used and methods combined in various ways (such as in FIG. 37) or with other techniques and improvements described above. As shown in FIG. 38, a memory capability (cloud- or hardware server-based) may be employed to store in a preferably secure location all various data and data sets described herein, including eye templates, ground truth data, product data, style and makeup look information, user information, training and learning information of all types and anything else the system requires for operation. The data should be accessible to programmable system software based on the algorithms and pseudo code noted herein, and executable on a processor in a controller herein"), comprising:
circuitry for receiving an image of a face of a subject (See Fu: Fig. 3, and [0113], "FIG. 3 shows a flow diagram of a method for virtually removing makeup in accordance with an embodiment of the present disclosure, generally referred to as embodiment 1000. Referring to FIG. 3, in Step 1010, an input is acquired by a user. The user input can be any facial image as that term is defined herein, including a single image, a repeat photo, a single video frame or a video having multiple frames, captured by the user using a mobile device or uploaded to the system of the present disclosure. Once acquired, the facial image is detected or identified from the user input. It is preferred that in the removal method, only one image is input, and that the face is detected once the image is uploaded, however, the invention is not limited to only one image input");
circuitry for detecting one or more eye landmarks within the image (See Fu: Fig. 3, and [0114], "Upon detection of the face of the image, in Step 1020, the facial landmarks are located using the input image. Landmarks can be preset and selected such as top of the chin, outside edge of each eye, inner edge of each eyebrow, and the like. Such landmarks are common to all faces and so are detected and evaluated using precise localization of their fiducial points (e.g. nose tip, mouth and eye corners) in color images of face foregrounds");
circuitry for adjusting an eyelash extension template based on the one or more eye landmarks to create an adjusted eyelash extension template (See Lotti: Fig. 9, and Col. 42 Lines 54-67 ~ Col. 43 Lines 1-17, "In an embodiment, 3D model 900 may be a morphological model. A morphological model can represent the shape and structure of objects (e.g., human faces) using morphological data. In some embodiments, morphological data can describe the form and structural relationships between geometry (e.g., vertices, lines, planes and/or landmarks) of the model and enables manipulation of the geometry based on those relationships. In some embodiments, a morphological model may include a template model (e.g., 3D template model) of a human face. The template model may be initialized with template 3D model values (e.g., template landmark data) reflecting average values (e.g., average positions, sizes, colors, etc.) for an object, such as a human face. The template 3D model values may be derived from a representative collection of objects, such as human faces or features thereof. In some embodiments, the template model can be used as a reference model that can be compared to values representing a subject's unique face. In some embodiments, the comparison can generate difference information (e.g., metric) reflecting differences (e.g., deltas or deviations) between the template 3D model values, and in particular the template landmark data, and values representing corresponding points and/or facial features of the subject's face. The difference information can be stored as part of 3D landmark data 812. To generate the 3D model of the subject's face, conversion system 820 may adjust the template model based on the difference information corresponding to a particular subject, which can contribute to computational efficiency in generating a 3D model. In some embodiments, a morphological model can be used with a PCA model to generate a 3D model, as described further below"); and
circuitry for generating an augmented reality presentation that includes the image of the face of the subject and the adjusted eyelash extension template (See Fu: Fig. 7, and [0156], "In each instance, the method after such histogram matching for a first effect, e.g., a level of gloss or shiny in steps 2025a and 2025b, the method provides a first image having the output effect. If multiple output effects are desired, steps 2020 and a further one of the other parallel steps 2030a and 2030b to provide shimmer and/or a natural effect in 2040a and 2040b or combinations thereof (or additional effects as desired as would be understood by one skilled in the art based on this disclosure) can be repeated to provide one or more additional images, each of which has the related output effect as desired. The first image having the first output effect and/or the additional images with their respective output effects are combined and blended with the original facial image of the user in step 2040 to create a resultant image in step 2050 having each of the output effects combined on the facial image of the user"; and [0010], "Further, while facial landmarks detection presents many potential attractive applications in augmented reality, virtual reality, human-computer interaction, and so on, and there are now applications that let people wear virtual make-up and recognize the faces using certain end points as facial landmarks, there are still issues with such developing technology from an accuracy standpoint. For example, when using these techniques there are always two primary problems that severely influence performance of such an application in a video: shaking problems and lag problems");
circuitry for presenting an interface for receiving a modification to one or more attributes of the adjusted eyelash extension template, wherein the one or more attributes include one or more of a length of the adjusted eyelash extension template, a density of the adjusted eyelash extension template, a thickness of the adjusted eyelash extension template, or a curve of the adjusted eyelash extension template (See Lotti: Figs. 1-2, and Col. 7 Lines 58-67 ~ Col. 8 Lines 1-9, "In some embodiments, a client device, such as client device 110, can implement or include one or more applications, such as application 119 executed at client device 110. In some embodiments, application 119 can be used to communicate (e.g., send and receive information) with beauty products platform 120 beauty products platform 120. In some embodiments, application 119 can implement user interfaces (Uls) (e.g., graphical user interfaces (GUls)), such as UI 112 that may be webpages rendered by a web browser and displayed on the client device 110 in a web browser window. In another embodiment, the Uls 112 of client application 119 may be included in a stand-alone application downloaded to the client device 110 and natively running on the client device 110 (also referred to as a "native application" or "native client application" herein). In some embodiments, preview module 151 can be implemented as part of application 119. In other embodiments, preview module 151 can be separate from application 119 and application 119 can interface with preview module 151"; Col. 19 Lines 1-21, "Lash configuration information (also referred to as "lash map" herein) can refer to information related to the selection of artificial lash extensions and/or the application of artificial lash extensions at the eye area of a user. In some embodiments, lash configuration information can identify the particular artificial lash extensions of a set of lash extensions (e.g., length, style, and/or color), a location at the underside of the natural lashes at which each particular artificial lash extension of the set of artificial lash extensions is to be applied, and/or the order of each artificial lash extension in the set of artificial lash extensions. In some embodiments and as described further below, lash configuration information can include one or more of style information, length information, color information, placement information, or order information for an eye or pair of eyes of a user. An example of lash configuration information is illustrated in element 235. Although described with respect to artificial lash extensions for purposes of illustration, rather than limitation, lash configuration information can apply to false eyelashes, generally, in some embodiments": and Fig. 6, and Col. 36 Lines 46-67 ~ Col. 37 Lines 1-3, "In some embodiments, an artificial lash extension can include a base. For example, artificial lash extension 600 includes base 606. In some embodiments, artificial lash extension 700 may or may not (as illustrated) include a base similar to base 606 of artificial lash extension 600. The base can include a top side (e.g., facing out of the page and towards the reader), a bottom side, a back side, a front side, and two ends (e.g., two lateral sides). In some embodiments, one or more of the multiple artificial hairs of artificial lash extension protrude out the front side of the base. When arranged at the underside of a natural lash, the backside of the artificial lash extension can point towards the user's eye. The thickness (e.g., between the top side and bottom side of the base can be between approximately 0.05 millimeters (mm) and approximately 0.15 mm (e.g., 0.05 mm+/-0.01 mm). In some embodiments, the thickness of the base can be less than 0.05 mm. In some embodiments, the low profile of the base is designed to allow the artificial lash extension to be light weight to better adhere to the underside of the natural lash and prevent obstruction of a user's view. The low profile of the base can at least in part be attributed to an attachment operation that forms the base and/or attaches clusters of artificial hairs to the base. For example, the attachment operation can include an application of heat that, at least in part, creates a base with a low profile");
circuitry for adjusting the adjusted eyelash extension template based the received modification to the one or more attributes of the adjusted eyelash extension template to create a further adjusted eyelash extension template (See Toyota: Fig. 2, and [0043], “The form alteration unit 16 determines whether the form that is configured by connecting the tips of plural eyelashes such as the upper eyelashes from the inner corner of the eye to the outer corner of the eye generated by the eyelash generation unit 15 has been altered by the user. If the form has been altered, the form alteration unit 16 obtains information on the form altered by the user, and alters the eyelash length and the eyelash direction (angle on a horizontal plane) based on the information on the altered form. It is noted that the eyelash form and the manner of altering the eyelash form are described in detail below”; and [0038], “It is noted that setting information relating to the eyelash shape may include parameter information such as the eyelash thickness, the eyelash length, the curvature of the eyelashes when they are curled, the number of eyelash strands, and the eyelash form, for example”); and
circuitry for generating an augmented reality presentation that includes the image of the face of the subject and the further adjusted eyelash extension template (See Toyota: Fig. 2, and [0046], “The simulation image generation unit 18 generates an eye image using the eyelashes generated by the eyelash generation unit 15 or the form altered by the form alteration unit 16. Also, the simulation image generation unit 18 may generate the eye image using the curvature altered by the curvature alteration unit 17”).
Regarding claim 16, Fu, Lotti and Toyota teach all the features with respect to claim 15 as outlined above. Further, Fu teaches that the computing system of claim 15, wherein generating the augmented reality presentation includes:
cropping the image of the face of the subject to an eye region based on the one or more eye landmarks (See Fu: Figs. 21 and 28A-E, and [0174], "To create the eye shadow add-on, the eye region landmarks are extracted from a landmarks detector for the frame. The eye region is cropped by the interested landmarks from 37 to 42 and 43 to 48. 100 points are generated b linear interpolation around the eye region from detected annotated landmarks (the landmarks may be annotated using an annotation system as described herein). From this, 100 points are generated around the eye shadow template based on the manually annotated landmarks. The template is applied to the frame by forward warping the 100 points from frame to template. There are many available warping algorithms such as forward warping, inverse warping and similarity transformation, affine transformation and the like. In the preferred embodiment, forward warping with affine transformation was adopted. Following this, the eye region image is cropped back into the original frame"); and
detecting one or more extension attachment points based on the cropped image of the eye region (See Fu: Figs. 21 and 28A-E, and [0174], "To create the eye shadow add-on, the eye region landmarks are extracted from a landmarks detector for the frame. The eye region is cropped by the interested landmarks from 37 to 42 and 43 to 48. 100 points are generated b linear interpolation around the eye region from detected annotated landmarks (the landmarks may be annotated using an annotation system as described herein). From this, 100 points are generated around the eye shadow template based on the manually annotated landmarks. The template is applied to the frame by forward warping the 100 points from frame to template. There are many available warping algorithms such as forward warping, inverse warping and similarity transformation, affine transformation and the like. In the preferred embodiment, forward warping with affine transformation was adopted. Following this, the eye region image is cropped back into the original frame". Note that after cropping, more points are added, which is mapped to "adjusting").
Regarding claim 17, Fu, Lotti and Toyota teach all the features with respect to claim 16 as outlined above. Further, Fu, Lotti and Toyota teach that the computing system of claim 16, wherein the eyelash extension template includes a two-dimensional image of the eyelash extension (See Fu: Figs. 28A-E, and [0173], "These effects are created using the following method. Eye templates are created as shown in FIGS. 28a-28e, wherein each template is respectively, an eye shadow template 438 (FIG. 28a), an eye middle template 440 (FIG. 28b), an eye corner template 442 (FIG. 28c), an eye tail template 444 (FIG. 28d) and an eye lash template 446 (FIG. 28e). Each such template is created by manually labeling landmarks using points on the templates according to a landmark protocol. An Example is shown in FIG. 29 wherein points identified as points 448 are applied to eye shadow template 438. The landmark locations of the points 448 are saved as a text file"); and
wherein generating the augmented reality presentation (See Fu: Fig. 3, and [0097], "Virtual Facial Makeup Simulation For Augmented Personalized Tutorials") includes:
adjusting the two-dimensional image of the eyelash extension based on the adjusted eyelash extension template (See Lotti: Fig. 9, and Col. 42 Lines 54-67 ~ Col. 43 Lines 1-17, "In an embodiment, 3D model 900 may be a morphological model. A morphological model can represent the shape and structure of objects (e.g., human faces) using morphological data. In some embodiments, morphological data can describe the form and structural relationships between geometry (e.g., vertices, lines, planes and/or landmarks) of the model and enables manipulation of the geometry based on those relationships. In some embodiments, a morphological model may include a template model (e.g., 3D template model) of a human face. The template model may be initialized with template 3D model values (e.g., template landmark data) reflecting average values (e.g., average positions, sizes, colors, etc.) for an object, such as a human face. The template 3D model values may be derived from a representative collection of objects, such as human faces or features thereof. In some embodiments, the template model can be used as a reference model that can be compared to values representing a subject's unique face. In some embodiments, the comparison can generate difference information (e.g., metric) reflecting differences (e.g., deltas or deviations) between the template 3D model values, and in particular the template landmark data, and values representing corresponding points and/or facial features of the subject's face. The difference information can be stored as part of 3D landmark data 812. To generate the 3D model of the subject's face, conversion system 820 may adjust the template model based on the difference information corresponding to a particular subject, which can contribute to computational efficiency in generating a 3D model. In some embodiments, a morphological model can be used with a PCA model to generate a 3D model, as described further below"); and
rendering the adjusted two-dimensional image of the eyelash extension based on the one or more extension attachment points (See Fu: Fig. 7, and [0156], "In each instance, the method after such histogram matching for a first effect, e.g., a level of gloss or shiny in steps 2025a and 2025b, the method provides a first image having the output effect. If multiple output effects are desired, steps 2020 and a further one of the other parallel steps 2030a and 2030b to provide shimmer and/or a natural effect in 2040a and 2040b or combinations thereof (or additional effects as desired as would be understood by one skilled in the art based on this disclosure) can be repeated to provide one or more additional images, each of which has the related output effect as desired. The first image having the first output effect and/or the additional images with their respective output effects are combined and blended with the original facial image of the user in step 2040 to create a resultant image in step 2050 having each of the output effects combined on the facial image of the user"; and [0010], "Further, while facial landmarks detection presents many potential attractive applications in augmented reality, virtual reality, human-computer interaction, and so on, and there are now applications that let people wear virtual make-up and recognize the faces using certain end points as facial landmarks, there are still issues with such developing technology from an accuracy standpoint. For example, when using these techniques there are always two primary problems that severely influence performance of such an application in a video: shaking problems and lag problems").
Regarding claim 18, Fu, Lotti and Toyota teach all the features with respect to claim 16 as outlined above. Further, Lotti teaches that the computing system of claim 16, wherein the eyelash extension template includes a three-dimensional model of the eyelash extension (See Lotti: Fig. 1, and Col. 10 Lines 10-32, "In some embodiments, beauty products platform 120 can implement preview module 151. In some embodiments, lash preview module 151 can implement one or more features, operations and/or embodiments as described herein. For example, in some embodiments, a video segment of a video stream having a representation of the subject's face can be received by the beauty products platform 120 and/or preview module 151. A computer vision operation can be performed on the video segment to track one or more points (e.g., two-dimensional (2D) points) of the video segment across multiple video frames. In some embodiments, beauty products platform 120 and/or preview module 151 can receive an indication of a user selection of false eyelashes among a set of false eyelashes for preview using AR. Using the tracked points, the beauty products platform 120 and/or preview module 151 can modify the video segment to include an overlay of a 3D model of false eyelashes (e.g., virtual element) proximate the tracked points (e.g., corresponding to the eye area of the subject). The modified video segment including the overlay of the 3D model of false eyelashes proximate the tracked points can be provided to the client device 110 for presentation in real-time in an AR environment"); and
wherein generating the augmented reality presentation (See Lotti: Fig. 3, and Col. 21 Lines 34-56, "FIG. 3 illustrates a flow diagram of an example method 300 for using augmented reality to preview artificial lash extensions, in accordance with some embodiments. Method 300 may be performed by processing logic that may comprise hardware (e.g., circuitry, dedicated logic, etc.), computer-readable instructions such as software or firmware (e.g., run on a general-purpose computing system or a dedicated machine), or a combination thereof. Method 300 may also be associated with sets of instructions stored on a non-transitory computer-readable medium (e.g., magnetic or optical disk, etc.). The instructions, when executed by a processing device, may cause the processing device to perform operations of method 300. In an embodiment, the operations of method 300 are performed by computing system 1200 of FIG. 12. In some embodiments, the operations of method 300 are performed by preview module 151 of FIG. lA-1B. In an embodiment, operations of a particular method depicted in FIG. 3 can be performed simultaneously or in different orders than depicted. Various embodiments may include additional operations not depicted in FIG. 3 or a subset of operations depicted in FIG. 3. In some embodiments, method 300 can be performed with the same, different, fewer, or more operations in the same or different order") includes:
adjusting the three-dimensional model of the eyelash extension based on the adjusted eyelash extension template (See Lotti: Fig. 9, and Col. 42 Lines 54-67 ~ Col. 43 Lines 1-17, "In an embodiment, 3D model 900 may be a morphological model. A morphological model can represent the shape and structure of objects (e.g., human faces) using morphological data. In some embodiments, morphological data can describe the form and structural relationships between geometry (e.g., vertices, lines, planes and/or landmarks) of the model and enables manipulation of the geometry based on those relationships. In some embodiments, a morphological model may include a template model (e.g., 3D template model) of a human face. The template model may be initialized with template 3D model values (e.g., template landmark data) reflecting average values (e.g., average positions, sizes, colors, etc.) for an object, such as a human face. The template 3D model values may be derived from a representative collection of objects, such as human faces or features thereof. In some embodiments, the template model can be used as a reference model that can be compared to values representing a subject's unique face. In some embodiments, the comparison can generate difference information (e.g., metric) reflecting differences (e.g., deltas or deviations) between the template 3D model values, and in particular the template landmark data, and values representing corresponding points and/or facial features of the subject's face. The difference information can be stored as part of 3D landmark data 812. To generate the 3D model of the subject's face, conversion system 820 may adjust the template model based on the difference information corresponding to a particular subject, which can contribute to computational efficiency in generating a 3D model. In some embodiments, a morphological model can be used with a PCA model to generate a 3D model, as described further below"); and
rendering the adjusted three-dimensional model of the eyelash extension based on the one or more extension attachment points (See Lotti: Fig. 3, and Col. 24 Lines 25-42, "At block 306, the processing logic can receive an indication of a user selection of the false eyelashes among a set of false eyelashes. For example, a user can select set of false eyelashes (e.g., artificial lash extensions) associated with specific lash configuration information that can identify one or more a style, length, color, placement, order, etc. In some embodiments, a user can select various artificial lash extensions or sets of artificial lash extensions and preview the artificial lash extensions or sets of artificial lash extensions. The preview can include showing the artificial lash extensions or sets of artificial lash extensions as virtual elements applied to a correct location in video frames of the video segment at the underside of natural eyelashes of the user. In some embodiments, by realistically rendering the artificial lash extensions as an overlay on video frames representing the user (e.g., still image or frames of the video stream/video segment) the artificial lash extensions as applied to the user can be virtually previewed").
Regarding claim 20, Fu, Lotti and Toyota teach all the features with respect to claim 15 as outlined above. Further, Fu teaches that the computing system of claim 15, further comprising: circuitry for transmitting instructions to an eyelash extension creation system to cause the eyelash extension creation system to apply the eyelash extension to the subject based on the adjusted eyelash extension template (See Fu: Fig. 38, and [0279], "It should also be noted that implementations of the systems and methods can be provided as one or more computer-readable programs embodied on or in one or more articles of manufacture. The program instructions can be encoded on an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus"; and Fig. 11, and [0207], "In FIG. 11, the makeup recommendation system generates personalized step-by-step makeup instructions using real products in the database. The trained models for different makeup styles 4040 may be taken from the deep learning system 4000 and annotation system 5000 which can be input into the makeup recommendation system 6000 to provide a personalized makeup recommendation 7050, and also optionally a virtual makeup tutorial may be provided as described below. The make-up recommendation 7050 can be derived from a makeup recommender 7020 from the trained system and models such as trained models 4040, although a separate trained model may be created solely for use with a recommendation system. Product matching 7030 can also be used using a makeup product database, which may be the same or different from the makeup database 7045 (as shown in FIG. 1, it is the same database). The makeup recommender and/or product matching can result in the personalized makeup recommendation 7050. Virtual tutorials may also be generated using segmented video pathways or taking information from product searching and identification using a trained products classifier from a beauty products database as discussed below").
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to GORDON G LIU whose telephone number is (571)270-0382. The examiner can normally be reached Monday - Friday 8:00-5:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Devona E Faulk can be reached at 571-272-7515. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/GORDON G LIU/Primary Examiner, Art Unit 2618
