DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
Applicant’s amendments and arguments received on 06/09/2026 have been considered. Claim 1-9 are pending in the application. Applicant’s amendments to the specifications and claims have overcome each and every objection previously set forth in the Non-Final Office Action mailed on 3/09/2026.
Response to Arguments
Applicant’s arguments with respect to claim(s) 1-9 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument. Please See Section 35 U.S.C 103 Rejection below.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1, 4, and 7-9 is/are rejected under 35 U.S.C. 103 as being unpatentable over Hu et al (US 20200401790 A1), Li et al (US 20200342209 A1), and Kim (US 10223767 B2), hereinafter Hu, Li, and Kim respectively.
Regarding claim 1, Hu teaches a computer implemented method comprising executing by one or more processors (processor, paragraph 0007) the steps of: rendering an output image derived from the input image using a rendering pipeline (“obtaining an adjustment amplitude by which the adjuster performs dynamic adjustment on the target face portion, and displaying a change effect of the target face portion based on the dynamic adjustment in a display interface” – Abstract. [NOTE: output image is displayed after performing adjustment on target (input) image). Fig 8 also shows the rendering pipeline]), the output image derived by applying one or more shape changes to a particular facial feature (“For example, a deformation operation may be performed on a mouth width, an eye size, or a face size” – Par 40, Lines 7-8), the one or more shape changes determined by: mapping a grid of spaced grid points to pixels of the particular facial feature and any associated facial features (“performing interpolation processing on pixels in the deformed grid region, to obtain pixel values corresponding to the deformed grid region; and constructing a deformed face portion according to the deformed grid region, and performing pixel rendering on the deformed face portion according to the pixel values, to obtain the deformed face image.” – Par 95); and warping at least some of the spaced grid points using respective shape changing functions, the warping changing location of at least some of the spaced grid points and changing locations of the face points of the particular facial feature (“adjustment positions of vertices or a plurality of pixel points of the target grid region included in each deformation unit are separately calculated according to the deformation type and the deformation intensity.” – Par 91, Lines 42-25. [NOTE: Hu discloses the changing of positions of vertices on the grid based on the face deformation changes. Face deformation can be understood by one of ordinary skill in the art as face warping. Rendering of pixel points of the face image is performed by interpolation processing to get the pixel values corresponding to the deformed grid, Par 96 ]) and wherein the rendering determines output pixels for the particular facial feature and any associated facial feature for the output image in response to the warping (“Then, the deformed face portion is constructed according to the deformed grid region, and the pixel rendering is performed on the deformed face portion according to the pixel values, to obtain the deformed face image.” – Par 96, Lines 9-12. [NOTE: pixels of the facial features are rendered to output an image that reflects the face warping]).
Hu does not teach processing an input image to localize facial features using a face tracking engine having one or more deep neural networks to respectively produce face points defining a contour for each of the facial features localized, wherein each of the face points is placed at a particular location on the contour of the facial feature it is defining and the one or more shape changes determined by: performing at least one warping function that takes at least some of the spaced grid points and the face points produced by the one or more deep neural networks as input. However, Li teaches processing an input image to localize facial features using a face tracking engine having one or more deep neural networks to respectively produce face points defining a contour for each of the facial features localized (“configure the computing device to process an image to determine respective locations of each of a plurality of landmarks by: processing the image using a Convolutional Neural Network (CNN)” – Par 6, Lines 4-7. [NOTE: The abstract specifically states the methods for facial landmarks with CNNs]). It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to modify Hu to incorporate Li’s teaching of using a CNN to produce face points for each facial feature. It is common in the art to use deep neural network face tracking engines to produce facial landmarks because of accuracy of detecting features such as eyes, noses, and mouths. Using the CNN as described by Li would have the predictable result of generating facial points that best depict where each complex face feature is on the input face image.
Hu in view of Li still does not teach wherein each of the face points is placed at a particular location on the contour of the facial feature it is defining; and performing at least one warping function that takes at least some of the spaced grid points and the face points produced by the one or more deep neural networks as input. However, Kim teaches wherein each of the face points is placed at a particular location on the contour of the facial feature it is defining;
PNG
media_image1.png
625
726
media_image1.png
Greyscale
“For instance, the additional landmark points can be added to areas surrounding a particular facial feature. In one specific implementation, the face mesh generator 114 is implemented to add additional landmark points corresponding to the eye by performing cubic interpolation on the landmark points” – Col 7, Lines 31-36 and Fig 2
NOTE: Kim discloses generating face mesh that includes landmark points to be used for face warping, see abstract. Additional landmark points can be added to any particular facial feature. One example as disclosed includes adding landmark points to define the eyes. This functionally corresponds to face points that are placed at a particular location on the contour of the facial feature it defines. After the combination, the method of placing the face landmarks on a particular location of the facial feature as taught by Kim can be performed using the generated face point using a convolutional network as taught by Li. This modification is then added to Hu’s system for warping facial features and rendering an output image of the modified face.
and performing at least one warping function that takes at least some of the spaced grid points and the face points produced by the one or more deep neural networks as input.
“The image processing application can generate a face mesh that includes landmark points, and then construct the deformation fields on the face mesh, where the deformation fields are defined by warpable elements (e.g., warpable vectors, warpable lines, warpable landmark points and/or warpable areas) formed from the landmark points.” – Col 3, Lines 1-6
NOTE: Kim teaches the construction of deformation fields defined by the face landmarks used for warping the face. Since the face landmarks are generated alongside the face mesh which makes up the spaced grid points, the warping of the face points would also include the warping of the spaced grid points. Therefore, the warping must take at least some of the spaced grid points and face points as input. Kim does not teach that the landmarks are generated by deep neural networks. After the combination, the method of generating facial landmarks with a convolutional network as taught by Li can be used to modify the face mesh taught by Kim so that the facial landmarks can be placed on the contour of a facial feature it is defining. This modification is further added to Hu’s method of warping facial features and rendering an output image derived by applying shape changes to a particular facial feature.
It would have been obvious to one of ordinary skill in the art before effective filing date of the present invention to modify Hu by incorporating the teachings of Kim to have the generated face points placed at particular locations on the contour of the facial feature it is defining and performing warping by taking the face points and spaced grid points as input. One would be motivated to make this combination to produce more accurate deformations if it has access to facial landmarks that define the facial features.
Regarding claim 9, the claim describes a system performs a function with the steps of claim 1. Therefore, system claim 9 corresponds to the method disclosed in claim 1 and is rejected for the same reasons obviousness as used above.
Regarding claim 4, Hu in view of Li and Kim teach the method of claim 1. Hu further teaches wherein the method comprises providing a user interface to receive input to define shape parameters for the one or more shape changes and wherein the rendering is responsive to the user input (“In the deformation operation interface, a user may select a to-be-deformed target face portion, an operation type, and an adjustment parameter. The deformation operation interface includes the portion selection interface and the type setting interface. The portion selection interface may be used for receiving the operation instruction, and the type setting interface may be used for selecting the operation type, and the like.” – Par 64, Lines 7-15. [NOTE: Hu discloses that the user can make selections for any type of deformation operation to which the deformation unit will process and apply to the initial input image and display on the interface to the user.]). It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to further incorporate the teachings of Hu to provide the user an interface to input deformation operations and display the changes. User Interfaces are common tools used for their intuitive nature. Using a UI to receive input from the user for their selection of face deformations and updating the image to reflect the modifications simplifies the process for the user.
Regarding claim 7, Hu in view of Li and Kim teach the method of claim 1. Hu further teaches wherein the particular facial feature defines a first facial feature and the step of rendering is repeated in respect of a second facial feature to product an output image having at least two shape changed facial features (“As shown in FIG. 10a, when the user selects a face and a corresponding whole of the face, the operation instruction may be generated, or as shown in FIG. 10b, when the user selects eyes and a corresponding eye height of the eyes, the operation instruction may be generated. ” – Par 109, Lines 6-10, Fig 10a-10b. [NOTE: Fig 10a-10b shows a display of a face with both a slimmer chin and bigger eyes]). It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to further incorporate the teachings of Hu to repeat the rendering step so that a second facial feature modification can be applied alongside the first facial feature modification. By allowing for multiple facial feature modifications to be displayed simultaneously, the user is given more customization to the face warping of the initial input image. It would also allow the user to make adjustments and see how each facial feature modification look together rather than separately.
Regarding claim 8, Hu in view of Li and Kim teaches the method of claim 1. Hu further teaches wherein the one or more shape changes applied to the particular facial feature comprise any one of a brow shaping, a nose shaping, a face contour change, an eye or eyelid change, or a lip change such as lip plumping (“For example, the operation types of the deformation corresponding to the face portion may include adjusting an angle, an eyebrow distance, and a position of the eyebrows, adjusting a size, a width, a thickness, and a position of the mouth, adjusting a size, a wing, a bridge, a tip, and a position of the nose, adjusting a size, an eye height, an eye distance, an inclination, an eye brightening degree, and eye bag removing of the eyes, and adjusting an overall contour, cheeks, a chin, and a forehead of the face” – Par 51, Lines 10-18). It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to further incorporate the teachings of Hu to include various types of facial feature shape changes such has eyebrow sizing, mouth width, nose positioning, eye height, etc. Adding various facial shape change possibilities allow for more creative control for the user to be able to modify and warp the initial image to their preferences.
Claim(s) 2 and 3 is/are rejected under 35 U.S.C. 103 as being unpatentable over Hu, Li, Kim and Sartori et al (US 11315173 B2), hereinafter Sartori.
Regarding claim 2, Hu in view of Li and Kim teaches the method of claim 1. Hu in view of Li does not teach wherein the face tracking engine and rendering pipeline are components of a VTO application for simulating the effects of a makeup product applied to facial features. However, Sartori teaches wherein the face tracking engine and rendering pipeline are components of a VTO application for simulating the effects of a makeup product applied to facial features (“Described herein are techniques for generating, calibrating, and applying virtual makeup products that simulate, in a photo realistic way, the application of real-world makeup products.”, Col 3, Lines 5-8. [NOTE: Sartori teaches a “virtual try-on” (VTO) application. To apply the makeup effects onto the input image, a rendering pipeline must exist to modify the image. After the combination, the face tracking engine as taught by Li and the VTO application as taught by Sartori can be added to Hu’s facial warping system so that makeup effects can be simulated on facial features that have been warped.]). It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to modify Hu by incorporating the teachings of Sartori to use a rendering pipeline of a VTO application to apply makeup effects to the face features after warping. Using the facial tracking engine to determine facial parts and a rendering pipeline for a VTO application will benefit the face warping system by allowing users to apply makeup to warped areas of the face. The VTO rendering pipeline could apply the makeup effect whether it is through eyeshadow, lipstick, blushes, etc. so that a user can simulate makeup effects without the need to physically put the makeup on.
Regarding claim 3, Hu in view of Li, Kim and Sartori teach the method of claim 2. Sartori further teaches wherein the rendering pipeline renders a makeup effect to the particular facial feature as shape changed such that the output image comprises the particular facial feature as shaped changed and with the makeup effect (“Process 400 is then performed to apply the virtual lipstick product to the base image resulting in a composite image including the base image and one or more makeup images that together simulate the application of the corresponding real-world makeup product. The user then selects a virtual eyeshadow product. Process 400 is then reapplied to generate one or more additional makeup images that simulate the application of the corresponding real-world eyeshadow product.” – Col 13, Lines 48-56. [NOTE: Sartori discloses that a user can select multiple makeup effects which is then combined together to create a composite image. This blending of makeup effects shows the rendering of a new image to display all of the makeup affects chosen. After the combination, the CNN face tracking engine taught by Li and the rendering pipeline for a VTO application as taught by Sartori can be used with the face warping system as taught by Hu to apply makeup effects to the warped face image to teach this claim element.]). It would have been obvious to one of ordinary skill in the art to modify Hu by further incorporating the teachings of Sartori to have the rendering pipeline apply makeup effects to warped facial features. This adds more utility features to the VTO application by giving more customizability for the user as they would not only be able to simulate makeup effects on their face, but also simulate makeup on face modifications based on their preferences.
Claim(s) 5 is/are rejected under 35 U.S.C. 103 as being unpatentable over Hu, Li, Kim and Kalarot et al (US 11907839 B2), hereinafter Kalarot.
Regarding claim 5, Hu in view of Li and Kim teaches the method of claim 4. However, Hu does not teach wherein the at least one warping function performs one or more of: curve matching to match a middle curve along a middle of the contour of the particular facial feature with a target curve defined by the shape parameters of the one or more shape changes, attenuating location changes to spaced grid points responsive to distance to the target curve; or point matching to match a discrete point of the contour of the particular facial feature to a target point defined by the shape parameters of the one or more shape changes; or area expansion or area compression to expand or compress an area along a face point curve or about a particular pixel, responsive to the shape parameters, the area expansion or area compression attenuated by an attenuation function responsive to distance from the face point curve or the particular pixel. However, Kalarot teaches point matching to match a discrete point of the contour of the particular facial feature to a target point defined by the shape parameters of the one or more shape changes (“The warper 204 receives an input image 106 with detected landmarks and a generated image 139 with detected landmarks from the landmark detector 202. The warper 204 warps one of the images to align the landmarks in the input image 106 with the landmarks in the generated image 139.” – Col 12, Lines 64-67. [NOTE: Kalarot teaches a point matching warp, but does not teach that is done on a user defined modified image based on the user’s input for defining the shape parameter. After the combination, the modified image as defined by the user as disclosed by Hu and the point matching to match landmarks as disclosed by Kalarot would then teach point matching to match a discrete point of the contour of the particular facial feature to a target point defined by the shape parameters of the one or more shape changes]). It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to modify Hu by incorporating the teachings of Kalarot to use point matching to match a point of a facial feature with a target point defined by shape changing parameters. With point matching, the discrete facial feature point can be aligned to the target point. This will allow user-specified shape parameters to produce consistent and accurate deformations.
Claim(s) 6 is/are rejected under 35 U.S.C. 103 as being unpatentable over Hu, Li, Kim and Liu et al (US 20210390789 A1), hereinafter Liu respectively.
Regarding claim 6, Hu in view of Li and Kim teaches the method of claim 1. However, Hu does not teach wherein determining the pixels comprises fitting triangles to vertices defined by the spaced grid points as warped and UV mapping the pixels of the particular facial feature and any associated feature onto the vertices. However, Kim teaches wherein determining the pixels comprises fitting triangles to vertices defined by the spaced grid points as warped (“The landmark points are representative of points, nodes, or markers that are positioned on a face and are combinable with other landmark points to form geometric shapes, such as triangles, that define the face mesh 116.” – Col 5 , Lines 54-58. [NOTE: Kim discloses the construction of triangles based on landmark points or vertices that define a face mesh. One of ordinary skill in the art could then use the same process for constructing triangles using face points with the spaced grid points taught by Hu.]). It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application modify Hu by incorporating the teachings of Kim to determine pixels by fitting triangles onto vertices defined by the space-grid points as warped. By fitting triangles to the vertices of the space grid, the warping can be done locally (in the area of interest such as the eye, nose, mouth etc.) without affecting other areas. This gives more control to the user by allowing them to modify certain features of their face without changing the entirety of it. Hu in view of Li and Kim still does not teach UV mapping the pixels of the particular facial feature and any associated feature onto the vertices. However, Liu further teaches UV mapping the pixels of the particular facial feature and any associated feature onto the vertices (“For example, the UV face position map can include a 2D texture map that maps points (e.g., pixels) in the 2D texture map to vertices and/or coordinates of a 3D representation (e.g., a 3D mesh, model, surface, geometry, etc.) of the face”- Par 62, Lines 11-15. [NOTE: Although Liu teaches the UV mapping of 2D texture to vertices in 3D meshes, the same process can be applied for 2D textures to 2D surfaces as discloses in the present application]). It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to modify Hu by incorporating the teachings of Liu to UV map the pixels of the facial feature onto the vertices. UV mapping is a common technique in the art for applying textures to 2D/3D shapes. By UV mapping the pixels of the facial feature onto the vertices, the makeup effects chosen by the user can accurately be applied to the correct areas even during facial deformation.
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DAVID V. NGUYEN whose telephone number is (571)272-6111. The examiner can normally be reached M-F 9:00-5:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, King Y Poon can be reached at 571-270-0728. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/DAVID VAN NGUYEN/Examiner, Art Unit 2617 /KING Y POON/Supervisory Patent Examiner, Art Unit 2617
Prosecution Insights