DETAILED ACTION
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries set forth in Graham v. John Deere Co., 383 U.S. 1, 148 USPQ 459 (1966), that are applied for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1 – 9, and 21 – 31 are rejected under 35 U.S.C. 103 as being unpatentable over Comploi et al. (Publication: US 2020/0312002 A1) in view of Cheng et al. (Publication: US 2020/0327309 A1) and Lee et al. (Publication: US 2014/0119642 A1).
Regarding claim 1, see rejection on claim 30.
Regarding claim 2, see rejection on claim 22.
Regarding claim 3, see rejection on claim 23.
Regarding claim 4, see rejection on claim 24.
Regarding claim 5, see rejection on claim 25.
Regarding claim 6, see rejection on claim 26.
Regarding claim 7, see rejection on claim 27.
Regarding claim 8, see rejection on claim 28.
Regarding claim 9, see rejection on claim 29.
Regarding claim 21, see rejection on claim 30.
Regarding claim 22, Comploi in view of Cheng, Lee disclose all the limitations of claim 21.
Cheng discloses wherein the target area comprises a forehead area ( [0052], [0054] - 466 of Fig. 8B “forehead”, removing hair and eyebrow features to show skin image based on Fig. 8A the capture user image. It expand because the features are remove and reveal the skim image.
PNG
media_image1.png
158
100
media_image1.png
Greyscale
).
Regarding claim 22, Comploi in view of Cheng, Lee disclose all the limitations of claim 22.
Cheng discloses wherein the forehead area is determined based on a key point of an eyebrow and a key point of a forehead contour in the first facial image ([0039] - the processor can analyze an image and determine based on color and/or location within the frame whether a particular pixel may be more likely to be a portion of the forehead or a portion of the eyebrow. In another example, an AI classifer can be used to conduct the image analysis and identify the facial features of users. To this end, the system may use algorithmic techniques, trained AI models, or the like, to determine the facial features. [0052] - Obstructions may be determined by using the eyes and mouth as landmarks and anything other than skin color detected above the eyes or around the mouth may be disregarded. In one example, the masks 320, 322 generated in operation 210 may be used to identify the location and shape of certain obstructions to allow their removable.).
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to modify Comploi in view of Cheng, Lee with wherein the forehead area is determined based on a key point of an eyebrow and a key point of a forehead contour in the first facial image by Cheng. The motivation for doing is to improve recognition as taught by Cheng.
Regarding claim 24, Comploi in view of Cheng, Lee disclose all the limitations of claim 21.
Cheng discloses wherein the target facial image is a mask of the target facial organ determined based on the target facial organ ([0256] - A mask m on facial may be obtained by Equation (13) as follows: m=abs(img.sub.1−img.sub.2)
[0258] The foreground part (pixels with value 1 or 255) of mask m may represent the major differences between the input image and the processed image, which may be the covering state of the second object. The corresponding part in the input image may be determined as the covering region, and the corresponding part in the processed image may be determined as the uncovering region.).
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to modify Comploi in view of Cheng, Lee with wherein the target facial image is a mask of the target facial organ determined based on the target facial organ by Cheng. The motivation for doing is to improve recognition as taught by Cheng.
Regarding claim 25, Comploi in view of Cheng, Lee disclose all the limitations of claim 24.
Cheng discloses wherein the mask of the target facial organ is determined based on a key point of the target facial organ in the first facial image ([0258] The foreground part (pixels with value 1 or 255) of mask m may represent the major differences between the input image and the processed image, which may be the covering state of the second object. The corresponding part in the input image may be determined as the covering region, and the corresponding part in the processed image may be determined as the uncovering region.).
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to modify Comploi in view of Cheng, Lee with wherein the target facial image is a mask of the target facial organ determined based on the target facial organ by Cheng. The motivation for doing is to improve recognition as taught by Cheng.
Regarding claim 26, Comploi in view of Cheng, Lee disclose all the limitations of claim 21 including skin image
Cheng discloses performing a mirror reflection process on the image in the target area ([0285] - A mirror mask G.sub.4 may be generated from mask G.sub.3 by turning G.sub.3 around its symmetric axis.) ; and
performing a stitching process on a reflected image obtained by the mirror reflection process and the image in the target area ([0285] - A mirror mask G.sub.4 may be generated from mask G.sub.3 by turning G.sub.3 around its symmetric axis. A matching may then be performed optionally between G.sub.3 and G.sub.4. For a pair of matched points p and p′ (p is from G.sub.3 and p′ is from G.sub.4), if I(p)−I(p′)≠0, a difference of the pixel value d may be determined from point p and the points around p. If d is within a predetermined range and I(p)−I(p′)<0, point p may be added into G.sub.3, “stitching” .).
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to modify Comploi in view of Cheng, Lee with performing a mirror reflection process on the image in the target area; and performing a stitching process on a reflected image obtained by the mirror reflection process and the image in the target area by Cheng. The motivation for doing is to improve recognition as taught by Cheng.
Regarding claim 27, Comploi in view of Cheng, Lee disclose all the limitations of claim 21 including skin image.
Cheng discloses performing a replication process on the image in the target area ([0285] - A mirror mask G.sub.4 may be generated from mask G.sub.3 by turning G.sub.3 around its symmetric axis.); and
performing a stitching processing on a plurality of replicated images obtained from the replication process ([0285] - A mirror mask G.sub.4 may be generated from mask G.sub.3 by turning G.sub.3 around its symmetric axis. A matching may then be performed optionally between G.sub.3 and G.sub.4. For a pair of matched points p and p′ (p is from G.sub.3 and p′ is from G.sub.4), if I(p)−I(p′)≠0, a difference of the pixel value d may be determined from point p and the points around p. If d is within a predetermined range and I(p)−I(p′)<0, point p may be added into G.sub.3, “stitching”)
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to modify Comploi in view of Cheng, Lee with performing a replication process on the image in the target area; and performing a stitching processing on a plurality of replicated images obtained from the replication process by Cheng. The motivation for doing is to improve recognition as taught by Cheng.
Regarding claim 28, see rejection on claim 31.
Regarding claim 29, Comploi in view of Cheng, Lee disclose all the limitations of claim 21 including smeared facial image.
Comploi discloses transferring a predetermined animation to the image to obtain a dynamic image ([0024] - Using the wrapped mesh, the 3D landmarks are measured and compared or classified relative to selected or stored avatar or design features, where the stored avatar features are designed to match an avatar design paradigm. In one example, a user nose width spanning across X number of mesh vertices or X inches may be compared against paradigm nose sizes, such as small, medium, and large, where each nose size has a vertices or inches interval (e.g., 0.3 to 0.6 inches is a small nose, 0.65-0.9 is a medium nose, etc.). Once the nose or other 3D feature is classified, the corresponding avatar 3D feature size and shape is selected based on the classification and applied to the 3D mesh. The 3D features are applied to the wrapped mesh and the user specific built 3D mesh is combined with the shape and color features classified and a complete user specific avatar can be generated.).
Regarding claim 30, Comploi discloses A non-transitory computer-readable storage medium, with a computer program stored thereon, the computer program being executable by a processor to implement a method comprising: obtaining a facial image to be processed ([0005] - a system for generating graphical representations of a person is disclosed. The system includes a camera for capturing images of the person and a computer in electronic communication with the camera. The computer includes a non-transitory memory component for storing a plurality of graphical features and a processor in electronic communication with the memory component. The processor is configured to perform operations .); and
performing a smearing process to a target facial organ in the facial image to be processed based on a pre-trained smearing model to obtain a smeared facial image corresponding to the facial image to be processed ( [0052] - Fig. 4, step 232 remove obstructions is based on the masks 320, 322 generated in operation 210 may be used to identify the location and shape of certain obstructions to allow their removable. a Boolean operation can be used to determine if there is 3D data that falls inside the mask or outside the mask.
[0041] - The feature masks 320, 322 are generated based on a perimeter shape as identified during the detection of the user features via trained AI models [0039] .),
wherein the smearing model is training based on a first facial image obtained without smearing the target facial organ and a second facial image obtained by smearing the target facial organ in the first facial image ([0029] - AI model is trained by images to estimate depth information.
[0054] After operation 232, operation 216 proceeds to step 234 and a user 3D mesh is generated. the processor 120 uses the landmark information and depth information detected in the user 3D information to generate a user mesh, e.g., a 3D geometric representation or point cloud corresponding to the user's features. FIGS. 8B and 8C illustrate examples of the initial user 3D mesh 464, 466.
[0053] FIGS. 8B and 8C illustrate user images with obstructions, facial hair and bangs, respectively, as compared to a first 3D shape generated “without” the obstructions removed and a second 3D shape generated with the obstructions removed.) and
changes pixel in the area based on a texture image ([0053] FIGS. 8B and 8C illustrate user images with obstructions, facial hair and bangs, respectively, as compared to a first 3D shape generated without the obstructions removed and a second 3D shape generated with the obstructions removed “changes pixel based on the comparison to a first 3D shape”.),
wherein the second facial image is generated based on a predetermined image generating model([0052] - Fig. 4, step 232 remove obstructions is based on the masks 320, 322 generated in operation 210 may be used to identify the location and shape of certain obstructions to allow their removable. a Boolean operation can be used to determine if there is 3D data that falls inside the mask or outside the mask “predetermined”. ),
the image generating model being trained based on a target texture image and a target facial image ([0057] - The factors for choosing a neural network (or a group of neural networks) may include feature(s) of object 130 (e.g., race, gender, age, facial expression, posture, type of object 136, or a combination thereof), properties of input image 135 (e.g., the quality, color of input image 135), and/or other factors including, for example, clothing, light conditions, or the like, or the combination thereof. For example, a neural network may be specifically trained to process a full-face color image including an expressionless male and to remove a pair of glasses.
[0240] To generate a training image, image database generator 1700 may recognize and locate certain part of the first object in image 1710. An image of the second object may be obtained or generated. The image of the second object may be merged into a copy of image 1710 at a location determined by one or more recognized parts of the first object. A training image (e.g., image 1721) may then be generated. In some embodiments, more than one images of the second object may be added into image 1710 to generate one training image. These images may include second objects of the same kind, (e.g., scars) or of different kinds (e.g. a pair of glass and eye shadow).).
wherein the target texture image is obtained by performing an expanding process on a skin image in a target area in the first facial image ([0052], [0054] - 466 of Fig. 8B “forehead”, removing hair and eyebrow features to show skin image based on Fig. 8A the capture user image. It “expands” because the features are remove and reveal the skim image.
[0053] FIGS. 8B and 8C illustrate user images with obstructions, facial hair and bangs, respectively, as compared to a first 3D shape generated without the obstructions removed and a second 3D shape generated with the obstructions removed “the target texture image is obtained by performing an expanding process”.
PNG
media_image1.png
158
100
media_image1.png
Greyscale
).
Comploi does not however Cheng discloses
wherein the model recognizes an area of pixels of the target facial organ in the facial image to be processed and changes characteristic of pixels in the area ([0011] In some embodiments, the locating the covering region may further include: determining, on the first image, a plurality of pixels, wherein the plurality of pixels are distributed on the covering region; locating a rough covering region basing on a sparse location; and refining the rough covering region, wherein the plurality of pixels are determined by an active shape model algorithm.
[0003] - then to remove the covering objects from the face .).
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to modify Comploi with wherein the model recognizes an area of pixels of the target facial organ in the facial image to be processed and changes characteristic of pixels in the area as taught by Cheng. The motivation for doing is to improve recognition as taught by Cheng.
Comploi in view of Cheng do not disclose; however, Lee discloses wherein the second image is generated bv inputting the first image and a mask of the target into the predetermined image generating model (As shown in Fig. 2 – received the detect face. Refine face and Hair Mask is generated based on the received detect face 201 and Face Mask 204 and Hair Mask 205, “bv inputting the first image and a mask of the target”.
[0050] At 204 of FIG. 2, an initial face mask is generated. The initial face mask may be found based upon shape models and/or color models. For example, a shape model may utilize the ASM control points to suggest the rough face area from the face box detected by the face detector. In the training phase, an average ASM shape model may be trained based on a set of training images. Then all the training images may be aligned by the average ASM model, and finally a generic face mask is generated by averaging all the registered target face masks in the training images. The meaning of each pixel in the generic face mask represents the likelihood probability of that pixel to be inside the face area. In the testing phase, this generic face mask is registered by ASM control points with the face found in the test image, and then the overlapped face area with high likelihood probability will be selected as the initial face mask, “predetermined image generating model”.).
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to modify Comploi in view Cheng with wherein the second image is generated bv inputting the first image and a mask of the target into the predetermined image generating model as taught by Lee. The motivation for doing is to incorporate the model thus to have automated abilities for creating presentation as taught by Cheng.
Regarding claim 31, Comploi in view of Cheng, Lee disclose all the limitations of claim 21.
Camploi discloses extracting a first organ image corresponding to the target facial organ in the facial image to be processed ([0022] - a hairline shape for the user is extracted from the user images and the shape is classified and matched to a selected paradigm shape, which is then used to build the user's avatar. the user's skin and eye color are extracted and classified within a paradigm scale to be matched to a paradigm selection of skin and eye color.);
adjusting a shape and/or a size of the target facial organ in the first organ image to obtain a second organ image ([0022] - a hairline shape for the user is extracted from the user images and the shape is classified and matched to a selected paradigm shape, which is then used to build the user's avatar. the user's skin and eye color are extracted and classified within a paradigm scale to be matched to a paradigm selection of skin and eye color.); and
adding the second organ image to the smeared facial image ([0065] – step 218, the processor 120 combines the selected facial features (e.g., facial hair, hairline, accessories, or the like), with the avatar colors (e.g., hair color, eye color, skin color), onto the avatar 3D facial shape to generate the actual avatar icon 492 (see FIG. 12).).
Response to Arguments
Claim Rejection Under 35 U.S.C. 103
Applicant asserts “Comploi does not disclose or suggest feature (4). Comploi's system is directed to generating a 3D avatar from user images. Comploi's AI model is trained to estimate depth information from facial images. See Comploi, ,r [0029]. The masks 320, 322 in Comploi are used to identify locations and shapes of obstructions for their removal. See Comploi, ,r [0052]. However, Comploi does not appear to disclose inputting a first facial image and a mask of a target facial organ into a predetennined image generating model to generate a second facial image. Comploi's obstruction removal process removes obstructions from 3D data using Boolean operations; it does not generate a second facial image by inputting the first facial image together with a mask into an image generating model. The concept of generating training data for a smearing model by inputting a facial image and an organ mask into an image generating model is entirely absent from Comploi. First, neither Comploi nor Cheng, taken individually or in combination, discloses or suggests the feature (4). As discussed in detail above, the claimed feature of inputting a first facial image and a mask of the target facial organ into a predetermined image generating model to generate a second facial image is not found in either reference. Even assuming, arguendo, that one of ordinary skill would be motivated to combine the references, the combination still would not arrive at the claimed invention because neither reference teaches or suggests the specific inputoutput relationship of the image generating model recited in feature (4).”
The argument has been fully considered and is persuasive. Therefore, the rejection has been withdrawn. However, upon further consideration, a new ground(s) of rejection is made in view of Lee reference.
Applicant asserts “Second, the technical problems addressed by the references differ fundamentally from the technical problem addressed by the present application. Comploi is directed to generating 3D avatar representations from user images, where the primary concern is estimating depth information and removing obstructions to produce a clean 3D mesh. Cheng is directed to removing covering objects from face images to improve facial recognition accuracy. By contrast, the present application is directed to performing a smearing process on target facial organs to enhance the visual interest and entertainment value of images. See Specification, ,-r~· [0003 ], [0025]. A person of ordinary skill in the art seeking to create an image smearing system for entertainment purposes would not have been motivated to look to Comploi's 3D avatar generation system or Cheng's facial recognition preprocessing system, because these references solve fundamentally different technical problems. The Office Action's assertion that the motivation is to "improve recognition" actually underscores this point-amended claim l is not directed to recognition improvement at all, but rather to image manipulation for entertainment purposes.”
Examiner disagrees.
The examiner recognizes that obviousness may be established by combining or modifying the teachings of the prior art to produce the claimed invention where there is some teaching, suggestion, or motivation to do so found either in the references themselves or in the knowledge generally available to one of ordinary skill in the art. See In re Fine, 837 F.2d 1071, 5 USPQ2d 1596 (Fed. Cir. 1988), In re Jones, 958 F.2d 347, 21 USPQ2d 1941 (Fed. Cir. 1992), and KSR International Co. v. Teleflex, Inc., 550 U.S. 398, 82 USPQ2d 1385 (2007).
In this case, each reference, Comploi, Cheng, or Lee, was directed to "image manipulation".
Regarding claims 2 – 9, 22 – 29, and 31, the Applicant asserts that they are not obvious over based on their dependency from independent claims 1, 21, and 30 respectively. The examiner cannot concur with the Applicant respectfully from same reason noted in the examiner’s response to argument asserted from claims 1, 21, and 30 respectively.
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Ming Wu whose telephone number is (571) 270-0724. The examiner can normally be reached on Monday-Thursday and alternate Fridays (9:30am - 6:00pm) EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Devona Faulk can be reached on 571-272-7515. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Ming Wu/
Primary Examiner, Art Unit 2618
Prosecution Insights