DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Receipt is acknowledged of certified copies of papers required by 37 CFR 1.55.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claim 18 is rejected under 35 U.S.C. 101 because the claimed invention is directed to non-statutory subject matter. The claim(s) does/do not fall within at least one of the four categories of patent eligible subject matter because the claimed invention is directed to a computer program per se.
Computer programs claimed as computer listings per se, i.e., the descriptions or expressions of the programs, are not physical "things." They are neither computer components nor statutory processes, as they are not "acts" being performed. Such claimed computer programs do not define any structural and functional interrelationships between the computer program and other claimed elements of a computer which permit the computer program's functionality to be realized. In contrast, a claimed non-transitory computer-readable medium encoded with a computer program is a computer element which defines structural and functional interrelationships between the computer program and the rest of the computer which permit the computer program's functionality to be realized, and is thus statutory. See Lowry, 32 F.3d at 1583-84, 32 USPQ2d at 1035.
As an additional note, a computer program claimed as stored on a non-transitory computer readable medium having executable programming instructions stored thereon is considered statutory.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-3, 8, 14, 16, and 18-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Wu et al. (WO 2022160701 A1), and in view of Cao et al. (CN 110738595 A).
Regarding Claim 1, Wu discloses An image processing method (¶3 reciting “a method, apparatus, device, and storage medium for generating special effects”. Fig. 10), comprising:
obtaining an image to be processed that comprises a target subject; (Fig. 10, step 1001. ¶126 reciting “In step 1001, the original face image can be detected to obtain facial key point information.”) and
determining facial attribute information of the target subject, (¶127 reciting “the original face image can be detected to obtain the location coordinates of 106 facial key points, which can be used as facial key point information” Further, ¶136 reciting “In step 1005, the face heatmap corresponding to the target face image is determined based on the face key point information.”) and
fusing a target special effect matching the facial attribute information for the target subject, to obtain a target special effect image corresponding to the image to be processed. (¶148 reciting “In step 1009, based on the fusion coefficient determined according to the fusion mask image, the blurred hair image and the texture image are fused to obtain the special effect image of the target face image.” Further, ¶154 reciting “After obtaining the effect image of the target face image, the effect image can be pasted back into the original face image to add effects on the original face image. For example, the pixel values in the effect image of the target face image can be overwritten with the corresponding pixel values of the original face image to obtain the original effect image.”)
However, Wu does not explicitly disclose obtaining the image to be processed in response to a special effect trigger operation.
It is well known in the art to obtain an image to process in repose to a trigger operation. In addition, Cao teaches “an image processing method, apparatus, device, and computer storage medium to improve the fit and realism of adding hair to a face.” (¶8). More specifically, Cao teaches obtaining an image in response to a trigger, and recites “. . ., the user can take a face image through the camera function of the terminal device 201 or select a face image from the album of the terminal device 201. Then, the face image is sent to the server 202. After receiving the face image, the server 202 will implement the image processing method provided in this application embodiment, add the hair corresponding to the hairstyle selected by the user to the face image, and return the final image after changing the hairstyle to the terminal device 201 for display.” (¶105).
It would have been obvious to one with ordinary skill, before the effective filing date of the claimed invention, to modify the method (taught by Wu) to obtain an image to process in response to a trigger (taught by Cao). The suggestions/motivations would have been to “improve the fit and realism of adding hair to a face” (¶8), and to apply a known technique to a known device (method, or product) ready for improvement to yield predictable results.
Regarding Claim 2. Wu in view of Cao discloses The method according to claim 1, wherein the special effect trigger operation comprises at least one of the following:
triggering a special effect processing control;
detecting voice information comprising a special effect adding instruction;
detecting that a display interface comprises a face image; and
detecting that, in a field of view corresponding to a target terminal, a body movement of the target subject is the same as a preset special effect feature.
(Cao, ¶105 disclosing the trigger operation comprises a processing control, i.e. a user selecting a face image from an album. The suggestions/motivations would have been the same as that of Claim 1 rejections.)
Regarding Claim 3. Wu in view of Cao discloses The method according to claim 1, wherein the facial attribute information comprises at least face deflection angle information, and the determining facial attribute information of the target subject comprises:
determining face deflection angle information of a face image of the target subject relative to a display device. (Cao, ¶95 teaching a face pose is related to a deflection angle of the face image relative to a display device, and reciting “Face pose: Pose usually refers to the relative orientation and position of an object relative to the camera. Specifically, for a face, it refers to the relative orientation and position of the face relative to the camera. Generally, the front view is used as the reference. When the head is rotated left and right or tilted up and down, the front view of a person will have a certain rotation angle or tilt angle relative to the fixed camera.” Further, ¶100 teaching the face pose of the target subject is used in matching (thus it is determined), and reciting “After obtaining the face image, the obtained face image can be matched with each mask image to obtain the mask image that is closest to the pose of the face image.” The suggestions/motivations would have been the same as that of Claim 1 rejections.)
Regarding Claim 8. Wu in view of Cao discloses The method according to claim 1, wherein the determining facial attribute information of the target subject, and fusing a target special effect matching the facial attribute information for the target subject, to obtain a target special effect image corresponding to the image to be processed comprises:
processing the input image to be processed based on a pre-trained target special effect rendering model, determining the facial attribute information of the image to be processed, and rendering the target special effect consistent with the facial attribute information to obtain the target special effect image.
(Cao, ¶110 reciting “Specifically, users can provide a large number of facial images and different hairstyle models, as well as other materials required by the image processing method in this embodiment. Based on the materials provided by the user, the image processing method in this embodiment adds hair to each facial image to obtain a large number of target composite images. The facial images and the corresponding target composite images are used as paired data as training samples for model training to train the constructed model and obtain an end-to-end model for adding hair to facial images.” The suggestions/motivations would have been the same as that of Claim 1 rejections.)
Regarding Claim 14. Wu in view of Cao discloses The method according to claim 1, wherein the target special effect comprises at least one of a pet head simulation special effect, an animal head simulation special effect, a cartoon image simulation special effect, a fluff simulation special effect, and a hairstyle simulation special effect to be fused with the face image. (Wu, Fig. 11 showing a hairstyle simulation special effect to be fused with the face image Img1.)
Regarding Claim 16. Wu in view of Cao discloses An electronic device (Wu, ¶3 reciting “a method, apparatus, device, and storage medium for generating special effects”, Fig. 14 showing a hardware diagram of a computer device.), comprising:
a processor; (Wu, Fig. 14, processor 1010) and
a storage (Wu, Fig. 14, storage 1020) apparatus configured to store a program, wherein the program, when executed by the processor, causes the processor to: (Wu, ¶173 reciting “The memory 1020 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program code is stored in the memory 1020 and is called and execute by the processor 1010.”)
obtain, in response to a special effect trigger operation, an image to be processed that comprises a target subject; and
determine facial attribute information of the target subject, and fusing a target special effect matching the facial attribute information for the target subject, to obtain a target special effect image corresponding to the image to be processed.
(See Claim 1 rejections for detailed analysis.)
Regarding Claim 18. Wu in view of Cao discloses A computer program product which, when executed by a computer, causes the computer to: (Wu, ¶173 reciting “The memory 1020 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program code is stored in the memory 1020 and is called and execute by the processor 1010.”)
obtain, in response to a special effect trigger operation, an image to be processed that comprises a target subject; and
determine facial attribute information of the target subject, and fusing a target special effect matching the facial attribute information for the target subject, to obtain a target special effect image corresponding to the image to be processed.
(See Claim 1 rejections for detailed analysis.)
Claim 19, has similar limitations as of Claim(s) 2 and 16, therefore it is rejected under the same rationale as Claim(s) 2 and 16.
Claim 20, has similar limitations as of Claim(s) 3 and 16, therefore it is rejected under the same rationale as Claim(s) 3 and 16.
Claim(s) 4-7 and 21-22 is/are rejected under 35 U.S.C. 103 as being unpatentable over Wu et al. (WO 2022160701 A1), and in view of Cao et al. (CN 110738595 A), and further in view of Wang (US 20190130166 A1).
Regarding Claim 4. Wu in view of Cao discloses The method according to claim 3.
Wu in view of Cao does not explicitly disclose wherein the determining face deflection angle information of a face image of the target subject relative to a display device comprises:
determining, based on a predetermined target center line, a deflection angle of the face image relative to the target center line, and using the deflection angle as the face deflection angle information, wherein the target center line is determined based on a historical face image, and a face deflection angle of the historical face image relative to the display device is less than a preset deflection angle threshold; or
segmenting the face image based on a preset grid, and determining the face deflection angle information of the face image relative to the display device based on a segmentation result; or
performing angle registration on the face image and all face images to be matched, to determine a target face image to be matched that corresponds to the face image, and using a face deflection angle of the target face image to be matched as the face deflection angle information of the target subject, wherein all the face images to be matched respectively correspond to different deflection angles, and a set of the different deflection angles covers 360 degrees; or
recognizing, based on a pre-trained face deflection angle determining model, the image to be processed to determine the face deflection angle information of the target subject.
Wang teaches “A deviation angle of the face region is determined according to the distances between the facial feature recognition points and the angles of the coordinate axis relative to the lines each connecting two of the facial feature recognition points.” (ABST). Further, ¶28 recites “The electronic device may acquire the deviation angle by means of artificial intelligence. The deviation angle of the face region refers to a rotation angle of the face region in the image relative to a standard face image. The standard face image is a front face image, i.e., an image shot when the face directly faces a camera. The deviation angle of the face region may be represented with three angles. Three straight lines perpendicular to each other in the three-dimensional space are intersected at a point to form a three-dimensional coordinate system, every two straight lines in the three straight lines may form a plane, and there are totally three planes. Then, the deviation angle of the face region is represented with rotation angles of the face region on the three planes relative to the standard face image.” Therefore, Wang teaches determining, based on a predetermined target center line (the axis lines in a standard face image plane), a deflection angle of the face image relative to the target center line, and using the deflection angle as the face deflection angle information, wherein the target center line is determined based on a historical face image (a front face image shot when the face directly faces a camera), and a face deflection angle of the historical face image relative to the display device is less than a preset deflection angle threshold. (a front face directly faces a camera)
It would have been obvious to one with ordinary skill, before the effective filing date of the claimed invention, to modify the method (taught by Wu in view of Cao) to determine a deflection angle of the face image relative to a target center line (taught by Wang). The suggestions/motivations would have been that “the deviation angle of the face region in the image is analyzed. The retouching template corresponding to the face region is acquired according to the deviation angle of the face region, so that the retouching template fits a face better.” (¶34), and to apply a known technique to a known device (method, or product) ready for improvement to yield predictable results.
Regarding Claim 5. Wu in view of Cao and Wang discloses The method according to claim 4, wherein the fusing a target special effect matching the facial attribute information for the target subject, to obtain a target special effect image corresponding to the image to be processed comprises:
obtaining a target fusion special effect model consistent with the facial attribute information from all fusion special effect models to be selected (Cao, ¶131 reciting “Step 302: Obtain a face image and determine the target mask image that matches the face pose in the face image from multiple mask images.”), wherein all the fusion special effect models to be selected are special effect models respectively corresponding to different face deflection angles; (Cao, ¶133 reciting “the target mask image refers to the image with the smallest deviation between the face pose and the face pose in the face image among multiple mask images. That is, the minimum deviation between face poses is defined as matching.”) and
fusing the target fusion special effect model and the face image of the target subject to obtain the target special effect image in which the target special effect is fused for the target subject.(Cao, ¶154 reciting “Step 303: Based on the hair region mask in the target mask image, merge the face image and the target hair material image corresponding to the target hairstyle to obtain the target composite image.”)
(The suggestions/motivations would have been the same as that of Claim 1 rejections.)
Regarding Claim 6. Wu in view of Cao and Wang discloses The method according to claim 5, wherein the fusing the target fusion special effect model and the face image of the target subject to obtain the target special effect image in which the target special effect is fused for the target subject comprises:
extracting a head image of the target subject (Wu, Fig. 11, Img 1. ¶130 reciting “the original face image can be adjusted to obtain the target face image Img1 in Figure 11.”), and fusing the head image into a target position in the target fusion special effect model to obtain a special effect image to be corrected, wherein the head image comprises the face image and a hair image (Wu, Fig. 11, Img9. ¶144 reciting “In step 1008, the face mask image and the hair mask image are fused together to obtain a fused mask image.” ¶153 reciting “In step 1010, the effect image of the target face image is fused with the original face image to obtain the original effect image.”); and
determining pixels to be corrected in the special effect image to be corrected, and processing the pixels to be corrected to obtain the target special effect image, wherein the pixels to be corrected comprise pixels corresponding to a hair area that is not covered by the target special effect and pixels on an edge of the face image that do not fit a target fusion special effect. (Wu, ¶154 reciting “After obtaining the effect image of the target face image, the effect image can be pasted back into the original face image to add effects on the original face image. For example, the pixel values in the effect image of the target face image can be overwritten with the corresponding pixel values of the original face image to obtain the original effect image.” In addition, ¶155 reciting “In step 1011, based on the gender information of the face in the original special effects image, the facial contour in the original special effects image is adjusted, and/or the original special effects image is beautified.” ¶156.)
Regarding Claim 7. Wu in view of Cao and Wang discloses The method according to claim 5, wherein the fusing the target fusion special effect model and the face image of the target subject to obtain the target special effect image in which the target special effect is fused for the target subject comprises:
determining at least one fusion key point in the target fusion special effect model and a corresponding target key point on the face image, to obtain at least one key point pair; and
determining a distortion parameter based on the at least one key point pair, so as to adapt the target fusion special effect model to the face image based on the distortion parameter, to obtain the target special effect image.
(Cao, ¶150 reciting “Specifically, based on the positional information of K key points in the target mask image and the positional information of K key points in the face image, the deviation value of the face pose between the target mask image and the face image can be calculated. The target mask image is then adjusted according to the deviation value so that the deviation value between the face pose in the adjusted target mask image and the face pose in the face image is less than a preset deviation value threshold, thereby compensating for the angular deviation between the two face poses.” The suggestions/motivations would have been the same as that of Claim 1 rejections.)
Claim 21, has similar limitations as of Claim(s) 4 and 16, therefore it is rejected under the same rationale as Claim(s) 4 and 16.
Claim 22, has similar limitations as of Claim(s) 5 and 16, therefore it is rejected under the same rationale as Claim(s) 5 and 16.
Claim(s) 9 is/are rejected under 35 U.S.C. 103 as being unpatentable over Wu et al. (WO 2022160701 A1), and in view of Cao et al. (CN 110738595 A), and further in view of Hall et al. (WO 2021056043 A1).
Regarding Claim 9. Wu in view of Cao discloses The method according to claim 8, wherein the method further comprises:
determining a special effect rendering model to be trained of a target network structure;
(Wu, ¶169 reciting “The components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units.”)
However, Wu in view of Cao does not explicitly disclose determining a master training special effect rendering model and a slave training special effect rendering model based on the special effect rendering model to be trained; and
obtaining the target special effect rendering model by training the master training special effect rendering model and the slave training special effect rendering model.
It is well known in the art that distributed training including Model Parallelism and Data Parallelism. Hall teaches distributed training models in ¶24. Further, Hall teaches a master training model and a slave training model in ¶122.
It would have been obvious to one with ordinary skill, before the effective filing date of the claimed invention, to adapt Hall’s teaching on distributed training model to the method taught by Wu in view of Cao. The suggestions/motivations would have been to improve training efficiency, and to apply a known technique to a known device (method, or product) ready for improvement to yield predictable results.
Allowable Subject Matter
Claims 10-13 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Regarding Claim 10, Wu in view of Cao and Hall discloses The method according to claim 9. Wu discloses a deep neural network with convolution filter in Fig. 9.
However, Claim 10 is distinguished from the closest known prior art alone or in reasonable combination, in consideration of the claim as a whole, particularly the limitations similar to ”obtaining at least one neural network to be selected, wherein the neural network to be selected comprises a convolutional layer, and the convolutional layer comprises at least one convolution each comprising a plurality of channel numbers; and determining, based on an amount of computation and an image processing effect of the at least one neural network to be selected, a neural network to be selected of the target network structure as the special effect rendering model to be trained, wherein the image processing effect is evaluated by a similarity between an output image and an actual image under a condition that model parameters in the at least one neural network to be selected are unified.” in combination with the remaining aspects of the claim and its base claims(s).
Regarding Claim 11. Wu in view of Cao discloses The method according to claim 9.
However, Claim 11 is distinguished from the closest known prior art alone or in reasonable combination, in consideration of the claim as a whole, particularly the limitations similar to “constructing, based on a number of channels of each convolution in the special effect rendering model to be trained, the master training special effect rendering model with a multiplied number of channels of the corresponding convolution; and using the special effect rendering model to be trained as the slave training special effect rendering model.” in combination with the remaining aspects of the claim and its base claims(s).
Regarding Claim 12. Wu in view of Cao and Hall discloses The method according to claim 9.
However, Claim 12 is distinguished from the closest known prior art alone or in reasonable combination, in consideration of the claim as a whole, particularly the limitations similar to “wherein the obtaining the target special effect rendering model by training the master training special effect rendering model and the slave training special effect rendering model comprises: obtaining a training sample set, wherein the training sample set comprises a plurality of training sample types each corresponding to different facial attribute information, each training sample comprises an original training image and a special effect superimposed image corresponding to the same facial attribute information, and the facial attribute information corresponds to a face deviation angle; inputting, for each training sample, the original training image in the current training sample separately into the master training special effect rendering model and the slave training special effect rendering model, to obtain a first special effect image and a second special effect image, wherein the first special effect image is an image output based on the master training special effect rendering model, and the second special effect image is an image output based on the slave training special effect rendering model; performing, based on loss functions for the master training special effect rendering model and the slave training special effect rendering model, loss processing on the first special effect image, the second special effect image, and the special effect superimposed image to obtain loss values, so as to correct model parameters in the master training special effect rendering model and the slave training special effect rendering model based on the loss values; using convergence in the loss functions as a training objective, to obtain a master special effect rendering model and a slave special effect rendering model; and using the trained slave special effect rendering model as the target special effect rendering model.” in combination with the remaining aspects of the claim and its base claims(s).
Claim 13 depends from Claim 12, and therefore also contain allowable subject matter.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to YI WANG whose telephone number is (571)272-6022. The examiner can normally be reached 9am - 5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Jason Chan can be reached at (571)272-3022. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/YI WANG/Primary Examiner, Art Unit 2619