Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Receipt is acknowledged of certified copies of papers required by 37 CFR 1.55.
Information Disclosure Statement
The IDS of (08/13/2024) has been considered by the examiner. The annotated copy is included herewith.
Claim Objections
Claim 5 is objected to because of the following informalities: At the end of line 3 should be an “or” connecting the two elements. Appropriate correction is required.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-2, 4-5, 7, and 18-20 are rejected under 35 U.S.C. 103 as being unpatentable over Qiu et al. (CN 113487709 A, IDS Ref), hereinafter referred to as Qiu, in view of Drummond et al. (US 20210406543 A1), hereinafter referred to as Drummond.
Regarding Claim 1, Qiu teaches a video image processing method, comprising: in response to an effect triggering operation, acquiring a current image to be processed comprising a target object,
Effect triggering operation:
Qiu [Page 8 Paragraphs 1-2] (firstly, determining the limb key point of the appointed limb part of the real anchor in the first video image by the limb detection model. then, under the condition of identifying the first video image comprises clear face image, obtaining the size of the face frame and/or position information of the face frame. after identifying the first video image comprises clear hand image, can obtain the size of the hand detection frame and/or the position information of the hand detection frame. under the condition of detecting the limb detection result containing the hand detection frame, further capable of performing gesture detection to the image located in the hand detection frame by the gesture recognition model, obtaining the gesture classification result, wherein the gesture classification result is a feature vector; the feature vector is used for representing the probability of the hand gesture of the real anchor (target object) in the first video image belonging to each preset gesture.)
Image with target object:
Examiner Note: Qiu teaches the target object being the real anchor in frame.
and determining event information of the target object;
Event information:
Qiu [Page 8 Paragraph 2] (further capable of performing gesture detection to the image located in the hand detection frame by the gesture recognition model, obtaining the gesture classification result,)
Qiu [Page 2 Paragraph 5] (the posture detection result comprises a limb detection result and/or gesture classification result)
determining part parameters of at least one model part (limbs) in a target animation model according to body part information of the target object in the current image to be processed;
Part parameters:
Qiu [Page 3 Paragraph 4] (based on the posture detection result, determining the first driving information for animation effect; wherein the first driving information is used for indicating animation jump information of animation effect of virtual live broadcast model displayed in the video live broadcast interface;)
Qiu [Page 11 Paragraph 3] (In the present disclosure, the first driving information may be a matrix of 1 * (P + Q), wherein P can be understood as the number of preset posture, Q can be understood as the number of the plurality of animation sequences corresponding to the preset posture.)
Examiner Note: Posture detection result is event information, and is used to determine first driving information, which contains preset postures and animation sequences, all which corresponds to part parameters.
Model part:
Examiner Note: The part parameters (preset posture and animation sequences) define how the model parts (limbs) are displayed and managed on the model.
Target animation model:
Qiu [Page 3 Paragraph 4] (the target animation effect of the virtual presenter model corresponding to the real anchor is determined according to the posture detection result)
Body part information:
Qiu [Page 2 Paragraph 5] (the posture detection result comprises a limb detection result and/or gesture classification result)
Examiner Note: The limb detection and gesture classification results are body part information
determining target effect display parameters of the target animation model based on the part parameters and the event information;
Qiu [Page 3 Paragraph 4] (the target animation effect of the virtual presenter model corresponding to the real anchor is determined according to the posture detection result, comprising: based on the posture detection result, determining the first driving information for animation effect; wherein the first driving information is used for indicating animation jump information of animation effect of virtual live broadcast model displayed in the video live broadcast interface; according to the first driving information, determining the animation sequence matched with the first driving information in a plurality of animation sequences corresponding to the posture detecting result, and determining the matched animation sequence as the target animation effect.)
Qiu [Page 4 Paragraph 5] (the target animation effect comprises a limb action special effect and/or rendering material effect for characterizing the virtual presenter model limb action, the video live interface corresponding to the real presenter display target animation effect of the virtual presenter model, comprising: displaying the limb action special effect of the limb action of the virtual presenter model in the video live broadcast interface; and/or displaying the rendering material effect at the target location associated with the limb action of the virtual presenter model)
Qiu [Page 2 Paragraph 5] (the posture detection result comprises a limb detection result and/or gesture classification result,)
Examiner Note: The target animation effect is gotten from the posture detection result (event information) and the first driving information (part parameters). This target animation effect is then later displayed at a target location. For a target location to be found, there must be some sort of parameters to determine how the target effect is to be displayed (target effect display parameters).
and determining a target video frame corresponding to the current image to be processed
Qiu [Page 6 Paragraph 1] (For example, the camera device of the live device can collect the video image of the real anchor, then, the real anchor of the video image contained in the capture, so as to obtain the real anchor posture information. after determining the attitude information, it can generate the corresponding driving signal, the driving signal for driving the video live picture display animation effect corresponding to the virtual presenter model.)
and playing the target video frame based on the target effect display parameters.
Qiu [Page 6 Paragraph 1] (For example, the camera device of the live device can collect the video image of the real anchor, then, the real anchor of the video image contained in the capture, so as to obtain the real anchor posture information. after determining the attitude information, it can generate the corresponding driving signal, the driving signal for driving the video live picture display animation effect corresponding to the virtual presenter model.)
However, Qiu doesn’t teach and fusing a target face image of the target object into the target animation model.
Drummond teaches fusing a target face image of the target object into the target animation model.
Drummond [0059] (Additionally, any objects can be processed using a computer animation model, such as a human's face and parts of a human body, animals, or non-living things such as chairs, cars, or other objects.)
Drummond [0064] (In some examples, a search for landmarks from the mean shape aligned to the position and size of the face determined by a global face detector is started. Such a search then repeats the steps of suggesting a tentative shape by adjusting the locations of shape points by template matching of the image texture around each point and then conforming the tentative shape to a global shape model until convergence occurs.)
Examiner Note: Qiu shows getting target face image of the target object, but doesn’t explicitly disclose a method of fusing into the target animation model. Drummond teaches the object processed could be a face, and it demonstrates fusing a target object and model together.
Qiu and Drummond are analogous in the art of gathering image data and animating a 3D model based off such data. It would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to modify Qiu’s invention of utilizing the face image to explicitly be used for fusing like in Drummond in order to create a functional facial animation system.
Regarding Claim 2, Qiu, in view of Drummond teaches
The method according to claim 1, wherein the acquiring a current image to be processed comprising a target object, and determining event information of the target object comprises: acquiring the current image to be processed comprising the target object and collected based on a camera apparatus;
Qiu [Page 6 Paragraph 1] (For example, the camera device of the live device can collect the video image of the real anchor, then, the real anchor of the video image contained in the capture, so as to obtain the real anchor posture information. after determining the attitude information, it can generate the corresponding driving signal, the driving signal for driving the video live picture display animation effect corresponding to the virtual presenter model.)
determining the event information triggered by the target object in the current image to be processed
Qiu [Page 2 Paragraph 5] (In one possible embodiment, the posture detection result comprises a limb detection result and/or gesture classification result, the first video image in the designated limb part of the real anchor posture detection, obtaining the posture detection result)
Qiu [Page 7 Paragraph 13] (by limb detection model, the first video image in the designated limb part of the real anchor for limb detection to obtain the limb detection result. Here, the limb detection result comprises at least one of the following: limb key point, size of the face frame, position information of the face frame, size of the hand detection frame, position information of the hand detection frame)
Qiu [Page 8 Paragraph 2] (under the condition of detecting the limb detection result containing the hand detection frame, further capable of performing gesture detection to the image located in the hand detection frame by the gesture recognition model)
However, Qiu doesn’t teach based on a preset feature detection algorithm.
Drummond teaches based on a preset feature detection algorithm.
Drummond [0062] “In some examples of a computer animation model to transform image data using face detection, the face is detected on an image with use of a specific face detection algorithm (e.g., Viola-Jones). Then, an Active Shape Model (ASM) algorithm is applied to the face region of an image to detect facial feature reference points.”
Examiner Note: Qiu mentions getting event information from detection models, but doesn’t give a specific example. Drummond gives an example of a preset feature detection algorithm that could be used. Implementing a standard, robust detection algorithm, such as an ASM as suggested by Drummond, to detect limbs or gestures in Qiu’s system would yield the same function as it does separately, but with improved accuracy or computational efficiency in the combined system, which is a predictable result.
Qiu and Drummond are analogous in the art of gathering image data and animating a 3D model based off such data. It would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to modify Qiu’s invention to gather its event information from a specific preset feature detection algorithm like in Drummond to detect/track the movement of objects within the image.
Regarding Claim 4, Qiu, in view of Drummond teaches
the method according to claim 2, wherein the determining the event information triggered by the target object in the current image to be processed based on a preset feature detection algorithm comprises:
determining the event information triggered by the target object based on current coordinate information of the plurality of preset detection parts and preset coordinate range information corresponding to the plurality of preset detection parts respectively.
Qiu [Page 10 Paragraph 9] (it can be determined that each difference value satisfies the preset difference threshold value, determining the real anchor in the first video image satisfies the gesture recognition condition; or, the number of the plurality of difference value satisfies the preset difference threshold value is greater than or equal to a certain threshold value, then determining the real anchor in the first video image satisfies the gesture recognition condition; otherwise, determining the real anchor in the first video image does not satisfy the gesture recognition condition.)
Examiner Note: “preset coordinate range information” is being interpreted as a preset or known threshold that corresponds to a trigger.
However, Qiu doesn’t teach determining current coordinate information of a plurality of preset detection parts in the target object based on the preset feature detection algorithm;
Drummond teaches determining current coordinate information of a plurality of preset detection parts in the target object based on the preset feature detection algorithm;
Drummond [0062] “In some examples of a computer animation model to transform image data using face detection, the face is detected on an image with use of a specific face detection algorithm (e.g., Viola-Jones). Then, an Active Shape Model (ASM) algorithm is applied to the face region of an image to detect facial feature reference points.”
Drummond [0063] “For facial landmarks, for example, the location of the left eye pupil may be used. If an initial landmark is not identifiable (e.g., if a person has an eyepatch), secondary landmarks may be used. Such landmark identification procedures may be used for any such objects. In some examples, a set of landmarks forms a shape. Shapes can be represented as vectors using the coordinates of the points in the shape. “
Examiner Note: Drummond teaches collecting key points based on a specific face detection algorithm, and using the coordinates of those key points in further operation.
Qiu and Drummond are analogous in the art of gathering image data and animating a 3D model based off such data. It would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to modify Qiu’s invention to contain key points with coordinate information from Drummond for precise position of said key points to perform further operation on.
Regarding Claim 5, Qiu fails to teach
the method according to claim 1, wherein the effect triggering operation comprises at least one of the following: triggering an effect prop corresponding to the target animation model; encompassing a face image in a detected view field region.
Drummond teaches the method according to claim 1, wherein the effect triggering operation comprises at least one of the following: triggering an effect prop corresponding to the target animation model; encompassing a face image in a detected view field region.
Drummond [0066] “The transformation system operating within the messaging client 104 determines the presence of a face within the image or video stream and provides modification icons associated with a computer animation model to transform image data, or the computer animation model can be present as associated with an interface described herein. The modification icons include changes that may be the basis for modifying the user's face within the image or video stream as part of the modification operation. Once a modification icon is selected, the transform system initiates a process to convert the image of the user to reflect the selected modification icon (e.g., generate a smiling face on the user). A modified image or video stream may be presented in a graphical user interface displayed on the client device 102 as soon as the image or video stream is captured, and a specified modification is selected.”
Examiner Note: This meets the limitation of “encompassing a face image in a detected view field region.”
Qiu and Drummond are analogous in the art of gathering image data and animating a 3D model based off such data. It would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to modify Qiu’s invention to have an effect triggering operation comprising a face being detected within the view field like in Drummond to detect/track the facial image of the target object.
Regarding Claim 7, Qiu, in view of Drummond further teaches
The method according to claim 1, wherein the body part information comprises head information,
Qiu [Page 8 Paragraph 5] (Here, the limb detection result comprises a limb key point, for example, limb key point of each appointed limb part. supposing, the appointed limb part comprises a head part and the upper half body part, then the limb key point comprises the key point of the head part, and the key point of the two arms in the upper half body part, the key point of the hand, and the key point of the upper body.)
and the determining part parameters of at least one model part in a target animation model according to body part information of the target object in the current image to be processed comprises: determining head attribute information corresponding to the head information of the target object based on a face image detection algorithm, wherein the head attribute information comprises head deflection angle information and head position information;
Qiu [Page 9 Paragraph 1] (It can be understood that the relative positional relationship comprises at least one of the following: the relative distance between each appointed limb part, the angle relationship between the associated limb part in each appointed limb part. wherein the associated limb part can be understood as adjacent designated limb part, or the same type of designated limb part.)
Qiu [Page 8 Paragraph 5] (Here, the limb detection result comprises a limb key point, for example, limb key point of each appointed limb part. supposing, the appointed limb part comprises a head part and the upper half body part, then the limb key point comprises the key point of the head part, and the key point of the two arms in the upper half body part, the key point of the hand, and the key point of the upper body.)
Examiner Note: It is demonstrated in Qiu that a limb part could be a head part, and to get the angle relationship (and relative distance), head deflection angle information and head position information are needed.
adjusting the part parameters of the head model in the target animation model according to the head attribute information,
Qiu [Page 3 Paragraph 4] (In one possible embodiment, the target animation effect of the virtual presenter model corresponding to the real anchor is determined according to the posture detection result)
Examiner Note: Posture detection result comprises the limb detection result (containing the attribute information), and the target animation model (presenter model) has its parameters of the head being adjusted according to these detected results.
wherein the part parameters comprise a deflection parameter and a movement parameter of the head model.
Qiu [Page 3 Paragraph 4] (In one possible embodiment, the target animation effect of the virtual presenter model corresponding to the real anchor is determined according to the posture detection result)
Examiner Note: Posture detection result comprises the limb detection result (containing the attribute information), and the target animation model (presenter model) has its parameters of the head being adjusted according to these detected results. Parameters of deflection and movement must be in the head model for it to be adjust accordingly.
Regarding Claim 18, it recites similar limitations to Claim 1, except it is an electronic device claim (Qiu Page 4 Paragraph 8) with at least one processor (See Qiu Page 4 Paragraph 8 “the memory stores a machine readable instruction executable by the processor”) and at least one storage (See Qiu Page 4 Paragraph 9 “the computer-readable storage medium is stored with a computer program, the computer program when executed by a processor executes the first aspect”); therefore, it is rejected under similar rationale as Claim 1.
Regarding Claim 20, it recites similar limitations to Claim 2, except it is an electronic device claim (Qiu Page 4 Paragraph 8), therefore it is rejected under similar rationale as Claim 2.
Regarding Claim 19, it recites similar limitations to Claim 1, except it is a computer-readable storage medium (Qiu Page 4 Paragraph 9), therefore it is rejected under similar rationale as Claim 1.
Claims 3, 6, and 21 are rejected under 35 U.S.C. 103 as being unpatentable over Qiu et al. (CN 113487709 A, IDS Ref), hereinafter referred to as Qiu, in view of Drummond et al. (US 20210406543 A1), hereinafter referred to as Drummond, in further view of Lu (US 20250037400 A1).
Regarding Claim 3, Qiu further teaches
Qiu [Page 2 Paragraph 5] (In one possible embodiment, the posture detection result comprises a limb detection result and/or gesture classification result, the first video image in the designated limb part of the real anchor posture detection, obtaining the posture detection result,)
However, Qiu fails to teach the method according to claim 2, wherein the determining the event information triggered by the target object in the current image to be processed based on a preset feature detection algorithm comprises: determining current key point coordinate information of a plurality of preset detection parts of the target object based on the preset feature detection algorithm; for a same preset detection part, determining movement information of a current preset detection part based on the key point coordinate information and historical key point coordinate information of the preset detection part corresponding to the same preset detection part in a historical image before the current image to be processed; determining the event information triggered by the target object based on the movement information of the plurality of preset detection parts.
Drummond teaches the method according to claim 2, wherein the determining the event information triggered by the target object in the current image to be processed based on a preset feature detection algorithm comprises: determining current key point coordinate information of a plurality of preset detection parts of the target object based on the preset feature detection algorithm;
Drummond [0062] “In some examples of a computer animation model to transform image data using face detection, the face is detected on an image with use of a specific face detection algorithm (e.g., Viola-Jones). Then, an Active Shape Model (ASM) algorithm is applied to the face region of an image to detect facial feature reference points.”
Drummond [0063] “For facial landmarks, for example, the location of the left eye pupil may be used. If an initial landmark is not identifiable (e.g., if a person has an eyepatch), secondary landmarks may be used. Such landmark identification procedures may be used for any such objects. In some examples, a set of landmarks forms a shape. Shapes can be represented as vectors using the coordinates of the points in the shape. “
Examiner Note: Drummond teaches collecting key points based on a specific face detection algorithm, and using the coordinates of those key points in further operation.
Qiu and Drummond are analogous in the art of gathering image data and animating a 3D model based off such data. It would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to modify Qiu’s invention to have an effect triggering operation comprising a face being detected within the view field like in Drummond to detect/track the facial image of the target object.
Qiu and Drummond fail to teach for a same preset detection part, determining movement information of a current preset detection part based on the key point coordinate information and historical key point coordinate information of the preset detection part corresponding to the same preset detection part in a historical image before the current image to be processed; determining the event information triggered by the target object based on the movement information of the plurality of preset detection parts.
Lu teaches for a same preset detection part, determining movement information of a current preset detection part based on the key point coordinate information and historical key point coordinate information of the preset detection part corresponding to the same preset detection part in a historical image before the current image to be processed; determining the event information triggered by the target object based on the movement information
Lu [0057] “The at least two video frames may be two video frames or a plurality of video frames, and a specific number of the video frames is matched with a specific demand. For example, the target object in the image to be processed is in a rotating posture, and in this case, if the item to be worn is a skirt, the skirt will flutter up, and then, the wearing effect to be processed is a fluttering effect. The height and angle of fluttering need to be determined in combination with the several previous video frames. In this case, a plurality of video frames before and after the image to be processed may be obtained, and motion information of the target object is determined by adopting a kinematics algorithm. For example, the motion information includes a rotational speed and intensity, and based on this, the wearing effect to be processed corresponding to the item to be worn may be determined.“
Examiner Note: Qiu demonstrates determining the event information (posture detection result) from a limb detection model. Lu further demonstrates comparing a video frame to a historical past video frame, comparing information between the same body parts in both frames.
Qiu, Drummond and Lu are analogous in the art of gathering image data and animating a 3D model based off such data. It would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to modify Qiu in view of Drummond’s method of determining event information to use the historical comparison of video frame data as shown in Lu to achieve the invention of determining event information based on movement information in order to assess event information with greater detail.
Regarding Claim 6, Qiu and Drummond fail to teach the method according to claim 1, wherein the event information is matched with limb action information of a plurality of preset detection parts.
Lu teaches the method according to claim 1, wherein the event information is matched with limb action information of a plurality of preset detection parts.
Lu [0057] “The at least two video frames may be two video frames or a plurality of video frames, and a specific number of the video frames is matched with a specific demand. For example, the target object in the image to be processed is in a rotating posture, and in this case, if the item to be worn is a skirt, the skirt will flutter up, and then, the wearing effect to be processed is a fluttering effect. The height and angle of fluttering need to be determined in combination with the several previous video frames. In this case, a plurality of video frames before and after the image to be processed may be obtained, and motion information of the target object is determined by adopting a kinematics algorithm. For example, the motion information includes a rotational speed and intensity, and based on this, the wearing effect to be processed corresponding to the item to be worn may be determined.”
Examiner Note: Lu demonstrates the event information (in this case the motion information) which matches limb action information (rotational speed and intensity). The limb action information would be coming from the preset detection parts (of the target object).
Qiu, Drummond, and Lu are analogous in the art of gathering image data and animating a 3D model based off such data. It would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to modify Qiu in view of Drummond’s invention of determining event information to use the historical comparison of video frame data as shown in Lu to achieve the invention of determining event information based on movement information in order to assess event information with greater detail.
Regarding Claim 21, it recites similar limitations to Claim 3, except it is an electronic device claim (Qiu Page 4 Paragraph 8), therefore it is rejected under similar rationale as Claim 3.
Claim 8 is rejected under 35 U.S.C. 103 as being unpatentable over Qiu et al. (CN 113487709 A, IDS Ref), hereinafter referred to as Qiu, in view of Drummond et al. (US 20210406543 A1), hereinafter referred to as Drummond, in further view of Lu (US 20250037400 A1), in even further view of Shin et al. (US 20200342218 A1), hereinafter referred to as Shin.
Regarding Claim 8, Qiu further teaches the method according to claim 7, further comprising:
and determining the part parameters of a plurality of other model parts to be determined in the target animation model other than the head model; wherein the model parts to be determined is matched with a limb and trunk of the target animation model.
Qiu [Page 9 Paragraph 1] (the limb detection result comprises a limb key point, for example, limb key point of each appointed limb part. supposing, the appointed limb part comprises a head part and the upper half body part, then the limb key point comprises the key point of the head part, and the key point of the two arms in the upper half body part, the key point of the hand, and the key point of the upper body.)
Examiner Note: Qiu shows that besides the head, other body parts can be used. In this case, the arms (limb) and upper half body part (trunk) correspond to other model parts that can be used. Limb detection is done on these parts as well.
However, Qiu in view of Drummond does not teach processing the part parameters based on an inverse kinematics algorithm.
Lu teaches processing the part parameters
Lu [0057] The at least two video frames may be two video frames or a plurality of video frames, and a specific number of the video frames is matched with a specific demand. For example, the target object in the image to be processed is in a rotating posture, and in this case, if the item to be worn is a skirt, the skirt will flutter up, and then, the wearing effect to be processed is a fluttering effect. The height and angle of fluttering need to be determined in combination with the several previous video frames. In this case, a plurality of video frames before and after the image to be processed may be obtained, and motion information of the target object is determined by adopting a kinematics algorithm. For example, the motion information includes a rotational speed and intensity, and based on this, the wearing effect to be processed corresponding to the item to be worn may be determined.
Qiu, Drummond, and Lu are analogous in the art of gathering image data and animating a 3D model based off such data. It would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to modify the combination Qiu and Drummond’s invention of determining part parameters to be done with kinematics as explicitly mentioned by Lu, as kinematics algorithms enable precise control, motion planning, and optimization.
However, Qiu, Drummond and Lu fail to teach based on an inverse kinematics algorithm,
Shin teaches based on an inverse kinematics algorithm,
Shin [0017] “The control unit 140 may perform the various processes by executing the computer executable instructions stored in the storage unit 130. The control unit 140 inputs the first position information on the portion of the body part acquired by the image analysis unit 120 into the three-dimensional skeleton model stored in the storage unit 130. Further, the control unit 140 acquires second position information on the remaining portion of the body part using Inverse Kinematics that is a calculation based on anatomical characteristics of the three-dimensional skeleton model. Furthermore, the control unit 140 recognizes a pose of the body part based on the acquired second position information and the first position information acquired in the image analysis unit 120. In this way, the control unit 140 may acquire the second position information on the remaining portion (apart from the portion) of the body part. Further, for a portion in the remaining portion where the first position information and the second position information are different, the control unit 140 may recognize the pose of the body part by modifying the first position information and the second position information based on interpolation between the first position information and the second position information.”
Qiu, Drummond, Lu, and Shin are analogous in the art of gathering image data and animating a 3D model based off such data. It would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to modify Qiu in view of Drummond in further view of Lu’s invention of determining part parameters to be done with specifically inverse kinematics as explicitly mentioned by Shin, as it enables efficient motion planning, realistic animation, and precise control by solving for joint movements in reverse.
Claim 9 is rejected under 35 U.S.C. 103 as being unpatentable over Qiu et al. (CN 113487709 A, IDS Ref), hereinafter referred to as Qiu, in view of Drummond et al. (US 20210406543 A1), hereinafter referred to as Drummond, in further view of Long et al. (US 20210279934 A1), hereinafter referred to as Long.
Regarding Claim 9, Qiu further teaches
The method according to claim 1, wherein the determining target effect display parameters of the target animation model based on part parameters and the event information comprises: determining a target animation effect
Qiu [Page 4 Paragraph 3] (the method further comprises: obtaining the virtual live scene corresponding to the real anchor; according to the posture detection result, determining target animation effect of virtual presenter model corresponding to the real presenter, comprising: obtaining the initial animation special effect matched with the posture detection result; determining the animation effect matched with the virtual live scene in the initial animation effect as the target animation effect)
Examiner Note: Qiu teaches determining a target animation effect based on event information, but not explicitly mentioning for fusing and based on an effect mapping relationship table.
determining the target effect display parameters based on the part parameters and the target animation effect
Qiu [Page 3 Paragraph 4] (the target animation effect of the virtual presenter model corresponding to the real anchor is determined according to the posture detection result, comprising: based on the posture detection result, determining the first driving information for animation effect; wherein the first driving information is used for indicating animation jump information of animation effect of virtual live broadcast model displayed in the video live broadcast interface; according to the first driving information, determining the animation sequence matched with the first driving information in a plurality of animation sequences corresponding to the posture detecting result, and determining the matched animation sequence as the target animation effect.)
Qiu [Page 4 Paragraph 5] (the target animation effect comprises a limb action special effect and/or rendering material effect for characterizing the virtual presenter model limb action, the video live interface corresponding to the real presenter display target animation effect of the virtual presenter model, comprising: displaying the limb action special effect of the limb action of the virtual presenter model in the video live broadcast interface; and/or displaying the rendering material effect at the target location associated with the limb action of the virtual presenter model)
Examiner Note: The target animation effect is gotten from the posture detection result (event information) and the first driving information (part parameters). This target animation effect is then later displayed at a target location. There must be some sort of parameters to determine how the target effect is to be displayed, for example in this case the target location associated with a limb.
However, Qiu and Drummond don’t teach animation effect to be fused consistent with the event information, according to a pre-established effect mapping relationship table, wherein the effect mapping relationship table comprises the event information and animation effects to be fused corresponding to the event information;
Long teaches animation effect to be fused consistent with the event information, according to a pre-established effect mapping relationship table, wherein the effect mapping relationship table comprises the event information and animation effects to be fused corresponding to the event information;
Long [0017] “inputting the reference virtual avatar and the expression parameter of at least one of the five sense organs into a pre-trained deep neural network to obtain a first virtual avatar that is associated with the attribute of the first avatar and has the expression of the first avatar”
Long [0048] “In practice, the executing body may determine the expression parameter corresponding to the first avatar by using a preset model or a preset corresponding relationship table. The preset model and the preset corresponding relationship table here may be used to represent the corresponding relationship between the avatar and the expression parameter of the avatar.”
Examiner Note: Long teaches mapping using a relationship table of the avatar (event information) to determine the expression parameter corresponding to the first avatar (target animation effect). This expression parameter is fused with a virtual model in [0017] to achieve the first avatar.
Qiu, Drummond, and Long are analogous in the art of gathering image data and animating a 3D model based off such data. It would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to modify Qiu in view of Drummond’s invention of determining target animation effect according to event information to be done with specifically a relationship table, as well as fused, as explicitly mentioned by Long, to provide a structured, predictable way to match event data to animation results.
Claim 10 is rejected under 35 U.S.C. 103 as being unpatentable over Qiu et al. (CN 113487709 A, IDS Ref), hereinafter referred to as Qiu, in view of Drummond et al. (US 20210406543 A1), hereinafter referred to as Drummond, in further view of Sadi et al. (US 20160086379 A1), hereinafter referred to as Sadi.
Regarding Claim 10, Qiu further teaches
The method according to claim 1, wherein before the determining part parameters of at least one model part in a target animation model according to body part information of the target object in the current image to be processed, the method further comprises:
Qiu teaches determining key points of head parts and limbs:
Qiu [Page 7 Paragraph 3] (Here, the designated limb part comprises: head part and upper half body part (two arm parts, hand part, upper half body part).)
Qiu [Page 8 Paragraph 5] (Here, the limb detection result comprises a limb key point, for example, limb key point of each appointed limb part. supposing, the appointed limb part comprises a head part and the upper half body part, then the limb key point comprises the key point of the head part, and the key point of the two arms in the upper half body part, the key point of the hand, and the key point of the upper body.)
However, Qiu and Drummond doesn’t teach performing offset processing on a scene to be corrected comprising the target animation model according to a preset head offset to obtain a target scene comprising the target animation model.
Sadi teaches performing offset processing on a scene to be corrected comprising the target animation model according to a preset head offset to obtain a target scene comprising the target animation model.
Sadi [0102] “In particular embodiments, the pairs of corresponding feature points include a respective one of the feature points from each of the first and second images. At step 340, the first or second image is spatially adjusted based on a calculated offset between each pair of corresponding feature points. At step 350, the first and second images are combined into a merged or stitched image based on the spatial adjustment.”
Examiner Note: Although, Sadi feature points correspond to a scene which may include, human bodies or animals (see figures 37 – 38 for example) and other objects, Sadi animation methodology describes offset processing on a scene to match a scene to a another, resulting in a target scene.
Qiu and Sadi are analogous in the art of gathering image data and animating a 3D model based off such data. It would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to modify the combination of Qiu in view of Drummond’s invention that shows head models that contain key points to be used as feature points with Sadi’s offset processing so the system can enhance the geometric precision of the resulting 3D models.
Claim 13 is rejected under 35 U.S.C. 103 as being unpatentable over Qiu et al. (CN 113487709 A, IDS Ref), hereinafter referred to as Qiu, in view of Drummond et al. (US 20210406543 A1), hereinafter referred to as Drummond, in further view of Sadi et al. (US 20160086379 A1), hereinafter referred to as Sadi, in even further view of Papandreou et al. (US 20210398351 A1), hereinafter referred to as Papandreou.
Regarding Claim 13, Qiu further teaches the method according to claim 10, wherein before the determining part parameters of at least one model part in a target animation model according to body part information of the target object in the current image to be processed, the method further comprises: determining a
Qiu [Page 8 Paragraph 1] (As shown in FIG. 2, firstly, determining the limb key point of the appointed limb part of the real anchor in the first video image by the limb detection model. then, under the condition of identifying the first video image comprises clear face image, obtaining the size of the face frame and/or position information of the face frame. after identifying the first video image comprises clear hand image, can obtain the size of the hand detection frame and/or the position information of the hand detection frame.)
Qiu [Page 3 Paragraph 4] (based on the posture detection result, determining the first driving information for animation effect; wherein the first driving information is used for indicating animation jump information of animation effect of virtual live broadcast model displayed in the video live broadcast interface;)
Qiu [Page 11 Paragraph 3] (In the present disclosure, the first driving information may be a matrix of 1 * (P + Q), wherein P can be understood as the number of preset posture, Q can be understood as the number of the plurality of animation sequences corresponding to the preset posture.)
Examiner Note: Qiu discloses acquiring a face image (in posture detection), and posture detection result is used in determining first driving information, which comprises a matrix that is used for indication animation information.
However, Qiu, in view of Drummond, in further view of Sadi teaches does not teach a displacement rotation scaling matrix.
Papandreou teaches a displacement rotation scaling matrix of a target face image based on a face image detection algorithm;
Papandreou [0142] “The 3D representations of the object parts are each determined using a part-specific model 814. A part-specific model is a “bottom up” model, such a CNN trained to reproduce a 3D model/representation of a particular object part 810 (e.g., hands, feet, heads, faces, torsos etc.) from an image 804 containing the object. Each object part may be associated with a different CNN, or a different branch of one CNN. The 3D representation of the object part 810 may be in the form of a mesh of the object part. The 3D representation of the object part 810 may be defined in a coordinate system relative to a key point of the object part (e.g., a wrist joint for a hand, a sternum for a torso, an ankle joint for a foot, a facial feature for a head/face).”
[0146] “Starting from a root joint 806 (i.e., i=0), the joint pose of a current joint, j, is determined from the joint pose of a previous joint, i, using a recursive relationship 808. First, a known (i.e., model-defined) offset, O.sub.j, if applied to the joint pose of the previous joint, P.sub.i. The known offset encodes the fixed relationships between joints, e.g., constraints on positions and rotations of joints relative to one another. The offset may comprise a position offset, d.sub.j, and a rotation offset E.sub.j. The offset may be represented by a 4×4 matrix… The product of the joint pose of the previous joint and the offset matrix, M.sub.i, converts the relative displacement/rotation of the joints into an absolute position.”
Examiner Note: Qiu discloses acquiring a face image (in posture detection), and posture detection result is used in determining first driving information, which comprises a matrix that is used for indication animation information. Papandreou more explicitly discloses a matrix that includes position and rotational offsets to transform positions from relative to absolute coordinates, which would be adapting something to match a targeted object based off detected body parts. This matrix could be substituted in place of the matrix used in Qiu to achieve the claimed limitation.
Qiu, Drummond, Sadi, and Papandreou are analogous in the art of gathering image data and animating a 3D model based off such data. It would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to modify the combination Qiu, Drummond, and Sadi’s invention that uses a matrix to represent the part parameters of the detected parts to specifically use Papandreou’s displacement rotation scaling matrix, as it allows for combining complex geometric operations into a single, efficient matrix multiplication.
Claims 11-12 are rejected under 35 U.S.C. 103 as being unpatentable over Qiu et al. (CN 113487709 A, IDS Ref), hereinafter referred to as Qiu, in view of Drummond et al. (US 20210406543 A1), hereinafter referred to as Drummond, in further view of Gronau et al. (US 20210392175 A1), hereinafter referred to as Gronau.
Regarding Claim 11, Qiu fails to teach the method according to claim 1, wherein the fusing a target face image of the target object into the target animation mode comprises:
As discussed in claim 1, Drummond teaches
Drummond [0059] (Additionally, any objects can be processed using a computer animation model, such as a human's face and parts of a human body, animals, or non-living things such as chairs, cars, or other objects.)
[0064] (In some examples, a search for landmarks from the mean shape aligned to the position and size of the face determined by a global face detector is started. Such a search then repeats the steps of suggesting a tentative shape by adjusting the locations of shape points by template matching of the image texture around each point and then conforming the tentative shape to a global shape model until convergence occurs.)
Examiner Note: Drummond teaches the object processed could be a face, and it demonstrates fusing a target object and model together.
Qiu and Drummond are analogous in the art of gathering image data and animating a 3D model based off such data. It would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to modify Qiu’s invention of utilizing the face image to explicitly be used for fusing like in Drummond in order to create a functional facial animation system.
However, Qui in view of Drummond doesn’t teach performing face segmentation processing on the current image to be processed based on a face image segmentation model or a face image segmentation algorithm to acquire a target face image corresponding to the target object.
Gronau teaches performing face segmentation processing on the current image to be processed based on a face image segmentation model or a face image segmentation algorithm to acquire a target face image corresponding to the target object.
Gronau [0028] “FIG. 13 is an example of an image of a face and a segmentation of the image;”
Gronau [0379] “The output of our method may be the geometry (deformation parameters and mesh models) per-frame, and a set of approximated camera parameters for each image.”
Examiner Note: Gronau teaches using facial segmentation, and the result of the segmentation being an approximated match to the camera parameters (target image).
Qiu, Drummond and Gronau are analogous in the art of processing facial data to apply to 3D models. It would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to modify the combination of Qiu and Drummond‘s invention that fuses a facial image into a 3D model to specifically use Gronau’s facial segmentation processing to allow for finer separation of facial regions to ensure more precise spatial details from the image.
Regarding Claim 12, Qiu further teaches the method according to claim 11, wherein the determining a target video frame corresponding to the current image to be processed and playing the target video frame based on the target effect display parameters comprises: adjusting a plurality of limbs and trunk in the target animation model based on the target effect display parameters to obtain the target video frame and play the target video frame.
Qiu [Page 2 Paragraph 4] (based on the posture detection result to determine the real presenter driving virtual presenter model of the target animation effect, and displaying the target animation effect in the video live interface. so as to realize triggering the target animation effect corresponding to the virtual presenter model on the video live broadcast interface through the posture detection result of the real anchor,)
Qiu [Page 8 Paragraph 5] (the limb detection result comprises a limb key point, for example, limb key point of each appointed limb part. supposing, the appointed limb part comprises a head part and the upper half body part, then the limb key point comprises the key point of the head part, and the key point of the two arms in the upper half body part, the key point of the hand, and the key point of the upper body.)
Qiu [Page 6 Paragraph 1] (For example, the camera device of the live device can collect the video image of the real anchor, then, the real anchor of the video image contained in the capture, so as to obtain the real anchor posture information. after determining the attitude information, it can generate the corresponding driving signal, the driving signal for driving the video live picture display animation effect corresponding to the virtual presenter model.)
Examiner Note: Qiu shows that besides the head, other body parts can be used. In this case, the arms (limb) and upper half body part (trunk) correspond to other model parts that can be used. Limb detection is done on these parts as well. Limb detection result goes into posture detection result, which is used to determine the target animation effect which would be applied to the virtual presenter model (target animation model) to be displayed in video.
Claims 14-16 are rejected under 35 U.S.C. 103 as being unpatentable over Qiu et al. (CN 113487709 A, IDS Ref), hereinafter referred to as Qiu, in view of Drummond et al. (US 20210406543 A1), hereinafter referred to as Drummond, in further view of Iwamoto et al. (US 20200349765 A1), hereinafter referred to as Iwamoto.
Regarding Claim 14, Qiu further teaches
The method according to claim 1, wherein the determining a target video frame corresponding to the current image to be processed and playing the target video frame based on the target effect display parameters comprises:
Qiu [Page 6 Paragraph 1] (For example, the camera device of the live device can collect the video image of the real anchor, then, the real anchor of the video image contained in the capture, so as to obtain the real anchor posture information. after determining the attitude information, it can generate the corresponding driving signal, the driving signal for driving the video live picture display animation effect corresponding to the virtual presenter model.)
Examiner Note: Qiu teaches taking a resulting target video frame and displaying it in a video corresponding to the target object.
However, Qiu in view of Drummond teach fusing a target face image into a target animation model, but don’t teach fusing a target effect corresponding to the target effect display parameter for the target animation model.
Iwamoto teaches fusing a target effect corresponding to the target effect display parameter for the target animation model
Iwamoto [0030] “for example, enabling a camera, collecting an image, generating a 3D model, obtaining a skeletal model or animation, storing the skeletal model or the animation, adding a special effect, and performing an interaction operation with a user.”
Iwamoto [0069] “Fuse the target skeletal model and the 3D model of the target object.”
Iwamoto [0073] “FIG. 3 shows a main process from object scanning to animation implementation. First, an object is scanned, to obtain a depth map by using the depth camera and obtain a color image by using the color camera; fusion is performed on the depth map and the color image to obtain a textured meshing model, that is, a 3D model of the object; and the 3D model is embedded into a skeletal model to, to animate the skeletal model according to a skeleton animation (it should be understood that movement of skeletons is usually invisible, but certainly may be visible to a user under special scenario requirements), so as to visually present an animation effect of the object. The following provides detailed descriptions with reference to examples.”
Iwamoto [0146] In one embodiment, a movement manner of a first object may be obtained, and the movement manner of the first object is used as the target movement manner. The first object may be an object that currently moves in real time (for example, a person that is running is captured, and a skeletal movement manner of the object is extracted by using a neural network). Alternatively, the first object may be a movement manner that is of an object and that was captured and stored (for example, a set of lovely actions of a dog were captured, and by using an algorithm, a movement manner of the actions was locally stored or stored in a cloud). Alternatively, the first object may be a preset movement manner of a specific object (for example, only a human-related movement manner is selected).
Examiner Note: Iwamoto shows a person that is running is captured (target effect display parameters), and then has a skeletal movement manner (target effect) extracted from it. The 3D model is embedded into the skeletal model (fused) to animation the skeletal model (target animation model).
Qiu, Drummond, and Iwamoto are analogous in the art of processing image data from a camera to apply to 3D models. It would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to modify the combination of Qiu and Drummond‘s invention of playing a target video frame corresponding to the target object with Iwamoto’s fusing of an animation effect into a target animation model to achieve such target video frame to ensure a precise fit of the model to the target object.
Regarding Claim 15, Qiu further teaches
The method according to claim 14, wherein the target effect display parameters comprise a current limb parameter, the part parameters and the animation effect parameters to be fused corresponding to the event information of each limb and trunk model in the target animation model;
Qiu [Page 8 Paragraph 2] (further capable of performing gesture detection to the image located in the hand detection frame by the gesture recognition model, obtaining the gesture classification result,)
[Page 2 Paragraph 5] (the posture detection result comprises a limb detection result and/or gesture classification result,)
[Page 3 Paragraph 4] (based on the posture detection result, determining the first driving information for animation effect; wherein the first driving information is used for indicating animation jump information of animation effect of virtual live broadcast model displayed in the video live broadcast interface;)
Examiner Note: Qiu teaches animation jump information (animation effect parameters) which is indicated by first driving information (part parameters) which corresponds to the posture detection result (event information) that contain limb parameters.
However, Qiu and Drummond doesn’t teach the target effect comprises a limb and trunk display effect of the target animation model corresponding to the current limb parameter and the part parameters, and the superimposed animation effect corresponding to the animation effect parameter to be fused; the animation effect is matched with the limb and trunk model corresponding to the animation effect.
Iwamoto teaches the target effect comprises a limb and trunk display effect of the target animation model corresponding to the current limb parameter and the part parameters,
Iwamoto [0155] “The animation of the 3D model of the object is represented as that the 3D model rigged with skeletons is mapped into a set of changing actions of the skeletal model. For each frame, deformation needs to be implemented on the surface of the 3D model (that is, an epidermis of the 3D model of the object) based on a change of a skeleton. Such a process is referred to as skinning. Therefore, this implements mapping from the 3D model to actions, thereby achieving an animation effect.”
Examiner Note: The animation effect is mapped from the skeletal model, meaning it comprises the limb and trunk. An animation of the whole skeleton would involve the two parts. The changing actions define the intended animation effect.
and the superimposed animation effect corresponding to the animation effect parameter to be fused;
Iwamoto [0073] “FIG. 3 shows a main process from object scanning to animation implementation. First, an object is scanned, to obtain a depth map by using the depth camera and obtain a color image by using the color camera; fusion is performed on the depth map and the color image to obtain a textured meshing model, that is, a 3D model of the object; and the 3D model is embedded into a skeletal model to, to animate the skeletal model according to a skeleton animation (it should be understood that movement of skeletons is usually invisible, but certainly may be visible to a user under special scenario requirements), so as to visually present an animation effect of the object. The following provides detailed descriptions with reference to examples.”
Examiner Note: The animation effect is achieved through the 3D model being embedded into a skeletal model, and is based off the skeleton animation of the skeletal model (animation effect parameter).
the animation effect is matched with the limb and trunk model corresponding to the animation effect.
Iwamoto [0155] “The animation of the 3D model of the object is represented as that the 3D model rigged with skeletons is mapped into a set of changing actions of the skeletal model. For each frame, deformation needs to be implemented on the surface of the 3D model (that is, an epidermis of the 3D model of the object) based on a change of a skeleton. Such a process is referred to as skinning. Therefore, this implements mapping from the 3D model to actions, thereby achieving an animation effect.”
Examiner Note: Iwamoto teaches a skeletal model (skeletal model would cover all limbs, including trunk) that has an effect associated with it. The animation effect is mapped from the skeletal model, meaning it comprises and corresponds to a limb(s) and trunk.
Qiu, Drummond, and Iwamoto are analogous in the art of processing image data from a camera to apply to 3D models. It would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to modify the combination of Qiu and Drummond‘s invention containing parameters associated with limb and trunk information with Iwamoto’s superimposed animation effect containing limb and trunk information to allow for higher fidelity in modeling the human body, particularly by better defining complex limb and trunk movements in 3D spaces.
Regarding Claim 16, Qiu fails to teach the method according to claim 15, further comprising:
Drummond teaches in response to detecting that an actual display duration of a fused animation corresponding to the event information reaches a preset display duration threshold, a fusion percentage of the fused animation is adjusted to a set value.
Drummond [0111] In one or more embodiments, during the scan operation, the messaging client 104 is configured to display a scanning graphic (not shown) to indicate that the messaging client 104 is performing the scan operation. For example, the scanning graphic corresponds to an animation that is displayed for the duration of the scan (e.g., a predetermined duration of 2 seconds).
Examiner Note: When an animation reaches a preset display threshold here (example 2 seconds) the animation is no longer displayed after (functionally, the fusion percentage of the animation is set to 0%, i.e. not being displayed anymore)
Qiu and Drummond are analogous in the art of gathering image data and animating a 3D model based off such data. It would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to modify Qiu’s invention that displays an animation with Drummond’s display threshold and adjustment of visibility to ensure visual clarity when the intended animation duration is over.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DAVID W SOON whose telephone number is (571)272-8113. The examiner can normally be reached M-F 7:30-5:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Alicia Harrington can be reached at (571) 272-2330. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/DAVID W SOON/ Examiner, Art Unit 2615
/ALICIA M HARRINGTON/ Supervisory Patent Examiner, Art Unit 2615